From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113123573668.4363949622615; Wed, 30 Jun 2021 21:18:43 -0700 (PDT) Received: from localhost ([::1]:54586 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo9y-0003UP-Do for importer@patchew.org; Thu, 01 Jul 2021 00:18:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42266) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4x-0008D6-4u for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36265) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4u-0000cC-8U for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:30 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-521-LBRZg99OO1eia2Uh6xpHug-1; Thu, 01 Jul 2021 00:13:23 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DFEEE800D62; Thu, 1 Jul 2021 04:13:22 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id D9E0F69CB4; Thu, 1 Jul 2021 04:13:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112807; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uj0PdPRLjNl0txxyzM579hJXU5jCT0LzejFgMBerdOE=; b=OMpC2x3ri/g/X45O7tBzP6RQ4BSzwKrw8dekn42B4FL4rLczyF2pE0kK6NuTnZbKDcqf5v eoapJDbnbzz0Ub4du0zPPpPoH2C3KtCpLACtJrI3ck1ZJzaTE25MAow2tE/E2XHoGcGW1j cfpQz5L4hlfdq/jhBELIcWjBDAEKjLo= X-MC-Unique: LBRZg99OO1eia2Uh6xpHug-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 01/20] python/pylint: Add exception for TypeVar names ('T') Date: Thu, 1 Jul 2021 00:12:54 -0400 Message-Id: <20210701041313.1696009-2-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" 'T' is a common TypeVar name, allow its use. See also https://github.com/PyCQA/pylint/issues/3401 -- In the future, we might be able to have a separate list of acceptable names for TypeVars exclusively. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/python/setup.cfg b/python/setup.cfg index 11f71d5312..cfbe17f0f6 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -100,6 +100,7 @@ good-names=3Di, fh, # fh =3D open(...) fd, # fd =3D os.open(...) c, # for c in string: ... + T, # for TypeVars. See pylint#3401 =20 [pylint.similarities] # Ignore imports when computing similarities. --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113010348849.706047945752; Wed, 30 Jun 2021 21:16:50 -0700 (PDT) Received: from localhost ([::1]:48374 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo89-0007nD-BQ for importer@patchew.org; Thu, 01 Jul 2021 00:16:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42276) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4y-0008Dy-1d for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:32 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:55751) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4v-0000cn-Cr for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:31 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-219-mUeSmgnpN2yL8wu5IN3zFw-1; Thu, 01 Jul 2021 00:13:25 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 03C501084F55; Thu, 1 Jul 2021 04:13:24 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0F1D669CB4; Thu, 1 Jul 2021 04:13:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112808; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jn62E4Ym9t68vPkoDgsDHqj7RMg1oFosYCUb+MArnxg=; b=hQ0HYhkxg7xQKRKEa6aHKguXN39v94Aoazu1vCN0iLqsiOkBb+RCQ8MKWEgZK21W5lQVAM G7hBIY5E3jh4YnLygJLNnDsanDmegWGr6v17YYswsvMbPefpCANfPcdP2RJZ/1ifght2yW 3S4zodeorF2klGkHhz8+2DY35R+MMOk= X-MC-Unique: mUeSmgnpN2yL8wu5IN3zFw-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 02/20] python/pylint: disable too-many-function-args Date: Thu, 1 Jul 2021 00:12:55 -0400 Message-Id: <20210701041313.1696009-3-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" too-many-function-args seems prone to failure when considering things like Method Resolution Order, which mypy gets correct. When dealing with multiple inheritance, pylint doesn't seem to understand which method will actually get called, while mypy does. Remove the less powerful, redundant check. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/setup.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/setup.cfg b/python/setup.cfg index cfbe17f0f6..e1c48eb706 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -87,7 +87,7 @@ ignore_missing_imports =3D True # --enable=3Dsimilarities". If you want to run only the classes checker, b= ut have # no Warning level messages displayed, use "--disable=3Dall --enable=3Dcla= sses # --disable=3DW". -disable=3D +disable=3Dtoo-many-function-args, # mypy handles this with less false pos= itives. =20 [pylint.basic] # Good variable names which should always be accepted, separated by a comm= a. --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113007542530.3034245890627; Wed, 30 Jun 2021 21:16:47 -0700 (PDT) Received: from localhost ([::1]:48208 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo86-0007gi-EH for importer@patchew.org; Thu, 01 Jul 2021 00:16:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42262) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4w-0008Cu-No for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:30 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22463) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4u-0000c7-3g for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:30 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-75-aeXe_Z6cO4GxfDDWLh136w-1; Thu, 01 Jul 2021 00:13:26 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 364EB80006E; Thu, 1 Jul 2021 04:13:25 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 262B469CB4; Thu, 1 Jul 2021 04:13:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112807; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3L1xpL9KIP3gebZ0aGqkBptjXDd0bvSXacnWJ+RLWiQ=; b=InZIAR4umZU/9bbbiioIHl9Rvafmk0vaVPc9QuAKilE4b6qCYya7SDxL9HSe3yjF6aFeuc axgmAOKgPwxKIucWYFkXggjX3rpW9eXkCYnz/7M1iD/Ny5NkljXG2Myabd6dtDYi2wqcJC jjTaRsmmUN7c/keW+gjea14mGlXDeWM= X-MC-Unique: aeXe_Z6cO4GxfDDWLh136w-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 03/20] python/aqmp: add asynchronous QMP (AQMP) subpackage Date: Thu, 1 Jul 2021 00:12:56 -0400 Message-Id: <20210701041313.1696009-4-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" For now, it's empty! Soon, it won't be. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 27 +++++++++++++++++++++++++++ python/qemu/aqmp/py.typed | 0 python/setup.cfg | 1 + 3 files changed, 28 insertions(+) create mode 100644 python/qemu/aqmp/__init__.py create mode 100644 python/qemu/aqmp/py.typed diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py new file mode 100644 index 0000000000..4c713b3ccf --- /dev/null +++ b/python/qemu/aqmp/__init__.py @@ -0,0 +1,27 @@ +""" +QEMU Monitor Protocol (QMP) development library & tooling. + +This package provides a fairly low-level class for communicating +asynchronously with QMP protocol servers, as implemented by QEMU, the +QEMU Guest Agent, and the QEMU Storage Daemon. + +:py:class:`~qmp_protocol.QMP` provides the main functionality of this +package. All errors raised by this library dervive from `AQMPError`, see +`aqmp.error` for additional detail. See `aqmp.events` for an in-depth +tutorial on managing QMP events. +""" + +# Copyright (C) 2020, 2021 John Snow for Red Hat, Inc. +# +# Authors: +# John Snow +# +# Based on earlier work by Luiz Capitulino . +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory. + + +# The order of these fields impact the Sphinx documentation order. +__all__ =3D ( +) diff --git a/python/qemu/aqmp/py.typed b/python/qemu/aqmp/py.typed new file mode 100644 index 0000000000..e69de29bb2 diff --git a/python/setup.cfg b/python/setup.cfg index e1c48eb706..bce8807702 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -27,6 +27,7 @@ packages =3D qemu.qmp qemu.machine qemu.utils + qemu.aqmp =20 [options.package_data] * =3D py.typed --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625112883435960.5146009209174; Wed, 30 Jun 2021 21:14:43 -0700 (PDT) Received: from localhost ([::1]:40736 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo66-0002bl-4m for importer@patchew.org; Thu, 01 Jul 2021 00:14:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4z-0008Gz-61 for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:20206) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4u-0000co-Ry for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:32 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-506-xtBaVglIOHGtS0_UN-ZfMQ-1; Thu, 01 Jul 2021 00:13:27 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 39EE2801F97; Thu, 1 Jul 2021 04:13:26 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E88069CB4; Thu, 1 Jul 2021 04:13:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112808; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u37O0ssgS4A0aLWMkEpMNTyBmETh8DDdyDuOTHgvjtg=; b=ds+KH2PasmcHi4gT/7/PGGo8E6cYR48V4qToUhfveNrm+OfQZCNiDB9Y1cTKrDvdqKQVJh y4cmTJbDDL+tzSnQd18rK1YQCgyDQkvDw5ks4cWi0dchbdRURN3z2A4sDx4jjz9JGQKSUi 28NxP+BD7rSazio6HXL62Hlbf2U4hBU= X-MC-Unique: xtBaVglIOHGtS0_UN-ZfMQ-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 04/20] python/aqmp: add error classes Date: Thu, 1 Jul 2021 00:12:57 -0400 Message-Id: <20210701041313.1696009-5-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 7 +++ python/qemu/aqmp/error.py | 97 ++++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 python/qemu/aqmp/error.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 4c713b3ccf..8e955d784d 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -21,7 +21,14 @@ # This work is licensed under the terms of the GNU GPL, version 2. See # the COPYING file in the top-level directory. =20 +from .error import AQMPError, MultiException + =20 # The order of these fields impact the Sphinx documentation order. __all__ =3D ( + # Exceptions + 'AQMPError', + + # Niche topics + 'MultiException', ) diff --git a/python/qemu/aqmp/error.py b/python/qemu/aqmp/error.py new file mode 100644 index 0000000000..126f77bb5c --- /dev/null +++ b/python/qemu/aqmp/error.py @@ -0,0 +1,97 @@ +""" +AQMP Error Classes + +This package seeks to provide semantic error classes that are intended +to be used directly by clients when they would like to handle particular +semantic failures (e.g. "failed to connect") without needing to know the +enumeration of possible reasons for that failure. + +AQMPError serves as the ancestor for *almost* all exceptions raised by +this package, and is suitable for use in handling semantic errors from +this library. In most cases, individual public methods will attempt to +catch and re-encapsulate various exceptions to provide a semantic +error-handling interface. + +.. caution:: + + The only exception that is not an `AQMPError` is + `MultiException`. It is special, and used to encapsulate one-or-more + exceptions of an arbitrary kind; this exception MAY be raised on + `disconnect()` when there are two or more exceptions from the AQMP + event loop to report back to the caller. + + Every pain has been taken to prevent this circumstance but in + certain cases these exceptions may occasionally be (unfortunately) + visible. See `MultiException` and `AsyncProtocol.disconnect()` for + more details. + + +.. admonition:: AQMP Exception Hierarchy Reference + + | `Exception` + | +-- `MultiException` + | +-- `AQMPError` + | +-- `ConnectError` + | +-- `StateError` + | +-- `ExecInterruptedError` + | +-- `ExecuteError` + | +-- `ListenerError` + | +-- `ProtocolError` + | +-- `DeserializationError` + | +-- `UnexpectedTypeError` + | +-- `ServerParseError` + | +-- `BadReplyError` + | +-- `GreetingError` + | +-- `NegotiationError` +""" + +from typing import Iterable, Iterator, List + + +class AQMPError(Exception): + """Abstract error class for all errors originating from this package."= "" + + +class ProtocolError(AQMPError): + """ + Abstract error class for protocol failures. + + Semantically, these errors are generally the fault of either the + protocol server or as a result of a bug in this this library. + + :param error_message: Human-readable string describing the error. + """ + def __init__(self, error_message: str): + super().__init__(error_message) + #: Human-readable error message, without any prefix. + self.error_message: str =3D error_message + + +class MultiException(Exception): + """ + Used for multiplexing exceptions. + + This exception is used in the case that errors were encountered in bot= h the + Reader and Writer tasks, and we must raise more than one. + + PEP 0654 seeks to remedy this clunky infrastructure, but it will not be + available for quite some time -- possibly Python 3.11 or even later. + + :param exceptions: An iterable of `BaseException` objects. + """ + def __init__(self, exceptions: Iterable[BaseException]): + super().__init__(exceptions) + self._exceptions: List[BaseException] =3D list(exceptions) + + def __str__(self) -> str: + ret =3D "------------------------------\n" + ret +=3D "Multiple Exceptions occurred:\n" + ret +=3D "\n" + for i, exc in enumerate(self._exceptions): + ret +=3D f"{i}) {str(exc)}\n" + ret +=3D "\n" + ret +=3D "-----------------------------\n" + return ret + + def __iter__(self) -> Iterator[BaseException]: + return iter(self._exceptions) --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625112888437835.4563285591257; Wed, 30 Jun 2021 21:14:48 -0700 (PDT) Received: from localhost ([::1]:40844 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo6B-0002gO-9O for importer@patchew.org; Thu, 01 Jul 2021 00:14:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42296) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo50-0008If-55 for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:34 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:34551) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4w-0000di-De for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:33 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-535-lXzhjoWcOg-VieWWaFshGA-1; Thu, 01 Jul 2021 00:13:28 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 83A5718414A0; Thu, 1 Jul 2021 04:13:27 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7774469CB4; Thu, 1 Jul 2021 04:13:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=seoFGKa3kTLAoNY38uU4GpNn7IHY7I0UEY0RjfUZw7w=; b=C7xqFEth81CaCYOJ0M/Fj9Ez6QaRMDKLAnZwR9TjVlx0JW2HE06oHI0jfI92wnae/Achw2 QxadSYXy8zzxU1MqWTsfBAsDUEGZKNtB58nfGUAocZw4O7LIsCZwfLQ4hqUQoE4qcageXO +5/sXiRCxvXQgu6EYrFoseyZxiV+qlw= X-MC-Unique: lXzhjoWcOg-VieWWaFshGA-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 05/20] python/aqmp: add asyncio compatibility wrappers Date: Thu, 1 Jul 2021 00:12:58 -0400 Message-Id: <20210701041313.1696009-6-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Python 3.6 does not have all of the goodies that Python 3.7 does, and I need to support both. Add some compatibility wrappers needed for this purpose. (Note: Python 3.6 is EOL December 2021.) Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/util.py | 77 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 python/qemu/aqmp/util.py diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py new file mode 100644 index 0000000000..c88a2201bc --- /dev/null +++ b/python/qemu/aqmp/util.py @@ -0,0 +1,77 @@ +""" +Miscellaneous Utilities + +This module primarily provides compatibility wrappers for Python 3.6 to +provide some features that otherwise become available in Python 3.7+. +""" + +import asyncio +import sys +from typing import ( + Any, + Coroutine, + Optional, + TypeVar, +) + + +T =3D TypeVar('T') + + +def create_task(coro: Coroutine[Any, Any, T], + loop: Optional[asyncio.AbstractEventLoop] =3D None + ) -> 'asyncio.Future[T]': + """ + Python 3.6-compatible `asyncio.create_task` wrapper. + + :param coro: The coroutine to execute in a task. + :param loop: Optionally, the loop to create the task in. + + :return: An `asyncio.Future` object. + """ + # Python 3.7+: + if sys.version_info >=3D (3, 7): + # pylint: disable=3Dno-member + if loop is not None: + return loop.create_task(coro) + return asyncio.create_task(coro) + + # Python 3.6: + return asyncio.ensure_future(coro, loop=3Dloop) + + +def is_closing(writer: asyncio.StreamWriter) -> bool: + """ + Python 3.6-compatible `asyncio.StreamWriter.is_closing` wrapper. + + :param writer: The `asyncio.StreamWriter` object. + :return: `True` if the writer is closing, or closed. + """ + if hasattr(writer, 'is_closing'): + # Python 3.7+ + return writer.is_closing() # type: ignore + + # Python 3.6: + transport =3D writer.transport + assert isinstance(transport, asyncio.WriteTransport) + return transport.is_closing() + + +async def wait_closed(writer: asyncio.StreamWriter) -> None: + """ + Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper. + + :param writer: The `asyncio.StreamWriter` to wait on. + """ + if hasattr(writer, 'wait_closed'): + # Python 3.7+ + await writer.wait_closed() # type: ignore + else: + # Python 3.6 + transport =3D writer.transport + assert isinstance(transport, asyncio.WriteTransport) + + while not transport.is_closing(): + await asyncio.sleep(0.0) + while transport.get_write_buffer_size() > 0: + await asyncio.sleep(0.0) --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113013735925.2104539635427; Wed, 30 Jun 2021 21:16:53 -0700 (PDT) Received: from localhost ([::1]:48622 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo8C-0007x8-Ll for importer@patchew.org; Thu, 01 Jul 2021 00:16:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo51-0008Kw-2O for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:35 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:45232) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4x-0000f0-Ki for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:34 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-34-0rudA-iVMV6ZdAyXt0xZ4g-1; Thu, 01 Jul 2021 00:13:29 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id ADEB8100C611; Thu, 1 Jul 2021 04:13:28 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id A5FCE69CB4; Thu, 1 Jul 2021 04:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112810; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dG3x9fQwpD05aHBNaJ/XnfHOuCgPvln6v/gI+g7EU4E=; b=X88EqzUQETVyh6hLTvGqtS4Ne8ubzGkgjhIJoSSa9cVTpN5I2J2+Gl/1Pw6ptzYbqBzQ3k edSVxWzgNT54iSL2LZZQC4zraTUTyn6NpMjVMGty8Mwt+QIqbp983PvGhyoDiHu/Qei7J2 xEiOtyYlyy9u1v/NGC5CwH+dtht6Y2g= X-MC-Unique: 0rudA-iVMV6ZdAyXt0xZ4g-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 06/20] python/aqmp: add generic async message-based protocol support Date: Thu, 1 Jul 2021 00:12:59 -0400 Message-Id: <20210701041313.1696009-7-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This is the bare minimum that you need to establish a full-duplex async message-based protocol with Python's asyncio. The features to be added in forthcoming commits are: - Runstate tracking - Logging - Support for incoming connections via accept() - _cb_outbound, _cb_inbound message hooks - _readline() method Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- A note for reviewers: If you believe that it is unsafe to call certain methods at certain times, you're absolutely correct! These interfaces are protected in the following commit. Some of the docstrings have dangling references, but they will resolve themselves within the next few commits. Forgive me for not wanting to rewrite them ... ! Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 4 +- python/qemu/aqmp/protocol.py | 523 +++++++++++++++++++++++++++++++++++ python/qemu/aqmp/util.py | 54 ++++ 3 files changed, 580 insertions(+), 1 deletion(-) create mode 100644 python/qemu/aqmp/protocol.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 8e955d784d..e003c898bd 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,12 +22,14 @@ # the COPYING file in the top-level directory. =20 from .error import AQMPError, MultiException +from .protocol import ConnectError =20 =20 # The order of these fields impact the Sphinx documentation order. __all__ =3D ( - # Exceptions + # Exceptions, most generic to most explicit 'AQMPError', + 'ConnectError', =20 # Niche topics 'MultiException', diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py new file mode 100644 index 0000000000..beb7e12d9c --- /dev/null +++ b/python/qemu/aqmp/protocol.py @@ -0,0 +1,523 @@ +""" +Generic Asynchronous Message-based Protocol Support + +This module provides a generic framework for sending and receiving +messages over an asyncio stream. `AsyncProtocol` is an abstract class +that implements the core mechanisms of a simple send/receive protocol, +and is designed to be extended. + +In this package, it is used as the implementation for the +:py:class:`~qmp_protocol.QMP` class. +""" + +import asyncio +from asyncio import StreamReader, StreamWriter +from ssl import SSLContext +# import exceptions will be removed in a forthcoming commit. +# The problem stems from pylint/flake8 believing that 'Any' +# is unused because of its only use in a string-quoted type. +from typing import ( # pylint: disable=3Dunused-import # noqa + Any, + Awaitable, + Callable, + Generic, + List, + Optional, + Tuple, + TypeVar, + Union, +) + +from .error import AQMPError, MultiException +from .util import ( + bottom_half, + create_task, + flush, + is_closing, + upper_half, + wait_closed, + wait_task_done, +) + + +T =3D TypeVar('T') +_TaskFN =3D Callable[[], Awaitable[None]] # aka ``async def func() -> Non= e`` +_FutureT =3D TypeVar('_FutureT', bound=3DOptional['asyncio.Future[Any]']) + + +class ConnectError(AQMPError): + """ + Raised when the initial connection process has failed. + + This Exception always wraps a "root cause" exception that can be + interrogated for additional information. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + def __init__(self, error_message: str, exc: Exception): + super().__init__(error_message) + #: Human-readable error string + self.error_message: str =3D error_message + #: Wrapped root cause exception + self.exc: Exception =3D exc + + def __str__(self) -> str: + return f"{self.error_message}: {self.exc!s}" + + +class AsyncProtocol(Generic[T]): + """ + AsyncProtocol implements a generic async message-based protocol. + + This protocol assumes the basic unit of information transfer between + client and server is a "message", the details of which are left up + to the implementation. It assumes the sending and receiving of these + messages is full-duplex and not necessarily correlated; i.e. it + supports asynchronous inbound messages. + + It is designed to be extended by a specific protocol which provides + the implementations for how to read and send messages. These must be + defined in `_do_recv()` and `_do_send()`, respectively. + + Other callbacks have a default implementation, but are intended to be + either extended or overridden: + + - `_begin_new_session`: + The base implementation starts the reader/writer tasks. + A protocol implementation can override this call, inserting + actions to be taken prior to starting the reader/writer tasks + before the super() call; actions needing to occur afterwards + can be written after the super() call. + - `_on_message`: + Actions to be performed when a message is received. + """ + # pylint: disable=3Dtoo-many-instance-attributes + + # ------------------------- + # Section: Public interface + # ------------------------- + + def __init__(self) -> None: + # stream I/O + self._reader: Optional[StreamReader] =3D None + self._writer: Optional[StreamWriter] =3D None + + # Outbound Message queue + self._outgoing: asyncio.Queue[T] =3D asyncio.Queue() + + # Special, long-running tasks: + self._reader_task: Optional[asyncio.Future[None]] =3D None + self._writer_task: Optional[asyncio.Future[None]] =3D None + + # Aggregate of the above tasks; this is useful for Exception + # aggregation when leaving the loop and ensuring that all tasks + # are done. + self._bh_tasks: Optional[asyncio.Future[Tuple[ + Optional[BaseException], + Optional[BaseException], + ]]] =3D None + + #: Disconnect task. The disconnect implementation runs in a task + #: so that asynchronous disconnects (initiated by the + #: reader/writer) are allowed to wait for the reader/writers to + #: exit. + self._dc_task: Optional[asyncio.Future[None]] =3D None + + @upper_half + async def connect(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] =3D None) -> None: + """ + Connect to the server and begin processing message queues. + + If this call fails, `runstate` is guaranteed to be set back to `ID= LE`. + + :param address: + Address to connect to; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise StateError: When the `Runstate` is not `IDLE`. + :raise ConnectError: If a connection cannot be made to the server. + """ + await self._new_session(address, ssl) + + @upper_half + async def disconnect(self, force: bool =3D False) -> None: + """ + Disconnect and wait for all tasks to fully stop. + + If there were exceptions that caused the reader/writers to termina= te + prematurely, they will be raised here. + + :param force: + When `False`, drain the outgoing message queue first before + terminating execution. When `True`, terminate immediately. + + :raise Exception: When the reader or writer terminate unexpectedly. + :raise MultiException: + When both the reader and writer terminate unexpectedly. This + exception is iterable and multiplexes both root cause exceptio= ns. + """ + self._schedule_disconnect(force) + await self._wait_disconnect() + + # -------------------------- + # Section: Session machinery + # -------------------------- + + @upper_half + async def _new_session(self, + address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] =3D None) -> None: + """ + Establish a new connection and initialize the session. + + Connect or accept a new connection, then begin the protocol + session machinery. If this call fails, `runstate` is guaranteed + to be set back to `IDLE`. + + :param address: + Address to connect to; + UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise ConnectError: + When a connection or session cannot be established. + + This exception will wrap a more concrete one. In most cases, + the wrapped exception will be `OSError` or `EOFError`. If a + protocol-level failure occurs while establishing a new + session, the wrapped error may also be an `AQMPError`. + """ + self._outgoing =3D asyncio.Queue() + + phase =3D "connection" + try: + await self._do_connect(address, ssl) + + phase =3D "session" + await self._begin_new_session() + + except Exception as err: + # Reset from CONNECTING back to IDLE. + await self.disconnect() + emsg =3D f"Failed to establish {phase}" + raise ConnectError(emsg, err) from err + + @upper_half + async def _do_connect(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] =3D None) -> None: + """ + Acting as the transport client, initiate a connection to a server. + + :param address: + Address to connect to; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise OSError: For stream-related errors. + """ + if isinstance(address, tuple): + connect =3D asyncio.open_connection(address[0], address[1], ss= l=3Dssl) + else: + connect =3D asyncio.open_unix_connection(path=3Daddress, ssl= =3Dssl) + self._reader, self._writer =3D await connect + + @upper_half + async def _begin_new_session(self) -> None: + """ + After a connection is established, start the bottom half machinery. + """ + reader_coro =3D self._bh_loop_forever(self._bh_recv_message) + writer_coro =3D self._bh_loop_forever(self._bh_send_message) + + self._reader_task =3D create_task(reader_coro) + self._writer_task =3D create_task(writer_coro) + + self._bh_tasks =3D asyncio.gather( + self._reader_task, + self._writer_task, + return_exceptions=3DTrue, + ) + + @upper_half + @bottom_half + def _schedule_disconnect(self, force: bool =3D False) -> None: + """ + Initiate a disconnect; idempotent. + + This method is used both in the upper-half as a direct + consequence of `disconnect()`, and in the bottom-half in the + case of unhandled exceptions in the reader/writer tasks. + + It can be invoked no matter what the `runstate` is. + + :param force: + When `False`, drain the outgoing message queue first before + terminating execution. When `True`, terminate immediately. + """ + if not self._dc_task: + self._dc_task =3D create_task(self._bh_disconnect(force)) + + @upper_half + def _results(self) -> None: + """ + Raises exception(s) from the finished tasks, if any. + + Called to fully quiesce the reader/writer. In the event of an + intentional cancellation by the user that completes gracefully, + this method will not raise any exceptions. + + In the event that both the reader and writer task should fail, + a `MultiException` will be raised containing both exceptions. + + :raise Exception: + Arbitrary exceptions re-raised on behalf of a failing Task. + :raise MultiException: + Iterable Exception used to multiplex multiple exceptions in the + event that multiple Tasks failed with non-cancellation reasons. + """ + exceptions: List[BaseException] =3D [] + + assert self._bh_tasks is None or self._bh_tasks.done() + results =3D self._bh_tasks.result() if self._bh_tasks else () + + for result in results: + if result is None: + continue + exceptions.append(result) + + if len(exceptions) =3D=3D 1: + raise exceptions.pop() + if len(exceptions) > 1: + # This is possible in theory, but I am not sure if it can + # occur in practice. The reader could suffer an Exception + # and then immediately schedule a disconnect. That + # disconnect is not guaranteed to run immediately, so the + # writer could be scheduled immediately afterwards instead + # of the disconnect task, and encounter another exception. + # + # An improved solution may be to raise the *first* exception + # which occurs, leaving the *second* exception to be merely + # logged. Still, what if the caller wants to interrogate + # that second exception? + raise MultiException(exceptions) + + @upper_half + async def _wait_disconnect(self) -> None: + """ + Waits for a previously scheduled disconnect to finish. + + This method will gather any bottom half exceptions and + re-raise them; so it is intended to be used in the upper half + call chain. + + If a single exception is encountered, it will be re-raised + faithfully. If multiple are found, they will be multiplexed + into a `MultiException`. + + :raise Exception: + Arbitrary exceptions re-raised on behalf of a failing Task. + :raise MultiException: + Iterable Exception used to multiplex multiple exceptions in the + event that multiple Tasks failed with non-cancellation reasons. + """ + assert self._dc_task + await self._dc_task + self._dc_task =3D None + + try: + self._results() # Raises BH errors here. + finally: + self._cleanup() + + @upper_half + def _cleanup(self) -> None: + """ + Fully reset this object to a clean state and return to `IDLE`. + """ + def _paranoid_task_erase(task: _FutureT) -> Optional[_FutureT]: + # Help to erase a task, ENSURING it is fully quiesced first. + assert (task is None) or task.done() + return None if (task and task.done()) else task + + self._dc_task =3D _paranoid_task_erase(self._dc_task) + self._reader_task =3D _paranoid_task_erase(self._reader_task) + self._writer_task =3D _paranoid_task_erase(self._writer_task) + self._bh_tasks =3D _paranoid_task_erase(self._bh_tasks) + + self._reader =3D None + self._writer =3D None + + # ---------------------------- + # Section: Bottom Half methods + # ---------------------------- + + @bottom_half + async def _bh_disconnect(self, force: bool =3D False) -> None: + """ + Disconnect and cancel all outstanding tasks. + + It is designed to be called from its task context, + :py:obj:`~AsyncProtocol._dc_task`. By running in its own task, + it is free to wait on any pending actions that may still need to + occur in either the reader or writer tasks. + + :param force: + When `False`, drain the outgoing message queue first before + terminating execution. When `True`, terminate immediately. + + """ + await self._bh_stop_writer(force) + await self._bh_stop_reader() + + # Next, close the writer stream itself. + # This implicitly closes the reader, too. + if self._writer: + if not is_closing(self._writer): + self._writer.close() + await wait_closed(self._writer) + + @bottom_half + async def _bh_stop_writer(self, force: bool =3D False) -> None: + # If we're not in a hurry, drain the outbound queue. + if self._writer_task and not self._writer_task.done(): + if not force: + await self._outgoing.join() + assert self._writer is not None + await flush(self._writer) + + # Cancel the writer task. + if self._writer_task and not self._writer_task.done(): + self._writer_task.cancel() + await wait_task_done(self._writer_task) + + @bottom_half + async def _bh_stop_reader(self) -> None: + if self._reader_task and not self._reader_task.done(): + self._reader_task.cancel() + await wait_task_done(self._reader_task) + + @bottom_half + async def _bh_loop_forever(self, async_fn: _TaskFN) -> None: + """ + Run one of the bottom-half methods in a loop forever. + + If the bottom half ever raises any exception, schedule a + disconnect that will terminate the entire loop. + + :param async_fn: The bottom-half method to run in a loop. + """ + try: + while True: + await async_fn() + except asyncio.CancelledError: + # We have been cancelled by _bh_disconnect, exit gracefully. + return + except BaseException: + self._schedule_disconnect(force=3DTrue) + raise + + @bottom_half + async def _bh_send_message(self) -> None: + """ + Wait for an outgoing message, then send it. + + Designed to be run in `_bh_loop_forever()`. + """ + msg =3D await self._outgoing.get() + try: + await self._send(msg) + finally: + self._outgoing.task_done() + + @bottom_half + async def _bh_recv_message(self) -> None: + """ + Wait for an incoming message and call `_on_message` to route it. + + Designed to be run in `_bh_loop_forever()`. + """ + msg =3D await self._recv() + await self._on_message(msg) + + # -------------------- + # Section: Message I/O + # -------------------- + + @upper_half + @bottom_half + async def _do_recv(self) -> T: + """ + Abstract: Read from the stream and return a message. + + Very low-level; intended to only be called by `_recv()`. + """ + raise NotImplementedError + + @upper_half + @bottom_half + async def _recv(self) -> T: + """ + Read an arbitrary protocol message. + + .. warning:: + This method is intended primarily for `_bh_recv_message()` + to use in an asynchronous task loop. Using it outside of + this loop will "steal" messages from the normal routing + mechanism. It is safe to use prior to `_begin_new_session()`, + but should not be used otherwise. + + This method uses `_do_recv()` to retrieve the raw message, and + then transforms it using `_cb_inbound()`. + + :return: A single (filtered, processed) protocol message. + """ + # A forthcoming commit makes this method less trivial. + return await self._do_recv() + + @upper_half + @bottom_half + def _do_send(self, msg: T) -> None: + """ + Abstract: Write a message to the stream. + + Very low-level; intended to only be called by `_send()`. + """ + raise NotImplementedError + + @upper_half + @bottom_half + async def _send(self, msg: T) -> None: + """ + Send an arbitrary protocol message. + + This method will transform any outgoing messages according to + `_cb_outbound()`. + + .. warning:: + Like `_recv()`, this method is intended to be called by + the writer task loop that processes outgoing + messages. Calling it directly may circumvent logic + implemented by the caller meant to correlate outgoing and + incoming messages. + + :raise OSError: For problems with the underlying stream. + """ + # A forthcoming commit makes this method less trivial. + self._do_send(msg) + + @bottom_half + async def _on_message(self, msg: T) -> None: + """ + Called to handle the receipt of a new message. + + .. caution:: + This is executed from within the reader loop, so be advised + that waiting on either the reader or writer task will lead + to deadlock. Additionally, any unhandled exceptions will + directly cause the loop to halt, so logic may be best-kept + to a minimum if at all possible. + + :param msg: The incoming message + """ + # Nothing to do in the abstract case. diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py index c88a2201bc..9ea91f2862 100644 --- a/python/qemu/aqmp/util.py +++ b/python/qemu/aqmp/util.py @@ -12,6 +12,7 @@ Coroutine, Optional, TypeVar, + cast, ) =20 =20 @@ -57,6 +58,20 @@ def is_closing(writer: asyncio.StreamWriter) -> bool: return transport.is_closing() =20 =20 +async def flush(writer: asyncio.StreamWriter) -> None: + """ + Utility function to ensure a StreamWriter is *fully* drained. + + `asyncio.StreamWriter.drain` only promises we will return to below + the "high-water mark". This function ensures we flush the entire + buffer. + """ + transport =3D cast(asyncio.WriteTransport, writer.transport) + + while transport.get_write_buffer_size() > 0: + await writer.drain() + + async def wait_closed(writer: asyncio.StreamWriter) -> None: """ Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper. @@ -75,3 +90,42 @@ async def wait_closed(writer: asyncio.StreamWriter) -> N= one: await asyncio.sleep(0.0) while transport.get_write_buffer_size() > 0: await asyncio.sleep(0.0) + + +async def wait_task_done(task: Optional['asyncio.Future[Any]']) -> None: + """ + Await a task to finish, but do not consume its result. + + :param task: The task to await completion for. + """ + while True: + if task and not task.done(): + await asyncio.sleep(0) # Yield + else: + break + + +def upper_half(func: T) -> T: + """ + Do-nothing decorator that annotates a method as an "upper-half" method. + + These methods must not call bottom-half functions directly, but can + schedule them to run. + """ + return func + + +def bottom_half(func: T) -> T: + """ + Do-nothing decorator that annotates a method as a "bottom-half" method. + + These methods must take great care to handle their own exceptions when= ever + possible. If they go unhandled, they will cause termination of the loo= p. + + These methods do not, in general, have the ability to directly + report information to a caller's context and will usually be + collected as a Task result instead. + + They must not call upper-half functions directly. + """ + return func --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113215081673.8309823087865; Wed, 30 Jun 2021 21:20:15 -0700 (PDT) Received: from localhost ([::1]:32850 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoBR-0007oe-Er for importer@patchew.org; Thu, 01 Jul 2021 00:20:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42304) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo51-0008L8-44 for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:35 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:38849) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4y-0000gO-Tp for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:34 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-587-HsMc9ZOBP3C4u0uODSJWNQ-1; Thu, 01 Jul 2021 00:13:31 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 30D4C800D62; Thu, 1 Jul 2021 04:13:30 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id D2E52604CC; Thu, 1 Jul 2021 04:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112812; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XKh+OOhOkA7eclMe7X5kXZhj9ThrlHpOSJDa0G90vWQ=; b=XRnpvhc9BCCf5xAymV4+I39udHg2nnKZHHs/sp4A8+k+xm++vqQUq6IeurvrYbkwHWcQPw Mn/iKRE9iRS8RqdSWQQdW83JHuMMVY0u9Hq8UL/wxaB48Je7SLZa/5IW4WQTd/VPOHYWcH 3mFraey9CRB6E+DjNSkgfMjInlh9Q+w= X-MC-Unique: HsMc9ZOBP3C4u0uODSJWNQ-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 07/20] python/aqmp: add runstate state machine to AsyncProtocol Date: Thu, 1 Jul 2021 00:13:00 -0400 Message-Id: <20210701041313.1696009-8-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This serves a few purposes: 1. Protect interfaces when it's not safe to call them (via @require) 2. Add an interface by which an async client can determine if the state has changed, for the purposes of connection management. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 5 +- python/qemu/aqmp/protocol.py | 133 +++++++++++++++++++++++++++++++++-- 2 files changed, 133 insertions(+), 5 deletions(-) diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index e003c898bd..5c44fabeea 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,11 +22,14 @@ # the COPYING file in the top-level directory. =20 from .error import AQMPError, MultiException -from .protocol import ConnectError +from .protocol import ConnectError, Runstate =20 =20 # The order of these fields impact the Sphinx documentation order. __all__ =3D ( + # Classes + 'Runstate', + # Exceptions, most generic to most explicit 'AQMPError', 'ConnectError', diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index beb7e12d9c..a99a191982 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -12,11 +12,10 @@ =20 import asyncio from asyncio import StreamReader, StreamWriter +from enum import Enum +from functools import wraps from ssl import SSLContext -# import exceptions will be removed in a forthcoming commit. -# The problem stems from pylint/flake8 believing that 'Any' -# is unused because of its only use in a string-quoted type. -from typing import ( # pylint: disable=3Dunused-import # noqa +from typing import ( Any, Awaitable, Callable, @@ -26,6 +25,7 @@ Tuple, TypeVar, Union, + cast, ) =20 from .error import AQMPError, MultiException @@ -45,6 +45,20 @@ _FutureT =3D TypeVar('_FutureT', bound=3DOptional['asyncio.Future[Any]']) =20 =20 +class Runstate(Enum): + """Protocol session runstate.""" + + #: Fully quiesced and disconnected. + IDLE =3D 0 + #: In the process of connecting or establishing a session. + CONNECTING =3D 1 + #: Fully connected and active session. + RUNNING =3D 2 + #: In the process of disconnecting. + #: Runstate may be returned to `IDLE` by calling `disconnect()`. + DISCONNECTING =3D 3 + + class ConnectError(AQMPError): """ Raised when the initial connection process has failed. @@ -66,6 +80,75 @@ def __str__(self) -> str: return f"{self.error_message}: {self.exc!s}" =20 =20 +class StateError(AQMPError): + """ + An API command (connect, execute, etc) was issued at an inappropriate = time. + + This error is raised when a command like + :py:meth:`~AsyncProtocol.connect()` is issued at an inappropriate + time. + + :param error_message: Human-readable string describing the state viola= tion. + :param state: The actual `Runstate` seen at the time of the violation. + :param required: The `Runstate` required to process this command. + + """ + def __init__(self, error_message: str, + state: Runstate, required: Runstate): + super().__init__(error_message) + self.error_message =3D error_message + self.state =3D state + self.required =3D required + + +F =3D TypeVar('F', bound=3DCallable[..., Any]) # pylint: disable=3Dinvali= d-name + + +# Don't Panic. +def require(required_state: Runstate) -> Callable[[F], F]: + """ + Decorator: protect a method so it can only be run in a certain `Runsta= te`. + + :param required_state: The `Runstate` required to invoke this method. + :raise StateError: When the required `Runstate` is not met. + """ + def _decorator(func: F) -> F: + # _decorator is the decorator that is built by calling the + # require() decorator factory; e.g.: + # + # @require(Runstate.IDLE) def # foo(): ... + # will replace 'foo' with the result of '_decorator(foo)'. + + @wraps(func) + def _wrapper(proto: 'AsyncProtocol[Any]', + *args: Any, **kwargs: Any) -> Any: + # _wrapper is the function that gets executed prior to the + # decorated method. + + if proto.runstate !=3D required_state: + if proto.runstate =3D=3D Runstate.CONNECTING: + emsg =3D "Client is currently connecting." + elif proto.runstate =3D=3D Runstate.DISCONNECTING: + emsg =3D ("Client is disconnecting." + " Call disconnect() to return to IDLE state.") + elif proto.runstate =3D=3D Runstate.RUNNING: + emsg =3D "Client is already connected and running." + elif proto.runstate =3D=3D Runstate.IDLE: + emsg =3D "Client is disconnected and idle." + else: + assert False + raise StateError(emsg, proto.runstate, required_state) + # No StateError, so call the wrapped method. + return func(proto, *args, **kwargs) + + # Return the decorated method; + # Transforming Func to Decorated[Func]. + return cast(F, _wrapper) + + # Return the decorator instance from the decorator factory. Phew! + return _decorator + + class AsyncProtocol(Generic[T]): """ AsyncProtocol implements a generic async message-based protocol. @@ -124,7 +207,18 @@ def __init__(self) -> None: #: exit. self._dc_task: Optional[asyncio.Future[None]] =3D None =20 + self._runstate =3D Runstate.IDLE + + #: An `asyncio.Event` that signals when `runstate` is changed. + self.runstate_changed: asyncio.Event =3D asyncio.Event() + + @property + def runstate(self) -> Runstate: + """The current `Runstate` of the connection.""" + return self._runstate + @upper_half + @require(Runstate.IDLE) async def connect(self, address: Union[str, Tuple[str, int]], ssl: Optional[SSLContext] =3D None) -> None: """ @@ -165,6 +259,21 @@ async def disconnect(self, force: bool =3D False) -> N= one: # Section: Session machinery # -------------------------- =20 + @upper_half + @bottom_half + def _set_state(self, state: Runstate) -> None: + """ + Change the `Runstate` of the protocol connection. + + Signals the `runstate_changed` event. + """ + if state =3D=3D self._runstate: + return + + self._runstate =3D state + self.runstate_changed.set() + self.runstate_changed.clear() + @upper_half async def _new_session(self, address: Union[str, Tuple[str, int]], @@ -189,6 +298,9 @@ async def _new_session(self, protocol-level failure occurs while establishing a new session, the wrapped error may also be an `AQMPError`. """ + assert self.runstate =3D=3D Runstate.IDLE + self._set_state(Runstate.CONNECTING) + self._outgoing =3D asyncio.Queue() =20 phase =3D "connection" @@ -204,6 +316,8 @@ async def _new_session(self, emsg =3D f"Failed to establish {phase}" raise ConnectError(emsg, err) from err =20 + assert self.runstate =3D=3D Runstate.RUNNING + @upper_half async def _do_connect(self, address: Union[str, Tuple[str, int]], ssl: Optional[SSLContext] =3D None) -> None: @@ -227,6 +341,8 @@ async def _begin_new_session(self) -> None: """ After a connection is established, start the bottom half machinery. """ + assert self.runstate =3D=3D Runstate.CONNECTING + reader_coro =3D self._bh_loop_forever(self._bh_recv_message) writer_coro =3D self._bh_loop_forever(self._bh_send_message) =20 @@ -239,6 +355,8 @@ async def _begin_new_session(self) -> None: return_exceptions=3DTrue, ) =20 + self._set_state(Runstate.RUNNING) + @upper_half @bottom_half def _schedule_disconnect(self, force: bool =3D False) -> None: @@ -276,6 +394,7 @@ def _results(self) -> None: Iterable Exception used to multiplex multiple exceptions in the event that multiple Tasks failed with non-cancellation reasons. """ + assert self.runstate =3D=3D Runstate.DISCONNECTING exceptions: List[BaseException] =3D [] =20 assert self._bh_tasks is None or self._bh_tasks.done() @@ -340,6 +459,7 @@ def _paranoid_task_erase(task: _FutureT) -> Optional[_F= utureT]: assert (task is None) or task.done() return None if (task and task.done()) else task =20 + assert self.runstate =3D=3D Runstate.DISCONNECTING self._dc_task =3D _paranoid_task_erase(self._dc_task) self._reader_task =3D _paranoid_task_erase(self._reader_task) self._writer_task =3D _paranoid_task_erase(self._writer_task) @@ -348,6 +468,8 @@ def _paranoid_task_erase(task: _FutureT) -> Optional[_F= utureT]: self._reader =3D None self._writer =3D None =20 + self._set_state(Runstate.IDLE) + # ---------------------------- # Section: Bottom Half methods # ---------------------------- @@ -367,6 +489,9 @@ async def _bh_disconnect(self, force: bool =3D False) -= > None: terminating execution. When `True`, terminate immediately. =20 """ + # Prohibit new calls to execute() et al. + self._set_state(Runstate.DISCONNECTING) + await self._bh_stop_writer(force) await self._bh_stop_reader() =20 --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625112896564130.54399151668576; Wed, 30 Jun 2021 21:14:56 -0700 (PDT) Received: from localhost ([::1]:41116 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo6G-0002s6-T7 for importer@patchew.org; Thu, 01 Jul 2021 00:14:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42314) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo53-0008Sm-65 for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22587) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo50-0000iN-TF for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:36 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-58-QswAubjBNKe0JHlnUgDMvQ-1; Thu, 01 Jul 2021 00:13:32 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 916A91084F5A; Thu, 1 Jul 2021 04:13:31 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 63A7169CB4; Thu, 1 Jul 2021 04:13:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bxg1hx/6n92aih6fXSibqMb6ZUcqzAfwooFlRpGuGVs=; b=hjb1bJuxPTtA0X9K5/H5qwlCmmHiXtOj+KKUaEZDnAOtVM86X14g5/+89g+e0WTlT5WZ+c MgRBqDuAMLhHb9UXHfwWitZtFViYAYAgrSquXSpQriMvmrp0nkZcLIJJQCcAyiUF4hb7oN l3i4HStnNa+fRGqCDQ41M9BPmI/DmX0= X-MC-Unique: QswAubjBNKe0JHlnUgDMvQ-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 08/20] python/aqmp: add logging to AsyncProtocol Date: Thu, 1 Jul 2021 00:13:01 -0400 Message-Id: <20210701041313.1696009-9-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Give the connection and the reader/writer tasks nicknames, and add logging statements throughout. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/protocol.py | 64 ++++++++++++++++++++++++++++++++---- python/qemu/aqmp/util.py | 32 ++++++++++++++++++ 2 files changed, 90 insertions(+), 6 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index a99a191982..dd8564ee02 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -14,6 +14,7 @@ from asyncio import StreamReader, StreamWriter from enum import Enum from functools import wraps +import logging from ssl import SSLContext from typing import ( Any, @@ -34,6 +35,7 @@ create_task, flush, is_closing, + pretty_traceback, upper_half, wait_closed, wait_task_done, @@ -174,14 +176,28 @@ class AsyncProtocol(Generic[T]): can be written after the super() call. - `_on_message`: Actions to be performed when a message is received. + + :param name: + Name used for logging messages, if any. By default, messages + will log to 'qemu.aqmp.protocol', but each individual connection + can be given its own logger by giving it a name; messages will + then log to 'qemu.aqmp.protocol.${name}'. """ # pylint: disable=3Dtoo-many-instance-attributes =20 + #: Logger object for debugging messages from this connection. + logger =3D logging.getLogger(__name__) + # ------------------------- # Section: Public interface # ------------------------- =20 - def __init__(self) -> None: + def __init__(self, name: Optional[str] =3D None) -> None: + #: The nickname for this connection, if any. + self.name: Optional[str] =3D name + if self.name is not None: + self.logger =3D self.logger.getChild(self.name) + # stream I/O self._reader: Optional[StreamReader] =3D None self._writer: Optional[StreamWriter] =3D None @@ -212,6 +228,15 @@ def __init__(self) -> None: #: An `asyncio.Event` that signals when `runstate` is changed. self.runstate_changed: asyncio.Event =3D asyncio.Event() =20 + def __repr__(self) -> str: + argstr =3D '' + if self.name is not None: + argstr +=3D f"name=3D{self.name}" + return "{:s}({:s})".format( + type(self).__name__, + argstr, + ) + @property def runstate(self) -> Runstate: """The current `Runstate` of the connection.""" @@ -301,6 +326,8 @@ async def _new_session(self, assert self.runstate =3D=3D Runstate.IDLE self._set_state(Runstate.CONNECTING) =20 + if not self._outgoing.empty(): + self.logger.warning("Outgoing message queue was not empty!") self._outgoing =3D asyncio.Queue() =20 phase =3D "connection" @@ -311,9 +338,15 @@ async def _new_session(self, await self._begin_new_session() =20 except Exception as err: - # Reset from CONNECTING back to IDLE. - await self.disconnect() emsg =3D f"Failed to establish {phase}" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + try: + # Reset from CONNECTING back to IDLE. + await self.disconnect() + except: + emsg =3D "Unexpected bottom half exceptions" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise raise ConnectError(emsg, err) from err =20 assert self.runstate =3D=3D Runstate.RUNNING @@ -330,12 +363,16 @@ async def _do_connect(self, address: Union[str, Tuple= [str, int]], =20 :raise OSError: For stream-related errors. """ + self.logger.debug("Connecting ...") + if isinstance(address, tuple): connect =3D asyncio.open_connection(address[0], address[1], ss= l=3Dssl) else: connect =3D asyncio.open_unix_connection(path=3Daddress, ssl= =3Dssl) self._reader, self._writer =3D await connect =20 + self.logger.debug("Connected.") + @upper_half async def _begin_new_session(self) -> None: """ @@ -343,8 +380,8 @@ async def _begin_new_session(self) -> None: """ assert self.runstate =3D=3D Runstate.CONNECTING =20 - reader_coro =3D self._bh_loop_forever(self._bh_recv_message) - writer_coro =3D self._bh_loop_forever(self._bh_send_message) + reader_coro =3D self._bh_loop_forever(self._bh_recv_message, 'Read= er') + writer_coro =3D self._bh_loop_forever(self._bh_send_message, 'Writ= er') =20 self._reader_task =3D create_task(reader_coro) self._writer_task =3D create_task(writer_coro) @@ -374,6 +411,7 @@ def _schedule_disconnect(self, force: bool =3D False) -= > None: terminating execution. When `True`, terminate immediately. """ if not self._dc_task: + self.logger.debug("scheduling disconnect.") self._dc_task =3D create_task(self._bh_disconnect(force)) =20 @upper_half @@ -499,8 +537,13 @@ async def _bh_disconnect(self, force: bool =3D False) = -> None: # This implicitly closes the reader, too. if self._writer: if not is_closing(self._writer): + self.logger.debug("Closing StreamWriter.") self._writer.close() + self.logger.debug("Waiting for writer to close.") await wait_closed(self._writer) + self.logger.debug("Writer closed.") + + self.logger.debug("Disconnected.") =20 @bottom_half async def _bh_stop_writer(self, force: bool =3D False) -> None: @@ -513,17 +556,19 @@ async def _bh_stop_writer(self, force: bool =3D False= ) -> None: =20 # Cancel the writer task. if self._writer_task and not self._writer_task.done(): + self.logger.debug("Cancelling writer task.") self._writer_task.cancel() await wait_task_done(self._writer_task) =20 @bottom_half async def _bh_stop_reader(self) -> None: if self._reader_task and not self._reader_task.done(): + self.logger.debug("Cancelling reader task.") self._reader_task.cancel() await wait_task_done(self._reader_task) =20 @bottom_half - async def _bh_loop_forever(self, async_fn: _TaskFN) -> None: + async def _bh_loop_forever(self, async_fn: _TaskFN, name: str) -> None: """ Run one of the bottom-half methods in a loop forever. =20 @@ -531,16 +576,23 @@ async def _bh_loop_forever(self, async_fn: _TaskFN) -= > None: disconnect that will terminate the entire loop. =20 :param async_fn: The bottom-half method to run in a loop. + :param name: The name of this task, used for logging. """ try: while True: await async_fn() except asyncio.CancelledError: # We have been cancelled by _bh_disconnect, exit gracefully. + self.logger.debug("Task.%s: cancelled.", name) return except BaseException: + self.logger.error( + "Task.%s: failure:\n%s\n", name, pretty_traceback() + ) self._schedule_disconnect(force=3DTrue) raise + finally: + self.logger.debug("Task.%s: exiting.", name) =20 @bottom_half async def _bh_send_message(self) -> None: diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py index 9ea91f2862..2311be5893 100644 --- a/python/qemu/aqmp/util.py +++ b/python/qemu/aqmp/util.py @@ -3,10 +3,14 @@ =20 This module primarily provides compatibility wrappers for Python 3.6 to provide some features that otherwise become available in Python 3.7+. + +It additionally provides `pretty_traceback()`, used for formatting +tracebacks for inclusion in the logging stream. """ =20 import asyncio import sys +import traceback from typing import ( Any, Coroutine, @@ -105,6 +109,34 @@ async def wait_task_done(task: Optional['asyncio.Futur= e[Any]']) -> None: break =20 =20 +def pretty_traceback(prefix: str =3D " | ") -> str: + """ + Formats the current traceback, indented to provide visual distinction. + + This is useful for printing a traceback within a traceback for + debugging purposes when encapsulating errors to deliver them up the + stack; when those errors are printed, this helps provide a nice + visual grouping to quickly identify the parts of the error that + belong to the inner exception. + + :param prefix: The prefix to append to each line of the traceback. + :return: A string, formatted something like the following:: + + | Traceback (most recent call last): + | File "foobar.py", line 42, in arbitrary_example + | foo.baz() + | ArbitraryError: [Errno 42] Something bad happened! + """ + output =3D "".join(traceback.format_exception(*sys.exc_info())) + + exc_lines =3D [] + for line in output.split('\n'): + exc_lines.append(prefix + line) + + # The last line is always empty, omit it + return "\n".join(exc_lines[:-1]) + + def upper_half(func: T) -> T: """ Do-nothing decorator that annotates a method as an "upper-half" method. --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113358263956.9821357334055; Wed, 30 Jun 2021 21:22:38 -0700 (PDT) Received: from localhost ([::1]:41030 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoDl-0004uB-9B for importer@patchew.org; Thu, 01 Jul 2021 00:22:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42336) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo56-0000Gw-NA for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:40 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:20072) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo54-0000lL-Py for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:40 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-372-rPLDLt0ZPBqcpJExoEksBg-1; Thu, 01 Jul 2021 00:13:37 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 37323362F9; Thu, 1 Jul 2021 04:13:36 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id B4EE8604CC; Thu, 1 Jul 2021 04:13:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IigzW1skemNVRbczM0Veq/ZUl4qm+ThwuBp/NS4dbPA=; b=Utxsb7wG0NeugfnbUB8SLnTDyebvP+AnegOEL86ueeVhYnKImQxnv8BF0gwNYw9+VPaAs6 ASbemx4Qy9t5AIKj4Vg5fY4YNpGM/TbgE+kkycaei0QQcwmRGcOpNdeIzSRhF6sRrK+0TB 0ZPmVJc7NYaWo+acuQWZ2qrXgDUDLHQ= X-MC-Unique: rPLDLt0ZPBqcpJExoEksBg-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 09/20] python/aqmp: add AsyncProtocol.accept() method Date: Thu, 1 Jul 2021 00:13:02 -0400 Message-Id: <20210701041313.1696009-10-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" It's a little messier than connect, because it wasn't designed to accept *precisely one* connection. Such is life. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/protocol.py | 85 ++++++++++++++++++++++++++++++++++-- 1 file changed, 82 insertions(+), 3 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index dd8564ee02..a32a8cbbf6 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -242,6 +242,24 @@ def runstate(self) -> Runstate: """The current `Runstate` of the connection.""" return self._runstate =20 + @upper_half + @require(Runstate.IDLE) + async def accept(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] =3D None) -> None: + """ + Accept a connection and begin processing message queues. + + If this call fails, `runstate` is guaranteed to be set back to `ID= LE`. + + :param address: + Address to listen to; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise StateError: When the `Runstate` is not `IDLE`. + :raise ConnectError: If a connection could not be accepted. + """ + await self._new_session(address, ssl, accept=3DTrue) + @upper_half @require(Runstate.IDLE) async def connect(self, address: Union[str, Tuple[str, int]], @@ -302,7 +320,8 @@ def _set_state(self, state: Runstate) -> None: @upper_half async def _new_session(self, address: Union[str, Tuple[str, int]], - ssl: Optional[SSLContext] =3D None) -> None: + ssl: Optional[SSLContext] =3D None, + accept: bool =3D False) -> None: """ Establish a new connection and initialize the session. =20 @@ -311,9 +330,10 @@ async def _new_session(self, to be set back to `IDLE`. =20 :param address: - Address to connect to; + Address to connect to/listen on; UNIX socket path or TCP address/port. :param ssl: SSL context to use, if any. + :param accept: Accept a connection instead of connecting when `Tru= e`. =20 :raise ConnectError: When a connection or session cannot be established. @@ -332,7 +352,10 @@ async def _new_session(self, =20 phase =3D "connection" try: - await self._do_connect(address, ssl) + if accept: + await self._do_accept(address, ssl) + else: + await self._do_connect(address, ssl) =20 phase =3D "session" await self._begin_new_session() @@ -351,6 +374,62 @@ async def _new_session(self, =20 assert self.runstate =3D=3D Runstate.RUNNING =20 + @upper_half + async def _do_accept(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] =3D None) -> None: + """ + Acting as the transport server, accept a single connection. + + :param address: + Address to listen on; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise OSError: For stream-related errors. + """ + self.logger.debug("Awaiting connection ...") + connected =3D asyncio.Event() + server: Optional[asyncio.AbstractServer] =3D None + + async def _client_connected_cb(reader: asyncio.StreamReader, + writer: asyncio.StreamWriter) -> No= ne: + """Used to accept a single incoming connection, see below.""" + nonlocal server + nonlocal connected + + # A connection has been accepted; stop listening for new ones. + assert server is not None + server.close() + await server.wait_closed() + server =3D None + + # Register this client as being connected + self._reader, self._writer =3D (reader, writer) + + # Signal back: We've accepted a client! + connected.set() + + if isinstance(address, tuple): + coro =3D asyncio.start_server( + _client_connected_cb, + host=3Daddress[0], + port=3Daddress[1], + ssl=3Dssl, + backlog=3D1, + ) + else: + coro =3D asyncio.start_unix_server( + _client_connected_cb, + path=3Daddress, + ssl=3Dssl, + backlog=3D1, + ) + + server =3D await coro # Starts listening + await connected.wait() # Waits for the callback to fire (and fini= sh) + assert server is None + + self.logger.debug("Connection accepted") + @upper_half async def _do_connect(self, address: Union[str, Tuple[str, int]], ssl: Optional[SSLContext] =3D None) -> None: --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113216814158.14892485062376; Wed, 30 Jun 2021 21:20:16 -0700 (PDT) Received: from localhost ([::1]:32926 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoBT-0007rp-IG for importer@patchew.org; Thu, 01 Jul 2021 00:20:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42392) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5I-00010W-1D for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:60284) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5E-0000sr-PP for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:51 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-280-F1Senn2MMdONVIBjeF6NVQ-1; Thu, 01 Jul 2021 00:13:44 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1A90018414A0; Thu, 1 Jul 2021 04:13:44 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5B0BA604CC; Thu, 1 Jul 2021 04:13:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eA51LRU27vMnzIkuOy/mZC5yOuGX5EVflxYDIfbxMqE=; b=TMflzlJqMIB/8CVpB0+R73fQsygYOp4b8H3t4R6Mueu/65m8jXTY+lLpJUMJmSfJMUTIJ0 lfFWh5J/UL65SKRrvl2shX068Vz4sH67hsxwcmC7TIMOmiqSAWZeio5iW8duesnsBQtOj7 MHuOcTgcKMiroHnFowooZ5GHHniBQWM= X-MC-Unique: F1Senn2MMdONVIBjeF6NVQ-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 10/20] python/aqmp: add _cb_inbound and _cb_inbound logging hooks Date: Thu, 1 Jul 2021 00:13:03 -0400 Message-Id: <20210701041313.1696009-11-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Add hooks designed to log/filter incoming/outgoing messages. The primary intent for these is to be able to support iotests which may want to log messages with specific filters for reproducible output. Another use is for plugging into Urwid frameworks; all messages in/out can be automatically added to a rendering list for the purposes of a qmp-shell like tool. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/protocol.py | 50 +++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index a32a8cbbf6..72c9e95198 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -176,6 +176,11 @@ class AsyncProtocol(Generic[T]): can be written after the super() call. - `_on_message`: Actions to be performed when a message is received. + - `_cb_outbound`: + Logging/Filtering hook for all outbound messages. + - `_cb_inbound`: + Logging/Filtering hook for all inbound messages. + This hook runs *before* `_on_message()`. =20 :param name: Name used for logging messages, if any. By default, messages @@ -700,6 +705,43 @@ async def _bh_recv_message(self) -> None: # Section: Message I/O # -------------------- =20 + @upper_half + @bottom_half + def _cb_outbound(self, msg: T) -> T: + """ + Callback: outbound message hook. + + This is intended for subclasses to be able to add arbitrary + hooks to filter or manipulate outgoing messages. The base + implementation does nothing but log the message without any + manipulation of the message. + + :param msg: raw outbound message + :return: final outbound message + """ + self.logger.debug("--> %s", str(msg)) + return msg + + @upper_half + @bottom_half + def _cb_inbound(self, msg: T) -> T: + """ + Callback: inbound message hook. + + This is intended for subclasses to be able to add arbitrary + hooks to filter or manipulate incoming messages. The base + implementation does nothing but log the message without any + manipulation of the message. + + This method does not "handle" incoming messages; it is a filter. + The actual "endpoint" for incoming messages is `_on_message()`. + + :param msg: raw inbound message + :return: processed inbound message + """ + self.logger.debug("<-- %s", str(msg)) + return msg + @upper_half @bottom_half async def _do_recv(self) -> T: @@ -728,8 +770,8 @@ async def _recv(self) -> T: =20 :return: A single (filtered, processed) protocol message. """ - # A forthcoming commit makes this method less trivial. - return await self._do_recv() + message =3D await self._do_recv() + return self._cb_inbound(message) =20 @upper_half @bottom_half @@ -759,7 +801,7 @@ async def _send(self, msg: T) -> None: =20 :raise OSError: For problems with the underlying stream. """ - # A forthcoming commit makes this method less trivial. + msg =3D self._cb_outbound(msg) self._do_send(msg) =20 @bottom_half @@ -774,6 +816,6 @@ async def _on_message(self, msg: T) -> None: directly cause the loop to halt, so logic may be best-kept to a minimum if at all possible. =20 - :param msg: The incoming message + :param msg: The incoming message, already logged/filtered. """ # Nothing to do in the abstract case. --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113125935577.5228399128221; Wed, 30 Jun 2021 21:18:45 -0700 (PDT) Received: from localhost ([::1]:54784 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoA0-0003cs-Py for importer@patchew.org; Thu, 01 Jul 2021 00:18:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42376) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5H-0000y9-EO for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:51 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:35576) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5E-0000st-PH for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:50 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-596-nyRNkiOJMqOL-lKEL22iUQ-1; Thu, 01 Jul 2021 00:13:47 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3450C100C609; Thu, 1 Jul 2021 04:13:46 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 49C3B604CC; Thu, 1 Jul 2021 04:13:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+r0GS8hd8wpaV11VOAVCy4OHgC4oU2ZRV6Q7dJgnRuY=; b=d3Wh8iyKDmXrmXry9drfQ/hFKteHFKTjeWeY4ftOfBqpJxCYyQrDdVd8Vd66+Y6CUEFFUV rBG2GMDdXpoU6oTu/X8W3BR+7IeF2Og8mlb8ZoOqiFo147mFBxibfNc3uVWt4yyuVT7IKU CBK/uq5ftARGiDEq8BDow3mRvu6jfAY= X-MC-Unique: nyRNkiOJMqOL-lKEL22iUQ-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 11/20] python/aqmp: add AsyncProtocol._readline() method Date: Thu, 1 Jul 2021 00:13:04 -0400 Message-Id: <20210701041313.1696009-12-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This is added as a courtesy: many protocols are line-based, including QMP. Putting it in AsyncProtocol lets us keep the QMP class implementation just a pinch more abstract. (And, if we decide to add a QTEST implementation later, it will need this, too. (Yes, I have a QTEST implementation.)) Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/protocol.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 72c9e95198..6a2a7be056 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -742,6 +742,36 @@ def _cb_inbound(self, msg: T) -> T: self.logger.debug("<-- %s", str(msg)) return msg =20 + @upper_half + @bottom_half + async def _readline(self) -> bytes: + """ + Wait for a newline from the incoming reader. + + This method is provided as a convenience for upper-layer + protocols, as many are line-based. + + This method *may* return a sequence of bytes without a trailing + newline if EOF occurs, but *some* bytes were received. In this + case, the next call will raise `EOFError`. It is assumed that + the layer 4 protocol will decide if there is anything meaningful + to be done with a partial message. + + :raise OSError: For stream-related errors. + :raise EOFError: + If the reader stream is at EOF and there are no bytes to retur= n. + :return: bytes, including the newline. + + """ + assert self._reader is not None + msg_bytes =3D await self._reader.readline() + + if not msg_bytes: + if self._reader.at_eof(): + raise EOFError + + return msg_bytes + @upper_half @bottom_half async def _do_recv(self) -> T: --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113318995132.09204945403735; Wed, 30 Jun 2021 21:21:58 -0700 (PDT) Received: from localhost ([::1]:39250 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoD7-0003jN-VL for importer@patchew.org; Thu, 01 Jul 2021 00:21:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42476) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5M-0001GD-DM for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:50232) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5H-0000v6-8d for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:56 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-134-1HdwwOINPe-sbnM7tgCc5Q-1; Thu, 01 Jul 2021 00:13:48 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 77956362F9; Thu, 1 Jul 2021 04:13:47 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F891604CC; Thu, 1 Jul 2021 04:13:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JRpDFNNJwzb5qxCP1zWhem3RH82wFJKoJsqX5HskYAs=; b=KR0/ElB74BvPBzS5ZOJPMRWXuVQp1WJbzN6w6HcOcgJPZsCzqpMgH7BDL529wa63WvSvea U0XeX9s/mpWkFmwhzI8va24jq4wdsXKlYV5AJB+viSrUY9waniSfyffyusGqQhcVcez95S DzLIzBhGNyHuJ8bY2sCksRa7EYb4Z+k= X-MC-Unique: 1HdwwOINPe-sbnM7tgCc5Q-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 12/20] python/aqmp: add QMP Message format Date: Thu, 1 Jul 2021 00:13:05 -0400 Message-Id: <20210701041313.1696009-13-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" The Message class is here primarily to serve as a solid type to use for mypy static typing for unambiguous annotation and documentation. We can also stuff JSON serialization and deserialization into this class itself so it can be re-used even outside this infrastructure. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 4 +- python/qemu/aqmp/message.py | 207 +++++++++++++++++++++++++++++++++++ 2 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 python/qemu/aqmp/message.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 5c44fabeea..c1ec68a023 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,12 +22,14 @@ # the COPYING file in the top-level directory. =20 from .error import AQMPError, MultiException +from .message import Message from .protocol import ConnectError, Runstate =20 =20 # The order of these fields impact the Sphinx documentation order. __all__ =3D ( - # Classes + # Classes, most to least important + 'Message', 'Runstate', =20 # Exceptions, most generic to most explicit diff --git a/python/qemu/aqmp/message.py b/python/qemu/aqmp/message.py new file mode 100644 index 0000000000..3a4b283032 --- /dev/null +++ b/python/qemu/aqmp/message.py @@ -0,0 +1,207 @@ +""" +QMP Message Format + +This module provides the `Message` class, which represents a single QMP +message sent to or from the server. +""" + +import json +from json import JSONDecodeError +from typing import ( + Dict, + Iterator, + Mapping, + MutableMapping, + Optional, + Union, +) + +from .error import ProtocolError + + +class Message(MutableMapping[str, object]): + """ + Represents a single QMP protocol message. + + QMP uses JSON objects as its basic communicative unit; so this + Python object is a :py:obj:`~collections.abc.MutableMapping`. It may + be instantiated from either another mapping (like a `dict`), or from + raw `bytes` that still need to be deserialized. + + Once instantiated, it may be treated like any other MutableMapping:: + + >>> msg =3D Message(b'{"hello": "world"}') + >>> assert msg['hello'] =3D=3D 'world' + >>> msg['id'] =3D 'foobar' + >>> print(msg) + { + "hello": "world", + "id": "foobar" + } + + It can be converted to `bytes`:: + + >>> msg =3D Message({"hello": "world"}) + >>> print(bytes(msg)) + b'{"hello":"world","id":"foobar"}' + + Or back into a garden-variety `dict`:: + + >>> dict(msg) + {'hello': 'world'} + + + :param value: Initial value, if any. + :param eager: + When `True`, attempt to serialize or deserialize the initial value + immediately, so that conversion exceptions are raised during + the call to ``__init__()``. + """ + # pylint: disable=3Dtoo-many-ancestors + + def __init__(self, + value: Union[bytes, Mapping[str, object]] =3D b'', *, + eager: bool =3D True): + self._data: Optional[bytes] =3D None + self._obj: Optional[Dict[str, object]] =3D None + + if isinstance(value, bytes): + self._data =3D value + if eager: + self._obj =3D self._deserialize(self._data) + else: + self._obj =3D dict(value) + if eager: + self._data =3D self._serialize(self._obj) + + # Methods necessary to implement the MutableMapping interface, see: + # https://docs.python.org/3/library/collections.abc.html#collections.a= bc.MutableMapping + + # We get pop, popitem, clear, update, setdefault, __contains__, + # keys, items, values, get, __eq__ and __ne__ for free. + + def __getitem__(self, key: str) -> object: + return self._object[key] + + def __setitem__(self, key: str, value: object) -> None: + self._object[key] =3D value + self._data =3D None + + def __delitem__(self, key: str) -> None: + del self._object[key] + self._data =3D None + + def __iter__(self) -> Iterator[str]: + return iter(self._object) + + def __len__(self) -> int: + return len(self._object) + + # Dunder methods not related to MutableMapping: + + def __repr__(self) -> str: + return f"Message({self._object!r})" + + def __str__(self) -> str: + """Pretty-printed representation of this QMP message.""" + return json.dumps(self._object, indent=3D2) + + def __bytes__(self) -> bytes: + """bytes representing this QMP message.""" + if self._data is None: + self._data =3D self._serialize(self._obj or {}) + return self._data + + # + + @property + def _object(self) -> Dict[str, object]: + """ + A `dict` representing this QMP message. + + Generated on-demand, if required. This property is private + because it returns an object that could be used to invalidate + the internal state of the `Message` object. + """ + if self._obj is None: + self._obj =3D self._deserialize(self._data or b'') + return self._obj + + @classmethod + def _serialize(cls, value: object) -> bytes: + """ + Serialize a JSON object as `bytes`. + + :raise ValueError: When the object cannot be serialized. + :raise TypeError: When the object cannot be serialized. + + :return: `bytes` ready to be sent over the wire. + """ + return json.dumps(value, separators=3D(',', ':')).encode('utf-8') + + @classmethod + def _deserialize(cls, data: bytes) -> Dict[str, object]: + """ + Deserialize JSON `bytes` into a native Python `dict`. + + :raise DeserializationError: + If JSON deserialization fails for any reason. + :raise UnexpectedTypeError: + If the data does not represent a JSON object. + + :return: A `dict` representing this QMP message. + """ + try: + obj =3D json.loads(data) + except JSONDecodeError as err: + emsg =3D "Failed to deserialize QMP message." + raise DeserializationError(emsg, data) from err + if not isinstance(obj, dict): + raise UnexpectedTypeError( + "QMP message is not a JSON object.", + obj + ) + return obj + + +class DeserializationError(ProtocolError): + """ + A QMP message was not understood as JSON. + + When this Exception is raised, ``__cause__`` will be set to the + `json.JSONDecodeError` Exception, which can be interrogated for + further details. + + :param error_message: Human-readable string describing the error. + :param raw: The raw `bytes` that prompted the failure. + """ + def __init__(self, error_message: str, raw: bytes): + super().__init__(error_message) + #: The raw `bytes` that were not understood as JSON. + self.raw: bytes =3D raw + + def __str__(self) -> str: + return "\n".join([ + super().__str__(), + f" raw bytes were: {str(self.raw)}", + ]) + + +class UnexpectedTypeError(ProtocolError): + """ + A QMP message was JSON, but not a JSON object. + + :param error_message: Human-readable string describing the error. + :param value: The deserialized JSON value that wasn't an object. + """ + def __init__(self, error_message: str, value: object): + super().__init__(error_message) + #: The JSON value that was expected to be an object. + self.value: object =3D value + + def __str__(self) -> str: + strval =3D json.dumps(self.value, indent=3D2) + return "\n".join([ + super().__str__(), + f" json value was: {strval}", + ]) --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113450910528.5841895590962; Wed, 30 Jun 2021 21:24:10 -0700 (PDT) Received: from localhost ([::1]:45320 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoFF-0007nW-Vh for importer@patchew.org; Thu, 01 Jul 2021 00:24:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42448) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5L-0001Cj-Gn for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44097) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5J-0000w5-7M for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:55 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-174-O5it89rGPCOIMVQoqQD9tA-1; Thu, 01 Jul 2021 00:13:49 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 91DF118414A2; Thu, 1 Jul 2021 04:13:48 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9C61769CB4; Thu, 1 Jul 2021 04:13:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dwUGGY5uyWyYQd5Zurxfm8CsvdBOc3cG50bLx3lkVDI=; b=DZ3xRpH37hCXtHKOJWg3XH+JOFsmLz6hXtYJcTVLuKOYzwil81ecU2QPlKaCVXJ84JPr6i ajgeCj/k3K2sGBxRIVVKZPKUMsjff5Zl8dp/d8SSXgRe9wUzMBxiH5qrlPHLmCKxRCvA+M Bxt4eTKOBfua7ebzt++aNj5gQmEhkpA= X-MC-Unique: O5it89rGPCOIMVQoqQD9tA-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 13/20] python/aqmp: add well-known QMP object models Date: Thu, 1 Jul 2021 00:13:06 -0400 Message-Id: <20210701041313.1696009-14-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" The QMP spec doesn't define very many objects that are iron-clad in their format, but there are a few. This module makes it trivial to validate them without relying on an external third-party library. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/models.py | 133 +++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 python/qemu/aqmp/models.py diff --git a/python/qemu/aqmp/models.py b/python/qemu/aqmp/models.py new file mode 100644 index 0000000000..24c94123ac --- /dev/null +++ b/python/qemu/aqmp/models.py @@ -0,0 +1,133 @@ +""" +QMP Data Models + +This module provides simplistic data classes that represent the few +structures that the QMP spec mandates; they are used to verify incoming +data to make sure it conforms to spec. +""" +# pylint: disable=3Dtoo-few-public-methods + +from collections import abc +from typing import ( + Any, + Mapping, + Optional, + Sequence, +) + + +class Model: + """ + Abstract data model, representing some QMP object of some kind. + + :param raw: The raw object to be validated. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + self._raw =3D raw + + def _check_key(self, key: str) -> None: + if key not in self._raw: + raise KeyError(f"'{self._name}' object requires '{key}' member= ") + + def _check_value(self, key: str, type_: type, typestr: str) -> None: + assert key in self._raw + if not isinstance(self._raw[key], type_): + raise TypeError( + f"'{self._name}' member '{key}' must be a {typestr}" + ) + + def _check_member(self, key: str, type_: type, typestr: str) -> None: + self._check_key(key) + self._check_value(key, type_, typestr) + + @property + def _name(self) -> str: + return type(self).__name__ + + def __repr__(self) -> str: + return f"{self._name}({self._raw!r})" + + +class Greeting(Model): + """ + Defined in qmp-spec.txt, section 2.2, "Server Greeting". + + :param raw: The raw Greeting object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'QMP' member + self.QMP: QMPGreeting # pylint: disable=3Dinvalid-name + + self._check_member('QMP', abc.Mapping, "JSON object") + self.QMP =3D QMPGreeting(self._raw['QMP']) + + +class QMPGreeting(Model): + """ + Defined in qmp-spec.txt, section 2.2, "Server Greeting". + + :param raw: The raw QMPGreeting object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'version' member + self.version: Mapping[str, object] + #: 'capabilities' member + self.capabilities: Sequence[object] + + self._check_member('version', abc.Mapping, "JSON object") + self.version =3D self._raw['version'] + + self._check_member('capabilities', abc.Sequence, "JSON array") + self.capabilities =3D self._raw['capabilities'] + + +class ErrorResponse(Model): + """ + Defined in qmp-spec.txt, section 2.4.2, "error". + + :param raw: The raw ErrorResponse object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'error' member + self.error: ErrorInfo + #: 'id' member + self.id: Optional[object] =3D None # pylint: disable=3Dinvalid-na= me + + self._check_member('error', abc.Mapping, "JSON object") + self.error =3D ErrorInfo(self._raw['error']) + + if 'id' in raw: + self.id =3D raw['id'] + + +class ErrorInfo(Model): + """ + Defined in qmp-spec.txt, section 2.4.2, "error". + + :param raw: The raw ErrorInfo object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'class' member, with an underscore to avoid conflicts in Python. + self.class_: str + #: 'desc' member + self.desc: str + + self._check_member('class', str, "string") + self.class_ =3D self._raw['class'] + + self._check_member('desc', str, "string") + self.desc =3D self._raw['desc'] --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113676588629.8635912949607; Wed, 30 Jun 2021 21:27:56 -0700 (PDT) Received: from localhost ([::1]:54318 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoIt-0005Uw-H5 for importer@patchew.org; Thu, 01 Jul 2021 00:27:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42544) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5R-0001XQ-7R for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:38595) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5L-0000yj-Rb for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:00 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-289-xwk57eyiOiasdP6QdbEcag-1; Thu, 01 Jul 2021 00:13:50 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C45F61084F55; Thu, 1 Jul 2021 04:13:49 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id B7319604CD; Thu, 1 Jul 2021 04:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0ojWLHfYCLSQAlkV7+YzXB1eKkc2lfnwZ4hcXW/UbNg=; b=NnAhfXwVZEUr7ultoRDXwgIjeC8g4Fb2V0w/6OkDiONCGaxUcgJtOcSWoEZTiywBl9JuCH BhckR64qNSSYJpk31/HMYtwXDItzcpHJL9Pkizu3fQdzNQdHmUw9zJiMwq2d1yLJfFg/ZY oW0xDcLKGIXqLzW1wyVIT8aUJZIkDFY= X-MC-Unique: xwk57eyiOiasdP6QdbEcag-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 14/20] python/aqmp: add QMP event support Date: Thu, 1 Jul 2021 00:13:07 -0400 Message-Id: <20210701041313.1696009-15-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) This class was designed as a "mix-in" primarily so that the feature could be given its own treatment in its own python file. It gets quite a bit too long otherwise. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- Yes, the docstring is long. I recommend looking at the generated Sphinx output for that part instead. You can review the markup itself if you are a masochist. Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 2 + python/qemu/aqmp/events.py | 878 +++++++++++++++++++++++++++++++++++ 2 files changed, 880 insertions(+) create mode 100644 python/qemu/aqmp/events.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index c1ec68a023..ae87436470 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,6 +22,7 @@ # the COPYING file in the top-level directory. =20 from .error import AQMPError, MultiException +from .events import EventListener from .message import Message from .protocol import ConnectError, Runstate =20 @@ -30,6 +31,7 @@ __all__ =3D ( # Classes, most to least important 'Message', + 'EventListener', 'Runstate', =20 # Exceptions, most generic to most explicit diff --git a/python/qemu/aqmp/events.py b/python/qemu/aqmp/events.py new file mode 100644 index 0000000000..140465255e --- /dev/null +++ b/python/qemu/aqmp/events.py @@ -0,0 +1,878 @@ +""" +AQMP Events and EventListeners + +Asynchronous QMP uses `EventListener` objects to listen for events. An +`EventListener` is a FIFO event queue that can be pre-filtered to listen +for only specific events. Each `EventListener` instance receives its own +copy of events that it hears, so events may be consumed without fear or +worry for depriving other listeners of events they need to hear. + + +EventListener Tutorial +---------------------- + +In all of the following examples, we assume that we have a +:py:class:`~qmp_protocol.QMP` object instantiated named ``qmp`` that is +already connected. + + +`listener()` context blocks with one name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most basic usage is by using the `listener()` context manager to +construct them: + +.. code:: python + + with qmp.listener('STOP') as listener: + await qmp.execute('stop') + await listener.get() + +The listener is active only for the duration of the =E2=80=98with=E2=80=99= block. This +instance listens only for =E2=80=98STOP=E2=80=99 events. + + +`listener()` context blocks with two or more names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Multiple events can be selected for by providing any ``Iterable[str]``: + +.. code:: python + + with qmp.listener(('STOP', 'RESUME')) as listener: + await qmp.execute('stop') + event =3D await listener.get() + assert event['event'] =3D=3D 'STOP' + + await qmp.execute('cont') + event =3D await listener.get() + assert event['event'] =3D=3D 'RESUME' + + +`listener()` context blocks with no names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By omitting names entirely, you can listen to ALL events. + +.. code:: python + + with qmp.listener() as listener: + await qmp.execute('stop') + event =3D await listener.get() + assert event['event'] =3D=3D 'STOP' + +This isn=E2=80=99t a very good use case for this feature: In a non-trivial +running system, we may not know what event will arrive next. Grabbing +the top of a FIFO queue returning multiple kinds of events may be prone +to error. + + +Using async iterators to retrieve events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you=E2=80=99d like to simply watch what events happen to arrive, you ca= n use +the listener as an async iterator: + +.. code:: python + + with qmp.listener() as listener: + async for event in listener: + print(f"Event arrived: {event['event']}") + +This is analogous to the following code: + +.. code:: python + + with qmp.listener() as listener: + while True: + event =3D listener.get() + print(f"Event arrived: {event['event']}") + +This event stream will never end, so these blocks will never terminate. + + +Using asyncio.Task to concurrently retrieve events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since a listener=E2=80=99s event stream will never terminate, it is not li= kely +useful to use that form in a script. For longer-running clients, we can +create event handlers by using `asyncio.Task` to create concurrent +coroutines: + +.. code:: python + + async def print_events(listener): + try: + async for event in listener: + print(f"Event arrived: {event['event']}") + except asyncio.CancelledError: + return + + with qmp.listener() as listener: + task =3D asyncio.Task(print_events(listener)) + await qmp.execute('stop') + await qmp.execute('cont') + task.cancel() + await task + +However, there is no guarantee that these events will be received by the +time we leave this context block. Once the context block is exited, the +listener will cease to hear any new events, and becomes inert. + +Be mindful of the timing: the above example will *probably*=E2=80=93 but d= oes +not *guarantee*=E2=80=93 that both STOP/RESUMED events will be printed. The +example below outlines how to use listeners outside of a context block. + + +Using `register_listener()` and `remove_listener()` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To create a listener with a longer lifetime, beyond the scope of a +single block, create a listener and then call `register_listener()`: + +.. code:: python + + class MyClient: + def __init__(self, qmp): + self.qmp =3D qmp + self.listener =3D EventListener() + + async def print_events(self): + try: + async for event in self.listener: + print(f"Event arrived: {event['event']}") + except asyncio.CancelledError: + return + + async def run(self): + self.task =3D asyncio.Task(self.print_events) + self.qmp.register_listener(self.listener) + await qmp.execute('stop') + await qmp.execute('cont') + + async def stop(self): + self.task.cancel() + await self.task + self.qmp.remove_listener(self.listener) + +The listener can be deactivated by using `remove_listener()`. When it is +removed, any possible pending events are cleared and it can be +re-registered at a later time. + + +Using the built-in all events listener +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :py:class:`~qmp_protocol.QMP` object creates its own default +listener named :py:obj:`~Events.events` that can be used for the same +purpose without having to create your own: + +.. code:: python + + async def print_events(listener): + try: + async for event in listener: + print(f"Event arrived: {event['event']}") + except asyncio.CancelledError: + return + + task =3D asyncio.Task(print_events(qmp.events)) + + await qmp.execute('stop') + await qmp.execute('cont') + + task.cancel() + await task + + +Using both .get() and async iterators +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The async iterator and `get()` methods pull events from the same FIFO +queue. If you mix the usage of both, be aware: Events are emitted +precisely once per listener. + +If multiple contexts try to pull events from the same listener instance, +events are still emitted only precisely once. + +This restriction can be lifted by creating additional listeners. + + +Creating multiple listeners +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Additional `EventListener` objects can be created at-will. Each one +receives its own copy of events, with separate FIFO event queues. + +.. code:: python + + my_listener =3D EventListener() + qmp.register_listener(my_listener) + + await qmp.execute('stop') + copy1 =3D await my_listener.get() + copy2 =3D await qmp.events.get() + + assert copy1 =3D=3D copy2 + +In this example, we await an event from both a user-created +`EventListener` and the built-in events listener. Both receive the same +event. + + +Clearing listeners +~~~~~~~~~~~~~~~~~~ + +`EventListener` objects can be cleared, clearing all events seen thus far: + +.. code:: python + + await qmp.execute('stop') + qmp.events.clear() + await qmp.execute('cont') + event =3D await qmp.events.get() + assert event['event'] =3D=3D 'RESUME' + +`EventListener` objects are FIFO queues. If events are not consumed, +they will remain in the queue until they are witnessed or discarded via +`clear()`. FIFO queues will be drained automatically upon leaving a +context block, or when calling `remove_listener()`. + + +Accessing listener history +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`EventListener` objects record their history. Even after being cleared, +you can obtain a record of all events seen so far: + +.. code:: python + + await qmp.execute('stop') + await qmp.execute('cont') + qmp.events.clear() + + assert len(qmp.events.history) =3D=3D 2 + assert qmp.events.history[0]['event'] =3D=3D 'STOP' + assert qmp.events.history[1]['event'] =3D=3D 'RESUME' + +The history is updated immediately and does not require the event to be +witnessed first. + + +Using event filters +~~~~~~~~~~~~~~~~~~~ + +`EventListener` objects can be given complex filtering criteria if names +are not sufficient: + +.. code:: python + + def job1_filter(event) -> bool: + event_data =3D event.get('data', {}) + event_job_id =3D event_data.get('id') + return event_job_id =3D=3D "job1" + + with qmp.listener('JOB_STATUS_CHANGE', job1_filter) as listener: + await qmp.execute('blockdev-backup', arguments=3D{'job-id': 'job1',= ...}) + async for event in listener: + if event['data']['status'] =3D=3D 'concluded': + break + +These filters might be most useful when parameterized. `EventListener` +objects expect a function that takes only a single argument (the raw +event, as a `Message`) and returns a bool; True if the event should be +accepted into the stream. You can create a function that adapts this +signature to accept configuration parameters: + +.. code:: python + + def job_filter(job_id: str) -> EventFilter: + def filter(event: Message) -> bool: + return event['data']['id'] =3D=3D job_id + return filter + + with qmp.listener('JOB_STATUS_CHANGE', job_filter('job2')) as listener: + await qmp.execute('blockdev-backup', arguments=3D{'job-id': 'job2',= ...}) + async for event in listener: + if event['data']['status'] =3D=3D 'concluded': + break + + +Activating an existing listener with `listen()` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Listeners with complex, long configurations can also be created manually +and activated temporarily by using `listen()` instead of `listener()`: + +.. code:: python + + listener =3D EventListener(('BLOCK_JOB_COMPLETED', 'BLOCK_JOB_CANCELLED= ', + 'BLOCK_JOB_ERROR', 'BLOCK_JOB_READY', + 'BLOCK_JOB_PENDING', 'JOB_STATUS_CHANGE')) + + with qmp.listen(listener): + await qmp.execute('blockdev-backup', arguments=3D{'job-id': 'job3',= ...}) + async for event in listener: + print(event) + if event['event'] =3D=3D 'BLOCK_JOB_COMPLETED': + break + +Any events that are not witnessed by the time the block is left will be +cleared from the queue; entering the block is an implicit +`register_listener()` and leaving the block is an implicit +`remove_listener()`. + + +Activating multiple existing listeners with `listen()` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While `listener()` is only capable of creating a single listener, +`listen()` is capable of activating multiple listeners simultaneously: + +.. code:: python + + def job_filter(job_id: str) -> EventFilter: + def filter(event: Message) -> bool: + return event['data']['id'] =3D=3D job_id + return filter + + jobA =3D EventListener('JOB_STATUS_CHANGE', job_filter('jobA')) + jobB =3D EventListener('JOB_STATUS_CHANGE', job_filter('jobB')) + + with qmp.listen(jobA, jobB): + qmp.execute('blockdev-create', arguments=3D{'job-id': 'jobA', ...}) + qmp.execute('blockdev-create', arguments=3D{'job-id': 'jobB', ...}) + + async for event in jobA.get(): + if event['data']['status'] =3D=3D 'concluded': + break + async for event in jobB.get(): + if event['data']['status'] =3D=3D 'concluded': + break + + +Extending the `EventListener` class +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the case that a more specialized `EventListener` is desired to +provide either more functionality or more compact syntax for specialized +cases, it can be extended. + +One of the key methods to extend or override is +:py:meth:`~EventListener.accept()`. The default implementation checks an +incoming message for: + +1. A qualifying name, if any :py:obj:`~EventListener.names` were + specified at initialization time +2. That :py:obj:`~EventListener.event_filter()` returns True. + +This can be modified however you see fit to change the criteria for +inclusion in the stream. + +For convenience, a ``JobListener`` class could be created that simply +bakes in configuration so it does not need to be repeated: + +.. code:: python + + class JobListener(EventListener): + def __init__(self, job_id: str): + super().__init__(('BLOCK_JOB_COMPLETED', 'BLOCK_JOB_CANCELLED', + 'BLOCK_JOB_ERROR', 'BLOCK_JOB_READY', + 'BLOCK_JOB_PENDING', 'JOB_STATUS_CHANGE')) + self.job_id =3D job_id + + def accept(self, event) -> bool: + if not super().accept(event): + return False + if event['event'] in ('BLOCK_JOB_PENDING', 'JOB_STATUS_CHANGE'): + return event['data']['id'] =3D=3D job_id + return event['data']['device'] =3D=3D job_id + +From here on out, you can conjure up a custom-purpose listener that +listens only for job-related events for a specific job-id easily: + +.. code:: python + + listener =3D JobListener('job4') + with qmp.listener(listener): + await qmp.execute('blockdev-backup', arguments=3D{'job-id': 'job4',= ...}) + async for event in listener: + print(event) + if event['event'] =3D=3D 'BLOCK_JOB_COMPLETED': + break + + +Experimental Interfaces & Design Issues +--------------------------------------- + +These interfaces are not ones I am sure I will keep or otherwise modify +heavily. + +Tertiary, or post-accept filtering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Primarily filtering is based on the names of events, and secondary +filtering is achieved through the use of event_filter callbacks. + +Tertiary filtering occurs after a listener has already accepted the +event, and takes place during the `get()` call. + +`get()` accepts optional ``**kwargs`` arguments that get matched +against the ``data`` field of an event to allow for trivial event +conditions: + +.. code:: python + + with qmp.listen('JOB_STATUS_CHANGE') as listener: + await qmp.execute('blockdev-backup', arguments=3D{'job-id': 'job5',= ...}) + await listener.get(status=3D'pending') + await qmp.execute('job-finalize', arguments=3D{'job-id': 'job5', ..= .}) + await listener.get(status=3D'null') + +The problem with this is that the tertiary filtering will drop events +that were not selected for on the floor, which reintroduces some of the +same problems that inspired the creation of `EventListener` to begin +with. + +Another problem is that this tertiary filtering is extremely +rudimentary: it is quite convenient for a listener configured to listen +only to ``JOB_STATUS_CHANGE``, but it does not allow for post-selection +of events with different names in the event that a listener was created +with a fairly wide selection criteria. + +A final problem is that the filtering is not very powerful: it matches +only fields in ``data`` for strict equality; it cannot perform subset +matching like the legacy `QEMUMachine` methods `event_wait()`, +`events_wait()`, or `event_match()`. + +However, those interfaces are =E2=80=A6 ugly, and a little complicated. Th= ey got +the job done years ago when I wrote them, but I think they=E2=80=99re over= ly +complex and too hard to use now. + +Still, I am left wondering if this isn=E2=80=99t flexible enough. + +- Dropping post-filtered events on the floor seems prone to error. +- Post-filtering on event name(s) might be nice, but further increases + risk related with accidentally discarding events. +- Post-filters could (perhaps) return a sequence of events they=E2=80=99ve + discarded, but that complicates the signature of `get()` a lot: + + .. code:: python + + event, discarded =3D await listener.get(status=3D'null') + event, _ =3D await listener.get(status=3D'null') + +- Maybe listeners could simply cache a =E2=80=9Cdiscarded=E2=80=9D list i= nto its object + state, and (possibly) emit a warning if these discarded events are + not cleared before the listener is unregistered. Still, the goals of co= mpact + syntax and safety are at odds here. Instructing `get()` that we're OK + with tossing events on the floor every time we use it will quickly clut= ter + up most unit tests. + +- ``kwargs`` syntax is convenient for the job filtering case in particula= r, + but is not really broadly flexible. + +- Maybe post-filtering can also be done with event filter functions, + the same kind as used for secondary pre-filtering. It=E2=80=99d at least + allow for maximum flexibility =E2=80=93 but the syntax would be less + convenient and compact than the kwargs post-filters: + + .. code:: python + + def event_filter(event) -> bool: + return event['data']['status'] =3D=3D 'null' + + event =3D await listener.get(event_filter) + +- The above suggestion also introduces a complexity if we want to + support both the ``**kwargs`` form and the ``event_filter`` form: + Whatever name is chosen for the ``event_filter`` argument implicitly + prohibits us from filtering against any possible data fields of the + same name. + + Items beginning with "__" are prohibited in the QMP spec, though, so + it may be safe to name the event filter argument something like + "__filter". + + Python 3.8=E2=80=99s PEP570 =E2=80=9CPositional Only Parameters=E2=80= =9D + https://www.python.org/dev/peps/pep-0570/ would be a good fit for + this feature, but we will not be able to use it for quite some time + in QEMU. (We will not be able to use 3.7 until some time in 2022.) + + +qmp.listener()=E2=80=99s type signature +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`listener()` does not return anything, because it was assumed the caller +already had a handle to the listener. However, for +``qmp.listener(EventListener())`` forms, the caller will not have saved +a handle to the listener. + +Because this function can accept *many* listeners, I found it hard to +accurately type in a way where it could be used in both =E2=80=9Cone=E2=80= =9D or =E2=80=9Cmany=E2=80=9D +forms conveniently and in a statically type-safe manner. + +Ultimately, I removed the return altogether, but perhaps with more time +I can work out a way to re-add it. + + +listener-dispatched callbacks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An earlier design allowed for users to directly set a callback on a +listener. + +It also allowed for a decorator to be used to easily morph a given +function into an event callback for an event of the same name: + +.. code:: python + + @qmp.event + async def stop(event): + print("QEMU has stopped!") + +Or to manually specify a list of events the handler was written for: + +.. code:: python + + @qmp.event(('STOP', 'RESUME')) + async def handler(event): + print(f"Got event '{event['event']}'!") + +This was very convenient for setting up dedicated functions that handle +specific events, setting up all-events loggers, etc. I didn't like this +in the end, for a few reasons: + +- When setting a callback on a listener, it meant that `get()` and the + async iterator became dead interfaces that would never return + anything. It felt like an abuse of the interface, ultimately. Forcing + the user to retrieve the event themselves felt like the =E2=80=9Ccleane= r=E2=80=9D + architecture =E2=80=93 though severely less convenient. + +- If the EventListener itself was responsible for executing an event + callback, it meant that the QMP bottom half itself was ultimately + responsible for calling user callbacks which may fault and cause the + bottom half to terminate. + + I didn't like the idea of a QMP client loop dying because of user + code; Normally, the design of the bottom half is such that =E2=80=9Cint= ernal=E2=80=9D + errors are hidden from the caller. In this case, the caller is likely + the only one who actually understands the error, which felt like an + inversion of concerns. + + +API Reference +------------- + +""" + +import asyncio +from contextlib import contextmanager +import logging +from typing import ( + AsyncIterator, + Callable, + Iterable, + Iterator, + List, + Mapping, + Optional, + Set, + Tuple, + Union, + cast, +) + +from .error import AQMPError +from .message import Message + + +EventNames =3D Union[str, Iterable[str], None] +EventFilter =3D Callable[[Message], bool] + + +class ListenerError(AQMPError): + """ + Generic error class for `EventListener`-related problems. + """ + + +class EventListener: + """ + Selectively listens for events with runtime configurable filtering. + + This class is designed to be directly usable for the most common cases, + but it can be extended to provide more rigorous control. + + :param names: + One or more names of events to listen for. + When not provided, listen for ALL events. + :param event_filter: + An optional event filtering function. + When names are also provided, this acts as a secondary filter. + + When ``names`` and ``event_filter`` are both provided, the names + will be filtered first, and then the filter function will be called + second. The event filter function can assume that the format of the + event is a known format. + """ + def __init__( + self, + names: EventNames =3D None, + event_filter: Optional[EventFilter] =3D None, + ): + # Queue of 'heard' events yet to be witnessed by a caller. + self._queue: 'asyncio.Queue[Message]' =3D asyncio.Queue() + + # Intended as a historical record, NOT a processing queue or backl= og. + self._history: List[Message] =3D [] + + #: Primary event filter, based on one or more event names. + self.names: Set[str] =3D set() + if isinstance(names, str): + self.names.add(names) + elif names is not None: + self.names.update(names) + + #: Optional, secondary event filter. + self.event_filter: Optional[EventFilter] =3D event_filter + + @property + def history(self) -> Tuple[Message, ...]: + """ + A read-only history of all events seen so far. + + This represents *every* event, including those not yet witnessed + via `get()` or ``async for``. It persists between `clear()` + calls and is immutable. + """ + return tuple(self._history) + + async def _get(self) -> Message: + """ + Wait for the very next event in this stream. + + If one is already available, return that one. + """ + return await self._queue.get() + + def accept(self, event: Message) -> bool: + """ + Determine if this listener accepts this event. + + This method determines which events will appear in the stream. + The default implementation simply checks the event against the + list of names and the event_filter to decide if this + `EventListener` accepts a given event. It can be + overridden/extended to provide custom listener behavior. + + User code is not expected to need to invoke this method. + + :param event: The event under consideration. + :return: `True`, if this listener accepts this event. + """ + name_ok =3D (not self.names) or (event['event'] in self.names) + return name_ok and ( + (not self.event_filter) or self.event_filter(event) + ) + + async def put(self, event: Message) -> None: + """ + Conditionally put a new event into the FIFO queue. + + This method is not designed to be invoked from user code, and it + should not need to be overridden. It is a public interface so + that :py:class:`~qmp_protocol.QMP` has an interface by which it + can inform registered listeners of new events. + + The event will be put into the queue if + :py:meth:`~EventListener.accept()` returns `True`. + + :param event: The new event to put into the FIFO queue. + """ + if not self.accept(event): + return + + self._history.append(event) + await self._queue.put(event) + + async def get(self, **kwargs: object) -> Message: + r""" + Wait for an event with optional tertiary filtering. + + :param \*\*kwargs: Optional tertiary filtering criteria. Each + keyword provided is treated as a key belonging to the + event's 'data' field. The value provided is matched against + the candidate event's data members. + + .. warning:: When tertiary filtering criteria are provided, + events that do not match tertiary criteria will be silently + dropped by this listener. All events that were accepted by + the listener will be visible in `history()`. + """ + if not kwargs: + return await self._get() + + def _tertiary_filter(event: Message) -> bool: + data =3D cast(Mapping[str, object], event.get('data', {})) + for key, value in kwargs.items(): + if key not in data: + return False + if data[key] !=3D value: + return False + return True + + async for event in self: + if _tertiary_filter(event): + return event + else: + assert False # Should be impossible to reach. + + def clear(self) -> None: + """ + Clear this listener of all pending events. + + Called when an `EventListener` is being unregistered, this clears = the + pending FIFO queue synchronously. It can be also be used to + manually clear any pending events, if desired. + + .. warning:: + Take care when discarding events. Cleared events will be + silently tossed on the floor. All events that were ever + accepted by this listener are visible in `history()`. + """ + while True: + try: + self._queue.get_nowait() + except asyncio.QueueEmpty: + break + + def __aiter__(self) -> AsyncIterator[Message]: + return self + + async def __anext__(self) -> Message: + """ + Enables the `EventListener` to function as an async iterator. + + It may be used like this: + + .. code:: python + + async for event in listener: + print(event) + + These iterators will never terminate of their own accord; you + must provide break conditions or otherwise prepare to run them + in an `asyncio.Task` that can be cancelled. + """ + return await self._get() + + +class Events: + """ + Events is a mix-in class that adds event functionality to the QMP clas= s. + + It's designed specifically as a mix-in for + :py:class:`~qmp_protocol.QMP`, and it relies upon the class it is + being mixed into having a 'logger' property. + """ + def __init__(self) -> None: + self._listeners: List[EventListener] =3D [] + + #: Default, all-events `EventListener`. + self.events: EventListener =3D EventListener() + self.register_listener(self.events) + + # Parent class needs to have a logger + self.logger: logging.Logger + + async def _event_dispatch(self, msg: Message) -> None: + """ + Given a new event, propagate it to all of the active listeners. + + :param msg: The event to propagate. + """ + for listener in self._listeners: + await listener.put(msg) + + def register_listener(self, listener: EventListener) -> None: + """ + Register and activate an `EventListener`. + + :param listener: The listener to activate. + :raise ListenerError: If the given listener is already registered. + """ + if listener in self._listeners: + raise ListenerError("Attempted to re-register existing listene= r") + self.logger.debug("Registering %s.", str(listener)) + self._listeners.append(listener) + + def remove_listener(self, listener: EventListener) -> None: + """ + Unregister and deactivate an `EventListener`. + + The removed listener will have its pending events cleared via + `clear()`. The listener can be re-registered later when + desired. + + :param listener: The listener to deactivate. + :raise ListenerError: If the given listener is not registered. + """ + if listener =3D=3D self.events: + raise ListenerError("Cannot remove the default listener.") + self.logger.debug("Removing %s.", str(listener)) + listener.clear() + self._listeners.remove(listener) + + @contextmanager + def listen(self, *listeners: EventListener) -> Iterator[None]: + r""" + Context manager: Temporarily listen with an `EventListener`. + + Accepts one or more `EventListener` objects and registers them, + activating them for the duration of the context block. + + `EventListener` objects will have any pending events in their + FIFO queue cleared upon exiting the context block, when they are + deactivated. + + :param \*listeners: One or more EventListeners to activate. + :raise ListenerError: If the given listener(s) are already active. + """ + _added =3D [] + + try: + for listener in listeners: + self.register_listener(listener) + _added.append(listener) + + yield + + finally: + for listener in _added: + self.remove_listener(listener) + + @contextmanager + def listener( + self, + names: EventNames =3D (), + event_filter: Optional[EventFilter] =3D None + ) -> Iterator[EventListener]: + """ + Context manager: Temporarily listen with a new `EventListener`. + + Creates an `EventListener` object and registers it, activating + it for the duration of the context block. + + :param names: + One or more names of events to listen for. + When not provided, listen for ALL events. + :param event_filter: + An optional event filtering function. + When names are also provided, this acts as a secondary filter. + + :return: The newly created and active `EventListener`. + """ + listener =3D EventListener(names, event_filter) + with self.listen(listener): + yield listener --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113593172521.4860308208678; Wed, 30 Jun 2021 21:26:33 -0700 (PDT) Received: from localhost ([::1]:52078 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoHY-0003xB-2G for importer@patchew.org; Thu, 01 Jul 2021 00:26:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42530) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5P-0001So-Oa for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:38395) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5K-0000yA-Qf for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:59 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-270-rt--H_1CN-643SnioJoD-g-1; Thu, 01 Jul 2021 00:13:51 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DCDCE362FB; Thu, 1 Jul 2021 04:13:50 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id E68BD69CB4; Thu, 1 Jul 2021 04:13:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yvCKp+a4gWOikI8Cdi3Sz7v0HoKXlDnKgk4n24Yb9mQ=; b=cQ7nho0u8lQHE50JwXsjMMgBN02evXMXpyYBgtuRLjNhoVvjTL+aXtoaN0idHKunYe6tkv moBL5xF1yGr9P8sQJAiIRiCNBzLzp8oeVv3nbRePU1WToHReq/YslkQl/l+XdDefL0kZ66 GuhqoQc8v8bf1e8RXEk6yn9oMuubXzA= X-MC-Unique: rt--H_1CN-643SnioJoD-g-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 15/20] python/aqmp: add QMP protocol support Date: Thu, 1 Jul 2021 00:13:08 -0400 Message-Id: <20210701041313.1696009-16-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" The star of our show! Add most of the QMP protocol, sans support for actually executing commands. No problem, that happens in the next two commits. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 2 + python/qemu/aqmp/qmp_protocol.py | 257 +++++++++++++++++++++++++++++++ 2 files changed, 259 insertions(+) create mode 100644 python/qemu/aqmp/qmp_protocol.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index ae87436470..68d98cca75 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -25,11 +25,13 @@ from .events import EventListener from .message import Message from .protocol import ConnectError, Runstate +from .qmp_protocol import QMP =20 =20 # The order of these fields impact the Sphinx documentation order. __all__ =3D ( # Classes, most to least important + 'QMP', 'Message', 'EventListener', 'Runstate', diff --git a/python/qemu/aqmp/qmp_protocol.py b/python/qemu/aqmp/qmp_protoc= ol.py new file mode 100644 index 0000000000..5872bfc017 --- /dev/null +++ b/python/qemu/aqmp/qmp_protocol.py @@ -0,0 +1,257 @@ +""" +QMP Protocol Implementation + +This module provides the `QMP` class, which can be used to connect and +send commands to a QMP server such as QEMU. The QMP class can be used to +either connect to a listening server, or used to listen and accept an +incoming connection from that server. +""" + +import logging +from typing import ( + Dict, + List, + Mapping, + Optional, +) + +from .error import ProtocolError +from .events import Events +from .message import Message +from .models import Greeting +from .protocol import AsyncProtocol +from .util import bottom_half, pretty_traceback, upper_half + + +class _WrappedProtocolError(ProtocolError): + """ + Abstract exception class for Protocol errors that wrap an Exception. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + def __init__(self, error_message: str, exc: Exception): + super().__init__(error_message) + self.exc =3D exc + + def __str__(self) -> str: + return f"{self.error_message}: {self.exc!s}" + + +class GreetingError(_WrappedProtocolError): + """ + An exception occurred during the Greeting phase. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + + +class NegotiationError(_WrappedProtocolError): + """ + An exception occurred during the Negotiation phase. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + + +class QMP(AsyncProtocol[Message], Events): + """ + Implements a QMP client connection. + + QMP can be used to establish a connection as either the transport + client or server, though this class always acts as the QMP client. + + :param name: Optional nickname for the connection, used for logging. + + Basic script-style usage looks like this:: + + qmp =3D QMP('my_virtual_machine_name') + await qmp.connect(('127.0.0.1', 1234)) + ... + res =3D await qmp.execute('block-query') + ... + await qmp.disconnect() + + Basic async client-style usage looks like this:: + + class Client: + def __init__(self, name: str): + self.qmp =3D QMP(name) + + async def watch_events(self): + try: + async for event in self.events: + print(f"Event: {event['event']}") + except asyncio.CancelledError: + return + + async def run(self, address=3D'/tmp/qemu.socket'): + await self.qmp.connect(address) + asyncio.create_task(self.watch_events()) + await self.qmp.runstate_changed.wait() + await self.disconnect() + + See `aqmp.events` for more detail on event handling patterns. + """ + #: Logger object used for debugging messages. + logger =3D logging.getLogger(__name__) + + def __init__(self, name: Optional[str] =3D None) -> None: + super().__init__(name) + Events.__init__(self) + + #: Whether or not to await a greeting after establishing a connect= ion. + self.await_greeting: bool =3D True + + #: Whether or not to perform capabilities negotiation upon connect= ion. + #: Implies `await_greeting`. + self.negotiate: bool =3D True + + # Cached Greeting, if one was awaited. + self._greeting: Optional[Greeting] =3D None + + @upper_half + async def _begin_new_session(self) -> None: + """ + Initiate the QMP session. + + Wait for the QMP greeting and perform capabilities negotiation. + + :raise GreetingError: When the greeting is not understood. + :raise NegotiationError: If the negotiation fails. + :raise EOFError: When the server unexpectedly hangs up. + :raise OSError: For underlying stream errors. + """ + if self.await_greeting or self.negotiate: + self._greeting =3D await self._get_greeting() + + if self.negotiate: + await self._negotiate() + + # This will start the reader/writers: + await super()._begin_new_session() + + @upper_half + async def _get_greeting(self) -> Greeting: + """ + :raise GreetingError: When the greeting is not understood. + :raise EOFError: When the server unexpectedly hangs up. + :raise OSError: For underlying stream errors. + + :return: the Greeting object given by the server. + """ + self.logger.debug("Awaiting greeting ...") + + try: + msg =3D await self._recv() + return Greeting(msg) + except (ProtocolError, KeyError, TypeError) as err: + emsg =3D "Did not understand Greeting" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise GreetingError(emsg, err) from err + except BaseException: + # EOFError, OSError, or something unexpected. + emsg =3D "Failed to receive Greeting" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise + + @upper_half + async def _negotiate(self) -> None: + """ + Perform QMP capabilities negotiation. + + :raise NegotiationError: When negotiation fails. + :raise EOFError: When the server unexpectedly hangs up. + :raise OSError: For underlying stream errors. + """ + self.logger.debug("Negotiating capabilities ...") + + arguments: Dict[str, List[str]] =3D {'enable': []} + if self._greeting and 'oob' in self._greeting.QMP.capabilities: + arguments['enable'].append('oob') + msg =3D self.make_execute_msg('qmp_capabilities', arguments=3Dargu= ments) + + # It's not safe to use execute() here, because the reader/writers + # aren't running. AsyncProtocol *requires* that a new session + # does not fail after the reader/writers are running! + try: + await self._send(msg) + reply =3D await self._recv() + assert 'return' in reply + assert 'error' not in reply + except (ProtocolError, AssertionError) as err: + emsg =3D "Negotiation failed" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise NegotiationError(emsg, err) from err + except BaseException: + # EOFError, OSError, or something unexpected. + emsg =3D "Negotiation failed" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise + + @bottom_half + async def _on_message(self, msg: Message) -> None: + """ + Add an incoming message to the appropriate queue/handler. + """ + # Incoming messages are not fully parsed/validated here; + # do only light peeking to know how to route the messages. + + if 'event' in msg: + await self._event_dispatch(msg) + return + + # Below, we assume everything left is an execute/exec-oob response. + # ... Which we'll implement in the next commit! + + @upper_half + @bottom_half + async def _do_recv(self) -> Message: + """ + :raise OSError: When a stream error is encountered. + :raise EOFError: When the stream is at EOF. + :raise ProtocolError: + When the Message is not understood. + See also `Message._deserialize`. + + :return: A single QMP `Message`. + """ + msg_bytes =3D await self._readline() + msg =3D Message(msg_bytes, eager=3DTrue) + return msg + + @upper_half + @bottom_half + def _do_send(self, msg: Message) -> None: + """ + :raise ValueError: JSON serialization failure + :raise TypeError: JSON serialization failure + :raise OSError: When a stream error is encountered. + """ + assert self._writer is not None + self._writer.write(bytes(msg)) + + @upper_half + def _cleanup(self) -> None: + super()._cleanup() + self._greeting =3D None + + @classmethod + def make_execute_msg(cls, cmd: str, + arguments: Optional[Mapping[str, object]] =3D Non= e, + oob: bool =3D False) -> Message: + """ + Create an executable message to be sent later. + + :param cmd: QMP command name. + :param arguments: Arguments (if any). Must be JSON-serializable. + :param oob: If `True`, execute "out of band". + + :return: An executable QMP `Message`. + """ + msg =3D Message({'exec-oob' if oob else 'execute': cmd}) + if arguments is not None: + msg['arguments'] =3D arguments + return msg --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113403812181.48527396422776; Wed, 30 Jun 2021 21:23:23 -0700 (PDT) Received: from localhost ([::1]:43664 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoEU-0006g3-O8 for importer@patchew.org; Thu, 01 Jul 2021 00:23:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42490) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5N-0001JC-7g for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:41594) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5K-0000y8-NA for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:57 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-397-cwWj02WlOxeSG4WSZdIsRQ-1; Thu, 01 Jul 2021 00:13:53 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 01C65804143; Thu, 1 Jul 2021 04:13:52 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0933669CB4; Thu, 1 Jul 2021 04:13:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K+o8mbvF9/lrr7ucl36zrO2mILjZDE5Y2AiqC/Y6US0=; b=Zg3Td8mb9DbU+1zRr3MKauTxlQ6KVOoureuoBw0ay27DAzVWVpwvrBSU0tlS2hBJf2PHma Ey/rruG1TGr9PRmQfosLod9vVUdIVH2zpYt3Zb1m/Dlk2pU4GiF5A4g7dpVL+yz2ahxcee 4tRvENANw4aVSAfPDSopkkpyh9Sbi7U= X-MC-Unique: cwWj02WlOxeSG4WSZdIsRQ-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 16/20] python/aqmp: Add message routing to QMP protocol Date: Thu, 1 Jul 2021 00:13:09 -0400 Message-Id: <20210701041313.1696009-17-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Add the ability to handle and route messages in qmp_protocol.py. The interface for actually sending anything still isn't added until next commit. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/qmp_protocol.py | 98 +++++++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 2 deletions(-) diff --git a/python/qemu/aqmp/qmp_protocol.py b/python/qemu/aqmp/qmp_protoc= ol.py index 5872bfc017..04c8a8cb54 100644 --- a/python/qemu/aqmp/qmp_protocol.py +++ b/python/qemu/aqmp/qmp_protocol.py @@ -7,15 +7,18 @@ incoming connection from that server. """ =20 +# The import workarounds here are fixed in the next commit. +import asyncio # pylint: disable=3Dunused-import # noqa import logging from typing import ( Dict, List, Mapping, Optional, + Union, ) =20 -from .error import ProtocolError +from .error import AQMPError, ProtocolError from .events import Events from .message import Message from .models import Greeting @@ -56,6 +59,53 @@ class NegotiationError(_WrappedProtocolError): """ =20 =20 +class ExecInterruptedError(AQMPError): + """ + Exception raised when an RPC is interrupted. + + This error is raised when an execute() statement could not be + completed. This can occur because the connection itself was + terminated before a reply was received. + + The true cause of the interruption will be available via `disconnect()= `. + """ + + +class _MsgProtocolError(ProtocolError): + """ + Abstract error class for protocol errors that have a `Message` object. + + This Exception class is used for protocol errors where the `Message` + was mechanically understood, but was found to be inappropriate or + malformed. + + :param error_message: Human-readable string describing the error. + :param msg: The QMP `Message` that caused the error. + """ + def __init__(self, error_message: str, msg: Message): + super().__init__(error_message) + #: The received `Message` that caused the error. + self.msg: Message =3D msg + + def __str__(self) -> str: + return "\n".join([ + super().__str__(), + f" Message was: {str(self.msg)}\n", + ]) + + +class ServerParseError(_MsgProtocolError): + """ + The Server sent a `Message` indicating parsing failure. + + i.e. A reply has arrived from the server, but it is missing the "ID" + field, indicating a parsing error. + + :param error_message: Human-readable string describing the error. + :param msg: The QMP `Message` that caused the error. + """ + + class QMP(AsyncProtocol[Message], Events): """ Implements a QMP client connection. @@ -98,6 +148,9 @@ async def run(self, address=3D'/tmp/qemu.socket'): #: Logger object used for debugging messages. logger =3D logging.getLogger(__name__) =20 + # Type alias for pending execute() result items + _PendingT =3D Union[Message, ExecInterruptedError] + def __init__(self, name: Optional[str] =3D None) -> None: super().__init__(name) Events.__init__(self) @@ -112,6 +165,9 @@ def __init__(self, name: Optional[str] =3D None) -> Non= e: # Cached Greeting, if one was awaited. self._greeting: Optional[Greeting] =3D None =20 + # Incoming RPC reply messages + self._pending: Dict[str, 'asyncio.Queue[QMP._PendingT]'] =3D {} + @upper_half async def _begin_new_session(self) -> None: """ @@ -191,10 +247,27 @@ async def _negotiate(self) -> None: self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) raise =20 + @bottom_half + async def _bh_disconnect(self, force: bool =3D False) -> None: + await super()._bh_disconnect(force) + + if self._pending: + self.logger.debug("Cancelling pending executions") + keys =3D self._pending.keys() + for key in keys: + self.logger.debug("Cancelling execution '%s'", key) + self._pending[key].put_nowait( + ExecInterruptedError("Disconnected") + ) + + self.logger.debug("QMP Disconnected.") + @bottom_half async def _on_message(self, msg: Message) -> None: """ Add an incoming message to the appropriate queue/handler. + + :raise ServerParseError: When Message has no 'event' nor 'id' memb= er """ # Incoming messages are not fully parsed/validated here; # do only light peeking to know how to route the messages. @@ -204,7 +277,27 @@ async def _on_message(self, msg: Message) -> None: return =20 # Below, we assume everything left is an execute/exec-oob response. - # ... Which we'll implement in the next commit! + + if 'id' not in msg: + # This is (very likely) a server parsing error. + # It doesn't inherently belong to any pending execution. + # Instead of performing clever recovery, just terminate. + # See "NOTE" in qmp-spec.txt, section 2.4.2 + raise ServerParseError("Server sent a message without an ID," + " indicating parse failure.", msg) + + assert 'id' in msg + exec_id =3D str(msg['id']) + + if exec_id not in self._pending: + # qmp-spec.txt, section 2.4: + # 'Clients should drop all the responses + # that have an unknown "id" field.' + self.logger.warning("Unknown ID '%s', message dropped.", exec_= id) + self.logger.debug("Unroutable message: %s", str(msg)) + return + + await self._pending[exec_id].put(msg) =20 @upper_half @bottom_half @@ -237,6 +330,7 @@ def _do_send(self, msg: Message) -> None: def _cleanup(self) -> None: super()._cleanup() self._greeting =3D None + assert not self._pending =20 @classmethod def make_execute_msg(cls, cmd: str, --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113516485162.7709221440124; Wed, 30 Jun 2021 21:25:16 -0700 (PDT) Received: from localhost ([::1]:47934 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoGJ-00017R-0C for importer@patchew.org; Thu, 01 Jul 2021 00:25:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42504) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5O-0001Nn-Fz for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:32320) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5L-0000yk-QO for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:58 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-406-WNHw-4kfNhaCPtIgcb7THw-1; Thu, 01 Jul 2021 00:13:53 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1B6B8804141; Thu, 1 Jul 2021 04:13:53 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2361C69CB4; Thu, 1 Jul 2021 04:13:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZZCoZ9GLFR+F15z9bX1ZZpaWNtGgb8XgD/xhrhFqTf4=; b=G+QdOkTQlxmErlfqNi4pWZ+RhickeSK7zHRrbT7EIyI/RF9yWIVNlDerbkLsls348NEyRK fdPaunMotKQ8o+7zE85vBDFvSI8LnYByKJ9XejvwLdsYqSQMI4T2NaS5FDzPAUf5uYwyFi 6EvOAOarlT0AEPa+og6LcTdQRMYTIxI= X-MC-Unique: WNHw-4kfNhaCPtIgcb7THw-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 17/20] python/aqmp: add execute() interfaces Date: Thu, 1 Jul 2021 00:13:10 -0400 Message-Id: <20210701041313.1696009-18-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Add execute() and execute_msg(). _execute() is split into _issue() and _reply() halves so that hypothetical subclasses of QMP that want to support different execution paradigms can do so. I anticipate a synchronous interface may have need of separating the send/reply phases. However, I do not wish to expose that interface here and want to actively discourage it, so they remain private interfaces. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 4 +- python/qemu/aqmp/qmp_protocol.py | 203 +++++++++++++++++++++++++++++-- 2 files changed, 199 insertions(+), 8 deletions(-) diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 68d98cca75..5cd7df87c6 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -25,7 +25,7 @@ from .events import EventListener from .message import Message from .protocol import ConnectError, Runstate -from .qmp_protocol import QMP +from .qmp_protocol import QMP, ExecInterruptedError, ExecuteError =20 =20 # The order of these fields impact the Sphinx documentation order. @@ -39,6 +39,8 @@ # Exceptions, most generic to most explicit 'AQMPError', 'ConnectError', + 'ExecuteError', + 'ExecInterruptedError', =20 # Niche topics 'MultiException', diff --git a/python/qemu/aqmp/qmp_protocol.py b/python/qemu/aqmp/qmp_protoc= ol.py index 04c8a8cb54..3c16cdc213 100644 --- a/python/qemu/aqmp/qmp_protocol.py +++ b/python/qemu/aqmp/qmp_protocol.py @@ -7,8 +7,7 @@ incoming connection from that server. """ =20 -# The import workarounds here are fixed in the next commit. -import asyncio # pylint: disable=3Dunused-import # noqa +import asyncio import logging from typing import ( Dict, @@ -21,8 +20,8 @@ from .error import AQMPError, ProtocolError from .events import Events from .message import Message -from .models import Greeting -from .protocol import AsyncProtocol +from .models import ErrorResponse, Greeting +from .protocol import AsyncProtocol, Runstate, require from .util import bottom_half, pretty_traceback, upper_half =20 =20 @@ -59,11 +58,32 @@ class NegotiationError(_WrappedProtocolError): """ =20 =20 +class ExecuteError(AQMPError): + """ + Exception raised by `QMP.execute()` on RPC failure. + + :param error_response: The RPC error response object. + :param sent: The sent RPC message that caused the failure. + :param received: The raw RPC error reply received. + """ + def __init__(self, error_response: ErrorResponse, + sent: Message, received: Message): + super().__init__(error_response.error.desc) + #: The sent `Message` that caused the failure + self.sent: Message =3D sent + #: The received `Message` that indicated failure + self.received: Message =3D received + #: The parsed error response + self.error: ErrorResponse =3D error_response + #: The QMP error class + self.error_class: str =3D error_response.error.class_ + + class ExecInterruptedError(AQMPError): """ - Exception raised when an RPC is interrupted. + Exception raised by `execute()` (et al) when an RPC is interrupted. =20 - This error is raised when an execute() statement could not be + This error is raised when an `execute()` statement could not be completed. This can occur because the connection itself was terminated before a reply was received. =20 @@ -106,6 +126,27 @@ class ServerParseError(_MsgProtocolError): """ =20 =20 +class BadReplyError(_MsgProtocolError): + """ + An execution reply was successfully routed, but not understood. + + If a QMP message is received with an 'id' field to allow it to be + routed, but is otherwise malformed, this exception will be raised. + + A reply message is malformed if it is missing either the 'return' or + 'error' keys, or if the 'error' value has missing keys or members of + the wrong type. + + :param error_message: Human-readable string describing the error. + :param msg: The malformed reply that was received. + :param sent: The message that was sent that prompted the error. + """ + def __init__(self, error_message: str, msg: Message, sent: Message): + super().__init__(error_message, msg) + #: The sent `Message` that caused the failure + self.sent =3D sent + + class QMP(AsyncProtocol[Message], Events): """ Implements a QMP client connection. @@ -165,6 +206,9 @@ def __init__(self, name: Optional[str] =3D None) -> Non= e: # Cached Greeting, if one was awaited. self._greeting: Optional[Greeting] =3D None =20 + # Command ID counter + self._execute_id =3D 0 + # Incoming RPC reply messages self._pending: Dict[str, 'asyncio.Queue[QMP._PendingT]'] =3D {} =20 @@ -332,12 +376,136 @@ def _cleanup(self) -> None: self._greeting =3D None assert not self._pending =20 + @upper_half + def _get_exec_id(self) -> str: + exec_id =3D f"__aqmp#{self._execute_id:05d}" + self._execute_id +=3D 1 + return exec_id + + @upper_half + async def _issue(self, msg: Message) -> str: + """ + Issue a QMP `Message` and do not wait for a reply. + + :param msg: The QMP `Message` to send to the server. + + :return: The ID of the `Message` sent. + """ + assert 'id' in msg + assert isinstance(msg['id'], str) + msg_id =3D msg['id'] + + queue: 'asyncio.Queue[QMP._PendingT]' =3D asyncio.Queue(maxsize=3D= 1) + self._pending[msg_id] =3D queue + await self._outgoing.put(msg) + + return msg_id + + @upper_half + async def _reply(self, msg_id: str) -> Message: + """ + Await a reply to a previously issued QMP message. + + :param msg_id: The ID of the previously issued message. + + :return: The reply from the server. + :raise ExecInterruptedError: + When the reply could not be retrieved because the connection + was lost, or some other problem. + """ + queue =3D self._pending[msg_id] + result =3D await queue.get() + + try: + if isinstance(result, ExecInterruptedError): + raise result + return result + finally: + del self._pending[msg_id] + + @upper_half + async def _execute(self, msg: Message, assign_id: bool =3D True) -> Me= ssage: + """ + Send a QMP `Message` to the server and await a reply. + + This method *assumes* you are sending some kind of an execute + statement that *will* receive a reply. + + An execution ID will be assigned if assign_id is `True`. It can be + disabled, but this requires that an ID is manually assigned + instead. For manually assigned IDs, you must not use the string + '__aqmp#' anywhere in the ID. + + :param msg: The QMP `Message` to execute. + :param assign_id: If True, assign a new execution ID. + + :return: Execution reply from the server. + :raise ExecInterruptedError: + When the reply could not be retrieved because the connection + was lost, or some other problem. + """ + if assign_id: + msg['id'] =3D self._get_exec_id() + else: + assert 'id' in msg + assert isinstance(msg['id'], str) + assert '__aqmp#' not in msg['id'] + + exec_id =3D await self._issue(msg) + return await self._reply(exec_id) + + @upper_half + @require(Runstate.RUNNING) + async def execute_msg(self, msg: Message) -> object: + """ + Execute a QMP command and return its value. + + :param msg: The QMP `Message` to execute. + + :return: + The command execution return value from the server. The type of + object returned depends on the command that was issued, + though most in QEMU return a `dict`. + :raise ValueError: + If the QMP `Message` does not have either the 'execute' or + 'exec-oob' fields set. + :raise ExecuteError: When the server returns an error response. + :raise ExecInterruptedError: if the connection was terminated earl= y. + """ + if not ('execute' in msg or 'exec-oob' in msg): + raise ValueError("Requires 'execute' or 'exec-oob' message") + + # Copy the Message so that the ID assigned by _execute() is + # local to this method; allowing the ID to be seen in raised + # Exceptions but without modifying the caller's held copy. + msg =3D Message(msg) + reply =3D await self._execute(msg) + + if 'error' in reply: + try: + error_response =3D ErrorResponse(reply) + except (KeyError, TypeError) as err: + # Error response was malformed. + raise BadReplyError( + "QMP error reply is malformed", reply, msg, + ) from err + + raise ExecuteError(error_response, msg, reply) + + if 'return' not in reply: + raise BadReplyError( + "QMP reply is missing a 'error' or 'return' member", + reply, msg, + ) + + return reply['return'] + @classmethod def make_execute_msg(cls, cmd: str, arguments: Optional[Mapping[str, object]] =3D Non= e, oob: bool =3D False) -> Message: """ - Create an executable message to be sent later. + Create an executable message to be sent by `execute_msg` later. =20 :param cmd: QMP command name. :param arguments: Arguments (if any). Must be JSON-serializable. @@ -349,3 +517,24 @@ def make_execute_msg(cls, cmd: str, if arguments is not None: msg['arguments'] =3D arguments return msg + + @upper_half + async def execute(self, cmd: str, + arguments: Optional[Mapping[str, object]] =3D None, + oob: bool =3D False) -> object: + """ + Execute a QMP command and return its value. + + :param cmd: QMP command name. + :param arguments: Arguments (if any). Must be JSON-serializable. + :param oob: If `True`, execute "out of band". + + :return: + The command execution return value from the server. The type of + object returned depends on the command that was issued, + though most in QEMU return a `dict`. + :raise ExecuteError: When the server returns an error response. + :raise ExecInterruptedError: if the connection was terminated earl= y. + """ + msg =3D self.make_execute_msg(cmd, arguments, oob=3Doob) + return await self.execute_msg(msg) --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113553856154.20549325026593; Wed, 30 Jun 2021 21:25:53 -0700 (PDT) Received: from localhost ([::1]:49588 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoGu-0002Jb-Rx for importer@patchew.org; Thu, 01 Jul 2021 00:25:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42546) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5R-0001ZS-Ph for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:56989) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5M-0000zD-NF for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:01 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-424-u7ieV3aLNryKS5XL_e0Umg-1; Thu, 01 Jul 2021 00:13:55 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2F31836308; Thu, 1 Jul 2021 04:13:54 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3D010604CC; Thu, 1 Jul 2021 04:13:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112836; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R9oVPt2wrGnnSHtYvijBTJbfNDXJHXzMXH4ecL6YbSQ=; b=YmtXwaNbKdsP7tG/4vZlM37YJu88Q4gGDZvTKncWO3nEU10V0pUjZK/xarw2sUqPnLSz8B 9H/luS+XRT9MTFc7ys0FnQF0LLSSfOspe621M1tGrx5Xr0bms8bKPM3/i6eSVKYUXClOD1 OxhYmdLswRtYkIcvmna654Kgd2s7y6w= X-MC-Unique: u7ieV3aLNryKS5XL_e0Umg-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 18/20] python/aqmp: add _raw() execution interface Date: Thu, 1 Jul 2021 00:13:11 -0400 Message-Id: <20210701041313.1696009-19-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This is added in anticipation of wanting it for a synchronous wrapper for the iotest interface. Normally, execute() and execute_msg() both raise QMP errors in the form of Python exceptions. Many iotests expect the entire reply as-is. To reduce churn there, add a private execution interface that will ease transition churn. However, I do not wish to encourage its use, so it will remain a private interface. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/qmp_protocol.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/python/qemu/aqmp/qmp_protocol.py b/python/qemu/aqmp/qmp_protoc= ol.py index 3c16cdc213..36baef9fb3 100644 --- a/python/qemu/aqmp/qmp_protocol.py +++ b/python/qemu/aqmp/qmp_protocol.py @@ -454,6 +454,31 @@ async def _execute(self, msg: Message, assign_id: bool= =3D True) -> Message: exec_id =3D await self._issue(msg) return await self._reply(exec_id) =20 + @upper_half + @require(Runstate.RUNNING) + async def _raw( + self, + msg: Union[Message, Mapping[str, object], bytes] + ) -> Message: + """ + Issue a fairly raw `Message` to the QMP server and await a reply. + + An AQMP execution ID will be assigned, so it isn't *truly* raw. + + :param msg: + A Message to send to the server. It may be a `Message`, any + Mapping (including Dict), or raw bytes. + + :return: Execution reply from the server. + :raise ExecInterruptedError: + When the reply could not be retrieved because the connection + was lost, or some other problem. + """ + # 1. convert generic Mapping or bytes to a QMP Message + # 2. copy Message objects so that we assign an ID only to the copy. + msg =3D Message(msg) + return await self._execute(msg) + @upper_half @require(Runstate.RUNNING) async def execute_msg(self, msg: Message) -> object: --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113130515871.4487083813095; Wed, 30 Jun 2021 21:18:50 -0700 (PDT) Received: from localhost ([::1]:55156 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoA5-0003t3-C5 for importer@patchew.org; Thu, 01 Jul 2021 00:18:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42552) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5S-0001bm-CO for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:02 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:20834) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5P-00011v-Cv for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:02 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-412-jCtHCQk-PX6y8e78xXfpsg-1; Thu, 01 Jul 2021 00:13:56 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 468D6100C611; Thu, 1 Jul 2021 04:13:55 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5124269CB4; Thu, 1 Jul 2021 04:13:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6UUZKsnxjk479MmTFaykmhlePb3zu5VVZKiuPKcifhU=; b=OyBGHbxZVdPI7dJYXENlfyB4fN8e5dhMGCOzjCdajI/mGtAfm6uPgAN2+B3uMHpkFWsVf/ VOPfP+d0fXRjG6RCkFsdI1Rvyzj/hqAzY79Gum5pSrBo25DMVbPhg6jMebmxmdabBajYBU p5SzHrsgKq03ZR5uN8xcoorCvwpSGFU= X-MC-Unique: jCtHCQk-PX6y8e78xXfpsg-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 19/20] python/aqmp: add asyncio_run compatibility wrapper Date: Thu, 1 Jul 2021 00:13:12 -0400 Message-Id: <20210701041313.1696009-20-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Merely as a convenience for users stuck on Python 3.6. It isn't used by the library itself. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/util.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py index 2311be5893..356323ac70 100644 --- a/python/qemu/aqmp/util.py +++ b/python/qemu/aqmp/util.py @@ -109,6 +109,26 @@ async def wait_task_done(task: Optional['asyncio.Futur= e[Any]']) -> None: break =20 =20 +def asyncio_run(coro: Coroutine[Any, Any, T]) -> T: + """ + Python 3.6-compatible `asyncio.run` wrapper. + + :param coro: A coroutine to execute now. + :return: The return value from the coroutine. + """ + # Python 3.7+ + if hasattr(asyncio, 'run'): + # pylint: disable=3Dno-member + return asyncio.run(coro) # type: ignore + + # Python 3.6 + loop =3D asyncio.get_event_loop() + ret =3D loop.run_until_complete(coro) + loop.close() + + return ret + + def pretty_traceback(prefix: str =3D " | ") -> str: """ Formats the current traceback, indented to provide visual distinction. --=20 2.31.1 From nobody Sat May 18 23:23:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625113222262400.8714813124874; Wed, 30 Jun 2021 21:20:22 -0700 (PDT) Received: from localhost ([::1]:33352 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyoBZ-000894-9V for importer@patchew.org; Thu, 01 Jul 2021 00:20:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42584) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5W-0001ro-KQ for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:28515) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo5U-00016c-Qy for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:14:06 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-477-4Eg0MGV1OMu7H0euwxkktA-1; Thu, 01 Jul 2021 00:14:00 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8FBE0100C611; Thu, 1 Jul 2021 04:13:59 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6849869CB4; Thu, 1 Jul 2021 04:13:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HjWAmsfOhxEV/4afQkaj7ZO9XgmR6HHngbUPDQA8fNU=; b=UoFmp5HbZOm2pmTh5uRcANB6EtqW83FvH0lxGidP0MkONGsjdmkZTHSe/GHts3hqbjZlJb 80tzGe3aRhHZeeBt2hKCd1cxU6i8SW714LJSHqBpULaCfEdcF1/LXTke81Mf2cp5VBP/Fr lUTwININMPFiC+vZ09xxEgWeGO2dn5E= X-MC-Unique: 4Eg0MGV1OMu7H0euwxkktA-1 From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 20/20] python/aqmp: add scary message Date: Thu, 1 Jul 2021 00:13:13 -0400 Message-Id: <20210701041313.1696009-21-jsnow@redhat.com> In-Reply-To: <20210701041313.1696009-1-jsnow@redhat.com> References: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Add a warning whenever AQMP is used to steer people gently away from using it for the time-being. Signed-off-by: John Snow Acked-by: Stefan Hajnoczi --- python/qemu/aqmp/__init__.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 5cd7df87c6..f85500e0a2 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -21,6 +21,8 @@ # This work is licensed under the terms of the GNU GPL, version 2. See # the COPYING file in the top-level directory. =20 +import warnings + from .error import AQMPError, MultiException from .events import EventListener from .message import Message @@ -28,6 +30,18 @@ from .qmp_protocol import QMP, ExecInterruptedError, ExecuteError =20 =20 +_WMSG =3D """ + +The Asynchronous QMP library is currently in development and its API +should be considered highly fluid and subject to change. It should +not be used by any other scripts checked into the QEMU tree. + +Proceed with caution! +""" + +warnings.warn(_WMSG, FutureWarning) + + # The order of these fields impact the Sphinx documentation order. __all__ =3D ( # Classes, most to least important --=20 2.31.1