From nobody Tue Apr 30 06:06:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1489713396769475.8459570576532; Thu, 16 Mar 2017 18:16:36 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 407CA15F56A; Fri, 17 Mar 2017 01:16:36 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 13CA76046B; Fri, 17 Mar 2017 01:16:36 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id C25761853D02; Fri, 17 Mar 2017 01:16:35 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v2H1GYNX014001 for ; Thu, 16 Mar 2017 21:16:34 -0400 Received: by smtp.corp.redhat.com (Postfix) id 502955C3FD; Fri, 17 Mar 2017 01:16:34 +0000 (UTC) Received: from mx1.redhat.com (ext-mx08.extmail.prod.ext.phx2.redhat.com [10.5.110.32]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 360225C3FA for ; Fri, 17 Mar 2017 01:16:32 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D8472C057FA9 for ; Fri, 17 Mar 2017 01:16:30 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 5D32120646 for ; Thu, 16 Mar 2017 21:16:29 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute5.internal (MEProxy); Thu, 16 Mar 2017 21:16:29 -0400 Received: from localhost.localdomain (exit1.ipredator.se [197.231.221.211]) by mail.messagingengine.com (Postfix) with ESMTPA id CB266241D8 for ; Thu, 16 Mar 2017 21:16:28 -0400 (EDT) Received: by localhost.localdomain (Postfix, from userid 1000) id 40D231C762; Fri, 17 Mar 2017 02:16:25 +0100 (CET) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 407CA15F56A Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 407CA15F56A Authentication-Results: mx1.redhat.com; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="V9deQYQ3" DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D8472C057FA9 Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=none smtp.mailfrom=woju@invisiblethingslab.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D8472C057FA9 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=plw8WYc9uWdtVnuO2T 19MN0IMAM=; b=V9deQYQ3+jrbzxvrLfcVwyIG8Gn0Wenf5/PkD60yeG6h16HLQg uI6OBcqZ6K8yH0BsDTMZzKt7GJug/s2oyMvDgvEOnrKgEaE5dhF6iYdWSqQ+aG7Y 9Hx2DFCAY+VcgMhy8niQ7zfAI1WcQ5s7bFWcHmq+qufjJYACErFqtfxGflzV7nQF fra/sD9dtV+wuPvfjZ6ha8VykWxN1/2+YyiGXyJ4zorJEwZxVER+BtyGVdLgQXHb c2+T2Lp5UKkfduipB0o7URVRCF+3farAgYyAyfw07qdHIZld/Ho71Yfw2o6AC3j0 jOSjHeTdNTYSXD/ddikygsaU83x5CnILGa4Q== X-ME-Sender: X-Sasl-enc: ED4wqdI7gICLGvObl+6AFENSZ3nhq6eU6zB2U2BLcKq9 1489713388 Date: Fri, 17 Mar 2017 02:16:25 +0100 From: Wojtek Porczyk To: libvirt-list@redhat.com Message-ID: <146cb29018f3eb5a571cfac147030f8285ac74f4.1489712246.git.woju@invisiblethingslab.com> Mail-Followup-To: libvirt-list@redhat.com References: MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 203 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 17 Mar 2017 01:16:31 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 17 Mar 2017 01:16:31 +0000 (UTC) for IP:'66.111.4.25' DOMAIN:'out1-smtp.messagingengine.com' HELO:'out1-smtp.messagingengine.com' FROM:'woju@invisiblethingslab.com' RCPT:'' X-RedHat-Spam-Score: -0.32 (BAYES_50, DCC_REPUT_00_12, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL) 66.111.4.25 out1-smtp.messagingengine.com 66.111.4.25 out1-smtp.messagingengine.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.32 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v1 1/2] Allow for ff callbacks to be called by custom event implementations X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============6871087044687516842==" Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 17 Mar 2017 01:16:36 +0000 (UTC) X-Zoho-Virus-Status: 1 X-ZohoMail: RSF_0 Z_629925259 SPT_0 --===============6871087044687516842== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="xHFwDpU9dbj6ez1V" Content-Disposition: inline --xHFwDpU9dbj6ez1V Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable The documentation says: > If the opaque user data requires free'ing when the handle is > unregistered, then a 2nd callback can be supplied for this purpose. > This callback needs to be invoked from a clean stack. If 'ff' > callbacks are invoked directly from the virEventRemoveHandleFunc they > will likely deadlock in libvirt. And they did deadlock. In removeTimeout too. Now we supply a custom function to pick it from the opaque blob and fire. Signed-off-by: Wojtek Porczyk --- libvirt-override.c | 36 ++++++++++++------------------------ libvirt-override.py | 39 +++++++++++++++++++++++++++++++++++++++ sanitytest.py | 5 +++-- 3 files changed, 54 insertions(+), 26 deletions(-) diff --git a/libvirt-override.c b/libvirt-override.c index 9e40f00..37f7ee2 100644 --- a/libvirt-override.c +++ b/libvirt-override.c @@ -5223,6 +5223,9 @@ libvirt_virEventAddHandleFunc(int fd, =20 VIR_PY_TUPLE_SET_GOTO(pyobj_args, 3, cb_args, cleanup); =20 + /* If changing contents of the opaque object, please also change + * virEventExecuteFFCallback() in libvirt-override.py + */ VIR_PY_TUPLE_SET_GOTO(cb_args, 0, libvirt_virEventHandleCallbackWrap(c= b), cleanup); VIR_PY_TUPLE_SET_GOTO(cb_args, 1, libvirt_virVoidPtrWrap(opaque), clea= nup); VIR_PY_TUPLE_SET_GOTO(cb_args, 2, libvirt_virFreeCallbackWrap(ff), cle= anup); @@ -5292,20 +5295,11 @@ libvirt_virEventRemoveHandleFunc(int watch) VIR_PY_TUPLE_SET_GOTO(pyobj_args, 0, libvirt_intWrap(watch), cleanup); =20 result =3D PyEval_CallObject(removeHandleObj, pyobj_args); - if (!result) { + if (result) { + retval =3D 0; + } else { PyErr_Print(); PyErr_Clear(); - } else if (!PyTuple_Check(result) || PyTuple_Size(result) !=3D 3) { - DEBUG("%s: %s must return opaque obj registered with %s" - "to avoid leaking libvirt memory\n", - __FUNCTION__, NAME(removeHandle), NAME(addHandle)); - } else { - opaque =3D PyTuple_GetItem(result, 1); - ff =3D PyTuple_GetItem(result, 2); - cff =3D PyvirFreeCallback_Get(ff); - if (cff) - (*cff)(PyvirVoidPtr_Get(opaque)); - retval =3D 0; } =20 cleanup: @@ -5350,6 +5344,9 @@ libvirt_virEventAddTimeoutFunc(int timeout, =20 VIR_PY_TUPLE_SET_GOTO(pyobj_args, 2, cb_args, cleanup); =20 + /* If changing contents of the opaque object, please also change + * virEventExecuteFFCallback() in libvirt-override.py + */ VIR_PY_TUPLE_SET_GOTO(cb_args, 0, libvirt_virEventTimeoutCallbackWrap(= cb), cleanup); VIR_PY_TUPLE_SET_GOTO(cb_args, 1, libvirt_virVoidPtrWrap(opaque), clea= nup); VIR_PY_TUPLE_SET_GOTO(cb_args, 2, libvirt_virFreeCallbackWrap(ff), cle= anup); @@ -5416,20 +5413,11 @@ libvirt_virEventRemoveTimeoutFunc(int timer) VIR_PY_TUPLE_SET_GOTO(pyobj_args, 0, libvirt_intWrap(timer), cleanup); =20 result =3D PyEval_CallObject(removeTimeoutObj, pyobj_args); - if (!result) { + if (result) { + retval =3D 0; + } else { PyErr_Print(); PyErr_Clear(); - } else if (!PyTuple_Check(result) || PyTuple_Size(result) !=3D 3) { - DEBUG("%s: %s must return opaque obj registered with %s" - "to avoid leaking libvirt memory\n", - __FUNCTION__, NAME(removeTimeout), NAME(addTimeout)); - } else { - opaque =3D PyTuple_GetItem(result, 1); - ff =3D PyTuple_GetItem(result, 2); - cff =3D PyvirFreeCallback_Get(ff); - if (cff) - (*cff)(PyvirVoidPtr_Get(opaque)); - retval =3D 0; } =20 cleanup: diff --git a/libvirt-override.py b/libvirt-override.py index 63f8ecb..3d09d63 100644 --- a/libvirt-override.py +++ b/libvirt-override.py @@ -16,6 +16,7 @@ except ImportError: if str(cyg_e).count("No module named"): raise lib_e =20 +import ctypes import types =20 # The root of all libvirt errors. @@ -211,3 +212,41 @@ def virEventAddTimeout(timeout, cb, opaque): ret =3D libvirtmod.virEventAddTimeout(timeout, cbData) if ret =3D=3D -1: raise libvirtError ('virEventAddTimeout() failed') return ret + + +# +# a caller for the ff callbacks for custom event loop implementations +# + +ctypes.pythonapi.PyCapsule_GetPointer.restype =3D ctypes.c_void_p +ctypes.pythonapi.PyCapsule_GetPointer.argtypes =3D ( + ctypes.py_object, ctypes.c_char_p) + +_virFreeCallback =3D ctypes.CFUNCTYPE(None, ctypes.c_void_p) + +def virEventExecuteFFCallback(opaque): + """ + Execute callback which frees the opaque buffer + + @opaque: the opaque object passed to addHandle or addTimeout + + WARNING: This function should not be called from any call by libvirt's + core. It will most probably cause deadlock in C-level libvirt code. + Instead it should be scheduled and called from implementation's stack. + + See https://libvirt.org/html/libvirt-libvirt-event.html#virEventAddHan= dleFunc + for more information. + + This function is not dependent on any event loop implementation. + """ + # The opaque object is really a 3-tuple, which contains a the real opa= que + # pointer and the ff callback, both of which are inside PyCapsules. If= not + # specified, the ff may be None. See libvirt-override.c. + + dummy, caps_opaque, caps_ff =3D opaque + ff =3D _virFreeCallback(ctypes.pythonapi.PyCapsule_GetPointer( + caps_ff, "virFreeCallback".encode("ascii"))) + if ff: + real_opaque =3D ctypes.pythonapi.PyCapsule_GetPointer( + caps_opaque, "void*".encode("ascii")) + ff(real_opaque) diff --git a/sanitytest.py b/sanitytest.py index a140ba2..6548831 100644 --- a/sanitytest.py +++ b/sanitytest.py @@ -345,11 +345,12 @@ for name in sorted(finalklassmap): =20 # Phase 6: Validate that every python API has a corresponding C API for klass in gotfunctions: - if klass =3D=3D "libvirtError": + if klass in ("libvirtError", "virEventAsyncIOImpl"): continue for func in sorted(gotfunctions[klass]): # These are pure python methods with no C APi - if func in ["connect", "getConnect", "domain", "getDomain"]: + if func in ["connect", "getConnect", "domain", "getDomain", + "virEventRegisterAsyncIOImpl"]: continue =20 key =3D "%s.%s" % (klass, func) --=20 2.5.5 --xHFwDpU9dbj6ez1V Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYyzjoAAoJEL9r2TIQOiNRM7UP/jckV5JYEQKil4h9+fT/e1iU enKIBY63OWuHi+r0KEbmsfgZ2OvH2I2Mj/xs7k4HbntcPZAvhZM3fHHU5RQJOvjA Bo8ICiApeq2bIxmRFSzJMhV6eQUpuyzcclqJ71ScVZ6rKL3GbJ07X+YKaSEhR2Gg k2/cg6G5Y7np5qGzZSSAJgtrruOLYnkElQhcYL2fY80guWVRVok32pyWu28eru9m oFnmtzHfmHp3lDT8Br+6xGLBYelJbW+7RBbfpTMQj0HIpiqUXvjYw31/vNbmXttj ch9W9Q0+uTZ5khgxwWvvDlpCnEP8JIe7UEBBIgmyN26R19ZA3OkRgFl+RKzjFOPh qjn3N7n45EieOs/lrmGSEWMiWSAty1VReQF2tF2Jm/8rbEbPKxHgTTeCFVzCbHZh K+5mjzsMf2kW179w5VhRfGBA+EtD4Iskz3xaeQfXU3XTLIHZu+rxckbR6ySKi04u YuM5RaSiPNpn+RkqBWGreB1je7qcvetlSbakffW/NoYt1iZstsGHbFBcy62ixuPN JdZ/g1wGnDutU5pC9jaBWhispSiJAQeTpkeKgfZmuRfB7+rFYwRDl1s1YHyfTity bP2Jc3dxvbA+qp/PZMmzfK4Uy2hR84TQsc5Ng5Ck1aXwH2AelzNktAUhT8e7rWvz XK14e5OlIgRjEeFCuZ/m =/voh -----END PGP SIGNATURE----- --xHFwDpU9dbj6ez1V-- --===============6871087044687516842== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list --===============6871087044687516842==-- From nobody Tue Apr 30 06:06:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 148971341558488.86499075633208; Thu, 16 Mar 2017 18:16:55 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D2ADC437F44; Fri, 17 Mar 2017 01:16:54 +0000 (UTC) Received: from colo-mx.corp.redhat.com (unknown [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A863817964; Fri, 17 Mar 2017 01:16:54 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 4E5E34BB75; Fri, 17 Mar 2017 01:16:54 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v2H1GqoB014026 for ; Thu, 16 Mar 2017 21:16:52 -0400 Received: by smtp.corp.redhat.com (Postfix) id D15D75C3FD; Fri, 17 Mar 2017 01:16:52 +0000 (UTC) Received: from mx1.redhat.com (ext-mx03.extmail.prod.ext.phx2.redhat.com [10.5.110.27]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C7FC95C3FA for ; Fri, 17 Mar 2017 01:16:51 +0000 (UTC) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2EFF6804ED for ; Fri, 17 Mar 2017 01:16:43 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id B133120646 for ; Thu, 16 Mar 2017 21:16:41 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute5.internal (MEProxy); Thu, 16 Mar 2017 21:16:41 -0400 Received: from localhost.localdomain (exit1.ipredator.se [197.231.221.211]) by mail.messagingengine.com (Postfix) with ESMTPA id 0C3CA241D8 for ; Thu, 16 Mar 2017 21:16:41 -0400 (EDT) Received: by localhost.localdomain (Postfix, from userid 1000) id 4DCC01C764; Fri, 17 Mar 2017 02:16:37 +0100 (CET) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D2ADC437F44 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=libvir-list-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D2ADC437F44 Authentication-Results: mx1.redhat.com; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="KiJW0DFO" DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 2EFF6804ED Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=none smtp.mailfrom=woju@invisiblethingslab.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 2EFF6804ED DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=iiYMT6Qtxmu3QNELnX /UeM4PAWk=; b=KiJW0DFOs8xNMDg4YD0O9uwd48ZLfDwKzqXlHAxVUNrULWtoSP SyHC00lC8EFn3nWqiqqfI7+PasHgV1wRX9lj0NfgE20I2vTALdN2HwyV4/muVtbT EJMQftHtFNoqi1tEzrRJfZbxNMNq/8l20ppy7Bhl1Mil32ukqMGTd4plteXXSliW ReowQ9G6JH8431qwe1epYs48x7xEw0cEEhWlLOjLHkyDmtSGQA0An4miySGK3Ln1 luKmE8DqwxzcSKXbR4E9bBGa6ic+v2msE9py1tMXMGgSLDYvNK2+WUaThQE1UMk2 ws6VSSNFOYWkLT5fiXlSckvZE8bmMCIi1hdA== X-ME-Sender: X-Sasl-enc: Lte9l1nzLQdlQEJiVpmz5bpjoGNSlXlUzzbIgqBuKvIr 1489713401 Date: Fri, 17 Mar 2017 02:16:37 +0100 From: Wojtek Porczyk To: libvirt-list@redhat.com Message-ID: Mail-Followup-To: libvirt-list@redhat.com References: MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 203 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Fri, 17 Mar 2017 01:16:43 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Fri, 17 Mar 2017 01:16:43 +0000 (UTC) for IP:'66.111.4.25' DOMAIN:'out1-smtp.messagingengine.com' HELO:'out1-smtp.messagingengine.com' FROM:'woju@invisiblethingslab.com' RCPT:'' X-RedHat-Spam-Score: 0.4 (BAYES_50, DCC_REPUT_00_12, DKIM_SIGNED, DKIM_VALID) 66.111.4.25 out1-smtp.messagingengine.com 66.111.4.25 out1-smtp.messagingengine.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.27 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v1 2/2] Add asyncio event loop implementation X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============7795668212155189053==" Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Fri, 17 Mar 2017 01:16:55 +0000 (UTC) X-Zoho-Virus-Status: 1 X-ZohoMail: RSF_0 Z_629925259 SPT_0 --===============7795668212155189053== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="sm4nu43k4a2Rpi4c" Content-Disposition: inline --sm4nu43k4a2Rpi4c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable This is usable only on python >=3D 3.4 (or 3.3 with out-of-tree asyncio), however it should be harmless for anyone with older python versions. In simplest case, to have the callbacks queued on the default loop: >>> import libvirt >>> libvirt.virEventRegisterAsyncIOImpl() The function is not present on non-compatible platforms. Signed-off-by: Wojtek Porczyk --- libvirt-override.py | 6 + libvirt-python.spec.in | 1 + libvirtaio.py | 398 +++++++++++++++++++++++++++++++++++++++++++++= ++++ sanitytest.py | 2 +- setup.py | 12 ++ 5 files changed, 418 insertions(+), 1 deletion(-) create mode 100644 libvirtaio.py diff --git a/libvirt-override.py b/libvirt-override.py index 3d09d63..6a28336 100644 --- a/libvirt-override.py +++ b/libvirt-override.py @@ -16,6 +16,12 @@ except ImportError: if str(cyg_e).count("No module named"): raise lib_e =20 +try: + from libvirtaio import virEventAsyncIOImpl, virEventRegisterAsyncIOImpl +except (ImportError, SyntaxError): + # python < 3.3, or 3.3 and no out-of-tree asyncio + pass + import ctypes import types =20 diff --git a/libvirt-python.spec.in b/libvirt-python.spec.in index 3021ebd..0ee535e 100644 --- a/libvirt-python.spec.in +++ b/libvirt-python.spec.in @@ -86,6 +86,7 @@ rm -f %{buildroot}%{_libdir}/python*/site-packages/*egg-i= nfo %defattr(-,root,root) %doc ChangeLog AUTHORS NEWS README COPYING COPYING.LESSER examples/ %{_libdir}/python3*/site-packages/libvirt.py* +%{_libdir}/python3*/site-packages/libvirtaio.py* %{_libdir}/python3*/site-packages/libvirt_qemu.py* %{_libdir}/python3*/site-packages/libvirt_lxc.py* %{_libdir}/python3*/site-packages/__pycache__/libvirt.cpython-*.py* diff --git a/libvirtaio.py b/libvirtaio.py new file mode 100644 index 0000000..44c9a5b --- /dev/null +++ b/libvirtaio.py @@ -0,0 +1,398 @@ +# +# Copyright 2017 Wojtek Porczyk +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an 'AS IS' BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +'''Libvirt event loop implementation using asyncio + +Register the implementation of default loop: + + >>> import libvirt + >>> libvirt.virEventRegisterAsyncIOImpl() + +.. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html +''' + +__author__ =3D 'Wojtek Porczyk ' +__license__ =3D 'Apache-2.0' +__all__ =3D ['virEventAsyncIOImpl', 'virEventRegisterAsyncIOImpl'] + +import asyncio +import itertools +import logging +import warnings + +import libvirt + +try: + from asyncio import ensure_future +except ImportError: + from asyncio import async as ensure_future + + +class Callback(object): + '''Base class for holding callback + + :param virEventAsyncIOImpl impl: the implementation in which we run + :param cb: the callback itself + :param opaque: the opaque tuple passed by libvirt + ''' + # pylint: disable=3Dtoo-few-public-methods + + _iden_counter =3D itertools.count() + + def __init__(self, impl, cb, opaque, *args, **kwargs): + super().__init__(*args, **kwargs) + self.iden =3D next(self._iden_counter) + self.impl =3D impl + self.cb =3D cb + self.opaque =3D opaque + + assert self.iden not in self.impl.callbacks, \ + 'found {} callback: {!r}'.format( + self.iden, self.impl.callbacks[self.iden]) + self.impl.callbacks[self.iden] =3D self + + def __repr__(self): + return '<{} iden=3D{}>'.format(self.__clas__.__name__, self.iden) + + def close(self): + '''Schedule *ff* callback''' + self.impl.log.debug('callback %d close(), scheduling ff', self.ide= n) + self.impl.schedule_ff_callback(self.opaque) + +# +# file descriptors +# + +class Descriptor(object): + '''Manager of one file descriptor + + :param virEventAsyncIOImpl impl: the implementation in which we run + :param int fd: the file descriptor + ''' + def __init__(self, impl, fd): + self.impl =3D impl + self.fd =3D fd + self.callbacks =3D {} + + def _handle(self, event): + '''Dispatch the event to the descriptors + + :param int event: The event (from libvirt's constants) being dispa= tched + ''' + for callback in self.callbacks.values(): + if callback.event is not None and callback.event & event: + callback.cb(callback.iden, self.fd, event, callback.opaque) + + def update(self): + '''Register or unregister callbacks at event loop + + This should be called after change of any ``.event`` in callbacks. + ''' + # It seems like loop.add_{reader,writer} can be run multiple times + # and will still register the callback only once. Likewise, + # remove_{reader,writer} may be run even if the reader/writer + # is not registered (and will just return False). + + # For the edge case of empty callbacks, any() returns False. + if any(callback.event & ~( + libvirt.VIR_EVENT_HANDLE_READABLE | + libvirt.VIR_EVENT_HANDLE_WRITABLE) + for callback in self.callbacks.values()): + warnings.warn( + 'The only event supported are VIR_EVENT_HANDLE_READABLE ' + 'and VIR_EVENT_HANDLE_WRITABLE', + UserWarning) + + if any(callback.event & libvirt.VIR_EVENT_HANDLE_READABLE + for callback in self.callbacks.values()): + self.impl.loop.add_reader( + self.fd, self._handle, libvirt.VIR_EVENT_HANDLE_READABLE) + else: + self.impl.loop.remove_reader(self.fd) + + if any(callback.event & libvirt.VIR_EVENT_HANDLE_WRITABLE + for callback in self.callbacks.values()): + self.impl.loop.add_writer( + self.fd, self._handle, libvirt.VIR_EVENT_HANDLE_WRITABLE) + else: + self.impl.loop.remove_writer(self.fd) + + def add_handle(self, callback): + '''Add a callback to the descriptor + + :param FDCallback callback: the callback to add + :rtype: None + + After adding the callback, it is immediately watched. + ''' + self.callbacks[callback.iden] =3D callback + self.update() + + def remove_handle(self, iden): + '''Remove a callback from the descriptor + + :param int iden: the identifier of the callback + :returns: the callback + :rtype: FDCallback + + After removing the callback, the descriptor may be unwatched, if t= here + are no more handles for it. + ''' + callback =3D self.callbacks.pop(iden) + self.update() + return callback + + def close(self): + '''''' + self.callbacks.clear() + self.update() + +class DescriptorDict(dict): + '''Descriptors collection + + This is used internally by virEventAsyncIOImpl to hold descriptors. + ''' + def __init__(self, impl): + super().__init__() + self.impl =3D impl + + def __missing__(self, fd): + descriptor =3D Descriptor(self.impl, fd) + self[fd] =3D descriptor + return descriptor + +class FDCallback(Callback): + '''Callback for file descriptor (watcher) + + :param Descriptor descriptor: the descriptor manager + :param int event: bitset of events on which to fire the callback + ''' + # pylint: disable=3Dtoo-few-public-methods + + def __init__(self, *args, descriptor, event, **kwargs): + super().__init__(*args, **kwargs) + self.descriptor =3D descriptor + self.event =3D event + + def __repr__(self): + return '<{} iden=3D{} fd=3D{} event=3D{}>'.format( + self.__class__.__name__, self.iden, self.descriptor.fd, self.e= vent) + + def update(self, *, event): + '''Update the callback and fix descriptor's watchers''' + self.event =3D event + self.descriptor.update() + +# +# timeouts +# + +class TimeoutCallback(Callback): + '''Callback for timer''' + def __init__(self, *args, timeout, **kwargs): + super().__init__(*args, **kwargs) + self.timeout =3D timeout + self._task =3D None + + def __repr__(self): + return '<{} iden=3D{} timeout=3D{}>'.format( + self.__class__.__name__, self.iden, self.timeout) + + @asyncio.coroutine + def _timer(self): + '''An actual timer running on the event loop. + + This is a coroutine. + ''' + while True: + assert self.timeout >=3D 0, \ + 'invalid timeout {} for running timer'.format(self.timeout) + + try: + if self.timeout > 0: + timeout =3D self.timeout * 1e-3 + self.impl.log.debug('sleeping %r', timeout) + yield from asyncio.sleep(timeout) + else: + # scheduling timeout for next loop iteration + yield + + except asyncio.CancelledError: + self.impl.log.debug('timer %d cancelled', self.iden) + break + + self.cb(self.iden, self.opaque) + self.impl.log.debug('timer %r callback ended', self.iden) + + def update(self, *, timeout=3DNone): + '''Start or the timer, possibly updating timeout''' + if timeout is not None: + self.timeout =3D timeout + + if self.timeout >=3D 0 and self._task is None: + self.impl.log.debug('timer %r start', self.iden) + self._task =3D ensure_future(self._timer(), + loop=3Dself.impl.loop) + + elif self.timeout < 0 and self._task is not None: + self.impl.log.debug('timer %r stop', self.iden) + self._task.cancel() # pylint: disable=3Dno-member + self._task =3D None + + def close(self): + '''Stop the timer and call ff callback''' + self.timeout =3D -1 + self.update() + super().close() + +# +# main implementation +# + +class virEventAsyncIOImpl(object): + '''Libvirt event adapter to asyncio. + + :param loop: asyncio's event loop + + If *loop* is not specified, the current (or default) event loop is use= d. + ''' + + def __init__(self, *, loop=3DNone): + self.loop =3D loop or asyncio.get_event_loop() + self.callbacks =3D {} + self.descriptors =3D DescriptorDict(self) + self.log =3D logging.getLogger(self.__class__.__name__) + + def register(self): + '''Register this instance as event loop implementation''' + # pylint: disable=3Dbad-whitespace + self.log.debug('register()') + libvirt.virEventRegisterImpl( + self._add_handle, self._update_handle, self._remove_handle, + self._add_timeout, self._update_timeout, self._remove_timeout) + return self + + def schedule_ff_callback(self, opaque): + '''Schedule a ff callback from one of the handles or timers''' + self.loop.call_soon(libvirt.virEventExecuteFFCallback, opaque) + + def is_idle(self): + '''Returns False if there are leftovers from a connection + + Those may happen if there are sematical problems while closing + a connection. For example, not deregistered events before .close(). + ''' + return not self.callbacks + + def _add_handle(self, fd, event, cb, opaque): + '''Register a callback for monitoring file handle events + + :param int fd: file descriptor to listen on + :param int event: bitset of events on which to fire the callback + :param cb: the callback to be called when an event occurrs + :param opaque: user data to pass to the callback + :rtype: int + :returns: handle watch number to be used for updating and unregist= ering for events + + .. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html#virEventAd= dHandleFuncFunc + ''' + self.log.debug('add_handle(fd=3D%d, event=3D%d, cb=3D%r, opaque=3D= %r)', + fd, event, cb, opaque) + callback =3D FDCallback(self, cb, opaque, + descriptor=3Dself.descriptors[fd], event=3Devent) + self.callbacks[callback.iden] =3D callback + self.descriptors[fd].add_handle(callback) + return callback.iden + + def _update_handle(self, watch, event): + '''Change event set for a monitored file handle + + :param int watch: file descriptor watch to modify + :param int event: new events to listen on + + .. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html#virEventUp= dateHandleFunc + ''' + self.log.debug('update_handle(watch=3D%d, event=3D%d)', watch, eve= nt) + return self.callbacks[watch].update(event=3Devent) + + def _remove_handle(self, watch): + '''Unregister a callback from a file handle. + + :param int watch: file descriptor watch to stop listening on + :returns: None (see source for explanation) + + .. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html#virEventRe= moveHandleFunc + ''' + self.log.debug('remove_handle(watch=3D%d)', watch) + callback =3D self.callbacks.pop(watch) + assert callback is self.descriptors.remove_handle(watch) + callback.close() + + def _add_timeout(self, timeout, cb, opaque): + '''Register a callback for a timer event + + :param int timeout: the timeout to monitor + :param cb: the callback to call when timeout has expired + :param opaque: user data to pass to the callback + :rtype: int + :returns: a timer value + + .. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html#virEventAd= dTimeoutFunc + ''' + self.log.debug('add_timeout(timeout=3D%d, cb=3D%r, opaque=3D%r)', + timeout, cb, opaque) + callback =3D TimeoutCallback(self, cb, opaque, timeout=3Dtimeout) + self.callbacks[callback.iden] =3D callback + callback.update() + return callback.iden + + def _update_timeout(self, timer, timeout): + '''Change frequency for a timer + + :param int timer: the timer to modify + :param int timeout: the new timeout value in ms + + .. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html#virEventUp= dateTimeoutFunc + ''' + self.log.debug('update_timeout(timer=3D%d, timeout=3D%d)', timer, = timeout) + return self.callbacks[timer].update(timeout=3Dtimeout) + + def _remove_timeout(self, timer): + '''Unregister a callback for a timer + + :param int timer: the timer to remove + :returns: None (see source for explanation) + + .. seealso:: + https://libvirt.org/html/libvirt-libvirt-event.html#virEventRe= moveTimeoutFunc + ''' + self.log.debug('remove_timeout(timer=3D%d)', timer) + callback =3D self.callbacks.pop(timer) + callback.close() + +def virEventRegisterAsyncIOImpl(*, loop=3DNone): + '''Arrange for libvirt's callbacks to be dispatched via asyncio event = loop + + The implementation object is returned, but in normal usage it can safe= ly be + discarded. + ''' + return virEventAsyncIOImpl(loop=3Dloop).register() diff --git a/sanitytest.py b/sanitytest.py index 6548831..53a739f 100644 --- a/sanitytest.py +++ b/sanitytest.py @@ -350,7 +350,7 @@ for klass in gotfunctions: for func in sorted(gotfunctions[klass]): # These are pure python methods with no C APi if func in ["connect", "getConnect", "domain", "getDomain", - "virEventRegisterAsyncIOImpl"]: + "virEventRegisterAsyncIOImpl", "virEventExecuteFFCallback"= ]: continue =20 key =3D "%s.%s" % (klass, func) diff --git a/setup.py b/setup.py index 120ddd5..bac9010 100755 --- a/setup.py +++ b/setup.py @@ -14,6 +14,7 @@ import sys import os import os.path import re +import shutil import time =20 MIN_LIBVIRT =3D "0.9.11" @@ -50,6 +51,12 @@ def have_libvirt_lxc(): except DistutilsExecError: return False =20 +def have_libvirtaio(): + # This depends on asyncio, which in turn depends on "yield from" synta= x. + # The asyncio module itself is in standard library since 3.4, but ther= e is + # an out-of-tree version compatible with 3.3. + return sys.version_info >=3D (3, 3) + def get_pkgconfig_data(args, mod, required=3DTrue): """Run pkg-config to and return content associated with it""" f =3D os.popen("%s %s %s" % (get_pkgcfg(), " ".join(args), mod)) @@ -124,6 +131,9 @@ def get_module_lists(): c_modules.append(modulelxc) py_modules.append("libvirt_lxc") =20 + if have_libvirtaio(): + py_modules.append("libvirtaio") + return c_modules, py_modules =20 =20 @@ -141,6 +151,8 @@ class my_build(build): self.spawn([sys.executable, "generator.py", "libvirt-qemu", apis[1= ]]) if have_libvirt_lxc(): self.spawn([sys.executable, "generator.py", "libvirt-lxc", api= s[2]]) + if have_libvirtaio(): + shutil.copy('libvirtaio.py', 'build') =20 build.run(self) =20 --=20 2.5.5 --sm4nu43k4a2Rpi4c Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYyzj0AAoJEL9r2TIQOiNRBOMQAKBh7G76+1LprRK4mdx1kE13 pIS9w1IdbyP7WSm1XfKRrNp+tQ9JtgtQI/huSd599T9OYGhYGFTLNeeX5DNS967u ifmM8skN4MWpasUNtZb+5GD0KZ/Pwg5spjtEuA3md+czT0xbFlT69N97oavr8per jXWssp7sDe1gnkgzOvESU8y6jUi/eALVgGNvi5zgfWEZMa+FgMEnJVEWvOwvtFph gNbUIqzplHy1kBzubzeYGtZqW4LC/ZMYvEAZ4JLVRElLYKprHaM7Yx0VCT7jf1vv EmQvBNcZFMSmyaK4fC/WGSX665kb9kavQ7b7eRRpFF+fkFz/Cd2lWXTvyfS1JOwY +fF95RbSB/BXiQgqvyuOFa49ds4A8aBd3CATMCRFQrmzds1KQXQgmegoLJDqfZh0 w5vOb1JFQHG/XJgAIxQLl9HWAtVair25KdpD0NFb7xSr6ld/wLiJ6fNTBZ9zgJHJ JjFbPr1pOtnNv+0DuvbXebtopz4pH7pRtkfsMoaI9InVxfWVK2vpgIP5ScgT8ADZ oetQ4qaxiVAvRu7/nNHGLR8hY1GMhk4HNS1G+zWDijh6CVoF3RnkoYXFIo30V/mP dFwXOQEJdGTUBPYYaWZWl8DjjRe6orddYd9whZcTlntaF4OOgrBOM3+Ag2ealmxN Gn1iWNLnDf882SdcZ1CR =6j2P -----END PGP SIGNATURE----- --sm4nu43k4a2Rpi4c-- --===============7795668212155189053== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list --===============7795668212155189053==--