Provide an standard asyncio event loop impl

[libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Daniel P. Berrange 7 years ago

This patch series is a followup to Wojtek's v2 posting:

  https://www.redhat.com/archives/libvir-list/2017-March/msg00838.html

It contains the fixes I pointed out in v1 / v2, along with a bunch of
updates to the event-test.py example, so we can use it to demonstrate
the asyncio impl. eg i'm using:

$ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session

Daniel P. Berrange (5):
  event-test: free opaque data when removing callbacks
  event-test: add timeout to exit event loop
  event-test: unregister callbacks & close conn on exit
  event-test: rename example event loop impl
  event-test: add ability to run the asyncio event loop

Wojtek Porczyk (2):
  Allow for ff callbacks to be called by custom event implementations
  Add asyncio event loop implementation

 MANIFEST.in            |   1 +
 examples/event-test.py | 194 +++++++++++++++---------
 libvirt-override.c     |  68 +++++----
 libvirt-override.py    |  23 +++
 libvirt-python.spec.in |   2 +
 libvirtaio.py          | 399 +++++++++++++++++++++++++++++++++++++++++++++++++
 sanitytest.py          |   3 +-
 setup.py               |  12 ++
 8 files changed, 603 insertions(+), 99 deletions(-)
 create mode 100644 libvirtaio.py

-- 
2.9.3

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Michal Privoznik 7 years ago

On 04/04/2017 04:31 PM, Daniel P. Berrange wrote:
> This patch series is a followup to Wojtek's v2 posting:
>
>   https://www.redhat.com/archives/libvir-list/2017-March/msg00838.html
>
> It contains the fixes I pointed out in v1 / v2, along with a bunch of
> updates to the event-test.py example, so we can use it to demonstrate
> the asyncio impl. eg i'm using:
>
> $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
>
> Daniel P. Berrange (5):
>   event-test: free opaque data when removing callbacks
>   event-test: add timeout to exit event loop
>   event-test: unregister callbacks & close conn on exit
>   event-test: rename example event loop impl
>   event-test: add ability to run the asyncio event loop
>
> Wojtek Porczyk (2):
>   Allow for ff callbacks to be called by custom event implementations
>   Add asyncio event loop implementation
>
>  MANIFEST.in            |   1 +
>  examples/event-test.py | 194 +++++++++++++++---------
>  libvirt-override.c     |  68 +++++----
>  libvirt-override.py    |  23 +++
>  libvirt-python.spec.in |   2 +
>  libvirtaio.py          | 399 +++++++++++++++++++++++++++++++++++++++++++++++++
>  sanitytest.py          |   3 +-
>  setup.py               |  12 ++
>  8 files changed, 603 insertions(+), 99 deletions(-)
>  create mode 100644 libvirtaio.py
>

The subject doesn't say so, but the diffstat does. This is against 
libvirt-python.


ACK series.

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Wojtek Porczyk 7 years ago

On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
> $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session

Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
event-test.py and also with our own code. Looks good to me.


I encountered one small problem, which I believe is orthogonal to the patch series:

On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> diff --git a/examples/event-test.py b/examples/event-test.py
> index 751a140..ac9fbe1 100755
> --- a/examples/event-test.py
> +++ b/examples/event-test.py

> +    netcallbacks = []
> +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))

With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
exception:

  libvirt.libvirtError: this function is not supported by the connection driver: virConnectNetworkEventRegisterAny

Commenting it out solves the problem.


-- 
pozdrawiam / best regards       _.-._
Wojtek Porczyk               .-^'   '^-.
Invisible Things Lab         |'-.-^-.-'|
                             |  |   |  |
 I do not fear computers,    |  '-.-'  |
 I fear lack of them.        '-._ :  ,-'
    -- Isaac Asimov             `^-^-_>
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Michal Privoznik 7 years ago

On 04/06/2017 10:47 PM, Wojtek Porczyk wrote:
> On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
>> $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
>
> Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
> event-test.py and also with our own code. Looks good to me.
>
>
> I encountered one small problem, which I believe is orthogonal to the patch series:
>
> On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
>> diff --git a/examples/event-test.py b/examples/event-test.py
>> index 751a140..ac9fbe1 100755
>> --- a/examples/event-test.py
>> +++ b/examples/event-test.py
>
>> +    netcallbacks = []
>> +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
>
> With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> exception:
>
>   libvirt.libvirtError: this function is not supported by the connection driver: virConnectNetworkEventRegisterAny

That's because we don't have a network driver for XEN. Usually, for 
stateful drivers (those which require daemon to run, e.g. qemu, lxc, ..) 
they can use our own bridge driver for network (it creates a bridge to 
which domain NICs are plugged in). However, then we have bunch of 
stateless driver for which libvirt is just a wrapper over their APIs. 
For those we would need a separate network driver that does the API 
wrapping. And it looks like there's none for XEN. Not sure whether 
there's something to wrap though. I mean I've no idea what XEN 
networking capabilities are.

But what we can do here is to surround networkEvenRegisterAny() call 
with try {} except () block and ignore ENOSUPP error.

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Wojtek Porczyk 7 years ago

On Fri, Apr 07, 2017 at 09:31:45AM +0200, Michal Privoznik wrote:
> On 04/06/2017 10:47 PM, Wojtek Porczyk wrote:
> >  libvirt.libvirtError: this function is not supported by the connection driver: virConnectNetworkEventRegisterAny
> 
> That's because we don't have a network driver for XEN. Usually, for stateful
> drivers (those which require daemon to run, e.g. qemu, lxc, ..) they can use
> our own bridge driver for network (it creates a bridge to which domain NICs
> are plugged in). However, then we have bunch of stateless driver for which
> libvirt is just a wrapper over their APIs. For those we would need a
> separate network driver that does the API wrapping. And it looks like
> there's none for XEN. Not sure whether there's something to wrap though. I
> mean I've no idea what XEN networking capabilities are.
> 
> But what we can do here is to surround networkEvenRegisterAny() call with
> try {} except () block and ignore ENOSUPP error.

Thanks for the explanation. Are there any other drivers which don't support
some events? If so, maybe all the event requests should be wrapped in that?


-- 
pozdrawiam / best regards       _.-._
Wojtek Porczyk               .-^'   '^-.
Invisible Things Lab         |'-.-^-.-'|
                             |  |   |  |
 I do not fear computers,    |  '-.-'  |
 I fear lack of them.        '-._ :  ,-'
    -- Isaac Asimov             `^-^-_>
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Daniel P. Berrange 7 years ago

On Fri, Apr 07, 2017 at 09:31:45AM +0200, Michal Privoznik wrote:
> On 04/06/2017 10:47 PM, Wojtek Porczyk wrote:
> > On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
> > > $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
> > 
> > Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
> > event-test.py and also with our own code. Looks good to me.
> > 
> > 
> > I encountered one small problem, which I believe is orthogonal to the patch series:
> > 
> > On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> > > diff --git a/examples/event-test.py b/examples/event-test.py
> > > index 751a140..ac9fbe1 100755
> > > --- a/examples/event-test.py
> > > +++ b/examples/event-test.py
> > 
> > > +    netcallbacks = []
> > > +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
> > 
> > With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> > exception:
> > 
> >   libvirt.libvirtError: this function is not supported by the connection driver: virConnectNetworkEventRegisterAny
> 
> That's because we don't have a network driver for XEN. Usually, for stateful
> drivers (those which require daemon to run, e.g. qemu, lxc, ..) they can use
> our own bridge driver for network (it creates a bridge to which domain NICs
> are plugged in). However, then we have bunch of stateless driver for which
> libvirt is just a wrapper over their APIs. For those we would need a
> separate network driver that does the API wrapping. And it looks like
> there's none for XEN. Not sure whether there's something to wrap though. I
> mean I've no idea what XEN networking capabilities are.

That doesn't make sense - the shared network driver should be operational
for Xen - it certainly worked in the past.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Daniel P. Berrange 7 years ago

On Thu, Apr 06, 2017 at 10:47:48PM +0200, Wojtek Porczyk wrote:
> On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
> > $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
> 
> Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
> event-test.py and also with our own code. Looks good to me.

Great, thank you.

> I encountered one small problem, which I believe is orthogonal to the patch series:
> 
> On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> > diff --git a/examples/event-test.py b/examples/event-test.py
> > index 751a140..ac9fbe1 100755
> > --- a/examples/event-test.py
> > +++ b/examples/event-test.py
> 
> > +    netcallbacks = []
> > +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
> 
> With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> exception:
> 
>   libvirt.libvirtError: this function is not supported by the
>    connection driver: virConnectNetworkEventRegisterAny
> 
> Commenting it out solves the problem.

Yes, I must say the event test is only validated with the KVM driver, but that said
I thought the Xen driver would get our default nework driver activated automatically,
so I'm surprised you see that.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Wojtek Porczyk 7 years ago

On Fri, Apr 07, 2017 at 10:22:38AM +0100, Daniel P. Berrange wrote:
> On Thu, Apr 06, 2017 at 10:47:48PM +0200, Wojtek Porczyk wrote:
> > On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
> > > $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
> > 
> > Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
> > event-test.py and also with our own code. Looks good to me.
> 
> Great, thank you.
> 
> > I encountered one small problem, which I believe is orthogonal to the patch series:
> > 
> > On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> > > diff --git a/examples/event-test.py b/examples/event-test.py
> > > index 751a140..ac9fbe1 100755
> > > --- a/examples/event-test.py
> > > +++ b/examples/event-test.py
> > 
> > > +    netcallbacks = []
> > > +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
> > 
> > With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> > exception:
> > 
> >   libvirt.libvirtError: this function is not supported by the
> >    connection driver: virConnectNetworkEventRegisterAny
> > 
> > Commenting it out solves the problem.
> 
> Yes, I must say the event test is only validated with the KVM driver, but that said
> I thought the Xen driver would get our default nework driver activated automatically,
> so I'm surprised you see that.

I don't know, maybe this is caused by our setup. For example we have no NIC in
dom0 (only loopback) and we also may have deliberately broken something around
the network.

Cc: marmarek

-- 
pozdrawiam / best regards       _.-._
Wojtek Porczyk               .-^'   '^-.
Invisible Things Lab         |'-.-^-.-'|
                             |  |   |  |
 I do not fear computers,    |  '-.-'  |
 I fear lack of them.        '-._ :  ,-'
    -- Isaac Asimov             `^-^-_>
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Marek Marczykowski-Górecki 7 years ago

On Fri, Apr 07, 2017 at 01:46:12PM +0200, Wojtek Porczyk wrote:
> On Fri, Apr 07, 2017 at 10:22:38AM +0100, Daniel P. Berrange wrote:
> > On Thu, Apr 06, 2017 at 10:47:48PM +0200, Wojtek Porczyk wrote:
> > > On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
> > > > $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
> > > 
> > > Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
> > > event-test.py and also with our own code. Looks good to me.
> > 
> > Great, thank you.
> > 
> > > I encountered one small problem, which I believe is orthogonal to the patch series:
> > > 
> > > On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> > > > diff --git a/examples/event-test.py b/examples/event-test.py
> > > > index 751a140..ac9fbe1 100755
> > > > --- a/examples/event-test.py
> > > > +++ b/examples/event-test.py
> > > 
> > > > +    netcallbacks = []
> > > > +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
> > > 
> > > With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> > > exception:
> > > 
> > >   libvirt.libvirtError: this function is not supported by the
> > >    connection driver: virConnectNetworkEventRegisterAny
> > > 
> > > Commenting it out solves the problem.
> > 
> > Yes, I must say the event test is only validated with the KVM driver, but that said
> > I thought the Xen driver would get our default nework driver activated automatically,
> > so I'm surprised you see that.
> 
> I don't know, maybe this is caused by our setup. For example we have no NIC in
> dom0 (only loopback) and we also may have deliberately broken something around
> the network.

Yes, our packages have ./configure (...) --without-network.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Wojtek Porczyk 7 years ago

On Fri, Apr 07, 2017 at 02:53:52PM +0200, Marek Marczykowski-Górecki wrote:
> On Fri, Apr 07, 2017 at 01:46:12PM +0200, Wojtek Porczyk wrote:
> > On Fri, Apr 07, 2017 at 10:22:38AM +0100, Daniel P. Berrange wrote:
> > > On Thu, Apr 06, 2017 at 10:47:48PM +0200, Wojtek Porczyk wrote:
> > > > I encountered one small problem, which I believe is orthogonal to the patch series:
> > > > 
> > > > On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> > > > > diff --git a/examples/event-test.py b/examples/event-test.py
> > > > > index 751a140..ac9fbe1 100755
> > > > > --- a/examples/event-test.py
> > > > > +++ b/examples/event-test.py
> > > > 
> > > > > +    netcallbacks = []
> > > > > +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
> > > > 
> > > > With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> > > > exception:
> > > > 
> > > >   libvirt.libvirtError: this function is not supported by the
> > > >    connection driver: virConnectNetworkEventRegisterAny
> > > > 
> > > > Commenting it out solves the problem.
> > > 
> > > Yes, I must say the event test is only validated with the KVM driver, but that said
> > > I thought the Xen driver would get our default nework driver activated automatically,
> > > so I'm surprised you see that.
> > 
> > I don't know, maybe this is caused by our setup. For example we have no NIC in
> > dom0 (only loopback) and we also may have deliberately broken something around
> > the network.
> 
> Yes, our packages have ./configure (...) --without-network.

So looks like our fault. Sorry for that.


-- 
pozdrawiam / best regards       _.-._
Wojtek Porczyk               .-^'   '^-.
Invisible Things Lab         |'-.-^-.-'|
                             |  |   |  |
 I do not fear computers,    |  '-.-'  |
 I fear lack of them.        '-._ :  ,-'
    -- Isaac Asimov             `^-^-_>
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Daniel P. Berrange 7 years ago

On Fri, Apr 07, 2017 at 06:35:54PM +0200, Wojtek Porczyk wrote:
> On Fri, Apr 07, 2017 at 02:53:52PM +0200, Marek Marczykowski-Górecki wrote:
> > On Fri, Apr 07, 2017 at 01:46:12PM +0200, Wojtek Porczyk wrote:
> > > On Fri, Apr 07, 2017 at 10:22:38AM +0100, Daniel P. Berrange wrote:
> > > > On Thu, Apr 06, 2017 at 10:47:48PM +0200, Wojtek Porczyk wrote:
> > > > > I encountered one small problem, which I believe is orthogonal to the patch series:
> > > > > 
> > > > > On Tue, Apr 04, 2017 at 03:31:32PM +0100, Daniel P. Berrange wrote:
> > > > > > diff --git a/examples/event-test.py b/examples/event-test.py
> > > > > > index 751a140..ac9fbe1 100755
> > > > > > --- a/examples/event-test.py
> > > > > > +++ b/examples/event-test.py
> > > > > 
> > > > > > +    netcallbacks = []
> > > > > > +    netcallbacks.append(vc.networkEventRegisterAny(None, libvirt.VIR_NETWORK_EVENT_ID_LIFECYCLE, myNetworkEventLifecycleCallback, None))
> > > > > 
> > > > > With vc = libvirt.open('xen:///') and with libvirt{,-python}-3.1.0 this line causes an
> > > > > exception:
> > > > > 
> > > > >   libvirt.libvirtError: this function is not supported by the
> > > > >    connection driver: virConnectNetworkEventRegisterAny
> > > > > 
> > > > > Commenting it out solves the problem.
> > > > 
> > > > Yes, I must say the event test is only validated with the KVM driver, but that said
> > > > I thought the Xen driver would get our default nework driver activated automatically,
> > > > so I'm surprised you see that.
> > > 
> > > I don't know, maybe this is caused by our setup. For example we have no NIC in
> > > dom0 (only loopback) and we also may have deliberately broken something around
> > > the network.
> > 
> > Yes, our packages have ./configure (...) --without-network.
> 
> So looks like our fault. Sorry for that.

No problem, good to know everything's working as expected


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 0/7] Provide an standard asyncio event loop impl

Posted by Daniel P. Berrange 7 years ago

On Thu, Apr 06, 2017 at 10:47:48PM +0200, Wojtek Porczyk wrote:
> On Tue, Apr 04, 2017 at 03:31:27PM +0100, Daniel P. Berrange wrote:
> > $ python3 ./examples/event-test.py --loop=asyncio --timeout=30 qemu:///session
> 
> Thank you for this update. I tested it backported to 3.1.0 with xen:/// using
> event-test.py and also with our own code. Looks good to me.

FYI, this series is now pushed to git master. Thanks for your contribution
to libvirt !

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list