[libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client

John Ferlan posted 9 patches 6 years, 2 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/20180119172311.15525-1-jferlan@redhat.com
There is a newer version of this series
daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
docs/news.xml            | 12 +++++++++
src/libvirt_private.syms |  1 +
src/libvirt_remote.syms  |  2 ++
src/qemu/qemu_driver.c   |  5 ++++
src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
src/rpc/virnetserver.h   |  4 +++
src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
src/util/virthreadpool.h |  2 ++
10 files changed, 204 insertions(+), 26 deletions(-)
[libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by John Ferlan 6 years, 2 months ago
RFC:
https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html

Adjustments since RFC...

Patches 1&2: No change, were already R-B'd
Patch 3: Removed code as noted in code review, update commit message
Patch 4: From old series removed, see below for more details
Patch 9: no change
NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
    are removed as they seemed to not be necessary

Replaced the former patch 4 with series of patches to (slowly) provide
support to disable new connections, handle removing waiting jobs, causing
the waiting workers to quit, and allow any running jobs to complete.

As it turns out, waiting for running jobs to complete cannot be done
from the virNetServerClose callbacks because that means the event loop
processing done during virNetServerRun will not allow any currently
long running worker job thread a means to complete.

So when virNetDaemonQuit is called as a result of the libvirtd signal
handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
to quit immediately, alter to using a quitRequested flag and then use
that quitRequested flag to check for long running worker threads before
causing the event loop to quit resulting in libvirtd being able to run
through the virNetDaemonClose processing.

John Ferlan (9):
  libvirtd: Alter refcnt processing for domain server objects
  libvirtd: Alter refcnt processing for server program objects
  netserver: Remove ServiceToggle during ServerDispose
  util: Introduce virThreadPoolDrain
  rpc: Introduce virNetServerQuitRequested
  rpc: Introduce virNetServerWorkerCount
  rpc: Alter virNetDaemonQuit processing
  docs: Add news article for libvirtd issue
  APPLY ONLY FOR TESTING PURPOSES

 daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
 docs/news.xml            | 12 +++++++++
 src/libvirt_private.syms |  1 +
 src/libvirt_remote.syms  |  2 ++
 src/qemu/qemu_driver.c   |  5 ++++
 src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
 src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
 src/rpc/virnetserver.h   |  4 +++
 src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
 src/util/virthreadpool.h |  2 ++
 10 files changed, 204 insertions(+), 26 deletions(-)

-- 
2.13.6

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Nikolay Shirokovskiy 6 years, 2 months ago
Hi, John.

Thank you for driving this issue! Unfortunately I have little time now
to take participation in reviweing etc.

Nikolay

On 19.01.2018 20:23, John Ferlan wrote:
> RFC:
> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
> 
> Adjustments since RFC...
> 
> Patches 1&2: No change, were already R-B'd
> Patch 3: Removed code as noted in code review, update commit message
> Patch 4: From old series removed, see below for more details
> Patch 9: no change
> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>     are removed as they seemed to not be necessary
> 
> Replaced the former patch 4 with series of patches to (slowly) provide
> support to disable new connections, handle removing waiting jobs, causing
> the waiting workers to quit, and allow any running jobs to complete.
> 
> As it turns out, waiting for running jobs to complete cannot be done
> from the virNetServerClose callbacks because that means the event loop
> processing done during virNetServerRun will not allow any currently
> long running worker job thread a means to complete.
> 
> So when virNetDaemonQuit is called as a result of the libvirtd signal
> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
> to quit immediately, alter to using a quitRequested flag and then use
> that quitRequested flag to check for long running worker threads before
> causing the event loop to quit resulting in libvirtd being able to run
> through the virNetDaemonClose processing.
> 
> John Ferlan (9):
>   libvirtd: Alter refcnt processing for domain server objects
>   libvirtd: Alter refcnt processing for server program objects
>   netserver: Remove ServiceToggle during ServerDispose
>   util: Introduce virThreadPoolDrain
>   rpc: Introduce virNetServerQuitRequested
>   rpc: Introduce virNetServerWorkerCount
>   rpc: Alter virNetDaemonQuit processing
>   docs: Add news article for libvirtd issue
>   APPLY ONLY FOR TESTING PURPOSES
> 
>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>  docs/news.xml            | 12 +++++++++
>  src/libvirt_private.syms |  1 +
>  src/libvirt_remote.syms  |  2 ++
>  src/qemu/qemu_driver.c   |  5 ++++
>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>  src/rpc/virnetserver.h   |  4 +++
>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>  src/util/virthreadpool.h |  2 ++
>  10 files changed, 204 insertions(+), 26 deletions(-)
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Nikolay Shirokovskiy 6 years, 2 months ago

On 19.01.2018 20:23, John Ferlan wrote:
> RFC:
> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
> 
> Adjustments since RFC...
> 
> Patches 1&2: No change, were already R-B'd
> Patch 3: Removed code as noted in code review, update commit message
> Patch 4: From old series removed, see below for more details
> Patch 9: no change
> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>     are removed as they seemed to not be necessary
> 
> Replaced the former patch 4 with series of patches to (slowly) provide
> support to disable new connections, handle removing waiting jobs, causing
> the waiting workers to quit, and allow any running jobs to complete.
> 
> As it turns out, waiting for running jobs to complete cannot be done
> from the virNetServerClose callbacks because that means the event loop
> processing done during virNetServerRun will not allow any currently
> long running worker job thread a means to complete.

Hi, John.

So you suggest a different appoarch. Instead of introducing means to 
help rpc threads to finish after event loop is finished (stateShutdown hook in my series)
you suggest to block futher new rpc processing and then waiting running
rpc calls to finish keeping event loop running for that purpose.

This could lead to libvirtd never finish its termination if one of
qemu processes do not respond for example. In case of long running
operation such as migration finishing can take much time. On the 
over hand if we finish rpc threads abruptly as in case of stateShutdown hook
we will deal with all possible error paths on daemon finishing. May
be good approach is to abort long running operation, then give rpc threads
some time to finish as you suggest and only after that finish them abruptly to handle
hanging qemu etc.

Also in this approach we keep event loop running for rpc threads only 
but there can be other threads that depend on the loop. For example if 
qemu is rebooted we spawn a thread that executes qemuProcessFakeReboot
that sends commands to qemu (i.e. depends on event loop). We need to await
such threads finishing too while keep event loop running. In approach of
stateShutdown hook we finish all threads that uses qemu monitor by closing
the monitor.

And hack with timeout in a loop... I think of a different aproach for waiting
rpc threads to finish their work. First let's make drain function only flush
job queue and take some callback which will be called when all workers will
be free from work (let's keep worker threads as Dan suggested). In this callback we
can use same technique as in virNetDaemonSignalHandler. That is make some
IO to set some flag in the event loop thread and cause virEventRunDefaultImpl
to finish and then test this flag in virNetDaemonRun.

Nikolay

> 
> So when virNetDaemonQuit is called as a result of the libvirtd signal
> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
> to quit immediately, alter to using a quitRequested flag and then use
> that quitRequested flag to check for long running worker threads before
> causing the event loop to quit resulting in libvirtd being able to run
> through the virNetDaemonClose processing.
> 
> John Ferlan (9):
>   libvirtd: Alter refcnt processing for domain server objects
>   libvirtd: Alter refcnt processing for server program objects
>   netserver: Remove ServiceToggle during ServerDispose
>   util: Introduce virThreadPoolDrain
>   rpc: Introduce virNetServerQuitRequested
>   rpc: Introduce virNetServerWorkerCount
>   rpc: Alter virNetDaemonQuit processing
>   docs: Add news article for libvirtd issue
>   APPLY ONLY FOR TESTING PURPOSES
> 
>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>  docs/news.xml            | 12 +++++++++
>  src/libvirt_private.syms |  1 +
>  src/libvirt_remote.syms  |  2 ++
>  src/qemu/qemu_driver.c   |  5 ++++
>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>  src/rpc/virnetserver.h   |  4 +++
>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>  src/util/virthreadpool.h |  2 ++
>  10 files changed, 204 insertions(+), 26 deletions(-)
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Daniel P. Berrangé 6 years, 2 months ago
On Fri, Jan 26, 2018 at 11:47:04AM +0300, Nikolay Shirokovskiy wrote:
> 
> 
> On 19.01.2018 20:23, John Ferlan wrote:
> > RFC:
> > https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
> > 
> > Adjustments since RFC...
> > 
> > Patches 1&2: No change, were already R-B'd
> > Patch 3: Removed code as noted in code review, update commit message
> > Patch 4: From old series removed, see below for more details
> > Patch 9: no change
> > NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
> >     are removed as they seemed to not be necessary
> > 
> > Replaced the former patch 4 with series of patches to (slowly) provide
> > support to disable new connections, handle removing waiting jobs, causing
> > the waiting workers to quit, and allow any running jobs to complete.
> > 
> > As it turns out, waiting for running jobs to complete cannot be done
> > from the virNetServerClose callbacks because that means the event loop
> > processing done during virNetServerRun will not allow any currently
> > long running worker job thread a means to complete.
> 
> Hi, John.
> 
> So you suggest a different appoarch. Instead of introducing means to 
> help rpc threads to finish after event loop is finished (stateShutdown hook in my series)
> you suggest to block futher new rpc processing and then waiting running
> rpc calls to finish keeping event loop running for that purpose.
> 
> This could lead to libvirtd never finish its termination if one of
> qemu processes do not respond for example. In case of long running
> operation such as migration finishing can take much time. On the 
> over hand if we finish rpc threads abruptly as in case of stateShutdown hook
> we will deal with all possible error paths on daemon finishing. May
> be good approach is to abort long running operation, then give rpc threads
> some time to finish as you suggest and only after that finish them abruptly
> to handle hanging qemu etc.

We should keep in mind why we are bothering todo any of this "graceful"
cleanup. 99% of the time it would be fine for libvirtd to just immediately
immediately exit and not run any cleanup code, since the OS cleans everything
up regardless. Really the only benefit of running cleanup is so that developers
can check for memory leaks across the process execution.

On balance it is *way* more important that libvirtd is guranteed to exit in
a short, finite amount of time, even if that means skipping cleanup, because
that is what is relevant to production deployment.

So this means while it is nice to wait for worker threads to complete their
current RPC option, we should not wait too long. eg Give them 15 seconds to
finish their work, and if they're not done then just unconditionally exit
libvirtd, possibly skipping remaining cleanup if that's neccessary to avoid
SEGV.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by John Ferlan 6 years, 1 month ago

On 01/26/2018 03:47 AM, Nikolay Shirokovskiy wrote:
> 
> 
> On 19.01.2018 20:23, John Ferlan wrote:
>> RFC:
>> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>>
>> Adjustments since RFC...
>>
>> Patches 1&2: No change, were already R-B'd
>> Patch 3: Removed code as noted in code review, update commit message
>> Patch 4: From old series removed, see below for more details
>> Patch 9: no change
>> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>>     are removed as they seemed to not be necessary
>>
>> Replaced the former patch 4 with series of patches to (slowly) provide
>> support to disable new connections, handle removing waiting jobs, causing
>> the waiting workers to quit, and allow any running jobs to complete.
>>
>> As it turns out, waiting for running jobs to complete cannot be done
>> from the virNetServerClose callbacks because that means the event loop
>> processing done during virNetServerRun will not allow any currently
>> long running worker job thread a means to complete.
> 
> Hi, John.

Sorry - been busy in other areas lately - trying to get back to this...

> 
> So you suggest a different appoarch. Instead of introducing means to 
> help rpc threads to finish after event loop is finished (stateShutdown hook in my series)
> you suggest to block futher new rpc processing and then waiting running
> rpc calls to finish keeping event loop running for that purpose.

Somewhere along the way in one of the reviews it was suggested to give
the workers perhaps a few extra cycles to complete anything they might
be doing. It was also suggested a mechanism to do that - which is what I
tried to accomplish here.

Based on that and that your mechanism was more of an "immediate
separation" by IIRC closing the monitor connection and letting that
close handler do whatever magic happens there.

This not to say your mechanism is the wrong way to go about it, but it
also hasn't garnered support nor have you actively attempted to champion
it. So I presented an alternative approach.

> 
> This could lead to libvirtd never finish its termination if one of
> qemu processes do not respond for example. In case of long running
> operation such as migration finishing can take much time. On the 
> over hand if we finish rpc threads abruptly as in case of stateShutdown hook
> we will deal with all possible error paths on daemon finishing. May
> be good approach is to abort long running operation, then give rpc threads
> some time to finish as you suggest and only after that finish them abruptly to handle
> hanging qemu etc.

True, but it's also easy enough to add something to the last and updated
patch to add the QuitTimer and force an exit without going through the
rest of the "friendly" quit (IOW: more or less what Dan has suggested).

There's an incredible amount of cycles being spent for what amounts to
trying to be nice from a SIGTERM, SIGQUIT, or SIGINT - IOW: eventual death.


> 
> Also in this approach we keep event loop running for rpc threads only 
> but there can be other threads that depend on the loop. For example if 
> qemu is rebooted we spawn a thread that executes qemuProcessFakeReboot
> that sends commands to qemu (i.e. depends on event loop). We need to await
> such threads finishing too while keep event loop running. In approach of
> stateShutdown hook we finish all threads that uses qemu monitor by closing
> the monitor.
> 
> And hack with timeout in a loop... I think of a different aproach for waiting
> rpc threads to finish their work. First let's make drain function only flush
> job queue and take some callback which will be called when all workers will
> be free from work (let's keep worker threads as Dan suggested). In this callback we
> can use same technique as in virNetDaemonSignalHandler. That is make some
> IO to set some flag in the event loop thread and cause virEventRunDefaultImpl
> to finish and then test this flag in virNetDaemonRun.
> 

Please feel free to spend as many cycles as you can making adjustments
to what I have. I'm working through some alterations and posting a v2 of
what I have and we'll see where it takes me/us.

John


> Nikolay
> 
>>
>> So when virNetDaemonQuit is called as a result of the libvirtd signal
>> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
>> to quit immediately, alter to using a quitRequested flag and then use
>> that quitRequested flag to check for long running worker threads before
>> causing the event loop to quit resulting in libvirtd being able to run
>> through the virNetDaemonClose processing.
>>
>> John Ferlan (9):
>>   libvirtd: Alter refcnt processing for domain server objects
>>   libvirtd: Alter refcnt processing for server program objects
>>   netserver: Remove ServiceToggle during ServerDispose
>>   util: Introduce virThreadPoolDrain
>>   rpc: Introduce virNetServerQuitRequested
>>   rpc: Introduce virNetServerWorkerCount
>>   rpc: Alter virNetDaemonQuit processing
>>   docs: Add news article for libvirtd issue
>>   APPLY ONLY FOR TESTING PURPOSES
>>
>>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>>  docs/news.xml            | 12 +++++++++
>>  src/libvirt_private.syms |  1 +
>>  src/libvirt_remote.syms  |  2 ++
>>  src/qemu/qemu_driver.c   |  5 ++++
>>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>>  src/rpc/virnetserver.h   |  4 +++
>>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>>  src/util/virthreadpool.h |  2 ++
>>  10 files changed, 204 insertions(+), 26 deletions(-)
>>

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Marc Hartmayer 5 years, 8 months ago
On Wed, Feb 14, 2018 at 11:36 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
> On 01/26/2018 03:47 AM, Nikolay Shirokovskiy wrote:
>>
>>
>> On 19.01.2018 20:23, John Ferlan wrote:
>>> RFC:
>>> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>>>
>>> Adjustments since RFC...
>>>
>>> Patches 1&2: No change, were already R-B'd
>>> Patch 3: Removed code as noted in code review, update commit message
>>> Patch 4: From old series removed, see below for more details
>>> Patch 9: no change
>>> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>>>     are removed as they seemed to not be necessary
>>>
>>> Replaced the former patch 4 with series of patches to (slowly) provide
>>> support to disable new connections, handle removing waiting jobs, causing
>>> the waiting workers to quit, and allow any running jobs to complete.
>>>
>>> As it turns out, waiting for running jobs to complete cannot be done
>>> from the virNetServerClose callbacks because that means the event loop
>>> processing done during virNetServerRun will not allow any currently
>>> long running worker job thread a means to complete.
>>
>> Hi, John.
>
> Sorry - been busy in other areas lately - trying to get back to this...
>
>>
>> So you suggest a different appoarch. Instead of introducing means to
>> help rpc threads to finish after event loop is finished (stateShutdown hook in my series)
>> you suggest to block futher new rpc processing and then waiting running
>> rpc calls to finish keeping event loop running for that purpose.
>
> Somewhere along the way in one of the reviews it was suggested to give
> the workers perhaps a few extra cycles to complete anything they might
> be doing. It was also suggested a mechanism to do that - which is what I
> tried to accomplish here.
>
> Based on that and that your mechanism was more of an "immediate
> separation" by IIRC closing the monitor connection and letting that
> close handler do whatever magic happens there.
>
> This not to say your mechanism is the wrong way to go about it, but it
> also hasn't garnered support nor have you actively attempted to champion
> it. So I presented an alternative approach.
>
>>
>> This could lead to libvirtd never finish its termination if one of
>> qemu processes do not respond for example. In case of long running
>> operation such as migration finishing can take much time. On the
>> over hand if we finish rpc threads abruptly as in case of stateShutdown hook
>> we will deal with all possible error paths on daemon finishing. May
>> be good approach is to abort long running operation, then give rpc threads
>> some time to finish as you suggest and only after that finish them abruptly to handle
>> hanging qemu etc.
>
> True, but it's also easy enough to add something to the last and updated
> patch to add the QuitTimer and force an exit without going through the
> rest of the "friendly" quit (IOW: more or less what Dan has suggested).
>
> There's an incredible amount of cycles being spent for what amounts to
> trying to be nice from a SIGTERM, SIGQUIT, or SIGINT - IOW: eventual death.
>
>
>>
>> Also in this approach we keep event loop running for rpc threads only
>> but there can be other threads that depend on the loop. For example if
>> qemu is rebooted we spawn a thread that executes qemuProcessFakeReboot
>> that sends commands to qemu (i.e. depends on event loop). We need to await
>> such threads finishing too while keep event loop running. In approach of
>> stateShutdown hook we finish all threads that uses qemu monitor by closing
>> the monitor.
>>
>> And hack with timeout in a loop... I think of a different aproach for waiting
>> rpc threads to finish their work. First let's make drain function only flush
>> job queue and take some callback which will be called when all workers will
>> be free from work (let's keep worker threads as Dan suggested). In this callback we
>> can use same technique as in virNetDaemonSignalHandler. That is make some
>> IO to set some flag in the event loop thread and cause virEventRunDefaultImpl
>> to finish and then test this flag in virNetDaemonRun.
>>
>
> Please feel free to spend as many cycles as you can making adjustments
> to what I have. I'm working through some alterations and posting a v2 of
> what I have and we'll see where it takes me/us.
>
> John

Is there any update so far? I’m asking because I’m still getting
segmentation faults and hang-ups on termination of libvirtd (using the
newest version of libvirt).

Example for a hang-up:
➤  bt
#0  0x000003fffca8df84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000003fffdac29ca in virCondWait (c=<optimized out>, m=<optimized out>) at ../../src/util/virthread.c:154
#2  0x000003fffdac381c in virThreadPoolFree (pool=<optimized out>) at ../../src/util/virthreadpool.c:290
#3  0x000003fffdbb21ae in virNetServerDispose (obj=0x1000cc640) at ../../src/rpc/virnetserver.c:803
#4  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
#5  0x000003fffda97a5a in virObjectFreeHashData (opaque=<optimized out>, name=<optimized out>) at ../../src/util/virobject.c:591
#6  0x000003fffda66576 in virHashFree (table=<optimized out>) at ../../src/util/virhash.c:305
#7  0x000003fffdbaff82 in virNetDaemonDispose (obj=0x1000cc3c0) at ../../src/rpc/virnetdaemon.c:105
#8  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
#9  0x0000000100026cd6 in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1487

And segmentation faults happen for RPC jobs that are still running.

[…snip]

--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by John Ferlan 5 years, 8 months ago
> 
> Is there any update so far? I’m asking because I’m still getting
> segmentation faults and hang-ups on termination of libvirtd (using the
> newest version of libvirt).
> 
> Example for a hang-up:
> ➤  bt
> #0  0x000003fffca8df84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1  0x000003fffdac29ca in virCondWait (c=<optimized out>, m=<optimized out>) at ../../src/util/virthread.c:154
> #2  0x000003fffdac381c in virThreadPoolFree (pool=<optimized out>) at ../../src/util/virthreadpool.c:290
> #3  0x000003fffdbb21ae in virNetServerDispose (obj=0x1000cc640) at ../../src/rpc/virnetserver.c:803
> #4  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
> #5  0x000003fffda97a5a in virObjectFreeHashData (opaque=<optimized out>, name=<optimized out>) at ../../src/util/virobject.c:591
> #6  0x000003fffda66576 in virHashFree (table=<optimized out>) at ../../src/util/virhash.c:305
> #7  0x000003fffdbaff82 in virNetDaemonDispose (obj=0x1000cc3c0) at ../../src/rpc/virnetdaemon.c:105
> #8  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
> #9  0x0000000100026cd6 in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1487
> 
> And segmentation faults happen for RPC jobs that are still running.
> 

There has been zero of my cycles spent thinking about this. Partially
because I'm busy in other areas, partially because I know Daniel is
planning changes in libvirtd
(https://www.redhat.com/archives/libvir-list/2018-May/msg01307.html),
and partially because I'm not sure I have a {reliable|simple} reproducer
(at least I don't recall).

I do still have various branches in various states of disarray that are
way behind current head (easy to happen it seems lately).

John

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Marc Hartmayer 5 years, 8 months ago
On Tue, Jul 03, 2018 at 09:21 PM +0200, John Ferlan <jferlan@redhat.com> wrote:
>>
>> Is there any update so far? I’m asking because I’m still getting
>> segmentation faults and hang-ups on termination of libvirtd (using the
>> newest version of libvirt).
>>
>> Example for a hang-up:
>> ➤  bt
>> #0  0x000003fffca8df84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>> #1  0x000003fffdac29ca in virCondWait (c=<optimized out>, m=<optimized out>) at ../../src/util/virthread.c:154
>> #2  0x000003fffdac381c in virThreadPoolFree (pool=<optimized out>) at ../../src/util/virthreadpool.c:290
>> #3  0x000003fffdbb21ae in virNetServerDispose (obj=0x1000cc640) at ../../src/rpc/virnetserver.c:803
>> #4  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
>> #5  0x000003fffda97a5a in virObjectFreeHashData (opaque=<optimized out>, name=<optimized out>) at ../../src/util/virobject.c:591
>> #6  0x000003fffda66576 in virHashFree (table=<optimized out>) at ../../src/util/virhash.c:305
>> #7  0x000003fffdbaff82 in virNetDaemonDispose (obj=0x1000cc3c0) at ../../src/rpc/virnetdaemon.c:105
>> #8  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
>> #9  0x0000000100026cd6 in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1487
>>
>> And segmentation faults happen for RPC jobs that are still running.
>>
>
> There has been zero of my cycles spent thinking about this. Partially
> because I'm busy in other areas, partially because I know Daniel is
> planning changes in libvirtd
> (https://www.redhat.com/archives/libvir-list/2018-May/msg01307.html),
> and partially because I'm not sure I have a {reliable|simple} reproducer
> (at least I don't recall).

I do a simple start/destroy loop and send a SIGTERM to libvirtd. This
leads in almost every case to a segmentation fault.

>
> I do still have various branches in various states of disarray that are
> way behind current head (easy to happen it seems lately).
>
> John
>
--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by John Ferlan 5 years, 8 months ago

On 07/06/2018 05:15 AM, Marc Hartmayer wrote:
> On Tue, Jul 03, 2018 at 09:21 PM +0200, John Ferlan <jferlan@redhat.com> wrote:
>>>
>>> Is there any update so far? I’m asking because I’m still getting
>>> segmentation faults and hang-ups on termination of libvirtd (using the
>>> newest version of libvirt).
>>>
>>> Example for a hang-up:
>>> ➤  bt
>>> #0  0x000003fffca8df84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1  0x000003fffdac29ca in virCondWait (c=<optimized out>, m=<optimized out>) at ../../src/util/virthread.c:154
>>> #2  0x000003fffdac381c in virThreadPoolFree (pool=<optimized out>) at ../../src/util/virthreadpool.c:290
>>> #3  0x000003fffdbb21ae in virNetServerDispose (obj=0x1000cc640) at ../../src/rpc/virnetserver.c:803
>>> #4  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
>>> #5  0x000003fffda97a5a in virObjectFreeHashData (opaque=<optimized out>, name=<optimized out>) at ../../src/util/virobject.c:591
>>> #6  0x000003fffda66576 in virHashFree (table=<optimized out>) at ../../src/util/virhash.c:305
>>> #7  0x000003fffdbaff82 in virNetDaemonDispose (obj=0x1000cc3c0) at ../../src/rpc/virnetdaemon.c:105
>>> #8  0x000003fffda97286 in virObjectUnref (anyobj=<optimized out>) at ../../src/util/virobject.c:350
>>> #9  0x0000000100026cd6 in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1487
>>>
>>> And segmentation faults happen for RPC jobs that are still running.
>>>
>>
>> There has been zero of my cycles spent thinking about this. Partially
>> because I'm busy in other areas, partially because I know Daniel is
>> planning changes in libvirtd
>> (https://www.redhat.com/archives/libvir-list/2018-May/msg01307.html),
>> and partially because I'm not sure I have a {reliable|simple} reproducer
>> (at least I don't recall).
> 
> I do a simple start/destroy loop and send a SIGTERM to libvirtd. This
> leads in almost every case to a segmentation fault.
> 
>>
>> I do still have various branches in various states of disarray that are
>> way behind current head (easy to happen it seems lately).
>>

I found/updated my branches and started recalling where things were left
off. I had R-By's for patch 1 & 2 until you noted the issue with the
global variables which was fixed in my branch.

Since there's been a lot of code motion since and what I have now is
different I figure I should repost the series and go from there. I
haven't addressed the use pipe instead of polling comment you had in
patch7.  I haven't put much thought into it, but in general given how
things work how would you expect pipe signaling to work in this model?
Essentially we have a client that either is not responding or is in the
middle of something large.

NB: I won't be around for the first half of next week to see any
responses or answer questions...  Perhaps it's vacation mode that isn't
allowing me to think about the pipe signaling logic right now ;-)

John

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Marc Hartmayer 6 years, 2 months ago
On Fri, Jan 19, 2018 at 06:23 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
> RFC:
> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>
> Adjustments since RFC...
>
> Patches 1&2: No change, were already R-B'd
> Patch 3: Removed code as noted in code review, update commit message
> Patch 4: From old series removed, see below for more details
> Patch 9: no change
> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>     are removed as they seemed to not be necessary
>
> Replaced the former patch 4 with series of patches to (slowly) provide
> support to disable new connections, handle removing waiting jobs, causing
> the waiting workers to quit, and allow any running jobs to complete.
>
> As it turns out, waiting for running jobs to complete cannot be done
> from the virNetServerClose callbacks because that means the event loop
> processing done during virNetServerRun will not allow any currently
> long running worker job thread a means to complete.
>
> So when virNetDaemonQuit is called as a result of the libvirtd signal
> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
> to quit immediately, alter to using a quitRequested flag and then use
> that quitRequested flag to check for long running worker threads before
> causing the event loop to quit resulting in libvirtd being able to run
> through the virNetDaemonClose processing.

Gave a quick test:
 + Didn't get a segmentation fault at the end of libvirtd (at least for
   my quick test)
 - a single SIGTERM doesn’t always lead to the termination of libvirtd
   now (debugged it: main thread is waiting for poll()). This behavior
   can be easily reproduced: Start libvirtd on the CLI, wait some
   seconds for the first initialization -> CTRL + C -> libvirtd doesn’t
   terminate, but also doesn’t accept new connections.

>
> John Ferlan (9):
>   libvirtd: Alter refcnt processing for domain server objects
>   libvirtd: Alter refcnt processing for server program objects
>   netserver: Remove ServiceToggle during ServerDispose
>   util: Introduce virThreadPoolDrain
>   rpc: Introduce virNetServerQuitRequested
>   rpc: Introduce virNetServerWorkerCount
>   rpc: Alter virNetDaemonQuit processing
>   docs: Add news article for libvirtd issue
>   APPLY ONLY FOR TESTING PURPOSES
>
>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>  docs/news.xml            | 12 +++++++++
>  src/libvirt_private.syms |  1 +
>  src/libvirt_remote.syms  |  2 ++
>  src/qemu/qemu_driver.c   |  5 ++++
>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>  src/rpc/virnetserver.h   |  4 +++
>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>  src/util/virthreadpool.h |  2 ++
>  10 files changed, 204 insertions(+), 26 deletions(-)
>
> --
> 2.13.6
>
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list
>
--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by John Ferlan 6 years, 2 months ago

On 01/23/2018 04:21 AM, Marc Hartmayer wrote:
> On Fri, Jan 19, 2018 at 06:23 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
>> RFC:
>> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>>
>> Adjustments since RFC...
>>
>> Patches 1&2: No change, were already R-B'd
>> Patch 3: Removed code as noted in code review, update commit message
>> Patch 4: From old series removed, see below for more details
>> Patch 9: no change
>> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>>     are removed as they seemed to not be necessary
>>
>> Replaced the former patch 4 with series of patches to (slowly) provide
>> support to disable new connections, handle removing waiting jobs, causing
>> the waiting workers to quit, and allow any running jobs to complete.
>>
>> As it turns out, waiting for running jobs to complete cannot be done
>> from the virNetServerClose callbacks because that means the event loop
>> processing done during virNetServerRun will not allow any currently
>> long running worker job thread a means to complete.
>>
>> So when virNetDaemonQuit is called as a result of the libvirtd signal
>> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
>> to quit immediately, alter to using a quitRequested flag and then use
>> that quitRequested flag to check for long running worker threads before
>> causing the event loop to quit resulting in libvirtd being able to run
>> through the virNetDaemonClose processing.
> 
> Gave a quick test:
>  + Didn't get a segmentation fault at the end of libvirtd (at least for
>    my quick test)
>  - a single SIGTERM doesn’t always lead to the termination of libvirtd
>    now (debugged it: main thread is waiting for poll()). This behavior
>    can be easily reproduced: Start libvirtd on the CLI, wait some
>    seconds for the first initialization -> CTRL + C -> libvirtd doesn’t
>    terminate, but also doesn’t accept new connections.
> 

Does this include the "update" in my response to patch 7?  It should be
extract-able and apply-able.

John

>>
>> John Ferlan (9):
>>   libvirtd: Alter refcnt processing for domain server objects
>>   libvirtd: Alter refcnt processing for server program objects
>>   netserver: Remove ServiceToggle during ServerDispose
>>   util: Introduce virThreadPoolDrain
>>   rpc: Introduce virNetServerQuitRequested
>>   rpc: Introduce virNetServerWorkerCount
>>   rpc: Alter virNetDaemonQuit processing
>>   docs: Add news article for libvirtd issue
>>   APPLY ONLY FOR TESTING PURPOSES
>>
>>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>>  docs/news.xml            | 12 +++++++++
>>  src/libvirt_private.syms |  1 +
>>  src/libvirt_remote.syms  |  2 ++
>>  src/qemu/qemu_driver.c   |  5 ++++
>>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>>  src/rpc/virnetserver.h   |  4 +++
>>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>>  src/util/virthreadpool.h |  2 ++
>>  10 files changed, 204 insertions(+), 26 deletions(-)
>>
>> --
>> 2.13.6
>>
>> --
>> libvir-list mailing list
>> libvir-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/libvir-list
>>
> --
> Beste Grüße / Kind regards
>    Marc Hartmayer
> 
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
> 

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Marc Hartmayer 6 years, 2 months ago
On Tue, Jan 23, 2018 at 04:01 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
> On 01/23/2018 04:21 AM, Marc Hartmayer wrote:
>> On Fri, Jan 19, 2018 at 06:23 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
>>> RFC:
>>> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>>>
>>> Adjustments since RFC...
>>>
>>> Patches 1&2: No change, were already R-B'd
>>> Patch 3: Removed code as noted in code review, update commit message
>>> Patch 4: From old series removed, see below for more details
>>> Patch 9: no change
>>> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>>>     are removed as they seemed to not be necessary
>>>
>>> Replaced the former patch 4 with series of patches to (slowly) provide
>>> support to disable new connections, handle removing waiting jobs, causing
>>> the waiting workers to quit, and allow any running jobs to complete.
>>>
>>> As it turns out, waiting for running jobs to complete cannot be done
>>> from the virNetServerClose callbacks because that means the event loop
>>> processing done during virNetServerRun will not allow any currently
>>> long running worker job thread a means to complete.
>>>
>>> So when virNetDaemonQuit is called as a result of the libvirtd signal
>>> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
>>> to quit immediately, alter to using a quitRequested flag and then use
>>> that quitRequested flag to check for long running worker threads before
>>> causing the event loop to quit resulting in libvirtd being able to run
>>> through the virNetDaemonClose processing.
>>
>> Gave a quick test:
>>  + Didn't get a segmentation fault at the end of libvirtd (at least for
>>    my quick test)
>>  - a single SIGTERM doesn’t always lead to the termination of libvirtd
>>    now (debugged it: main thread is waiting for poll()). This behavior
>>    can be easily reproduced: Start libvirtd on the CLI, wait some
>>    seconds for the first initialization -> CTRL + C -> libvirtd doesn’t
>>    terminate, but also doesn’t accept new connections.
>>
>
> Does this include the "update" in my response to patch 7?  It should be
> extract-able and apply-able.

No, I forgot this one :( I’ll try it tomorrow.

>
> John
>
>>>
>>> John Ferlan (9):
>>>   libvirtd: Alter refcnt processing for domain server objects
>>>   libvirtd: Alter refcnt processing for server program objects
>>>   netserver: Remove ServiceToggle during ServerDispose
>>>   util: Introduce virThreadPoolDrain
>>>   rpc: Introduce virNetServerQuitRequested
>>>   rpc: Introduce virNetServerWorkerCount
>>>   rpc: Alter virNetDaemonQuit processing
>>>   docs: Add news article for libvirtd issue
>>>   APPLY ONLY FOR TESTING PURPOSES
>>>
>>>  daemon/libvirtd.c        | 43 +++++++++++++++++++++++---------
>>>  docs/news.xml            | 12 +++++++++
>>>  src/libvirt_private.syms |  1 +
>>>  src/libvirt_remote.syms  |  2 ++
>>>  src/qemu/qemu_driver.c   |  5 ++++
>>>  src/rpc/virnetdaemon.c   | 45 +++++++++++++++++++++++++++++++++-
>>>  src/rpc/virnetserver.c   | 52 ++++++++++++++++++++++++++++++++++++---
>>>  src/rpc/virnetserver.h   |  4 +++
>>>  src/util/virthreadpool.c | 64 ++++++++++++++++++++++++++++++++++++++++--------
>>>  src/util/virthreadpool.h |  2 ++
>>>  10 files changed, 204 insertions(+), 26 deletions(-)
>>>
>>> --
>>> 2.13.6
>>>
>>> --
>>> libvir-list mailing list
>>> libvir-list@redhat.com
>>> https://www.redhat.com/mailman/listinfo/libvir-list
>>>
>> --
>> Beste Grüße / Kind regards
>>    Marc Hartmayer
>>
>> IBM Deutschland Research & Development GmbH
>> Vorsitzende des Aufsichtsrats: Martina Koederitz
>> Geschäftsführung: Dirk Wittkopp
>> Sitz der Gesellschaft: Böblingen
>> Registergericht: Amtsgericht Stuttgart, HRB 243294
>>
>
--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 0/9] Resolve libvirtd hang on termination with connected long running client
Posted by Marc Hartmayer 6 years, 2 months ago
On Tue, Jan 23, 2018 at 05:41 PM +0100, Marc Hartmayer <mhartmay@linux.vnet.ibm.com> wrote:
> On Tue, Jan 23, 2018 at 04:01 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
>> On 01/23/2018 04:21 AM, Marc Hartmayer wrote:
>>> On Fri, Jan 19, 2018 at 06:23 PM +0100, John Ferlan <jferlan@redhat.com> wrote:
>>>> RFC:
>>>> https://www.redhat.com/archives/libvir-list/2018-January/msg00318.html
>>>>
>>>> Adjustments since RFC...
>>>>
>>>> Patches 1&2: No change, were already R-B'd
>>>> Patch 3: Removed code as noted in code review, update commit message
>>>> Patch 4: From old series removed, see below for more details
>>>> Patch 9: no change
>>>> NB: Patches 5-8 and 10 from Nikolay Shirokovskiy <nshirokovskiy@virtuozzo.com>
>>>>     are removed as they seemed to not be necessary
>>>>
>>>> Replaced the former patch 4 with series of patches to (slowly) provide
>>>> support to disable new connections, handle removing waiting jobs, causing
>>>> the waiting workers to quit, and allow any running jobs to complete.
>>>>
>>>> As it turns out, waiting for running jobs to complete cannot be done
>>>> from the virNetServerClose callbacks because that means the event loop
>>>> processing done during virNetServerRun will not allow any currently
>>>> long running worker job thread a means to complete.
>>>>
>>>> So when virNetDaemonQuit is called as a result of the libvirtd signal
>>>> handlers for SIG{QUIT|INT|TERM}, instead of just causing virNetServerRun
>>>> to quit immediately, alter to using a quitRequested flag and then use
>>>> that quitRequested flag to check for long running worker threads before
>>>> causing the event loop to quit resulting in libvirtd being able to run
>>>> through the virNetDaemonClose processing.
>>>
>>> Gave a quick test:
>>>  + Didn't get a segmentation fault at the end of libvirtd (at least for
>>>    my quick test)
>>>  - a single SIGTERM doesn’t always lead to the termination of libvirtd
>>>    now (debugged it: main thread is waiting for poll()). This behavior
>>>    can be easily reproduced: Start libvirtd on the CLI, wait some
>>>    seconds for the first initialization -> CTRL + C -> libvirtd doesn’t
>>>    terminate, but also doesn’t accept new connections.
>>>
>>
>> Does this include the "update" in my response to patch 7?  It should be
>> extract-able and apply-able.
>
> No, I forgot this one :( I’ll try it tomorrow.

It works with the new patch 7.

[…snip]

--
Beste Grüße / Kind regards
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list