[libvirt] [PATCH 0/3] fix crash on libvirtd termination

Nikolay Shirokovskiy posted 3 patches 6 years, 6 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/1506516338-372467-1-git-send-email-nshirokovskiy@virtuozzo.com
daemon/libvirtd.c        |  2 ++
src/driver-state.h       |  4 ++++
src/libvirt.c            | 18 ++++++++++++++++++
src/libvirt_internal.h   |  1 +
src/libvirt_private.syms |  1 +
src/qemu/qemu_driver.c   | 39 +++++++++++++++++++++++++++++++++++++++
src/qemu/qemu_monitor.c  | 27 +++++++++++++--------------
src/rpc/virnetserver.c   |  5 +++--
8 files changed, 81 insertions(+), 16 deletions(-)
[libvirt] [PATCH 0/3] fix crash on libvirtd termination
Posted by Nikolay Shirokovskiy 6 years, 6 months ago
Libvirtd termination can crash. One can use [2] patch to trigger it. Call
domstats function and send TERM to libvirtd. You'd probably see stacktrace [1].
The problem is that threads with clients requests are joined after drivers
cleanup. This patch series address this issue.

[1] Crash stacktrace

    Program received signal SIGSEGV, Segmentation fault.

    Thread 5 (Thread 0x7fffe6a4d700 (LWP 921916)):
    #0  0x00007fffd9cb3f14 in qemuDomainObjBeginJobInternal (driver=driver@entry=0x7fffcc103e40, 
        obj=obj@entry=0x7fffcc1a6ca0, job=job@entry=QEMU_JOB_QUERY, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE)
        at qemu/qemu_domain.c:4114
    #1  0x00007fffd9cb82ab in qemuDomainObjBeginJob (driver=driver@entry=0x7fffcc103e40, obj=obj@entry=0x7fffcc1a6ca0, 
        job=job@entry=QEMU_JOB_QUERY) at qemu/qemu_domain.c:4240
    #2  0x00007fffd9d23094 in qemuConnectGetAllDomainStats (conn=0x7fffcc1bc140, doms=<optimized out>, 
        ndoms=<optimized out>, stats=127, retStats=0x7fffe6a4cb10, flags=<optimized out>) at qemu/qemu_driver.c:20116
    #3  0x00007ffff744a166 in virDomainListGetStats (doms=0x7fffa8000a10, stats=0, 
        retStats=retStats@entry=0x7fffe6a4cb10, flags=0) at libvirt-domain.c:11592
    #4  0x000055555557af15 in remoteDispatchConnectGetAllDomainStats (server=<optimized out>, msg=<optimized out>, 
        ret=0x7fffa80008e0, args=0x7fffa80008c0, rerr=0x7fffe6a4cc50, client=<optimized out>) at remote.c:6532
    #5  remoteDispatchConnectGetAllDomainStatsHelper (server=<optimized out>, client=<optimized out>, 
        msg=<optimized out>, rerr=0x7fffe6a4cc50, args=0x7fffa80008c0, ret=0x7fffa80008e0) at remote_dispatch.h:615
    #6  0x00007ffff74abba2 in virNetServerProgramDispatchCall (msg=0x55555583bf50, client=0x55555583c580, 
        server=0x555555810f40, prog=0x55555583a140) at rpc/virnetserverprogram.c:437
    #7  virNetServerProgramDispatch (prog=0x55555583a140, server=server@entry=0x555555810f40, client=0x55555583c580, 
        msg=0x55555583bf50) at rpc/virnetserverprogram.c:307
    #8  0x00005555555ae10d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, 
        srv=0x555555810f40) at rpc/virnetserver.c:148
    #9  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x555555810f40) at rpc/virnetserver.c:169
    #10 0x00007ffff7390fd1 in virThreadPoolWorker (opaque=opaque@entry=0x5555558057a0) at util/virthreadpool.c:167
    #11 0x00007ffff7390358 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
    #12 0x00007ffff457be25 in start_thread () from /lib64/libpthread.so.0
    #13 0x00007ffff42a934d in clone () from /lib64/libc.so.6

    Thread 1 (Thread 0x7ffff7fae880 (LWP 921909)):
    #0  0x00007ffff457f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
    #1  0x00007ffff73905c6 in virCondWait (c=<optimized out>, m=<optimized out>) at util/virthread.c:154
    #2  0x00007ffff73911e0 in virThreadPoolFree (pool=0x555555811030) at util/virthreadpool.c:290
    #3  0x00005555555adb44 in virNetServerDispose (obj=0x555555810f40) at rpc/virnetserver.c:767
    #4  0x00007ffff736f62b in virObjectUnref (anyobj=<optimized out>) at util/virobject.c:356
    #5  0x00007ffff7343e19 in virHashFree (table=0x55555581ba40) at util/virhash.c:318
    #6  0x00007ffff74a46b5 in virNetDaemonDispose (obj=0x555555812c50) at rpc/virnetdaemon.c:105
    #7  0x00007ffff736f62b in virObjectUnref (anyobj=anyobj@entry=0x555555812c50) at util/virobject.c:356
    #8  0x0000555555570479 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1539


[2] patch to trigger crash
    #   diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
    #   index cf5e4ad..39a57aa 100644
    #   --- a/src/qemu/qemu_driver.c
    #   +++ b/src/qemu/qemu_driver.c
    #   @@ -20144,6 +20144,8 @@ qemuConnectGetAllDomainStats(virConnectPtr conn,
    #        domflags = 0;
    #        vm = vms[i];
    #
    #   +    sleep(5);
    #   +
    #        virObjectLock(vm);
    #
    #        if (HAVE_JOB(privflags) &&

Nikolay Shirokovskiy (3):
  daemon: finish threads on close
  qemu: monitor: check monitor not closed on send
  qemu: implement state driver shutdown function

 daemon/libvirtd.c        |  2 ++
 src/driver-state.h       |  4 ++++
 src/libvirt.c            | 18 ++++++++++++++++++
 src/libvirt_internal.h   |  1 +
 src/libvirt_private.syms |  1 +
 src/qemu/qemu_driver.c   | 39 +++++++++++++++++++++++++++++++++++++++
 src/qemu/qemu_monitor.c  | 27 +++++++++++++--------------
 src/rpc/virnetserver.c   |  5 +++--
 8 files changed, 81 insertions(+), 16 deletions(-)

-- 
1.8.3.1

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list