From nobody Tue May 7 22:59:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) client-ip=207.211.31.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586963597; cv=none; d=zohomail.com; s=zohoarc; b=fC+y/8X3W1IRRZ6xifiGD729o+wGNTmIeJgcVbxMnltGQz59vb2yOs0xtkTdocihHsPCJs4FmhkPiTRMJ9tlU0H59a6P5Y0Sv8IqX04x3jQDM995da75SO9LO5r5+zdkobYYJUUcTyzIqpCWRlLSByMBf2YNLOcyQbXMjPGNaEk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586963597; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=Qgk9/GNqisdhC3VCdS9pV2d648wW9J3bQUV4GcLLm/A=; b=FVyma7JYFyCnSxFNAsye2urwYJe79a/XEvig9xU1yzbcI7omXE5CCEb1DR7ISMG+0rJ098EHZndUfDxwlXI7kf7IVn6y3DmjOUl9d3mcs6ItmnyBaRpwcNJQz+vYaMkAQN4534OIe/LG09CJzIooEmDpmoWZN0ngUAwsUXOVXPo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) by mx.zohomail.com with SMTPS id 1586963597232538.2504227161384; Wed, 15 Apr 2020 08:13:17 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-257-PRAmEKKJMTe2x7qPTU17Gw-1; Wed, 15 Apr 2020 11:13:04 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E1A0D149D3; Wed, 15 Apr 2020 15:12:55 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 75F2560BF3; Wed, 15 Apr 2020 15:12:55 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 143AB18089CD; Wed, 15 Apr 2020 15:12:55 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 03FFAvLR009060 for ; Wed, 15 Apr 2020 11:10:57 -0400 Received: by smtp.corp.redhat.com (Postfix) id 4E991101D482; Wed, 15 Apr 2020 15:10:57 +0000 (UTC) Received: from lpt.redhat.com (unknown [10.40.208.65]) by smtp.corp.redhat.com (Postfix) with ESMTP id 85D0A101D481 for ; Wed, 15 Apr 2020 15:10:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586963595; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=Qgk9/GNqisdhC3VCdS9pV2d648wW9J3bQUV4GcLLm/A=; b=H1qaK+wXibCA/7LYW+edtYAgClpGH5Vqe2hLbPFizBmnPjjNJUqmEfsR+uYmNmP5x/Ngbw N9wMfjvZlCp5qAtuSJa8xUClAq9pitYhLCI2ZDhOdrOlKrvSeahM7WrZyGEIYEGc/tU5WR VOqbI5Zza1661T+ptzhE1uNCKh9+No0= X-MC-Unique: PRAmEKKJMTe2x7qPTU17Gw-1 From: =?UTF-8?q?J=C3=A1n=20Tomko?= To: libvir-list@redhat.com Subject: [libvirt PATCH] docs: kbase: add a page about debugging Date: Wed, 15 Apr 2020 17:10:45 +0200 Message-Id: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id 03FFAvLR009060 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Migrate the following wiki articles: https://wiki.libvirt.org/page/DebugLogs https://wiki.libvirt.org/page/Debugging With the syntax changed to rST and rewrapped. Signed-off-by: J=C3=A1n Tomko --- I might've changed an article or two, too. Or rather: the article 'the'. docs/kbase.html.in | 3 + docs/kbase/debugging.rst | 303 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 306 insertions(+) create mode 100644 docs/kbase/debugging.rst diff --git a/docs/kbase.html.in b/docs/kbase.html.in index c586e0f676..1a50428805 100644 --- a/docs/kbase.html.in +++ b/docs/kbase.html.in @@ -36,6 +36,9 @@ =20
Virtio-FS
Share a filesystem between the guest and the host
+ +
Debugging libvirt
+
How to capture debug logs and use a debugger on libvirt
=20 diff --git a/docs/kbase/debugging.rst b/docs/kbase/debugging.rst new file mode 100644 index 0000000000..a9bc02aa55 --- /dev/null +++ b/docs/kbase/debugging.rst @@ -0,0 +1,303 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Debugging libvirt +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. contents:: + + +If you `report a bug `_ against libvirt, in= most +cases you will be asked to attach debug logs. These are bare text files wh= ich +tracks transition between different states of libvirtd, what it has tried = to +achieve, etc. Because of client -- server schema used in libvirt, the logs= can +be either client or server too. Usually, it's server side that matters as +nearly all interesting work takes place there. Moreover, libvirt catches s= tderr +of all running domains. These can be useful as well. + + +How to turn on debug logs for libvirt +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Persistent setting +------------------ + +The daemon configuration files location is dependent on the `connection URI +`_. For ``qemu:///system``: + +* open ``/etc/libvirt/libvirtd.conf`` in your favourite editor +* find & replace, or set these variables:: + + # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 SEE BELOW! # + log_level =3D 1 + log_filters=3D"1:qemu 3:remote 4:event 3:util.json 3:rpc" + log_outputs=3D"1:file:/var/log/libvirt/libvirtd.log" + + + # PREFERRED SETTINGS AFTER LIBVIRT 4.4.0 # + log_filters=3D"3:remote 4:event 3:util.json 3:rpc 1:*" + log_outputs=3D"1:file:/var/log/libvirt/libvirtd.log" + +* save and exit +* restart libvirtd service:: + + systemctl restart libvirtd.service + +In the config variables above, we have set logging level to 1 (debug level= ), +set some filters (to filter out noise), e.g. from rpc only warnings (=3Dle= vel 3) +and above will be reported. The logs are saved into +``/var/log/libvirt/libvirtd.log``. Since libvirt *4.4.0* log filters +support shell globbing, therefore the usage of ``log_level`` is considered +deprecated in favour of pure usage of ``log_filters``. + +In case you want to get the client logs, you need to set this environment = variable:: + + export LIBVIRT_LOG_OUTPUTS=3D"1:file:/tmp/libvirt_client.log" + +However, when you are using the session mode ``qemu:///session`` or you ru= n the +``libvirtd`` as unprivileged user you will find configuration file under +``$XDG_CONFIG_HOME/libvirt/libvirt.conf``. + + +Runtime setting +--------------- + +Debugging anomalies can be very painful, especially when trying to reprodu= ce it +after the daemon restarts, since the new session can make the anomaly +"disappear". Therefore, it's possible to enable the debug logs during runt= ime +using libvirt administration API. To use it conveniently, there's a ``virt= -admin`` +client provided by the ``libvirt-admin`` package. Use the package manager = provided +by your distribution to install this package. Once you have it installed, = run +the following as root to see the set of log filters currently being active= :: + + # virt-admin daemon-log-filters + Logging filters: 3:remote 4:util.json 4:rpc + +In order to change this set, run the same command as root, this time with = your own set of filters:: + + ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## + # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1= :nodedev 1:qemu" + + ## CURRENT APPROACH USING SHELL GLOBBING ## + # virt-admin daemon-log-filters "3:remote 4:util.json 4:rpc 1:*" + +Analogically, the same procedure can be performed with log outputs:: + + # virt-admin daemon-log-outputs + Logging outputs: 3:syslog:libvirtd + # virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" + +NOTE: It's always good practice to return the settings to the original sta= te +once you're finished debugging, just remember to save the original sets of +filters and outputs and restore them at the end the same way as described +above. + + +Removing filters and outputs +---------------------------- + +It's also possible to remove all the filters and produce an enormous log f= ile, +but it is not recommended since some of libvirt's modules can produce a la= rge +amount of noise. However, should you really want to do this, you can speci= fy an +empty set of filters:: + + # virt-admin daemon-log-filters "" + Logging filters: + +The situation is a bit different with outputs, since libvirt always has to= log +somewhere and resetting the outputs to an empty set will restore the defau= lt +setting which depends on the host configuration, ''journald'' in our case:: + + # virt-admin daemon-log-outputs + Logging outputs: 1:file:/var/log/libvirt/libvirtd.log + # virt-admin daemon-log-outputs "" + Logging outputs: 2:journald + + +What to attach? +--------------- + +Now you should go and reproduce the bug. Once you're finished, attach: + +* ``/var/log/libvirt/libvirtd.log`` or whatever path you set for the daemo= n logs. +* If the problem is related to a domain attach ``/var/log/libvirt/qemu/$do= m.log`` + then. Or substitute ``qemu`` with whatever hypervisor you are using. +* If you are asked for client logs, ``/tmp/libvirt_client.log``. + + +Using a debugger +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +When trying to figure out what libvirt does, debug logs might not always be +enough. And sometimes you might want to get some information from a user,= but +you do not want to waste both your and their time by explaining how to do = stuff +in gdb to, for example, get a backtrace. Here are some useful tips that y= ou +might use. + + +Prerequisites +------------- + +In cases where you want to see details of what is happening, you need to h= ave +debugging symbols installed, at least for the package you are trying to de= bug. +Although having debugging symbols for all dependent libraries is usually +helpful as well. Usually ``gdb`` will tell you what you need to do in orde= r to +get the proper data to your machine when you run it with a binary. + +=3D=3D=3D Example: =3D=3D=3D + +Running this command on 32bit Fedora 29 tells us what to install in order = to +get the proper debugging symbols:: + + # gdb $(which libvirtd) + GNU gdb (GDB) Fedora 8.2-6.fc29 + ... + Reading symbols from /usr/sbin/libvirtd...(no debugging symbols found).= ..done. + Missing separate debuginfos, use: dnf debuginfo-install libvirt-daemon-= 4.7.0-1.fc29.i686 + +When the package is installed, we can break on main and run until then (gd= b's +command ``start`` is perfect for this):: + + # gdb $(which libvirtd) + GNU gdb (GDB) Fedora 8.2-6.fc29 + ... + Reading symbols from /usr/sbin/libvirtd...Reading symbols from /usr/lib= /debug/usr/sbin/libvirtd-4.7.0-1.fc29.i386.debug...done. + done. + (gdb) start + Temporary breakpoint 1 at 0x18fc0: file remote/remote_daemon.c, line 10= 30. + Starting program: /usr/sbin/libvirtd + Missing separate debuginfos, use: dnf debuginfo-install glibc-2.28-26.f= c29.i686 + Missing separate debuginfo for /lib/libvirt-lxc.so.0 + Try: dnf --enablerepo=3D'*debug*' install /usr/lib/debug/.build-id/4d/1= 6496b686ec54ca4201bd769b04293f6c756b3.debug + Missing separate debuginfo for /lib/libvirt-qemu.so.0 + Try: dnf --enablerepo=3D'*debug*' install /usr/lib/debug/.build-id/ea/9= 1d5346bd3e265ffb12ae641ca93643443e6e7.debug + Missing separate debuginfo for /lib/libvirt.so.0 + Try: dnf --enablerepo=3D'*debug*' install /usr/lib/debug/.build-id/02/a= f3a96fc6227ed5e3a447344bcbb672bde14ba.debug + ... + Temporary breakpoint 1, main (argc=3D1, argv=3D0xbffff614) at remote/re= mote_daemon.c:1030 + 1030 int main(int argc, char **argv) { + Missing separate debuginfos, use: dnf debuginfo-install audit-libs-3.0-= 0.5.20181218gitbdb72c0.fc29.i686 avahi-libs-0.7-16.fc29.i686 brotli-1.0.5-1= .fc29.i686 cyrus-sasl-lib-2.1.27-0.3rc7.fc29.i686 dbus-libs-1.12.12-1.fc29.= i686 device-mapper-libs-1.02.154-1.fc29.i686 gmp-6.1.2-9.fc29.i686 gnutls-3= .6.6-1.fc29.i686 keyutils-libs-1.5.10-8.fc29.i686 krb5-libs-1.16.1-25.fc29.= i686 libacl-2.2.53-2.fc29.i686 libattr-2.4.48-3.fc29.i686 libblkid-2.32.1-1= .fc29.i686 libcap-2.25-12.fc29.i686 libcap-ng-0.7.9-5.fc29.i686 libcom_err-= 1.44.4-1.fc29.i686 libcurl-7.61.1-10.fc29.i686 libffi-3.1-18.fc29.i686 libg= crypt-1.8.4-1.fc29.i686 libidn2-2.1.1a-1.fc29.i686 libmount-2.32.1-1.fc29.i= 686 libnghttp2-1.34.0-1.fc29.i686 libnl3-3.4.0-6.fc29.i686 libpsl-0.20.2-5.= fc29.i686 libselinux-2.8-6.fc29.i686 libsepol-2.8-3.fc29.i686 libssh-0.8.7-= 1.fc29.i686 libssh2-1.8.1-1.fc29.i686 libtirpc-1.1.4-2.rc2.fc29.i686 libuni= string-0.9.10-4.fc29.i686 libuuid-2.32.1-1.fc29.i686 libwsman1-2.6.5-8.fc29= .i686 libxcrypt-4.4.4-2 .fc29.i686 libxml2-2.9.8-5.fc29.i686 lz4-libs-1.8.3-1.fc29.i686 numactl-li= bs-2.0.12-1.fc29.i686 openldap-2.4.46-10.fc29.i686 openssl-libs-1.1.1b-3.fc= 29.i686 p11-kit-0.23.15-2.fc29.i686 pcre2-10.32-8.fc29.i686 xz-libs-5.2.4-3= .fc29.i686 yajl-2.1.0-11.fc29.i686 zlib-1.2.11-14.fc29.i686 + +You might need to run the above commands for more complete output. It is = very +dependent on the actually problem, whether you need this or not, but it wi= ll +never hurt to actually have all the data installed. + + +When libvirt hangs +------------------ + +When a process hangs, we usually ask for a backtrace. To avoid problems w= ith +paging and so on, it is usually very helpful to just get a backtrace for o= ne +instance of the particular process. For that you can use something like t= his:: + + # gdb -batch -p $(pidof libvirtd) -ex 't a a bt f' + +This command will attach to currently running libvirtd process and run ``t= a a +bt f``, which is short for ``thread apply all backtrace full``, feel free = to +combine with ``sudo`` for users. If you are using this for virsh, or any = other +binary which might have multiple processes running, then make sure you sup= ply +the right pid for the ``-p`` option. For more info, read below about how = to +automate gdb. + + +When libvirt crashes +-------------------- + +Different distros have different mechanisms of catching and reporting cras= hes. +The automated ones are usually enabled only for the packaged binaries, but= that +should be enough for users. Developers will have their own way of doing t= hings +anyway. + +* **systemd-coredump** -- ``coredumpctl show`` shows all needed information + (even a backtrace) of the last crash, use ``coredumpctl ls`` to list all + crashes cordumpctl knows about. + +* **abrt** -- ``abrt-cli`` works similarly to the above (TBD: how to get t= he + backtrace using abrt-cli) + +* **setup your own** -- you can do one of these things: + + * set the ulimit for the service (depends on your init system) and look= for + the file that gets created + * set kernel.core_pattern using sysctl to a command (rather than a file= name + template) that gets ran with each core dump. This one does not need = any + ulimit setting, but you need to know what to specify there. + +For more information see related documentation. + + +Automating gdb +-------------- + +When you need more specific behaviour from gdb, you can automate that, but= for +multiline commands you need an input redirection or execute them from the = file. + + +Multiline example +----------------- + +Simple example that will print backtrace when ``abort()`` is reached:: + + $ cat >/var/lib/libvirt/gdbabortscript </etc/systemd/system/libvirtd.service.d/override.conf <