From nobody Tue Feb 10 14:49:46 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1770648595; cv=none; d=zohomail.com; s=zohoarc; b=MPNYquNCPQzsM4H5Nf5344i6Hiyu9dkDeGBa0k040v+tyHlTIUeU5X7oXJk8IMo4z1FBhwNkrvDyHUlqUK4VdTl4M8wNDXfuWeTcfUZwmnw/7NB7SymWn/7St3pDf9WkfqsWMM3aRnq3Z30jtnAQXX0Kn0iJvQXKV14UmdXSIS8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1770648595; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=7GPYTbIijqvYuqXnn3XuXd7bbQuY3nUYJe0SRBL25F8=; b=eIPKNkixGDJ33xbuuHTYcea9qYnS95Sr2V0kJN6S4e/d5qWBlyH7HG9vvCAGuXsRbY/EUw1F+1/FcdVW6P/693WeivaKSRJRSfYV4Y+SiT/ibVVz4VYS9bd6Z06xnmtWDj1rPqxToHHuNgSL5JcNMrp5Qx1WRiNKmXnKtpRVM7Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1770648595134746.8226913722136; Mon, 9 Feb 2026 06:49:55 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vpSWs-0007GG-CM; Mon, 09 Feb 2026 09:46:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vpSW0-00072g-4A for qemu-devel@nongnu.org; Mon, 09 Feb 2026 09:45:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vpSVx-0005jL-He for qemu-devel@nongnu.org; Mon, 09 Feb 2026 09:45:27 -0500 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-171-2bkRSxdBNoaUKsjwYWSt4g-1; Mon, 09 Feb 2026 09:45:21 -0500 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 10DA719560B3; Mon, 9 Feb 2026 14:45:19 +0000 (UTC) Received: from localhost (unknown [10.2.16.94]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EBAF418004BB; Mon, 9 Feb 2026 14:45:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1770648324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7GPYTbIijqvYuqXnn3XuXd7bbQuY3nUYJe0SRBL25F8=; b=eZLrLLt/SD33zNZOnNrCqmzRheetZ043WzsqFrDw6R3ku80ZWU4oY2/HL8CZ2cFhiRNJlr L9dkGx5PziPmbUohjuUty4qImR+Kh6PavMkXlwht73vLrKkF6M+RBgVcEQgTCcCy1xOpoH FMiREJ5kX/cVSewpv/PZHhDBFP1FEeY= X-MC-Unique: 2bkRSxdBNoaUKsjwYWSt4g-1 X-Mimecast-MFC-AGG-ID: 2bkRSxdBNoaUKsjwYWSt4g_1770648319 From: Stefan Hajnoczi To: qemu-devel@nongnu.org Cc: Eduardo Habkost , Marcel Apfelbaum , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Fam Zheng , Paolo Bonzini , Pierrick Bouvier , Yanan Wang , qemu-block@nongnu.org, Zhao Liu , Peter Maydell , pkrempa@redhat.com, Andrey Drobyshev , Vladimir Sementsov-Ogievskiy , Peter Xu , Vladimir Sementsov-Ogievskiy , Stefan Hajnoczi Subject: [PULL 4/9] scripts/qemugdb: coroutine: Add option for obtaining detailed trace in coredump Date: Mon, 9 Feb 2026 09:45:02 -0500 Message-ID: <20260209144508.45268-5-stefanha@redhat.com> In-Reply-To: <20260209144508.45268-1-stefanha@redhat.com> References: <20260209144508.45268-1-stefanha@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1770648596876158500 Content-Type: text/plain; charset="utf-8" From: Andrey Drobyshev Commit 772f86839f ("scripts/qemu-gdb: Support coroutine dumps in coredumps") introduced coroutine traces in coredumps using raw stack unwinding. While this works, this approach does not allow to view the function arguments in the corresponding stack frames. As an alternative, we can obtain saved registers from the coroutine's jmpbuf, patch them into the coredump's struct elf_prstatus in place, and execute another gdb subprocess to get backtrace from the patched temporary coredump. While providing more detailed info, this alternative approach, however, is more invasive as it might potentially corrupt the coredump file. We do take precautions by saving the original registers values into a separate binary blob /path/to/coredump.ptregs, so that it can be restores in the next GDB session. Still, instead of making it a new deault, let's keep raw unwi= nd the default behaviour, but add the '--detailed' option for 'qemu bt' and 'qemu coroutine' command which would enforce the new behaviour. That's how this looks: (gdb) qemu coroutine 0x7fda9335a508 #0 0x5602bdb41c26 in qemu_coroutine_switch<+214> () at ../util/coroutine= -ucontext.c:321 #1 0x5602bdb3e8fe in qemu_aio_coroutine_enter<+493> () at ../util/qemu-c= oroutine.c:293 #2 0x5602bdb3c4eb in co_schedule_bh_cb<+538> () at ../util/async.c:547 #3 0x5602bdb3b518 in aio_bh_call<+119> () at ../util/async.c:172 #4 0x5602bdb3b79a in aio_bh_poll<+457> () at ../util/async.c:219 #5 0x5602bdb10f22 in aio_poll<+1201> () at ../util/aio-posix.c:719 #6 0x5602bd8fb1ac in iothread_run<+123> () at ../iothread.c:63 #7 0x5602bdb18a24 in qemu_thread_start<+355> () at ../util/qemu-thread-p= osix.c:393 (gdb) qemu coroutine 0x7fda9335a508 --detailed patching core file /tmp/tmpq4hmk2qc found "CORE" at 0x10c48 assume pt_regs at 0x10cbc write r15 at 0x10cbc write r14 at 0x10cc4 write r13 at 0x10ccc write r12 at 0x10cd4 write rbp at 0x10cdc write rbx at 0x10ce4 write rip at 0x10d3c write rsp at 0x10d54 #0 0x00005602bdb41c26 in qemu_coroutine_switch (from_=3D0x7fda9335a508, = to_=3D0x7fda8400c280, action=3DCOROUTINE_ENTER) at ../util/coroutine-uconte= xt.c:321 #1 0x00005602bdb3e8fe in qemu_aio_coroutine_enter (ctx=3D0x5602bf7147c0,= co=3D0x7fda8400c280) at ../util/qemu-coroutine.c:293 #2 0x00005602bdb3c4eb in co_schedule_bh_cb (opaque=3D0x5602bf7147c0) at = ../util/async.c:547 #3 0x00005602bdb3b518 in aio_bh_call (bh=3D0x5602bf714a40) at ../util/as= ync.c:172 #4 0x00005602bdb3b79a in aio_bh_poll (ctx=3D0x5602bf7147c0) at ../util/a= sync.c:219 #5 0x00005602bdb10f22 in aio_poll (ctx=3D0x5602bf7147c0, blocking=3Dtrue= ) at ../util/aio-posix.c:719 #6 0x00005602bd8fb1ac in iothread_run (opaque=3D0x5602bf42b100) at ../io= thread.c:63 #7 0x00005602bdb18a24 in qemu_thread_start (args=3D0x5602bf7164a0) at ..= /util/qemu-thread-posix.c:393 #8 0x00007fda9e89f7f2 in start_thread (arg=3D) at pthread= _create.c:443 #9 0x00007fda9e83f450 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/= clone3.S:81 CC: Vladimir Sementsov-Ogievskiy CC: Peter Xu Originally-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Drobyshev Reviewed-by: Stefan Hajnoczi Message-id: 20251204105019.455060-5-andrey.drobyshev@virtuozzo.com Signed-off-by: Stefan Hajnoczi --- scripts/qemugdb/coroutine.py | 257 +++++++++++++++++++++++++++++++++-- 1 file changed, 242 insertions(+), 15 deletions(-) diff --git a/scripts/qemugdb/coroutine.py b/scripts/qemugdb/coroutine.py index e98fc48a4b..29f57ae84e 100644 --- a/scripts/qemugdb/coroutine.py +++ b/scripts/qemugdb/coroutine.py @@ -9,10 +9,119 @@ # This work is licensed under the terms of the GNU GPL, version 2 # or later. See the COPYING file in the top-level directory. =20 +import atexit import gdb +import os +import pty +import re +import struct +import textwrap + +from collections import OrderedDict +from copy import deepcopy =20 VOID_PTR =3D gdb.lookup_type('void').pointer() =20 +# Registers in the same order they're present in ELF coredump file. +# See asm/ptrace.h +PT_REGS =3D ['r15', 'r14', 'r13', 'r12', 'rbp', 'rbx', 'r11', 'r10', 'r9', + 'r8', 'rax', 'rcx', 'rdx', 'rsi', 'rdi', 'orig_rax', 'rip', 'cs= ', + 'eflags', 'rsp', 'ss'] + +coredump =3D None + + +class Coredump: + _ptregs_suff =3D '.ptregs' + + def __init__(self, coredump, executable): + atexit.register(self._cleanup) + + self.coredump =3D coredump + self.executable =3D executable + self._ptregs_blob =3D coredump + self._ptregs_suff + self._dirty =3D False + + with open(coredump, 'rb') as f: + while f.read(4) !=3D b'CORE': + pass + gdb.write(f'core file {coredump}: found "CORE" at 0x{f.tell():= x}\n') + + # Looking for struct elf_prstatus and pr_reg field in it (an a= rray + # of general purpose registers). See sys/procfs.h. + + # lseek(f.fileno(), 4, SEEK_CUR): go to elf_prstatus + f.seek(4, 1) + + # lseek(f.fileno(), 112, SEEK_CUR): + # offsetof(struct elf_prstatus, pr_reg) + f.seek(112, 1) + + self._ptregs_offset =3D f.tell() + + # If binary blob with the name /path/to/coredump + '.ptregs' + # exists, that means proper cleanup didn't happen during previ= ous + # GDB session with the same coredump, and registers in the dump + # itself might've remained patched. Thus we restore original + # registers values from this blob + if os.path.exists(self._ptregs_blob): + with open(self._ptregs_blob, 'rb') as b: + orig_ptregs_bytes =3D b.read() + self._dirty =3D True + else: + orig_ptregs_bytes =3D f.read(len(PT_REGS) * 8) + + values =3D struct.unpack(f"=3D{len(PT_REGS)}q", orig_ptregs_by= tes) + self._orig_ptregs =3D OrderedDict(zip(PT_REGS, values)) + + if not os.path.exists(self._ptregs_blob): + gdb.write(f'saving original pt_regs in {self._ptregs_blob}= \n') + with open(self._ptregs_blob, 'wb') as b: + b.write(orig_ptregs_bytes) + + gdb.write('\n') + + def patch_regs(self, regs): + # Set dirty flag early on to make sure regs are restored upon clea= nup + self._dirty =3D True + + gdb.write(f'patching core file {self.coredump}\n') + patched_ptregs =3D deepcopy(self._orig_ptregs) + int_regs =3D {k: int(v) for k, v in regs.items()} + patched_ptregs.update(int_regs) + + with open(self.coredump, 'ab') as f: + gdb.write(f'assume pt_regs at 0x{self._ptregs_offset:x}\n') + f.seek(self._ptregs_offset, 0) + gdb.write('writing regs:\n') + for reg in self._orig_ptregs.keys(): + if reg in int_regs: + gdb.write(f" {reg}: {int_regs[reg]:#16x}\n") + f.write(struct.pack(f"=3D{len(PT_REGS)}q", *patched_ptregs.val= ues())) + + gdb.write('\n') + + def restore_regs(self): + if not self._dirty: + return + + gdb.write(f'\nrestoring original regs in core file {self.coredump}= \n') + with open(self.coredump, 'ab') as f: + gdb.write(f'assume pt_regs at 0x{self._ptregs_offset:x}\n') + f.seek(self._ptregs_offset, 0) + f.write(struct.pack(f"=3D{len(PT_REGS)}q", + *self._orig_ptregs.values())) + + self._dirty =3D False + gdb.write('\n') + + def _cleanup(self): + if os.path.exists(self._ptregs_blob): + self.restore_regs() + gdb.write(f'\nremoving saved pt_regs file {self._ptregs_blob}\= n') + os.unlink(self._ptregs_blob) + + def pthread_self(): '''Fetch the base address of TLS.''' return gdb.parse_and_eval("$fs_base") @@ -77,6 +186,55 @@ def symbol_lookup(addr): =20 return f"{func_str} at {path}:{line}" =20 +def run_with_pty(cmd): + # Create a PTY pair + master_fd, slave_fd =3D pty.openpty() + + pid =3D os.fork() + if pid =3D=3D 0: # Child + os.close(master_fd) + # Attach stdin/stdout/stderr to the PTY slave side + os.dup2(slave_fd, 0) + os.dup2(slave_fd, 1) + os.dup2(slave_fd, 2) + os.close(slave_fd) + os.execvp("gdb", cmd) # Runs gdb and doesn't return + + # Parent + os.close(slave_fd) + + output =3D bytearray() + try: + while True: + data =3D os.read(master_fd, 65536) + if not data: + break + output.extend(data) + except OSError: # in case subprocess exits and we get EBADF on read() + pass + finally: + try: + os.close(master_fd) + except OSError: # in case we get EBADF on close() + pass + + # Wait for child to finish (reap zombie) + os.waitpid(pid, 0) + + return output.decode('utf-8') + +def dump_backtrace_patched(regs): + cmd =3D ['gdb', '-batch', + '-ex', 'set debuginfod enabled off', + '-ex', 'set complaints 0', + '-ex', 'set style enabled on', + '-ex', 'python print("----split----")', + '-ex', 'bt', coredump.executable, coredump.coredump] + + coredump.patch_regs(regs) + out =3D run_with_pty(cmd).split('----split----')[1] + gdb.write(out) + def dump_backtrace(regs): ''' Backtrace dump with raw registers, mimic GDB command 'bt'. @@ -120,7 +278,7 @@ def dump_backtrace_live(regs): =20 selected_frame.select() =20 -def bt_jmpbuf(jmpbuf): +def bt_jmpbuf(jmpbuf, detailed=3DFalse): '''Backtrace a jmpbuf''' regs =3D get_jmpbuf_regs(jmpbuf) try: @@ -128,8 +286,12 @@ def bt_jmpbuf(jmpbuf): # but only works with live sessions. dump_backtrace_live(regs) except: - # If above doesn't work, fallback to poor man's unwind - dump_backtrace(regs) + if detailed: + # Obtain detailed trace by patching regs in copied coredump + dump_backtrace_patched(regs) + else: + # If above doesn't work, fallback to poor man's unwind + dump_backtrace(regs) =20 def co_cast(co): return co.cast(gdb.lookup_type('CoroutineUContext').pointer()) @@ -138,28 +300,90 @@ def coroutine_to_jmpbuf(co): coroutine_pointer =3D co_cast(co) return coroutine_pointer['env']['__jmpbuf'] =20 +def init_coredump(): + global coredump + + files =3D gdb.execute('info files', False, True).split('\n') + + if not 'core dump' in files[1]: + return False + + core_path =3D re.search("`(.*)'", files[2]).group(1) + exec_path =3D re.match('^Symbols from "(.*)".$', files[0]).group(1) + + if coredump is None: + coredump =3D Coredump(core_path, exec_path) + + return True =20 class CoroutineCommand(gdb.Command): - '''Display coroutine backtrace''' + __doc__ =3D textwrap.dedent("""\ + Display coroutine backtrace + + Usage: qemu coroutine COROPTR [--detailed] + Show backtrace for a coroutine specified by COROPTR + + --detailed obtain detailed trace by copying coredump, patc= hing + regs in it, and runing gdb subprocess to get + backtrace from the patched coredump + """) + def __init__(self): gdb.Command.__init__(self, 'qemu coroutine', gdb.COMMAND_DATA, gdb.COMPLETE_NONE) =20 + def _usage(self): + gdb.write('usage: qemu coroutine [--detailed]\= n') + return + def invoke(self, arg, from_tty): argv =3D gdb.string_to_argv(arg) - if len(argv) !=3D 1: - gdb.write('usage: qemu coroutine \n') + argc =3D len(argv) + if argc =3D=3D 0 or argc > 2 or (argc =3D=3D 2 and argv[1] !=3D '-= -detailed'): + return self._usage() + detailed =3D True if argc =3D=3D 2 else False + + is_coredump =3D init_coredump() + if detailed and not is_coredump: + gdb.write('--detailed is only valid when debugging core dumps\= n') return =20 - bt_jmpbuf(coroutine_to_jmpbuf(gdb.parse_and_eval(argv[0]))) + try: + bt_jmpbuf(coroutine_to_jmpbuf(gdb.parse_and_eval(argv[0])), + detailed=3Ddetailed) + finally: + coredump.restore_regs() =20 class CoroutineBt(gdb.Command): - '''Display backtrace including coroutine switches''' + __doc__ =3D textwrap.dedent("""\ + Display backtrace including coroutine switches + + Usage: qemu bt [--detailed] + + --detailed obtain detailed trace by copying coredump, patc= hing + regs in it, and runing gdb subprocess to get + backtrace from the patched coredump + """) + def __init__(self): gdb.Command.__init__(self, 'qemu bt', gdb.COMMAND_STACK, gdb.COMPLETE_NONE) =20 + def _usage(self): + gdb.write('usage: qemu bt [--detailed]\n') + return + def invoke(self, arg, from_tty): + argv =3D gdb.string_to_argv(arg) + argc =3D len(argv) + if argc > 1 or (argc =3D=3D 1 and argv[0] !=3D '--detailed'): + return self._usage() + detailed =3D True if argc =3D=3D 1 else False + + is_coredump =3D init_coredump() + if detailed and not is_coredump: + gdb.write('--detailed is only valid when debugging core dumps\= n') + return =20 gdb.execute("bt") =20 @@ -173,13 +397,16 @@ def invoke(self, arg, from_tty): if co_ptr =3D=3D False: return =20 - while True: - co =3D co_cast(co_ptr) - co_ptr =3D co["base"]["caller"] - if co_ptr =3D=3D 0: - break - gdb.write("Coroutine at " + str(co_ptr) + ":\n") - bt_jmpbuf(coroutine_to_jmpbuf(co_ptr)) + try: + while True: + co =3D co_cast(co_ptr) + co_ptr =3D co["base"]["caller"] + if co_ptr =3D=3D 0: + break + gdb.write("\nCoroutine at " + str(co_ptr) + ":\n") + bt_jmpbuf(coroutine_to_jmpbuf(co_ptr), detailed=3Ddetailed) + finally: + coredump.restore_regs() =20 class CoroutineSPFunction(gdb.Function): def __init__(self): --=20 2.52.0