From nobody Sun Sep 28 16:36:38 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1758534312; cv=none; d=zohomail.com; s=zohoarc; b=hHZtSf5R41z0lQp/uwNFO8NdSWz/4UCmj7deBF/5ybpf2HaPf5LOY4Ny07AbJaNeku2a43HLACUL9x7w2rYpLw0vQZ4+mkhi6zWdpO5v5xg27+uaWAXw5ULR/CHsM1cUTd5x/ObFtYZROsTp8/w7pDTf0HEmS/e32HEIeO1+ihI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1758534312; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=30/Sw7/xAd9JXNBC4JguewcrlL8rhREEZz71xctzWtw=; b=Mxo+RaSNrd/UXnNCqjEwQx6Psh/bEc4skLOEZICJvKfFOrDC+zze4I3fm0EIBEt5CrwyFztuZTNc29r4kXyNtiml2m3moHqvkBcENieu4lUkwZYkqjWZdJBcc3LZnLybBDojxGhs9IX2O1rJ2u48WIcH0Yxsz+383ZzWYCvbAOc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1758534312526337.42914488421536; Mon, 22 Sep 2025 02:45:12 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1v0d5z-0006DQ-SO; Mon, 22 Sep 2025 05:44:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v0d5A-0005Sa-8S for qemu-devel@nongnu.org; Mon, 22 Sep 2025 05:43:43 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1v0d4p-000219-Kw for qemu-devel@nongnu.org; Mon, 22 Sep 2025 05:43:38 -0400 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-46b7bf21fceso13604645e9.3 for ; Mon, 22 Sep 2025 02:43:19 -0700 (PDT) Received: from draig.lan ([185.126.160.19]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3ee07407d33sm19370945f8f.18.2025.09.22.02.43.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Sep 2025 02:43:14 -0700 (PDT) Received: from draig.lan (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id EBA335FA6C; Mon, 22 Sep 2025 10:37:13 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1758534197; x=1759138997; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=30/Sw7/xAd9JXNBC4JguewcrlL8rhREEZz71xctzWtw=; b=K8PYXxe96pRYX8Fcz/B4YzSUpoJtA+WLySxAmEP6sP1DPYIhhGCfqrK3/pJyh3+GuF awKaGNzdBaoCk9RS9IqjsjNCY0SfS3M7b1j4Fs6BKmq6ob+XxagwWL3TPlRMTjOID9WU MvlMOf7PwPGTb6vnqnx3KCg1b0KwFJmj0TUS/Ot2aZVgXH37fDSyAFbPYC6XFC4o6Qz6 WovH1MO/o6qK9C1apJiLpkfRJBvg9bWNQGsIVWm6G/rBpDvnf4qMb1ygfZ7HVKKXf5uf lU+M8lviXif+z5npSVH2rLHVKFY9P4J9UmOv5HQcxIZI8x/TMl0cJTE2KkEqZJjvsnuN w4iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758534197; x=1759138997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=30/Sw7/xAd9JXNBC4JguewcrlL8rhREEZz71xctzWtw=; b=wdiV5cS2mZoslGMs+YmALwkhmegV9R9PDEXHgo3IFpqX12UJ/ctiLrs1XmZvy5iJhg yL4tGwYKgo2dKoOGLV6seFCR9ff0qelq9fww3XyUzEVeMOJX2RLiLIcSIWq9ZBuiXORF oZQfTIt9TdJ6A6vGIeg2tKoJXED9fb05Gek0BmdjlsxeFsaiVWlOsFVlo72RW9kPCQde cxBjf1NxV4DKpnCPsm7iynmf7jpK0ZWGZXp/tzZXOFCaHrzn1ZcmU+p+lg2VjXbfQpwC M67P3iIUseWOJRpm/J89pwEcRNJxpV09yTGYH/UNHpF2u4SqqWt7nQ4i7Jk3yGNjVRN3 vVJQ== X-Gm-Message-State: AOJu0YxNltLxSvylqeolS781944tapozEEBt9EW7PHlzkHPIv82P5eC5 swyVXwhzLuGDMSLEhg9czZeQusS9PiYpL3gzu5hNfLy454/HtWKDFawk3zeK3JPsTIw= X-Gm-Gg: ASbGnct8kPhlF0hdi3KtcH3nwISJkcZIY2+CGu8eYjiYL+kckKOTBR4Dh0wimktr4fr kfCilNGwzxw5++OHJFc5TdRa6VfZX9W1Wg6OjuD8zX9O2j9NrIbEplxFpcIZvparMsCDAPOWxHx jpq4yli5rPC5SPosr6g6K3hrBXmRh9u5utk9UEhLiLk1WN+DhLzpOjm+yIXrEuGO8UpqZNZomiL D7FEgC8VZwxbO4FWgZmEjrg9VoJlDhU1D4rD3URCq1i5Zhksyovqd3zcZKgMc7dPitwEjCusc2l aAiLEKhoLpTx/AZdcyylIbTT79VAcMm6zZ7B5cUcKr6ooM2EhoDhGX9feQ3eUUe6+s1ZumoeVyk ZO1DHqb+LbxbLYkSsRc7eitQ= X-Google-Smtp-Source: AGHT+IEr4H07owG+xBQBwEkJEDkgU6g+J+fTCesMM1MjRJs0OT/DRbC4Ba+MvUjyq+UGC2HTtIP2Tg== X-Received: by 2002:a05:600c:1c03:b0:46d:fe0b:d55a with SMTP id 5b1f17b1804b1-46dfe0bd756mr11149635e9.33.1758534197144; Mon, 22 Sep 2025 02:43:17 -0700 (PDT) From: =?UTF-8?q?Alex=20Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: Palmer Dabbelt , Alexandre Iooss , Mahmoud Mandour , qemu-riscv@nongnu.org, Daniel Henrique Barboza , Thomas Huth , Peter Maydell , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Alistair Francis , qemu-arm@nongnu.org, =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Weiwei Li , Pierrick Bouvier , Liu Zhiwei , Manos Pitsidianakis Subject: [PATCH 25/25] contrib/plugins/uftrace: add documentation Date: Mon, 22 Sep 2025 10:37:10 +0100 Message-ID: <20250922093711.2768983-26-alex.bennee@linaro.org> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20250922093711.2768983-1-alex.bennee@linaro.org> References: <20250922093711.2768983-1-alex.bennee@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x333.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1758534314013116600 From: Pierrick Bouvier This documentation summarizes how to use the plugin, and present two examples of the possibilities offered by it, in system and user mode. As well, it explains how to rebuild and reproduce those examples. Reviewed-by: Manos Pitsidianakis Signed-off-by: Pierrick Bouvier Message-ID: <20250902075042.223990-10-pierrick.bouvier@linaro.org> Signed-off-by: Alex Benn=C3=A9e --- docs/about/emulation.rst | 199 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst index 456d01d5b08..8a5e128f677 100644 --- a/docs/about/emulation.rst +++ b/docs/about/emulation.rst @@ -816,6 +816,205 @@ This plugin can limit the number of Instructions Per = Second that are executed:: The lower the number the more accurate time will be, but the less ef= ficient the plugin. Defaults to ips/10 =20 +Uftrace +....... + +``contrib/plugins/uftrace.c`` + +This plugin generates a binary trace compatible with +`uftrace `_. + +Plugin supports aarch64 and x64, and works in user and system mode, allowi= ng to +trace a system boot, which is not something possible usually. + +In user mode, the memory mapping is directly copied from ``/proc/self/maps= `` at +the end of execution. Uftrace should be able to retrieve symbols by itself, +without any additional step. +In system mode, the default memory mapping is empty, and you can generate +one (and associated symbols) using ``contrib/plugins/uftrace_symbols.py``. +Symbols must be present in ELF binaries. + +It tracks the call stack (based on frame pointer analysis). Thus, your pro= gram +and its dependencies must be compiled using ``-fno-omit-frame-pointer +-mno-omit-leaf-frame-pointer``. In 2024, `Ubuntu and Fedora enabled it by +default again on x64 +`_. +On aarch64, this is less of a problem, as they are usually part of the ABI, +except for leaf functions. That's true for user space applications, but not +necessarily for bare metal code. You can read this `section +` to easily build a system with frame pointe= rs. + +When tracing long scenarios (> 1 min), the generated trace can become very= long, +making it hard to extract data from it. In this case, a simple solution is= to +trace execution while generating a timestamped output log using +``qemu-system-aarch64 ... | ts "%s"``. Then, ``uftrace --time-range=3Dstar= t~end`` +can be used to reduce trace for only this part of execution. + +Performance wise, overhead compared to normal tcg execution is around x5-x= 15. + +.. list-table:: Uftrace plugin arguments + :widths: 20 80 + :header-rows: 1 + + * - Option + - Description + * - trace-privilege-level=3D[on|off] + - Generate separate traces for each privilege level (Exception Level + + Security State on aarch64, Rings on x64). + +.. list-table:: uftrace_symbols.py arguments + :widths: 20 80 + :header-rows: 1 + + * - Option + - Description + * - elf_file [elf_file ...] + - path to an ELF file. Use /path/to/file:0xdeadbeef to add a mapping o= ffset. + * - --prefix-symbols + - prepend binary name to symbols + +Example user trace +++++++++++++++++++ + +As an example, we can trace qemu itself running git:: + + $ ./build/qemu-aarch64 -plugin \ + build/contrib/plugins/libuftrace.so \ + ./build/qemu-aarch64 /usr/bin/git --help + + # and generate a chrome trace directly + $ uftrace dump --chrome | gzip > ~/qemu_aarch64_git_help.json.gz + +For convenience, you can download this trace `qemu_aarch64_git_help.json.gz +`_. +Download it and open this trace on https://ui.perfetto.dev/. You can zoom = in/out +using :kbd:`W`, :kbd:`A`, :kbd:`S`, :kbd:`D` keys. +Some sequences taken from this trace: + +- Loading program and its interpreter + +.. image:: https://fileserver.linaro.org/s/fie8JgX76yyL5cq/preview + :height: 200px + +- open syscall + +.. image:: https://fileserver.linaro.org/s/rsXPTeZZPza4PcE/preview + :height: 200px + +- TB creation + +.. image:: https://fileserver.linaro.org/s/GXY6NKMw5EeRCew/preview + :height: 200px + +It's usually better to use ``uftrace record`` directly. However, tracing +binaries through qemu-user can be convenient when you don't want to recomp= ile +them (``uftrace record`` requires instrumentation), as long as symbols are +present. + +Example system trace +++++++++++++++++++++ + +A full trace example (chrome trace, from instructions below) generated fro= m a +system boot can be found `here +`_. +Download it and open this trace on https://ui.perfetto.dev/. You can see c= ode +executed for all privilege levels, and zoom in/out using +:kbd:`W`, :kbd:`A`, :kbd:`S`, :kbd:`D` keys. You can find below some seque= nces +taken from this trace: + +- Two first stages of boot sequence in Arm Trusted Firmware (EL3 and S-EL1) + +.. image:: https://fileserver.linaro.org/s/kkxBS552W7nYESX/preview + :height: 200px + +- U-boot initialization (until code relocation, after which we can't track= it) + +.. image:: https://fileserver.linaro.org/s/LKTgsXNZFi5GFNC/preview + :height: 200px + +- Stat and open syscalls in kernel + +.. image:: https://fileserver.linaro.org/s/dXe4MfraKg2F476/preview + :height: 200px + +- Timer interrupt + +.. image:: https://fileserver.linaro.org/s/TM5yobYzJtP7P3C/preview + :height: 200px + +- Poweroff sequence (from kernel back to firmware, NS-EL2 to EL3) + +.. image:: https://fileserver.linaro.org/s/oR2PtyGKJrqnfRf/preview + :height: 200px + +Build and run system example +++++++++++++++++++++++++++++ + +.. _uftrace_build_system_example: + +Building a full system image with frame pointers is not trivial. + +We provide a `simple way `= _ to +build an aarch64 system, combining Arm Trusted firmware, U-boot, Linux ker= nel +and debian userland. It's based on containers (``podman`` only) and +``qemu-user-static (binfmt)`` to make sure it's easily reproducible and do= es not depend +on machine where you build it. + +You can follow the exact same instructions for a x64 system, combining edk= 2, +Linux, and Ubuntu, simply by switching to +`x86_64 `_ bra= nch. + +To build the system:: + + # Install dependencies + $ sudo apt install -y podman qemu-user-static + + $ git clone https://github.com/pbo-linaro/qemu-linux-stack + $ cd qemu-linux-stack + $ ./build.sh + + # system can be started using: + $ ./run.sh /path/to/qemu-system-aarch64 + +To generate a uftrace for a system boot from that:: + + # run true and poweroff the system + $ env INIT=3Dtrue ./run.sh path/to/qemu-system-aarch64 \ + -plugin path/to/contrib/plugins/libuftrace.so,trace-privilege-level= =3Don + + # generate symbols and memory mapping + $ path/to/contrib/plugins/uftrace_symbols.py \ + --prefix-symbols \ + arm-trusted-firmware/build/qemu/debug/bl1/bl1.elf \ + arm-trusted-firmware/build/qemu/debug/bl2/bl2.elf \ + arm-trusted-firmware/build/qemu/debug/bl31/bl31.elf \ + u-boot/u-boot:0x60000000 \ + linux/vmlinux + + # inspect trace with + $ uftrace replay + +Uftrace allows to filter the trace, and dump flamegraphs, or a chrome trac= e. +This last one is very interesting to see visually the boot process:: + + $ uftrace dump --chrome > boot.json + # Open your browser, and load boot.json on https://ui.perfetto.dev/. + +Long visual chrome traces can't be easily opened, thus, it might be +interesting to generate them around a particular point of execution:: + + # execute qemu and timestamp output log + $ env INIT=3Dtrue ./run.sh path/to/qemu-system-aarch64 \ + -plugin path/to/contrib/plugins/libuftrace.so,trace-privilege-level= =3Don |& + ts "%s" | tee exec.log + + $ cat exec.log | grep 'Run /init' + 1753122320 [ 11.834391] Run /init as init process + # init was launched at 1753122320 + + # generate trace around init execution (2 seconds): + $ uftrace dump --chrome --time-range=3D1753122320~1753122322 > init.js= on + Other emulation features ------------------------ =20 --=20 2.47.3