From nobody Sun Nov 24 04:43:53 2024 Received: from sender4-pp-f112.zoho.com (sender4-pp-f112.zoho.com [136.143.188.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1188E17BB0D; Thu, 7 Nov 2024 14:00:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=136.143.188.112 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730988057; cv=pass; b=ijSH4PYVArz0LuBdiYhszenuSM+4L41Ir7Nt5Xc+A/z7iMMuO6DN7VQ134OL4TI3oRdCsZh/xLZvKn62jXLpiHQe3BEAX/HiyYlvuiI+fQM2Y5Qky46gtz8osqwg9JXXbWO6tv/s4EX5IbZAMA1f4yKFzZ3q7gMaKDvlMrFfghk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730988057; c=relaxed/simple; bh=dPUgn2OA+01/YHhSGiAuDl9wzX61SpjRLz6r3hhup0M=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=IY3Ly/uvEhyNVmD6ynYJQuKVmpjS78rXk0fiw07gKiNZCPDV49Ej90rL+m6RLraZixhJ9pJ4ZwwY7Cnfc48Bg/+4hP3wgVsP6Rbsh2SLx7CPiWFf3ROFb4Vgu46ERRlmFSLW+iKTBxEa2KgUeCFQxtVxl/yRI4+5AU8kEKx3a48= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (1024-bit key) header.d=collabora.com header.i=sebastian.fricke@collabora.com header.b=XTWCRvhh; arc=pass smtp.client-ip=136.143.188.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=collabora.com header.i=sebastian.fricke@collabora.com header.b="XTWCRvhh" ARC-Seal: i=1; a=rsa-sha256; t=1730988037; cv=none; d=zohomail.com; s=zohoarc; b=HX3eKeh0XcbFTzhmXCaXSNfXzCRNVv/qmGTSLohQYgM7Vrqv/IKWhPJwN8XSCQJYLViLWdCXjsYzDKXso7Fr8DlADaji2Y5Q1n23RhUkIcvQktBTevbHbUUeHWVyKQqhDh8c5cJi7gceABwzTSEYPAc8sEO1Og/CljnkRYKvmmg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1730988037; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=nbtfIV1xa7iMd01yGQAcBD58sY6EzPQH0wYNkg8hMHQ=; b=G6ku5JjIIUb/m7WfK59di3fq1qe1a+4g+ANfiAsLfpnCXrv6BZeXE911rCtvNxbq+Q9iuLs35eqOV7GIb8K0T10DUy+jedegPS+TTXNDU+tmHTQItZZMYMmsjCNXNDlXC2VBQNGyHsbTCUCx7l4K9jbyG4RJO83fhgwD4CySlRo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=collabora.com; spf=pass smtp.mailfrom=sebastian.fricke@collabora.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1730988037; s=zohomail; d=collabora.com; i=sebastian.fricke@collabora.com; h=From:From:Date:Date:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Message-Id:References:In-Reply-To:To:To:Cc:Cc:Reply-To; bh=nbtfIV1xa7iMd01yGQAcBD58sY6EzPQH0wYNkg8hMHQ=; b=XTWCRvhhdMv7IEukq7vL4kUNDvJKOMDJ677VMoIgZvPOO0tosXnmbItdYToc8LsL vK2WVN2qq0AzAsA4eNbS14dV72Wia3atGpR+kV6PdmgWphACH0jOSfmDLMFv7QXaK96 xgCmwHoN+TykInZsdp+XkrR79BPE8wijFrIkm/YA= Received: by mx.zohomail.com with SMTPS id 1730988036034523.1380231511052; Thu, 7 Nov 2024 06:00:36 -0800 (PST) From: Sebastian Fricke Date: Thu, 07 Nov 2024 15:00:23 +0100 Subject: [PATCH 1/2] docs: Add guides section for debugging Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20241028-media_docs_improve_v3-v1-1-2b1b486c223e@collabora.com> References: <20241028-media_docs_improve_v3-v1-0-2b1b486c223e@collabora.com> In-Reply-To: <20241028-media_docs_improve_v3-v1-0-2b1b486c223e@collabora.com> To: Jonathan Corbet Cc: bagasdotme@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, laurent.pinchart@ideasonboard.com, hverkuil-cisco@xs4all.nl, mauro.chehab@linux.intel.com, kernel@collabora.com, bob.beckett@collabora.com, nicolas.dufresne@collabora.com, Sebastian Fricke X-Mailer: b4 0.11.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1730988026; l=23438; i=sebastian.fricke@collabora.com; s=linux-media; h=from:subject:message-id; bh=dPUgn2OA+01/YHhSGiAuDl9wzX61SpjRLz6r3hhup0M=; b=3deVEj/TrtNEQ69Oznwc5vbwOEk35GAu61kOTzHKa9ozJThnBVGPzp8RE6WKQCsySX0FPuO26olc ejA8z3/8BcZ7e39NP3iN5bzWEL69ozctAz6CgWip+/QKLhm05Jo1 X-Developer-Key: i=sebastian.fricke@collabora.com; a=ed25519; pk=pYXedPwrTtErcj7ERYeo/IpTrpe4QbJuEzSB52fslBg= X-ZohoMailClient: External This idea was formed after noticing that new developers experience certain difficulty to navigate within the multitude of different debugging options in the Kernel and while there often is good documentation for the tools, the developer has to know first that they exist and where to find them. Add a general debugging section to the Kernel documentation, as an easily locatable entry point to other documentation and as a general guideline for the topic. Signed-off-by: Sebastian Fricke --- Documentation/index.rst | 2 + .../driver_development_debugging_guide.rst | 206 +++++++++++++++ Documentation/process/debugging/general_advice.rst | 48 ++++ Documentation/process/debugging/index.rst | 21 ++ .../debugging/userspace_debugging_guide.rst | 278 +++++++++++++++++= ++++ 5 files changed, 555 insertions(+) diff --git a/Documentation/index.rst b/Documentation/index.rst index 36e61783437c..be19f0a79a6a 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -57,6 +57,7 @@ Various other manuals with useful information for all ker= nel developers. dev-tools/testing-overview kernel-hacking/index trace/index + process/debugging/index fault-injection/index livepatch/index rust/index @@ -76,6 +77,7 @@ developers seeking information on the kernel's user-space= APIs. The kernel build system admin-guide/reporting-issues.rst User-space tools + process/debugging/userspace_debugging_guide.rst userspace-api/index =20 See also: the `Linux man pages `_, diff --git a/Documentation/process/debugging/driver_development_debugging_g= uide.rst b/Documentation/process/debugging/driver_development_debugging_gui= de.rst new file mode 100644 index 000000000000..f3d9c2c192c4 --- /dev/null +++ b/Documentation/process/debugging/driver_development_debugging_guide.rst @@ -0,0 +1,206 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Debugging advice for driver development +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This document serves as a general starting point and lookup for debugging = device +drivers. +While this guide focuses on debugging that requires re-compiling the +module/kernel, the :doc:`userspace debugging guide +` will guide +you through tools like dynamic debug, ftrace and other tools useful for +debugging issues and behavior. +For general debugging advice, see the :doc:`general advice document +`. + +.. contents:: + :depth: 3 + +The following sections show you the available tools. + +Printk & friends +---------------- + +These are derivatives of printf() with varying destinations and support for +being dynamically turned on or off, or lack thereof. + +Simple printk +~~~~~~~~~~~~~ + +The classic, can be used to great effect for quick and dirty development +of new modules or to extract arbitrary necessary data for troubleshooting. + +Prerequisite: `CONFIG_PRINTK` (usually enabled by default) + +**Pros**: + +- No need to learn anything, simple to use +- Easy to modify exactly to your needs (formatting of the data (See: + :doc:`/core-api/printk-formats`), visibility in the log) +- Can cause delays in the execution of the code (beneficial to confirm whe= ther + timing is a factor) + +**Cons**: + +- Requires rebuilding the kernel/module +- Can cause delays in the execution of the code (which can cause issues to= be + not reproducible) + +For the full documentation see :doc:`/core-api/printk-basics` + +Trace_printk +~~~~~~~~~~~~ + +Prerequisite: `CONFIG_DYNAMIC_FTRACE` & `#include ` + +It is a tiny bit less comfortable to use than printk(), because you will h= ave +to read the messages from the trace file (See: :ref:`read_ftrace_log` +instead of from the kernel log, but very useful when printk() adds unwanted +delays into the code execution, causing issues to be flaky or hidden.) + +If the processing of this still causes timing issues then you can try +trace_puts(). + +For the full Documentation see trace_printk() + +dev_dbg +~~~~~~~ + +Print statement, which can be targeted by :ref:`process/debugging/userspac= e_debugging_guide:dynamic debug` +that contains additional information about the device used within the cont= ext. + +**When is it appropriate to leave a debug print in the code?** + +Permanent debug statements have to be useful for a developer to troublesho= ot +driver misbehavior. Judging that is a bit more of an art than a science, b= ut +some guidelines are in the :ref:`Coding style guidelines ` + +Custom printk +~~~~~~~~~~~~~ + +Example:: + + #define core_dbg(fmt, arg...) do { \ + if (core_debug) \ + printk(KERN_DEBUG pr_fmt("core: " fmt), ## arg); \ + } while (0) + +**When should you do this?** + +It is better to just use a pr_debug(), which can later be turned on/off wi= th +dynamic debug. Additionally, a lot of drivers activate these prints via a +variable like `core_debug` set by a module parameter. However, Module +parameters `are not recommended anymore +`_. + +Ftrace +------ + +Creating a custom Ftrace tracepoint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Here is a basic description of :ref:`how to implement new tracepoints +`. + +For the full event tracing documentation see :doc:`/trace/events` + +For the full Ftrace documentation see :doc:`/trace/ftrace` + +DebugFS +------- + +Prerequisite: `CONFIG_DEBUG_FS` & `#include ` + +DebugFS differs from the other approaches of debugging, as it doesn't write +messages to the kernel log nor add traces to the code. Instead it allows t= he +developer to handle a set of files. +With these files you can either store values of variables or make +register/memory dumps or you can make these files writable and modify +values/settings in the driver. + +Possible use-cases among others: + +- Store register values +- Keep track of variables +- Store errors +- Store settings +- Toggle a setting like debug on/off +- Error injection + +This is especially useful, when the size of a data dump would be hard to d= igest +as part of the general kernel log (for example when dumping raw bitstream = data) +or when you are not interested in all the values all the time, but with the +possibility to inspect them. + +The general idea is: + +- Create a directory during probe (`struct dentry *parent =3D + debugfs_create_dir("my_driver", NULL);`) +- Create a file (`debugfs_create_u32("my_value", 444, parent, &my_variable= );`) + + - In this example the file is found in `/sys/kernel/debug/my_driver/my_v= alue` + (with read permissions for user/group/all) + - any update of `my_variable` will update the value in the file + +- Clean up the folder when removing the device + (`debugfs_remove_recursive(parent);`) + +For the full documentation see :doc:`/filesystems/debugfs`. + +KASAN, UBSAN, lockdep and other error checkers +---------------------------------------------- + +KASAN (Kernel Address Sanitizer) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Prerequisite: `CONFIG_KASAN` + +KASAN is a dynamic memory error detector that helps to find use-after-free= and +out-of-bounds bugs. It uses compile-time instrumentation to check every me= mory +access. + +For the full documentation see :doc:`/dev-tools/kasan`. + +UBSAN (Undefined Behavior Sanitizer) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Prerequisite: `CONFIG_UBSAN` + +UBSAN relies on compiler instrumentation and runtime checks to detect unde= fined +behavior. It is designed to find a variety of issues, including signed int= eger +overflow, array index out of bounds, and more. + +For the full documentation see :doc:`/dev-tools/ubsan` + +lockdep (Lock Dependency Validator) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Prerequisite: `CONFIG_DEBUG_LOCKDEP` + +lockdep is a runtime lock dependency validator that detects potential dead= locks +and other locking-related issues in the kernel. +It tracks lock acquisitions and releases, building a dependency graph that= is +analyzed for potential deadlocks. +lockdep is especially useful for validating the correctness of lock orderi= ng in +the kernel. + +For the full documentation see :doc:`/RCU/lockdep` + +device coredump +--------------- + +Prerequisite: `#include ` + +Provides the infrastructure for a driver to provide arbitrary data to user= land. +It is most often used in conjunction with udev or similar userland applica= tion +to listen for kernel uevents, which indicate that the dump is ready. Udev = has +rules to copy that file somewhere for long-term storage and analysis, as by +default, the data for the dump is automatically cleaned up after 5 minutes. +That data is analyzed with driver-specific tools or GDB. + +You can find an example implementation at: +`drivers/media/platform/qcom/venus/core.c +`__ + +**Copyright** =C2=A92024 : Collabora diff --git a/Documentation/process/debugging/general_advice.rst b/Documenta= tion/process/debugging/general_advice.rst new file mode 100644 index 000000000000..631430656d53 --- /dev/null +++ b/Documentation/process/debugging/general_advice.rst @@ -0,0 +1,48 @@ +.. SPDX-License-Identifier: GPL-2.0 + +General debugging advice +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Depending on the issue, a different set of tools is available to track dow= n the +problem or even to realize whether there is one in the first place. + +As a first step you have to figure out what kind of issue you want to debu= g. +Depending on the answer, your methodology and choice of tools may vary. + +Do I need to debug with limited access? +--------------------------------------- + +Do you have limited access to the machine or are you unable to stop the ru= nning +execution? + +In this case your debugging capability depends on built-in debugging suppo= rt of +provided distro kernel. +The :doc:`/process/debugging/userspace_debugging_guide` provides a brief +overview over range of possible debugging tools in that situation. You can +check the capability of your kernel, in most cases, by looking into config= file +within the /boot folder. + +Do I have root access to the system? +------------------------------------ + +Are you easily able to replace the module in question or to install a new +kernel? + +In that case your range of available tools is a lot bigger, you can find t= he +tools in the :doc:`/process/debugging/driver_development_debugging_guide`. + +Is timing a factor? +------------------- + +It is important to understand if the problem you want to debug manifests i= tself +consistently (i.e. given a set of inputs you always get the same, incorrect +output), or inconsistently. If it manifests itself inconsistently, some ti= ming +factor might be at play. If inserting delays into the code does change the +behavior, then quite likely timing is a factor. + +When timing does alter the outcome of the code execution using a simple +printk() for debugging purposes won't work, a similar alternative is to use +trace_printk() , which logs the debug messages to the trace file instead o= f the +kernel log. + +**Copyright** =C2=A92024 : Collabora diff --git a/Documentation/process/debugging/index.rst b/Documentation/proc= ess/debugging/index.rst new file mode 100644 index 000000000000..c200ede7c955 --- /dev/null +++ b/Documentation/process/debugging/index.rst @@ -0,0 +1,21 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Debugging advice for Linux Kernel developers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. toctree:: + :maxdepth: 1 + + general_advice + driver_development_debugging_guide + userspace_debugging_guide + +.. only:: subproject and html + + Indices + =3D=3D=3D=3D=3D=3D=3D + + * :ref:`genindex` + +**Copyright** =C2=A92024 : Collabora diff --git a/Documentation/process/debugging/userspace_debugging_guide.rst = b/Documentation/process/debugging/userspace_debugging_guide.rst new file mode 100644 index 000000000000..0afe35c468a9 --- /dev/null +++ b/Documentation/process/debugging/userspace_debugging_guide.rst @@ -0,0 +1,278 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +Userspace debugging advice +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + +A brief overview of common tools to debug the Linux Kernel from userspace. +For debugging advice aimed at driver developer go :doc:`here +`. +For general debugging advice, see :doc:`general-debugging-guide +`. + +.. contents:: + :depth: 3 + +The following sections show you the available tools. + +Dynamic debug +------------- + +Mechanism to filter what ends up in the kernel log by dis-/en-abling log +messages. + +Prerequisite: `CONFIG_DYNAMIC_DEBUG` + +.. _valid_dyndbg_prints: + +Dynamic debug is only able to target: + +- pr_debug() +- dev_dbg() +- print_hex_dump_debug() +- print_hex_dump_bytes() + +Therefore the usability of this tool is, as of now, quite limited as there= is +no uniform rule for adding debug prints to the codebase, resulting in a va= riety +of ways these prints are implemented. + +Also, note that most debug statements are implemented as a variation of +dprintk(), which have to be activated via a parameter in respective module, +dynamic debug is unable to do that step for you. + +Here is one example, that enables all available pr_debug() 's within the f= ile:: + + $ alias ddcmd=3D'echo $* > /proc/dynamic_debug/control' + $ ddcmd '-p; file v4l2-h264.c +p' + $ grep =3Dp /proc/dynamic_debug/control + drivers/media/v4l2-core/v4l2-h264.c:372 [v4l2_h264]print_ref_list_b =3Dp + "ref_pic_list_b%u (cur_poc %u%c) %s" + drivers/media/v4l2-core/v4l2-h264.c:333 [v4l2_h264]print_ref_list_p =3Dp + "ref_pic_list_p (cur_poc %u%c) %s\n" + +**When should you use this over** `Ftrace`_ **?** + +- When the code contains one of the :ref:`valid print statements + ` or when you have added multiple pr_debug() + statements during development +- When timing is not an issue, meaning if multiple pr_debug() statements in + the code won't cause delays +- When you care more about receiving specific log messages than tracing the + pattern of how a function is called + +For the full documentation see :doc:`/admin-guide/dynamic-debug-howto` + +Ftrace +------ + +Prerequisite: `CONFIG_DYNAMIC_FTRACE` + +You can find the tracing folder in either `/sys/kernel/` or `/sys/debug/ke= rnel/`. + +Some of the most important operations for debugging are: + +- You can perform a function trace by adding a function name to the + `set_ftrace_filter` file (which accepts any function name found within t= he + `available_filter_functions` file) or you can specifically disable certa= in + functions by adding their names to the `set_ftrace_notrace` file (More i= nfo + at: :ref:`trace/ftrace:dynamic ftrace`). +- In order to find out where the calls originates from you can activate the + `func_stack_trace` option under `options/func_stack_trace`. +- Tracing the children of a function call and showing the return values is + possible by adding the desired function in the `set_graph_function` file + (requires config `FUNCTION_GRAPH_RETVAL`) more info at + :ref:`trace/ftrace:dynamic ftrace with the function graph tracer`. + +For the full Ftrace documentation see :doc:`/trace/ftrace` + +Or you could also trace for specific events by :ref:`using event tracing +`, which can be defined as described = here: +:ref:`Creating a custom Ftrace tracepoint +`. + +For the full Ftrace event tracing documentation see :doc:`/trace/events` + +.. _read_ftrace_log: + +Reading the ftrace log +~~~~~~~~~~~~~~~~~~~~~~ + +The `trace` file can be read just like any other file (`cat`, `tail`, `hea= d`, +`vim`, etc.), the size of the file is limited by the `buffer_size_kb` (``e= cho +1000 > buffer_size_kb``). The :ref:`trace/ftrace:trace_pipe` will behave +similar to the `trace` file, but whenever you read from the file the conte= nt is +consumed. + +Kernelshark +~~~~~~~~~~~ + +A GUI interface to visualize the traces as a graph and list view from the +output of the `trace-cmd +`__ applica= tion. + +For the full documentation see ``__ + +Perf & alternatives +------------------- + +The tools mentioned above provide ways to inspect kernel code, results, va= riable values, etc. +Sometimes you have to find out first where to look and for those cases, a = box of +performance tracking tools can help you to frame the issue. + +Why should you do a performance analysis? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A performance analysis is a good first step when among other reasons: + +- you cannot define the issue +- you do not know where it occurs +- the running system should not be interrupted or it is a remote system, w= here + you cannot install a new module/kernel + +How to do a simple analysis with linux tools? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For the start of a performance analysis, you can start with the usual tools +like: + +- `top` / `htop` / `atop` (*get an overview of the system load, see spikes= on + specific processes*) +- `mpstat -P ALL` (*look at the load distribution among CPUs*) +- `iostat -x` (*observe input and output devices utilization and performan= ce*) +- `vmstat` (*overview of memory usage on the system*) +- `pidstat` (*similar to* `vmstat` *but per process, to dial it down to the + target*) +- `strace -tp $PID` (*once you know the process, you can figure out how it + communicates with the Kernel*) + +These should help to narrow down the areas to look at sufficiently. + +Diving deeper with perf +~~~~~~~~~~~~~~~~~~~~~~~ + +The **perf** tool provides a series of metrics and events to further dial = down +on issues. + +Prerequisite: build or install perf on your system + +Gather statistics data for finding all files starting with `gcc` in `/usr`= :: + + # perf stat -d find /usr -name 'gcc*' | wc -l + + Performance counter stats for 'find /usr -name gcc*': + + 1277.81 msec task-clock # 0.997 CPUs utilized + 9 context-switches # 7.043 /sec + 1 cpu-migrations # 0.783 /sec + 704 page-faults # 550.943 /sec + 766548897 cycles # 0.600 GHz = (97.15%) + 798285467 instructions # 1.04 insn per cycle = (97.15%) + 57582731 branches # 45.064 M/sec = (2.85%) + 3842573 branch-misses # 6.67% of all branches = (97.15%) + 281616097 L1-dcache-loads # 220.390 M/sec = (97.15%) + 4220975 L1-dcache-load-misses # 1.50% of all L1-dcache ac= cesses (97.15%) + LLC-loads + LLC-load-misses + + 1.281746009 seconds time elapsed + + 0.508796000 seconds user + 0.773209000 seconds sys + + + 52 + +The availability of events and metrics depends on the system you are runni= ng. + +For the full documentation see +``__ + +Perfetto +~~~~~~~~ + +A set of tools to measure and analyze how well applications and systems pe= rform. +You can use it to: + +* identify bottlenecks +* optimize code +* make software run faster and more efficiently. + +**What is the difference between perfetto and perf?** + +* perf is tool as part of and specialized for the Linux Kernel and has CLI= user + interface. +* perfetto cross-platform performance analysis stack, has extended + functionality into userspace and provides a WEB user interface. + +For the full documentation see ``__ + +Kernel panic analysis tools +--------------------------- + + To analyse the crash dump please use `Kdump` & `Kexec`. + + For the full documentation see the :doc:`/admin-guide/kdump/kdump` + + In order to find the corresponding line in the code you can use `faddr2l= ine + `__,= note + that you need to enable `CONFIG_DEBUG_INFO` for that to work. + + An alternative to using `faddr2line` is the use of `objdump` (and it's + derivatives for the different platforms like `aarch64-linux-gnu-objdump`= ), + take this line as an example: + + `[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]`. + + We can find the corresponding line of code by executing:: + + aarch64-linux-gnu-objdump -dS drivers/staging/media/rkvdec/rockchip-vd= ec.ko | grep rkvdec_device_run\>: -A 40 + 0000000000000ac8 : + ac8: d503201f nop + acc: d503201f nop + { + ad0: d503233f paciasp + ad4: a9bd7bfd stp x29, x30, [sp, #-48]! + ad8: 910003fd mov x29, sp + adc: a90153f3 stp x19, x20, [sp, #16] + ae0: a9025bf5 stp x21, x22, [sp, #32] + const struct rkvdec_coded_fmt_desc *desc =3D ctx->coded_fmt_desc; + ae4: f9411814 ldr x20, [x0, #560] + struct rkvdec_dev *rkvdec =3D ctx->dev; + ae8: f9418015 ldr x21, [x0, #768] + if (WARN_ON(!desc)) + aec: b4000654 cbz x20, bb4 + ret =3D pm_runtime_resume_and_get(rkvdec->dev); + af0: f943d2b6 ldr x22, [x21, #1952] + ret =3D __pm_runtime_resume(dev, RPM_GET_PUT); + af4: aa0003f3 mov x19, x0 + af8: 52800081 mov w1, #0x4 // #4 + afc: aa1603e0 mov x0, x22 + b00: 94000000 bl 0 <__pm_runtime_resume> + if (ret < 0) { + b04: 37f80340 tbnz w0, #31, b6c + dev_warn(rkvdec->dev, "Not good\n"); + b08: f943d2a0 ldr x0, [x21, #1952] + b0c: 90000001 adrp x1, 0 + b10: 91000021 add x1, x1, #0x0 + b14: 94000000 bl 0 <_dev_warn> + *bad =3D 1; + b18: d2800001 mov x1, #0x0 // #0 + ... + + Meaning, in this line from the crash dump:: + + [ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec] + + I can take the `0x50` as offset, which I have to add to the base address + of the corresponding function, which I find in this line:: + + 0000000000000ac8 : + + The result of `0xac8 + 0x50 =3D 0xb18` + And when I search for that address within the function I get the + following line:: + + *bad =3D 1; + b18: d2800001 mov x1, #0x0 + +**Copyright** =C2=A92024 : Collabora --=20 2.25.1 From nobody Sun Nov 24 04:43:53 2024 Received: from sender4-pp-f112.zoho.com (sender4-pp-f112.zoho.com [136.143.188.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8EE528F1; Thu, 7 Nov 2024 14:00:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=136.143.188.112 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730988058; cv=pass; b=BlZdUjS1iGyOFPXSUdWy6o9famgR41ypIykHxiqLN1IuZwrVXr6MNK+5XYr5ivlbT0tv9GycvsMwIWnM672cW/h6urZkhXCu73HACpAyhhXx5unroCopiD6Fnj5ZbNjV1FMqjaqWjfHsnFOKRLkOO+geeWm7aGj6blRhgugTVgk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730988058; c=relaxed/simple; bh=4NcsKRtVZifoMZpv/hWeYJyhhc5NM9VV7JwBM8iZYu4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mi2XcqOiWFjMPxGWvcIwTZIxeFsUx6L524mnyBbBrUhaAB0Gw1HZ9MxAHtCjyxbofmvmKbm5S5uBRaBsZte1rVJ+uyVmhXBQjZQAmlp52Mo2SpAewas4tk7V/5/Q25RRPa+BuBBOAtvcU1xG7eEyDm1B+bmn6gggnUXxC8j23PY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (1024-bit key) header.d=collabora.com header.i=sebastian.fricke@collabora.com header.b=dCGAV1O3; arc=pass smtp.client-ip=136.143.188.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=collabora.com header.i=sebastian.fricke@collabora.com header.b="dCGAV1O3" ARC-Seal: i=1; a=rsa-sha256; t=1730988043; cv=none; d=zohomail.com; s=zohoarc; b=FPzo6oCh3divIIwH605kd/PhJXr0AK8WevAN6er8ZLUtrKSC4cHNcA6EQdaq/ookC2XW4XD+sVOM1+OTWoAPGDCaOZE91zJYVi0KqjQMOmUwHiCZBG87BLRxmSoFzYnrsy8fhllNiMiWD8bUtYGNWGEy6SkopeNFqqMIe9o7QB4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1730988043; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=FQlNU2ZVOPW7ydKhB4DJDtUqfzE3jqalhsT9d8MtRhc=; b=T3ZFHP7Aocc0l73a8c1QCF0r/ItZF9T5OfjRke9vTQny69kZDt3zFJIUPEGYHFKRCJa0a4J4Y73zKrtE2v6DaVg+anUuPSZfDB1sfReO3FkGBbYksy/gQZ95RFvdMZdPYNsQEW/kYdC1iR4XAzaZxnqpAfxA1yEFkn8EjUlAHkA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=collabora.com; spf=pass smtp.mailfrom=sebastian.fricke@collabora.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1730988043; s=zohomail; d=collabora.com; i=sebastian.fricke@collabora.com; h=From:From:Date:Date:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Message-Id:References:In-Reply-To:To:To:Cc:Cc:Reply-To; bh=FQlNU2ZVOPW7ydKhB4DJDtUqfzE3jqalhsT9d8MtRhc=; b=dCGAV1O3473pj6EBP6vlotZ+6OpeOIYS0Z/f7bAKnWluzsz3lnKaxjqS4fg3w2Y6 peHkvDiPPUdb2fy++RctYiCbOWqZDfoqtT2Y/jbhUgP/1tcWp36N3fB6JmZ2uMVkroT C9eXw9x5vA3B7dCuSj2Pj4BG7uN8GhNjohFDERa4= Received: by mx.zohomail.com with SMTPS id 1730988040967873.7786360092949; Thu, 7 Nov 2024 06:00:40 -0800 (PST) From: Sebastian Fricke Date: Thu, 07 Nov 2024 15:00:24 +0100 Subject: [PATCH 2/2] docs: media: Debugging guide for the media subsystem Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20241028-media_docs_improve_v3-v1-2-2b1b486c223e@collabora.com> References: <20241028-media_docs_improve_v3-v1-0-2b1b486c223e@collabora.com> In-Reply-To: <20241028-media_docs_improve_v3-v1-0-2b1b486c223e@collabora.com> To: Jonathan Corbet Cc: bagasdotme@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, laurent.pinchart@ideasonboard.com, hverkuil-cisco@xs4all.nl, mauro.chehab@linux.intel.com, kernel@collabora.com, bob.beckett@collabora.com, nicolas.dufresne@collabora.com, Sebastian Fricke X-Mailer: b4 0.11.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1730988026; l=8175; i=sebastian.fricke@collabora.com; s=linux-media; h=from:subject:message-id; bh=4NcsKRtVZifoMZpv/hWeYJyhhc5NM9VV7JwBM8iZYu4=; b=Frt57lxx3bQHq1CrSJu82LhWrtDlJ+PDHIoTpHWtcaBPg583hUxvbg8rdo7QnoJrs+b+T7KaPeTu X5zjDHTvCKdVVxHXrDj+Pv4kVp36u5cnA2r0AGtIAnaAq74TtTCv X-Developer-Key: i=sebastian.fricke@collabora.com; a=ed25519; pk=pYXedPwrTtErcj7ERYeo/IpTrpe4QbJuEzSB52fslBg= X-ZohoMailClient: External Provide a guide for developers on how to debug code with a focus on the media subsystem. This document aims to provide a rough overview over the possibilities and a rational to help choosing the right tool for the given circumstances. Signed-off-by: Sebastian Fricke --- Documentation/process/debugging/index.rst | 1 + .../debugging/media_specific_debugging_guide.rst | 178 +++++++++++++++++= ++++ 2 files changed, 179 insertions(+) diff --git a/Documentation/process/debugging/index.rst b/Documentation/proc= ess/debugging/index.rst index c200ede7c955..322e33d65ca3 100644 --- a/Documentation/process/debugging/index.rst +++ b/Documentation/process/debugging/index.rst @@ -10,6 +10,7 @@ Debugging advice for Linux Kernel developers general_advice driver_development_debugging_guide userspace_debugging_guide + media_specific_debugging_guide =20 .. only:: subproject and html =20 diff --git a/Documentation/process/debugging/media_specific_debugging_guide= .rst b/Documentation/process/debugging/media_specific_debugging_guide.rst new file mode 100644 index 000000000000..ab840e695af9 --- /dev/null +++ b/Documentation/process/debugging/media_specific_debugging_guide.rst @@ -0,0 +1,178 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Debugging and tracing in the media subsystem +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This document serves as a starting point and lookup for debugging device +drivers in the media subsystem and to debug these drivers from userspace. + +.. contents:: + :depth: 3 + +General debugging advice +------------------------ + +For general advice see the :doc:`/process/debugging/general_advice`. + +The following sections show you some of the available tools. + +dev_debug module parameter +-------------------------- + +Every video device provides a `dev_debug` parameter, which allows to get +further insights into the IOCTLs in the background.:: + + # cat /sys/class/video4linux/video3/name + rkvdec + # echo 0xff > /sys/class/video4linux/video3/dev_debug + # dmesg -wH + [...] videodev: v4l2_open: video3: open (0) + [ +0.000036] video3: VIDIOC_QUERYCAP: driver=3Drkvdec, card=3Drkvdec, + bus=3Dplatform:rkvdec, version=3D0x00060900, capabilities=3D0x84204000, + device_caps=3D0x04204000 + +For the full documentation see :ref:`driver-api/media/v4l2-dev:video devic= e debugging` + +dev_dbg / v4l2_dbg +------------------ + +Two debug print statements, which are specific for devices and for the v4l2 +subsystem, avoid adding these to your final submission unless they have +long-term value for investigations. + +For a general overview please see the +:ref:`process/debugging/driver_development_debugging_guide:printk & friend= s` +guide. + +- Difference between both? + + - v4l2_dbg utilizes v4l2_printk under the hood, which further uses printk + directly, thus it cannot be targeted by dynamic debug + - dev_dbg can be targeted by dynamic debug + - v4l2_dbg has a more specific prefix format for the media subsystem, wh= ile + dev_dbg only highlights the driver name and the location of the log + +Dynamic debug +------------- + +A method to trim down the debug output to your needs. + +For general advice see the +:ref:`process/debugging/userspace_debugging_guide:dynamic debug` guide. + +Here is one example, that enables all available pr_debug()'s within the fi= le:: + + $ alias ddcmd=3D'echo $* > /proc/dynamic_debug/control' + $ ddcmd '-p; file v4l2-h264.c +p' + $ grep =3Dp /proc/dynamic_debug/control + drivers/media/v4l2-core/v4l2-h264.c:372 [v4l2_h264]print_ref_list_b =3Dp + "ref_pic_list_b%u (cur_poc %u%c) %s" + drivers/media/v4l2-core/v4l2-h264.c:333 [v4l2_h264]print_ref_list_p =3Dp + "ref_pic_list_p (cur_poc %u%c) %s\n" + +Ftrace +------ + +An internal kernel tracer that can trace static predefined events, function +calls, etc. Very useful for debugging problems without changing the kernel= and +understanding the behavior of subsystems. + +For general advice see the +:ref:`process/debugging/userspace_debugging_guide:ftrace` guide. + +DebugFS +------- + +This tool allows you to dump or modify internal values of your driver to f= iles +in a custom filesystem. + +For general advice see the +:ref:`process/debugging/driver_development_debugging_guide:debugfs` guide. + +Perf & alternatives +------------------- + +Tools to measure the various stats on a running system to diagnose issues. + +For general advice see the +:ref:`process/debugging/userspace_debugging_guide:perf & alternatives` gui= de. + +Example for media devices: + +Gather statistics data for a decoding job: (This example is on a RK3399 SoC +with the rkvdec codec driver using the `fluster test suite +`__):: + + perf stat -d python3 fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -ts + JVT-AVC_V1 -tv AUD_MW_E -j1 + ... + Performance counter stats for 'python3 fluster.py run -d + GStreamer-H.264-V4L2SL-Gst1.0 -ts JVT-AVC_V1 -tv AUD_MW_E -j1 -v': + + 7794.23 msec task-clock:u # 0.697 CPUs uti= lized + 0 context-switches:u # 0.000 /sec + 0 cpu-migrations:u # 0.000 /sec + 11901 page-faults:u # 1.527 K/sec + 882671556 cycles:u # 0.113 GHz = (95.79%) + 711708695 instructions:u # 0.81 insn per= cycle (95.79%) + 10581935 branches:u # 1.358 M/sec = (15.13%) + 6871144 branch-misses:u # 64.93% of all b= ranches (95.79%) + 281716547 L1-dcache-loads:u # 36.144 M/sec = (95.79%) + 9019581 L1-dcache-load-misses:u # 3.20% of all L= 1-dcache accesses (95.79%) + LLC-loads:u + LLC-load-misses:u + + 11.180830431 seconds time elapsed + + 1.502318000 seconds user + 6.377221000 seconds sys + +The availability of events and metrics depends on the system you are runni= ng. + +Error checking & panic analysis +------------------------------- + +Various Kernel configuration options to enhance error detection of the Lin= ux +Kernel with the cost of lowering performance. + +For general advice see the +:ref:`process/debugging/driver_development_debugging_guide:kasan, ubsan, +lockdep and other error checkers` guide. + +Driver verification with v4l2-compliance +---------------------------------------- + +To verify, that a driver adheres to the v4l2 API, the tool v4l2-compliance= is +used, which is part of the `v4l_utils +`__, a suite of userspace tools to = work +with the media subsystem. + +To see the detailed media topology (and check it) use:: + + v4l2-compliance -M /dev/mediaX --verbose + +You can also run a full compliance check for all devices referenced in the +media topology with:: + + v4l2-compliance -m /dev/mediaX + +Debugging problems with receiving video +--------------------------------------- + +Implementing vidioc_log_status in the driver: this can log the current sta= tus +to the kernel log. It's called by v4l2-ctl --log-status. Very useful for +debugging problems with receiving video (TV/S-Video/HDMI/etc) since the vi= deo +signal is external (so unpredictable). Less useful with camera sensor inpu= ts +since you have control over what the camera sensor does. + +Usually you can just assign the default:: + + .vidioc_log_status =3D v4l2_ctrl_log_status, + +But you can also create your own callback, to create a custom status log. + +You can find an example in the cobalt driver +(`drivers/media/pci/cobalt/cobalt-v4l2.c `__). + +**Copyright** =C2=A92024 : Collabora --=20 2.25.1