From nobody Fri Dec 19 04:25:33 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FEF8C7EE2E for ; Fri, 26 May 2023 04:21:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242002AbjEZEVA (ORCPT ); Fri, 26 May 2023 00:21:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241981AbjEZEUo (ORCPT ); Fri, 26 May 2023 00:20:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 504DAE5E; Thu, 25 May 2023 21:20:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5E1F163A23; Fri, 26 May 2023 04:19:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07735C433D2; Fri, 26 May 2023 04:19:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685074798; bh=8MRjZRiswdJA7BkVmwwwg6IT4p9+fB1e3FMsiIY4h0c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PTHaUTstR/AtgyMdNL8aU2w0sl3hScvUdG4IKk5VBKuYEeltmo4uEUt+198sqqxNG a1hPFH0YMPjUUpfRSMIjcF4br7nAT/sZgOqJBrIWt8QcGODGG/oFtAAm5rsTi3K9Bc aHLSNCqHKxUbffO/uGA0RwLNZNqjzuOp2iI9IMmtcJAFO+lCOduzIQVA9Skrc17jwF qlZeIxERwcdJ7eBgsFJMNW8qgqbpbm+9Oo9FfNgRdLkSr6cyj8VwbeLHuBE/0hV6az pkn1OuP0EY9MTZKM8I+dObJzrp+FMNHzwbqtzaVEqK0t4lB3B38Yh6QAF1ErTQ78Kd JCUpW9dV8RkyQ== From: "Masami Hiramatsu (Google)" To: linux-trace-kernel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Steven Rostedt , mhiramat@kernel.org, Florent Revest , Mark Rutland , Will Deacon , Mathieu Desnoyers , Martin KaFai Lau , bpf@vger.kernel.org, Bagas Sanjaya Subject: [PATCH v13 12/12] Documentation: tracing/probes: Add fprobe event tracing document Date: Fri, 26 May 2023 12:19:53 +0800 Message-ID: <168507479345.913472.2804569685436422001.stgit@mhiramat.roam.corp.google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog In-Reply-To: <168507466597.913472.10572827237387849017.stgit@mhiramat.roam.corp.google.com> References: <168507466597.913472.10572827237387849017.stgit@mhiramat.roam.corp.google.com> User-Agent: StGit/0.19 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Masami Hiramatsu (Google) Add a documentation about fprobe event tracing including tracepoint probe event and BTF argument. Signed-off-by: Masami Hiramatsu (Google) Reviewed-by: Bagas Sanjaya --- Changes in v12: - Use double backquotes for keywords and identifiers. - Use the ref link instead of file. Changes in v11: - Fix $$args to $arg*. - Update the document. Changes in v10: - Fix space before tab. Changes in v7: - Update about BTF auto type casting for $retval. --- Documentation/trace/fprobetrace.rst | 188 +++++++++++++++++++++++++++++++= ++++ Documentation/trace/index.rst | 1=20 Documentation/trace/kprobetrace.rst | 2=20 3 files changed, 191 insertions(+) create mode 100644 Documentation/trace/fprobetrace.rst diff --git a/Documentation/trace/fprobetrace.rst b/Documentation/trace/fpro= betrace.rst new file mode 100644 index 000000000000..e949bc0cff05 --- /dev/null +++ b/Documentation/trace/fprobetrace.rst @@ -0,0 +1,188 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +Fprobe-based Event Tracing +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + +.. Author: Masami Hiramatsu + +Overview +-------- + +Fprobe event is similar to the kprobe event, but limited to probe on +the function entry and exit only. It is good enough for many use cases +which only traces some specific functions. + +This document also covers tracepoint probe events (tprobe) since this +is also works only on the tracepoint entry. User can trace a part of +tracepoint argument, or the tracepoint without trace-event, which is +not exposed on tracefs. + +As same as other dynamic events, fprobe events and tracepoint probe +events are defined via `dynamic_events` interface file on tracefs. + +Synopsis of fprobe-events +------------------------- +:: + + f[:[GRP1/][EVENT1]] SYM [FETCHARGS] : Probe on fun= ction entry + f[MAXACTIVE][:[GRP1/][EVENT1]] SYM%return [FETCHARGS] : Probe on fun= ction exit + t[:[GRP2/][EVENT2]] TRACEPOINT [FETCHARGS] : Probe on tra= cepoint + + GRP1 : Group name for fprobe. If omitted, use "fprobes" for it. + GRP2 : Group name for tprobe. If omitted, use "tracepoints" for= it. + EVENT1 : Event name for fprobe. If omitted, the event name is + "SYM__entry" or "SYM__exit". + EVENT2 : Event name for tprobe. If omitted, the event name is + the same as "TRACEPOINT", but if the "TRACEPOINT" starts + with a digit character, "_TRACEPOINT" is used. + MAXACTIVE : Maximum number of instances of the specified function th= at + can be probed simultaneously, or 0 for the default value + as defined in Documentation/trace/fprobes.rst + + FETCHARGS : Arguments. Each probe can have up to 128 args. + ARG : Fetch "ARG" function argument using BTF (only for functi= on + entry or tracepoint.) (\*1) + @ADDR : Fetch memory at ADDR (ADDR should be in kernel) + @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbo= l) + $stackN : Fetch Nth entry of stack (N >=3D 0) + $stack : Fetch stack address. + $argN : Fetch the Nth function argument. (N >=3D 1) (\*2) + $retval : Fetch return value.(\*3) + $comm : Fetch current task comm. + +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*4)(\= *5) + \IMM : Store an immediate value to the argument. + NAME=3DFETCHARG : Set NAME as the argument name of FETCHARG. + FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types + (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types + (x8/x16/x32/x64), "char", "string", "ustring", "symbol",= "symstr" + and bitfield are supported. + + (\*1) This is available only when BTF is enabled. + (\*2) only for the probe on function entry (offs =3D=3D 0). + (\*3) only for return probe. + (\*4) this is useful for fetching a field of data structures. + (\*5) "u" means user-space dereference. + +For the details of TYPE, see :ref:`kprobetrace documentation `. + +BTF arguments +------------- +BTF (BPF Type Format) argument allows user to trace function and tracepoint +parameters by its name instead of ``$argN``. This feature is available if = the +kernel is configured with CONFIG_BPF_SYSCALL and CONFIG_DEBUG_INFO_BTF. +If user only specify the BTF argument, the event's argument name is also +automatically set by the given name. :: + + # echo 'f:myprobe vfs_read count pos' >> dynamic_events + # cat dynamic_events + f:fprobes/myprobe vfs_read count=3Dcount pos=3Dpos + +It also chooses the fetch type from BTF information. For example, in the a= bove +example, the ``count`` is unsigned long, and the ``pos`` is a pointer. Thu= s, both +are converted to 64bit unsigned long, but only ``pos`` has "%Lx" print-for= mat as +below :: + + # cat events/fprobes/myprobe/format + name: myprobe + ID: 1313 + format: + field:unsigned short common_type; offset:0; size:2; signed:0; + field:unsigned char common_flags; offset:2; size:1; signed:0; + field:unsigned char common_preempt_count; offset:3; size:1; signed:0; + field:int common_pid; offset:4; size:4; signed:1; + + field:unsigned long __probe_ip; offset:8; size:8; signed:0; + field:u64 count; offset:16; size:8; signed:0; + field:u64 pos; offset:24; size:8; signed:0; + + print fmt: "(%lx) count=3D%Lu pos=3D0x%Lx", REC->__probe_ip, REC->count, = REC->pos + +If user unsures the name of arguments, ``$arg*`` will be helpful. The ``$a= rg*`` +is expanded to all function arguments of the function or the tracepoint. :: + + # echo 'f:myprobe vfs_read $arg*' >> dynamic_events + # cat dynamic_events + f:fprobes/myprobe vfs_read file=3Dfile buf=3Dbuf count=3Dcount pos=3Dpos + +BTF also affects the ``$retval``. If user doesn't set any type, the retval= type is +automatically picked from the BTF. If the function returns ``void``, ``$re= tval`` +is rejected. + +Usage examples +-------------- +Here is an example to add fprobe events on ``vfs_read()`` function entry +and exit, with BTF arguments. +:: + + # echo 'f vfs_read $arg*' >> dynamic_events + # echo 'f vfs_read%return $retval' >> dynamic_events + # cat dynamic_events + f:fprobes/vfs_read__entry vfs_read file=3Dfile buf=3Dbuf count=3Dcount po= s=3Dpos + f:fprobes/vfs_read__exit vfs_read%return arg1=3D$retval + # echo 1 > events/fprobes/enable + # head -n 20 trace | tail + # TASK-PID CPU# ||||| TIMESTAMP FUNCTION + # | | | ||||| | | + sh-70 [000] ...1. 335.883195: vfs_read__entry: (vfs_= read+0x4/0x340) file=3D0xffff888005cf9a80 buf=3D0x7ffef36c6879 count=3D1 po= s=3D0xffffc900005aff08 + sh-70 [000] ..... 335.883208: vfs_read__exit: (ksys_= read+0x75/0x100 <- vfs_read) arg1=3D1 + sh-70 [000] ...1. 335.883220: vfs_read__entry: (vfs_= read+0x4/0x340) file=3D0xffff888005cf9a80 buf=3D0x7ffef36c6879 count=3D1 po= s=3D0xffffc900005aff08 + sh-70 [000] ..... 335.883224: vfs_read__exit: (ksys_= read+0x75/0x100 <- vfs_read) arg1=3D1 + sh-70 [000] ...1. 335.883232: vfs_read__entry: (vfs_= read+0x4/0x340) file=3D0xffff888005cf9a80 buf=3D0x7ffef36c687a count=3D1 po= s=3D0xffffc900005aff08 + sh-70 [000] ..... 335.883237: vfs_read__exit: (ksys_= read+0x75/0x100 <- vfs_read) arg1=3D1 + sh-70 [000] ...1. 336.050329: vfs_read__entry: (vfs_= read+0x4/0x340) file=3D0xffff888005cf9a80 buf=3D0x7ffef36c6879 count=3D1 po= s=3D0xffffc900005aff08 + sh-70 [000] ..... 336.050343: vfs_read__exit: (ksys_= read+0x75/0x100 <- vfs_read) arg1=3D1 + +You can see all function arguments and return values are recorded as signe= d int. + +Also, here is an example of tracepoint events on ``sched_switch`` tracepoi= nt. +To compare the result, this also enables the ``sched_switch`` traceevent t= oo. +:: + + # echo 't sched_switch $arg*' >> dynamic_events + # echo 1 > events/sched/sched_switch/enable + # echo 1 > events/tracepoints/sched_switch/enable + # echo > trace + # head -n 20 trace | tail + # TASK-PID CPU# ||||| TIMESTAMP FUNCTION + # | | | ||||| | | + sh-70 [000] d..2. 3912.083993: sched_switch: prev_com= m=3Dsh prev_pid=3D70 prev_prio=3D120 prev_state=3DS =3D=3D> next_comm=3Dswa= pper/0 next_pid=3D0 next_prio=3D120 + sh-70 [000] d..3. 3912.083995: sched_switch: (__probe= stub_sched_switch+0x4/0x10) preempt=3D0 prev=3D0xffff88800664e100 next=3D0x= ffffffff828229c0 prev_state=3D1 + -0 [000] d..2. 3912.084183: sched_switch: prev_com= m=3Dswapper/0 prev_pid=3D0 prev_prio=3D120 prev_state=3DR =3D=3D> next_comm= =3Drcu_preempt next_pid=3D16 next_prio=3D120 + -0 [000] d..3. 3912.084184: sched_switch: (__probe= stub_sched_switch+0x4/0x10) preempt=3D0 prev=3D0xffffffff828229c0 next=3D0x= ffff888004208000 prev_state=3D0 + rcu_preempt-16 [000] d..2. 3912.084196: sched_switch: prev_com= m=3Drcu_preempt prev_pid=3D16 prev_prio=3D120 prev_state=3DI =3D=3D> next_c= omm=3Dswapper/0 next_pid=3D0 next_prio=3D120 + rcu_preempt-16 [000] d..3. 3912.084196: sched_switch: (__probe= stub_sched_switch+0x4/0x10) preempt=3D0 prev=3D0xffff888004208000 next=3D0x= ffffffff828229c0 prev_state=3D1026 + -0 [000] d..2. 3912.085191: sched_switch: prev_com= m=3Dswapper/0 prev_pid=3D0 prev_prio=3D120 prev_state=3DR =3D=3D> next_comm= =3Drcu_preempt next_pid=3D16 next_prio=3D120 + -0 [000] d..3. 3912.085191: sched_switch: (__probe= stub_sched_switch+0x4/0x10) preempt=3D0 prev=3D0xffffffff828229c0 next=3D0x= ffff888004208000 prev_state=3D0 + +As you can see, the ``sched_switch`` trace-event shows *cooked* parameters= , on +the other hand, the ``sched_switch`` tracepoint probe event shows *raw* +parameters. This means you can access any field values in the task +structure pointed by the ``prev`` and ``next`` arguments. + +For example, usually ``task_struct::start_time`` is not traced, but with t= his +traceprobe event, you can trace it as below. +:: + + # echo 't sched_switch comm=3D+1896(next):string start_time=3D+1728(next= ):u64' > dynamic_events + # head -n 20 trace | tail + # TASK-PID CPU# ||||| TIMESTAMP FUNCTION + # | | | ||||| | | + sh-70 [000] d..3. 5606.686577: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"rcu_preempt" usage=3D1 start_time=3D245= 000000 + rcu_preempt-16 [000] d..3. 5606.686602: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"sh" usage=3D1 start_time=3D1596095526 + sh-70 [000] d..3. 5606.686637: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"swapper/0" usage=3D2 start_time=3D0 + -0 [000] d..3. 5606.687190: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"rcu_preempt" usage=3D1 start_time=3D245= 000000 + rcu_preempt-16 [000] d..3. 5606.687202: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"swapper/0" usage=3D2 start_time=3D0 + -0 [000] d..3. 5606.690317: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"kworker/0:1" usage=3D1 start_time=3D137= 000000 + kworker/0:1-14 [000] d..3. 5606.690339: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"swapper/0" usage=3D2 start_time=3D0 + -0 [000] d..3. 5606.692368: sched_switch: (__probe= stub_sched_switch+0x4/0x10) comm=3D"kworker/0:1" usage=3D1 start_time=3D137= 000000 + +Currently, to find the offset of a specific field in the data structure, +you need to build kernel with debuginfo and run `perf probe` command with +`-D` option. e.g. +:: + + # perf probe -D "__probestub_sched_switch next->comm:string next->start_t= ime" + p:probe/__probestub_sched_switch __probestub_sched_switch+0 comm=3D+1896(= %cx):string start_time=3D+1728(%cx):u64 + +And replace the ``%cx`` with the ``next``. diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst index ea25a9220f92..5092d6c13af5 100644 --- a/Documentation/trace/index.rst +++ b/Documentation/trace/index.rst @@ -13,6 +13,7 @@ Linux Tracing Technologies kprobes kprobetrace uprobetracer + fprobetrace tracepoints events events-kmem diff --git a/Documentation/trace/kprobetrace.rst b/Documentation/trace/kpro= betrace.rst index 651f9ab53f3e..8a2dfee38145 100644 --- a/Documentation/trace/kprobetrace.rst +++ b/Documentation/trace/kprobetrace.rst @@ -66,6 +66,8 @@ Synopsis of kprobe_events (\*3) this is useful for fetching a field of data structures. (\*4) "u" means user-space dereference. See :ref:`user_mem_access`. =20 +.. _kprobetrace_types: + Types ----- Several types are supported for fetchargs. Kprobe tracer will access memory