From: Donglin Peng <pengdonglin@xiaomi.com>
Use BTF to print symbolic names for enum-type function arguments,
improving trace readability.
Before:
count_memcg_events(memcg=0xffff..., idx=20, count=0x1) {
After:
count_memcg_events(memcg=0xffff..., idx=20 [PGFAULT], count=0x1) {
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: linux-trace-kernel@vger.kernel.org
Signed-off-by: Donglin Peng <pengdonglin@xiaomi.com>
---
kernel/trace/trace_output.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c
index cc2d3306bb60..c395f768c3b8 100644
--- a/kernel/trace/trace_output.c
+++ b/kernel/trace/trace_output.c
@@ -695,12 +695,13 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
{
const struct btf_param *param;
const struct btf_type *t;
+ const struct btf_enum *enump;
const char *param_name;
char name[KSYM_NAME_LEN];
unsigned long arg;
struct btf *btf;
s32 tid, nr = 0;
- int a, p, x;
+ int a, p, x, i;
u16 encode;
trace_seq_printf(s, "(");
@@ -754,6 +755,14 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
break;
case BTF_KIND_ENUM:
trace_seq_printf(s, "%ld", arg);
+ for_each_enum(i, t, enump) {
+ if (arg == enump->val) {
+ trace_seq_printf(s, " [%s]",
+ btf_name_by_offset(btf,
+ enump->name_off));
+ break;
+ }
+ }
break;
default:
/* This does not handle complex arguments */
--
2.34.1On Mon, 2 Feb 2026 19:15:48 +0800
Donglin Peng <dolinux.peng@gmail.com> wrote:
> From: Donglin Peng <pengdonglin@xiaomi.com>
>
> Use BTF to print symbolic names for enum-type function arguments,
> improving trace readability.
>
> Before:
> count_memcg_events(memcg=0xffff..., idx=20, count=0x1) {
>
> After:
> count_memcg_events(memcg=0xffff..., idx=20 [PGFAULT], count=0x1) {
Hmm, it is interesting idea. Maybe I can also introduce the same
feature to fetcharg for fprobe/kprobes as a new type.
Since the enum type should be solved in parsing the event definition,
for fprobe/kprobes the performance will not be a performance problem.
Thanks,
>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: linux-trace-kernel@vger.kernel.org
> Signed-off-by: Donglin Peng <pengdonglin@xiaomi.com>
> ---
> kernel/trace/trace_output.c | 11 ++++++++++-
> 1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c
> index cc2d3306bb60..c395f768c3b8 100644
> --- a/kernel/trace/trace_output.c
> +++ b/kernel/trace/trace_output.c
> @@ -695,12 +695,13 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
> {
> const struct btf_param *param;
> const struct btf_type *t;
> + const struct btf_enum *enump;
> const char *param_name;
> char name[KSYM_NAME_LEN];
> unsigned long arg;
> struct btf *btf;
> s32 tid, nr = 0;
> - int a, p, x;
> + int a, p, x, i;
> u16 encode;
>
> trace_seq_printf(s, "(");
> @@ -754,6 +755,14 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
> break;
> case BTF_KIND_ENUM:
> trace_seq_printf(s, "%ld", arg);
> + for_each_enum(i, t, enump) {
> + if (arg == enump->val) {
> + trace_seq_printf(s, " [%s]",
> + btf_name_by_offset(btf,
> + enump->name_off));
> + break;
> + }
> + }
> break;
> default:
> /* This does not handle complex arguments */
> --
> 2.34.1
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
On Mon, 2 Feb 2026 19:15:48 +0800
Donglin Peng <dolinux.peng@gmail.com> wrote:
> @@ -754,6 +755,14 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
> break;
> case BTF_KIND_ENUM:
> trace_seq_printf(s, "%ld", arg);
> + for_each_enum(i, t, enump) {
> + if (arg == enump->val) {
> + trace_seq_printf(s, " [%s]",
> + btf_name_by_offset(btf,
> + enump->name_off));
> + break;
> + }
> + }
> break;
I have to ask; how big is that enum list?
Do we really want to do a linear search for every enum we come across?
What we could do is for the first time we hit an enum, create an array of
enums, sort them, and do a binary search from then on. This could be saved
in the tracing code itself (in kernel/trace/trace_btf.c), if BPF doesn't
care about it.
-- Steve
On Tue, Feb 3, 2026 at 12:12 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 2 Feb 2026 19:15:48 +0800
> Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> > @@ -754,6 +755,14 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
> > break;
> > case BTF_KIND_ENUM:
> > trace_seq_printf(s, "%ld", arg);
> > + for_each_enum(i, t, enump) {
> > + if (arg == enump->val) {
> > + trace_seq_printf(s, " [%s]",
> > + btf_name_by_offset(btf,
> > + enump->name_off));
> > + break;
> > + }
> > + }
> > break;
>
> I have to ask; how big is that enum list?
Based on local vmlinux analysis (6.18 kernel):
- 78% enums have vlen ≤ 10
- 95% enums have vlen ≤ 50
- Max observed vlen: 349 (extremely rare case)
$ bpftool btf dump file x86_64/vmlinux | grep "^\[" | grep ENUM | grep
-v "(anon)" | awk -F '[ =]' '{
for(i=1; i<=NF; i++) {
if ($i == "vlen") {
count[$(i+1)]++
}
}
}
END {
for (vlen in count) {
print "vlen=" vlen ": " count[vlen] " times"
}
}' | sort -k 2 -n
vlen=104: 1 times
vlen=114: 1 times
vlen=116: 1 times
vlen=129: 1 times
vlen=146: 1 times
vlen=167: 1 times
vlen=181: 1 times
vlen=183: 1 times
vlen=201: 1 times
vlen=213: 1 times
vlen=33: 1 times
vlen=349: 1 times
vlen=42: 1 times
vlen=45: 1 times
vlen=48: 1 times
vlen=50: 1 times
vlen=54: 1 times
vlen=55: 1 times
vlen=56: 1 times
vlen=59: 1 times
vlen=60: 1 times
vlen=61: 1 times
vlen=64: 1 times
vlen=68: 1 times
vlen=71: 1 times
vlen=74: 1 times
vlen=83: 1 times
vlen=85: 1 times
vlen=87: 1 times
vlen=122: 2 times
vlen=38: 2 times
vlen=39: 2 times
vlen=43: 2 times
vlen=44: 2 times
vlen=49: 2 times
vlen=52: 2 times
vlen=58: 2 times
vlen=73: 2 times
vlen=79: 2 times
vlen=35: 3 times
vlen=41: 3 times
vlen=46: 3 times
vlen=25: 4 times
vlen=27: 4 times
vlen=28: 4 times
vlen=29: 4 times
vlen=34: 4 times
vlen=37: 4 times
vlen=40: 4 times
vlen=19: 5 times
vlen=30: 5 times
vlen=20: 6 times
vlen=22: 6 times
vlen=24: 6 times
vlen=16: 8 times
vlen=23: 8 times
vlen=26: 8 times
vlen=31: 8 times
vlen=32: 8 times
vlen=17: 11 times
vlen=18: 12 times
vlen=21: 12 times
vlen=15: 13 times
vlen=14: 21 times
vlen=12: 25 times
vlen=13: 25 times
vlen=10: 34 times
vlen=11: 35 times
vlen=1: 58 times
vlen=8: 70 times
vlen=9: 74 times
vlen=7: 102 times
vlen=6: 108 times
vlen=5: 170 times
vlen=2: 193 times
vlen=4: 250 times
vlen=3: 333 times
>
> Do we really want to do a linear search for every enum we come across?
I think linear search is efficient for typical enum value lists (vlen < 10), I
also support using binary search where feasible.
>
> What we could do is for the first time we hit an enum, create an array of
> enums, sort them, and do a binary search from then on. This could be saved
> in the tracing code itself (in kernel/trace/trace_btf.c), if BPF doesn't
> care about it.
I think this approach adds unnecessary complexity. Instead, we could
leverage btf2btf during the resolve_btfids phase to proactively sort the
btf_enum entries of enum-type btf_type by their value, enabling efficient
binary search lookups.
>
> -- Steve
On Tue, Feb 3, 2026 at 10:17 AM Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> On Tue, Feb 3, 2026 at 12:12 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Mon, 2 Feb 2026 19:15:48 +0800
> > Donglin Peng <dolinux.peng@gmail.com> wrote:
> >
> > > @@ -754,6 +755,14 @@ void print_function_args(struct trace_seq *s, unsigned long *args,
> > > break;
> > > case BTF_KIND_ENUM:
> > > trace_seq_printf(s, "%ld", arg);
> > > + for_each_enum(i, t, enump) {
> > > + if (arg == enump->val) {
> > > + trace_seq_printf(s, " [%s]",
> > > + btf_name_by_offset(btf,
> > > + enump->name_off));
> > > + break;
> > > + }
> > > + }
> > > break;
> >
> > I have to ask; how big is that enum list?
>
> Based on local vmlinux analysis (6.18 kernel):
> - 78% enums have vlen ≤ 10
> - 95% enums have vlen ≤ 50
> - Max observed vlen: 349 (extremely rare case)
>
> $ bpftool btf dump file x86_64/vmlinux | grep "^\[" | grep ENUM | grep
> -v "(anon)" | awk -F '[ =]' '{
> for(i=1; i<=NF; i++) {
> if ($i == "vlen") {
> count[$(i+1)]++
> }
> }
> }
> END {
> for (vlen in count) {
> print "vlen=" vlen ": " count[vlen] " times"
> }
> }' | sort -k 2 -n
> vlen=104: 1 times
> vlen=114: 1 times
> vlen=116: 1 times
> vlen=129: 1 times
> vlen=146: 1 times
> vlen=167: 1 times
> vlen=181: 1 times
> vlen=183: 1 times
> vlen=201: 1 times
> vlen=213: 1 times
> vlen=33: 1 times
> vlen=349: 1 times
> vlen=42: 1 times
> vlen=45: 1 times
> vlen=48: 1 times
> vlen=50: 1 times
> vlen=54: 1 times
> vlen=55: 1 times
> vlen=56: 1 times
> vlen=59: 1 times
> vlen=60: 1 times
> vlen=61: 1 times
> vlen=64: 1 times
> vlen=68: 1 times
> vlen=71: 1 times
> vlen=74: 1 times
> vlen=83: 1 times
> vlen=85: 1 times
> vlen=87: 1 times
> vlen=122: 2 times
> vlen=38: 2 times
> vlen=39: 2 times
> vlen=43: 2 times
> vlen=44: 2 times
> vlen=49: 2 times
> vlen=52: 2 times
> vlen=58: 2 times
> vlen=73: 2 times
> vlen=79: 2 times
> vlen=35: 3 times
> vlen=41: 3 times
> vlen=46: 3 times
> vlen=25: 4 times
> vlen=27: 4 times
> vlen=28: 4 times
> vlen=29: 4 times
> vlen=34: 4 times
> vlen=37: 4 times
> vlen=40: 4 times
> vlen=19: 5 times
> vlen=30: 5 times
> vlen=20: 6 times
> vlen=22: 6 times
> vlen=24: 6 times
> vlen=16: 8 times
> vlen=23: 8 times
> vlen=26: 8 times
> vlen=31: 8 times
> vlen=32: 8 times
> vlen=17: 11 times
> vlen=18: 12 times
> vlen=21: 12 times
> vlen=15: 13 times
> vlen=14: 21 times
> vlen=12: 25 times
> vlen=13: 25 times
> vlen=10: 34 times
> vlen=11: 35 times
> vlen=1: 58 times
> vlen=8: 70 times
> vlen=9: 74 times
> vlen=7: 102 times
> vlen=6: 108 times
> vlen=5: 170 times
> vlen=2: 193 times
> vlen=4: 250 times
> vlen=3: 333 times
>
> >
> > Do we really want to do a linear search for every enum we come across?
>
> I think linear search is efficient for typical enum value lists (vlen < 10), I
> also support using binary search where feasible.
>
> >
> > What we could do is for the first time we hit an enum, create an array of
> > enums, sort them, and do a binary search from then on. This could be saved
> > in the tracing code itself (in kernel/trace/trace_btf.c), if BPF doesn't
> > care about it.
>
> I think this approach adds unnecessary complexity. Instead, we could
> leverage btf2btf during the resolve_btfids phase to proactively sort the
> btf_enum entries of enum-type btf_type by their value, enabling efficient
> binary search lookups.
Testing revealed that sorting within resolve_btfids introduces issues with
btf__dedup. Therefore, I plan to move the sorting logic directly into
btf__add_enum_value and btf__add_enum64_value in libbpf, which are
invoked by pahole. However, it means that we need a newer pahole
version.
>
> >
> > -- Steve
On Tue, 3 Feb 2026 21:50:47 +0800 Donglin Peng <dolinux.peng@gmail.com> wrote: > Testing revealed that sorting within resolve_btfids introduces issues with > btf__dedup. Therefore, I plan to move the sorting logic directly into > btf__add_enum_value and btf__add_enum64_value in libbpf, which are > invoked by pahole. However, it means that we need a newer pahole > version. Sorting isn't a requirement just something I wanted to bring up. If it's too complex and doesn't achieve much benefit then let's not do it. My worry is because "cat trace" takes quite a long time just reading the BTF arguments. I'm worried it will just get worse with enums as well. I have trace-cmd reading BTF now (just haven't officially released it) and doing an extract and reading the trace.dat file is much faster than reading the trace file with arguments. I'll need to implement the enum logic too in libtraceevent. -- Steve
On Tue, Feb 3, 2026 at 11:16 PM Steven Rostedt <rostedt@goodmis.org> wrote: > > On Tue, 3 Feb 2026 21:50:47 +0800 > Donglin Peng <dolinux.peng@gmail.com> wrote: > > > Testing revealed that sorting within resolve_btfids introduces issues with > > btf__dedup. Therefore, I plan to move the sorting logic directly into > > btf__add_enum_value and btf__add_enum64_value in libbpf, which are > > invoked by pahole. However, it means that we need a newer pahole > > version. > > Sorting isn't a requirement just something I wanted to bring up. If it's > too complex and doesn't achieve much benefit then let's not do it. Thanks for clarifying. Analysis shows most enum-type btf_type have small vlen (78% ≤ 10, 95% ≤50), so I think that linear search could be acceptable. > > My worry is because "cat trace" takes quite a long time just reading the > BTF arguments. I'm worried it will just get worse with enums as well. The delay stems from `btf_find_by_name_kind`’s linear search over vmlinux BTF (10k+ types). This was resolved by adding binary search to `btf_find_by_name_kind` [1]. Performance tests [2] confirm the improvement: 1. Original funcgraph-retval: # time cat trace | wc -l 101024 real 0m0.682s user 0m0.000s sys 0m0.695s 2. Enhanced funcgraph-retval: # time cat trace | wc -l 99326 real 0m12.886s user 0m0.010s sys 0m12.680s 3. Enhanced funcgraph-retval + optimizined btf_find_by_name_kind: # time cat trace | wc -l 102922 real 0m0.794s user 0m0.000s sys 0m0.810s Binary search reduces overhead to near-negligible levels (0.794s vs. 12.886s). [1] https://lore.kernel.org/bpf/20260109130003.3313716-1-dolinux.peng@gmail.com/ [2] https://lore.kernel.org/lkml/20251209121349.525641-1-dolinux.peng@gmail.com/ > > I have trace-cmd reading BTF now (just haven't officially released it) and > doing an extract and reading the trace.dat file is much faster than reading > the trace file with arguments. I'll need to implement the enum logic too in > libtraceevent. > > -- Steve
On Tue, Feb 3, 2026 at 7:16 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 3 Feb 2026 21:50:47 +0800
> Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> > Testing revealed that sorting within resolve_btfids introduces issues with
> > btf__dedup. Therefore, I plan to move the sorting logic directly into
> > btf__add_enum_value and btf__add_enum64_value in libbpf, which are
> > invoked by pahole. However, it means that we need a newer pahole
> > version.
>
> Sorting isn't a requirement just something I wanted to bring up. If it's
> too complex and doesn't achieve much benefit then let's not do it.
>
> My worry is because "cat trace" takes quite a long time just reading the
> BTF arguments. I'm worried it will just get worse with enums as well.
>
> I have trace-cmd reading BTF now (just haven't officially released it) and
> doing an extract and reading the trace.dat file is much faster than reading
> the trace file with arguments. I'll need to implement the enum logic too in
> libtraceevent.
If you mean to do pretty printing of the trace in user space then +1 from me.
I don't like sorting enums either in resolve_btfid, pahole or kernel.
Sorted BTF by name was ok, since it doesn't change original semantics.
While sorting enums by value gets us to the grey zone where
the sequence of enum names in vmlinux.h becomes different than in dwarf.
Also id->name mapping in general is not precise.
There is no requirement for enums to be unique.
Just grabbing the first one:
ATA_PIO0 = 1,
ATA_PIO1 = 3,
ATA_PIO2 = 7,
ATA_UDMA0 = 1,
ATA_UDMA1 = 3,
ATA_UDMA2 = 7,
ATA_ID_CYLS = 1,
ATA_ID_HEADS = 3,
SCR_ERROR = 1,
SCR_CONTROL = 2,
SCR_ACTIVE = 3,
All these names are part of the same enum type.
Which one to print? First one?
another example:
enum {
BPF_LOCAL_STORAGE_GET_F_CREATE = 1,
BPF_SK_STORAGE_GET_F_CREATE = 1,
};
and another:
enum {
BPF_SKB_TSTAMP_UNSPEC = 0,
BPF_SKB_TSTAMP_DELIVERY_MONO = 1,
BPF_SKB_CLOCK_REALTIME = 0,
BPF_SKB_CLOCK_MONOTONIC = 1,
BPF_SKB_CLOCK_TAI = 2,
};
I'd rather not print any and keep it integer only.
pw-bot: cr
On Wed, Feb 4, 2026 at 12:00 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Feb 3, 2026 at 7:16 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Tue, 3 Feb 2026 21:50:47 +0800
> > Donglin Peng <dolinux.peng@gmail.com> wrote:
> >
> > > Testing revealed that sorting within resolve_btfids introduces issues with
> > > btf__dedup. Therefore, I plan to move the sorting logic directly into
> > > btf__add_enum_value and btf__add_enum64_value in libbpf, which are
> > > invoked by pahole. However, it means that we need a newer pahole
> > > version.
> >
> > Sorting isn't a requirement just something I wanted to bring up. If it's
> > too complex and doesn't achieve much benefit then let's not do it.
> >
> > My worry is because "cat trace" takes quite a long time just reading the
> > BTF arguments. I'm worried it will just get worse with enums as well.
> >
> > I have trace-cmd reading BTF now (just haven't officially released it) and
> > doing an extract and reading the trace.dat file is much faster than reading
> > the trace file with arguments. I'll need to implement the enum logic too in
> > libtraceevent.
>
> If you mean to do pretty printing of the trace in user space then +1 from me.
>
> I don't like sorting enums either in resolve_btfid, pahole or kernel.
> Sorted BTF by name was ok, since it doesn't change original semantics.
> While sorting enums by value gets us to the grey zone where
> the sequence of enum names in vmlinux.h becomes different than in dwarf.
Thanks, I agreed.
>
> Also id->name mapping in general is not precise.
> There is no requirement for enums to be unique.
> Just grabbing the first one:
> ATA_PIO0 = 1,
> ATA_PIO1 = 3,
> ATA_PIO2 = 7,
> ATA_UDMA0 = 1,
> ATA_UDMA1 = 3,
> ATA_UDMA2 = 7,
> ATA_ID_CYLS = 1,
> ATA_ID_HEADS = 3,
> SCR_ERROR = 1,
> SCR_CONTROL = 2,
> SCR_ACTIVE = 3,
>
> All these names are part of the same enum type.
> Which one to print? First one?
I think these cases are not very common and printing the first
one would be helpful enough, and we can add documentation
notes in ftrace to guide users.
>
> another example:
> enum {
> BPF_LOCAL_STORAGE_GET_F_CREATE = 1,
> BPF_SK_STORAGE_GET_F_CREATE = 1,
> };
>
> and another:
> enum {
> BPF_SKB_TSTAMP_UNSPEC = 0,
> BPF_SKB_TSTAMP_DELIVERY_MONO = 1,
> BPF_SKB_CLOCK_REALTIME = 0,
> BPF_SKB_CLOCK_MONOTONIC = 1,
> BPF_SKB_CLOCK_TAI = 2,
> };
>
> I'd rather not print any and keep it integer only.
>
> pw-bot: cr
On Wed, 4 Feb 2026 22:52:11 +0800
Donglin Peng <dolinux.peng@gmail.com> wrote:
> > > I have trace-cmd reading BTF now (just haven't officially released it) and
> > > doing an extract and reading the trace.dat file is much faster than reading
> > > the trace file with arguments. I'll need to implement the enum logic too in
> > > libtraceevent.
> >
> > If you mean to do pretty printing of the trace in user space then +1 from me.
> >
> > I don't like sorting enums either in resolve_btfid, pahole or kernel.
> > Sorted BTF by name was ok, since it doesn't change original semantics.
> > While sorting enums by value gets us to the grey zone where
> > the sequence of enum names in vmlinux.h becomes different than in dwarf.
>
> Thanks, I agreed.
BTW, I just officially released trace-cmd v3.4 (where you can see whats
new in that release here[1]).
The biggest change is that it saves the BTF file in the trace.dat file
and parses it on the report (it requires libtraceevent v1.9):
~# trace-cmd record -p function_graph -O funcgraph-args -g do_sys_openat2
[..]
~# trace-cmd report
trace-cmd-50935 [002] ...1. 3490.518138: funcgraph_entry: | do_sys_openat2(dfd=4294967196, filename=0x557bb9e3ee10, how=0xffff88815220fea8) {
trace-cmd-50935 [002] ...1. 3490.518141: funcgraph_entry: | getname_flags(filename=0x557bb9e3ee10, flags=0) {
trace-cmd-50935 [002] ...1. 3490.518142: funcgraph_entry: | getname_flags.part.0(filename=0x557bb9e3ee10, flags=0) {
trace-cmd-50935 [002] ...1. 3490.518143: funcgraph_entry: | kmem_cache_alloc_noprof(s=0xffff888106c7e000, gfpflags=0xcc0) {
trace-cmd-50935 [002] ...1. 3490.518145: funcgraph_entry: | stack_trace_save(store=0xffff88815220fac8, size=0x40, skipnr=0x0) {
trace-cmd-50935 [002] ...1. 3490.518147: funcgraph_entry: | arch_stack_walk(consume_entry=0xffffffff94d9dfe0, cookie=0xffff88815220fa58, task=0xffff88812d4c3580, regs=0x0) {
trace-cmd-50935 [002] ...1. 3490.518148: funcgraph_entry: | __unwind_start(state=0xffff88815220f988, task=0xffff88812d4c3580, regs=0x0, first_frame=0xffff88815220fa28) {
trace-cmd-50935 [002] ...1. 3490.518149: funcgraph_entry: 1.518 us | get_stack_info(stack=0xffff88815220f938, task=0xffff88812d4c3580, info=0xffff88815220f988, visit_mask=0xffff88815220f9a8); (ret=0x0)
trace-cmd-50935 [002] ...1. 3490.518152: funcgraph_entry: | unwind_next_frame(state=0xffff88815220f988) {
trace-cmd-50935 [002] ...1. 3490.518153: funcgraph_entry: 0.951 us | __rcu_read_lock(); (ret=0xffff88812d4c3580)
-- Steve
[1] https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/tag/?h=trace-cmd-v3.4
On Fri, Feb 6, 2026 at 2:05 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 4 Feb 2026 22:52:11 +0800
> Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> > > > I have trace-cmd reading BTF now (just haven't officially released it) and
> > > > doing an extract and reading the trace.dat file is much faster than reading
> > > > the trace file with arguments. I'll need to implement the enum logic too in
> > > > libtraceevent.
> > >
> > > If you mean to do pretty printing of the trace in user space then +1 from me.
> > >
> > > I don't like sorting enums either in resolve_btfid, pahole or kernel.
> > > Sorted BTF by name was ok, since it doesn't change original semantics.
> > > While sorting enums by value gets us to the grey zone where
> > > the sequence of enum names in vmlinux.h becomes different than in dwarf.
> >
> > Thanks, I agreed.
>
> BTW, I just officially released trace-cmd v3.4 (where you can see whats
> new in that release here[1]).
>
> The biggest change is that it saves the BTF file in the trace.dat file
> and parses it on the report (it requires libtraceevent v1.9):
>
> ~# trace-cmd record -p function_graph -O funcgraph-args -g do_sys_openat2
> [..]
> ~# trace-cmd report
> trace-cmd-50935 [002] ...1. 3490.518138: funcgraph_entry: | do_sys_openat2(dfd=4294967196, filename=0x557bb9e3ee10, how=0xffff88815220fea8) {
> trace-cmd-50935 [002] ...1. 3490.518141: funcgraph_entry: | getname_flags(filename=0x557bb9e3ee10, flags=0) {
> trace-cmd-50935 [002] ...1. 3490.518142: funcgraph_entry: | getname_flags.part.0(filename=0x557bb9e3ee10, flags=0) {
> trace-cmd-50935 [002] ...1. 3490.518143: funcgraph_entry: | kmem_cache_alloc_noprof(s=0xffff888106c7e000, gfpflags=0xcc0) {
> trace-cmd-50935 [002] ...1. 3490.518145: funcgraph_entry: | stack_trace_save(store=0xffff88815220fac8, size=0x40, skipnr=0x0) {
> trace-cmd-50935 [002] ...1. 3490.518147: funcgraph_entry: | arch_stack_walk(consume_entry=0xffffffff94d9dfe0, cookie=0xffff88815220fa58, task=0xffff88812d4c3580, regs=0x0) {
> trace-cmd-50935 [002] ...1. 3490.518148: funcgraph_entry: | __unwind_start(state=0xffff88815220f988, task=0xffff88812d4c3580, regs=0x0, first_frame=0xffff88815220fa28) {
> trace-cmd-50935 [002] ...1. 3490.518149: funcgraph_entry: 1.518 us | get_stack_info(stack=0xffff88815220f938, task=0xffff88812d4c3580, info=0xffff88815220f988, visit_mask=0xffff88815220f9a8); (ret=0x0)
> trace-cmd-50935 [002] ...1. 3490.518152: funcgraph_entry: | unwind_next_frame(state=0xffff88815220f988) {
> trace-cmd-50935 [002] ...1. 3490.518153: funcgraph_entry: 0.951 us | __rcu_read_lock(); (ret=0xffff88812d4c3580)
Thanks, I will implement this feature in libtraceevent.
>
> -- Steve
>
>
>
> [1] https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/tag/?h=trace-cmd-v3.4
On Wed, Feb 4, 2026 at 10:52 PM Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> On Wed, Feb 4, 2026 at 12:00 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Tue, Feb 3, 2026 at 7:16 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > On Tue, 3 Feb 2026 21:50:47 +0800
> > > Donglin Peng <dolinux.peng@gmail.com> wrote:
> > >
> > > > Testing revealed that sorting within resolve_btfids introduces issues with
> > > > btf__dedup. Therefore, I plan to move the sorting logic directly into
> > > > btf__add_enum_value and btf__add_enum64_value in libbpf, which are
> > > > invoked by pahole. However, it means that we need a newer pahole
> > > > version.
> > >
> > > Sorting isn't a requirement just something I wanted to bring up. If it's
> > > too complex and doesn't achieve much benefit then let's not do it.
> > >
> > > My worry is because "cat trace" takes quite a long time just reading the
> > > BTF arguments. I'm worried it will just get worse with enums as well.
> > >
> > > I have trace-cmd reading BTF now (just haven't officially released it) and
> > > doing an extract and reading the trace.dat file is much faster than reading
> > > the trace file with arguments. I'll need to implement the enum logic too in
> > > libtraceevent.
> >
> > If you mean to do pretty printing of the trace in user space then +1 from me.
> >
> > I don't like sorting enums either in resolve_btfid, pahole or kernel.
> > Sorted BTF by name was ok, since it doesn't change original semantics.
> > While sorting enums by value gets us to the grey zone where
> > the sequence of enum names in vmlinux.h becomes different than in dwarf.
>
> Thanks, I agreed.
>
> >
> > Also id->name mapping in general is not precise.
> > There is no requirement for enums to be unique.
> > Just grabbing the first one:
> > ATA_PIO0 = 1,
> > ATA_PIO1 = 3,
> > ATA_PIO2 = 7,
> > ATA_UDMA0 = 1,
> > ATA_UDMA1 = 3,
> > ATA_UDMA2 = 7,
> > ATA_ID_CYLS = 1,
> > ATA_ID_HEADS = 3,
> > SCR_ERROR = 1,
> > SCR_CONTROL = 2,
> > SCR_ACTIVE = 3,
> >
> > All these names are part of the same enum type.
> > Which one to print? First one?
Another option is to print all matching entries, incurring increased
overhead and extended trace log length. However, I prefer printing
the first matching entry, though it might be inaccurate in rare cases.
>
> I think these cases are not very common and printing the first
> one would be helpful enough, and we can add documentation
> notes in ftrace to guide users.
>
> >
> > another example:
> > enum {
> > BPF_LOCAL_STORAGE_GET_F_CREATE = 1,
> > BPF_SK_STORAGE_GET_F_CREATE = 1,
> > };
> >
> > and another:
> > enum {
> > BPF_SKB_TSTAMP_UNSPEC = 0,
> > BPF_SKB_TSTAMP_DELIVERY_MONO = 1,
> > BPF_SKB_CLOCK_REALTIME = 0,
> > BPF_SKB_CLOCK_MONOTONIC = 1,
> > BPF_SKB_CLOCK_TAI = 2,
> > };
> >
> > I'd rather not print any and keep it integer only.
> >
> > pw-bot: cr
On Thu, Feb 5, 2026 at 1:21 AM Donglin Peng <dolinux.peng@gmail.com> wrote: > > On Wed, Feb 4, 2026 at 10:52 PM Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > On Wed, Feb 4, 2026 at 12:00 AM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > > > > On Tue, Feb 3, 2026 at 7:16 AM Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > > > > On Tue, 3 Feb 2026 21:50:47 +0800 > > > > Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > > > > > > Testing revealed that sorting within resolve_btfids introduces issues with > > > > > btf__dedup. Therefore, I plan to move the sorting logic directly into > > > > > btf__add_enum_value and btf__add_enum64_value in libbpf, which are > > > > > invoked by pahole. However, it means that we need a newer pahole > > > > > version. > > > > > > > > Sorting isn't a requirement just something I wanted to bring up. If it's > > > > too complex and doesn't achieve much benefit then let's not do it. > > > > > > > > My worry is because "cat trace" takes quite a long time just reading the > > > > BTF arguments. I'm worried it will just get worse with enums as well. > > > > > > > > I have trace-cmd reading BTF now (just haven't officially released it) and > > > > doing an extract and reading the trace.dat file is much faster than reading > > > > the trace file with arguments. I'll need to implement the enum logic too in > > > > libtraceevent. > > > > > > If you mean to do pretty printing of the trace in user space then +1 from me. > > > > > > I don't like sorting enums either in resolve_btfid, pahole or kernel. > > > Sorted BTF by name was ok, since it doesn't change original semantics. > > > While sorting enums by value gets us to the grey zone where > > > the sequence of enum names in vmlinux.h becomes different than in dwarf. > > > > Thanks, I agreed. > > > > > > > > Also id->name mapping in general is not precise. > > > There is no requirement for enums to be unique. > > > Just grabbing the first one: > > > ATA_PIO0 = 1, > > > ATA_PIO1 = 3, > > > ATA_PIO2 = 7, > > > ATA_UDMA0 = 1, > > > ATA_UDMA1 = 3, > > > ATA_UDMA2 = 7, > > > ATA_ID_CYLS = 1, > > > ATA_ID_HEADS = 3, > > > SCR_ERROR = 1, > > > SCR_CONTROL = 2, > > > SCR_ACTIVE = 3, > > > > > > All these names are part of the same enum type. > > > Which one to print? First one? > > Another option is to print all matching entries, incurring increased > overhead and extended trace log length. However, I prefer printing > the first matching entry, though it might be inaccurate in rare cases. I disagree. It's not rare. I wouldn't print anything. Let user space deal with it.
On Thu, Feb 5, 2026 at 11:56 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Thu, Feb 5, 2026 at 1:21 AM Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > On Wed, Feb 4, 2026 at 10:52 PM Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > > > On Wed, Feb 4, 2026 at 12:00 AM Alexei Starovoitov > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > On Tue, Feb 3, 2026 at 7:16 AM Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > > > > > > On Tue, 3 Feb 2026 21:50:47 +0800 > > > > > Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > > > > > > > > Testing revealed that sorting within resolve_btfids introduces issues with > > > > > > btf__dedup. Therefore, I plan to move the sorting logic directly into > > > > > > btf__add_enum_value and btf__add_enum64_value in libbpf, which are > > > > > > invoked by pahole. However, it means that we need a newer pahole > > > > > > version. > > > > > > > > > > Sorting isn't a requirement just something I wanted to bring up. If it's > > > > > too complex and doesn't achieve much benefit then let's not do it. > > > > > > > > > > My worry is because "cat trace" takes quite a long time just reading the > > > > > BTF arguments. I'm worried it will just get worse with enums as well. > > > > > > > > > > I have trace-cmd reading BTF now (just haven't officially released it) and > > > > > doing an extract and reading the trace.dat file is much faster than reading > > > > > the trace file with arguments. I'll need to implement the enum logic too in > > > > > libtraceevent. > > > > > > > > If you mean to do pretty printing of the trace in user space then +1 from me. > > > > > > > > I don't like sorting enums either in resolve_btfid, pahole or kernel. > > > > Sorted BTF by name was ok, since it doesn't change original semantics. > > > > While sorting enums by value gets us to the grey zone where > > > > the sequence of enum names in vmlinux.h becomes different than in dwarf. > > > > > > Thanks, I agreed. > > > > > > > > > > > Also id->name mapping in general is not precise. > > > > There is no requirement for enums to be unique. > > > > Just grabbing the first one: > > > > ATA_PIO0 = 1, > > > > ATA_PIO1 = 3, > > > > ATA_PIO2 = 7, > > > > ATA_UDMA0 = 1, > > > > ATA_UDMA1 = 3, > > > > ATA_UDMA2 = 7, > > > > ATA_ID_CYLS = 1, > > > > ATA_ID_HEADS = 3, > > > > SCR_ERROR = 1, > > > > SCR_CONTROL = 2, > > > > SCR_ACTIVE = 3, > > > > > > > > All these names are part of the same enum type. > > > > Which one to print? First one? > > > > Another option is to print all matching entries, incurring increased > > overhead and extended trace log length. However, I prefer printing > > the first matching entry, though it might be inaccurate in rare cases. > > I disagree. It's not rare. > I wouldn't print anything. Let user space deal with it. Okay, I will implement this in libtraceevent first. By the way, would the first patch [1] introducing the for_each_enumand for_each_enum64 helper macros be acceptable? [1] https://lore.kernel.org/lkml/20260202111548.3555306-2-dolinux.peng@gmail.com/
On Thu, Feb 5, 2026 at 8:10 PM Donglin Peng <dolinux.peng@gmail.com> wrote: > > On Thu, Feb 5, 2026 at 11:56 PM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Thu, Feb 5, 2026 at 1:21 AM Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > > > On Wed, Feb 4, 2026 at 10:52 PM Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > > > > > On Wed, Feb 4, 2026 at 12:00 AM Alexei Starovoitov > > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > > > On Tue, Feb 3, 2026 at 7:16 AM Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > > > > > > > > On Tue, 3 Feb 2026 21:50:47 +0800 > > > > > > Donglin Peng <dolinux.peng@gmail.com> wrote: > > > > > > > > > > > > > Testing revealed that sorting within resolve_btfids introduces issues with > > > > > > > btf__dedup. Therefore, I plan to move the sorting logic directly into > > > > > > > btf__add_enum_value and btf__add_enum64_value in libbpf, which are > > > > > > > invoked by pahole. However, it means that we need a newer pahole > > > > > > > version. > > > > > > > > > > > > Sorting isn't a requirement just something I wanted to bring up. If it's > > > > > > too complex and doesn't achieve much benefit then let's not do it. > > > > > > > > > > > > My worry is because "cat trace" takes quite a long time just reading the > > > > > > BTF arguments. I'm worried it will just get worse with enums as well. > > > > > > > > > > > > I have trace-cmd reading BTF now (just haven't officially released it) and > > > > > > doing an extract and reading the trace.dat file is much faster than reading > > > > > > the trace file with arguments. I'll need to implement the enum logic too in > > > > > > libtraceevent. > > > > > > > > > > If you mean to do pretty printing of the trace in user space then +1 from me. > > > > > > > > > > I don't like sorting enums either in resolve_btfid, pahole or kernel. > > > > > Sorted BTF by name was ok, since it doesn't change original semantics. > > > > > While sorting enums by value gets us to the grey zone where > > > > > the sequence of enum names in vmlinux.h becomes different than in dwarf. > > > > > > > > Thanks, I agreed. > > > > > > > > > > > > > > Also id->name mapping in general is not precise. > > > > > There is no requirement for enums to be unique. > > > > > Just grabbing the first one: > > > > > ATA_PIO0 = 1, > > > > > ATA_PIO1 = 3, > > > > > ATA_PIO2 = 7, > > > > > ATA_UDMA0 = 1, > > > > > ATA_UDMA1 = 3, > > > > > ATA_UDMA2 = 7, > > > > > ATA_ID_CYLS = 1, > > > > > ATA_ID_HEADS = 3, > > > > > SCR_ERROR = 1, > > > > > SCR_CONTROL = 2, > > > > > SCR_ACTIVE = 3, > > > > > > > > > > All these names are part of the same enum type. > > > > > Which one to print? First one? > > > > > > Another option is to print all matching entries, incurring increased > > > overhead and extended trace log length. However, I prefer printing > > > the first matching entry, though it might be inaccurate in rare cases. > > > > I disagree. It's not rare. > > I wouldn't print anything. Let user space deal with it. > > Okay, I will implement this in libtraceevent first. By the way, would the first > patch [1] introducing the for_each_enumand for_each_enum64 helper > macros be acceptable? > > [1] https://lore.kernel.org/lkml/20260202111548.3555306-2-dolinux.peng@gmail.com/ Just that patch alone? What's the point? Refactor for what? Does it read better? No.
© 2016 - 2026 Red Hat, Inc.