From nobody Tue Dec 2 02:06:08 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9F6730B525; Thu, 20 Nov 2025 23:48:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763682496; cv=none; b=VrwOLI7wsd+/di2sJWib+jjwoW0SRRTGaANzckfVCINXLxHhacOm2YWZ9hatO+jE13pLCODdkdbP2zUv2ABwwQik1zJjIazBUaPinbTaGjbkBLt7bA3B4l8jtEUUaBJ7vaDq/JB4YJiCe2nElVHA5KXMYT6DBEQsDEUPBg9h+F8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763682496; c=relaxed/simple; bh=ftkesYbwPx4XJAoR1xc+HVyQKo51smR6BoXfYy+LCs8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZJiDX+Nud0R4I8CYzmKJNxW+SUjth1noBbI/YVpoQs3jioOlKf64APQ5L8EQSK+8ytJ+xqyz3Rimhr7GjlvEfdYoW3DUJ9u5CWaq+c4tkt9xax3e5jbRW6DIf5wLBuHYcRNiosXUg3uiCjVq9gqOtc73IE2kfbA1p2Jsz4KqvqU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oOw2vU/k; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oOw2vU/k" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F20BEC19422; Thu, 20 Nov 2025 23:48:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763682496; bh=ftkesYbwPx4XJAoR1xc+HVyQKo51smR6BoXfYy+LCs8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oOw2vU/k3mTFEfqKZbJtEcQxAF4axO20TyXMMrtrA5+CnEnl6KtpUG30rULde53mU wU7J9r+xNTjs5jReOX9DaChaCYA90R3Rt4Ogvph/ryeQD24uDAw8b4IWsjoaDnJR2z vuGJpytMRfaSaImBSo+1RMgxIxG9YBxTICzQqqzwMMOOTirGu2sE3MFG/OXwcKxJER iAh0anKAxuiwLRtpD4ggWgWVr5bSFb6W1u/kyIfCslmcHcCGC+oB+vbWwgVI8yTcXZ d6xQwaREJV0oepWo777ce97n19pQYWiYgESaMrXTvJBafgNRqXZZNObJZxXP1zvbkj oPuEb+L0V51kw== From: Namhyung Kim To: Arnaldo Carvalho de Melo , Ian Rogers , James Clark Cc: Jiri Olsa , Adrian Hunter , Peter Zijlstra , Ingo Molnar , LKML , linux-perf-users@vger.kernel.org, Steven Rostedt , Josh Poimboeuf , Indu Bhagat , Jens Remus , Mathieu Desnoyers , linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v6 5/6] perf tools: Merge deferred user callchains Date: Thu, 20 Nov 2025 15:48:03 -0800 Message-ID: <20251120234804.156340-6-namhyung@kernel.org> X-Mailer: git-send-email 2.52.0.rc2.455.g230fcf2819-goog In-Reply-To: <20251120234804.156340-1-namhyung@kernel.org> References: <20251120234804.156340-1-namhyung@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Save samples with deferred callchains in a separate list and deliver them after merging the user callchains. If users don't want to merge they can set tool->merge_deferred_callchains to false to prevent the behavior. With previous result, now perf script will show the merged callchains. $ perf script ... pwd 2312 121.163435: 249113 cpu/cycles/P: ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms]) ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms]) ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms]) ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms]) ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms]) ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms]) ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kal= lsyms]) 7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86= -64.so.2) 7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-= linux-x86-64.so.2) 7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-lin= ux-x86-64.so.2) ... The old output can be get using --no-merge-callchain option. Also perf report can get the user callchain entry at the end. $ perf report --no-children --stdio -q -S __build_id_parse.isra.0 # symbol: __build_id_parse.isra.0 8.40% pwd [kernel.kallsyms] | ---__build_id_parse.isra.0 perf_event_mmap mprotect_fixup do_mprotect_pkey __x64_sys_mprotect do_syscall_64 entry_SYSCALL_64_after_hwframe mprotect _dl_sysdep_start _dl_start_user Signed-off-by: Namhyung Kim --- tools/perf/Documentation/perf-script.txt | 5 ++ tools/perf/builtin-inject.c | 1 + tools/perf/builtin-report.c | 1 + tools/perf/builtin-script.c | 4 ++ tools/perf/util/callchain.c | 29 +++++++++ tools/perf/util/callchain.h | 3 + tools/perf/util/evlist.c | 1 + tools/perf/util/evlist.h | 2 + tools/perf/util/session.c | 79 +++++++++++++++++++++++- tools/perf/util/tool.c | 2 + tools/perf/util/tool.h | 1 + 11 files changed, 127 insertions(+), 1 deletion(-) diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Document= ation/perf-script.txt index 28bec7e78bc858ba..03d1129606328d6d 100644 --- a/tools/perf/Documentation/perf-script.txt +++ b/tools/perf/Documentation/perf-script.txt @@ -527,6 +527,11 @@ include::itrace.txt[] The known limitations include exception handing such as setjmp/longjmp will have calls/returns not match. =20 +--merge-callchains:: + Enable merging deferred user callchains if available. This is the + default behavior. If you want to see separate CALLCHAIN_DEFERRED + records for some reason, use --no-merge-callchains explicitly. + :GMEXAMPLECMD: script :GMEXAMPLESUBCMD: include::guest-files.txt[] diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c index bd9245d2dd41aa48..51d2721b6db9dccb 100644 --- a/tools/perf/builtin-inject.c +++ b/tools/perf/builtin-inject.c @@ -2527,6 +2527,7 @@ int cmd_inject(int argc, const char **argv) inject.tool.auxtrace =3D perf_event__repipe_auxtrace; inject.tool.bpf_metadata =3D perf_event__repipe_op2_synth; inject.tool.dont_split_sample_group =3D true; + inject.tool.merge_deferred_callchains =3D false; inject.session =3D __perf_session__new(&data, &inject.tool, /*trace_event_repipe=3D*/inject.output.is_pipe, /*host_env=3D*/NULL); diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c index 2bc269f5fcef8023..add6b1c2aaf04270 100644 --- a/tools/perf/builtin-report.c +++ b/tools/perf/builtin-report.c @@ -1614,6 +1614,7 @@ int cmd_report(int argc, const char **argv) report.tool.event_update =3D perf_event__process_event_update; report.tool.feature =3D process_feature_event; report.tool.ordering_requires_timestamps =3D true; + report.tool.merge_deferred_callchains =3D !dump_trace; =20 session =3D perf_session__new(&data, &report.tool); if (IS_ERR(session)) { diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c index 85b42205a71b3993..62e43d3c5ad731a0 100644 --- a/tools/perf/builtin-script.c +++ b/tools/perf/builtin-script.c @@ -4009,6 +4009,7 @@ int cmd_script(int argc, const char **argv) bool header_only =3D false; bool script_started =3D false; bool unsorted_dump =3D false; + bool merge_deferred_callchains =3D true; char *rec_script_path =3D NULL; char *rep_script_path =3D NULL; struct perf_session *session; @@ -4162,6 +4163,8 @@ int cmd_script(int argc, const char **argv) "Guest code can be found in hypervisor process"), OPT_BOOLEAN('\0', "stitch-lbr", &script.stitch_lbr, "Enable LBR callgraph stitching approach"), + OPT_BOOLEAN('\0', "merge-callchains", &merge_deferred_callchains, + "Enable merge deferred user callchains"), OPTS_EVSWITCH(&script.evswitch), OPT_END() }; @@ -4418,6 +4421,7 @@ int cmd_script(int argc, const char **argv) script.tool.throttle =3D process_throttle_event; script.tool.unthrottle =3D process_throttle_event; script.tool.ordering_requires_timestamps =3D true; + script.tool.merge_deferred_callchains =3D merge_deferred_callchains; session =3D perf_session__new(&data, &script.tool); if (IS_ERR(session)) return PTR_ERR(session); diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c index 2884187ccbbecfdc..71dc5a070065dd2a 100644 --- a/tools/perf/util/callchain.c +++ b/tools/perf/util/callchain.c @@ -1838,3 +1838,32 @@ int sample__for_each_callchain_node(struct thread *t= hread, struct evsel *evsel, } return 0; } + +int sample__merge_deferred_callchain(struct perf_sample *sample_orig, + struct perf_sample *sample_callchain) +{ + u64 nr_orig =3D sample_orig->callchain->nr - 1; + u64 nr_deferred =3D sample_callchain->callchain->nr; + struct ip_callchain *callchain; + + if (sample_orig->callchain->nr < 2) { + sample_orig->deferred_callchain =3D false; + return -EINVAL; + } + + callchain =3D calloc(1 + nr_orig + nr_deferred, sizeof(u64)); + if (callchain =3D=3D NULL) { + sample_orig->deferred_callchain =3D false; + return -ENOMEM; + } + + callchain->nr =3D nr_orig + nr_deferred; + /* copy original including PERF_CONTEXT_USER_DEFERRED (but the cookie) */ + memcpy(callchain->ips, sample_orig->callchain->ips, nr_orig * sizeof(u64)= ); + /* copy deferred user callchains */ + memcpy(&callchain->ips[nr_orig], sample_callchain->callchain->ips, + nr_deferred * sizeof(u64)); + + sample_orig->callchain =3D callchain; + return 0; +} diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h index d5ae4fbb7ce5fa44..2a52af8c80ace33c 100644 --- a/tools/perf/util/callchain.h +++ b/tools/perf/util/callchain.h @@ -318,4 +318,7 @@ int sample__for_each_callchain_node(struct thread *thre= ad, struct evsel *evsel, struct perf_sample *sample, int max_stack, bool symbols, callchain_iter_fn cb, void *data); =20 +int sample__merge_deferred_callchain(struct perf_sample *sample_orig, + struct perf_sample *sample_callchain); + #endif /* __PERF_CALLCHAIN_H */ diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c index e8217efdda5323c6..03674d2cbd015e4f 100644 --- a/tools/perf/util/evlist.c +++ b/tools/perf/util/evlist.c @@ -85,6 +85,7 @@ void evlist__init(struct evlist *evlist, struct perf_cpu_= map *cpus, evlist->ctl_fd.pos =3D -1; evlist->nr_br_cntr =3D -1; metricgroup__rblist_init(&evlist->metric_events); + INIT_LIST_HEAD(&evlist->deferred_samples); } =20 struct evlist *evlist__new(void) diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h index 5e71e3dc60423079..911834ae7c2a6f76 100644 --- a/tools/perf/util/evlist.h +++ b/tools/perf/util/evlist.h @@ -92,6 +92,8 @@ struct evlist { * of struct metric_expr. */ struct rblist metric_events; + /* samples with deferred_callchain would wait here. */ + struct list_head deferred_samples; }; =20 struct evsel_str_handler { diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c index 361e15c1f26a96d0..dc570ad47ccc2c63 100644 --- a/tools/perf/util/session.c +++ b/tools/perf/util/session.c @@ -1285,6 +1285,66 @@ static int evlist__deliver_sample(struct evlist *evl= ist, const struct perf_tool per_thread); } =20 +/* + * Samples with deferred callchains should wait for the next matching + * PERF_RECORD_CALLCHAIN_RECORD entries. Keep the events in a list and + * deliver them once it finds the callchains. + */ +struct deferred_event { + struct list_head list; + union perf_event *event; +}; + +static int evlist__deliver_deferred_callchain(struct evlist *evlist, + const struct perf_tool *tool, + union perf_event *event, + struct perf_sample *sample, + struct machine *machine) +{ + struct deferred_event *de, *tmp; + struct evsel *evsel; + int ret =3D 0; + + if (!tool->merge_deferred_callchains) { + evsel =3D evlist__id2evsel(evlist, sample->id); + return tool->callchain_deferred(tool, event, sample, + evsel, machine); + } + + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) { + struct perf_sample orig_sample; + + ret =3D evlist__parse_sample(evlist, de->event, &orig_sample); + if (ret < 0) { + pr_err("failed to parse original sample\n"); + break; + } + + if (sample->tid !=3D orig_sample.tid) + continue; + + if (event->callchain_deferred.cookie =3D=3D orig_sample.deferred_cookie) + sample__merge_deferred_callchain(&orig_sample, sample); + else + orig_sample.deferred_callchain =3D false; + + evsel =3D evlist__id2evsel(evlist, orig_sample.id); + ret =3D evlist__deliver_sample(evlist, tool, de->event, + &orig_sample, evsel, machine); + + if (orig_sample.deferred_callchain) + free(orig_sample.callchain); + + list_del(&de->list); + free(de->event); + free(de); + + if (ret) + break; + } + return ret; +} + static int machines__deliver_event(struct machines *machines, struct evlist *evlist, union perf_event *event, @@ -1313,6 +1373,22 @@ static int machines__deliver_event(struct machines *= machines, return 0; } dump_sample(evsel, event, sample, perf_env__arch(machine->env)); + if (sample->deferred_callchain && tool->merge_deferred_callchains) { + struct deferred_event *de =3D malloc(sizeof(*de)); + size_t sz =3D event->header.size; + + if (de =3D=3D NULL) + return -ENOMEM; + + de->event =3D malloc(sz); + if (de->event =3D=3D NULL) { + free(de); + return -ENOMEM; + } + memcpy(de->event, event, sz); + list_add_tail(&de->list, &evlist->deferred_samples); + return 0; + } return evlist__deliver_sample(evlist, tool, event, sample, evsel, machin= e); case PERF_RECORD_MMAP: return tool->mmap(tool, event, sample, machine); @@ -1372,7 +1448,8 @@ static int machines__deliver_event(struct machines *m= achines, return tool->aux_output_hw_id(tool, event, sample, machine); case PERF_RECORD_CALLCHAIN_DEFERRED: dump_deferred_callchain(evsel, event, sample); - return tool->callchain_deferred(tool, event, sample, evsel, machine); + return evlist__deliver_deferred_callchain(evlist, tool, event, + sample, machine); default: ++evlist->stats.nr_unknown_events; return -1; diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c index e77f0e2ecc1f79db..27ba5849c74a2e7d 100644 --- a/tools/perf/util/tool.c +++ b/tools/perf/util/tool.c @@ -266,6 +266,7 @@ void perf_tool__init(struct perf_tool *tool, bool order= ed_events) tool->cgroup_events =3D false; tool->no_warn =3D false; tool->show_feat_hdr =3D SHOW_FEAT_NO_HEADER; + tool->merge_deferred_callchains =3D true; =20 tool->sample =3D process_event_sample_stub; tool->mmap =3D process_event_stub; @@ -448,6 +449,7 @@ void delegate_tool__init(struct delegate_tool *tool, st= ruct perf_tool *delegate) tool->tool.cgroup_events =3D delegate->cgroup_events; tool->tool.no_warn =3D delegate->no_warn; tool->tool.show_feat_hdr =3D delegate->show_feat_hdr; + tool->tool.merge_deferred_callchains =3D delegate->merge_deferred_callcha= ins; =20 tool->tool.sample =3D delegate_sample; tool->tool.read =3D delegate_read; diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h index 9b9f0a8cbf3de4b5..e96b69d25a5b737d 100644 --- a/tools/perf/util/tool.h +++ b/tools/perf/util/tool.h @@ -90,6 +90,7 @@ struct perf_tool { bool cgroup_events; bool no_warn; bool dont_split_sample_group; + bool merge_deferred_callchains; enum show_feature_header show_feat_hdr; }; =20 --=20 2.52.0.rc2.455.g230fcf2819-goog