From nobody Sun Feb 8 05:28:59 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC775246768; Sat, 15 Nov 2025 23:41:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250068; cv=none; b=i0srw5YlLpxhxlj76mAHn8xOI1Q962su+1m1pmigOLNr/4XxTwlXObuRy3xMFKx9EelSi3aMYJg17O//W7j5UHW6q9++Mexi0lz8s2V/8voCBg+cqQCkl/RgN7z8LHo/2eUpKDDyJFYhwExVlBfRB4IT0a5QR004hejQCtb7jXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250068; c=relaxed/simple; bh=ojTHLSjZBXCP+lYeZxCWYORNJqxatfl7RNTwhIJgLEE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qeiOvHgchC8X7RuzlnoqZEMFn6XB8VL4xw/A4Nl/QRzCb1iqj151wJPdZREGzN5s7BhXEFL3ZgBvGuV4hoY3pifYYt0Z2aBwvMtK0v9wCi4DxPkzqfCPbjn2pG3oGROC6/58/m1NG5WhLT2zcGA39gvSTQ5836SiF7llWvdgpcE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iAMtqO99; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iAMtqO99" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5B9DC4AF1D; Sat, 15 Nov 2025 23:41:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763250068; bh=ojTHLSjZBXCP+lYeZxCWYORNJqxatfl7RNTwhIJgLEE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iAMtqO99TRakrBOkqn1IBcz8FDBT5X190PUoX0JEj84BoCOQIY9Uyh/v+91mhHBr3 dFaGiAQYPuJTigaJcDPPecRWQv5M4SpZQT2p/cF1csGiOIVsafXa547UdsO9fOuYOI 4zNkWGoRbldcMJwnO6wQdhqX0A42AT83Al36aegmDO6SCTl0lOhNPUy4dKbE2Pt5FA VnMW3WgBNmuu7A6rr6Meli3MEm1AB4d8mBF/6nfojmeY+1DvqBZpcpsI606pYaxv+A a5UwYtwcRHp3Z+UCxmyCl2Pwy/ycNm4IUBodK1mm7d7wEISPfBIkG+9HU5FJZWEIvF 1VjjD5/eKxNvA== From: Namhyung Kim To: Arnaldo Carvalho de Melo , Ian Rogers , James Clark Cc: Jiri Olsa , Adrian Hunter , Peter Zijlstra , Ingo Molnar , LKML , linux-perf-users@vger.kernel.org, Steven Rostedt , Josh Poimboeuf , Indu Bhagat , Jens Remus , Mathieu Desnoyers , linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v4 1/5] tools headers UAPI: Sync linux/perf_event.h for deferred callchains Date: Sat, 15 Nov 2025 15:41:02 -0800 Message-ID: <20251115234106.348571-2-namhyung@kernel.org> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog In-Reply-To: <20251115234106.348571-1-namhyung@kernel.org> References: <20251115234106.348571-1-namhyung@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It needs to sync with the kernel to support user space changes for the deferred callchains. Signed-off-by: Namhyung Kim Reviewed-by: Ian Rogers --- tools/include/uapi/linux/perf_event.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tools/include/uapi/linux/perf_event.h b/tools/include/uapi/lin= ux/perf_event.h index 78a362b8002776e5..d292f96bc06f86bc 100644 --- a/tools/include/uapi/linux/perf_event.h +++ b/tools/include/uapi/linux/perf_event.h @@ -463,7 +463,9 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THR= EAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* request PERF_RECORD_CALLCHAIN_DEFERRED records= */ + defer_output : 1, /* output PERF_RECORD_CALLCHAIN_DEFERRED records = */ + __reserved_1 : 24; =20 union { __u32 wakeup_events; /* wake up every n events */ @@ -1239,6 +1241,22 @@ enum perf_event_type { */ PERF_RECORD_AUX_OUTPUT_HW_ID =3D 21, =20 + /* + * This user callchain capture was deferred until shortly before + * returning to user space. Previous samples would have kernel + * callchains only and they need to be stitched with this to make full + * callchains. + * + * struct { + * struct perf_event_header header; + * u64 cookie; + * u64 nr; + * u64 ips[nr]; + * struct sample_id sample_id; + * }; + */ + PERF_RECORD_CALLCHAIN_DEFERRED =3D 22, + PERF_RECORD_MAX, /* non-ABI */ }; =20 @@ -1269,6 +1287,7 @@ enum perf_callchain_context { PERF_CONTEXT_HV =3D (__u64)-32, PERF_CONTEXT_KERNEL =3D (__u64)-128, PERF_CONTEXT_USER =3D (__u64)-512, + PERF_CONTEXT_USER_DEFERRED =3D (__u64)-640, =20 PERF_CONTEXT_GUEST =3D (__u64)-2048, PERF_CONTEXT_GUEST_KERNEL =3D (__u64)-2176, --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:59 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20D4F2773C3; Sat, 15 Nov 2025 23:41:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250070; cv=none; b=b1iaq5jhZ2WZq1C7WSQ0G6dihVIqszkitZAMB+HrqUco+o2FtpOpmwtjhzSXWhOsez8GTKq6+463KRrl/sNVsN+ixWN4URGj3cYCZ+6onuihe2LvC9wZqFOxDNNCNl2s7BnfPk9Zp8J9vkCrxE28xrhuvZ1tQEc+zB1jXwgXB/0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250070; c=relaxed/simple; bh=xJnJc9u7RJ2aTT8BEwhXyb++4h/xod80JdHranqM8pU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MM9lGXDK0OsSJpzDezOgsquflEYorLkXtV6T6qENUWap6XVOL1lkyu9v/+iyyGP7Ci1uaO456gBkQGomjjiVQsHcBs/+k/vFQMLgveBDW5BNpadtFAA2d0tKGn1HIxdpRqnRy8VUs8fjTPtLbK2He3oHKjfUP/z69umLvUpPvAw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kR2qVoRQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kR2qVoRQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6009CC4AF54; Sat, 15 Nov 2025 23:41:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763250068; bh=xJnJc9u7RJ2aTT8BEwhXyb++4h/xod80JdHranqM8pU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kR2qVoRQLp0WZwujU7TTAalBDna0bjPr5EnxSXV83E+lK3ox25Wq+ZhVtLFrMovxK L35MMwruBA08azPo4r74/wkviO6I7BJ/Q19hSK3R3NvG4iUNFQtKizNxr6OYujKHdN 25Oa+pJqYy4Wt9ih5SYsFf+HyXlTbd131w9hx5EIsDCm0YgQV7R3IQKMZZxuuUo4R3 n9hl+NWx5Dkm7c5uFaHUO/5alFMFuCWvK9CbJ13l64o5IUm+23Au9tVfEiaAgGSP+I 00JdbYpsa1FpJgIIbv95WlMCSQc7y5On6FuWDFw1ZuVm3R1jyo+bwOW7nT2CtZ1SDb dxPEaq3TLqfeA== From: Namhyung Kim To: Arnaldo Carvalho de Melo , Ian Rogers , James Clark Cc: Jiri Olsa , Adrian Hunter , Peter Zijlstra , Ingo Molnar , LKML , linux-perf-users@vger.kernel.org, Steven Rostedt , Josh Poimboeuf , Indu Bhagat , Jens Remus , Mathieu Desnoyers , linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v4 2/5] perf tools: Minimal DEFERRED_CALLCHAIN support Date: Sat, 15 Nov 2025 15:41:03 -0800 Message-ID: <20251115234106.348571-3-namhyung@kernel.org> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog In-Reply-To: <20251115234106.348571-1-namhyung@kernel.org> References: <20251115234106.348571-1-namhyung@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a new event type for deferred callchains and a new callback for the struct perf_tool. For now it doesn't actually handle the deferred callchains but it just marks the sample if it has the PERF_CONTEXT_ USER_DEFFERED in the callchain array. At least, perf report can dump the raw data with this change. Actually this requires the next commit to enable attr.defer_callchain, but if you already have a data file, it'll show the following result. $ perf report -D ... 0x2158@perf.data [0x40]: event: 22 . . ... raw event: size 64 bytes . 0000: 16 00 00 00 02 00 40 00 06 00 00 00 0b 00 00 00 ......@.......= .. . 0010: 03 00 00 00 00 00 00 00 a7 7f 33 fe 18 7f 00 00 ..........3...= .. . 0020: 0f 0e 33 fe 18 7f 00 00 48 14 33 fe 18 7f 00 00 ..3.....H.3...= .. . 0030: 08 09 00 00 08 09 00 00 e6 7a e7 35 1c 00 00 00 .........z.5..= .. 121163447014 0x2158 [0x40]: PERF_RECORD_CALLCHAIN_DEFERRED(IP, 0x2): 2312= /2312: 0xb00000006 ... FP chain: nr:3 ..... 0: 00007f18fe337fa7 ..... 1: 00007f18fe330e0f ..... 2: 00007f18fe331448 : unhandled! Signed-off-by: Namhyung Kim Reviewed-by: Ian Rogers --- tools/lib/perf/include/perf/event.h | 13 ++++++++++ tools/perf/util/event.c | 1 + tools/perf/util/evsel.c | 31 +++++++++++++++++++++-- tools/perf/util/machine.c | 1 + tools/perf/util/perf_event_attr_fprintf.c | 2 ++ tools/perf/util/sample.h | 2 ++ tools/perf/util/session.c | 20 +++++++++++++++ tools/perf/util/tool.c | 1 + tools/perf/util/tool.h | 3 ++- 9 files changed, 71 insertions(+), 3 deletions(-) diff --git a/tools/lib/perf/include/perf/event.h b/tools/lib/perf/include/p= erf/event.h index aa1e91c97a226e1a..43a8cb04994fa033 100644 --- a/tools/lib/perf/include/perf/event.h +++ b/tools/lib/perf/include/perf/event.h @@ -151,6 +151,18 @@ struct perf_record_switch { __u32 next_prev_tid; }; =20 +struct perf_record_callchain_deferred { + struct perf_event_header header; + /* + * This is to match kernel and (deferred) user stacks together. + * The kernel part will be in the sample callchain array after + * the PERF_CONTEXT_USER_DEFERRED entry. + */ + __u64 cookie; + __u64 nr; + __u64 ips[]; +}; + struct perf_record_header_attr { struct perf_event_header header; struct perf_event_attr attr; @@ -523,6 +535,7 @@ union perf_event { struct perf_record_read read; struct perf_record_throttle throttle; struct perf_record_sample sample; + struct perf_record_callchain_deferred callchain_deferred; struct perf_record_bpf_event bpf; struct perf_record_ksymbol ksymbol; struct perf_record_text_poke_event text_poke; diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c index fcf44149feb20c35..4c92cc1a952c1d9f 100644 --- a/tools/perf/util/event.c +++ b/tools/perf/util/event.c @@ -61,6 +61,7 @@ static const char *perf_event__names[] =3D { [PERF_RECORD_CGROUP] =3D "CGROUP", [PERF_RECORD_TEXT_POKE] =3D "TEXT_POKE", [PERF_RECORD_AUX_OUTPUT_HW_ID] =3D "AUX_OUTPUT_HW_ID", + [PERF_RECORD_CALLCHAIN_DEFERRED] =3D "CALLCHAIN_DEFERRED", [PERF_RECORD_HEADER_ATTR] =3D "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] =3D "EVENT_TYPE", [PERF_RECORD_HEADER_TRACING_DATA] =3D "TRACING_DATA", diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c index 989c56d4a23f74f4..5ee3e7dee93fbbcb 100644 --- a/tools/perf/util/evsel.c +++ b/tools/perf/util/evsel.c @@ -3089,6 +3089,20 @@ int evsel__parse_sample(struct evsel *evsel, union p= erf_event *event, data->data_src =3D PERF_MEM_DATA_SRC_NONE; data->vcpu =3D -1; =20 + if (event->header.type =3D=3D PERF_RECORD_CALLCHAIN_DEFERRED) { + const u64 max_callchain_nr =3D UINT64_MAX / sizeof(u64); + + data->callchain =3D (struct ip_callchain *)&event->callchain_deferred.nr; + if (data->callchain->nr > max_callchain_nr) + return -EFAULT; + + data->deferred_cookie =3D event->callchain_deferred.cookie; + + if (evsel->core.attr.sample_id_all) + perf_evsel__parse_id_sample(evsel, event, data); + return 0; + } + if (event->header.type !=3D PERF_RECORD_SAMPLE) { if (!evsel->core.attr.sample_id_all) return 0; @@ -3213,12 +3227,25 @@ int evsel__parse_sample(struct evsel *evsel, union = perf_event *event, =20 if (type & PERF_SAMPLE_CALLCHAIN) { const u64 max_callchain_nr =3D UINT64_MAX / sizeof(u64); + u64 callchain_nr; =20 OVERFLOW_CHECK_u64(array); data->callchain =3D (struct ip_callchain *)array++; - if (data->callchain->nr > max_callchain_nr) + callchain_nr =3D data->callchain->nr; + if (callchain_nr > max_callchain_nr) return -EFAULT; - sz =3D data->callchain->nr * sizeof(u64); + sz =3D callchain_nr * sizeof(u64); + /* + * Save the cookie for the deferred user callchain. The last 2 + * entries in the callchain should be the context marker and the + * cookie. The cookie will be used to match PERF_RECORD_ + * CALLCHAIN_DEFERRED later. + */ + if (evsel->core.attr.defer_callchain && callchain_nr >=3D 2 && + data->callchain->ips[callchain_nr - 2] =3D=3D PERF_CONTEXT_USER_DEFE= RRED) { + data->deferred_cookie =3D data->callchain->ips[callchain_nr - 1]; + data->deferred_callchain =3D true; + } OVERFLOW_CHECK(array, sz, max_size); array =3D (void *)array + sz; } diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index b5dd42588c916d91..841b711d970e9457 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -2124,6 +2124,7 @@ static int add_callchain_ip(struct thread *thread, *cpumode =3D PERF_RECORD_MISC_KERNEL; break; case PERF_CONTEXT_USER: + case PERF_CONTEXT_USER_DEFERRED: *cpumode =3D PERF_RECORD_MISC_USER; break; default: diff --git a/tools/perf/util/perf_event_attr_fprintf.c b/tools/perf/util/pe= rf_event_attr_fprintf.c index 66b666d9ce649dd7..741c3d657a8b6ae7 100644 --- a/tools/perf/util/perf_event_attr_fprintf.c +++ b/tools/perf/util/perf_event_attr_fprintf.c @@ -343,6 +343,8 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_even= t_attr *attr, PRINT_ATTRf(inherit_thread, p_unsigned); PRINT_ATTRf(remove_on_exec, p_unsigned); PRINT_ATTRf(sigtrap, p_unsigned); + PRINT_ATTRf(defer_callchain, p_unsigned); + PRINT_ATTRf(defer_output, p_unsigned); =20 PRINT_ATTRn("{ wakeup_events, wakeup_watermark }", wakeup_events, p_unsig= ned, false); PRINT_ATTRf(bp_type, p_unsigned); diff --git a/tools/perf/util/sample.h b/tools/perf/util/sample.h index fae834144ef42105..a8307b20a9ea8066 100644 --- a/tools/perf/util/sample.h +++ b/tools/perf/util/sample.h @@ -107,6 +107,8 @@ struct perf_sample { /** @weight3: On x86 holds retire_lat, on powerpc holds p_stage_cyc. */ u16 weight3; bool no_hw_idx; /* No hw_idx collected in branch_stack */ + bool deferred_callchain; /* Has deferred user callchains */ + u64 deferred_cookie; char insn[MAX_INSN]; void *raw_data; struct ip_callchain *callchain; diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c index 4b0236b2df2913e1..361e15c1f26a96d0 100644 --- a/tools/perf/util/session.c +++ b/tools/perf/util/session.c @@ -720,6 +720,7 @@ static perf_event__swap_op perf_event__swap_ops[] =3D { [PERF_RECORD_CGROUP] =3D perf_event__cgroup_swap, [PERF_RECORD_TEXT_POKE] =3D perf_event__text_poke_swap, [PERF_RECORD_AUX_OUTPUT_HW_ID] =3D perf_event__all64_swap, + [PERF_RECORD_CALLCHAIN_DEFERRED] =3D perf_event__all64_swap, [PERF_RECORD_HEADER_ATTR] =3D perf_event__hdr_attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] =3D perf_event__event_type_swap, [PERF_RECORD_HEADER_TRACING_DATA] =3D perf_event__tracing_data_swap, @@ -854,6 +855,9 @@ static void callchain__printf(struct evsel *evsel, for (i =3D 0; i < callchain->nr; i++) printf("..... %2d: %016" PRIx64 "\n", i, callchain->ips[i]); + + if (sample->deferred_callchain) + printf("...... (deferred)\n"); } =20 static void branch_stack__printf(struct perf_sample *sample, @@ -1123,6 +1127,19 @@ static void dump_sample(struct evsel *evsel, union p= erf_event *event, sample_read__printf(sample, evsel->core.attr.read_format); } =20 +static void dump_deferred_callchain(struct evsel *evsel, union perf_event = *event, + struct perf_sample *sample) +{ + if (!dump_trace) + return; + + printf("(IP, 0x%x): %d/%d: %#" PRIx64 "\n", + event->header.misc, sample->pid, sample->tid, sample->deferred_coo= kie); + + if (evsel__has_callchain(evsel)) + callchain__printf(evsel, sample); +} + static void dump_read(struct evsel *evsel, union perf_event *event) { struct perf_record_read *read_event =3D &event->read; @@ -1353,6 +1370,9 @@ static int machines__deliver_event(struct machines *m= achines, return tool->text_poke(tool, event, sample, machine); case PERF_RECORD_AUX_OUTPUT_HW_ID: return tool->aux_output_hw_id(tool, event, sample, machine); + case PERF_RECORD_CALLCHAIN_DEFERRED: + dump_deferred_callchain(evsel, event, sample); + return tool->callchain_deferred(tool, event, sample, evsel, machine); default: ++evlist->stats.nr_unknown_events; return -1; diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c index 22a8a4ffe05f778e..f732d33e7f895ed4 100644 --- a/tools/perf/util/tool.c +++ b/tools/perf/util/tool.c @@ -287,6 +287,7 @@ void perf_tool__init(struct perf_tool *tool, bool order= ed_events) tool->read =3D process_event_sample_stub; tool->throttle =3D process_event_stub; tool->unthrottle =3D process_event_stub; + tool->callchain_deferred =3D process_event_sample_stub; tool->attr =3D process_event_synth_attr_stub; tool->event_update =3D process_event_synth_event_update_stub; tool->tracing_data =3D process_event_synth_tracing_data_stub; diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h index 88337cee1e3e2be3..9b9f0a8cbf3de4b5 100644 --- a/tools/perf/util/tool.h +++ b/tools/perf/util/tool.h @@ -44,7 +44,8 @@ enum show_feature_header { =20 struct perf_tool { event_sample sample, - read; + read, + callchain_deferred; event_op mmap, mmap2, comm, --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:59 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20DBB2773E4; Sat, 15 Nov 2025 23:41:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250070; cv=none; b=bm3HbWm/ZNyVWRrDO2f/ETkA6Cd9HpM0GQpRqZRsjBfC3HC9JboYPppy26RN+1og8KyOujRF7uGmr4ugIKHyBcQEk3cOpmblfz13YHzM84cc1LzkcQOn13r5RNsPY5Qki5MtAUw8t3uxVqLtS5Ngf9wQalGpVXi4MkmhF6FvULQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250070; c=relaxed/simple; bh=kou+fW5zSUIurqEMH1ts4VKI93KS9bTnRZtP6XhgKQI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=shH9i0WnYWNfZ8/5hHhU2rsD7f3aK5CgnqroAR+h7qlDn5Gd2+bNCaGYWHjb99A7V0ib7eiNsGSqzBvRuNu5R/Iy8ebtj+C6zs+D/tolAQMYE8dffoy7oZf8eRA7xMOEIoHKuFIn9pIe7z6Zq53oQPE5Bo7IpE5qCv+KYm0sJS4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=N9SLmM9y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="N9SLmM9y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17FE3C4DDE7; Sat, 15 Nov 2025 23:41:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763250069; bh=kou+fW5zSUIurqEMH1ts4VKI93KS9bTnRZtP6XhgKQI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N9SLmM9yajuhAafdjdYCAJN9PALp2GpIbYtpDbZbmIY7UBV/siUlEO8YLMDZMUlc7 q0l7tz1LqU3bzBOscQtTrG7LFJPmOx7NUPa9WQrH0kxS8jZisOaGu3vRC8qaz2iiqe UsuUvIzgPIDmmFgAdIwTO+AQAVZRVLyOZON0jaZi3gs41M0YI4AF1ENTSnaUgtK0hm XsGsVl58h/iuJtYorClb5Sjm/KjrvI1rja1XGBPChm7tAm21EoEHtoBhn7GHs+yWes 4Sl9N0a58lChHRXNY/A/WpAgmuh5c09/TYB+fxHS3gfOqnSse0r41RKmcCeH48dAsG BZ9SoChkxTzSQ== From: Namhyung Kim To: Arnaldo Carvalho de Melo , Ian Rogers , James Clark Cc: Jiri Olsa , Adrian Hunter , Peter Zijlstra , Ingo Molnar , LKML , linux-perf-users@vger.kernel.org, Steven Rostedt , Josh Poimboeuf , Indu Bhagat , Jens Remus , Mathieu Desnoyers , linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v4 3/5] perf record: Add --call-graph fp,defer option for deferred callchains Date: Sat, 15 Nov 2025 15:41:04 -0800 Message-ID: <20251115234106.348571-4-namhyung@kernel.org> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog In-Reply-To: <20251115234106.348571-1-namhyung@kernel.org> References: <20251115234106.348571-1-namhyung@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a new callchain record mode option for deferred callchains. For now it only works with FP (frame-pointer) mode. And add the missing feature detection logic to clear the flag on old kernels. $ perf record --call-graph fp,defer -vv true ... ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) size 136 config 0 (PERF_COUNT_HW_CPU_CYCLES) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|CALLCHAIN|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 defer_callchain 1 defer_output 1 ------------------------------------------------------------ sys_perf_event_open: pid 162755 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 switching off deferred callchain support Signed-off-by: Namhyung Kim Reviewed-by: Ian Rogers --- tools/perf/Documentation/perf-config.txt | 3 +++ tools/perf/Documentation/perf-record.txt | 4 ++++ tools/perf/util/callchain.c | 16 +++++++++++++--- tools/perf/util/callchain.h | 1 + tools/perf/util/evsel.c | 19 +++++++++++++++++++ tools/perf/util/evsel.h | 1 + 6 files changed, 41 insertions(+), 3 deletions(-) diff --git a/tools/perf/Documentation/perf-config.txt b/tools/perf/Document= ation/perf-config.txt index c6f33565966735fe..642d1c490d9e3bcd 100644 --- a/tools/perf/Documentation/perf-config.txt +++ b/tools/perf/Documentation/perf-config.txt @@ -452,6 +452,9 @@ Variables kernel space is controlled not by this option but by the kernel config (CONFIG_UNWINDER_*). =20 + The 'defer' mode can be used with 'fp' mode to enable deferred + user callchains (like 'fp,defer'). + call-graph.dump-size:: The size of stack to dump in order to do post-unwinding. Default is 8192= (byte). When using dwarf into record-mode, the default size will be used if omit= ted. diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Document= ation/perf-record.txt index 067891bd7da6edc8..e8b9aadbbfa50574 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -325,6 +325,10 @@ OPTIONS by default. User can change the number by passing it after comma like "--call-graph fp,32". =20 + Also "defer" can be used with "fp" (like "--call-graph fp,defer") to + enable deferred user callchain which will collect user-space callchains + when the thread returns to the user space. + -q:: --quiet:: Don't print any warnings or messages, useful for scripting. diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c index d7b7eef740b9d6ed..2884187ccbbecfdc 100644 --- a/tools/perf/util/callchain.c +++ b/tools/perf/util/callchain.c @@ -275,9 +275,13 @@ int parse_callchain_record(const char *arg, struct cal= lchain_param *param) if (tok) { unsigned long size; =20 - size =3D strtoul(tok, &name, 0); - if (size < (unsigned) sysctl__max_stack()) - param->max_stack =3D size; + if (!strncmp(tok, "defer", sizeof("defer"))) { + param->defer =3D true; + } else { + size =3D strtoul(tok, &name, 0); + if (size < (unsigned) sysctl__max_stack()) + param->max_stack =3D size; + } } break; =20 @@ -314,6 +318,12 @@ int parse_callchain_record(const char *arg, struct cal= lchain_param *param) } while (0); =20 free(buf); + + if (param->defer && param->record_mode !=3D CALLCHAIN_FP) { + pr_err("callchain: deferred callchain only works with FP\n"); + return -EINVAL; + } + return ret; } =20 diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h index 86ed9e4d04f9ee7b..d5ae4fbb7ce5fa44 100644 --- a/tools/perf/util/callchain.h +++ b/tools/perf/util/callchain.h @@ -98,6 +98,7 @@ extern bool dwarf_callchain_users; =20 struct callchain_param { bool enabled; + bool defer; enum perf_call_graph_mode record_mode; u32 dump_size; enum chain_mode mode; diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c index 5ee3e7dee93fbbcb..7772ee9cfe3ac1c7 100644 --- a/tools/perf/util/evsel.c +++ b/tools/perf/util/evsel.c @@ -1065,6 +1065,9 @@ static void __evsel__config_callchain(struct evsel *e= vsel, struct record_opts *o pr_info("Disabling user space callchains for function trace event.\n"); attr->exclude_callchain_user =3D 1; } + + if (param->defer && !attr->exclude_callchain_user) + attr->defer_callchain =3D 1; } =20 void evsel__config_callchain(struct evsel *evsel, struct record_opts *opts, @@ -1511,6 +1514,7 @@ void evsel__config(struct evsel *evsel, struct record= _opts *opts, attr->mmap2 =3D track && !perf_missing_features.mmap2; attr->comm =3D track; attr->build_id =3D track && opts->build_id; + attr->defer_output =3D track && callchain->defer; =20 /* * ksymbol is tracked separately with text poke because it needs to be @@ -2199,6 +2203,10 @@ static int __evsel__prepare_open(struct evsel *evsel= , struct perf_cpu_map *cpus, =20 static void evsel__disable_missing_features(struct evsel *evsel) { + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_callc= hain) + evsel->core.attr.defer_callchain =3D 0; + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_outpu= t) + evsel->core.attr.defer_output =3D 0; if (perf_missing_features.inherit_sample_read && evsel->core.attr.inherit= && (evsel->core.attr.sample_type & PERF_SAMPLE_READ)) evsel->core.attr.inherit =3D 0; @@ -2473,6 +2481,13 @@ static bool evsel__detect_missing_features(struct ev= sel *evsel, struct perf_cpu =20 /* Please add new feature detection here. */ =20 + attr.defer_callchain =3D true; + if (has_attr_feature(&attr, /*flags=3D*/0)) + goto found; + perf_missing_features.defer_callchain =3D true; + pr_debug2("switching off deferred callchain support\n"); + attr.defer_callchain =3D false; + attr.inherit =3D true; attr.sample_type =3D PERF_SAMPLE_READ | PERF_SAMPLE_TID; if (has_attr_feature(&attr, /*flags=3D*/0)) @@ -2584,6 +2599,10 @@ static bool evsel__detect_missing_features(struct ev= sel *evsel, struct perf_cpu errno =3D old_errno; =20 check: + if ((evsel->core.attr.defer_callchain || evsel->core.attr.defer_output) && + perf_missing_features.defer_callchain) + return true; + if (evsel->core.attr.inherit && (evsel->core.attr.sample_type & PERF_SAMPLE_READ) && perf_missing_features.inherit_sample_read) diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h index 3ae4ac8f9a37e009..a08130ff2e47a887 100644 --- a/tools/perf/util/evsel.h +++ b/tools/perf/util/evsel.h @@ -221,6 +221,7 @@ struct perf_missing_features { bool branch_counters; bool aux_action; bool inherit_sample_read; + bool defer_callchain; }; =20 extern struct perf_missing_features perf_missing_features; --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:59 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B357C27E060; Sat, 15 Nov 2025 23:41:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250070; cv=none; b=liAFHmwUk1ZZjv8d5yjhzmhZBQAF02H5l1cNdkLcLwdf+djioF/M+AeY0uV+gZSmwL1PbdvA8jn6038MW4TbxAouffPNJnocw73YWT+8ryIWRuQpHcvlGBpwvltyM4GpkuQ/aeuU248wo3hmqDSMFJ/ta9yETzm7P4PVxKNm9js= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250070; c=relaxed/simple; bh=8aI5d44+B13FpjrPrJrk5w8fPEAqNUfYH0gw+zGEJw4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t11oNIbBWRqIeq10RhzvN5C/HGahiNeV3LFt/GqHcnqZxGbUjC1LwsFSqrWkAg1mj4HIOvv1rYH916mOVAwP0R+CP06qIlN9RPjXPNUJOIshlXSgmYlryZ0R6N201lAUgxPoJk3/5TBxHWDu2jq24+Dg3sPxQhZidn0WOW5caD4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZBvlq6lo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZBvlq6lo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C4485C4AF50; Sat, 15 Nov 2025 23:41:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763250070; bh=8aI5d44+B13FpjrPrJrk5w8fPEAqNUfYH0gw+zGEJw4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZBvlq6lo5xu3mKUYYqfdp9cX3l5N/ezuLSLQ7zjY1qy7D/vDU7rSq8ASuR8VyAIpd 1IgKqW0Gbb1ychlfcNRXylYfm5+Q6XPBVPEiuFVwAwxcExeMAKKoj2xMwFx+LNTfxa +OOj3b6DuphFe7qxoJaLxkYPTDCM9RC34zEBkdUG4eGOB/BseVdlpTVVubrzPXxZW8 7ZnQlXEUrUfDo0sOGshhylmyRQTUmrhUvjVdYp3rWFT5To88WGzO4ixojaABLXYXhx L3aj7aCYtoAczhQ4SSHARVI9mTXCXze8rytqDWuxw6V+O2G0GLIjS9wf6TOJKOqg0i ukQ6xfxTrTe1Q== From: Namhyung Kim To: Arnaldo Carvalho de Melo , Ian Rogers , James Clark Cc: Jiri Olsa , Adrian Hunter , Peter Zijlstra , Ingo Molnar , LKML , linux-perf-users@vger.kernel.org, Steven Rostedt , Josh Poimboeuf , Indu Bhagat , Jens Remus , Mathieu Desnoyers , linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v4 4/5] perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED Date: Sat, 15 Nov 2025 15:41:05 -0800 Message-ID: <20251115234106.348571-5-namhyung@kernel.org> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog In-Reply-To: <20251115234106.348571-1-namhyung@kernel.org> References: <20251115234106.348571-1-namhyung@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Handle the deferred callchains in the script output. $ perf script ... pwd 2312 121.163435: 249113 cpu/cycles/P: ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms]) ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms]) ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms]) ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms]) ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms]) ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms]) ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kal= lsyms]) b00000006 (cookie) ([unknown]) pwd 2312 121.163447: DEFERRED CALLCHAIN [cookie: b00000006] 7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86= -64.so.2) 7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-= linux-x86-64.so.2) 7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-lin= ux-x86-64.so.2) Signed-off-by: Namhyung Kim Reviewed-by: Ian Rogers --- tools/perf/builtin-script.c | 89 +++++++++++++++++++++++++++++++++ tools/perf/util/evsel_fprintf.c | 5 +- 2 files changed, 93 insertions(+), 1 deletion(-) diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c index 011962e1ee0f6898..85b42205a71b3993 100644 --- a/tools/perf/builtin-script.c +++ b/tools/perf/builtin-script.c @@ -2706,6 +2706,94 @@ static int process_sample_event(const struct perf_to= ol *tool, return ret; } =20 +static int process_deferred_sample_event(const struct perf_tool *tool, + union perf_event *event, + struct perf_sample *sample, + struct evsel *evsel, + struct machine *machine) +{ + struct perf_script *scr =3D container_of(tool, struct perf_script, tool); + struct perf_event_attr *attr =3D &evsel->core.attr; + struct evsel_script *es =3D evsel->priv; + unsigned int type =3D output_type(attr->type); + struct addr_location al; + FILE *fp =3D es->fp; + int ret =3D 0; + + if (output[type].fields =3D=3D 0) + return 0; + + /* Set thread to NULL to indicate addr_al and al are not initialized */ + addr_location__init(&al); + + if (perf_time__ranges_skip_sample(scr->ptime_range, scr->range_num, + sample->time)) { + goto out_put; + } + + if (debug_mode) { + if (sample->time < last_timestamp) { + pr_err("Samples misordered, previous: %" PRIu64 + " this: %" PRIu64 "\n", last_timestamp, + sample->time); + nr_unordered++; + } + last_timestamp =3D sample->time; + goto out_put; + } + + if (filter_cpu(sample)) + goto out_put; + + if (machine__resolve(machine, &al, sample) < 0) { + pr_err("problem processing %d event, skipping it.\n", + event->header.type); + ret =3D -1; + goto out_put; + } + + if (al.filtered) + goto out_put; + + if (!show_event(sample, evsel, al.thread, &al, NULL)) + goto out_put; + + if (evswitch__discard(&scr->evswitch, evsel)) + goto out_put; + + perf_sample__fprintf_start(scr, sample, al.thread, evsel, + PERF_RECORD_CALLCHAIN_DEFERRED, fp); + fprintf(fp, "DEFERRED CALLCHAIN [cookie: %llx]", + (unsigned long long)event->callchain_deferred.cookie); + + if (PRINT_FIELD(IP)) { + struct callchain_cursor *cursor =3D NULL; + + if (symbol_conf.use_callchain && sample->callchain) { + cursor =3D get_tls_callchain_cursor(); + if (thread__resolve_callchain(al.thread, cursor, evsel, + sample, NULL, NULL, + scripting_max_stack)) { + pr_info("cannot resolve deferred callchains\n"); + cursor =3D NULL; + } + } + + fputc(cursor ? '\n' : ' ', fp); + sample__fprintf_sym(sample, &al, 0, output[type].print_ip_opts, + cursor, symbol_conf.bt_stop_list, fp); + } + + fprintf(fp, "\n"); + + if (verbose > 0) + fflush(fp); + +out_put: + addr_location__exit(&al); + return ret; +} + // Used when scr->per_event_dump is not set static struct evsel_script es_stdout; =20 @@ -4303,6 +4391,7 @@ int cmd_script(int argc, const char **argv) =20 perf_tool__init(&script.tool, !unsorted_dump); script.tool.sample =3D process_sample_event; + script.tool.callchain_deferred =3D process_deferred_sample_event; script.tool.mmap =3D perf_event__process_mmap; script.tool.mmap2 =3D perf_event__process_mmap2; script.tool.comm =3D perf_event__process_comm; diff --git a/tools/perf/util/evsel_fprintf.c b/tools/perf/util/evsel_fprint= f.c index 103984b29b1e10ae..10f1a03c28601e36 100644 --- a/tools/perf/util/evsel_fprintf.c +++ b/tools/perf/util/evsel_fprintf.c @@ -168,7 +168,10 @@ int sample__fprintf_callchain(struct perf_sample *samp= le, int left_alignment, node_al.addr =3D addr; node_al.map =3D map__get(map); =20 - if (print_symoffset) { + if (sample->deferred_callchain && + sample->deferred_cookie =3D=3D node->ip) { + printed +=3D fprintf(fp, "(cookie)"); + } else if (print_symoffset) { printed +=3D __symbol__fprintf_symname_offs(sym, &node_al, print_unknown_as_addr, true, fp); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Sun Feb 8 05:28:59 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 900272877C3; Sat, 15 Nov 2025 23:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250072; cv=none; b=Z0TytFQAB7wsjP9Eh/fcNvNsfDuSg4pD2ECVL6ITBMG984yARCnjkAsYKOQvy/pxgOqoTdfznpDlhTpKD77qS5MvBh/P/xcPDLFjYWYnh2drkU1ZL1S+bjSMq4f45Ul51eUdMlt24pi7YZHaGAd8OAxqawGmNfwfAJR0sCEHhlA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763250072; c=relaxed/simple; bh=YxjJn6agdNjSzJJRyp6VEqAOPGCfHety6pcDGYrqvxI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kD3NCx/ApiTOLUFRHhr0VCfwFpSgQhj2aNTbhjQQdXQ0Wc8MvCmVfwHxeq+PZezqFhubCjnQscYHhGe5Kps4dzW5OcOlxtKyZWVPlcXXU9gdzXlFhP7mFtitovJ+Sk4wTwDUtHyKja2HyhefCMXmtLgg5i0T6AzBzTlY5zd3sxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H2hG2Dfp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H2hG2Dfp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 822F8C4AF1D; Sat, 15 Nov 2025 23:41:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763250071; bh=YxjJn6agdNjSzJJRyp6VEqAOPGCfHety6pcDGYrqvxI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H2hG2DfpTIusEsgOjI59ZBsxWRp0ZD6bm6G5sbOpeStMcI+Nj5v1c51br2GvfFreR qhmuFeVv/dXXgFVUWHv6yaSnqhXh5lKde4tn3pBoo3lgA2hQNrD2n8e3JXEVVLGNGD vcexKT3kew9c+x/Vy5TRDhlOa5dWZ5NWMYVsyb8BGnsCdyHWRu17Rtlqt1bPY+y5N6 QEuoaxi7ZhITimHtqUGJRFCKEytvjd9SGjo5Ew31QTXxh73k3LPXUgtEQmECO17Lly 31r/0hDcPpgWqkqskmjeq7GfTukBRlLAYdWnkM3X7WUkvNKFAfBIo9Plua4TATcw+9 JueRdSnBPfXUw== From: Namhyung Kim To: Arnaldo Carvalho de Melo , Ian Rogers , James Clark Cc: Jiri Olsa , Adrian Hunter , Peter Zijlstra , Ingo Molnar , LKML , linux-perf-users@vger.kernel.org, Steven Rostedt , Josh Poimboeuf , Indu Bhagat , Jens Remus , Mathieu Desnoyers , linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v4 5/5] perf tools: Merge deferred user callchains Date: Sat, 15 Nov 2025 15:41:06 -0800 Message-ID: <20251115234106.348571-6-namhyung@kernel.org> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog In-Reply-To: <20251115234106.348571-1-namhyung@kernel.org> References: <20251115234106.348571-1-namhyung@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Save samples with deferred callchains in a separate list and deliver them after merging the user callchains. If users don't want to merge they can set tool->merge_deferred_callchains to false to prevent the behavior. With previous result, now perf script will show the merged callchains. $ perf script ... pwd 2312 121.163435: 249113 cpu/cycles/P: ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms]) ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms]) ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms]) ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms]) ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms]) ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms]) ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kal= lsyms]) 7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86= -64.so.2) 7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-= linux-x86-64.so.2) 7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-lin= ux-x86-64.so.2) ... The old output can be get using --no-merge-callchain option. Also perf report can get the user callchain entry at the end. $ perf report --no-children --stdio -q -S __build_id_parse.isra.0 # symbol: __build_id_parse.isra.0 8.40% pwd [kernel.kallsyms] | ---__build_id_parse.isra.0 perf_event_mmap mprotect_fixup do_mprotect_pkey __x64_sys_mprotect do_syscall_64 entry_SYSCALL_64_after_hwframe mprotect _dl_sysdep_start _dl_start_user Signed-off-by: Namhyung Kim --- tools/perf/Documentation/perf-script.txt | 5 ++ tools/perf/builtin-inject.c | 1 + tools/perf/builtin-report.c | 1 + tools/perf/builtin-script.c | 4 ++ tools/perf/util/callchain.c | 29 ++++++++++ tools/perf/util/callchain.h | 3 ++ tools/perf/util/evlist.c | 1 + tools/perf/util/evlist.h | 2 + tools/perf/util/session.c | 67 +++++++++++++++++++++++- tools/perf/util/tool.c | 1 + tools/perf/util/tool.h | 1 + 11 files changed, 114 insertions(+), 1 deletion(-) diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Document= ation/perf-script.txt index 28bec7e78bc858ba..03d1129606328d6d 100644 --- a/tools/perf/Documentation/perf-script.txt +++ b/tools/perf/Documentation/perf-script.txt @@ -527,6 +527,11 @@ include::itrace.txt[] The known limitations include exception handing such as setjmp/longjmp will have calls/returns not match. =20 +--merge-callchains:: + Enable merging deferred user callchains if available. This is the + default behavior. If you want to see separate CALLCHAIN_DEFERRED + records for some reason, use --no-merge-callchains explicitly. + :GMEXAMPLECMD: script :GMEXAMPLESUBCMD: include::guest-files.txt[] diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c index bd9245d2dd41aa48..51d2721b6db9dccb 100644 --- a/tools/perf/builtin-inject.c +++ b/tools/perf/builtin-inject.c @@ -2527,6 +2527,7 @@ int cmd_inject(int argc, const char **argv) inject.tool.auxtrace =3D perf_event__repipe_auxtrace; inject.tool.bpf_metadata =3D perf_event__repipe_op2_synth; inject.tool.dont_split_sample_group =3D true; + inject.tool.merge_deferred_callchains =3D false; inject.session =3D __perf_session__new(&data, &inject.tool, /*trace_event_repipe=3D*/inject.output.is_pipe, /*host_env=3D*/NULL); diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c index 2bc269f5fcef8023..add6b1c2aaf04270 100644 --- a/tools/perf/builtin-report.c +++ b/tools/perf/builtin-report.c @@ -1614,6 +1614,7 @@ int cmd_report(int argc, const char **argv) report.tool.event_update =3D perf_event__process_event_update; report.tool.feature =3D process_feature_event; report.tool.ordering_requires_timestamps =3D true; + report.tool.merge_deferred_callchains =3D !dump_trace; =20 session =3D perf_session__new(&data, &report.tool); if (IS_ERR(session)) { diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c index 85b42205a71b3993..62e43d3c5ad731a0 100644 --- a/tools/perf/builtin-script.c +++ b/tools/perf/builtin-script.c @@ -4009,6 +4009,7 @@ int cmd_script(int argc, const char **argv) bool header_only =3D false; bool script_started =3D false; bool unsorted_dump =3D false; + bool merge_deferred_callchains =3D true; char *rec_script_path =3D NULL; char *rep_script_path =3D NULL; struct perf_session *session; @@ -4162,6 +4163,8 @@ int cmd_script(int argc, const char **argv) "Guest code can be found in hypervisor process"), OPT_BOOLEAN('\0', "stitch-lbr", &script.stitch_lbr, "Enable LBR callgraph stitching approach"), + OPT_BOOLEAN('\0', "merge-callchains", &merge_deferred_callchains, + "Enable merge deferred user callchains"), OPTS_EVSWITCH(&script.evswitch), OPT_END() }; @@ -4418,6 +4421,7 @@ int cmd_script(int argc, const char **argv) script.tool.throttle =3D process_throttle_event; script.tool.unthrottle =3D process_throttle_event; script.tool.ordering_requires_timestamps =3D true; + script.tool.merge_deferred_callchains =3D merge_deferred_callchains; session =3D perf_session__new(&data, &script.tool); if (IS_ERR(session)) return PTR_ERR(session); diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c index 2884187ccbbecfdc..71dc5a070065dd2a 100644 --- a/tools/perf/util/callchain.c +++ b/tools/perf/util/callchain.c @@ -1838,3 +1838,32 @@ int sample__for_each_callchain_node(struct thread *t= hread, struct evsel *evsel, } return 0; } + +int sample__merge_deferred_callchain(struct perf_sample *sample_orig, + struct perf_sample *sample_callchain) +{ + u64 nr_orig =3D sample_orig->callchain->nr - 1; + u64 nr_deferred =3D sample_callchain->callchain->nr; + struct ip_callchain *callchain; + + if (sample_orig->callchain->nr < 2) { + sample_orig->deferred_callchain =3D false; + return -EINVAL; + } + + callchain =3D calloc(1 + nr_orig + nr_deferred, sizeof(u64)); + if (callchain =3D=3D NULL) { + sample_orig->deferred_callchain =3D false; + return -ENOMEM; + } + + callchain->nr =3D nr_orig + nr_deferred; + /* copy original including PERF_CONTEXT_USER_DEFERRED (but the cookie) */ + memcpy(callchain->ips, sample_orig->callchain->ips, nr_orig * sizeof(u64)= ); + /* copy deferred user callchains */ + memcpy(&callchain->ips[nr_orig], sample_callchain->callchain->ips, + nr_deferred * sizeof(u64)); + + sample_orig->callchain =3D callchain; + return 0; +} diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h index d5ae4fbb7ce5fa44..2a52af8c80ace33c 100644 --- a/tools/perf/util/callchain.h +++ b/tools/perf/util/callchain.h @@ -318,4 +318,7 @@ int sample__for_each_callchain_node(struct thread *thre= ad, struct evsel *evsel, struct perf_sample *sample, int max_stack, bool symbols, callchain_iter_fn cb, void *data); =20 +int sample__merge_deferred_callchain(struct perf_sample *sample_orig, + struct perf_sample *sample_callchain); + #endif /* __PERF_CALLCHAIN_H */ diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c index e8217efdda5323c6..03674d2cbd015e4f 100644 --- a/tools/perf/util/evlist.c +++ b/tools/perf/util/evlist.c @@ -85,6 +85,7 @@ void evlist__init(struct evlist *evlist, struct perf_cpu_= map *cpus, evlist->ctl_fd.pos =3D -1; evlist->nr_br_cntr =3D -1; metricgroup__rblist_init(&evlist->metric_events); + INIT_LIST_HEAD(&evlist->deferred_samples); } =20 struct evlist *evlist__new(void) diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h index 5e71e3dc60423079..911834ae7c2a6f76 100644 --- a/tools/perf/util/evlist.h +++ b/tools/perf/util/evlist.h @@ -92,6 +92,8 @@ struct evlist { * of struct metric_expr. */ struct rblist metric_events; + /* samples with deferred_callchain would wait here. */ + struct list_head deferred_samples; }; =20 struct evsel_str_handler { diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c index 361e15c1f26a96d0..2e777fd1bcf6707b 100644 --- a/tools/perf/util/session.c +++ b/tools/perf/util/session.c @@ -1285,6 +1285,60 @@ static int evlist__deliver_sample(struct evlist *evl= ist, const struct perf_tool per_thread); } =20 +struct deferred_event { + struct list_head list; + union perf_event *event; +}; + +static int evlist__deliver_deferred_samples(struct evlist *evlist, + const struct perf_tool *tool, + union perf_event *event, + struct perf_sample *sample, + struct machine *machine) +{ + struct deferred_event *de, *tmp; + struct evsel *evsel; + int ret =3D 0; + + if (!tool->merge_deferred_callchains) { + evsel =3D evlist__id2evsel(evlist, sample->id); + return tool->callchain_deferred(tool, event, sample, + evsel, machine); + } + + list_for_each_entry_safe(de, tmp, &evlist->deferred_samples, list) { + struct perf_sample orig_sample; + + ret =3D evlist__parse_sample(evlist, de->event, &orig_sample); + if (ret < 0) { + pr_err("failed to parse original sample\n"); + break; + } + + if (sample->tid !=3D orig_sample.tid) + continue; + + if (event->callchain_deferred.cookie =3D=3D orig_sample.deferred_cookie) + sample__merge_deferred_callchain(&orig_sample, sample); + else + orig_sample.deferred_callchain =3D false; + + evsel =3D evlist__id2evsel(evlist, orig_sample.id); + ret =3D evlist__deliver_sample(evlist, tool, de->event, + &orig_sample, evsel, machine); + + if (orig_sample.deferred_callchain) + free(orig_sample.callchain); + + list_del(&de->list); + free(de); + + if (ret) + break; + } + return ret; +} + static int machines__deliver_event(struct machines *machines, struct evlist *evlist, union perf_event *event, @@ -1313,6 +1367,16 @@ static int machines__deliver_event(struct machines *= machines, return 0; } dump_sample(evsel, event, sample, perf_env__arch(machine->env)); + if (sample->deferred_callchain && tool->merge_deferred_callchains) { + struct deferred_event *de =3D malloc(sizeof(*de)); + + if (de =3D=3D NULL) + return -ENOMEM; + + de->event =3D event; + list_add_tail(&de->list, &evlist->deferred_samples); + return 0; + } return evlist__deliver_sample(evlist, tool, event, sample, evsel, machin= e); case PERF_RECORD_MMAP: return tool->mmap(tool, event, sample, machine); @@ -1372,7 +1436,8 @@ static int machines__deliver_event(struct machines *m= achines, return tool->aux_output_hw_id(tool, event, sample, machine); case PERF_RECORD_CALLCHAIN_DEFERRED: dump_deferred_callchain(evsel, event, sample); - return tool->callchain_deferred(tool, event, sample, evsel, machine); + return evlist__deliver_deferred_samples(evlist, tool, event, + sample, machine); default: ++evlist->stats.nr_unknown_events; return -1; diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c index f732d33e7f895ed4..c5d3b464b2a433b3 100644 --- a/tools/perf/util/tool.c +++ b/tools/perf/util/tool.c @@ -266,6 +266,7 @@ void perf_tool__init(struct perf_tool *tool, bool order= ed_events) tool->cgroup_events =3D false; tool->no_warn =3D false; tool->show_feat_hdr =3D SHOW_FEAT_NO_HEADER; + tool->merge_deferred_callchains =3D true; =20 tool->sample =3D process_event_sample_stub; tool->mmap =3D process_event_stub; diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h index 9b9f0a8cbf3de4b5..e96b69d25a5b737d 100644 --- a/tools/perf/util/tool.h +++ b/tools/perf/util/tool.h @@ -90,6 +90,7 @@ struct perf_tool { bool cgroup_events; bool no_warn; bool dont_split_sample_group; + bool merge_deferred_callchains; enum show_feature_header show_feat_hdr; }; =20 --=20 2.52.0.rc1.455.g30608eb744-goog