The vdso dumped from process memory (in buildid-cache) lacks debugging
info. To annotate vdso symbols with source lines we need specify a
debugging version.
For x86, we can find them from your local build as
arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
/lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
the buildid has to match.
$ sudo perf record -a
$ sudo perf report --objdump=llvm-objdump \
--vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
__vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
Percent│ movq -48(%rbp),%rsi
│ testq %rax,%rax
│ ; return vread_hvclock();
│ movq %rax,%rdx
│ ; if (unlikely(!vdso_cycles_ok(cycles)))
│ ↑ js eb
│ ↑ jmp 74
│ ; ts->tv_sec = vdso_ts->sec;
0.02 │147: leaq 2(%rbx),%rax
│ shlq $4, %rax
│ addq %r10,%rax
│ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
9.38 │152: movl (%r10),%ecx
When doing cross platform analysis, we also need specify the vdso path if
we are interested in its symbols.
v2: update documentation.
Signed-off-by: Changbin Du <changbin.du@huawei.com>
---
tools/perf/Documentation/perf-annotate.txt | 3 +
tools/perf/Documentation/perf-c2c.txt | 3 +
tools/perf/Documentation/perf-inject.txt | 3 +
tools/perf/Documentation/perf-report.txt | 3 +
tools/perf/Documentation/perf-script.txt | 3 +
tools/perf/Documentation/perf-top.txt | 3 +
tools/perf/builtin-annotate.c | 2 +
tools/perf/builtin-c2c.c | 2 +
tools/perf/builtin-inject.c | 2 +
tools/perf/builtin-report.c | 2 +
tools/perf/builtin-script.c | 2 +
tools/perf/builtin-top.c | 2 +
tools/perf/util/disasm.c | 7 +-
tools/perf/util/symbol.c | 82 +++++++++++++++++++++-
tools/perf/util/symbol_conf.h | 5 ++
15 files changed, 119 insertions(+), 5 deletions(-)
diff --git a/tools/perf/Documentation/perf-annotate.txt b/tools/perf/Documentation/perf-annotate.txt
index b95524bea021..4b6692f9a793 100644
--- a/tools/perf/Documentation/perf-annotate.txt
+++ b/tools/perf/Documentation/perf-annotate.txt
@@ -58,6 +58,9 @@ OPTIONS
--ignore-vmlinux::
Ignore vmlinux files.
+--vdso=<vdso1[,vdso2]>::
+ Specify vdso pathnames. You can specify up to two for multiarch-support.
+
--itrace::
Options for decoding instruction tracing data. The options are:
diff --git a/tools/perf/Documentation/perf-c2c.txt b/tools/perf/Documentation/perf-c2c.txt
index 856f0dfb8e5a..7c07efca7542 100644
--- a/tools/perf/Documentation/perf-c2c.txt
+++ b/tools/perf/Documentation/perf-c2c.txt
@@ -71,6 +71,9 @@ REPORT OPTIONS
--vmlinux=<file>::
vmlinux pathname
+--vdso=<vdso1[,vdso2]>::
+ Specify vdso pathnames. You can specify up to two for multiarch-support.
+
-v::
--verbose::
Be more verbose (show counter open errors, etc).
diff --git a/tools/perf/Documentation/perf-inject.txt b/tools/perf/Documentation/perf-inject.txt
index c972032f4ca0..3c88967b4c7f 100644
--- a/tools/perf/Documentation/perf-inject.txt
+++ b/tools/perf/Documentation/perf-inject.txt
@@ -62,6 +62,9 @@ OPTIONS
--kallsyms=<file>::
kallsyms pathname
+--vdso=<vdso1[,vdso2]>::
+ Specify vdso pathnames. You can specify up to two for multiarch-support.
+
--itrace::
Decode Instruction Tracing data, replacing it with synthesized events.
Options are:
diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index d2b1593ef700..8a3ba5f74cac 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -345,6 +345,9 @@ OPTIONS
Load module symbols. WARNING: This should only be used with -k and
a LIVE kernel.
+--vdso=<vdso1[,vdso2]>::
+ Specify vdso pathnames. You can specify up to two for multiarch-support.
+
-f::
--force::
Don't do ownership validation.
diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
index ff086ef05a0c..48f9974ca4c5 100644
--- a/tools/perf/Documentation/perf-script.txt
+++ b/tools/perf/Documentation/perf-script.txt
@@ -296,6 +296,9 @@ OPTIONS
--kallsyms=<file>::
kallsyms pathname
+--vdso=<vdso1[,vdso2]>::
+ Specify vdso pathnames. You can specify up to two for multiarch-support.
+
--symfs=<directory>::
Look for files with symbols relative to this directory.
diff --git a/tools/perf/Documentation/perf-top.txt b/tools/perf/Documentation/perf-top.txt
index a754875fa5bb..1cf3ee16a0c1 100644
--- a/tools/perf/Documentation/perf-top.txt
+++ b/tools/perf/Documentation/perf-top.txt
@@ -76,6 +76,9 @@ Default is to monitor all CPUS.
--kallsyms=<file>::
kallsyms pathname
+--vdso=<vdso1[,vdso2]>::
+ Specify vdso pathnames. You can specify up to two for multiarch-support.
+
-m <pages>::
--mmap-pages=<pages>::
Number of mmap data pages (must be a power of two) or size
diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 50d2fb222d48..ff466882065d 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -742,6 +742,8 @@ int cmd_annotate(int argc, const char **argv)
"file", "vmlinux pathname"),
OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
"load module symbols - WARNING: use only with -k and LIVE kernel"),
+ OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
+ parse_vdso_pathnames),
OPT_BOOLEAN('l', "print-line", &annotate_opts.print_lines,
"print matching source lines (may be slow)"),
OPT_BOOLEAN('P', "full-paths", &annotate_opts.full_path,
diff --git a/tools/perf/builtin-c2c.c b/tools/perf/builtin-c2c.c
index c157bd31f2e5..4764f9139661 100644
--- a/tools/perf/builtin-c2c.c
+++ b/tools/perf/builtin-c2c.c
@@ -3018,6 +3018,8 @@ static int perf_c2c__report(int argc, const char **argv)
const struct option options[] = {
OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name,
"file", "vmlinux pathname"),
+ OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
+ parse_vdso_pathnames),
OPT_STRING('i', "input", &input_name, "file",
"the input file to process"),
OPT_INCR('N', "node-info", &c2c.node_info,
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index a212678d47be..e774e83d0a0f 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -2247,6 +2247,8 @@ int cmd_inject(int argc, const char **argv)
"don't load vmlinux even if found"),
OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name, "file",
"kallsyms pathname"),
+ OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
+ parse_vdso_pathnames),
OPT_BOOLEAN('f', "force", &data.force, "don't complain, do it"),
OPT_CALLBACK_OPTARG(0, "itrace", &inject.itrace_synth_opts,
NULL, "opts", "Instruction Tracing options\n"
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 69618fb0110b..a64b48460dce 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -1324,6 +1324,8 @@ int cmd_report(int argc, const char **argv)
"don't load vmlinux even if found"),
OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
"file", "kallsyms pathname"),
+ OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
+ parse_vdso_pathnames),
OPT_BOOLEAN('f', "force", &symbol_conf.force, "don't complain, do it"),
OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
"load module symbols - WARNING: use only with -k and LIVE kernel"),
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index c16224b1fef3..2e358922a8d1 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -3965,6 +3965,8 @@ int cmd_script(int argc, const char **argv)
"file", "vmlinux pathname"),
OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
"file", "kallsyms pathname"),
+ OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
+ parse_vdso_pathnames),
OPT_BOOLEAN('G', "hide-call-graph", &no_callchain,
"When printing symbols do not display call chain"),
OPT_CALLBACK(0, "symfs", NULL, "directory",
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 1d6aef51c122..a3cce4e76eb9 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -1479,6 +1479,8 @@ int cmd_top(int argc, const char **argv)
"file", "kallsyms pathname"),
OPT_BOOLEAN('K', "hide_kernel_symbols", &top.hide_kernel_symbols,
"hide kernel symbols"),
+ OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
+ parse_vdso_pathnames),
OPT_CALLBACK('m', "mmap-pages", &opts->mmap_pages, "pages",
"number of mmap data pages", evlist__parse_mmap_pages),
OPT_INTEGER('r', "realtime", &top.realtime_prio,
diff --git a/tools/perf/util/disasm.c b/tools/perf/util/disasm.c
index 72aec8f61b94..8d9d78d3150c 100644
--- a/tools/perf/util/disasm.c
+++ b/tools/perf/util/disasm.c
@@ -16,6 +16,7 @@
#include "debug.h"
#include "disasm.h"
#include "dso.h"
+#include "vdso.h"
#include "env.h"
#include "evsel.h"
#include "map.h"
@@ -1126,7 +1127,7 @@ static int dso__disassemble_filename(struct dso *dso, char *filename, size_t fil
if (pos && strlen(pos) < SBUILD_ID_SIZE - 2)
dirname(build_id_path);
- if (dso__is_kcore(dso))
+ if (dso__is_kcore(dso) || dso__is_vdso(dso))
goto fallback;
len = readlink(build_id_path, linkname, sizeof(linkname) - 1);
@@ -1134,7 +1135,7 @@ static int dso__disassemble_filename(struct dso *dso, char *filename, size_t fil
goto fallback;
linkname[len] = '\0';
- if (strstr(linkname, DSO__NAME_KALLSYMS) ||
+ if (strstr(linkname, DSO__NAME_KALLSYMS) || strstr(linkname, DSO__NAME_VDSO) ||
access(filename, R_OK)) {
fallback:
/*
@@ -1142,7 +1143,7 @@ static int dso__disassemble_filename(struct dso *dso, char *filename, size_t fil
* cache, or is just a kallsyms file, well, lets hope that this
* DSO is the same as when 'perf record' ran.
*/
- if (dso__kernel(dso) && dso__long_name(dso)[0] == '/')
+ if ((dso__kernel(dso) || dso__is_vdso(dso)) && dso__long_name(dso)[0] == '/')
snprintf(filename, filename_size, "%s", dso__long_name(dso));
else
__symbol__join_symfs(filename, filename_size, dso__long_name(dso));
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 9e5940b5bc59..a90c647d37e1 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -19,6 +19,7 @@
#include "build-id.h"
#include "cap.h"
#include "dso.h"
+#include "vdso.h"
#include "util.h" // lsdir()
#include "debug.h"
#include "event.h"
@@ -44,6 +45,7 @@
static int dso__load_kernel_sym(struct dso *dso, struct map *map);
static int dso__load_guest_kernel_sym(struct dso *dso, struct map *map);
+static int dso__load_vdso_sym(struct dso *dso, struct map *map);
static bool symbol__is_idle(const char *name);
int vmlinux_path__nr_entries;
@@ -1833,6 +1835,12 @@ int dso__load(struct dso *dso, struct map *map)
goto out;
}
+ if (dso__is_vdso(dso)) {
+ ret = dso__load_vdso_sym(dso, map);
+ if (ret > 0)
+ goto out;
+ }
+
dso__set_adjust_symbols(dso, false);
if (perfmap) {
@@ -2017,12 +2025,14 @@ int dso__load_vmlinux(struct dso *dso, struct map *map,
dso__set_binary_type(dso, DSO_BINARY_TYPE__VMLINUX);
err = dso__load_sym(dso, map, &ss, &ss, 0);
- symsrc__destroy(&ss);
-
if (err > 0) {
dso__set_loaded(dso);
pr_debug("Using %s for symbols\n", symfs_vmlinux);
+
+ if (symsrc__has_symtab(&ss) && !dso__symsrc_filename(dso))
+ dso__set_symsrc_filename(dso, strdup(symfs_vmlinux));
}
+ symsrc__destroy(&ss);
return err;
}
@@ -2349,6 +2359,74 @@ static int vmlinux_path__init(struct perf_env *env)
return -1;
}
+int parse_vdso_pathnames(const struct option *opt __maybe_unused,
+ const char *arg, int unset __maybe_unused)
+{
+ char *tmp, *tok, *str = strdup(arg);
+ unsigned int i = 0;
+
+ for (tok = strtok_r(str, ",", &tmp); tok && i < ARRAY_SIZE(symbol_conf.vdso_name);
+ tok = strtok_r(NULL, ",", &tmp)) {
+ symbol_conf.vdso_name[i++] = strdup(tok);
+ }
+
+ free(str);
+ return 0;
+}
+
+static int dso__load_vdso(struct dso *dso, struct map *map,
+ const char *vdso)
+{
+ int err = -1;
+ struct symsrc ss;
+ char symfs_vdso[PATH_MAX];
+
+ if (vdso[0] == '/')
+ snprintf(symfs_vdso, sizeof(symfs_vdso), "%s", vdso);
+ else
+ symbol__join_symfs(symfs_vdso, vdso);
+
+ if (symsrc__init(&ss, dso, symfs_vdso, DSO_BINARY_TYPE__SYSTEM_PATH_DSO))
+ return -1;
+
+ /*
+ * dso__load_sym() may copy 'dso' which will result in the copies having
+ * an incorrect long name unless we set it here first.
+ */
+ dso__set_long_name(dso, vdso, false);
+ dso__set_binary_type(dso, DSO_BINARY_TYPE__SYSTEM_PATH_DSO);
+
+ err = dso__load_sym(dso, map, &ss, &ss, 0);
+ if (err > 0) {
+ dso__set_loaded(dso);
+ pr_debug("Using %s for %s symbols\n", symfs_vdso, dso__short_name(dso));
+
+ if (symsrc__has_symtab(&ss) && !dso__symsrc_filename(dso))
+ dso__set_symsrc_filename(dso, strdup(symfs_vdso));
+ }
+ symsrc__destroy(&ss);
+
+ return err;
+}
+
+static int dso__load_vdso_sym(struct dso *dso, struct map *map)
+{
+ int ret;
+
+ if (!dso__is_vdso(dso))
+ return -1;
+
+ for (unsigned int i = 0; i < ARRAY_SIZE(symbol_conf.vdso_name); i++) {
+ if (symbol_conf.vdso_name[i] != NULL) {
+ ret = dso__load_vdso(dso, map, symbol_conf.vdso_name[i]);
+ if (ret > 0)
+ return ret;
+ }
+ }
+
+ return -1;
+}
+
int setup_list(struct strlist **list, const char *list_str,
const char *list_name)
{
diff --git a/tools/perf/util/symbol_conf.h b/tools/perf/util/symbol_conf.h
index c114bbceef40..108356e3c981 100644
--- a/tools/perf/util/symbol_conf.h
+++ b/tools/perf/util/symbol_conf.h
@@ -3,6 +3,7 @@
#define __PERF_SYMBOL_CONF 1
#include <stdbool.h>
+#include <subcmd/parse-options.h>
struct strlist;
struct intlist;
@@ -55,6 +56,7 @@ struct symbol_conf {
const char *default_guest_vmlinux_name,
*default_guest_kallsyms,
*default_guest_modules;
+ const char *vdso_name[2];
const char *guestmount;
const char *dso_list_str,
*comm_list_str,
@@ -85,4 +87,7 @@ struct symbol_conf {
extern struct symbol_conf symbol_conf;
+int parse_vdso_pathnames(const struct option *opt __maybe_unused,
+ const char *arg, int unset __maybe_unused);
+
#endif // __PERF_SYMBOL_CONF
--
2.34.1
On 25/06/24 06:37, Changbin Du wrote:
> The vdso dumped from process memory (in buildid-cache) lacks debugging
> info. To annotate vdso symbols with source lines we need specify a
> debugging version.
>
> For x86, we can find them from your local build as
> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
> the buildid has to match.
>
> $ sudo perf record -a
> $ sudo perf report --objdump=llvm-objdump \
> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
>
> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> Percent│ movq -48(%rbp),%rsi
> │ testq %rax,%rax
> │ ; return vread_hvclock();
> │ movq %rax,%rdx
> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
> │ ↑ js eb
> │ ↑ jmp 74
> │ ; ts->tv_sec = vdso_ts->sec;
> 0.02 │147: leaq 2(%rbx),%rax
> │ shlq $4, %rax
> │ addq %r10,%rax
> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
> 9.38 │152: movl (%r10),%ecx
>
> When doing cross platform analysis, we also need specify the vdso path if
> we are interested in its symbols.
Would it be possible to add vdso and vdso debug to the build-id
cache and ensure perf can find it there?
Typically, getting dsos from another machine is handled via
build-id cache e.g. what perf-archive does
>
> v2: update documentation.
>
> Signed-off-by: Changbin Du <changbin.du@huawei.com>
> ---
> tools/perf/Documentation/perf-annotate.txt | 3 +
> tools/perf/Documentation/perf-c2c.txt | 3 +
> tools/perf/Documentation/perf-inject.txt | 3 +
> tools/perf/Documentation/perf-report.txt | 3 +
> tools/perf/Documentation/perf-script.txt | 3 +
> tools/perf/Documentation/perf-top.txt | 3 +
> tools/perf/builtin-annotate.c | 2 +
> tools/perf/builtin-c2c.c | 2 +
> tools/perf/builtin-inject.c | 2 +
> tools/perf/builtin-report.c | 2 +
> tools/perf/builtin-script.c | 2 +
> tools/perf/builtin-top.c | 2 +
> tools/perf/util/disasm.c | 7 +-
> tools/perf/util/symbol.c | 82 +++++++++++++++++++++-
> tools/perf/util/symbol_conf.h | 5 ++
> 15 files changed, 119 insertions(+), 5 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-annotate.txt b/tools/perf/Documentation/perf-annotate.txt
> index b95524bea021..4b6692f9a793 100644
> --- a/tools/perf/Documentation/perf-annotate.txt
> +++ b/tools/perf/Documentation/perf-annotate.txt
> @@ -58,6 +58,9 @@ OPTIONS
> --ignore-vmlinux::
> Ignore vmlinux files.
>
> +--vdso=<vdso1[,vdso2]>::
> + Specify vdso pathnames. You can specify up to two for multiarch-support.
> +
> --itrace::
> Options for decoding instruction tracing data. The options are:
>
> diff --git a/tools/perf/Documentation/perf-c2c.txt b/tools/perf/Documentation/perf-c2c.txt
> index 856f0dfb8e5a..7c07efca7542 100644
> --- a/tools/perf/Documentation/perf-c2c.txt
> +++ b/tools/perf/Documentation/perf-c2c.txt
> @@ -71,6 +71,9 @@ REPORT OPTIONS
> --vmlinux=<file>::
> vmlinux pathname
>
> +--vdso=<vdso1[,vdso2]>::
> + Specify vdso pathnames. You can specify up to two for multiarch-support.
> +
> -v::
> --verbose::
> Be more verbose (show counter open errors, etc).
> diff --git a/tools/perf/Documentation/perf-inject.txt b/tools/perf/Documentation/perf-inject.txt
> index c972032f4ca0..3c88967b4c7f 100644
> --- a/tools/perf/Documentation/perf-inject.txt
> +++ b/tools/perf/Documentation/perf-inject.txt
> @@ -62,6 +62,9 @@ OPTIONS
> --kallsyms=<file>::
> kallsyms pathname
>
> +--vdso=<vdso1[,vdso2]>::
> + Specify vdso pathnames. You can specify up to two for multiarch-support.
> +
> --itrace::
> Decode Instruction Tracing data, replacing it with synthesized events.
> Options are:
> diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
> index d2b1593ef700..8a3ba5f74cac 100644
> --- a/tools/perf/Documentation/perf-report.txt
> +++ b/tools/perf/Documentation/perf-report.txt
> @@ -345,6 +345,9 @@ OPTIONS
> Load module symbols. WARNING: This should only be used with -k and
> a LIVE kernel.
>
> +--vdso=<vdso1[,vdso2]>::
> + Specify vdso pathnames. You can specify up to two for multiarch-support.
> +
> -f::
> --force::
> Don't do ownership validation.
> diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
> index ff086ef05a0c..48f9974ca4c5 100644
> --- a/tools/perf/Documentation/perf-script.txt
> +++ b/tools/perf/Documentation/perf-script.txt
> @@ -296,6 +296,9 @@ OPTIONS
> --kallsyms=<file>::
> kallsyms pathname
>
> +--vdso=<vdso1[,vdso2]>::
> + Specify vdso pathnames. You can specify up to two for multiarch-support.
> +
> --symfs=<directory>::
> Look for files with symbols relative to this directory.
>
> diff --git a/tools/perf/Documentation/perf-top.txt b/tools/perf/Documentation/perf-top.txt
> index a754875fa5bb..1cf3ee16a0c1 100644
> --- a/tools/perf/Documentation/perf-top.txt
> +++ b/tools/perf/Documentation/perf-top.txt
> @@ -76,6 +76,9 @@ Default is to monitor all CPUS.
> --kallsyms=<file>::
> kallsyms pathname
>
> +--vdso=<vdso1[,vdso2]>::
> + Specify vdso pathnames. You can specify up to two for multiarch-support.
> +
> -m <pages>::
> --mmap-pages=<pages>::
> Number of mmap data pages (must be a power of two) or size
> diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
> index 50d2fb222d48..ff466882065d 100644
> --- a/tools/perf/builtin-annotate.c
> +++ b/tools/perf/builtin-annotate.c
> @@ -742,6 +742,8 @@ int cmd_annotate(int argc, const char **argv)
> "file", "vmlinux pathname"),
> OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
> "load module symbols - WARNING: use only with -k and LIVE kernel"),
> + OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
> + parse_vdso_pathnames),
> OPT_BOOLEAN('l', "print-line", &annotate_opts.print_lines,
> "print matching source lines (may be slow)"),
> OPT_BOOLEAN('P', "full-paths", &annotate_opts.full_path,
> diff --git a/tools/perf/builtin-c2c.c b/tools/perf/builtin-c2c.c
> index c157bd31f2e5..4764f9139661 100644
> --- a/tools/perf/builtin-c2c.c
> +++ b/tools/perf/builtin-c2c.c
> @@ -3018,6 +3018,8 @@ static int perf_c2c__report(int argc, const char **argv)
> const struct option options[] = {
> OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name,
> "file", "vmlinux pathname"),
> + OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
> + parse_vdso_pathnames),
> OPT_STRING('i', "input", &input_name, "file",
> "the input file to process"),
> OPT_INCR('N', "node-info", &c2c.node_info,
> diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
> index a212678d47be..e774e83d0a0f 100644
> --- a/tools/perf/builtin-inject.c
> +++ b/tools/perf/builtin-inject.c
> @@ -2247,6 +2247,8 @@ int cmd_inject(int argc, const char **argv)
> "don't load vmlinux even if found"),
> OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name, "file",
> "kallsyms pathname"),
> + OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
> + parse_vdso_pathnames),
> OPT_BOOLEAN('f', "force", &data.force, "don't complain, do it"),
> OPT_CALLBACK_OPTARG(0, "itrace", &inject.itrace_synth_opts,
> NULL, "opts", "Instruction Tracing options\n"
> diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> index 69618fb0110b..a64b48460dce 100644
> --- a/tools/perf/builtin-report.c
> +++ b/tools/perf/builtin-report.c
> @@ -1324,6 +1324,8 @@ int cmd_report(int argc, const char **argv)
> "don't load vmlinux even if found"),
> OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
> "file", "kallsyms pathname"),
> + OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
> + parse_vdso_pathnames),
> OPT_BOOLEAN('f', "force", &symbol_conf.force, "don't complain, do it"),
> OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
> "load module symbols - WARNING: use only with -k and LIVE kernel"),
> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
> index c16224b1fef3..2e358922a8d1 100644
> --- a/tools/perf/builtin-script.c
> +++ b/tools/perf/builtin-script.c
> @@ -3965,6 +3965,8 @@ int cmd_script(int argc, const char **argv)
> "file", "vmlinux pathname"),
> OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
> "file", "kallsyms pathname"),
> + OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
> + parse_vdso_pathnames),
> OPT_BOOLEAN('G', "hide-call-graph", &no_callchain,
> "When printing symbols do not display call chain"),
> OPT_CALLBACK(0, "symfs", NULL, "directory",
> diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
> index 1d6aef51c122..a3cce4e76eb9 100644
> --- a/tools/perf/builtin-top.c
> +++ b/tools/perf/builtin-top.c
> @@ -1479,6 +1479,8 @@ int cmd_top(int argc, const char **argv)
> "file", "kallsyms pathname"),
> OPT_BOOLEAN('K', "hide_kernel_symbols", &top.hide_kernel_symbols,
> "hide kernel symbols"),
> + OPT_CALLBACK(0, "vdso", NULL, "vdso1[,vdso2]", "vdso pathnames",
> + parse_vdso_pathnames),
> OPT_CALLBACK('m', "mmap-pages", &opts->mmap_pages, "pages",
> "number of mmap data pages", evlist__parse_mmap_pages),
> OPT_INTEGER('r', "realtime", &top.realtime_prio,
> diff --git a/tools/perf/util/disasm.c b/tools/perf/util/disasm.c
> index 72aec8f61b94..8d9d78d3150c 100644
> --- a/tools/perf/util/disasm.c
> +++ b/tools/perf/util/disasm.c
> @@ -16,6 +16,7 @@
> #include "debug.h"
> #include "disasm.h"
> #include "dso.h"
> +#include "vdso.h"
> #include "env.h"
> #include "evsel.h"
> #include "map.h"
> @@ -1126,7 +1127,7 @@ static int dso__disassemble_filename(struct dso *dso, char *filename, size_t fil
> if (pos && strlen(pos) < SBUILD_ID_SIZE - 2)
> dirname(build_id_path);
>
> - if (dso__is_kcore(dso))
> + if (dso__is_kcore(dso) || dso__is_vdso(dso))
> goto fallback;
>
> len = readlink(build_id_path, linkname, sizeof(linkname) - 1);
> @@ -1134,7 +1135,7 @@ static int dso__disassemble_filename(struct dso *dso, char *filename, size_t fil
> goto fallback;
>
> linkname[len] = '\0';
> - if (strstr(linkname, DSO__NAME_KALLSYMS) ||
> + if (strstr(linkname, DSO__NAME_KALLSYMS) || strstr(linkname, DSO__NAME_VDSO) ||
> access(filename, R_OK)) {
> fallback:
> /*
> @@ -1142,7 +1143,7 @@ static int dso__disassemble_filename(struct dso *dso, char *filename, size_t fil
> * cache, or is just a kallsyms file, well, lets hope that this
> * DSO is the same as when 'perf record' ran.
> */
> - if (dso__kernel(dso) && dso__long_name(dso)[0] == '/')
> + if ((dso__kernel(dso) || dso__is_vdso(dso)) && dso__long_name(dso)[0] == '/')
> snprintf(filename, filename_size, "%s", dso__long_name(dso));
> else
> __symbol__join_symfs(filename, filename_size, dso__long_name(dso));
> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> index 9e5940b5bc59..a90c647d37e1 100644
> --- a/tools/perf/util/symbol.c
> +++ b/tools/perf/util/symbol.c
> @@ -19,6 +19,7 @@
> #include "build-id.h"
> #include "cap.h"
> #include "dso.h"
> +#include "vdso.h"
> #include "util.h" // lsdir()
> #include "debug.h"
> #include "event.h"
> @@ -44,6 +45,7 @@
>
> static int dso__load_kernel_sym(struct dso *dso, struct map *map);
> static int dso__load_guest_kernel_sym(struct dso *dso, struct map *map);
> +static int dso__load_vdso_sym(struct dso *dso, struct map *map);
> static bool symbol__is_idle(const char *name);
>
> int vmlinux_path__nr_entries;
> @@ -1833,6 +1835,12 @@ int dso__load(struct dso *dso, struct map *map)
> goto out;
> }
>
> + if (dso__is_vdso(dso)) {
> + ret = dso__load_vdso_sym(dso, map);
> + if (ret > 0)
> + goto out;
> + }
> +
> dso__set_adjust_symbols(dso, false);
>
> if (perfmap) {
> @@ -2017,12 +2025,14 @@ int dso__load_vmlinux(struct dso *dso, struct map *map,
> dso__set_binary_type(dso, DSO_BINARY_TYPE__VMLINUX);
>
> err = dso__load_sym(dso, map, &ss, &ss, 0);
> - symsrc__destroy(&ss);
> -
> if (err > 0) {
> dso__set_loaded(dso);
> pr_debug("Using %s for symbols\n", symfs_vmlinux);
> +
> + if (symsrc__has_symtab(&ss) && !dso__symsrc_filename(dso))
> + dso__set_symsrc_filename(dso, strdup(symfs_vmlinux));
> }
> + symsrc__destroy(&ss);
>
> return err;
> }
> @@ -2349,6 +2359,74 @@ static int vmlinux_path__init(struct perf_env *env)
> return -1;
> }
>
> +int parse_vdso_pathnames(const struct option *opt __maybe_unused,
> + const char *arg, int unset __maybe_unused)
> +{
> + char *tmp, *tok, *str = strdup(arg);
> + unsigned int i = 0;
> +
> + for (tok = strtok_r(str, ",", &tmp); tok && i < ARRAY_SIZE(symbol_conf.vdso_name);
> + tok = strtok_r(NULL, ",", &tmp)) {
> + symbol_conf.vdso_name[i++] = strdup(tok);
> + }
> +
> + free(str);
> + return 0;
> +}
> +
> +static int dso__load_vdso(struct dso *dso, struct map *map,
> + const char *vdso)
> +{
> + int err = -1;
> + struct symsrc ss;
> + char symfs_vdso[PATH_MAX];
> +
> + if (vdso[0] == '/')
> + snprintf(symfs_vdso, sizeof(symfs_vdso), "%s", vdso);
> + else
> + symbol__join_symfs(symfs_vdso, vdso);
> +
> + if (symsrc__init(&ss, dso, symfs_vdso, DSO_BINARY_TYPE__SYSTEM_PATH_DSO))
> + return -1;
> +
> + /*
> + * dso__load_sym() may copy 'dso' which will result in the copies having
> + * an incorrect long name unless we set it here first.
> + */
> + dso__set_long_name(dso, vdso, false);
> + dso__set_binary_type(dso, DSO_BINARY_TYPE__SYSTEM_PATH_DSO);
> +
> + err = dso__load_sym(dso, map, &ss, &ss, 0);
> + if (err > 0) {
> + dso__set_loaded(dso);
> + pr_debug("Using %s for %s symbols\n", symfs_vdso, dso__short_name(dso));
> +
> + if (symsrc__has_symtab(&ss) && !dso__symsrc_filename(dso))
> + dso__set_symsrc_filename(dso, strdup(symfs_vdso));
> + }
> + symsrc__destroy(&ss);
> +
> + return err;
> +}
> +
> +static int dso__load_vdso_sym(struct dso *dso, struct map *map)
> +{
> + int ret;
> +
> + if (!dso__is_vdso(dso))
> + return -1;
> +
> + for (unsigned int i = 0; i < ARRAY_SIZE(symbol_conf.vdso_name); i++) {
> + if (symbol_conf.vdso_name[i] != NULL) {
> + ret = dso__load_vdso(dso, map, symbol_conf.vdso_name[i]);
> + if (ret > 0)
> + return ret;
> + }
> + }
> +
> + return -1;
> +}
> +
> int setup_list(struct strlist **list, const char *list_str,
> const char *list_name)
> {
> diff --git a/tools/perf/util/symbol_conf.h b/tools/perf/util/symbol_conf.h
> index c114bbceef40..108356e3c981 100644
> --- a/tools/perf/util/symbol_conf.h
> +++ b/tools/perf/util/symbol_conf.h
> @@ -3,6 +3,7 @@
> #define __PERF_SYMBOL_CONF 1
>
> #include <stdbool.h>
> +#include <subcmd/parse-options.h>
>
> struct strlist;
> struct intlist;
> @@ -55,6 +56,7 @@ struct symbol_conf {
> const char *default_guest_vmlinux_name,
> *default_guest_kallsyms,
> *default_guest_modules;
> + const char *vdso_name[2];
> const char *guestmount;
> const char *dso_list_str,
> *comm_list_str,
> @@ -85,4 +87,7 @@ struct symbol_conf {
>
> extern struct symbol_conf symbol_conf;
>
> +int parse_vdso_pathnames(const struct option *opt __maybe_unused,
> + const char *arg, int unset __maybe_unused);
> +
> #endif // __PERF_SYMBOL_CONF
On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
> On 25/06/24 06:37, Changbin Du wrote:
> > The vdso dumped from process memory (in buildid-cache) lacks debugging
> > info. To annotate vdso symbols with source lines we need specify a
> > debugging version.
> >
> > For x86, we can find them from your local build as
> > arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
> > /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
> > the buildid has to match.
> >
> > $ sudo perf record -a
> > $ sudo perf report --objdump=llvm-objdump \
> > --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
> >
> > Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> > __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> > Percent│ movq -48(%rbp),%rsi
> > │ testq %rax,%rax
> > │ ; return vread_hvclock();
> > │ movq %rax,%rdx
> > │ ; if (unlikely(!vdso_cycles_ok(cycles)))
> > │ ↑ js eb
> > │ ↑ jmp 74
> > │ ; ts->tv_sec = vdso_ts->sec;
> > 0.02 │147: leaq 2(%rbx),%rax
> > │ shlq $4, %rax
> > │ addq %r10,%rax
> > │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
> > 9.38 │152: movl (%r10),%ecx
> >
> > When doing cross platform analysis, we also need specify the vdso path if
> > we are interested in its symbols.
>
> Would it be possible to add vdso and vdso debug to the build-id
> cache and ensure perf can find it there?
>
> Typically, getting dsos from another machine is handled via
> build-id cache e.g. what perf-archive does
>
Hmm. I agree this is better alternative approach for cross-machine analysis.
When collecting vdsos to buildid cache, I think we can use the local searched
objects (with debug symbols) instead if its build-id matches vdsos from process
dumping (the real code ran).
Currently I just follow what perf does for vmlinux so to reuse most of existing
code. Maybe vmlinux is too big to add to buildid-cahce?
Can we keep our current strategy for now? I'll think about above options when
I have more time.
--
Cheers,
Changbin Du
On 26/06/24 05:26, duchangbin wrote:
> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
>> On 25/06/24 06:37, Changbin Du wrote:
>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
>>> info. To annotate vdso symbols with source lines we need specify a
>>> debugging version.
>>>
>>> For x86, we can find them from your local build as
>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
>>> the buildid has to match.
>>>
>>> $ sudo perf record -a
>>> $ sudo perf report --objdump=llvm-objdump \
>>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
>>>
>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
>>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
>>> Percent│ movq -48(%rbp),%rsi
>>> │ testq %rax,%rax
>>> │ ; return vread_hvclock();
>>> │ movq %rax,%rdx
>>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
>>> │ ↑ js eb
>>> │ ↑ jmp 74
>>> │ ; ts->tv_sec = vdso_ts->sec;
>>> 0.02 │147: leaq 2(%rbx),%rax
>>> │ shlq $4, %rax
>>> │ addq %r10,%rax
>>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
>>> 9.38 │152: movl (%r10),%ecx
>>>
>>> When doing cross platform analysis, we also need specify the vdso path if
>>> we are interested in its symbols.
>>
>> Would it be possible to add vdso and vdso debug to the build-id
>> cache and ensure perf can find it there?
>>
>> Typically, getting dsos from another machine is handled via
>> build-id cache e.g. what perf-archive does
>>
> Hmm. I agree this is better alternative approach for cross-machine analysis.
> When collecting vdsos to buildid cache, I think we can use the local searched
> objects (with debug symbols) instead if its build-id matches vdsos from process
> dumping (the real code ran).
>
> Currently I just follow what perf does for vmlinux so to reuse most of existing
> code. Maybe vmlinux is too big to add to buildid-cahce?
>
> Can we keep our current strategy for now? I'll think about above options when
> I have more time.
>
I tried adding vdso via perf buildid-cache. It doesn't work only
because the lookup expects the basename to be "vdso" but it is
"elf".
Adding a link from "vdso" to "elf" made it work e.g.
$ cat gettimeofday-test.c
#include <stdio.h>
#include <sys/time.h>
int main()
{
struct timeval tv;
int ret;
ret = gettimeofday(&tv, NULL);
if (ret == -1) {
fprintf(stderr, "gettimeofday failed\n");
return 1;
}
printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
return 0;
$ perf record -e intel_pt//u ./gettimeofday-test
1719397042.892837
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.026 MB perf.data ]
$ perf script --itrace=e
$ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
$ perf script --itrace=e
Warning:
2 instruction trace errors
instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
$ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
$ perf script --itrace=e
Warning:
2 instruction trace errors
instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
$ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
total 36
-rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
-rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
$ perf script --itrace=e
$
So maybe a change could be made to build_id_cache__add() to add
the extra link if the file name matches vdso
Hello guys,
On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
> On 26/06/24 05:26, duchangbin wrote:
> > On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
> >> On 25/06/24 06:37, Changbin Du wrote:
> >>> The vdso dumped from process memory (in buildid-cache) lacks debugging
> >>> info. To annotate vdso symbols with source lines we need specify a
> >>> debugging version.
> >>>
> >>> For x86, we can find them from your local build as
> >>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
> >>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
> >>> the buildid has to match.
> >>>
> >>> $ sudo perf record -a
> >>> $ sudo perf report --objdump=llvm-objdump \
> >>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
> >>>
> >>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> >>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> >>> Percent│ movq -48(%rbp),%rsi
> >>> │ testq %rax,%rax
> >>> │ ; return vread_hvclock();
> >>> │ movq %rax,%rdx
> >>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
> >>> │ ↑ js eb
> >>> │ ↑ jmp 74
> >>> │ ; ts->tv_sec = vdso_ts->sec;
> >>> 0.02 │147: leaq 2(%rbx),%rax
> >>> │ shlq $4, %rax
> >>> │ addq %r10,%rax
> >>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
> >>> 9.38 │152: movl (%r10),%ecx
> >>>
> >>> When doing cross platform analysis, we also need specify the vdso path if
> >>> we are interested in its symbols.
> >>
> >> Would it be possible to add vdso and vdso debug to the build-id
> >> cache and ensure perf can find it there?
> >>
> >> Typically, getting dsos from another machine is handled via
> >> build-id cache e.g. what perf-archive does
> >>
> > Hmm. I agree this is better alternative approach for cross-machine analysis.
> > When collecting vdsos to buildid cache, I think we can use the local searched
> > objects (with debug symbols) instead if its build-id matches vdsos from process
> > dumping (the real code ran).
> >
> > Currently I just follow what perf does for vmlinux so to reuse most of existing
> > code. Maybe vmlinux is too big to add to buildid-cahce?
> >
> > Can we keep our current strategy for now? I'll think about above options when
> > I have more time.
> >
>
> I tried adding vdso via perf buildid-cache. It doesn't work only
> because the lookup expects the basename to be "vdso" but it is
> "elf".
>
> Adding a link from "vdso" to "elf" made it work e.g.
>
> $ cat gettimeofday-test.c
> #include <stdio.h>
> #include <sys/time.h>
>
> int main()
> {
> struct timeval tv;
> int ret;
>
> ret = gettimeofday(&tv, NULL);
> if (ret == -1) {
> fprintf(stderr, "gettimeofday failed\n");
> return 1;
> }
>
> printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
>
> return 0;
> $ perf record -e intel_pt//u ./gettimeofday-test
> 1719397042.892837
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.026 MB perf.data ]
> $ perf script --itrace=e
> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> $ perf script --itrace=e
> Warning:
> 2 instruction trace errors
> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> $ perf script --itrace=e
> Warning:
> 2 instruction trace errors
> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
> total 36
> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
> -rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
> lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
> $ perf script --itrace=e
> $
>
> So maybe a change could be made to build_id_cache__add() to add
> the extra link if the file name matches vdso
Thanks for doing this! I noticed buildid_cache__basename() will handle
the name properly once it realizes the file is a vdso. Maybe we can
check the filepath with some patterns, or simply add an command line
option to say it's a vdso.
Thanks,
Namhyung
On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
> Hello guys,
>
> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
> > On 26/06/24 05:26, duchangbin wrote:
> > > On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
> > >> On 25/06/24 06:37, Changbin Du wrote:
> > >>> The vdso dumped from process memory (in buildid-cache) lacks debugging
> > >>> info. To annotate vdso symbols with source lines we need specify a
> > >>> debugging version.
> > >>>
> > >>> For x86, we can find them from your local build as
> > >>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
> > >>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
> > >>> the buildid has to match.
> > >>>
> > >>> $ sudo perf record -a
> > >>> $ sudo perf report --objdump=llvm-objdump \
> > >>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
> > >>>
> > >>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> > >>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> > >>> Percent│ movq -48(%rbp),%rsi
> > >>> │ testq %rax,%rax
> > >>> │ ; return vread_hvclock();
> > >>> │ movq %rax,%rdx
> > >>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
> > >>> │ ↑ js eb
> > >>> │ ↑ jmp 74
> > >>> │ ; ts->tv_sec = vdso_ts->sec;
> > >>> 0.02 │147: leaq 2(%rbx),%rax
> > >>> │ shlq $4, %rax
> > >>> │ addq %r10,%rax
> > >>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
> > >>> 9.38 │152: movl (%r10),%ecx
> > >>>
> > >>> When doing cross platform analysis, we also need specify the vdso path if
> > >>> we are interested in its symbols.
> > >>
> > >> Would it be possible to add vdso and vdso debug to the build-id
> > >> cache and ensure perf can find it there?
> > >>
> > >> Typically, getting dsos from another machine is handled via
> > >> build-id cache e.g. what perf-archive does
> > >>
> > > Hmm. I agree this is better alternative approach for cross-machine analysis.
> > > When collecting vdsos to buildid cache, I think we can use the local searched
> > > objects (with debug symbols) instead if its build-id matches vdsos from process
> > > dumping (the real code ran).
> > >
> > > Currently I just follow what perf does for vmlinux so to reuse most of existing
> > > code. Maybe vmlinux is too big to add to buildid-cahce?
> > >
> > > Can we keep our current strategy for now? I'll think about above options when
> > > I have more time.
> > >
> >
> > I tried adding vdso via perf buildid-cache. It doesn't work only
> > because the lookup expects the basename to be "vdso" but it is
> > "elf".
> >
> > Adding a link from "vdso" to "elf" made it work e.g.
> >
> > $ cat gettimeofday-test.c
> > #include <stdio.h>
> > #include <sys/time.h>
> >
> > int main()
> > {
> > struct timeval tv;
> > int ret;
> >
> > ret = gettimeofday(&tv, NULL);
> > if (ret == -1) {
> > fprintf(stderr, "gettimeofday failed\n");
> > return 1;
> > }
> >
> > printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
> >
> > return 0;
> > $ perf record -e intel_pt//u ./gettimeofday-test
> > 1719397042.892837
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.026 MB perf.data ]
> > $ perf script --itrace=e
> > $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> > $ perf script --itrace=e
> > Warning:
> > 2 instruction trace errors
> > instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> > instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> > $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> > $ perf script --itrace=e
> > Warning:
> > 2 instruction trace errors
> > instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> > instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> > $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
> > ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
> > ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
> > total 36
> > -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
> > -rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
> > lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
> > /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
> > $ perf script --itrace=e
> > $
> >
> > So maybe a change could be made to build_id_cache__add() to add
> > the extra link if the file name matches vdso
>
> Thanks for doing this! I noticed buildid_cache__basename() will handle
> the name properly once it realizes the file is a vdso. Maybe we can
> check the filepath with some patterns, or simply add an command line
> option to say it's a vdso.
>
> Thanks,
> Namhyung
>
I added some tricks for vdso in build_id_cache__add(). It replace vdso object
with debugging one if found. Is this okay?
+static char *build_id_cache__find_debug_vdso(const char *sbuild_id)
+{
+ char sbuild_id_tmp[SBUILD_ID_SIZE];
+ struct build_id bid;
+ int i, ret = 0;
+
+ printf("Looking at the vdso_path (%d entries long)\n",
+ vdso_paths.nr_entries + 1);
+
+ for (i = 0; i < vdso_paths.nr_entries; ++i) {
+ ret = filename__read_build_id(vdso_paths.paths[i], &bid);
+ if (ret < 0)
+ continue;
+
+ build_id__sprintf(&bid, sbuild_id_tmp);
+ if (!strcmp(sbuild_id, sbuild_id_tmp)) {
+ printf("Found debugging vdso %s\n", vdso_paths.paths[i]);
+ return strdup(vdso_paths.paths[i]);
+ }
+ }
+
+ return NULL;
+}
+
int
build_id_cache__add(const char *sbuild_id, const char *name, const char *realname,
struct nsinfo *nsi, bool is_kallsyms, bool is_vdso,
const char *proper_name, const char *root_dir)
{
const size_t size = PATH_MAX;
- char *filename = NULL, *dir_name = NULL, *linkname = zalloc(size), *tmp;
+ char *filename = NULL, *dir_name = NULL, *vdso_name = NULL, *linkname = zalloc(size), *tmp;
char *debugfile = NULL;
int err = -1;
+ /* replace vdso object with debugging one if found */
+ if (is_vdso) {
+ vdso_name = build_id_cache__find_debug_vdso(sbuild_id);
+ if (vdso_name)
+ name = realname = vdso_name;
+ }
+
if (!proper_name)
proper_name = name;
>
--
Cheers,
Changbin Du
On 28/06/24 07:21, duchangbin wrote:
> On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
>> Hello guys,
>>
>> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
>>> On 26/06/24 05:26, duchangbin wrote:
>>>> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
>>>>> On 25/06/24 06:37, Changbin Du wrote:
>>>>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
>>>>>> info. To annotate vdso symbols with source lines we need specify a
>>>>>> debugging version.
>>>>>>
>>>>>> For x86, we can find them from your local build as
>>>>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
>>>>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
>>>>>> the buildid has to match.
>>>>>>
>>>>>> $ sudo perf record -a
>>>>>> $ sudo perf report --objdump=llvm-objdump \
>>>>>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
>>>>>>
>>>>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
>>>>>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
>>>>>> Percent│ movq -48(%rbp),%rsi
>>>>>> │ testq %rax,%rax
>>>>>> │ ; return vread_hvclock();
>>>>>> │ movq %rax,%rdx
>>>>>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
>>>>>> │ ↑ js eb
>>>>>> │ ↑ jmp 74
>>>>>> │ ; ts->tv_sec = vdso_ts->sec;
>>>>>> 0.02 │147: leaq 2(%rbx),%rax
>>>>>> │ shlq $4, %rax
>>>>>> │ addq %r10,%rax
>>>>>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
>>>>>> 9.38 │152: movl (%r10),%ecx
>>>>>>
>>>>>> When doing cross platform analysis, we also need specify the vdso path if
>>>>>> we are interested in its symbols.
>>>>>
>>>>> Would it be possible to add vdso and vdso debug to the build-id
>>>>> cache and ensure perf can find it there?
>>>>>
>>>>> Typically, getting dsos from another machine is handled via
>>>>> build-id cache e.g. what perf-archive does
>>>>>
>>>> Hmm. I agree this is better alternative approach for cross-machine analysis.
>>>> When collecting vdsos to buildid cache, I think we can use the local searched
>>>> objects (with debug symbols) instead if its build-id matches vdsos from process
>>>> dumping (the real code ran).
>>>>
>>>> Currently I just follow what perf does for vmlinux so to reuse most of existing
>>>> code. Maybe vmlinux is too big to add to buildid-cahce?
>>>>
>>>> Can we keep our current strategy for now? I'll think about above options when
>>>> I have more time.
>>>>
>>>
>>> I tried adding vdso via perf buildid-cache. It doesn't work only
>>> because the lookup expects the basename to be "vdso" but it is
>>> "elf".
>>>
>>> Adding a link from "vdso" to "elf" made it work e.g.
>>>
>>> $ cat gettimeofday-test.c
>>> #include <stdio.h>
>>> #include <sys/time.h>
>>>
>>> int main()
>>> {
>>> struct timeval tv;
>>> int ret;
>>>
>>> ret = gettimeofday(&tv, NULL);
>>> if (ret == -1) {
>>> fprintf(stderr, "gettimeofday failed\n");
>>> return 1;
>>> }
>>>
>>> printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
>>>
>>> return 0;
>>> $ perf record -e intel_pt//u ./gettimeofday-test
>>> 1719397042.892837
>>> [ perf record: Woken up 1 times to write data ]
>>> [ perf record: Captured and wrote 0.026 MB perf.data ]
>>> $ perf script --itrace=e
>>> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>> $ perf script --itrace=e
>>> Warning:
>>> 2 instruction trace errors
>>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>> $ perf script --itrace=e
>>> Warning:
>>> 2 instruction trace errors
>>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
>>> total 36
>>> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
>>> -rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
>>> lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
>>> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
>>> $ perf script --itrace=e
>>> $
>>>
>>> So maybe a change could be made to build_id_cache__add() to add
>>> the extra link if the file name matches vdso
>>
>> Thanks for doing this! I noticed buildid_cache__basename() will handle
>> the name properly once it realizes the file is a vdso. Maybe we can
>> check the filepath with some patterns, or simply add an command line
>> option to say it's a vdso.
>>
>> Thanks,
>> Namhyung
>>
> I added some tricks for vdso in build_id_cache__add(). It replace vdso object
> with debugging one if found. Is this okay?
perf has support for storing debug files in the build-id
cache using the base name "debug" instead of "elf", so really
it would be better to make that work
>
> +static char *build_id_cache__find_debug_vdso(const char *sbuild_id)
> +{
> + char sbuild_id_tmp[SBUILD_ID_SIZE];
> + struct build_id bid;
> + int i, ret = 0;
> +
> + printf("Looking at the vdso_path (%d entries long)\n",
> + vdso_paths.nr_entries + 1);
> +
> + for (i = 0; i < vdso_paths.nr_entries; ++i) {
> + ret = filename__read_build_id(vdso_paths.paths[i], &bid);
> + if (ret < 0)
> + continue;
> +
> + build_id__sprintf(&bid, sbuild_id_tmp);
> + if (!strcmp(sbuild_id, sbuild_id_tmp)) {
> + printf("Found debugging vdso %s\n", vdso_paths.paths[i]);
> + return strdup(vdso_paths.paths[i]);
> + }
> + }
> +
> + return NULL;
> +}
> +
> int
> build_id_cache__add(const char *sbuild_id, const char *name, const char *realname,
> struct nsinfo *nsi, bool is_kallsyms, bool is_vdso,
> const char *proper_name, const char *root_dir)
> {
> const size_t size = PATH_MAX;
> - char *filename = NULL, *dir_name = NULL, *linkname = zalloc(size), *tmp;
> + char *filename = NULL, *dir_name = NULL, *vdso_name = NULL, *linkname = zalloc(size), *tmp;
> char *debugfile = NULL;
> int err = -1;
>
> + /* replace vdso object with debugging one if found */
> + if (is_vdso) {
> + vdso_name = build_id_cache__find_debug_vdso(sbuild_id);
> + if (vdso_name)
> + name = realname = vdso_name;
> + }
> +
> if (!proper_name)
> proper_name = name;
>
>>
>
On Fri, Jun 28, 2024 at 08:27:55PM +0300, Adrian Hunter wrote:
> On 28/06/24 07:21, duchangbin wrote:
> > On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
> >> Hello guys,
> >>
> >> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
> >>> On 26/06/24 05:26, duchangbin wrote:
> >>>> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
> >>>>> On 25/06/24 06:37, Changbin Du wrote:
> >>>>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
> >>>>>> info. To annotate vdso symbols with source lines we need specify a
> >>>>>> debugging version.
> >>>>>>
> >>>>>> For x86, we can find them from your local build as
> >>>>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
> >>>>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
> >>>>>> the buildid has to match.
> >>>>>>
> >>>>>> $ sudo perf record -a
> >>>>>> $ sudo perf report --objdump=llvm-objdump \
> >>>>>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
> >>>>>>
> >>>>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> >>>>>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> >>>>>> Percent│ movq -48(%rbp),%rsi
> >>>>>> │ testq %rax,%rax
> >>>>>> │ ; return vread_hvclock();
> >>>>>> │ movq %rax,%rdx
> >>>>>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
> >>>>>> │ ↑ js eb
> >>>>>> │ ↑ jmp 74
> >>>>>> │ ; ts->tv_sec = vdso_ts->sec;
> >>>>>> 0.02 │147: leaq 2(%rbx),%rax
> >>>>>> │ shlq $4, %rax
> >>>>>> │ addq %r10,%rax
> >>>>>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
> >>>>>> 9.38 │152: movl (%r10),%ecx
> >>>>>>
> >>>>>> When doing cross platform analysis, we also need specify the vdso path if
> >>>>>> we are interested in its symbols.
> >>>>>
> >>>>> Would it be possible to add vdso and vdso debug to the build-id
> >>>>> cache and ensure perf can find it there?
> >>>>>
> >>>>> Typically, getting dsos from another machine is handled via
> >>>>> build-id cache e.g. what perf-archive does
> >>>>>
> >>>> Hmm. I agree this is better alternative approach for cross-machine analysis.
> >>>> When collecting vdsos to buildid cache, I think we can use the local searched
> >>>> objects (with debug symbols) instead if its build-id matches vdsos from process
> >>>> dumping (the real code ran).
> >>>>
> >>>> Currently I just follow what perf does for vmlinux so to reuse most of existing
> >>>> code. Maybe vmlinux is too big to add to buildid-cahce?
> >>>>
> >>>> Can we keep our current strategy for now? I'll think about above options when
> >>>> I have more time.
> >>>>
> >>>
> >>> I tried adding vdso via perf buildid-cache. It doesn't work only
> >>> because the lookup expects the basename to be "vdso" but it is
> >>> "elf".
> >>>
> >>> Adding a link from "vdso" to "elf" made it work e.g.
> >>>
> >>> $ cat gettimeofday-test.c
> >>> #include <stdio.h>
> >>> #include <sys/time.h>
> >>>
> >>> int main()
> >>> {
> >>> struct timeval tv;
> >>> int ret;
> >>>
> >>> ret = gettimeofday(&tv, NULL);
> >>> if (ret == -1) {
> >>> fprintf(stderr, "gettimeofday failed\n");
> >>> return 1;
> >>> }
> >>>
> >>> printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
> >>>
> >>> return 0;
> >>> $ perf record -e intel_pt//u ./gettimeofday-test
> >>> 1719397042.892837
> >>> [ perf record: Woken up 1 times to write data ]
> >>> [ perf record: Captured and wrote 0.026 MB perf.data ]
> >>> $ perf script --itrace=e
> >>> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> >>> $ perf script --itrace=e
> >>> Warning:
> >>> 2 instruction trace errors
> >>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> >>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> >>> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> >>> $ perf script --itrace=e
> >>> Warning:
> >>> 2 instruction trace errors
> >>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> >>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> >>> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
> >>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
> >>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
> >>> total 36
> >>> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
> >>> -rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
> >>> lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
> >>> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
> >>> $ perf script --itrace=e
> >>> $
> >>>
> >>> So maybe a change could be made to build_id_cache__add() to add
> >>> the extra link if the file name matches vdso
> >>
> >> Thanks for doing this! I noticed buildid_cache__basename() will handle
> >> the name properly once it realizes the file is a vdso. Maybe we can
> >> check the filepath with some patterns, or simply add an command line
> >> option to say it's a vdso.
> >>
> >> Thanks,
> >> Namhyung
> >>
> > I added some tricks for vdso in build_id_cache__add(). It replace vdso object
> > with debugging one if found. Is this okay?
>
> perf has support for storing debug files in the build-id
> cache using the base name "debug" instead of "elf", so really
> it would be better to make that work
>
My thoughts are:
1. add debugging vdso file as "debug" in buildid cache.
2. add a new perf config item 'annotate.prefer_debug_file' (default false) to
annotate with debugging object files.
--
Cheers,
Changbin Du
On 29/06/24 05:32, duchangbin wrote:
> On Fri, Jun 28, 2024 at 08:27:55PM +0300, Adrian Hunter wrote:
>> On 28/06/24 07:21, duchangbin wrote:
>>> On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
>>>> Hello guys,
>>>>
>>>> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
>>>>> On 26/06/24 05:26, duchangbin wrote:
>>>>>> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
>>>>>>> On 25/06/24 06:37, Changbin Du wrote:
>>>>>>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
>>>>>>>> info. To annotate vdso symbols with source lines we need specify a
>>>>>>>> debugging version.
>>>>>>>>
>>>>>>>> For x86, we can find them from your local build as
>>>>>>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
>>>>>>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
>>>>>>>> the buildid has to match.
>>>>>>>>
>>>>>>>> $ sudo perf record -a
>>>>>>>> $ sudo perf report --objdump=llvm-objdump \
>>>>>>>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
>>>>>>>>
>>>>>>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
>>>>>>>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
>>>>>>>> Percent│ movq -48(%rbp),%rsi
>>>>>>>> │ testq %rax,%rax
>>>>>>>> │ ; return vread_hvclock();
>>>>>>>> │ movq %rax,%rdx
>>>>>>>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
>>>>>>>> │ ↑ js eb
>>>>>>>> │ ↑ jmp 74
>>>>>>>> │ ; ts->tv_sec = vdso_ts->sec;
>>>>>>>> 0.02 │147: leaq 2(%rbx),%rax
>>>>>>>> │ shlq $4, %rax
>>>>>>>> │ addq %r10,%rax
>>>>>>>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
>>>>>>>> 9.38 │152: movl (%r10),%ecx
>>>>>>>>
>>>>>>>> When doing cross platform analysis, we also need specify the vdso path if
>>>>>>>> we are interested in its symbols.
>>>>>>>
>>>>>>> Would it be possible to add vdso and vdso debug to the build-id
>>>>>>> cache and ensure perf can find it there?
>>>>>>>
>>>>>>> Typically, getting dsos from another machine is handled via
>>>>>>> build-id cache e.g. what perf-archive does
>>>>>>>
>>>>>> Hmm. I agree this is better alternative approach for cross-machine analysis.
>>>>>> When collecting vdsos to buildid cache, I think we can use the local searched
>>>>>> objects (with debug symbols) instead if its build-id matches vdsos from process
>>>>>> dumping (the real code ran).
>>>>>>
>>>>>> Currently I just follow what perf does for vmlinux so to reuse most of existing
>>>>>> code. Maybe vmlinux is too big to add to buildid-cahce?
>>>>>>
>>>>>> Can we keep our current strategy for now? I'll think about above options when
>>>>>> I have more time.
>>>>>>
>>>>>
>>>>> I tried adding vdso via perf buildid-cache. It doesn't work only
>>>>> because the lookup expects the basename to be "vdso" but it is
>>>>> "elf".
>>>>>
>>>>> Adding a link from "vdso" to "elf" made it work e.g.
>>>>>
>>>>> $ cat gettimeofday-test.c
>>>>> #include <stdio.h>
>>>>> #include <sys/time.h>
>>>>>
>>>>> int main()
>>>>> {
>>>>> struct timeval tv;
>>>>> int ret;
>>>>>
>>>>> ret = gettimeofday(&tv, NULL);
>>>>> if (ret == -1) {
>>>>> fprintf(stderr, "gettimeofday failed\n");
>>>>> return 1;
>>>>> }
>>>>>
>>>>> printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
>>>>>
>>>>> return 0;
>>>>> $ perf record -e intel_pt//u ./gettimeofday-test
>>>>> 1719397042.892837
>>>>> [ perf record: Woken up 1 times to write data ]
>>>>> [ perf record: Captured and wrote 0.026 MB perf.data ]
>>>>> $ perf script --itrace=e
>>>>> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>>>> $ perf script --itrace=e
>>>>> Warning:
>>>>> 2 instruction trace errors
>>>>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>>>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>>>> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>>>> $ perf script --itrace=e
>>>>> Warning:
>>>>> 2 instruction trace errors
>>>>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>>>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>>>> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
>>>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
>>>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
>>>>> total 36
>>>>> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
>>>>> -rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
>>>>> lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
>>>>> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
>>>>> $ perf script --itrace=e
>>>>> $
>>>>>
>>>>> So maybe a change could be made to build_id_cache__add() to add
>>>>> the extra link if the file name matches vdso
>>>>
>>>> Thanks for doing this! I noticed buildid_cache__basename() will handle
>>>> the name properly once it realizes the file is a vdso. Maybe we can
>>>> check the filepath with some patterns, or simply add an command line
>>>> option to say it's a vdso.
>>>>
>>>> Thanks,
>>>> Namhyung
>>>>
>>> I added some tricks for vdso in build_id_cache__add(). It replace vdso object
>>> with debugging one if found. Is this okay?
>>
>> perf has support for storing debug files in the build-id
>> cache using the base name "debug" instead of "elf", so really
>> it would be better to make that work
>>
> My thoughts are:
> 1. add debugging vdso file as "debug" in buildid cache.
> 2. add a new perf config item 'annotate.prefer_debug_file' (default false) to
> annotate with debugging object files.
Is that needed? Isn't the debug information read from
the "debug" file? If not, does that need fixing?
On Sat, Jun 29, 2024 at 09:47:10PM +0300, Adrian Hunter wrote:
> On 29/06/24 05:32, duchangbin wrote:
> > On Fri, Jun 28, 2024 at 08:27:55PM +0300, Adrian Hunter wrote:
> >> On 28/06/24 07:21, duchangbin wrote:
> >>> On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
> >>>> Hello guys,
> >>>>
> >>>> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
> >>>>> On 26/06/24 05:26, duchangbin wrote:
> >>>>>> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
> >>>>>>> On 25/06/24 06:37, Changbin Du wrote:
> >>>>>>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
> >>>>>>>> info. To annotate vdso symbols with source lines we need specify a
> >>>>>>>> debugging version.
> >>>>>>>>
> >>>>>>>> For x86, we can find them from your local build as
> >>>>>>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
> >>>>>>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
> >>>>>>>> the buildid has to match.
> >>>>>>>>
> >>>>>>>> $ sudo perf record -a
> >>>>>>>> $ sudo perf report --objdump=llvm-objdump \
> >>>>>>>> --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
> >>>>>>>>
> >>>>>>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> >>>>>>>> __vdso_clock_gettime /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> >>>>>>>> Percent│ movq -48(%rbp),%rsi
> >>>>>>>> │ testq %rax,%rax
> >>>>>>>> │ ; return vread_hvclock();
> >>>>>>>> │ movq %rax,%rdx
> >>>>>>>> │ ; if (unlikely(!vdso_cycles_ok(cycles)))
> >>>>>>>> │ ↑ js eb
> >>>>>>>> │ ↑ jmp 74
> >>>>>>>> │ ; ts->tv_sec = vdso_ts->sec;
> >>>>>>>> 0.02 │147: leaq 2(%rbx),%rax
> >>>>>>>> │ shlq $4, %rax
> >>>>>>>> │ addq %r10,%rax
> >>>>>>>> │ ; while ((seq = READ_ONCE(vd->seq)) & 1) {
> >>>>>>>> 9.38 │152: movl (%r10),%ecx
> >>>>>>>>
> >>>>>>>> When doing cross platform analysis, we also need specify the vdso path if
> >>>>>>>> we are interested in its symbols.
> >>>>>>>
> >>>>>>> Would it be possible to add vdso and vdso debug to the build-id
> >>>>>>> cache and ensure perf can find it there?
> >>>>>>>
> >>>>>>> Typically, getting dsos from another machine is handled via
> >>>>>>> build-id cache e.g. what perf-archive does
> >>>>>>>
> >>>>>> Hmm. I agree this is better alternative approach for cross-machine analysis.
> >>>>>> When collecting vdsos to buildid cache, I think we can use the local searched
> >>>>>> objects (with debug symbols) instead if its build-id matches vdsos from process
> >>>>>> dumping (the real code ran).
> >>>>>>
> >>>>>> Currently I just follow what perf does for vmlinux so to reuse most of existing
> >>>>>> code. Maybe vmlinux is too big to add to buildid-cahce?
> >>>>>>
> >>>>>> Can we keep our current strategy for now? I'll think about above options when
> >>>>>> I have more time.
> >>>>>>
> >>>>>
> >>>>> I tried adding vdso via perf buildid-cache. It doesn't work only
> >>>>> because the lookup expects the basename to be "vdso" but it is
> >>>>> "elf".
> >>>>>
> >>>>> Adding a link from "vdso" to "elf" made it work e.g.
> >>>>>
> >>>>> $ cat gettimeofday-test.c
> >>>>> #include <stdio.h>
> >>>>> #include <sys/time.h>
> >>>>>
> >>>>> int main()
> >>>>> {
> >>>>> struct timeval tv;
> >>>>> int ret;
> >>>>>
> >>>>> ret = gettimeofday(&tv, NULL);
> >>>>> if (ret == -1) {
> >>>>> fprintf(stderr, "gettimeofday failed\n");
> >>>>> return 1;
> >>>>> }
> >>>>>
> >>>>> printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
> >>>>>
> >>>>> return 0;
> >>>>> $ perf record -e intel_pt//u ./gettimeofday-test
> >>>>> 1719397042.892837
> >>>>> [ perf record: Woken up 1 times to write data ]
> >>>>> [ perf record: Captured and wrote 0.026 MB perf.data ]
> >>>>> $ perf script --itrace=e
> >>>>> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> >>>>> $ perf script --itrace=e
> >>>>> Warning:
> >>>>> 2 instruction trace errors
> >>>>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> >>>>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> >>>>> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
> >>>>> $ perf script --itrace=e
> >>>>> Warning:
> >>>>> 2 instruction trace errors
> >>>>> instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
> >>>>> instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
> >>>>> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
> >>>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
> >>>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
> >>>>> total 36
> >>>>> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
> >>>>> -rw-r----- 1 ahunter ahunter 0 Jun 26 13:17 probes
> >>>>> lrwxrwxrwx 1 ahunter ahunter 3 Jun 26 13:18 vdso -> elf
> >>>>> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
> >>>>> $ perf script --itrace=e
> >>>>> $
> >>>>>
> >>>>> So maybe a change could be made to build_id_cache__add() to add
> >>>>> the extra link if the file name matches vdso
> >>>>
> >>>> Thanks for doing this! I noticed buildid_cache__basename() will handle
> >>>> the name properly once it realizes the file is a vdso. Maybe we can
> >>>> check the filepath with some patterns, or simply add an command line
> >>>> option to say it's a vdso.
> >>>>
> >>>> Thanks,
> >>>> Namhyung
> >>>>
> >>> I added some tricks for vdso in build_id_cache__add(). It replace vdso object
> >>> with debugging one if found. Is this okay?
> >>
> >> perf has support for storing debug files in the build-id
> >> cache using the base name "debug" instead of "elf", so really
> >> it would be better to make that work
> >>
> > My thoughts are:
> > 1. add debugging vdso file as "debug" in buildid cache.
> > 2. add a new perf config item 'annotate.prefer_debug_file' (default false) to
> > annotate with debugging object files.
>
> Is that needed? Isn't the debug information read from
> the "debug" file? If not, does that need fixing?
>
Is it by design as you mentioned before?
https://lore.kernel.org/r/all/395cfff7-9692-4123-96b6-353752007f46@intel.com/:
"In the past, there have been cases where the debugging version has not
worked for reading object code. I don't remember the details, but the
symbols and debugging information was OK while the object code was not.
In general, using anything other than the file that was actually executed
for reading object code seems like a bad idea."
I don't know whether this really matter mostly. I can fix it if it's a bug.
https://lore.kernel.org/r/all/395cfff7-9692-4123-96b6-353752007f46@intel.com/
>
--
Cheers,
Changbin Du
© 2016 - 2025 Red Hat, Inc.