The vdso dumped from process memory (in buildid-cache) lacks debugging
info. To annotate vdso symbols with source lines we need a debugging
version.

For x86, we can find them from your local build as
'arch/x86/entry/vdso/vdso{32,64}.so.dbg'. Or they may resides in
'/lib/modules/<version>/vdso/vdso{32,64}.so' on Ubuntu. But notice that the
builid has to match.

If user doesn't specify the path, perf will search them internally as long
as vmlinux when recording samples. The searched debugging vdso will add to
buildid cache as ~/.debug/[vdso]/debug.

Below samples are captured on my local build kernel. perf succesfully
find debugging version vdso and we can annotate with source without
specifying vdso path.

$ sudo perf record -a
$ sudo perf report --objdump=llvm-objdump

Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
__vdso_clock_gettime  /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
Percent│       movq    -48(%rbp),%rsi
       │       testq   %rax,%rax
       │     ;               return vread_hvclock();
       │       movq    %rax,%rdx
       │     ;               if (unlikely(!vdso_cycles_ok(cycles)))
       │     ↑ js      eb
       │     ↑ jmp     74
       │     ;               ts->tv_sec = vdso_ts->sec;
  0.02 │147:   leaq    2(%rbx),%rax
       │       shlq    $4, %rax
       │       addq    %r10,%rax
       │     ;               while ((seq = READ_ONCE(vd->seq)) & 1) {
  9.38 │152:   movl    (%r10),%ecx

When doing cross platform analysis, we need to specify the vdso path manually if
we are interested in its symbols, because they are not installed on local
machine. At most two vdso can be given. Also you can pack your buildid cache with
perf-archive if the debugging vdso can be found on the sampled machine.

$ sudo perf report --objdump=llvm-objdump \
      --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg

I also improved perf-buildid-cache command to recognize vdso when adding files,
then place it at correct place.

$ sudo perf buildid-cache --add /work/linux/arch/x86/entry/vdso/vdso64.so.dbg

Then it will be copied to ~/.debug/[vdso]/debug.

v6:
  - split "perf: build-id: try to search debugging vdso and add to cache" 
    by functional logical. (suggested by Adrian)
v5:
  - Searching the vdso in record stage instead of report. So the debugging
    vdso will be in build-id cache. This is friendly for cross-machine analysis.
  - Improve perf-buildid-cache command recognize vdso when adding files
v4:
  - split the refactoring from the actual change.
v3:
  - update documentation.
v2:
  - now search vdso automatically as long as vmlinux, as suggested by Adrian.
  - remove change 'prefer symsrc_filename for filename'.

Changbin Du (8):
  perf: support specify vdso path in cmdline
  perf: disasm: refactor function dso__disassemble_filename
  perf: disasm: use build_id_path if fallback failed
  perf: symbol: generalize vmlinux path searching
  perf: build-id: add support for build-id cache vdso debug
  perf: build-id: extend build_id_cache__find_debug() to find local
    debugging vdso
  perf: disasm: prefer debugging files in build-id cache
  perf buildid-cache: recognize vdso when adding files

 tools/perf/Documentation/perf-annotate.txt |   3 +
 tools/perf/Documentation/perf-c2c.txt      |   3 +
 tools/perf/Documentation/perf-inject.txt   |   3 +
 tools/perf/Documentation/perf-report.txt   |   3 +
 tools/perf/Documentation/perf-script.txt   |   3 +
 tools/perf/Documentation/perf-top.txt      |   3 +
 tools/perf/builtin-annotate.c              |   2 +
 tools/perf/builtin-buildid-cache.c         |  26 ++-
 tools/perf/builtin-c2c.c                   |   2 +
 tools/perf/builtin-inject.c                |   2 +
 tools/perf/builtin-report.c                |   2 +
 tools/perf/builtin-script.c                |   2 +
 tools/perf/builtin-top.c                   |   2 +
 tools/perf/util/build-id.c                 |  57 +++++-
 tools/perf/util/disasm.c                   | 131 ++++++++-----
 tools/perf/util/machine.c                  |   4 +-
 tools/perf/util/symbol.c                   | 209 ++++++++++++++++-----
 tools/perf/util/symbol.h                   |   9 +-
 tools/perf/util/symbol_conf.h              |   5 +
 19 files changed, 359 insertions(+), 112 deletions(-)

-- 
2.34.1

Hi Changbin,

On Fri, Aug 16, 2024 at 06:58:03PM +0800, Changbin Du wrote:
> The vdso dumped from process memory (in buildid-cache) lacks debugging
> info. To annotate vdso symbols with source lines we need a debugging
> version.
> 
> For x86, we can find them from your local build as
> 'arch/x86/entry/vdso/vdso{32,64}.so.dbg'. Or they may resides in
> '/lib/modules/<version>/vdso/vdso{32,64}.so' on Ubuntu. But notice that the
> builid has to match.
> 
> If user doesn't specify the path, perf will search them internally as long
> as vmlinux when recording samples. The searched debugging vdso will add to
> buildid cache as ~/.debug/[vdso]/debug.
> 
> Below samples are captured on my local build kernel. perf succesfully
> find debugging version vdso and we can annotate with source without
> specifying vdso path.
> 
> $ sudo perf record -a
> $ sudo perf report --objdump=llvm-objdump
> 
> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
> __vdso_clock_gettime  /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
> Percent│       movq    -48(%rbp),%rsi
>        │       testq   %rax,%rax
>        │     ;               return vread_hvclock();
>        │       movq    %rax,%rdx
>        │     ;               if (unlikely(!vdso_cycles_ok(cycles)))
>        │     ↑ js      eb
>        │     ↑ jmp     74
>        │     ;               ts->tv_sec = vdso_ts->sec;
>   0.02 │147:   leaq    2(%rbx),%rax
>        │       shlq    $4, %rax
>        │       addq    %r10,%rax
>        │     ;               while ((seq = READ_ONCE(vd->seq)) & 1) {
>   9.38 │152:   movl    (%r10),%ecx
> 
> When doing cross platform analysis, we need to specify the vdso path manually if
> we are interested in its symbols, because they are not installed on local
> machine. At most two vdso can be given. Also you can pack your buildid cache with
> perf-archive if the debugging vdso can be found on the sampled machine.
> 
> $ sudo perf report --objdump=llvm-objdump \
>       --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
With latest upstream kernel, I still can't annotate symbols in vdso.
With your patch and this --vdso command, it works great on x86 and arm64
platform. So
   Tested-by: Yin Fengwei <fengwei_yin@linux.alibaba.com>


Regards
Yin, Fengwei

> 
> I also improved perf-buildid-cache command to recognize vdso when adding files,
> then place it at correct place.
> 
> $ sudo perf buildid-cache --add /work/linux/arch/x86/entry/vdso/vdso64.so.dbg