Linking against libcapstone and libLLVM can be a significant increase
in dependencies and size of memory footprint. For something like `perf
record` the disassembler and addr2line functionality won't be
used. Support dynamically loading these libraries using dlopen and
then calling the appropriate functions found using dlsym.

BUILD_NONDISTRO is used to build perf against the license incompatible
libbfd and libiberty libraries. As this has been opt-in for nearly 2
years, commit dd317df07207 ("perf build: Make binutil libraries opt
in"), remove the code to simplify the code base.

The patch series:
1) does some initial clean up;
2) moves the capstone and LLVM code to their own C files,
3) simplifies a little the capstone code;
4) adds perf_ variants of the functions that will either directly call
   the function or use dlsym to discover it;
5) adds BPF JIT disassembly support to LLVM and capstone disassembly;
6) removes the BUILD_NONDISTRO code, reduces scope and removes what's possible.

The addr2line LLVM functionality is written in C++. To avoid linking
against libLLVM for this, a new LIBLLVM_DYNAMIC option is added where
the C++ code with the libLLVM dependency will be built into a
libperf-llvm.so and that dlsym-ed and called against. Ideally LLVM
would extend their C API to avoid this.

The libbfd BPF disassembly supported source lines, this wasn't ported
to the capstone and LLVM disassembly.

v2: Add mangling of the function names in libperf-llvm.so to avoid
    potential infinite recursion. Add BPF JIT disassembly support to
    LLVM and capstone. Add/rebase the BUILD_NONDISTRO cleanup onto the
    series from:
    https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/
    Some other minor additional clean up.

Ian Rogers (17):
  perf build: Remove libtracefs configuration
  perf map: Constify objdump offset/address conversion APIs
  perf capstone: Move capstone functionality into its own file
  perf llvm: Move llvm functionality into its own file
  perf capstone: Remove open_capstone_handle
  perf capstone: Support for dlopen-ing libcapstone.so
  perf llvm: Support for dlopen-ing libLLVM.so
  perf llvm: Mangle libperf-llvm.so function names
  perf dso: Move read_symbol from llvm/capstone to dso
  perf dso: Support BPF programs in dso__read_symbol
  perf llvm: Disassemble cleanup
  perf dso: Clean up read_symbol error handling
  perf build: Remove libbfd support
  perf build: Remove libiberty support
  perf build: Remove unused defines
  perf disasm: Remove disasm_bpf
  perf disasm: Make ins__scnprintf and ins__is_nop static

 tools/perf/Documentation/perf-check.txt |   1 -
 tools/perf/Makefile.config              |  90 +---
 tools/perf/Makefile.perf                |  35 +-
 tools/perf/builtin-check.c              |   1 -
 tools/perf/builtin-script.c             |   2 -
 tools/perf/tests/Build                  |   1 -
 tools/perf/tests/builtin-test.c         |   1 -
 tools/perf/tests/make                   |   4 +-
 tools/perf/tests/pe-file-parsing.c      | 101 ----
 tools/perf/tests/tests.h                |   1 -
 tools/perf/util/Build                   |   5 +-
 tools/perf/util/annotate.h              |   1 -
 tools/perf/util/capstone.c              | 682 ++++++++++++++++++++++++
 tools/perf/util/capstone.h              |  24 +
 tools/perf/util/demangle-cxx.cpp        |  22 +-
 tools/perf/util/disasm.c                | 632 +---------------------
 tools/perf/util/disasm.h                |   5 +-
 tools/perf/util/disasm_bpf.c            | 195 -------
 tools/perf/util/disasm_bpf.h            |  12 -
 tools/perf/util/dso.c                   |  98 ++++
 tools/perf/util/dso.h                   |   4 +
 tools/perf/util/llvm-c-helpers.cpp      | 120 ++++-
 tools/perf/util/llvm-c-helpers.h        |  24 +-
 tools/perf/util/llvm.c                  | 489 +++++++++++++++++
 tools/perf/util/llvm.h                  |  24 +
 tools/perf/util/map.c                   |  19 +-
 tools/perf/util/map.h                   |   6 +-
 tools/perf/util/print_insn.c            | 117 +---
 tools/perf/util/srcline.c               | 306 +----------
 tools/perf/util/srcline.h               |   6 +
 tools/perf/util/symbol-elf.c            |  95 ----
 tools/perf/util/symbol.c                | 135 -----
 tools/perf/util/symbol.h                |   4 -
 33 files changed, 1552 insertions(+), 1710 deletions(-)
 delete mode 100644 tools/perf/tests/pe-file-parsing.c
 create mode 100644 tools/perf/util/capstone.c
 create mode 100644 tools/perf/util/capstone.h
 delete mode 100644 tools/perf/util/disasm_bpf.c
 delete mode 100644 tools/perf/util/disasm_bpf.h
 create mode 100644 tools/perf/util/llvm.c
 create mode 100644 tools/perf/util/llvm.h

-- 
2.48.0.rc2.279.g1de40edade-goog

On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote:
> Linking against libcapstone and libLLVM can be a significant increase
> in dependencies and size of memory footprint. For something like `perf
> record` the disassembler and addr2line functionality won't be
> used. Support dynamically loading these libraries using dlopen and
> then calling the appropriate functions found using dlsym.

It's unclear to me what this actually fixes. If the code is not used
it should not be faulted in and the dynamic linker is lazy too, so 
if it's not used, it won't even be linked. 

I don't see any numbers, but it won't surprise me if it improved
actual run time or memory usage significantly.

-Andi

On Wed, Jan 22, 2025 at 7:21 AM Andi Kleen <ak@linux.intel.com> wrote:
>
> On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote:
> > Linking against libcapstone and libLLVM can be a significant increase
> > in dependencies and size of memory footprint. For something like `perf
> > record` the disassembler and addr2line functionality won't be
> > used. Support dynamically loading these libraries using dlopen and
> > then calling the appropriate functions found using dlsym.
>
> It's unclear to me what this actually fixes. If the code is not used
> it should not be faulted in and the dynamic linker is lazy too, so
> if it's not used, it won't even be linked.
>
> I don't see any numbers, but it won't surprise me if it improved
> actual run time or memory usage significantly.

In certain scenarios, like data centers, it can be useful to
statically link all your dependencies to avoid dll hell. The X86
disassembler alone in libllvm is of a size comparable to the perf tool
- I think this speaks to us doing a reasonably good job of size
optimization of the events/metrics in the perf tool. We want these
dependencies for the performance over forking objdump and addr2line,
but we don't want it baked in - unless the person doing the build
wants this and this is still the default if the libraries are detected
by Makefile.config. Using dlopen also means distributions can have a
perf tool that doesn't drag in libLLVM.so and a universe of
dependencies, but when it is installed get the performance advantages.
In data centres having fast disassembly/addr2line is less of a
priority over the binary size cost replicated over 10,000s of machines
because those machines don't tend to be running the annotate/report
commands.

Fwiw, Namhyung's uftrace is doing something similar for python:
https://github.com/namhyung/uftrace/blob/master/utils/script-python.c#L139
and I wish the perf tool were also doing this. I think it is much
nicer to have the tool fail at runtime because of a missing dependency
which you can then install should you want it, rather than doing an
equivalent within the code base with #ifdefs and needing users to
recompile. This patch series significantly reduces the #ifdefs in
places like the core disasm code.

Thanks,
Ian

> In certain scenarios, like data centers, it can be useful to
> statically link all your dependencies to avoid dll hell.

Yes but it won't be loaded into memory if not used. Executable
loading is all lazy. Maybe look a page fault trace for loading
perf if you don't believe me.

So you're trying to optimize disk space here?

I didn't see that in the cover letter.

It doesn't seem like a very good reason for such an intrusive patch kit.

If it's a serious concern maybe investigate an executable compressor?

> The X86
> disassembler alone in libllvm is of a size comparable to the perf tool

I agree that LLVM is a serious bloat and DLL hell concern, but I don't think 
dlopen is the answer here.

-Andi

On Thu, Jan 23, 2025 at 10:19 AM Andi Kleen <ak@linux.intel.com> wrote:
>
> > In certain scenarios, like data centers, it can be useful to
> > statically link all your dependencies to avoid dll hell.
>
> Yes but it won't be loaded into memory if not used. Executable
> loading is all lazy. Maybe look a page fault trace for loading
> perf if you don't believe me.
>
> So you're trying to optimize disk space here?
>
> I didn't see that in the cover letter.

For me yes, for distributions it is dependencies. This is already in
the v3 message:
https://lore.kernel.org/lkml/20250122174308.350350-1-irogers@google.com/

> It doesn't seem like a very good reason for such an intrusive patch kit.

The capstone and LLVM code is preexisting. Moving the capstone/llvm
code to their own files isn't dependent on dlopen, it does make it
nicer to have a single place we're doing dlopen. The change to shim
the capstone/LLVM calls looks like this:
https://github.com/googleprodkernel/linux-perf/blob/google_tools_master/tools/perf/util/llvm.c#L160-L182
That is a shim is introduced that either calls through to the function
if we're linking against libcapstone/llvm or does the dlsym. There are
7 such functions in the LLVM code. I don't think shimming 7 functions
is at the scale of hugely intrusive.

> If it's a serious concern maybe investigate an executable compressor?

Perhaps just have a squashfs partition.

Fwiw, excluding dependencies I think compression on the events is a
good solution. Convert json events/metrics to a sysfs file with the
cpuid in the path, add the compressed file to the binary as data, find
"json" events by iterating the directories in the compressed file,
etc. A single filesystem approach to event lookup can mean we do some
kind of unionfs style lookup of events, which could support users
adding their own events/metrics in a directory. Zip doesn't support
compressing across files, which is something of a requirement here,
other formats do but it's a case of optimizing for some kind of
libarchive sweet spot. The opportunity here is that about 70% of the
binary is event encodings, a compressed file is about 30% of the
current binary size, so we could reduce the binary size by about 40%.

> > The X86
> > disassembler alone in libllvm is of a size comparable to the perf tool
>
> I agree that LLVM is a serious bloat and DLL hell concern, but I don't think
> dlopen is the answer here.

Agreed, but it's where the code is at. addr2line command or use LLVM
for some performance. I think having an inbuilt solution would be best
longer term - we spend energy trying to parse and understand text
output from tools/libraries when the information is just sitting there
in the instruction encoding. Such a solution would be brittle for
things like new dwarf information, so we may want to have fallbacks
like LLVM but having a loosely coupled dependency using dlopen feels
preferable there, to aid package maintainers.

Thanks,
Ian

On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote:
> Linking against libcapstone and libLLVM can be a significant increase
> in dependencies and size of memory footprint. For something like `perf
> record` the disassembler and addr2line functionality won't be
> used. Support dynamically loading these libraries using dlopen and
> then calling the appropriate functions found using dlsym.

It's not clear from the description how you would use dlopen/dlsym.
Based on an offline discussion, you want to leave the current linking
model as is, and to support dlopen/dlsym when it's NOT detected at
build-time, right?

For that, you need to carry some definitions of the functions and types
for the used APIs.  But I'm not sure if it's right to carry them in the
perf code base.

> 
> BUILD_NONDISTRO is used to build perf against the license incompatible
> libbfd and libiberty libraries. As this has been opt-in for nearly 2
> years, commit dd317df07207 ("perf build: Make binutil libraries opt
> in"), remove the code to simplify the code base.

This part can be a separate series.

> 
> The patch series:
> 1) does some initial clean up;
> 2) moves the capstone and LLVM code to their own C files,
> 3) simplifies a little the capstone code;

I like changes up to this in general.  Let me take a look at the
patches.

Thanks,
Namhyung


> 4) adds perf_ variants of the functions that will either directly call
>    the function or use dlsym to discover it;
> 5) adds BPF JIT disassembly support to LLVM and capstone disassembly;
> 6) removes the BUILD_NONDISTRO code, reduces scope and removes what's possible.
> 
> The addr2line LLVM functionality is written in C++. To avoid linking
> against libLLVM for this, a new LIBLLVM_DYNAMIC option is added where
> the C++ code with the libLLVM dependency will be built into a
> libperf-llvm.so and that dlsym-ed and called against. Ideally LLVM
> would extend their C API to avoid this.
> 
> The libbfd BPF disassembly supported source lines, this wasn't ported
> to the capstone and LLVM disassembly.
> 
> v2: Add mangling of the function names in libperf-llvm.so to avoid
>     potential infinite recursion. Add BPF JIT disassembly support to
>     LLVM and capstone. Add/rebase the BUILD_NONDISTRO cleanup onto the
>     series from:
>     https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/
>     Some other minor additional clean up.
> 
> Ian Rogers (17):
>   perf build: Remove libtracefs configuration
>   perf map: Constify objdump offset/address conversion APIs
>   perf capstone: Move capstone functionality into its own file
>   perf llvm: Move llvm functionality into its own file
>   perf capstone: Remove open_capstone_handle
>   perf capstone: Support for dlopen-ing libcapstone.so
>   perf llvm: Support for dlopen-ing libLLVM.so
>   perf llvm: Mangle libperf-llvm.so function names
>   perf dso: Move read_symbol from llvm/capstone to dso
>   perf dso: Support BPF programs in dso__read_symbol
>   perf llvm: Disassemble cleanup
>   perf dso: Clean up read_symbol error handling
>   perf build: Remove libbfd support
>   perf build: Remove libiberty support
>   perf build: Remove unused defines
>   perf disasm: Remove disasm_bpf
>   perf disasm: Make ins__scnprintf and ins__is_nop static
> 
>  tools/perf/Documentation/perf-check.txt |   1 -
>  tools/perf/Makefile.config              |  90 +---
>  tools/perf/Makefile.perf                |  35 +-
>  tools/perf/builtin-check.c              |   1 -
>  tools/perf/builtin-script.c             |   2 -
>  tools/perf/tests/Build                  |   1 -
>  tools/perf/tests/builtin-test.c         |   1 -
>  tools/perf/tests/make                   |   4 +-
>  tools/perf/tests/pe-file-parsing.c      | 101 ----
>  tools/perf/tests/tests.h                |   1 -
>  tools/perf/util/Build                   |   5 +-
>  tools/perf/util/annotate.h              |   1 -
>  tools/perf/util/capstone.c              | 682 ++++++++++++++++++++++++
>  tools/perf/util/capstone.h              |  24 +
>  tools/perf/util/demangle-cxx.cpp        |  22 +-
>  tools/perf/util/disasm.c                | 632 +---------------------
>  tools/perf/util/disasm.h                |   5 +-
>  tools/perf/util/disasm_bpf.c            | 195 -------
>  tools/perf/util/disasm_bpf.h            |  12 -
>  tools/perf/util/dso.c                   |  98 ++++
>  tools/perf/util/dso.h                   |   4 +
>  tools/perf/util/llvm-c-helpers.cpp      | 120 ++++-
>  tools/perf/util/llvm-c-helpers.h        |  24 +-
>  tools/perf/util/llvm.c                  | 489 +++++++++++++++++
>  tools/perf/util/llvm.h                  |  24 +
>  tools/perf/util/map.c                   |  19 +-
>  tools/perf/util/map.h                   |   6 +-
>  tools/perf/util/print_insn.c            | 117 +---
>  tools/perf/util/srcline.c               | 306 +----------
>  tools/perf/util/srcline.h               |   6 +
>  tools/perf/util/symbol-elf.c            |  95 ----
>  tools/perf/util/symbol.c                | 135 -----
>  tools/perf/util/symbol.h                |   4 -
>  33 files changed, 1552 insertions(+), 1710 deletions(-)
>  delete mode 100644 tools/perf/tests/pe-file-parsing.c
>  create mode 100644 tools/perf/util/capstone.c
>  create mode 100644 tools/perf/util/capstone.h
>  delete mode 100644 tools/perf/util/disasm_bpf.c
>  delete mode 100644 tools/perf/util/disasm_bpf.h
>  create mode 100644 tools/perf/util/llvm.c
>  create mode 100644 tools/perf/util/llvm.h
> 
> -- 
> 2.48.0.rc2.279.g1de40edade-goog
>

On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Tue, Jan 21, 2025 at 10:23:15PM -0800, Ian Rogers wrote:
> > Linking against libcapstone and libLLVM can be a significant increase
> > in dependencies and size of memory footprint. For something like `perf
> > record` the disassembler and addr2line functionality won't be
> > used. Support dynamically loading these libraries using dlopen and
> > then calling the appropriate functions found using dlsym.
>
> It's not clear from the description how you would use dlopen/dlsym.
> Based on an offline discussion, you want to leave the current linking
> model as is, and to support dlopen/dlsym when it's NOT detected at
> build-time, right?

Yep. Current behavior is no header file than these options fail, new
behavior is that we try to use dlopen/dlsym and fail if the dlopen
fails.

> For that, you need to carry some definitions of the functions and types
> for the used APIs.  But I'm not sure if it's right to carry them in the
> perf code base.

Right. I mention that here:
https://lore.kernel.org/lkml/CAP-5=fUhNuybCU-2_5EgcCwgwXnxvyFMvyhzKe=ZP1bssQwXHw@mail.gmail.com/
For LLVM we need 3 typedefs and 5 #defines, for capstone we need 2
structs and 5 enums (if we #ifdef some x86 only formatting code).

The problem in not carrying those definitions is:
1) if the header file isn't present a build won't support
LLVM/capstone even by dlopen - everything falls through to
objdump/addr2line that have known performance issues;
2) package maintainers either need to spot a warning message to
realize they've done this by having a missing header file (hard to
spot and brittle) or we require the build to fail and people without
capstone.h opt out of the build error with NO_CAPSTONE=1 - something
perf developers will probably not like;
3) the LLVM/capstone code needs #ifdefs and __maybe_unused to suppress
compiler warnings, or perhaps we have a minimal version of those
files, leading to extra code complexity.

I believe the approach here is no worse than what we do with vmlinux.h
for BPF code and is robust as depending on dlsym being able to look up
the function names. It is not perfect but I think it is more perfect
and less complex than the alternative.

> >
> > BUILD_NONDISTRO is used to build perf against the license incompatible
> > libbfd and libiberty libraries. As this has been opt-in for nearly 2
> > years, commit dd317df07207 ("perf build: Make binutil libraries opt
> > in"), remove the code to simplify the code base.
>
> This part can be a separate series.

Right, I posted it as a series here:
https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/
as mentioned in the v2 notes below. The issue was that Arnaldo pointed
out removing BUILD_NONDISTRO removed disassemble_bpf that had only
been implemented for libbfd. This series adds the LLVM/capstone
definitions built into the symbol__disassemble refactor. Merging that
series would conflict with this series, so I posted everything
together to avoid having series of patches depending upon one another.
I also wanted to check that what is in disasm.c in the end is
reasonable, which I believe it is with significantly reduced
ifdef-ery.

> >
> > The patch series:
> > 1) does some initial clean up;
> > 2) moves the capstone and LLVM code to their own C files,
> > 3) simplifies a little the capstone code;
>
> I like changes up to this in general.  Let me take a look at the
> patches.

Thanks,
Ian
Ian

> Thanks,
> Namhyung

On Thu, Jan 23, 2025 at 3:36 PM Ian Rogers <irogers@google.com> wrote:
> On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > I like changes up to this in general.  Let me take a look at the
> > patches.

So it would be nice to make progress with this series given some level
of happiness, I don't see any actions currently on the patch series as
is. If I may be so bold as to recap the issues that have come up:

1) Andi Kleen mentions that dlopen is inferior to linking against
libraries and those libraries aren't a memory overhead if unused.

I agree but pointed-out the data center use case means that saving
size on binaries can be important to some (me). We've also been trying
to reduce perf's dependencies for distributions as perf dragging in
say the whole of libLLVM can be annoying for making minimal
distributions that contain perf. Perhaps somebody (Arnaldo?) more
involved with distributions can confirm or deny the distribution
problem, I'm hoping it is self-evident.

2) Namhyung Kim was uncomfortable with the code defining
types/constants that were in header files as the two may drift over
time

I agree but in the same way as a function name is an ABI for dlysym,
the types/constants are too. Yes a header file may change, but in
doing so the ABI has changed and so it would be an incompatible change
and everything would be broken. We'd need to fix the code for this,
say as we did when libbpf moved to version 1.0, but using a header
file would only weakly guard against this problem. The problem with
including the header files is that then the build either breaks
without the header or we need to support a no linking against a
library and not using dlopen case. I suspect a lot of distributions
wouldn't understand the build subtlety in this, the necessary build
options and things installed, and we'd end up not using things like
libLLVM even when it is known to be a large performance win. I also
hope one day we can move from parsing text out of forked commands, as
it is slower and more brittle, to just directly using libraries.
Making dlopen the fallback (probably with a warning on failure) seems
like the right direction for this except we won't get it if we need to
drag in extra dependency header files for the build to succeed (well
we could have a no library or dlopen option, but then we'd probably
find distributions packaging this and things like perf annotate
getting broken as they don't even know how to dlopen a library).

3) Namhyung Kim (and I) also raises that the libcapstone patch can be
smaller by dropping the print_capstone_detail support on x86

Note, given the similarity between capstone and libLLVM for
disassembly, it is curious that only capstone gives the extra detail.

I agree. Given the capstone disassembly output will be compromised we
should warn for this, probably in Makefile.config to avoid running
afoul of -Werror. It isn't clear that having a warning is a good move
given the handful of structs needed to support print_capstone_detail.
I'd prefer to keep the structs so that we haven't got a warning that
looks like it needs cleaning up.

4) Namhyung Kim raised concerns over #if placement

Namhyung raised that he'd prefer:
```
#if HAVE_LIBCAPSTONE_SUPPORT
// lots of code
#else
// lots of code
#endif
```
rather than the #ifs being inside or around individual functions. I
raised that the large #ifs is a problem in the current code as you
lose context when trying to understand a function. You may look at a
function but not realize it isn't being used because of a #if 10s or
100s of lines above. Namhyung raised that the large #ifs is closer to
kernel style, I disagreed as I think kernel style is only doing this
when it stubs out a bunch of API functions, not when more context
would be useful. Hopefully as the person writing the patches the style
choice I've made can be respected.

5) Daniel Xu raised issues with the removal of libbfd for Rust
support, as the code implies libbfd C++ demangling is a pre-requisite
of legacy rust symbol demangling

A separate patch was posted adding Rust v0 symbol demangling with no
libbfd dependency:
https://lore.kernel.org/lkml/20250129193037.573431-1-irogers@google.com/
The legacy support should work with the non-libbfd demanglers as
that's what we have today. We should really clean up Rust demangling
and have tests. This is blocked on the Rust community responding to:
https://github.com/rust-lang/rust/issues/60705

Thanks,
Ian

On Mon, Feb 10, 2025 at 10:06 AM Ian Rogers <irogers@google.com> wrote:
>
> On Thu, Jan 23, 2025 at 3:36 PM Ian Rogers <irogers@google.com> wrote:
> > On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > I like changes up to this in general.  Let me take a look at the
> > > patches.
>
> So it would be nice to make progress with this series given some level
> of happiness, I don't see any actions currently on the patch series as
> is. If I may be so bold as to recap the issues that have come up:
>
> 1) Andi Kleen mentions that dlopen is inferior to linking against
> libraries and those libraries aren't a memory overhead if unused.
>
> I agree but pointed-out the data center use case means that saving
> size on binaries can be important to some (me). We've also been trying
> to reduce perf's dependencies for distributions as perf dragging in
> say the whole of libLLVM can be annoying for making minimal
> distributions that contain perf. Perhaps somebody (Arnaldo?) more
> involved with distributions can confirm or deny the distribution
> problem, I'm hoping it is self-evident.
>
> 2) Namhyung Kim was uncomfortable with the code defining
> types/constants that were in header files as the two may drift over
> time
>
> I agree but in the same way as a function name is an ABI for dlysym,
> the types/constants are too. Yes a header file may change, but in
> doing so the ABI has changed and so it would be an incompatible change
> and everything would be broken. We'd need to fix the code for this,
> say as we did when libbpf moved to version 1.0, but using a header
> file would only weakly guard against this problem. The problem with
> including the header files is that then the build either breaks
> without the header or we need to support a no linking against a
> library and not using dlopen case. I suspect a lot of distributions
> wouldn't understand the build subtlety in this, the necessary build
> options and things installed, and we'd end up not using things like
> libLLVM even when it is known to be a large performance win. I also
> hope one day we can move from parsing text out of forked commands, as
> it is slower and more brittle, to just directly using libraries.
> Making dlopen the fallback (probably with a warning on failure) seems
> like the right direction for this except we won't get it if we need to
> drag in extra dependency header files for the build to succeed (well
> we could have a no library or dlopen option, but then we'd probably
> find distributions packaging this and things like perf annotate
> getting broken as they don't even know how to dlopen a library).
>
> 3) Namhyung Kim (and I) also raises that the libcapstone patch can be
> smaller by dropping the print_capstone_detail support on x86
>
> Note, given the similarity between capstone and libLLVM for
> disassembly, it is curious that only capstone gives the extra detail.
>
> I agree. Given the capstone disassembly output will be compromised we
> should warn for this, probably in Makefile.config to avoid running
> afoul of -Werror. It isn't clear that having a warning is a good move
> given the handful of structs needed to support print_capstone_detail.
> I'd prefer to keep the structs so that we haven't got a warning that
> looks like it needs cleaning up.
>
> 4) Namhyung Kim raised concerns over #if placement
>
> Namhyung raised that he'd prefer:
> ```
> #if HAVE_LIBCAPSTONE_SUPPORT
> // lots of code
> #else
> // lots of code
> #endif
> ```
> rather than the #ifs being inside or around individual functions. I
> raised that the large #ifs is a problem in the current code as you
> lose context when trying to understand a function. You may look at a
> function but not realize it isn't being used because of a #if 10s or
> 100s of lines above. Namhyung raised that the large #ifs is closer to
> kernel style, I disagreed as I think kernel style is only doing this
> when it stubs out a bunch of API functions, not when more context
> would be useful. Hopefully as the person writing the patches the style
> choice I've made can be respected.
>
> 5) Daniel Xu raised issues with the removal of libbfd for Rust
> support, as the code implies libbfd C++ demangling is a pre-requisite
> of legacy rust symbol demangling
>
> A separate patch was posted adding Rust v0 symbol demangling with no
> libbfd dependency:
> https://lore.kernel.org/lkml/20250129193037.573431-1-irogers@google.com/
> The legacy support should work with the non-libbfd demanglers as
> that's what we have today. We should really clean up Rust demangling
> and have tests. This is blocked on the Rust community responding to:
> https://github.com/rust-lang/rust/issues/60705

Ping.

Thanks,
Ian

Hi Ian,

On Wed, Mar 12, 2025 at 02:04:30PM -0700, Ian Rogers wrote:
> On Mon, Feb 10, 2025 at 10:06 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Thu, Jan 23, 2025 at 3:36 PM Ian Rogers <irogers@google.com> wrote:
> > > On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > I like changes up to this in general.  Let me take a look at the
> > > > patches.
> >
> > So it would be nice to make progress with this series given some level
> > of happiness, I don't see any actions currently on the patch series as
> > is. If I may be so bold as to recap the issues that have come up:
> >
> > 1) Andi Kleen mentions that dlopen is inferior to linking against
> > libraries and those libraries aren't a memory overhead if unused.
> >
> > I agree but pointed-out the data center use case means that saving
> > size on binaries can be important to some (me). We've also been trying
> > to reduce perf's dependencies for distributions as perf dragging in
> > say the whole of libLLVM can be annoying for making minimal
> > distributions that contain perf. Perhaps somebody (Arnaldo?) more
> > involved with distributions can confirm or deny the distribution
> > problem, I'm hoping it is self-evident.
> >
> > 2) Namhyung Kim was uncomfortable with the code defining
> > types/constants that were in header files as the two may drift over
> > time
> >
> > I agree but in the same way as a function name is an ABI for dlysym,
> > the types/constants are too. Yes a header file may change, but in
> > doing so the ABI has changed and so it would be an incompatible change
> > and everything would be broken. We'd need to fix the code for this,
> > say as we did when libbpf moved to version 1.0, but using a header
> > file would only weakly guard against this problem. The problem with
> > including the header files is that then the build either breaks
> > without the header or we need to support a no linking against a
> > library and not using dlopen case. I suspect a lot of distributions
> > wouldn't understand the build subtlety in this, the necessary build
> > options and things installed, and we'd end up not using things like
> > libLLVM even when it is known to be a large performance win. I also
> > hope one day we can move from parsing text out of forked commands, as
> > it is slower and more brittle, to just directly using libraries.
> > Making dlopen the fallback (probably with a warning on failure) seems
> > like the right direction for this except we won't get it if we need to
> > drag in extra dependency header files for the build to succeed (well
> > we could have a no library or dlopen option, but then we'd probably
> > find distributions packaging this and things like perf annotate
> > getting broken as they don't even know how to dlopen a library).
> >
> > 3) Namhyung Kim (and I) also raises that the libcapstone patch can be
> > smaller by dropping the print_capstone_detail support on x86
> >
> > Note, given the similarity between capstone and libLLVM for
> > disassembly, it is curious that only capstone gives the extra detail.
> >
> > I agree. Given the capstone disassembly output will be compromised we
> > should warn for this, probably in Makefile.config to avoid running
> > afoul of -Werror. It isn't clear that having a warning is a good move
> > given the handful of structs needed to support print_capstone_detail.
> > I'd prefer to keep the structs so that we haven't got a warning that
> > looks like it needs cleaning up.
> >
> > 4) Namhyung Kim raised concerns over #if placement
> >
> > Namhyung raised that he'd prefer:
> > ```
> > #if HAVE_LIBCAPSTONE_SUPPORT
> > // lots of code
> > #else
> > // lots of code
> > #endif
> > ```
> > rather than the #ifs being inside or around individual functions. I
> > raised that the large #ifs is a problem in the current code as you
> > lose context when trying to understand a function. You may look at a
> > function but not realize it isn't being used because of a #if 10s or
> > 100s of lines above. Namhyung raised that the large #ifs is closer to
> > kernel style, I disagreed as I think kernel style is only doing this
> > when it stubs out a bunch of API functions, not when more context
> > would be useful. Hopefully as the person writing the patches the style
> > choice I've made can be respected.
> >
> > 5) Daniel Xu raised issues with the removal of libbfd for Rust
> > support, as the code implies libbfd C++ demangling is a pre-requisite
> > of legacy rust symbol demangling
> >
> > A separate patch was posted adding Rust v0 symbol demangling with no
> > libbfd dependency:
> > https://lore.kernel.org/lkml/20250129193037.573431-1-irogers@google.com/
> > The legacy support should work with the non-libbfd demanglers as
> > that's what we have today. We should really clean up Rust demangling
> > and have tests. This is blocked on the Rust community responding to:
> > https://github.com/rust-lang/rust/issues/60705

I think #ifdef placements is not a big deal, but I still don't want to
pull libcapstone details into the perf tree.

For LLVM, I think you should to build llvm-c-helpers anyway which means
you still need LLVM headers and don't need to redefine the structures.

Can we do the same for capstone?  I think it's best to use capstone
headers directly and add a build option to use dlopen().

Thanks,
Namhyung

On Thu, Mar 13, 2025 at 03:24:22PM -0700, Namhyung Kim wrote:
> I think #ifdef placements is not a big deal, but I still don't want to
> pull libcapstone details into the perf tree.
 
> For LLVM, I think you should to build llvm-c-helpers anyway which means
> you still need LLVM headers and don't need to redefine the structures.
 
> Can we do the same for capstone?  I think it's best to use capstone
> headers directly and add a build option to use dlopen().

My two cents: if one wants to support some library, then have its devel
packages available at build time.

Then, perf nowadays has lots of dependencies, we need to rein on that,
making the good to have but not always used things to be dlopen'ed.

Like we did with gtk (that at this point I think is really deprecated,
BTW).

gdb has prior art in this area that we could use, it is not even a TUI
but it asks if debuginfo should be used and if so it goes on on
potentially lenghty updates of the local buildid cache they keep (which
is not the one we use, it should be).

And in the recent discussion with Dmitry Vyukov the possibility doing a
question to the user about a default behaviour to be set and then using
.perfconfig not to bother anymore the user about things is part of
helping the user to deal with the myriad possibilites perf offers.

gdb could use that as well, why ask at every session if debuginfod
should be used? Annoying.

I think perf should try to use what is available, both at build and at
run time, and it shouldn't change the way it output things, but should
warn the user about recent developments, things we over time figured out
are problematic and thus a new default would be better, but then obtain
consent if the user cares about it, and allow for backtracking, to go
and change .perfconfig when the user realises the old output/behaviour
is not really nice.

But keeping the grass green as it used to be should be the priority.

- Arnaldo

On Fri, Mar 14, 2025 at 10:06 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Thu, Mar 13, 2025 at 03:24:22PM -0700, Namhyung Kim wrote:
> > I think #ifdef placements is not a big deal, but I still don't want to
> > pull libcapstone details into the perf tree.
>
> > For LLVM, I think you should to build llvm-c-helpers anyway which means
> > you still need LLVM headers and don't need to redefine the structures.
>
> > Can we do the same for capstone?  I think it's best to use capstone
> > headers directly and add a build option to use dlopen().
>
> My two cents: if one wants to support some library, then have its devel
> packages available at build time.
>
> Then, perf nowadays has lots of dependencies, we need to rein on that,
> making the good to have but not always used things to be dlopen'ed.
>
> Like we did with gtk (that at this point I think is really deprecated,
> BTW).
>
> gdb has prior art in this area that we could use, it is not even a TUI
> but it asks if debuginfo should be used and if so it goes on on
> potentially lenghty updates of the local buildid cache they keep (which
> is not the one we use, it should be).
>
> And in the recent discussion with Dmitry Vyukov the possibility doing a
> question to the user about a default behaviour to be set and then using
> .perfconfig not to bother anymore the user about things is part of
> helping the user to deal with the myriad possibilites perf offers.
>
> gdb could use that as well, why ask at every session if debuginfod
> should be used? Annoying.
>
> I think perf should try to use what is available, both at build and at
> run time, and it shouldn't change the way it output things, but should
> warn the user about recent developments, things we over time figured out
> are problematic and thus a new default would be better, but then obtain
> consent if the user cares about it, and allow for backtracking, to go
> and change .perfconfig when the user realises the old output/behaviour
> is not really nice.
>
> But keeping the grass green as it used to be should be the priority.

So I don't understand what you are saying. If I have libcapstone
installed should the build link against it or just use its headers for
dlopen? Is declaring structs/constants to avoid needing the library
acceptable?

The patches as they are will link against libcapstone if it's
installed (just as currently happens) if not it will try to use dlopen
at runtime, if that fails then you get the current no libcapstone not
available at build time behavior. To support this a minimal amount of
structs and constants are necessary. See here:
https://github.com/googleprodkernel/linux-perf/blob/google_tools_master/tools/perf/util/llvm.c#L21-L58
https://github.com/googleprodkernel/linux-perf/blob/google_tools_master/tools/perf/util/capstone.c#L23-L132

The patches try to make it look as though the same function exists
with dlopen or directly calling by turning for example "cs_disasm"
into:
```
static size_t perf_cs_disasm(csh handle, const uint8_t *code, size_t code_size,
uint64_t address, size_t count, struct cs_insn **insn)
{
#ifdef HAVE_LIBCAPSTONE_SUPPORT
return cs_disasm(handle, code, code_size, address, count, insn);
#else
static bool fn_init;
static enum cs_err (*fn)(csh handle, const uint8_t *code, size_t code_size,
uint64_t address, size_t count, struct cs_insn **insn);

if (!fn_init) {
fn = dlsym(perf_cs_dll_handle(), "cs_disasm");
if (!fn)
pr_debug("dlsym failed for cs_disasm\n");
fn_init = true;
}
if (!fn)
return CS_ERR_HANDLE;
return fn(handle, code, code_size, address, count, insn);
#endif
}
```
That is with the library linked it is just a direct call otherwise
dlopen/dlsym are used. This means that the perf code using the library
is unchanged except turning "cs_disasm" into "perf_cs_disasm" and we
can unconditionally assume the perf_cs_disasm function will be
available.

If we don't create the structs/constants ourselves, which I think is
where controversy lies, then any function that calls cs_disasm needs
to handle a situation where there is no cs_disasm at build time. I was
trying to avoid this clutter in the code and just have things at the
library boundary. The structs/constants become necessary as we can't
assume we have access to the header file to support perf's current
build.

Thanks,
Ian

On Thu, Mar 13, 2025 at 3:24 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Hi Ian,
>
> On Wed, Mar 12, 2025 at 02:04:30PM -0700, Ian Rogers wrote:
> > On Mon, Feb 10, 2025 at 10:06 AM Ian Rogers <irogers@google.com> wrote:
> > >
> > > On Thu, Jan 23, 2025 at 3:36 PM Ian Rogers <irogers@google.com> wrote:
> > > > On Thu, Jan 23, 2025 at 1:59 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > > > I like changes up to this in general.  Let me take a look at the
> > > > > patches.
> > >
> > > So it would be nice to make progress with this series given some level
> > > of happiness, I don't see any actions currently on the patch series as
> > > is. If I may be so bold as to recap the issues that have come up:
> > >
> > > 1) Andi Kleen mentions that dlopen is inferior to linking against
> > > libraries and those libraries aren't a memory overhead if unused.
> > >
> > > I agree but pointed-out the data center use case means that saving
> > > size on binaries can be important to some (me). We've also been trying
> > > to reduce perf's dependencies for distributions as perf dragging in
> > > say the whole of libLLVM can be annoying for making minimal
> > > distributions that contain perf. Perhaps somebody (Arnaldo?) more
> > > involved with distributions can confirm or deny the distribution
> > > problem, I'm hoping it is self-evident.
> > >
> > > 2) Namhyung Kim was uncomfortable with the code defining
> > > types/constants that were in header files as the two may drift over
> > > time
> > >
> > > I agree but in the same way as a function name is an ABI for dlysym,
> > > the types/constants are too. Yes a header file may change, but in
> > > doing so the ABI has changed and so it would be an incompatible change
> > > and everything would be broken. We'd need to fix the code for this,
> > > say as we did when libbpf moved to version 1.0, but using a header
> > > file would only weakly guard against this problem. The problem with
> > > including the header files is that then the build either breaks
> > > without the header or we need to support a no linking against a
> > > library and not using dlopen case. I suspect a lot of distributions
> > > wouldn't understand the build subtlety in this, the necessary build
> > > options and things installed, and we'd end up not using things like
> > > libLLVM even when it is known to be a large performance win. I also
> > > hope one day we can move from parsing text out of forked commands, as
> > > it is slower and more brittle, to just directly using libraries.
> > > Making dlopen the fallback (probably with a warning on failure) seems
> > > like the right direction for this except we won't get it if we need to
> > > drag in extra dependency header files for the build to succeed (well
> > > we could have a no library or dlopen option, but then we'd probably
> > > find distributions packaging this and things like perf annotate
> > > getting broken as they don't even know how to dlopen a library).
> > >
> > > 3) Namhyung Kim (and I) also raises that the libcapstone patch can be
> > > smaller by dropping the print_capstone_detail support on x86
> > >
> > > Note, given the similarity between capstone and libLLVM for
> > > disassembly, it is curious that only capstone gives the extra detail.
> > >
> > > I agree. Given the capstone disassembly output will be compromised we
> > > should warn for this, probably in Makefile.config to avoid running
> > > afoul of -Werror. It isn't clear that having a warning is a good move
> > > given the handful of structs needed to support print_capstone_detail.
> > > I'd prefer to keep the structs so that we haven't got a warning that
> > > looks like it needs cleaning up.
> > >
> > > 4) Namhyung Kim raised concerns over #if placement
> > >
> > > Namhyung raised that he'd prefer:
> > > ```
> > > #if HAVE_LIBCAPSTONE_SUPPORT
> > > // lots of code
> > > #else
> > > // lots of code
> > > #endif
> > > ```
> > > rather than the #ifs being inside or around individual functions. I
> > > raised that the large #ifs is a problem in the current code as you
> > > lose context when trying to understand a function. You may look at a
> > > function but not realize it isn't being used because of a #if 10s or
> > > 100s of lines above. Namhyung raised that the large #ifs is closer to
> > > kernel style, I disagreed as I think kernel style is only doing this
> > > when it stubs out a bunch of API functions, not when more context
> > > would be useful. Hopefully as the person writing the patches the style
> > > choice I've made can be respected.
> > >
> > > 5) Daniel Xu raised issues with the removal of libbfd for Rust
> > > support, as the code implies libbfd C++ demangling is a pre-requisite
> > > of legacy rust symbol demangling
> > >
> > > A separate patch was posted adding Rust v0 symbol demangling with no
> > > libbfd dependency:
> > > https://lore.kernel.org/lkml/20250129193037.573431-1-irogers@google.com/
> > > The legacy support should work with the non-libbfd demanglers as
> > > that's what we have today. We should really clean up Rust demangling
> > > and have tests. This is blocked on the Rust community responding to:
> > > https://github.com/rust-lang/rust/issues/60705
>
> I think #ifdef placements is not a big deal, but I still don't want to
> pull libcapstone details into the perf tree.
>
> For LLVM, I think you should to build llvm-c-helpers anyway which means
> you still need LLVM headers and don't need to redefine the structures.
>
> Can we do the same for capstone?  I think it's best to use capstone
> headers directly and add a build option to use dlopen().

So I don't disagree. If we have headers but someone wants dlopen,
let's use the headers and let them use dlopen. Unfortunately the way
the build is set up isn't for that. The build assumes either:
1) you have libcapstone and its headers and want to link against it,
2) you don't have libcapstone and support is just removed.

In the changes the options become:
1) unchanged, if you have libcapstone then use the headers and link
against - this is Andi's preferred approach,
2) if you don't have libcapstone dlopen is used with constants derived
from pahole and baked into the sources - much as we do for vmlinux.h
in BPF programs.

One way to achieve what you are asking for is to make the build do:
1) the link against libcapstone approach that needs the headers,
2) the dlopen with capstone.h headers,
but we need a way to build without libcapstone, so we get:
3) possibly dlopen with pahole derived constants - but if we have that
then why do 2?
4) a no support option.

The problem is that if we don't do (3) then (4) will become the no
libcapstone option and frankly people who care will do (1) and so (2)
becomes redundant.
It was intentional doing the changes the way I have so that when you
don't build with libcapstone, were you later to add the library you
would gain support for it. This is with the down side of a small
number of constants being in the code. Potentially these could change
in the header files, but any such change is an ABI breakage and so
unlikely to happen. So I think the way the patches have things set up
is best.

Thanks,
Ian