Linking against libcapstone and libLLVM can be a significant increase
in dependencies and file size if building statically. For something
like `perf record` the disassembler and addr2line functionality won't
be used. Support dynamically loading these libraries using dlopen and
then calling the appropriate functions found using dlsym.

BUILD_NONDISTRO is used to build perf against the license incompatible
libbfd and libiberty libraries. As this has been opt-in for nearly 2
years, commit dd317df07207 ("perf build: Make binutil libraries opt
in"), remove the code to simplify the code base.

The patch series:
1) does some initial clean up;
2) moves the capstone and LLVM code to their own C files,
3) simplifies a little the capstone code;
4) adds perf_ variants of the functions that will either directly call
   the function or use dlsym to discover it;
5) adds BPF JIT disassembly support to LLVM and capstone disassembly;
6) removes the BUILD_NONDISTRO code, reduces scope and removes what's possible;
7) adds fallback to srcline's addr2line so that llvm_addr2line is
   tried first and then the forked command tried next, moving the code
   for forking out of the main srcline.c file in the process.

The addr2line LLVM functionality is written in C++. To avoid linking
against libLLVM for this, a new LIBLLVM_DYNAMIC option is added where
the C++ code with the libLLVM dependency will be built into a
libperf-llvm.so and that dlsym-ed and called against. Ideally LLVM
would extend their C API to avoid this.

The libbfd BPF disassembly supported source lines, this wasn't ported
to the capstone and LLVM disassembly.

v5: Rebase and comment typo fix.
v4: Rebase and addition of a patch removing an unused struct variable.
v3: Add srcline addr2line fallback trying LLVM first then forking a
    process. This came up in conversation with Steinar Gunderson
    <sesse@google.com>.
    Tweak the cover letter message to try to address Andi Kleen's
    <ak@linux.intel.com> feedback that the series doesn't really
    achieve anything.
v2: Add mangling of the function names in libperf-llvm.so to avoid
    potential infinite recursion. Add BPF JIT disassembly support to
    LLVM and capstone. Add/rebase the BUILD_NONDISTRO cleanup onto the
    series from:
    https://lore.kernel.org/lkml/20250111202851.1075338-1-irogers@google.com/
    Some other minor additional clean up.

Ian Rogers (19):
  perf build: Remove libtracefs configuration
  perf map: Constify objdump offset/address conversion APIs
  perf capstone: Move capstone functionality into its own file
  perf llvm: Move llvm functionality into its own file
  perf capstone: Remove open_capstone_handle
  perf capstone: Support for dlopen-ing libcapstone.so
  perf llvm: Support for dlopen-ing libLLVM.so
  perf llvm: Mangle libperf-llvm.so function names
  perf dso: Move read_symbol from llvm/capstone to dso
  perf dso: Support BPF programs in dso__read_symbol
  perf llvm: Disassemble cleanup
  perf dso: Clean up read_symbol error handling
  perf build: Remove libbfd support
  perf build: Remove libiberty support
  perf build: Remove unused defines
  perf disasm: Remove disasm_bpf
  perf disasm: Make ins__scnprintf and ins__is_nop static
  perf srcline: Fallback between addr2line implementations
  perf disasm: Remove unused evsel from annotate_args

 tools/perf/Documentation/perf-check.txt |   1 -
 tools/perf/Makefile.config              |  95 +--
 tools/perf/Makefile.perf                |  35 +-
 tools/perf/builtin-check.c              |   1 -
 tools/perf/builtin-script.c             |   2 -
 tools/perf/tests/Build                  |   1 -
 tools/perf/tests/builtin-test.c         |   1 -
 tools/perf/tests/make                   |   4 +-
 tools/perf/tests/pe-file-parsing.c      | 101 ----
 tools/perf/tests/tests.h                |   1 -
 tools/perf/util/Build                   |   6 +-
 tools/perf/util/addr2line.c             | 439 ++++++++++++++
 tools/perf/util/addr2line.h             |  20 +
 tools/perf/util/annotate.c              |   1 -
 tools/perf/util/annotate.h              |   1 -
 tools/perf/util/capstone.c              | 682 +++++++++++++++++++++
 tools/perf/util/capstone.h              |  24 +
 tools/perf/util/config.c                |   2 +-
 tools/perf/util/demangle-cxx.cpp        |  22 +-
 tools/perf/util/disasm.c                | 628 +------------------
 tools/perf/util/disasm.h                |   6 +-
 tools/perf/util/disasm_bpf.c            | 195 ------
 tools/perf/util/disasm_bpf.h            |  12 -
 tools/perf/util/dso.c                   |  98 +++
 tools/perf/util/dso.h                   |   4 +
 tools/perf/util/llvm-c-helpers.cpp      | 120 +++-
 tools/perf/util/llvm-c-helpers.h        |  24 +-
 tools/perf/util/llvm.c                  | 484 +++++++++++++++
 tools/perf/util/llvm.h                  |  21 +
 tools/perf/util/map.c                   |  19 +-
 tools/perf/util/map.h                   |   6 +-
 tools/perf/util/print_insn.c            | 117 +---
 tools/perf/util/srcline.c               | 772 +-----------------------
 tools/perf/util/srcline.h               |   7 +-
 tools/perf/util/symbol-elf.c            |  85 ---
 tools/perf/util/symbol.c                | 145 -----
 tools/perf/util/symbol.h                |   4 -
 37 files changed, 2020 insertions(+), 2166 deletions(-)
 delete mode 100644 tools/perf/tests/pe-file-parsing.c
 create mode 100644 tools/perf/util/addr2line.c
 create mode 100644 tools/perf/util/addr2line.h
 create mode 100644 tools/perf/util/capstone.c
 create mode 100644 tools/perf/util/capstone.h
 delete mode 100644 tools/perf/util/disasm_bpf.c
 delete mode 100644 tools/perf/util/disasm_bpf.h
 create mode 100644 tools/perf/util/llvm.c
 create mode 100644 tools/perf/util/llvm.h

-- 
2.51.0.rc2.233.g662b1ed5c5-goog

> As this has been opt-in for nearly 2 years, commit dd317df07207 ("perf build: Make binutil libraries opt
> in"), remove the code to simplify the code base.

I don't think this is a reason to remove it by itself. We've been
enabling it in Gentoo and happily using it.

Anyway, my main concern (though I have a few) is that annotate support
will be lacking. A few months ago, objdump was the only way to get
source line support [0]. Is that still the case?

[0] https://lore.kernel.org/all/Z_S_JB_L_t9viciV@google.com/

> A few months ago, objdump was the only way to get
> source line support [0]. Is that still the case?

... or is this perhaps handled by "[PATCH v5 18/19] perf srcline:
Fallback between addr2line implementations", in which case, shouldn't
that really land first so people can try the LLVM impl and use the
binutils one if it fails?

On Fri, Aug 22, 2025 at 11:52 PM Sam James <sam@gentoo.org> wrote:
>
> > A few months ago, objdump was the only way to get
> > source line support [0]. Is that still the case?
>
> ... or is this perhaps handled by "[PATCH v5 18/19] perf srcline:
> Fallback between addr2line implementations", in which case, shouldn't
> that really land first so people can try the LLVM impl and use the
> binutils one if it fails?

So my opinion, BUILD_NON_DISTRO isn't supported and the code behind it
should go away. Please don't do anything to the contrary or enable it
for your distribution - this was supposed to be implied by the name.
The forking and running addr2line gets around the license issue that
is GPLv3* but comes with a performance overhead. It also has a
maintenance overhead supporting llvm and binutil addr2line, when the
addr2line output changes things break, etc. (LLVM has been evolving
their output but I'm not aware of it breaking things yet). We should
(imo) delete the forking and running addr2line support, it fits the
billing of something we can do when capstone and libLLVM support
aren't there but the code is a hot mess and we don't do exhaustive
testing against the many addr2line flavors, the best case is buyer
beware. Capstone is derived from libLLVM, I'm not sure it makes sense
having 2 libraries for this stuff. There's libLLVM but what it
provides through a C API is a mess requiring the C++ shimming. Tbh, I
think most of what these libraries provide we should just get over
ourselves and provide in perf itself. For example, does it make sense
to be trying to add type annotations to objdump output, to just update
objdump or have a disassembler library where we can annotate things as
we see fit? Library bindings don't break when text output formats get
tweaked. Given we're doing so much dwarf processing, do we need a
library for that or should that just be in-house? We can side step
most of this mess by starting again in python as is being shown in the
textual changes that bring with it stuff like console flame graphs:
https://lore.kernel.org/lkml/CAP-5=fU=z8kcY4zjezoxSwYf9vczYzHztiMSBvJxdwwBPVWv2Q@mail.gmail.com/
So I think long term we make the perf tool minimal with minimal
dependencies (ie no addr2line, libLLVM, etc.), we work on having nice
stuff in the python stuff where we can reuse or build new libraries
for addr2line, objdump-ing, etc. Use >1 thread, use asyncio, etc.

For where we are now, ie no python stuff, BUILD_NON_DISTRO should go
away as nobody is maintaining it and hasn't for 2 years (what happens
when libbfd and libiberty change?). We should focus on making the best
of what we have via libraries/tools that are supported - while not
forcing the libraries to be there or making the perf binary massive by
dragging in say libLLVM. The patch series pushes in that direction and
I commend it to the reader.

No, reordering the patches to compare performance of binutils doesn't
make sense, just build with and without the patch series if you want
to do this, but also don't do this as BUILD_NON_DISTRO should go away.

Thanks,
Ian

* (As I understand the issue IANAL) GPLv3 and GPLv2 can't be linked
together. Why not just use GPLv3? A major issue for me is that GPLv3
adds a requirement for “Installation Information” to be provided,
which means placing a binary in a cryptographically signed OS
partition you'd need to reveal the signing key which defeats the
purpose of signing the partition to ensure you aren't hacked. I like
open source and using the code, I don't want to be hacked by giving to
the hackers my signing keys.

Ian Rogers <irogers@google.com> writes:

> On Fri, Aug 22, 2025 at 11:52 PM Sam James <sam@gentoo.org> wrote:
>>
>> > A few months ago, objdump was the only way to get
>> > source line support [0]. Is that still the case?
>>
>> ... or is this perhaps handled by "[PATCH v5 18/19] perf srcline:
>> Fallback between addr2line implementations", in which case, shouldn't
>> that really land first so people can try the LLVM impl and use the
>> binutils one if it fails?
>
> So my opinion, BUILD_NON_DISTRO isn't supported and the code behind it
> should go away. Please don't do anything to the contrary or enable it
> for your distribution - this was supposed to be implied by the name.

We're principally a source-based distribution, so it's not as much of an
issue.

> The forking and running addr2line gets around the license issue that
> is GPLv3* but comes with a performance overhead. It also has a
> maintenance overhead supporting llvm and binutil addr2line, when the
> addr2line output changes things break, etc. (LLVM has been evolving
> their output but I'm not aware of it breaking things yet). We should
> (imo) delete the forking and running addr2line support, it fits the
> billing of something we can do when capstone and libLLVM support
> aren't there but the code is a hot mess and we don't do exhaustive
> testing against the many addr2line flavors, the best case is buyer
> beware. Capstone is derived from libLLVM, I'm not sure it makes sense
> having 2 libraries for this stuff. There's libLLVM but what it
> provides through a C API is a mess requiring the C++ shimming. Tbh, I
> think most of what these libraries provide we should just get over
> ourselves and provide in perf itself. For example, does it make sense
> to be trying to add type annotations to objdump output, to just update
> objdump or have a disassembler library where we can annotate things as
> we see fit? Library bindings don't break when text output formats get
> tweaked. Given we're doing so much dwarf processing, do we need a
> library for that or should that just be in-house? We can side step
> most of this mess by starting again in python as is being shown in the
> textual changes that bring with it stuff like console flame graphs:
> https://lore.kernel.org/lkml/CAP-5=fU=z8kcY4zjezoxSwYf9vczYzHztiMSBvJxdwwBPVWv2Q@mail.gmail.com/
> So I think long term we make the perf tool minimal with minimal
> dependencies (ie no addr2line, libLLVM, etc.), we work on having nice
> stuff in the python stuff where we can reuse or build new libraries
> for addr2line, objdump-ing, etc. Use >1 thread, use asyncio, etc.

Yeah, this absolutely sounds like the right direction indeed. I'm glad
to hear it.

>
> For where we are now, ie no python stuff, BUILD_NON_DISTRO should go
> away as nobody is maintaining it and hasn't for 2 years (what happens
> when libbfd and libiberty change?)

They don't change often, though. The fixes are usually trivial when they
do arise.

> We should focus on making the best
> of what we have via libraries/tools that are supported - while not
> forcing the libraries to be there or making the perf binary massive by
> dragging in say libLLVM. The patch series pushes in that direction and
> I commend it to the reader.
>
> No, reordering the patches to compare performance of binutils doesn't
> make sense, just build with and without the patch series if you want
> to do this, but also don't do this as BUILD_NON_DISTRO should go away.

I was asking purely because of the *functionality loss*, though, not
performance. In the thread I linked from just a few months ago with Ingo
Molnar, there was a real issue with llvm or capstone-based disassembly
not showing source information. I'd hit the same problem. Is that fixed now?

This is my principal concern rather than the LLVM dependency (even if
I'd love to avoid that, I understand and appreciate the arguments you're
making above and intent on future direction).

>
> Thanks,
> Ian
>
> * (As I understand the issue IANAL) GPLv3 and GPLv2 can't be linked
> together. Why not just use GPLv3? A major issue for me is that GPLv3
> adds a requirement for “Installation Information” to be provided,
> which means placing a binary in a cryptographically signed OS
> partition you'd need to reveal the signing key which defeats the
> purpose of signing the partition to ensure you aren't hacked. I like
> open source and using the code, I don't want to be hacked by giving to
> the hackers my signing keys.

I think the way people usually handle this is allowing a custom key to
be added, but then it taints the device. That's how I think routers seem
to handle it often anyway.

On Sat, Aug 23, 2025 at 3:04 PM Sam James <sam@gentoo.org> wrote:
>
> Ian Rogers <irogers@google.com> writes:
>
> > On Fri, Aug 22, 2025 at 11:52 PM Sam James <sam@gentoo.org> wrote:
> >>
> >> > A few months ago, objdump was the only way to get
> >> > source line support [0]. Is that still the case?
> >>
> >> ... or is this perhaps handled by "[PATCH v5 18/19] perf srcline:
> >> Fallback between addr2line implementations", in which case, shouldn't
> >> that really land first so people can try the LLVM impl and use the
> >> binutils one if it fails?
> >
> > So my opinion, BUILD_NON_DISTRO isn't supported and the code behind it
> > should go away. Please don't do anything to the contrary or enable it
> > for your distribution - this was supposed to be implied by the name.
>
> We're principally a source-based distribution, so it's not as much of an
> issue.

Agreed, but I don't think the perf tool should be made to be a source
only distribution :-)

> > The forking and running addr2line gets around the license issue that
> > is GPLv3* but comes with a performance overhead. It also has a
> > maintenance overhead supporting llvm and binutil addr2line, when the
> > addr2line output changes things break, etc. (LLVM has been evolving
> > their output but I'm not aware of it breaking things yet). We should
> > (imo) delete the forking and running addr2line support, it fits the
> > billing of something we can do when capstone and libLLVM support
> > aren't there but the code is a hot mess and we don't do exhaustive
> > testing against the many addr2line flavors, the best case is buyer
> > beware. Capstone is derived from libLLVM, I'm not sure it makes sense
> > having 2 libraries for this stuff. There's libLLVM but what it
> > provides through a C API is a mess requiring the C++ shimming. Tbh, I
> > think most of what these libraries provide we should just get over
> > ourselves and provide in perf itself. For example, does it make sense
> > to be trying to add type annotations to objdump output, to just update
> > objdump or have a disassembler library where we can annotate things as
> > we see fit? Library bindings don't break when text output formats get
> > tweaked. Given we're doing so much dwarf processing, do we need a
> > library for that or should that just be in-house? We can side step
> > most of this mess by starting again in python as is being shown in the
> > textual changes that bring with it stuff like console flame graphs:
> > https://lore.kernel.org/lkml/CAP-5=fU=z8kcY4zjezoxSwYf9vczYzHztiMSBvJxdwwBPVWv2Q@mail.gmail.com/
> > So I think long term we make the perf tool minimal with minimal
> > dependencies (ie no addr2line, libLLVM, etc.), we work on having nice
> > stuff in the python stuff where we can reuse or build new libraries
> > for addr2line, objdump-ing, etc. Use >1 thread, use asyncio, etc.
>
> Yeah, this absolutely sounds like the right direction indeed. I'm glad
> to hear it.

Well it takes effort and what drags against that is supporting things
that can't be distributed.

> >
> > For where we are now, ie no python stuff, BUILD_NON_DISTRO should go
> > away as nobody is maintaining it and hasn't for 2 years (what happens
> > when libbfd and libiberty change?)
>
> They don't change often, though. The fixes are usually trivial when they
> do arise.

So just taking the feature tests as an example of the burden, we have:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/build/feature/test-disassembler-four-args.c?h=perf-tools-next
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/build/feature/test-disassembler-init-styled.c?h=perf-tools-next
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/build/feature/test-libbfd-buildid.c?h=perf-tools-next
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/build/feature/test-libbfd.c?h=perf-tools-next
so it isn't just supporting a small libbfd wrapper, it is maintaining
addr2line, objdump, etc. functionality for all these configurations.
As it doesn't get tested, or tested with these features both true and
false, it is liable to bitrot. However, the biggest issue/drag is in
improving/refactoring the tool and having to make sure these new
things work for a situation that in most distributions really, really
doesn't matter.

> > We should focus on making the best
> > of what we have via libraries/tools that are supported - while not
> > forcing the libraries to be there or making the perf binary massive by
> > dragging in say libLLVM. The patch series pushes in that direction and
> > I commend it to the reader.
> >
> > No, reordering the patches to compare performance of binutils doesn't
> > make sense, just build with and without the patch series if you want
> > to do this, but also don't do this as BUILD_NON_DISTRO should go away.
>
> I was asking purely because of the *functionality loss*, though, not
> performance. In the thread I linked from just a few months ago with Ingo
> Molnar, there was a real issue with llvm or capstone-based disassembly
> not showing source information. I'd hit the same problem. Is that fixed now?

Can you link to the issue? Capstone and LLVM don't replace objdump, in
perf there is a list of disassemblers and in your ~/.perfconfig and
you can always set LLVM and capstone to be your least preferred if
they lack a feature you need:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/Documentation/perf-config.txt?h=perf-tools-next#n250

> This is my principal concern rather than the LLVM dependency (even if
> I'd love to avoid that, I understand and appreciate the arguments you're
> making above and intent on future direction).

I think adding a missing feature to the LLVM disassembler is forward
progress, like fixing the BPF disassembly for LLVM done in this
series. I need to know more about the issue but it sounds less of a
libbfd thing and more an objdump -S thing.

This patch series was reposted as I carry it in:
https://github.com/googleprodkernel/linux-perf
to avoid the cost of linking in libLLVM/libcapstone in contexts (data
centers) where that functionality isn't used.

Thanks,
Ian

> BUILD_NONDISTRO is used to build perf against the license incompatible
> libbfd and libiberty libraries. As this has been opt-in for nearly 2
> years, commit dd317df07207 ("perf build: Make binutil libraries opt
> in"), remove the code to simplify the code base.

The last time I tried the LLVM stuff was totally broken, couldn't
resolve many things. The only workaround was to go back to the actually
working libbfd. Please don't remove the only working option.

Thanks,

-Andi

On Sat, Aug 23, 2025 at 4:34 PM Andi Kleen <ak@linux.intel.com> wrote:
>
> > BUILD_NONDISTRO is used to build perf against the license incompatible
> > libbfd and libiberty libraries. As this has been opt-in for nearly 2
> > years, commit dd317df07207 ("perf build: Make binutil libraries opt
> > in"), remove the code to simplify the code base.
>
> The last time I tried the LLVM stuff was totally broken, couldn't
> resolve many things. The only workaround was to go back to the actually
> working libbfd. Please don't remove the only working option.

Andi, did you report a bug? Does the non-libbfd stuff work? Yes, as
every distribution has shipped it since binutils went to GPLv3 (in
2007?). While I don't want LLVM stuff not to work, bugs happen, libbfd
is license incompatible with GPLv2 and maintaining it in the source
tree is a burden. When something doesn't work, create a test to repeat
the issue and then we can make it work and ensure it doesn't break
again. In general I've not had issues with the LLVM code.

Thanks,
Ian