Commit ffa915f46193 ("Merge branch 'bpf_metadata'"), from September 2020,
added support to the kernel, libbpf, and bpftool to treat read-only BPF
variables that have names starting with 'bpf_metadata_' specially. This
patch series updates perf to handle these variables similarly, allowing a
perf.data file to capture relevant information about BPF programs on the
system being profiled.

When it encounters a BPF program, it reads the program's maps to find an
'.rodata' map with 'bpf_metadata_' variables. If it finds one, it extracts
their values as strings, and creates a new PERF_RECORD_BPF_METADATA
synthetic event using that data. It does this both for BPF programs that
were loaded when a 'perf record' starts, as well as for programs that are
loaded while the profile is running. For the latter case, it stores the
metadata for the duration of the profile, and then dumps it at the end of
the profile, where it's in a better context to do so.

The PERF_RECORD_BPF_METADATA event holds an array of key-value pairs, where
the key is the variable name (minus the "bpf_metadata_" prefix) and the
value is the variable's value, formatted as a string. There is one such
event generated for each BPF subprogram. Generating it per subprogram
rather than per program allows it to be correlated with PERF_RECORD_KSYMBOL
events; the metadata event's "prog_name" is designed to be identical to the
"name" field of a perf_record_ksymbol. This allows specific BPF metadata to
be associated with each BPF address range in the collection.

Changes:

* v2 -> v3:
  - Split out event collection from event display.
  - Resync with tmp.perf-tools-next.
  - Link to v2:
    https://lore.kernel.org/linux-perf-users/20250605233934.1881839-1-blakejones@google.com/T/#t

* v1 -> v2:
  - Split out libbpf change and send it to the bpf tree.
  - Add feature detection to perf to detect the libbpf change.
  - Allow the feature to be skipped if the libbpf support is not found.
  - Add an example of a PERF_RECORD_BPF_METADATA record.
  - Change calloc() calls to zalloc().
  - Don't check for NULL before calling free().
  - Update the perf_event header when it is created, rather than
    storing the event size and updating it later.
  - Add a BPF metadata variable (with the perf version) to all
    perf BPF programs.
  - Update the selftest to look for the new perf_version variable.
  - Split out the selftest into its own patch.
  - Link to v1:
    https://lore.kernel.org/linux-perf-users/20250521222725.3895192-1-blakejones@google.com/T/#t

Blake Jones (5):
  perf: detect support for libbpf's emit_strings option
  perf: collect BPF metadata from existing BPF programs
  perf: collect BPF metadata from new programs
  perf: display the new PERF_RECORD_BPF_METADATA event
  perf: add test for PERF_RECORD_BPF_METADATA collection

 tools/build/Makefile.feature                |   1 +
 tools/build/feature/Makefile                |   4 +
 tools/build/feature/test-libbpf-strings.c   |  10 +
 tools/lib/perf/include/perf/event.h         |  18 +
 tools/perf/Documentation/perf-check.txt     |   1 +
 tools/perf/Makefile.config                  |  12 +
 tools/perf/Makefile.perf                    |   3 +-
 tools/perf/builtin-check.c                  |   1 +
 tools/perf/builtin-inject.c                 |   1 +
 tools/perf/builtin-record.c                 |   8 +
 tools/perf/builtin-script.c                 |  15 +-
 tools/perf/tests/shell/test_bpf_metadata.sh |  76 ++++
 tools/perf/util/bpf-event.c                 | 378 ++++++++++++++++++++
 tools/perf/util/bpf-event.h                 |  13 +
 tools/perf/util/bpf_skel/perf_version.h     |  17 +
 tools/perf/util/env.c                       |  19 +-
 tools/perf/util/env.h                       |   4 +
 tools/perf/util/event.c                     |  21 ++
 tools/perf/util/event.h                     |   1 +
 tools/perf/util/header.c                    |   1 +
 tools/perf/util/session.c                   |   4 +
 tools/perf/util/synthetic-events.h          |   2 +
 tools/perf/util/tool.c                      |  14 +
 tools/perf/util/tool.h                      |   3 +-
 24 files changed, 622 insertions(+), 5 deletions(-)
 create mode 100644 tools/build/feature/test-libbpf-strings.c
 create mode 100755 tools/perf/tests/shell/test_bpf_metadata.sh
 create mode 100644 tools/perf/util/bpf_skel/perf_version.h

-- 
2.50.0.rc0.604.gd4ff7b7c86-goog

Hi Blake,

On Fri, Jun 06, 2025 at 02:52:41PM -0700, Blake Jones wrote:
> Commit ffa915f46193 ("Merge branch 'bpf_metadata'"), from September 2020,
> added support to the kernel, libbpf, and bpftool to treat read-only BPF
> variables that have names starting with 'bpf_metadata_' specially. This
> patch series updates perf to handle these variables similarly, allowing a
> perf.data file to capture relevant information about BPF programs on the
> system being profiled.
> 
> When it encounters a BPF program, it reads the program's maps to find an
> '.rodata' map with 'bpf_metadata_' variables. If it finds one, it extracts
> their values as strings, and creates a new PERF_RECORD_BPF_METADATA
> synthetic event using that data. It does this both for BPF programs that
> were loaded when a 'perf record' starts, as well as for programs that are
> loaded while the profile is running. For the latter case, it stores the
> metadata for the duration of the profile, and then dumps it at the end of
> the profile, where it's in a better context to do so.
> 
> The PERF_RECORD_BPF_METADATA event holds an array of key-value pairs, where
> the key is the variable name (minus the "bpf_metadata_" prefix) and the
> value is the variable's value, formatted as a string. There is one such
> event generated for each BPF subprogram. Generating it per subprogram
> rather than per program allows it to be correlated with PERF_RECORD_KSYMBOL
> events; the metadata event's "prog_name" is designed to be identical to the
> "name" field of a perf_record_ksymbol. This allows specific BPF metadata to
> be associated with each BPF address range in the collection.
> 
> Changes:
> 
> * v2 -> v3:
>   - Split out event collection from event display.
>   - Resync with tmp.perf-tools-next.
>   - Link to v2:
>     https://lore.kernel.org/linux-perf-users/20250605233934.1881839-1-blakejones@google.com/T/#t
> 
> * v1 -> v2:
>   - Split out libbpf change and send it to the bpf tree.
>   - Add feature detection to perf to detect the libbpf change.
>   - Allow the feature to be skipped if the libbpf support is not found.
>   - Add an example of a PERF_RECORD_BPF_METADATA record.
>   - Change calloc() calls to zalloc().
>   - Don't check for NULL before calling free().
>   - Update the perf_event header when it is created, rather than
>     storing the event size and updating it later.
>   - Add a BPF metadata variable (with the perf version) to all
>     perf BPF programs.
>   - Update the selftest to look for the new perf_version variable.
>   - Split out the selftest into its own patch.
>   - Link to v1:
>     https://lore.kernel.org/linux-perf-users/20250521222725.3895192-1-blakejones@google.com/T/#t
> 
> Blake Jones (5):
>   perf: detect support for libbpf's emit_strings option
>   perf: collect BPF metadata from existing BPF programs
>   perf: collect BPF metadata from new programs
>   perf: display the new PERF_RECORD_BPF_METADATA event
>   perf: add test for PERF_RECORD_BPF_METADATA collection

I tried to process your patches but it failed to build like below:

  util/bpf-event.h: In function 'bpf_metadata_free':
  util/bpf-event.h:68:59: error: unused parameter 'metadata' [-Werror=unused-parameter]
     68 | static inline void bpf_metadata_free(struct bpf_metadata *metadata)
        |                                      ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~

It was a simple to fix by adding __maybe_unused but after that I got
another build error so I stopped.

  /usr/bin/ld: /tmp/tmp.N9VJQ2A3pl/perf-in.o: in function `cmd_record':
  (.text+0x191ae): undefined reference to `perf_event__synthesize_final_bpf_metadata'
  collect2: error: ld returned 1 exit status
  make[4]: *** [Makefile.perf:804: /tmp/tmp.N9VJQ2A3pl/perf] Error 1
  make[4]: *** Waiting for unfinished jobs....
  make[3]: *** [Makefile.perf:290: sub-make] Error 2
  make[2]: *** [Makefile:76: all] Error 2
  make[1]: *** [tests/make:341: make_no_libbpf_O] Error 1
  make: *** [Makefile:109: build-test] Error 2

Please run 'make build-test' and send v4.

Thanks,
Namhyung

> 
>  tools/build/Makefile.feature                |   1 +
>  tools/build/feature/Makefile                |   4 +
>  tools/build/feature/test-libbpf-strings.c   |  10 +
>  tools/lib/perf/include/perf/event.h         |  18 +
>  tools/perf/Documentation/perf-check.txt     |   1 +
>  tools/perf/Makefile.config                  |  12 +
>  tools/perf/Makefile.perf                    |   3 +-
>  tools/perf/builtin-check.c                  |   1 +
>  tools/perf/builtin-inject.c                 |   1 +
>  tools/perf/builtin-record.c                 |   8 +
>  tools/perf/builtin-script.c                 |  15 +-
>  tools/perf/tests/shell/test_bpf_metadata.sh |  76 ++++
>  tools/perf/util/bpf-event.c                 | 378 ++++++++++++++++++++
>  tools/perf/util/bpf-event.h                 |  13 +
>  tools/perf/util/bpf_skel/perf_version.h     |  17 +
>  tools/perf/util/env.c                       |  19 +-
>  tools/perf/util/env.h                       |   4 +
>  tools/perf/util/event.c                     |  21 ++
>  tools/perf/util/event.h                     |   1 +
>  tools/perf/util/header.c                    |   1 +
>  tools/perf/util/session.c                   |   4 +
>  tools/perf/util/synthetic-events.h          |   2 +
>  tools/perf/util/tool.c                      |  14 +
>  tools/perf/util/tool.h                      |   3 +-
>  24 files changed, 622 insertions(+), 5 deletions(-)
>  create mode 100644 tools/build/feature/test-libbpf-strings.c
>  create mode 100755 tools/perf/tests/shell/test_bpf_metadata.sh
>  create mode 100644 tools/perf/util/bpf_skel/perf_version.h
> 
> -- 
> 2.50.0.rc0.604.gd4ff7b7c86-goog
>

Hi Namhyung,

On Wed, Jun 11, 2025 at 11:29 AM Namhyung Kim <namhyung@kernel.org> wrote:
> I tried to process your patches but it failed to build like below:
> [...]
> Please run 'make build-test' and send v4.

Very sorry about that. I've fixed the two issues you noticed, as well as
one additional one where I was using the wrong include path to check for
the presence of the libbpf-strings feature.

I'm trying to test my fixes using "make build-test", but it's proving a bit
of a challenge. I installed libgtk-4-dev, binutils-dev, and libopencsd-dev
to fix build problems as they came up; I also installed libtraceevent-dev,
but somehow it still wasn't detected by the build process and so I had to
use NO_LIBTRACEEVENT=1.

Even after installing these libraries, I'm still hitting errors when doing
"make build-test" on a copy of the perf source *without* my changes:

    In file included from util/disasm_bpf.c:18:
    .../tools/include/tools/dis-asm-compat.h:10:6:
        error: redeclaration of 'enum disassembler_style'
       10 | enum disassembler_style {DISASSEMBLER_STYLE_NOT_EMPTY};
          |      ^~~~~~~~~~~~~~~~~~
    In file included from util/disasm_bpf.c:15:
    /usr/include/dis-asm.h:53:6: note: originally defined here
       53 | enum disassembler_style
          |      ^~~~~~~~~~~~~~~~~~

I noticed that tools/perf/BUILD_TEST_FEATURE_DUMP has
"feature-disassembler-four-args=0" and "feature-disassembler-init-styled=0"
as of when this failed, which seems to be upstream of the observed failure
(the version of binutils-dev that I installed seems to have newer-style
versions of these interfaces).

Is there anything written up about how to set up a machine so that
"make build-test" works reliably?

Thanks.

Blake

On Wed, Jun 11, 2025 at 5:39 PM Blake Jones <blakejones@google.com> wrote:
>
> Hi Namhyung,
>
> On Wed, Jun 11, 2025 at 11:29 AM Namhyung Kim <namhyung@kernel.org> wrote:
> > I tried to process your patches but it failed to build like below:
> > [...]
> > Please run 'make build-test' and send v4.
>
> Very sorry about that. I've fixed the two issues you noticed, as well as
> one additional one where I was using the wrong include path to check for
> the presence of the libbpf-strings feature.
>
> I'm trying to test my fixes using "make build-test", but it's proving a bit
> of a challenge. I installed libgtk-4-dev, binutils-dev, and libopencsd-dev
> to fix build problems as they came up; I also installed libtraceevent-dev,
> but somehow it still wasn't detected by the build process and so I had to
> use NO_LIBTRACEEVENT=1.
>
> Even after installing these libraries, I'm still hitting errors when doing
> "make build-test" on a copy of the perf source *without* my changes:
>
>     In file included from util/disasm_bpf.c:18:
>     .../tools/include/tools/dis-asm-compat.h:10:6:
>         error: redeclaration of 'enum disassembler_style'
>        10 | enum disassembler_style {DISASSEMBLER_STYLE_NOT_EMPTY};
>           |      ^~~~~~~~~~~~~~~~~~
>     In file included from util/disasm_bpf.c:15:
>     /usr/include/dis-asm.h:53:6: note: originally defined here
>        53 | enum disassembler_style
>           |      ^~~~~~~~~~~~~~~~~~
>
> I noticed that tools/perf/BUILD_TEST_FEATURE_DUMP has
> "feature-disassembler-four-args=0" and "feature-disassembler-init-styled=0"
> as of when this failed, which seems to be upstream of the observed failure
> (the version of binutils-dev that I installed seems to have newer-style
> versions of these interfaces).

Fwiw, binutils is GPLv3 and license incompatible with perf which is
largely GPLv2. This patch series deletes the code in perf using it and
migrates the BPF disassembly to using capstone or libLLVM:
https://lore.kernel.org/lkml/20250417230740.86048-1-irogers@google.com/
The series isn't merged into upstream Linux but is in:
https://github.com/googleprodkernel/linux-perf

Thanks,
Ian

> Is there anything written up about how to set up a machine so that
> "make build-test" works reliably?
>
> Thanks.
>
> Blake

On Wed, Jun 11, 2025 at 10:19 PM Ian Rogers <irogers@google.com> wrote:
> Fwiw, binutils is GPLv3 and license incompatible with perf which is
> largely GPLv2. This patch series deletes the code in perf using it and
> migrates the BPF disassembly to using capstone or libLLVM:
> https://lore.kernel.org/lkml/20250417230740.86048-1-irogers@google.com/
> The series isn't merged into upstream Linux but is in:
> https://github.com/googleprodkernel/linux-perf

Good to know. I'd be a bit concerned about the validity of testing my
changes with a 19-patch unmerged changeset, so I don't think I'm going
to try that for now.

Perhaps I'll just send along v4 of my changes, which has the fixes and
testing mentioned upthread.

Blake

On Wed, Jun 11, 2025 at 5:39 PM Blake Jones <blakejones@google.com> wrote:
> Is there anything written up about how to set up a machine so that
> "make build-test" works reliably?

Barring that, I've confirmed that each of my new patches builds successfully
under the following build commands: (I have a copy of libbpf that supports
".emit_strings" in /usr/local/include)

    cd tools/perf
    make clean
    make NO_LIBTRACEEVENT=1 LIBBPF_DYNAMIC=1 LIBBPF_INCLUDE=/usr/local/include
    ./perf check feature libbpf-strings

    make clean
    make NO_LIBTRACEEVENT=1
    ./perf check feature libbpf-strings

    make clean
    make NO_LIBTRACEEVENT=1 NO_LIBBPF=1
    ./perf check feature libbpf-strings

Please let me know if that seems like sufficient testing.

Blake

Hi Blake,

On Wed, Jun 11, 2025 at 06:02:55PM -0700, Blake Jones wrote:
> On Wed, Jun 11, 2025 at 5:39 PM Blake Jones <blakejones@google.com> wrote:
> > Is there anything written up about how to set up a machine so that
> > "make build-test" works reliably?

Sorry for the trouble.  I found it needs more work.

> 
> Barring that, I've confirmed that each of my new patches builds successfully
> under the following build commands: (I have a copy of libbpf that supports
> ".emit_strings" in /usr/local/include)
> 
>     cd tools/perf
>     make clean
>     make NO_LIBTRACEEVENT=1 LIBBPF_DYNAMIC=1 LIBBPF_INCLUDE=/usr/local/include
>     ./perf check feature libbpf-strings
> 
>     make clean
>     make NO_LIBTRACEEVENT=1
>     ./perf check feature libbpf-strings
> 
>     make clean
>     make NO_LIBTRACEEVENT=1 NO_LIBBPF=1
>     ./perf check feature libbpf-strings
> 
> Please let me know if that seems like sufficient testing.

Looks ok to me and thanks for doing this.

Namhyung