This commit introduces an option "--bpf-aux-pause" for loading BPF
program to trigger AUX pause and resume.

After:

  perf record -e cs_etm/aux-action=start-paused/ \
    --bpf-aux-pause="kretprobe:p:__arm64_sys_openat,kprobe:r:__arm64_sys_openat,tp:r:sched:sched_switch" \
    -a -- ls

  perf record -e cs_etm/aux-action=start-paused/ \
    --bpf-aux-pause="kretprobe:p:__arm64_sys_openat,kprobe:r:__arm64_sys_openat,tp:r:sched:sched_switch" \
    --per-thread -- ls

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/perf/builtin-record.c | 20 +++++++++++++++++++-
 tools/perf/util/evsel.c     |  6 ++++++
 tools/perf/util/record.h    |  1 +
 3 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 8059bce85a510b2dddc66f1b8b0013276840eddc..793609c0a59e8abcd248608f55e0af03a253138d 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -880,7 +880,12 @@ static int record__auxtrace_init(struct record *rec)
 	if (err)
 		return err;
 
-	return auxtrace_parse_filters(rec->evlist);
+	err = auxtrace_parse_filters(rec->evlist);
+	if (err)
+		return err;
+
+	return auxtrace__prepare_bpf(rec->itr,
+				     rec->opts.auxtrace_bpf_aux_pause_opts);
 }
 
 #else
@@ -2506,6 +2511,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 	evlist__config(rec->evlist, opts, &callchain_param);
 
+	err = auxtrace__set_bpf_filter(rec->evlist, opts);
+	if (err)
+		goto out_free_threads;
+
 	/* Debug message used by test scripts */
 	pr_debug3("perf record opening and mmapping events\n");
 	if (record__open(rec) != 0) {
@@ -2579,6 +2588,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (record__start_threads(rec))
 		goto out_free_threads;
 
+	if (auxtrace__enable_bpf())
+		goto out_free_threads;
+
 	/*
 	 * When perf is starting the traced process, all the events
 	 * (apart from group members) have enable_on_exec=1 set,
@@ -2907,6 +2919,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	}
 
 out_delete_session:
+	auxtrace__cleanup_bpf();
 #ifdef HAVE_EVENTFD_SUPPORT
 	if (done_fd >= 0) {
 		fd = done_fd;
@@ -3629,6 +3642,11 @@ static struct option __record_options[] = {
 	OPT_CALLBACK(0, "off-cpu-thresh", &record.opts, "ms",
 		     "Dump off-cpu samples if off-cpu time exceeds this threshold (in milliseconds). (Default: 500ms)",
 		     record__parse_off_cpu_thresh),
+	OPT_STRING_OPTARG(0, "bpf-aux-pause", &record.opts.auxtrace_bpf_aux_pause_opts,
+			  "{kprobe|kretprobe}:{p|r}:function_name\n"
+			  "\t\t\t  {uprobe|uretprobe}:{p|r}:executable:function_name\n"
+			  "\t\t\t  {tp|tracepoint}:{p|r}:category:tracepoint\n",
+			  "Enable AUX pause with BPF backend", ""),
 	OPT_END()
 };
 
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index d55482f094bf95ac7b5c5173c1341baeb0fa9c93..f240e48f41a3e7ca5ba81733efc58a25c5c829ba 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -2650,6 +2650,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 
 			bpf_counter__install_pe(evsel, idx, fd);
 
+			/* Update event info into BPF map for AUX trace */
+			if (auxtrace__update_bpf_map(evsel, idx, fd)) {
+				err = -EINVAL;
+				goto out_close;
+			}
+
 			if (unlikely(test_attr__enabled())) {
 				test_attr__open(&evsel->core.attr, pid, cpu,
 						fd, group_fd, evsel->open_flags);
diff --git a/tools/perf/util/record.h b/tools/perf/util/record.h
index ea3a6c4657eefb743dc3d54b0b791ea39117cc10..1b1c2ed7fcadae8b56408b3b1b154faef3996eb3 100644
--- a/tools/perf/util/record.h
+++ b/tools/perf/util/record.h
@@ -65,6 +65,7 @@ struct record_opts {
 	size_t	      auxtrace_snapshot_size;
 	const char    *auxtrace_snapshot_opts;
 	const char    *auxtrace_sample_opts;
+	const char    *auxtrace_bpf_aux_pause_opts;
 	bool	      sample_transaction;
 	bool	      use_clockid;
 	clockid_t     clockid;

-- 
2.34.1

Hello,

kernel test robot noticed "perf-sanity-tests.Convert_perf_time_to_TSC.Perf_time_to_TSC.fail" on:

commit: e350af63969b875598f0656a20d801bbcaa7bd76 ("[PATCH RESEND v3 5/6] perf record: Support AUX pause and resume with BPF")
url: https://github.com/intel-lab-lkp/linux/commits/Leo-Yan/perf-core-Make-perf_event_aux_pause-as-external-function/20250725-181647
patch link: https://lore.kernel.org/all/20250725-perf_aux_pause_resume_bpf_rebase-v3-5-ae21deb49d1a@arm.com/
patch subject: [PATCH RESEND v3 5/6] perf record: Support AUX pause and resume with BPF

in testcase: perf-sanity-tests
version: 
with following parameters:

	perf_compiler: gcc
	group: group-01



config: x86_64-rhel-9.4-bpf
compiler: gcc-12
test machine: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake-E) with 32G memory

(please refer to attached dmesg/kmsg for entire log/backtrace)


besides, we also noticed other tests failed which can pass on parent:

90bef873a01451e8 e350af63969b875598f0656a20d
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
           :5           83%           5:5     perf-sanity-tests.Convert_perf_time_to_TSC.Perf_time_to_TSC.fail
           :5           83%           5:5     perf-sanity-tests.Zstd_perf.data_compression/decompression.fail
           :5           83%           5:5     perf-sanity-tests.build_id_cache_operations.fail
           :5           83%           5:5     perf-sanity-tests.daemon_operations.fail
           :5           83%           5:5     perf-sanity-tests.perf_all_metricgroups_test.fail
           :5           83%           5:5     perf-sanity-tests.perf_diff_tests.fail
           :5           83%           5:5     perf-sanity-tests.perf_stat_JSON_output_linter.fail
           :5           83%           5:5     perf-sanity-tests.perf_stat_STD_output_linter.fail



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202507310818.a05d2380-lkp@intel.com



2025-07-30 19:06:25 sudo /usr/src/linux-perf-x86_64-rhel-9.4-bpf-e350af63969b875598f0656a20d801bbcaa7bd76/tools/perf/perf test 64 -v
 64: Convert perf time to TSC                                        :
 64.1: TSC support                                                   : Running (2 active)
 64.1: TSC support                                                   : Ok
--- start ---
test child forked, pid 7359
Using CPUID GenuineIntel-6-9E-D
evlist__open() failed
---- end(-1) ----
 64.2: Perf time to TSC                                              : FAILED!



The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250731/202507310818.a05d2380-lkp@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Hi,

On Thu, Jul 31, 2025 at 03:26:01PM +0800, kernel test robot wrote:

[...]

> 2025-07-30 19:06:25 sudo /usr/src/linux-perf-x86_64-rhel-9.4-bpf-e350af63969b875598f0656a20d801bbcaa7bd76/tools/perf/perf test 64 -v
>  64: Convert perf time to TSC                                        :
>  64.1: TSC support                                                   : Running (2 active)
>  64.1: TSC support                                                   : Ok
> --- start ---
> test child forked, pid 7359
> Using CPUID GenuineIntel-6-9E-D
> evlist__open() failed
> ---- end(-1) ----
>  64.2: Perf time to TSC                                              : FAILED!

I roughly read the job.yaml file, seems it does not install clang.
Thus, the building will not enalbe the option BUILD_BPF_SKEL=1.

As a result, auxtrace__update_bpf_map() will return a failure. I will
fix this issue in next spin.

Thanks for test and reporting.

Leo

> The kernel config and materials to reproduce are available at:
> https://download.01.org/0day-ci/archive/20250731/202507310818.a05d2380-lkp@intel.com
> 
> 
> 
> -- 
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki
>

Hi,

On Thu, Jul 31, 2025 at 10:11:48AM +0100, Leo Yan wrote:
> Hi,
> 
> On Thu, Jul 31, 2025 at 03:26:01PM +0800, kernel test robot wrote:
> 
> [...]
> 
> > 2025-07-30 19:06:25 sudo /usr/src/linux-perf-x86_64-rhel-9.4-bpf-e350af63969b875598f0656a20d801bbcaa7bd76/tools/perf/perf test 64 -v
> >  64: Convert perf time to TSC                                        :
> >  64.1: TSC support                                                   : Running (2 active)
> >  64.1: TSC support                                                   : Ok
> > --- start ---
> > test child forked, pid 7359
> > Using CPUID GenuineIntel-6-9E-D
> > evlist__open() failed
> > ---- end(-1) ----
> >  64.2: Perf time to TSC                                              : FAILED!
> 
> I roughly read the job.yaml file, seems it does not install clang.
> Thus, the building will not enalbe the option BUILD_BPF_SKEL=1.
> 
> As a result, auxtrace__update_bpf_map() will return a failure. I will
> fix this issue in next spin.
> 
> Thanks for test and reporting.

I played a bit and found that if with the option "BUILD_BPF_SKEL := 0",
I even failed to build perf (I will fix in new version).

I think you should have built perf with enabled BUILD_BPF_SKEL. However,
in this case, I cannot reproduce the test failure, both on my x86_64 and
Arm64 machines. I reviewed the code and have no clue this patch will
cause the failure.

Is it possible to run the test at your side with the option "perf test
64 -vvv" for debugging info?  Or if it is possible related to the
setting kernel.perf_event_paranoid?

Thanks,
Leo