In perf-record, when an event is triggered, default behavior is to
save sample data to perf.data. Sometimes, we may just want to do
some lightweight actions, such as printing a log.
    
Based on this requirement, add the --action option to the event to
specify the behavior when the event occurs.

This patchset uses bpf prog to attach to tracepoint event, and save sample
to bpf perf_event ringbuffer in handler. perf-tool read the data and run actions.

Currently only one call is supported, that is, print(),
and some commonly used builtin variables are also supported.

For example:

  # perf record -e sched:sched_switch --action 'print("[%03d][%llu]comm=%s, pid=%d, tid=%d\n", cpu, time, comm, pid, tid)' true
  [003][795464100275136]comm=perf, pid=141580, tid=141580
  [003][795464100278234]comm=swapper/3, pid=0, tid=0
  [003][795464100288984]comm=perf, pid=141580, tid=141580
  [003][795464100457865]comm=swapper/3, pid=0, tid=0
  [003][795464100485547]comm=perf, pid=141580, tid=141580
  [003][795464100491398]comm=kworker/u36:1, pid=139834, tid=139834
  [003][795464100493647]comm=perf, pid=141580, tid=141580
  [003][795464100494967]comm=kworker/u36:1, pid=139834, tid=139834
  [003][795464100498146]comm=perf, pid=141580, tid=141580
  ...

  # perf record -e cycles --action 'print("test\n");' true
  bpf record action only supports specifying for tracepoint tracer

  # perf record -e sched:sched_switch --action 'print("[%llu]comm=%s, cpu=%d, pid=%d, tid=%d\n", time, comm, cpu, pid)' true
  print() arguments number for format string mismatch: 5 expected, 4 provided
  parse action option failed

   Usage: perf record [<options>] [<command>]
      or: perf record [<options>] -- <command> [<options>]

          --action <action>
                            event action

  # perf record -e sched:sched_switch --action 'print("test\n");' true
  test
  test
  test
  test
  test
  test
  test
  test
  test
  test
  ...

This patchset implements simple features and can be extended as needed.

TODO LIST:
1. Support common operations such as logical operations and bit operations
2. Support other calls such as dumpstack(), count()
3. Support specify actions for kprobe events
4. For builds that disable bpf_skel, support real-time parsing of perf record mmap ringbuffer data (similar to perf top)
5. Link libllvm to support dynamic generation of bpf progs

Yang Jihong (12):
  perf record: Add event action support
  perf event action: Add parsing const expr support
  perf event action: Add parsing const integer expr support
  perf event action: Add parsing const string expr support
  perf event action: Add parsing call expr support
  perf event action: Add parsing print() call expr support
  perf event action: Add parsing builtin expr support
  perf event action: Add parsing builtin cpu expr support
  perf event action: Add parsing builtin pid expr support
  perf event action: Add parsing builtin tid expr support
  perf event action: Add parsing builtin comm expr support
  perf event action: Add parsing builtin time expr support

 tools/perf/Documentation/perf-record.txt     |   8 +
 tools/perf/Makefile.perf                     |   1 +
 tools/perf/builtin-record.c                  |  31 +
 tools/perf/util/Build                        |  18 +
 tools/perf/util/bpf_skel/bpf_record_action.h |  24 +
 tools/perf/util/bpf_skel/record_action.bpf.c | 151 ++++
 tools/perf/util/parse-action.c               | 729 +++++++++++++++++++
 tools/perf/util/parse-action.h               |  98 +++
 tools/perf/util/parse-action.l               | 190 +++++
 tools/perf/util/parse-action.y               | 156 ++++
 tools/perf/util/record_action.c              | 380 ++++++++++
 tools/perf/util/record_action.h              |  30 +
 12 files changed, 1816 insertions(+)
 create mode 100644 tools/perf/util/bpf_skel/bpf_record_action.h
 create mode 100644 tools/perf/util/bpf_skel/record_action.bpf.c
 create mode 100644 tools/perf/util/parse-action.c
 create mode 100644 tools/perf/util/parse-action.h
 create mode 100644 tools/perf/util/parse-action.l
 create mode 100644 tools/perf/util/parse-action.y
 create mode 100644 tools/perf/util/record_action.c
 create mode 100644 tools/perf/util/record_action.h

-- 
2.25.1

On Thu, Nov 28, 2024 at 09:35:41PM +0800, Yang Jihong wrote:
> In perf-record, when an event is triggered, default behavior is to
> save sample data to perf.data. Sometimes, we may just want to do
> some lightweight actions, such as printing a log.

> Based on this requirement, add the --action option to the event to
> specify the behavior when the event occurs.

'perf record' is centered on saving data to disk without processing
events, while it has sideband events for some needs, like processing BPF
related events (PERF_RECORD_BPF_EVENT to catch PERF_BPF_EVENT_PROG_LOAD
and UNLOAD), doing things in a "live" way as your patchkit does seems
more appropriate to do in 'perf trace' :-)

I'll take a look at the rest of the patch series.

- Arnaldo
 
> This patchset uses bpf prog to attach to tracepoint event, and save sample
> to bpf perf_event ringbuffer in handler. perf-tool read the data and run actions.
> 
> Currently only one call is supported, that is, print(),
> and some commonly used builtin variables are also supported.
> 
> For example:
> 
>   # perf record -e sched:sched_switch --action 'print("[%03d][%llu]comm=%s, pid=%d, tid=%d\n", cpu, time, comm, pid, tid)' true
>   [003][795464100275136]comm=perf, pid=141580, tid=141580
>   [003][795464100278234]comm=swapper/3, pid=0, tid=0
>   [003][795464100288984]comm=perf, pid=141580, tid=141580
>   [003][795464100457865]comm=swapper/3, pid=0, tid=0
>   [003][795464100485547]comm=perf, pid=141580, tid=141580
>   [003][795464100491398]comm=kworker/u36:1, pid=139834, tid=139834
>   [003][795464100493647]comm=perf, pid=141580, tid=141580
>   [003][795464100494967]comm=kworker/u36:1, pid=139834, tid=139834
>   [003][795464100498146]comm=perf, pid=141580, tid=141580
>   ...
> 
>   # perf record -e cycles --action 'print("test\n");' true
>   bpf record action only supports specifying for tracepoint tracer
> 
>   # perf record -e sched:sched_switch --action 'print("[%llu]comm=%s, cpu=%d, pid=%d, tid=%d\n", time, comm, cpu, pid)' true
>   print() arguments number for format string mismatch: 5 expected, 4 provided
>   parse action option failed
> 
>    Usage: perf record [<options>] [<command>]
>       or: perf record [<options>] -- <command> [<options>]
> 
>           --action <action>
>                             event action
> 
>   # perf record -e sched:sched_switch --action 'print("test\n");' true
>   test
>   test
>   test
>   test
>   test
>   test
>   test
>   test
>   test
>   test
>   ...
> 
> This patchset implements simple features and can be extended as needed.
> 
> TODO LIST:
> 1. Support common operations such as logical operations and bit operations
> 2. Support other calls such as dumpstack(), count()
> 3. Support specify actions for kprobe events
> 4. For builds that disable bpf_skel, support real-time parsing of perf record mmap ringbuffer data (similar to perf top)
> 5. Link libllvm to support dynamic generation of bpf progs
> 
> Yang Jihong (12):
>   perf record: Add event action support
>   perf event action: Add parsing const expr support
>   perf event action: Add parsing const integer expr support
>   perf event action: Add parsing const string expr support
>   perf event action: Add parsing call expr support
>   perf event action: Add parsing print() call expr support
>   perf event action: Add parsing builtin expr support
>   perf event action: Add parsing builtin cpu expr support
>   perf event action: Add parsing builtin pid expr support
>   perf event action: Add parsing builtin tid expr support
>   perf event action: Add parsing builtin comm expr support
>   perf event action: Add parsing builtin time expr support
> 
>  tools/perf/Documentation/perf-record.txt     |   8 +
>  tools/perf/Makefile.perf                     |   1 +
>  tools/perf/builtin-record.c                  |  31 +
>  tools/perf/util/Build                        |  18 +
>  tools/perf/util/bpf_skel/bpf_record_action.h |  24 +
>  tools/perf/util/bpf_skel/record_action.bpf.c | 151 ++++
>  tools/perf/util/parse-action.c               | 729 +++++++++++++++++++
>  tools/perf/util/parse-action.h               |  98 +++
>  tools/perf/util/parse-action.l               | 190 +++++
>  tools/perf/util/parse-action.y               | 156 ++++
>  tools/perf/util/record_action.c              | 380 ++++++++++
>  tools/perf/util/record_action.h              |  30 +
>  12 files changed, 1816 insertions(+)
>  create mode 100644 tools/perf/util/bpf_skel/bpf_record_action.h
>  create mode 100644 tools/perf/util/bpf_skel/record_action.bpf.c
>  create mode 100644 tools/perf/util/parse-action.c
>  create mode 100644 tools/perf/util/parse-action.h
>  create mode 100644 tools/perf/util/parse-action.l
>  create mode 100644 tools/perf/util/parse-action.y
>  create mode 100644 tools/perf/util/record_action.c
>  create mode 100644 tools/perf/util/record_action.h
> 
> -- 
> 2.25.1

Hello,

On Thu, Nov 28, 2024 at 05:14:53PM -0300, Arnaldo Carvalho de Melo wrote:
> On Thu, Nov 28, 2024 at 09:35:41PM +0800, Yang Jihong wrote:
> > In perf-record, when an event is triggered, default behavior is to
> > save sample data to perf.data. Sometimes, we may just want to do
> > some lightweight actions, such as printing a log.
> 
> > Based on this requirement, add the --action option to the event to
> > specify the behavior when the event occurs.
> 
> 'perf record' is centered on saving data to disk without processing
> events, while it has sideband events for some needs, like processing BPF
> related events (PERF_RECORD_BPF_EVENT to catch PERF_BPF_EVENT_PROG_LOAD
> and UNLOAD), doing things in a "live" way as your patchkit does seems
> more appropriate to do in 'perf trace' :-)

Agreed, 'perf trace' looks like a better place as you seem to target
tracepoint events mostly.

Thanks,
Namhyung

Hello,

On 12/3/24 05:46, Namhyung Kim wrote:
> Hello,
> 
> On Thu, Nov 28, 2024 at 05:14:53PM -0300, Arnaldo Carvalho de Melo wrote:
>> On Thu, Nov 28, 2024 at 09:35:41PM +0800, Yang Jihong wrote:
>>> In perf-record, when an event is triggered, default behavior is to
>>> save sample data to perf.data. Sometimes, we may just want to do
>>> some lightweight actions, such as printing a log.
>>
>>> Based on this requirement, add the --action option to the event to
>>> specify the behavior when the event occurs.
>>
>> 'perf record' is centered on saving data to disk without processing
>> events, while it has sideband events for some needs, like processing BPF
>> related events (PERF_RECORD_BPF_EVENT to catch PERF_BPF_EVENT_PROG_LOAD
>> and UNLOAD), doing things in a "live" way as your patchkit does seems
>> more appropriate to do in 'perf trace' :-)
> 
> Agreed, 'perf trace' looks like a better place as you seem to target
> tracepoint events mostly.
> 
Okay, will do it in 'perf trace'.

Attaching to a kprobe events will also be supported in the future.

Thanks,
Yang

On Wed, Dec 4, 2024 at 12:35 AM Yang Jihong <yangjihong@bytedance.com> wrote:
>
> Hello,
>
> On 12/3/24 05:46, Namhyung Kim wrote:
> > Hello,
> >
> > On Thu, Nov 28, 2024 at 05:14:53PM -0300, Arnaldo Carvalho de Melo wrote:
> >> On Thu, Nov 28, 2024 at 09:35:41PM +0800, Yang Jihong wrote:
> >>> In perf-record, when an event is triggered, default behavior is to
> >>> save sample data to perf.data. Sometimes, we may just want to do
> >>> some lightweight actions, such as printing a log.
> >>
> >>> Based on this requirement, add the --action option to the event to
> >>> specify the behavior when the event occurs.
> >>
> >> 'perf record' is centered on saving data to disk without processing
> >> events, while it has sideband events for some needs, like processing BPF
> >> related events (PERF_RECORD_BPF_EVENT to catch PERF_BPF_EVENT_PROG_LOAD
> >> and UNLOAD), doing things in a "live" way as your patchkit does seems
> >> more appropriate to do in 'perf trace' :-)
> >
> > Agreed, 'perf trace' looks like a better place as you seem to target
> > tracepoint events mostly.
> >
> Okay, will do it in 'perf trace'.
>
> Attaching to a kprobe events will also be supported in the future.

Hi Yang,

Just some extra information in case it is useful on python scripting,
which could be reused to avoid writing a new interpreter as in this
series. There are quite a few perf script examples here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/scripts/python?h=perf-tools-next
There are also standalone scripts here including for tracepoints:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/python?h=perf-tools-next
One of the things missing for standalone scripts was being able to
conveniently parse events - you'd need to manually determine PMU type
and config values. That is fixed in this reviewed but not merged
series:
https://lore.kernel.org/lkml/20241119011644.971342-1-irogers@google.com/

I'd like to see the python code improved as it allows quicker command
creation like your example. Another example I'd recently tweaked is:
https://lore.kernel.org/lkml/20241119180130.19160-1-irogers@google.com/
Ultimately I think it would be nice to be creating tools using UIs
like textualize:
https://www.textualize.io/
and our Gecko Google summer-of-code project was an improvement here:
https://lore.kernel.org/lkml/ZOrsiZA+C0zbWEQS@yoga/

Other than basic functionality, perf's python support could be
improved in a number of ways that we've discussed in office hours or
on the list. Some things that come to mind:
1) The perf script callback mechanism doesn't feel particularly
pythonic, generators or coroutines could be better.
2) We should probably have more library python code rather than adding
new functions to `tools/perf/util/python.c` - I may do this to add
type hints for mypy. If we could have a subset that works without the
C code then it could be a basis for `perf.data` file analysis not tied
to Linux, or `pip install`-able. There is a similar library with
simpleperf, but shipping something with the perf codebase means we can
keep the file format and library synchronized.
3) We'd like to reduce the number of dependencies `perf` as a command
has, and libpython is a big one. Something `uftrace` did was dlopen
libpython to avoid needing to link against it. A project similarly
rethinking python binding with C code is HPy.

Thanks,
Ian

Hello,

On 12/5/24 03:54, Ian Rogers wrote:
> On Wed, Dec 4, 2024 at 12:35 AM Yang Jihong <yangjihong@bytedance.com> wrote:
>>
>> Hello,
>>
>> On 12/3/24 05:46, Namhyung Kim wrote:
>>> Hello,
>>>
>>> On Thu, Nov 28, 2024 at 05:14:53PM -0300, Arnaldo Carvalho de Melo wrote:
>>>> On Thu, Nov 28, 2024 at 09:35:41PM +0800, Yang Jihong wrote:
>>>>> In perf-record, when an event is triggered, default behavior is to
>>>>> save sample data to perf.data. Sometimes, we may just want to do
>>>>> some lightweight actions, such as printing a log.
>>>>
>>>>> Based on this requirement, add the --action option to the event to
>>>>> specify the behavior when the event occurs.
>>>>
>>>> 'perf record' is centered on saving data to disk without processing
>>>> events, while it has sideband events for some needs, like processing BPF
>>>> related events (PERF_RECORD_BPF_EVENT to catch PERF_BPF_EVENT_PROG_LOAD
>>>> and UNLOAD), doing things in a "live" way as your patchkit does seems
>>>> more appropriate to do in 'perf trace' :-)
>>>
>>> Agreed, 'perf trace' looks like a better place as you seem to target
>>> tracepoint events mostly.
>>>
>> Okay, will do it in 'perf trace'.
>>
>> Attaching to a kprobe events will also be supported in the future.
> 
> Hi Yang,
> 
> Just some extra information in case it is useful on python scripting,
> which could be reused to avoid writing a new interpreter as in this
> series. There are quite a few perf script examples here:

Thanks for your suggestion. I think using python scripting, similar to 
bcc, is more suitable for scenarios with complex requirements.
For some simple and lightweight scenarios, such as just printing some 
information in real time, one-line command (similar to dtrace)may be 
more suitable to implement user-defined sample.
This is just my idea, hope to discuss to improve the usability of 
perf-tool. :-)

> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/scripts/python?h=perf-tools-next
> There are also standalone scripts here including for tracepoints:
> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/python?h=perf-tools-next
> One of the things missing for standalone scripts was being able to
> conveniently parse events - you'd need to manually determine PMU type
> and config values. That is fixed in this reviewed but not merged
> series:
> https://lore.kernel.org/lkml/20241119011644.971342-1-irogers@google.com/
> 
> I'd like to see the python code improved as it allows quicker command
> creation like your example. Another example I'd recently tweaked is:
> https://lore.kernel.org/lkml/20241119180130.19160-1-irogers@google.com/
> Ultimately I think it would be nice to be creating tools using UIs
> like textualize:
> https://www.textualize.io/
> and our Gecko Google summer-of-code project was an improvement here:
> https://lore.kernel.org/lkml/ZOrsiZA+C0zbWEQS@yoga/
> 
> Other than basic functionality, perf's python support could be
> improved in a number of ways that we've discussed in office hours or
> on the list. Some things that come to mind:
> 1) The perf script callback mechanism doesn't feel particularly
> pythonic, generators or coroutines could be better.
> 2) We should probably have more library python code rather than adding
> new functions to `tools/perf/util/python.c` - I may do this to add
> type hints for mypy. If we could have a subset that works without the
> C code then it could be a basis for `perf.data` file analysis not tied
> to Linux, or `pip install`-able. There is a similar library with
> simpleperf, but shipping something with the perf codebase means we can
> keep the file format and library synchronized.
> 3) We'd like to reduce the number of dependencies `perf` as a command
> has, and libpython is a big one. Something `uftrace` did was dlopen
> libpython to avoid needing to link against it. A project similarly
> rethinking python binding with C code is HPy.
> 
Yes, we can greatly simplify the development work by using python.
In the end, if we can allow users to import perf python library and use 
a few simple codes to implement a customized data collection, it will be 
of great help to improve the usability of the perf tool.


Actually, I think perf-tool has a lot of functions and is very helpful 
for performance optimization, but the threshold is relatively high for 
beginners. (I don’t know if this is appropriate :-))

I have some considerations in improving the usability of the tool, but I 
don’t know if it is appropriate:
1. Provide rich prompt information, such as perf-help can provide some 
user requirements in an interactive way, provide common perf commands 
and usage methods; or provide some usage cases through keywords
2. Use some graphical interfaces as the backend so that users can use it 
in an easy-to-understand way
3. Based on the python you mentioned above, I think it is a good way to 
allow some developers to customize sampling data more easily.


Thanks,
Yang.

Hello,

On 11/29/24 04:14, Arnaldo Carvalho de Melo wrote:
> On Thu, Nov 28, 2024 at 09:35:41PM +0800, Yang Jihong wrote:
>> In perf-record, when an event is triggered, default behavior is to
>> save sample data to perf.data. Sometimes, we may just want to do
>> some lightweight actions, such as printing a log.
> 
>> Based on this requirement, add the --action option to the event to
>> specify the behavior when the event occurs.
> 
> 'perf record' is centered on saving data to disk without processing
> events, while it has sideband events for some needs, like processing BPF
> related events (PERF_RECORD_BPF_EVENT to catch PERF_BPF_EVENT_PROG_LOAD
> and UNLOAD), doing things in a "live" way as your patchkit does seems
> more appropriate to do in 'perf trace' :-)

Ok, we can put it in "perf trace".

> 
> I'll take a look at the rest of the patch series.

Thanks for reviewing this patchset. Currently, this patchset only 
implements a print() call and only supports attaching to tracepoints. 
The functionality is relatively simple.

Eventually, we can use bpf prog to allow users to customize the sampled 
data to and the processing of the data.

we can continue to expand the functionality (for example, attaching to 
kprobe events, printing the parameters and return values ​​of the target 
function)


Thanks,
Yang

On 28/11/24 15:35, Yang Jihong wrote:
> In perf-record, when an event is triggered, default behavior is to
> save sample data to perf.data. Sometimes, we may just want to do
> some lightweight actions, such as printing a log.

Why not just pipe 'perf record' to 'perf script'?

# perf record -e sched:sched_switch | perf script | head
            perf  768231 [000] 2318380.474267: sched:sched_switch: perf:768231 [120] R ==> migration/0:18 [0]
     migration/0      18 [000] 2318380.474294: sched:sched_switch: migration/0:18 [0] S ==> swapper/0:0 [120]
            perf  768231 [001] 2318380.474353: sched:sched_switch: perf:768231 [120] R ==> migration/1:23 [0]
     migration/1      23 [001] 2318380.474382: sched:sched_switch: migration/1:23 [0] S ==> swapper/1:0 [120]
            perf  768231 [002] 2318380.474477: sched:sched_switch: perf:768231 [120] R ==> migration/2:29 [0]
     migration/2      29 [002] 2318380.474503: sched:sched_switch: migration/2:29 [0] S ==> swapper/2:0 [120]
            perf  768231 [003] 2318380.474513: sched:sched_switch: perf:768231 [120] R ==> migration/3:35 [0]
     migration/3      35 [003] 2318380.474523: sched:sched_switch: migration/3:35 [0] S ==> swapper/3:0 [120]
            perf  768231 [004] 2318380.474534: sched:sched_switch: perf:768231 [120] R ==> migration/4:41 [0]
     migration/4      41 [004] 2318380.474541: sched:sched_switch: migration/4:41 [0] S ==> swapper/4:0 [120]

Hello,

On 11/28/24 21:53, Adrian Hunter wrote:
> On 28/11/24 15:35, Yang Jihong wrote:
>> In perf-record, when an event is triggered, default behavior is to
>> save sample data to perf.data. Sometimes, we may just want to do
>> some lightweight actions, such as printing a log.
> 
> Why not just pipe 'perf record' to 'perf script'?
> 
> # perf record -e sched:sched_switch | perf script | head
>              perf  768231 [000] 2318380.474267: sched:sched_switch: perf:768231 [120] R ==> migration/0:18 [0]
>       migration/0      18 [000] 2318380.474294: sched:sched_switch: migration/0:18 [0] S ==> swapper/0:0 [120]
>              perf  768231 [001] 2318380.474353: sched:sched_switch: perf:768231 [120] R ==> migration/1:23 [0]
>       migration/1      23 [001] 2318380.474382: sched:sched_switch: migration/1:23 [0] S ==> swapper/1:0 [120]
>              perf  768231 [002] 2318380.474477: sched:sched_switch: perf:768231 [120] R ==> migration/2:29 [0]
>       migration/2      29 [002] 2318380.474503: sched:sched_switch: migration/2:29 [0] S ==> swapper/2:0 [120]
>              perf  768231 [003] 2318380.474513: sched:sched_switch: perf:768231 [120] R ==> migration/3:35 [0]
>       migration/3      35 [003] 2318380.474523: sched:sched_switch: migration/3:35 [0] S ==> swapper/3:0 [120]
>              perf  768231 [004] 2318380.474534: sched:sched_switch: perf:768231 [120] R ==> migration/4:41 [0]
>       migration/4      41 [004] 2318380.474541: sched:sched_switch: migration/4:41 [0] S ==> swapper/4:0 [120]

pipe 'perf record' to 'perf script' has limited extensions.
We can use bpf prog to do more (such as saving user-defined data (such 
as registers, function parameters), and customizing the format of 
user-printed logs).

Now we only implement a simple print() call, and we can support more 
calls later.

Thanks,
Yang