From: Petar Gligoric <petar.gligoric@rohde-schwarz.com>

This patch-series introduces the task analyzer and adds the feature to
output csv files for further analysis in thirds party script 
(e.g. pandas and friends). 

The task analyzer dissects recorded perf.data files based on
sched:sched_switch events. It outputs useful information for the user
of each task, like times between schedule in/schedule out of the same
task.

    Switched-In      Switched-Out CPU      PID      TID             Comm    Runtime     Time Out-In
15576.658891407   15576.659156086   4     2412     2428            gdbus        265            1949
15576.659111320   15576.659455410   0     2412     2412      gnome-shell        344            2267
15576.659491326   15576.659506173   2       74       74      kworker/2:1         15           13145
[...]
 
The user can modify the output to his liking and his necessity. He can
either limit the output to tasks he wants or filter tasks he does not
present in the output. The output can also be limited via the time or
specific tasks can be highlight via colors. A combination of those 
options is also possible.
Additionally the user can print out a summary of all tasks, which is a
table of information from all tasks throughout the whole trace.
Information, like total runtime, how many times the tasks have been 
scheduled in, what the max runtime was and when it occurred, just to 
name few.

Summary
    Task Information                       Runtime Information
PID   TID            Comm Runs Accumulated    Mean  Median  Min   Max          Max At
 14    14     ksoftirqd/0   13         334      26      15    9   127 15571.621211956
 15    15     rcu_preempt  133        1778      13      13    2    33 15572.581176024
 16    16     migration/0    3          49      16      13   12    24 15571.608915425
[...]

The standard task as well as the summary can be printed in either
nanoseconds, milliseconds or microseconds(standard). 
Both standard as well as summary can be saved in a user specified file
in csv format.        

v2:
 - Add tests for new perf task-analyzer suggested by Ian Rogers

Hagen Paul Pfeifer (1):
  perf script: introduce task analyzer

Petar Gligoric (2):
  perf script: task-analyzer add csv support
  perf test: Add new task-analyzer tests

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: Namhyung Kim <namhyung@kernel.org>

 .../scripts/python/bin/tasks-analyzer-record  |   2 +
 .../scripts/python/bin/tasks-analyzer-report  |   3 +
 tools/perf/scripts/python/tasks-analyzer.py   | 937 ++++++++++++++++++
 3 files changed, 942 insertions(+)
 create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-record
 create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-report
 create mode 100755 tools/perf/scripts/python/tasks-analyzer.py
-- 
2.30.2

Hello,

On Tue, Dec 6, 2022 at 7:44 AM Petar Gligoric <petar.gligor@gmail.com> wrote:
>
> From: Petar Gligoric <petar.gligoric@rohde-schwarz.com>
>
> This patch-series introduces the task analyzer and adds the feature to
> output csv files for further analysis in thirds party script
> (e.g. pandas and friends).
>
> The task analyzer dissects recorded perf.data files based on
> sched:sched_switch events. It outputs useful information for the user
> of each task, like times between schedule in/schedule out of the same
> task.
>
>     Switched-In      Switched-Out CPU      PID      TID             Comm    Runtime     Time Out-In
> 15576.658891407   15576.659156086   4     2412     2428            gdbus        265            1949
> 15576.659111320   15576.659455410   0     2412     2412      gnome-shell        344            2267
> 15576.659491326   15576.659506173   2       74       74      kworker/2:1         15           13145
> [...]
>
> The user can modify the output to his liking and his necessity. He can
> either limit the output to tasks he wants or filter tasks he does not
> present in the output. The output can also be limited via the time or
> specific tasks can be highlight via colors. A combination of those
> options is also possible.
> Additionally the user can print out a summary of all tasks, which is a
> table of information from all tasks throughout the whole trace.
> Information, like total runtime, how many times the tasks have been
> scheduled in, what the max runtime was and when it occurred, just to
> name few.
>
> Summary
>     Task Information                       Runtime Information
> PID   TID            Comm Runs Accumulated    Mean  Median  Min   Max          Max At
>  14    14     ksoftirqd/0   13         334      26      15    9   127 15571.621211956
>  15    15     rcu_preempt  133        1778      13      13    2    33 15572.581176024
>  16    16     migration/0    3          49      16      13   12    24 15571.608915425
> [...]
>
> The standard task as well as the summary can be printed in either
> nanoseconds, milliseconds or microseconds(standard).
> Both standard as well as summary can be saved in a user specified file
> in csv format.

Have you looked at 'perf sched timehist' ?
I think it has the common functionality and can be easily extended if there's
missing one.

Thanks,
Namhyung

>
> v2:
>  - Add tests for new perf task-analyzer suggested by Ian Rogers
>
> Hagen Paul Pfeifer (1):
>   perf script: introduce task analyzer
>
> Petar Gligoric (2):
>   perf script: task-analyzer add csv support
>   perf test: Add new task-analyzer tests
>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Andi Kleen <ak@linux.intel.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Ian Rogers <irogers@google.com>
> Cc: Namhyung Kim <namhyung@kernel.org>
>
>  .../scripts/python/bin/tasks-analyzer-record  |   2 +
>  .../scripts/python/bin/tasks-analyzer-report  |   3 +
>  tools/perf/scripts/python/tasks-analyzer.py   | 937 ++++++++++++++++++
>  3 files changed, 942 insertions(+)
>  create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-record
>  create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-report
>  create mode 100755 tools/perf/scripts/python/tasks-analyzer.py
> --
> 2.30.2
>

On Tue, Dec 06, 2022 at 10:32:35AM -0800, Namhyung Kim wrote:
> Hello,
> 
> Have you looked at 'perf sched timehist' ?
> I think it has the common functionality and can be easily extended if there's
> missing one.
> 
> Thanks,
> Namhyung
> 

Thanks for the input! For this patchset we explicitly decided against
extending "perf sched timehist" - after some pros and cons. Mainly we
didn't want to break existing programs (which might parse the output of
perf sched) and also the goal of the task-analyzer is a bit different.
E.g what will follow as a follow-up patch, is to show IRQs visually
pleasing intermixed with tasks to show potential sources of task
latency. This will be offered as an option for the task-analyzer, but
would be too much functionality for "perf sched timehist". This was the
main reason why we decided against the extension.

Best Regards,
Petar

On Tue, Dec 6, 2022 at 2:21 PM Petar Gligoric <petar.gligor@gmail.com> wrote:
>
> On Tue, Dec 06, 2022 at 10:32:35AM -0800, Namhyung Kim wrote:
> > Hello,
> >
> > Have you looked at 'perf sched timehist' ?
> > I think it has the common functionality and can be easily extended if there's
> > missing one.
> >
> > Thanks,
> > Namhyung
> >
>
> Thanks for the input! For this patchset we explicitly decided against
> extending "perf sched timehist" - after some pros and cons. Mainly we
> didn't want to break existing programs (which might parse the output of
> perf sched) and also the goal of the task-analyzer is a bit different.
> E.g what will follow as a follow-up patch, is to show IRQs visually
> pleasing intermixed with tasks to show potential sources of task
> latency. This will be offered as an option for the task-analyzer, but
> would be too much functionality for "perf sched timehist". This was the
> main reason why we decided against the extension.

Then you might want to add a new sub-command under perf sched.
But I guess we can just add a new option for the different output
format in the perf sched timehist.

Anyway, "perf script" is a generic tool not targeting specific events.
This functionality requires sched_switch (and more?) then we need
the record part to make sure the data has the events.  That's why
it's natural to have it in perf sched IMHO.

Thanks,
Namhyung

> > Thanks for the input! For this patchset we explicitly decided against
> > extending "perf sched timehist" - after some pros and cons. Mainly we
> > didn't want to break existing programs (which might parse the output of
> > perf sched) and also the goal of the task-analyzer is a bit different.
> > E.g what will follow as a follow-up patch, is to show IRQs visually
> > pleasing intermixed with tasks to show potential sources of task
> > latency. This will be offered as an option for the task-analyzer, but
> > would be too much functionality for "perf sched timehist". This was the
> > main reason why we decided against the extension.
> 
> Then you might want to add a new sub-command under perf sched.
> But I guess we can just add a new option for the different output
> format in the perf sched timehist.
> 
> Anyway, "perf script" is a generic tool not targeting specific events.
> This functionality requires sched_switch (and more?) then we need
> the record part to make sure the data has the events.  That's why
> it's natural to have it in perf sched IMHO.
> 
> Thanks,
> Namhyung

We assumed that python scripts should not only be used as a "generic tool".
There are a number of tools like flamegraph, netdev-times, dropmonitor and
other scripts that analyze and look at *very specific* events. The question is
rather, why should this be implemented in C? When using Python - with batteries
included - less code has to be written for the identical result, and it is less
error-prone than C (less technical debt). We have measured the performance,
even for very large perf.data files, and again we see no problems.

But maybe this should be clarified in principle! What do you say Arnaldo, Ian,
...?

Petar

On Thu, Dec 8, 2022 at 4:15 AM Petar Gligoric <petar.gligor@gmail.com> wrote:
>
> > > Thanks for the input! For this patchset we explicitly decided against
> > > extending "perf sched timehist" - after some pros and cons. Mainly we
> > > didn't want to break existing programs (which might parse the output of
> > > perf sched) and also the goal of the task-analyzer is a bit different.
> > > E.g what will follow as a follow-up patch, is to show IRQs visually
> > > pleasing intermixed with tasks to show potential sources of task
> > > latency. This will be offered as an option for the task-analyzer, but
> > > would be too much functionality for "perf sched timehist". This was the
> > > main reason why we decided against the extension.
> >
> > Then you might want to add a new sub-command under perf sched.
> > But I guess we can just add a new option for the different output
> > format in the perf sched timehist.
> >
> > Anyway, "perf script" is a generic tool not targeting specific events.
> > This functionality requires sched_switch (and more?) then we need
> > the record part to make sure the data has the events.  That's why
> > it's natural to have it in perf sched IMHO.
> >
> > Thanks,
> > Namhyung
>
> We assumed that python scripts should not only be used as a "generic tool".

Oh, I didn't know you worked on a separate python script.
Sorry, I just read the cover letter (but not the diffstat) and
I thought you added it to the builtin-script.c. ;-)

> There are a number of tools like flamegraph, netdev-times, dropmonitor and
> other scripts that analyze and look at *very specific* events. The question is
> rather, why should this be implemented in C? When using Python - with batteries
> included - less code has to be written for the identical result, and it is less
> error-prone than C (less technical debt). We have measured the performance,
> even for very large perf.data files, and again we see no problems.

Then I have no problem with python.  Please take a look at the
scripts/python/bin directory to provide shortcuts for record/report.

Thanks,
Namhyung


>
> But maybe this should be clarified in principle! What do you say Arnaldo, Ian,
> ...?
>
> Petar
>

>On Thu, Dec 08, 2022 at 11:10:58AM -0800, Namhyung Kim wrote:
> 
> Then I have no problem with python.  Please take a look at the
> scripts/python/bin directory to provide shortcuts for record/report.
> 
> Thanks,
> Namhyung
> 
> 

Great that we could resolve it! Regarding wrapper: I thought I added
them or did you mean something else?

.../scripts/python/bin/tasks-analyzer-record |  2 +
.../scripts/python/bin/tasks-analyzer-report |  3 +

Best Regards,
Petar

On Thu, Dec 8, 2022 at 12:11 PM Petar Gligoric <petar.gligor@gmail.com> wrote:
>
> >On Thu, Dec 08, 2022 at 11:10:58AM -0800, Namhyung Kim wrote:
> >
> > Then I have no problem with python.  Please take a look at the
> > scripts/python/bin directory to provide shortcuts for record/report.
>
> Great that we could resolve it! Regarding wrapper: I thought I added
> them or did you mean something else?
>
> .../scripts/python/bin/tasks-analyzer-record |  2 +
> .../scripts/python/bin/tasks-analyzer-report |  3 +

Right, that's what I meant.  You got it already! :)

Thanks,
Namhyung

Em Thu, Dec 08, 2022 at 12:15:13PM -0800, Namhyung Kim escreveu:
> On Thu, Dec 8, 2022 at 12:11 PM Petar Gligoric <petar.gligor@gmail.com> wrote:
> >
> > >On Thu, Dec 08, 2022 at 11:10:58AM -0800, Namhyung Kim wrote:
> > >
> > > Then I have no problem with python.  Please take a look at the
> > > scripts/python/bin directory to provide shortcuts for record/report.
> >
> > Great that we could resolve it! Regarding wrapper: I thought I added
> > them or did you mean something else?
> >
> > .../scripts/python/bin/tasks-analyzer-record |  2 +
> > .../scripts/python/bin/tasks-analyzer-report |  3 +
> 
> Right, that's what I meant.  You got it already! :)

I don't see any objection to merging this, comes with a set of 'perf
test' to make sure we don't regress, I see no reason not to merge, done.

- Arnaldo