This patch series builds up to the addition of a new ilist app written
in python using textual [1] for the UI. The app presents perf PMUs and
events, displays the event information as in `perf list` while at the
bottom of the console showing recent activity of the event in total
and across all CPUs.

The first part of the patches are a few perf and perf python C API
fixes, most importantly the counter reading in python supports tool
PMUs.

The second part of the patches adds event json for the software PMU
and makes the tracepoint PMU support iteration of events and the
like. Without these improvements the tracepoint and software PMUs will
appear to have no events in the ilist app. As the software PMU moves
parsing to json, the legacy hard coded parsing is removed. This has
proven controversial for hardware events and so that cleanup isn't
done here.

The final patch adds the ilist command. To run it you need the updated
perf.cpython.so in your PYTHONPATH and then execute the
script. Expanding PMUs and then selecting events will cause event
informatin to be displayed in the top-right and the counters values to
be displayed as sparklines and counts in the bottom half of the
screen.

Some thoughts on the series:

 - The PMU changes will conflict with the addition of the DRM PMU:
   https://lore.kernel.org/lkml/20250403202439.57791-1-irogers@google.com/
   when these two are merged together ilist will show yet more
   counters. It'd be nice if the DRM stuff could land and then I can
   rebase these patches.

 - The parse-events clean up of the software and tracepoint PMU. The
   software PMU hard coding to be legacy first has similar issues and
   will conflict with the clean up in:
   https://lore.kernel.org/lkml/20250416045117.876775-1-irogers@google.com/
   Moving the software work to json means we don't need special parse
   events terms for software events, etc. We can just treat things
   like regular PMUs with json, etc. I'd much rather we had less
   special case logic so that series is best rebased on top of this
   work and it should drop the changes for software terms, etc. which
   this series removes. Maybe one day the whole event parsing can be
   much more regular in how PMUs are treated but there's always
   "cycles".

 - Should python libraries have feature tests? How does this get
   packaged outside the kernel tree? I think these are open
   questions. Clearly textual is kind of a big dependency and we've
   largely been moving in the direction of fewer dependencies
   recently. Hopefully the app makes it clear why I think this one is
   worth carrying. We carry libslang as a dependency and I think
   textual clearly far surpasses it.

 - How to launch? Currently I run tools/perf/python/ilist.py but it
   would be much nicer if we could do `perf ilist` much as we do for
   perf-archive.sh. There are probably other scripts that should be
   perf commands like flamegraph and gecko. It'd be nice to follow up
   the series with something to make using these commands easy.

 - Additional thoughts were captured on the mailing list:
   https://lore.kernel.org/lkml/CAP-5=fWC+doaVd5rEMWJXSQi_db_Wu2tyAe5Lm6jnQjcwXkF+w@mail.gmail.com/
 
[1] https://textual.textualize.io/

v4: No conflict rebase. Picks up perf-tools-next DRM PMU which
    displays as expected.

v3: Add a search dialog to the ilist app with 'n'ext and 'p'revious
    keys. No changes in the ground work first 14 patches.

v2: In the jevents event description duplication, some minor changes
    accidentally missed from v1 meaning that in v1 the descriptions
    were still duplicated. Expand the cover letter with some thoughts
    on the series.

Ian Rogers (15):
  perf hwmon_pmu: Avoid shortening hwmon PMU name
  perf parse-events: Minor tidy up of event_type helper
  perf python: In str(evsel) use the evsel__pmu_name helper
  perf python: Fix thread check in pyrf_evsel__read
  perf python: Correct pyrf_evsel__read for tool PMUs
  perf python: Add basic PMU abstraction and pmus sequence
  perf python: Add function returning dictionary of all events on a PMU
  perf jevents: If the long_desc and desc are identical then drop the
    long_desc
  perf jevents: Add common software event json
  perf pmu: Tolerate failure to read the type for wellknown PMUs
  perf parse-events: Remove non-json software events
  perf tp_pmu: Factor existing tracepoint logic to new file
  perf tp_pmu: Add event APIs
  perf list: Remove tracepoint printing code
  perf ilist: Add new python ilist command

 tools/perf/builtin-list.c                     |  47 ++-
 .../arch/common/common/software.json          |  92 +++++
 tools/perf/pmu-events/empty-pmu-events.c      | 266 ++++++++-----
 tools/perf/pmu-events/jevents.py              |  18 +-
 tools/perf/python/ilist.py                    | 376 ++++++++++++++++++
 tools/perf/util/Build                         |   1 +
 tools/perf/util/evsel.c                       |  21 +-
 tools/perf/util/hwmon_pmu.c                   |   2 +-
 tools/perf/util/parse-events.c                | 225 ++++-------
 tools/perf/util/parse-events.h                |   3 +-
 tools/perf/util/parse-events.l                |  38 +-
 tools/perf/util/parse-events.y                |  29 +-
 tools/perf/util/pmu.c                         |  44 +-
 tools/perf/util/print-events.c                |  95 -----
 tools/perf/util/print-events.h                |   1 -
 tools/perf/util/python.c                      | 248 +++++++++++-
 tools/perf/util/tp_pmu.c                      | 209 ++++++++++
 tools/perf/util/tp_pmu.h                      |  19 +
 18 files changed, 1282 insertions(+), 452 deletions(-)
 create mode 100644 tools/perf/pmu-events/arch/common/common/software.json
 create mode 100755 tools/perf/python/ilist.py
 create mode 100644 tools/perf/util/tp_pmu.c
 create mode 100644 tools/perf/util/tp_pmu.h

-- 
2.50.0.727.gbf7dc18ff4-goog

Hi Ian,

Sorry for the delay.

On Fri, Jun 27, 2025 at 05:09:14PM -0700, Ian Rogers wrote:
> This patch series builds up to the addition of a new ilist app written
> in python using textual [1] for the UI. The app presents perf PMUs and
> events, displays the event information as in `perf list` while at the
> bottom of the console showing recent activity of the event in total
> and across all CPUs.
> 
> The first part of the patches are a few perf and perf python C API
> fixes, most importantly the counter reading in python supports tool
> PMUs.

The first part looks ok.  It'd be nice if you can send it separately.

> 
> The second part of the patches adds event json for the software PMU
> and makes the tracepoint PMU support iteration of events and the
> like. Without these improvements the tracepoint and software PMUs will
> appear to have no events in the ilist app. As the software PMU moves
> parsing to json, the legacy hard coded parsing is removed. This has
> proven controversial for hardware events and so that cleanup isn't
> done here.

I think it's ok to move hardware events to JSON as well.  Then we can
probably remove the hard coded parsing code.

The tricky part is the hybrid machines.  Would it be possible to define
common hardware with standard (legacy) encoding and to mark it "weak" or
overridable by arch-specific events with the same name?

> 
> The final patch adds the ilist command. To run it you need the updated
> perf.cpython.so in your PYTHONPATH and then execute the
> script. Expanding PMUs and then selecting events will cause event
> informatin to be displayed in the top-right and the counters values to
> be displayed as sparklines and counts in the bottom half of the
> screen.
> 
> Some thoughts on the series:
> 
>  - The PMU changes will conflict with the addition of the DRM PMU:
>    https://lore.kernel.org/lkml/20250403202439.57791-1-irogers@google.com/
>    when these two are merged together ilist will show yet more
>    counters. It'd be nice if the DRM stuff could land and then I can
>    rebase these patches.

Yep, it's merged now.  Please rebase the series.

> 
>  - The parse-events clean up of the software and tracepoint PMU. The
>    software PMU hard coding to be legacy first has similar issues and
>    will conflict with the clean up in:
>    https://lore.kernel.org/lkml/20250416045117.876775-1-irogers@google.com/
>    Moving the software work to json means we don't need special parse
>    events terms for software events, etc. We can just treat things
>    like regular PMUs with json, etc. I'd much rather we had less
>    special case logic so that series is best rebased on top of this
>    work and it should drop the changes for software terms, etc. which
>    this series removes. Maybe one day the whole event parsing can be
>    much more regular in how PMUs are treated but there's always
>    "cycles".

As I said, moving the hardware events would be a solution as long as it
can handle complexities in hybrid.

> 
>  - Should python libraries have feature tests? How does this get
>    packaged outside the kernel tree? I think these are open
>    questions. Clearly textual is kind of a big dependency and we've
>    largely been moving in the direction of fewer dependencies
>    recently. Hopefully the app makes it clear why I think this one is
>    worth carrying. We carry libslang as a dependency and I think
>    textual clearly far surpasses it.

Hmm.. I think it can be checked at runtime.  And maybe it can ask users
to install the required packages.

> 
>  - How to launch? Currently I run tools/perf/python/ilist.py but it
>    would be much nicer if we could do `perf ilist` much as we do for
>    perf-archive.sh. There are probably other scripts that should be
>    perf commands like flamegraph and gecko. It'd be nice to follow up
>    the series with something to make using these commands easy.

Sounds good.

Thanks,
Namhyung

> 
>  - Additional thoughts were captured on the mailing list:
>    https://lore.kernel.org/lkml/CAP-5=fWC+doaVd5rEMWJXSQi_db_Wu2tyAe5Lm6jnQjcwXkF+w@mail.gmail.com/
>  
> [1] https://textual.textualize.io/
> 
> v4: No conflict rebase. Picks up perf-tools-next DRM PMU which
>     displays as expected.
> 
> v3: Add a search dialog to the ilist app with 'n'ext and 'p'revious
>     keys. No changes in the ground work first 14 patches.
> 
> v2: In the jevents event description duplication, some minor changes
>     accidentally missed from v1 meaning that in v1 the descriptions
>     were still duplicated. Expand the cover letter with some thoughts
>     on the series.
> 
> Ian Rogers (15):
>   perf hwmon_pmu: Avoid shortening hwmon PMU name
>   perf parse-events: Minor tidy up of event_type helper
>   perf python: In str(evsel) use the evsel__pmu_name helper
>   perf python: Fix thread check in pyrf_evsel__read
>   perf python: Correct pyrf_evsel__read for tool PMUs
>   perf python: Add basic PMU abstraction and pmus sequence
>   perf python: Add function returning dictionary of all events on a PMU
>   perf jevents: If the long_desc and desc are identical then drop the
>     long_desc
>   perf jevents: Add common software event json
>   perf pmu: Tolerate failure to read the type for wellknown PMUs
>   perf parse-events: Remove non-json software events
>   perf tp_pmu: Factor existing tracepoint logic to new file
>   perf tp_pmu: Add event APIs
>   perf list: Remove tracepoint printing code
>   perf ilist: Add new python ilist command
> 
>  tools/perf/builtin-list.c                     |  47 ++-
>  .../arch/common/common/software.json          |  92 +++++
>  tools/perf/pmu-events/empty-pmu-events.c      | 266 ++++++++-----
>  tools/perf/pmu-events/jevents.py              |  18 +-
>  tools/perf/python/ilist.py                    | 376 ++++++++++++++++++
>  tools/perf/util/Build                         |   1 +
>  tools/perf/util/evsel.c                       |  21 +-
>  tools/perf/util/hwmon_pmu.c                   |   2 +-
>  tools/perf/util/parse-events.c                | 225 ++++-------
>  tools/perf/util/parse-events.h                |   3 +-
>  tools/perf/util/parse-events.l                |  38 +-
>  tools/perf/util/parse-events.y                |  29 +-
>  tools/perf/util/pmu.c                         |  44 +-
>  tools/perf/util/print-events.c                |  95 -----
>  tools/perf/util/print-events.h                |   1 -
>  tools/perf/util/python.c                      | 248 +++++++++++-
>  tools/perf/util/tp_pmu.c                      | 209 ++++++++++
>  tools/perf/util/tp_pmu.h                      |  19 +
>  18 files changed, 1282 insertions(+), 452 deletions(-)
>  create mode 100644 tools/perf/pmu-events/arch/common/common/software.json
>  create mode 100755 tools/perf/python/ilist.py
>  create mode 100644 tools/perf/util/tp_pmu.c
>  create mode 100644 tools/perf/util/tp_pmu.h
> 
> -- 
> 2.50.0.727.gbf7dc18ff4-goog
>

On Wed, Jul 9, 2025 at 9:35 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Hi Ian,
>
> Sorry for the delay.
>
> On Fri, Jun 27, 2025 at 05:09:14PM -0700, Ian Rogers wrote:
> > This patch series builds up to the addition of a new ilist app written
> > in python using textual [1] for the UI. The app presents perf PMUs and
> > events, displays the event information as in `perf list` while at the
> > bottom of the console showing recent activity of the event in total
> > and across all CPUs.
> >
> > The first part of the patches are a few perf and perf python C API
> > fixes, most importantly the counter reading in python supports tool
> > PMUs.
>
> The first part looks ok.  It'd be nice if you can send it separately.

Thanks Namhyung!

I'm working on a longer series to add metrics to the display. Breaking
up this series I have:

Fixes/cleanup:
  perf hwmon_pmu: Avoid shortening hwmon PMU name
  perf parse-events: Minor tidy up of event_type helper
  perf python: In str(evsel) use the evsel__pmu_name helper
  perf python: Fix thread check in pyrf_evsel__read
  perf python: Correct pyrf_evsel__read for tool PMUs

New python PMU/PMUs abstractions:
  perf python: Add basic PMU abstraction and pmus sequence
  perf python: Add function returning dictionary of all events on a PMU

jevents changes and a proper "software" PMU:
  perf jevents: If the long_desc and desc are identical then drop the
    long_desc
  perf jevents: Add common software event json
  perf pmu: Tolerate failure to read the type for wellknown PMUs
  perf parse-events: Remove non-json software events

tracepoint PMU:
  perf tp_pmu: Factor existing tracepoint logic to new file
  perf tp_pmu: Add event APIs
  perf list: Remove tracepoint printing code

ilist app:
  perf ilist: Add new python ilist command

I'll resend the fixes/cleanup to start with.

> >
> > The second part of the patches adds event json for the software PMU
> > and makes the tracepoint PMU support iteration of events and the
> > like. Without these improvements the tracepoint and software PMUs will
> > appear to have no events in the ilist app. As the software PMU moves
> > parsing to json, the legacy hard coded parsing is removed. This has
> > proven controversial for hardware events and so that cleanup isn't
> > done here.
>
> I think it's ok to move hardware events to JSON as well.  Then we can
> probably remove the hard coded parsing code.

A complication here is that in the json we have a "Unit" entry that
maps to a named PMU. For "software" the
`/sys/bus/event_source/devices/software` exists, similarly for
tracepoint, but for legacy hardware and hardware cache we may need to
keep the legacy type and not use the PMU type, the naming of the PMU
isn't consistent, etc.

> The tricky part is the hybrid machines.  Would it be possible to define
> common hardware with standard (legacy) encoding and to mark it "weak" or
> overridable by arch-specific events with the same name?

My thought is that eventually things could work like unionfs. The code
looks for a PMU/events in one place, then if that misses look in the
next, etc. This way we could allow users to have a directory where
they define things like events and metrics, expanding perf's
capabilities. We could also compress the event/metric data which may
be able to half the perf binary size (beyond the pretty simple
reuse/offset optimizations jevents.py tries today). With the unionfs
approach there would be a default prioritization of where to look
probably matching:
https://lore.kernel.org/lkml/20250416045117.876775-1-irogers@google.com/
(maybe we need to treat cycles as a special case given ARM). We can
then have a config setting to make the order different. There are two
issues though, the event encoding/description/.. legacy vs
sysfs/json/user encoding and the behavior of
wildcarding/case-sensitivity/.. the non-legacy stuff inherently
supports wildcarding and case insensitivity, but this may surprise
people.

> > The final patch adds the ilist command. To run it you need the updated
> > perf.cpython.so in your PYTHONPATH and then execute the
> > script. Expanding PMUs and then selecting events will cause event
> > informatin to be displayed in the top-right and the counters values to
> > be displayed as sparklines and counts in the bottom half of the
> > screen.
> >
> > Some thoughts on the series:
> >
> >  - The PMU changes will conflict with the addition of the DRM PMU:
> >    https://lore.kernel.org/lkml/20250403202439.57791-1-irogers@google.com/
> >    when these two are merged together ilist will show yet more
> >    counters. It'd be nice if the DRM stuff could land and then I can
> >    rebase these patches.
>
> Yep, it's merged now.  Please rebase the series.

Ack. Thanks!

> >
> >  - The parse-events clean up of the software and tracepoint PMU. The
> >    software PMU hard coding to be legacy first has similar issues and
> >    will conflict with the clean up in:
> >    https://lore.kernel.org/lkml/20250416045117.876775-1-irogers@google.com/
> >    Moving the software work to json means we don't need special parse
> >    events terms for software events, etc. We can just treat things
> >    like regular PMUs with json, etc. I'd much rather we had less
> >    special case logic so that series is best rebased on top of this
> >    work and it should drop the changes for software terms, etc. which
> >    this series removes. Maybe one day the whole event parsing can be
> >    much more regular in how PMUs are treated but there's always
> >    "cycles".
>
> As I said, moving the hardware events would be a solution as long as it
> can handle complexities in hybrid.
>
> >
> >  - Should python libraries have feature tests? How does this get
> >    packaged outside the kernel tree? I think these are open
> >    questions. Clearly textual is kind of a big dependency and we've
> >    largely been moving in the direction of fewer dependencies
> >    recently. Hopefully the app makes it clear why I think this one is
> >    worth carrying. We carry libslang as a dependency and I think
> >    textual clearly far surpasses it.
>
> Hmm.. I think it can be checked at runtime.  And maybe it can ask users
> to install the required packages.

Yeah, rather than:
```
import textual
```
we can do something like:
```
import importlib
try:
  textual = importlib.import_module('textual')
except ModuleNotFoundError:
  print("The textual library isn't found. Please install with pip, uv
or your distributions package manager")
  exit(1)
```
I'll add it to the patch with the app in it.

> >  - How to launch? Currently I run tools/perf/python/ilist.py but it
> >    would be much nicer if we could do `perf ilist` much as we do for
> >    perf-archive.sh. There are probably other scripts that should be
> >    perf commands like flamegraph and gecko. It'd be nice to follow up
> >    the series with something to make using these commands easy.
>
> Sounds good.

Thanks,
Ian

> Thanks,
> Namhyung
>
> >
> >  - Additional thoughts were captured on the mailing list:
> >    https://lore.kernel.org/lkml/CAP-5=fWC+doaVd5rEMWJXSQi_db_Wu2tyAe5Lm6jnQjcwXkF+w@mail.gmail.com/
> >
> > [1] https://textual.textualize.io/
> >
> > v4: No conflict rebase. Picks up perf-tools-next DRM PMU which
> >     displays as expected.
> >
> > v3: Add a search dialog to the ilist app with 'n'ext and 'p'revious
> >     keys. No changes in the ground work first 14 patches.
> >
> > v2: In the jevents event description duplication, some minor changes
> >     accidentally missed from v1 meaning that in v1 the descriptions
> >     were still duplicated. Expand the cover letter with some thoughts
> >     on the series.
> >
> > Ian Rogers (15):
> >   perf hwmon_pmu: Avoid shortening hwmon PMU name
> >   perf parse-events: Minor tidy up of event_type helper
> >   perf python: In str(evsel) use the evsel__pmu_name helper
> >   perf python: Fix thread check in pyrf_evsel__read
> >   perf python: Correct pyrf_evsel__read for tool PMUs
> >   perf python: Add basic PMU abstraction and pmus sequence
> >   perf python: Add function returning dictionary of all events on a PMU
> >   perf jevents: If the long_desc and desc are identical then drop the
> >     long_desc
> >   perf jevents: Add common software event json
> >   perf pmu: Tolerate failure to read the type for wellknown PMUs
> >   perf parse-events: Remove non-json software events
> >   perf tp_pmu: Factor existing tracepoint logic to new file
> >   perf tp_pmu: Add event APIs
> >   perf list: Remove tracepoint printing code
> >   perf ilist: Add new python ilist command
> >
> >  tools/perf/builtin-list.c                     |  47 ++-
> >  .../arch/common/common/software.json          |  92 +++++
> >  tools/perf/pmu-events/empty-pmu-events.c      | 266 ++++++++-----
> >  tools/perf/pmu-events/jevents.py              |  18 +-
> >  tools/perf/python/ilist.py                    | 376 ++++++++++++++++++
> >  tools/perf/util/Build                         |   1 +
> >  tools/perf/util/evsel.c                       |  21 +-
> >  tools/perf/util/hwmon_pmu.c                   |   2 +-
> >  tools/perf/util/parse-events.c                | 225 ++++-------
> >  tools/perf/util/parse-events.h                |   3 +-
> >  tools/perf/util/parse-events.l                |  38 +-
> >  tools/perf/util/parse-events.y                |  29 +-
> >  tools/perf/util/pmu.c                         |  44 +-
> >  tools/perf/util/print-events.c                |  95 -----
> >  tools/perf/util/print-events.h                |   1 -
> >  tools/perf/util/python.c                      | 248 +++++++++++-
> >  tools/perf/util/tp_pmu.c                      | 209 ++++++++++
> >  tools/perf/util/tp_pmu.h                      |  19 +
> >  18 files changed, 1282 insertions(+), 452 deletions(-)
> >  create mode 100644 tools/perf/pmu-events/arch/common/common/software.json
> >  create mode 100755 tools/perf/python/ilist.py
> >  create mode 100644 tools/perf/util/tp_pmu.c
> >  create mode 100644 tools/perf/util/tp_pmu.h
> >
> > --
> > 2.50.0.727.gbf7dc18ff4-goog
> >