In this RFC series, we are introducing a C extension module to allow
python programs to call the libperf API functions. Currently libperf can
be used by C programs, but expanding the support to python is beneficial
for python users.

The structure of the patch series is as follows:
1. Patch 1 : Create wrappers for the perf structs which are used by
examples/counting.c

2. Patch 2: Create the C extension module that maps and exposes the
libperf functions to python programs

2. Patch 3: A python variant of counting.c - counting.py to demonstrate
the usage of libperf from python

We have not added support for entire libperf, as we want to get
community feedback on the approach taken in this series.

Gautam Menghani (3):
  libperf: Introduce wrappers for perf structs to be exposed to python
  libperf: Introduce a C extension module for python
  libperf: Add counting.py example to demonstrate libperf usage from
    python

 tools/lib/perf/Build                          |   1 +
 .../perf/Documentation/examples/counting.py   |  74 +++
 tools/lib/perf/Makefile                       |  12 +-
 tools/lib/perf/include/perf/py_perf.h         | 431 ++++++++++++++++++
 tools/lib/perf/libperf.map                    |   1 +
 tools/lib/perf/py_perf.c                      | 262 +++++++++++
 6 files changed, 779 insertions(+), 2 deletions(-)
 create mode 100755 tools/lib/perf/Documentation/examples/counting.py
 create mode 100644 tools/lib/perf/include/perf/py_perf.h
 create mode 100644 tools/lib/perf/py_perf.c

-- 
2.47.0

Hello Gautam

On Thu, Mar 13, 2025 at 01:21:21PM +0530, Gautam Menghani wrote:
> In this RFC series, we are introducing a C extension module to allow
> python programs to call the libperf API functions. Currently libperf can
> be used by C programs, but expanding the support to python is beneficial
> for python users.
> 
> The structure of the patch series is as follows:
> 1. Patch 1 : Create wrappers for the perf structs which are used by
> examples/counting.c
> 
> 2. Patch 2: Create the C extension module that maps and exposes the
> libperf functions to python programs

May I ask why you are not using SWIG? With libcpupower the kernel has
already been using SWIG to generate Python bindings for a C user-space API.

This has several advantages including a much smaller footprint (you only
need to copy the header definitions into a .swg file), can generate for
several languages (Perl, Ruby, Java and C#), and SWIG is an active,
tested, and mature piece of software code that has been around for
almost as long as the Linux kernel.

Python bindings including the makefile as an example:
https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python

How to use the bindings in a script:
https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python/test_raw_pylibcpupower.py

Original discussion:
https://lore.kernel.org/linux-pm/20240724221122.54601-1-jwyatt@redhat.com/

SWIG has been pretty useful as it helped me find two issues in the
libcpupower API that have been around for over 10 years:
https://lore.kernel.org/linux-pm/20240905021916.15938-1-jwyatt@redhat.com/T/#mf04b4ba93f79fe68c20c1d88d8ed966164a1c7d7
https://lore.kernel.org/linux-pm/20250305210901.24177-1-jwyatt@redhat.com/

> 
> 2. Patch 3: A python variant of counting.c - counting.py to demonstrate
> the usage of libperf from python
> 
> We have not added support for entire libperf, as we want to get
> community feedback on the approach taken in this series.
> 
> Gautam Menghani (3):
>   libperf: Introduce wrappers for perf structs to be exposed to python
>   libperf: Introduce a C extension module for python
>   libperf: Add counting.py example to demonstrate libperf usage from
>     python
> 
>  tools/lib/perf/Build                          |   1 +
>  .../perf/Documentation/examples/counting.py   |  74 +++
>  tools/lib/perf/Makefile                       |  12 +-
>  tools/lib/perf/include/perf/py_perf.h         | 431 ++++++++++++++++++
>  tools/lib/perf/libperf.map                    |   1 +
>  tools/lib/perf/py_perf.c                      | 262 +++++++++++
>  6 files changed, 779 insertions(+), 2 deletions(-)
>  create mode 100755 tools/lib/perf/Documentation/examples/counting.py
>  create mode 100644 tools/lib/perf/include/perf/py_perf.h
>  create mode 100644 tools/lib/perf/py_perf.c
> 
> -- 
> 2.47.0
> 

-- 
Sincerely,
John Wyatt
Software Engineer, Core Kernel
Red Hat

On Thu, Mar 13, 2025 at 7:12 AM John B. Wyatt IV <jwyatt@redhat.com> wrote:
>
> Hello Gautam
>
> On Thu, Mar 13, 2025 at 01:21:21PM +0530, Gautam Menghani wrote:
> > In this RFC series, we are introducing a C extension module to allow
> > python programs to call the libperf API functions. Currently libperf can
> > be used by C programs, but expanding the support to python is beneficial
> > for python users.
> >
> > The structure of the patch series is as follows:
> > 1. Patch 1 : Create wrappers for the perf structs which are used by
> > examples/counting.c
> >
> > 2. Patch 2: Create the C extension module that maps and exposes the
> > libperf functions to python programs
>
> May I ask why you are not using SWIG? With libcpupower the kernel has
> already been using SWIG to generate Python bindings for a C user-space API.
>
> This has several advantages including a much smaller footprint (you only
> need to copy the header definitions into a .swg file), can generate for
> several languages (Perl, Ruby, Java and C#), and SWIG is an active,
> tested, and mature piece of software code that has been around for
> almost as long as the Linux kernel.
>
> Python bindings including the makefile as an example:
> https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python
>
> How to use the bindings in a script:
> https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python/test_raw_pylibcpupower.py
>
> Original discussion:
> https://lore.kernel.org/linux-pm/20240724221122.54601-1-jwyatt@redhat.com/
>
> SWIG has been pretty useful as it helped me find two issues in the
> libcpupower API that have been around for over 10 years:
> https://lore.kernel.org/linux-pm/20240905021916.15938-1-jwyatt@redhat.com/T/#mf04b4ba93f79fe68c20c1d88d8ed966164a1c7d7
> https://lore.kernel.org/linux-pm/20250305210901.24177-1-jwyatt@redhat.com/

So I think we should probably get rid of libperf and re-integrate it
back into the perf code. There are issues in the code, like removing a
list element:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/perf/evlist.c?h=tmp.perf-tools-next#n58
just leaks the removed element. Deleting the element means that the
container evsel, rather than the libperf perf_evsel, leaks things like
the name variable. We can add yet more call backs and complexity but
I'm not sure what we're winning. Perhaps we can move things the other
way, perf code into libperf, like machine, session, pmus, .. I'd
prefer if we were to do that we refactored the code and adopted the
same license as libbpf as both libraries have similar packaging
issues. The viral GPLv2 on libperf is something of an issue. Perhaps
we can also migrate this code to Rust.

SWIG is fine, there is also CLIF, I'm way of dependencies as even a
python dependency in the perf build is optional.

We already have perf python bindings, and I've been working to extend
those for example in:
https://lore.kernel.org/lkml/20250228222308.626803-1-irogers@google.com/
and I've been working to expand those for things like hybrid CPUs. It
seems a shame to reinvent all of that logic again on top of libperf.

Thanks,
Ian

> >
> > 2. Patch 3: A python variant of counting.c - counting.py to demonstrate
> > the usage of libperf from python
> >
> > We have not added support for entire libperf, as we want to get
> > community feedback on the approach taken in this series.
> >
> > Gautam Menghani (3):
> >   libperf: Introduce wrappers for perf structs to be exposed to python
> >   libperf: Introduce a C extension module for python
> >   libperf: Add counting.py example to demonstrate libperf usage from
> >     python
> >
> >  tools/lib/perf/Build                          |   1 +
> >  .../perf/Documentation/examples/counting.py   |  74 +++
> >  tools/lib/perf/Makefile                       |  12 +-
> >  tools/lib/perf/include/perf/py_perf.h         | 431 ++++++++++++++++++
> >  tools/lib/perf/libperf.map                    |   1 +
> >  tools/lib/perf/py_perf.c                      | 262 +++++++++++
> >  6 files changed, 779 insertions(+), 2 deletions(-)
> >  create mode 100755 tools/lib/perf/Documentation/examples/counting.py
> >  create mode 100644 tools/lib/perf/include/perf/py_perf.h
> >  create mode 100644 tools/lib/perf/py_perf.c
> >
> > --
> > 2.47.0
> >
>
> --
> Sincerely,
> John Wyatt
> Software Engineer, Core Kernel
> Red Hat
>

On Thu, Mar 13, 2025 at 09:01:10AM -0700, Ian Rogers wrote:
> On Thu, Mar 13, 2025 at 7:12 AM John B. Wyatt IV <jwyatt@redhat.com> wrote:
> >
> > Hello Gautam
> >
> > On Thu, Mar 13, 2025 at 01:21:21PM +0530, Gautam Menghani wrote:
> > > In this RFC series, we are introducing a C extension module to allow
> > > python programs to call the libperf API functions. Currently libperf can
> > > be used by C programs, but expanding the support to python is beneficial
> > > for python users.
> > >
> > > The structure of the patch series is as follows:
> > > 1. Patch 1 : Create wrappers for the perf structs which are used by
> > > examples/counting.c
> > >
> > > 2. Patch 2: Create the C extension module that maps and exposes the
> > > libperf functions to python programs
> >
> > May I ask why you are not using SWIG? With libcpupower the kernel has
> > already been using SWIG to generate Python bindings for a C user-space API.
> >
> > This has several advantages including a much smaller footprint (you only
> > need to copy the header definitions into a .swg file), can generate for
> > several languages (Perl, Ruby, Java and C#), and SWIG is an active,
> > tested, and mature piece of software code that has been around for
> > almost as long as the Linux kernel.
> >
> > Python bindings including the makefile as an example:
> > https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python
> >
> > How to use the bindings in a script:
> > https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python/test_raw_pylibcpupower.py
> >
> > Original discussion:
> > https://lore.kernel.org/linux-pm/20240724221122.54601-1-jwyatt@redhat.com/
> >
> > SWIG has been pretty useful as it helped me find two issues in the
> > libcpupower API that have been around for over 10 years:
> > https://lore.kernel.org/linux-pm/20240905021916.15938-1-jwyatt@redhat.com/T/#mf04b4ba93f79fe68c20c1d88d8ed966164a1c7d7
> > https://lore.kernel.org/linux-pm/20250305210901.24177-1-jwyatt@redhat.com/
> 
> So I think we should probably get rid of libperf and re-integrate it
> back into the perf code. There are issues in the code, like removing a
> list element:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/perf/evlist.c?h=tmp.perf-tools-next#n58
> just leaks the removed element. Deleting the element means that the
> container evsel, rather than the libperf perf_evsel, leaks things like
> the name variable. We can add yet more call backs and complexity but
> I'm not sure what we're winning. Perhaps we can move things the other
> way, perf code into libperf, like machine, session, pmus, .. I'd
> prefer if we were to do that we refactored the code and adopted the
> same license as libbpf as both libraries have similar packaging
> issues. The viral GPLv2 on libperf is something of an issue. Perhaps
> we can also migrate this code to Rust.
> 
> SWIG is fine, there is also CLIF, I'm way of dependencies as even a
> python dependency in the perf build is optional.
> 
> We already have perf python bindings, and I've been working to extend
> those for example in:
> https://lore.kernel.org/lkml/20250228222308.626803-1-irogers@google.com/
> and I've been working to expand those for things like hybrid CPUs. It
> seems a shame to reinvent all of that logic again on top of libperf.

Thanks for the feedback and pointers. We'll look more into this
approach.

Thanks,
Gautam