perf lock contention: Add BPF support (v1)

[PATCH 0/3] perf lock contention: Add BPF support (v1)

Posted by Namhyung Kim 3 years, 8 months ago

Hello,

This patchset adds -b/--use-bpf option and others to use BPF to
collect kernel lock contention stats.  With this option it doesn't
require a separate `perf lock record` step.  Basic filtering on cpu
(with -a or -C option) and on task (with -p and --tid option) is
supported as usual.

  $ sudo perf lock con -a -b sleep 1
     contended   total wait     max wait     avg wait         type   caller

            42    192.67 us     13.64 us      4.59 us     spinlock   queue_work_on+0x20
            23     85.54 us     10.28 us      3.72 us     spinlock   worker_thread+0x14a
             6     13.92 us      6.51 us      2.32 us        mutex   kernfs_iop_permission+0x30
             3     11.59 us     10.04 us      3.86 us        mutex   kernfs_dop_revalidate+0x3c
             1      7.52 us      7.52 us      7.52 us     spinlock   kthread+0x115
             1      7.24 us      7.24 us      7.24 us     rwlock:W   sys_epoll_wait+0x148
             2      7.08 us      3.99 us      3.54 us     spinlock   delayed_work_timer_fn+0x1b
             1      6.41 us      6.41 us      6.41 us     spinlock   idle_balance+0xa06
             2      2.50 us      1.83 us      1.25 us        mutex   kernfs_iop_lookup+0x2f
             1      1.71 us      1.71 us      1.71 us        mutex   kernfs_iop_getattr+0x2c

It seems my system had some contentions on the workqueue spinlock and
the kernfs mutex.

The code is available at perf/lock-bpf-v1 branch in

  git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung


Namhyung Kim (3):
  perf lock: Pass machine pointer to is_lock_function()
  perf lock: Use BPF for lock contention analysis
  perf lock: Implement cpu and task filters for BPF

 tools/perf/Documentation/perf-lock.txt        |  22 ++
 tools/perf/Makefile.perf                      |   2 +-
 tools/perf/builtin-lock.c                     | 226 ++++++++----------
 tools/perf/util/Build                         |   1 +
 tools/perf/util/bpf_lock_contention.c         | 181 ++++++++++++++
 .../perf/util/bpf_skel/lock_contention.bpf.c  | 170 +++++++++++++
 tools/perf/util/lock-contention.h             | 140 +++++++++++
 7 files changed, 614 insertions(+), 128 deletions(-)
 create mode 100644 tools/perf/util/bpf_lock_contention.c
 create mode 100644 tools/perf/util/bpf_skel/lock_contention.bpf.c
 create mode 100644 tools/perf/util/lock-contention.h

-- 
2.37.1.455.g008518b4e5-goog

Re: [PATCH 0/3] perf lock contention: Add BPF support (v1)

Posted by Arnaldo Carvalho de Melo 3 years, 8 months ago

Em Fri, Jul 29, 2022 at 01:07:53PM -0700, Namhyung Kim escreveu:
> Hello,
> 
> This patchset adds -b/--use-bpf option and others to use BPF to
> collect kernel lock contention stats.  With this option it doesn't
> require a separate `perf lock record` step.  Basic filtering on cpu
> (with -a or -C option) and on task (with -p and --tid option) is
> supported as usual.
> 
>   $ sudo perf lock con -a -b sleep 1
>      contended   total wait     max wait     avg wait         type   caller
> 
>             42    192.67 us     13.64 us      4.59 us     spinlock   queue_work_on+0x20
>             23     85.54 us     10.28 us      3.72 us     spinlock   worker_thread+0x14a
>              6     13.92 us      6.51 us      2.32 us        mutex   kernfs_iop_permission+0x30
>              3     11.59 us     10.04 us      3.86 us        mutex   kernfs_dop_revalidate+0x3c
>              1      7.52 us      7.52 us      7.52 us     spinlock   kthread+0x115
>              1      7.24 us      7.24 us      7.24 us     rwlock:W   sys_epoll_wait+0x148
>              2      7.08 us      3.99 us      3.54 us     spinlock   delayed_work_timer_fn+0x1b
>              1      6.41 us      6.41 us      6.41 us     spinlock   idle_balance+0xa06
>              2      2.50 us      1.83 us      1.25 us        mutex   kernfs_iop_lookup+0x2f
>              1      1.71 us      1.71 us      1.71 us        mutex   kernfs_iop_getattr+0x2c
> 
> It seems my system had some contentions on the workqueue spinlock and
> the kernfs mutex.
> 
> The code is available at perf/lock-bpf-v1 branch in
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks, applied.

- Arnaldo

 
> Thanks,
> Namhyung
> 
> 
> Namhyung Kim (3):
>   perf lock: Pass machine pointer to is_lock_function()
>   perf lock: Use BPF for lock contention analysis
>   perf lock: Implement cpu and task filters for BPF
> 
>  tools/perf/Documentation/perf-lock.txt        |  22 ++
>  tools/perf/Makefile.perf                      |   2 +-
>  tools/perf/builtin-lock.c                     | 226 ++++++++----------
>  tools/perf/util/Build                         |   1 +
>  tools/perf/util/bpf_lock_contention.c         | 181 ++++++++++++++
>  .../perf/util/bpf_skel/lock_contention.bpf.c  | 170 +++++++++++++
>  tools/perf/util/lock-contention.h             | 140 +++++++++++
>  7 files changed, 614 insertions(+), 128 deletions(-)
>  create mode 100644 tools/perf/util/bpf_lock_contention.c
>  create mode 100644 tools/perf/util/bpf_skel/lock_contention.bpf.c
>  create mode 100644 tools/perf/util/lock-contention.h
> 
> -- 
> 2.37.1.455.g008518b4e5-goog

-- 

- Arnaldo