Extend the rv userspace tool to load BPF monitors, those can be found in
specific locations (e.g. /usr/share/rv/bpf_monitors/) and are plain
object files including BTF data.

This type of BPF monitors can be generated from rvgen using the -b flag
just like in-kernel monitors and, after manual adaptation, can be built
and run transparently by the rv userspace tool.

Only DA monitors are supported (as they are the only class the rv tool
currenly supports) and the tool aims to work in the same way for both
in-kernel and BPF monitors.

The da_monitor header is adapted to be included directly in BPF programs
and share as much logic as possible.

BPF monitors are implemented using the struct_ops framework to integrate
as much as possible with in-kernel monitor. After registering, they
look like standard monitors in the tracefs, although to properly enable
them we need to load handlers from userspace and this is done by the rv
tool. BPF monitors cannot generate tracepoints so the rv -t for those
monitors is reading events from a ringbuffer.

Selftests (make -C tools/verification/rv check) are present to validate
and demonstrate the usage, which boils down to:

  rv bpf register          # load struct_ops and maps for all monitors
  rv list                  # unmodified, list BPF monitors transparently
  rv mon -t <mon>          # run the monitor with tracing (ringbuffer)
  rv mon -r printk <mon>   # run the monitor with a reactor
  rv bpf unregister        # remove all BPF monitors

Patch 1 adds tracepoints required in a later monitor example.
Patch 2 improves the rv tool selftest reporting.
Patches 3-9 prepare and add support for BPF monitors in the kernel.
Patch 10 adds a feature check for tools/build
Patch 11-14 prepare and add support for BPF monitors in the rv tool.
Patch 15-17 prepare and include BPF monitors examples
Patch 18 adds support for generating BPF monitors in rvgen
Patch 19-20 adds BPF selftests (make check) for the rv and rvgen tools

To: linux-trace-kernel@vger.kernel.org
To: bpf@vger.kernel.org
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Nam Cao <namcao@linutronix.de>
Cc: Wen Yang <wen.yang@linux.dev>
Cc: Tobias Schaffner <tobias.schaffner@siemens.com>
Cc: Viktor Malik <vmalik@redhat.com>

Gabriele Monaco (19):
  tools/rv: Skip empty pid error in selftest if command failed
  rv: Refactor da_trace() functions to get strings internally
  rv: Use static arrays for rv_monitor name and description
  rv: Add in-kernel support for BPF monitors
  rv: Add rv_get_monitor_by_name()
  rv: Add reactors support to BPF monitors
  rv: Cast result of model_get_*_name()
  rv: Handle unregistered monitors safely in tracefs
  tools/build: Add a feature test for bpftool-btf
  tools/rv: Move argument parsing from in_kernel to utils
  tools/rv: Export functionality for in_kernel monitors
  tools/rv: Implement BPF monitor loading and tracing
  tools/rv: Implement BPF monitor registration logic
  tools/rv: Copy stripped bpf_atomic.h from libarena
  tools/rv: Add BPF monitors
  tools/rv: Define CONFIG_X86_64 statically for BPF monitors
  verification/rvgen: Add support for BPF monitors
  tools/rv: Add selftest for rv bpf
  verification/rvgen: Add selftest for rvgen -b

Nam Cao (1):
  sched: Add task enqueue/dequeue trace points

 include/linux/rv.h                            |    9 +-
 include/rv/automata.h                         |    4 +-
 include/rv/da_monitor.h                       |   67 +-
 include/trace/events/sched.h                  |    8 +
 kernel/sched/core.c                           |   12 +-
 kernel/sched/sched.h                          |    2 +
 kernel/trace/rv/Kconfig                       |   10 +
 kernel/trace/rv/Makefile                      |    1 +
 kernel/trace/rv/rv.c                          |   68 +-
 kernel/trace/rv/rv.h                          |    5 +-
 kernel/trace/rv/rv_bpf.c                      |  174 +++
 kernel/trace/rv/rv_reactors.c                 |    6 +
 tools/build/Makefile.feature                  |    1 +
 tools/build/feature/Makefile                  |    7 +-
 tools/verification/models/nohz.dot            |   16 +
 tools/verification/models/tqueue.dot          |   15 +
 tools/verification/rv/Makefile                |   51 +-
 tools/verification/rv/Makefile.config         |   49 +
 tools/verification/rv/Makefile.rv             |    5 +
 tools/verification/rv/bpf_monitors/.gitignore |    2 +
 .../verification/rv/bpf_monitors/bpf_atomic.h |  105 ++
 .../rv/bpf_monitors/da_monitor_bpf.h          |  400 +++++++
 tools/verification/rv/bpf_monitors/nohz.c     |   47 +
 tools/verification/rv/bpf_monitors/nohz.h     |   49 +
 tools/verification/rv/bpf_monitors/tqueue.c   |   35 +
 tools/verification/rv/bpf_monitors/tqueue.h   |   47 +
 tools/verification/rv/include/bpf_monitor.h   |   21 +
 tools/verification/rv/include/in_kernel.h     |    5 +
 tools/verification/rv/include/utils.h         |   17 +-
 tools/verification/rv/src/Build               |    5 +
 tools/verification/rv/src/bpf_monitor.c       | 1025 +++++++++++++++++
 tools/verification/rv/src/in_kernel.c         |  205 ++--
 tools/verification/rv/src/rv.c                |   10 +-
 tools/verification/rv/src/utils.c             |   95 +-
 tools/verification/rv/tests/rv_bpf.t          |  104 ++
 tools/verification/rvgen/__main__.py          |   15 +-
 tools/verification/rvgen/rvgen/dot2c.py       |   15 +-
 tools/verification/rvgen/rvgen/dot2k.py       |   22 +-
 tools/verification/rvgen/rvgen/generator.py   |   26 +-
 .../rvgen/rvgen/templates/dot2k/main_bpf.c    |   26 +
 .../tests/golden/da_bpf_cpu/da_bpf_cpu.c      |   40 +
 .../tests/golden/da_bpf_cpu/da_bpf_cpu.h      |   47 +
 .../tests/golden/da_bpf_obj/da_bpf_obj.c      |   54 +
 .../tests/golden/da_bpf_obj/da_bpf_obj.h      |   47 +
 .../verification/rvgen/tests/rvgen_monitor.t  |   11 +
 tools/verification/tests/engine.sh            |    5 +-
 46 files changed, 2770 insertions(+), 220 deletions(-)
 create mode 100644 kernel/trace/rv/rv_bpf.c
 create mode 100644 tools/verification/models/nohz.dot
 create mode 100644 tools/verification/models/tqueue.dot
 create mode 100644 tools/verification/rv/bpf_monitors/.gitignore
 create mode 100644 tools/verification/rv/bpf_monitors/bpf_atomic.h
 create mode 100644 tools/verification/rv/bpf_monitors/da_monitor_bpf.h
 create mode 100644 tools/verification/rv/bpf_monitors/nohz.c
 create mode 100644 tools/verification/rv/bpf_monitors/nohz.h
 create mode 100644 tools/verification/rv/bpf_monitors/tqueue.c
 create mode 100644 tools/verification/rv/bpf_monitors/tqueue.h
 create mode 100644 tools/verification/rv/include/bpf_monitor.h
 create mode 100644 tools/verification/rv/src/bpf_monitor.c
 create mode 100644 tools/verification/rv/tests/rv_bpf.t
 create mode 100644 tools/verification/rvgen/rvgen/templates/dot2k/main_bpf.c
 create mode 100644 tools/verification/rvgen/tests/golden/da_bpf_cpu/da_bpf_cpu.c
 create mode 100644 tools/verification/rvgen/tests/golden/da_bpf_cpu/da_bpf_cpu.h
 create mode 100644 tools/verification/rvgen/tests/golden/da_bpf_obj/da_bpf_obj.c
 create mode 100644 tools/verification/rvgen/tests/golden/da_bpf_obj/da_bpf_obj.h


base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.55.0

Gabriele Monaco <gmonaco@redhat.com> writes:
> Extend the rv userspace tool to load BPF monitors, those can be found in
> specific locations (e.g. /usr/share/rv/bpf_monitors/) and are plain
> object files including BTF data.
>
> This type of BPF monitors can be generated from rvgen using the -b flag
> just like in-kernel monitors and, after manual adaptation, can be built
> and run transparently by the rv userspace tool.

I am not familiar with BPF. What is the benefit of BPF monitors,
compared to the existing DA monitors?

Nam

On Tue, 2026-09-01 at 20:35 +0200, Nam Cao wrote:
> Gabriele Monaco <gmonaco@redhat.com> writes:
> > Extend the rv userspace tool to load BPF monitors, those can be found in
> > specific locations (e.g. /usr/share/rv/bpf_monitors/) and are plain
> > object files including BTF data.
> > 
> > This type of BPF monitors can be generated from rvgen using the -b flag
> > just like in-kernel monitors and, after manual adaptation, can be built
> > and run transparently by the rv userspace tool.
> 
> I am not familiar with BPF. What is the benefit of BPF monitors,
> compared to the existing DA monitors?

I should definitely have included it in the cover letter.. I'm writing it
everywhere (will present at LPC) but forgot it here.

Essentially BPF monitors can be pluggable, folks writing their own monitors
won't need to submit a patch or maintain a separate tree, which is useful for
domain-specific models.
By being pluggable you also don't need to reboot to use a new/updated monitor.

Think of being able to distribute a more granular set of rules for RTapp, I
remember we had conversation along those lines, not all rules apply to all
contexts and what you send upstream has to be general, what you keep for
yourself doesn't.

Having monitors in BPF brings also other perks over kernel modules: a whole
bunch of readily available probe types (uprobes, fprobes, all unexported
tracepoints that are cumbersome for modules), the map infrastructure for
allocation is arguably easier and the code is verified when loaded against
common issues (NULL pointer access, unbound loops, etc.).

That said, I try to mimic as much as possible the in-kernel functionality, but
some things are not the same (event/error tracepoints).

These support DA only because BPF loading needs a userspace component and the RV
tool doesn't support LTL and HA yet, there shouldn't be any technical reason not
to extend to those in the future.

Gabriele

On Tue Sep 1, 2026 at 11:52 PM PDT, Gabriele Monaco wrote:
> On Tue, 2026-09-01 at 20:35 +0200, Nam Cao wrote:
>> Gabriele Monaco <gmonaco@redhat.com> writes:
>> > Extend the rv userspace tool to load BPF monitors, those can be found in
>> > specific locations (e.g. /usr/share/rv/bpf_monitors/) and are plain
>> > object files including BTF data.
>> > 
>> > This type of BPF monitors can be generated from rvgen using the -b flag
>> > just like in-kernel monitors and, after manual adaptation, can be built
>> > and run transparently by the rv userspace tool.
>> 
>> I am not familiar with BPF. What is the benefit of BPF monitors,
>> compared to the existing DA monitors?
>
> I should definitely have included it in the cover letter.. I'm writing it
> everywhere (will present at LPC) but forgot it here.
>
> Essentially BPF monitors can be pluggable, folks writing their own monitors
> won't need to submit a patch or maintain a separate tree, which is useful for
> domain-specific models.
> By being pluggable you also don't need to reboot to use a new/updated monitor.
>
> Think of being able to distribute a more granular set of rules for RTapp, I
> remember we had conversation along those lines, not all rules apply to all
> contexts and what you send upstream has to be general, what you keep for
> yourself doesn't.
>
> Having monitors in BPF brings also other perks over kernel modules: a whole
> bunch of readily available probe types (uprobes, fprobes, all unexported
> tracepoints that are cumbersome for modules), the map infrastructure for
> allocation is arguably easier and the code is verified when loaded against
> common issues (NULL pointer access, unbound loops, etc.).
>
> That said, I try to mimic as much as possible the in-kernel functionality, but
> some things are not the same (event/error tracepoints).
>
> These support DA only because BPF loading needs a userspace component and the RV
> tool doesn't support LTL and HA yet, there shouldn't be any technical reason not
> to extend to those in the future.

I don't think bpf fits here. I haven't seen active use of RV and even less
so of any request from people who want this kind of programmability.

So Nack for now. Sorry.

On Thu, Sep 3, 2026 at 4:03 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> I don't think bpf fits here. I haven't seen active use of RV and even less
> so of any request from people who want this kind of programmability.
>
> So Nack for now. Sorry.
>

A major application of RV is monitoring the combined behavior of the
kernel and an external application, see [1] [2]. Adopting RV for these
use cases is difficult because the monitors are currently built into
the kernel. Blocking BPF adoption because it is not widely used risks
a deadlock: RV not getting enough users, because it cannot use BPF,
and BPF rejecting RV because it lacks sufficient use.

Of course, I might be entirely wrong: RV may not end up being
interesting enough to users even with the BPF hooks in place, or it
might grow enough with only in-kernel monitors for you to take it
seriously. However, we cannot predict that in advance.

(Note also that the idea to use BPF in RV dates back to 2022 [3].)

[1] https://lore.kernel.org/linux-trace-kernel/20260827072400.45734-1-tobias.schaffner@siemens.com/T/#e8a8b21136a60ad7c8f67b07c66f71de0ec0c1075
[2] https://docs.kernel.org/trace/rv/monitor_rtapp.html
[3] https://gitlab.com/linux-rv-tools/dot2bpf

Tomas

On Wed, 2026-09-02 at 18:57 -0700, Alexei Starovoitov wrote:
> I don't think bpf fits here. I haven't seen active use of RV and even less
> so of any request from people who want this kind of programmability.
> 
> So Nack for now. Sorry.

Thanks for your opinion, however it sounds a bit harsh to me.

RV is clearly not a popular tool, but I wouldn't say it doesn't have active use:
we're actively using it to validate the deadline scheduler/server changes and it
already found a number of issues (e.g. [1], [2]).
The RTapp monitors are used to validate if userspace real-time applications
interact with the kernel in the best way for predictability, as far as I'm aware
that's regularly used [3].

In my eyes, the missing piece of RV is in fact this programmability. Adding a
monitor now requires to write in-tree built-in kernel code.

There are strong use cases for domain-specific rules that aren't general enough
to belong in the kernel tree:
* stricter RTapp rules that may not apply to all use-cases
* verify some exotic system within Linux (e.g. a co-kernel [4])
* modelling for functional safety certification process (used for in-vehicle
OSes like RHIVOS)

BPF seems the natural tool for the job to me. Mind that RV monitors would be
simple users of BPF, requiring only modifications in the RV subsystem but using
standard BPF facilities.

I'm totally open to discuss on this though.

Thanks,
Gabriele

[1] - https://lore.kernel.org/lkml/20260522125833.264145-1-gmonaco@redhat.com
[2] - https://lore.kernel.org/lkml/20260113085159.114226-3-gmonaco@redhat.com
[3] -
https://lore.kernel.org/linux-rt-users/20260826132153.2476006-1-bigeasy@linutronix.de/T/#t
[4] -
https://lore.kernel.org/lkml/20260827072400.45734-1-tobias.schaffner@siemens.com

On Thu, 03 Sep 2026 09:21:18 +0200
Gabriele Monaco <gmonaco@redhat.com> wrote:

> On Wed, 2026-09-02 at 18:57 -0700, Alexei Starovoitov wrote:
> > I don't think bpf fits here. I haven't seen active use of RV and even less
> > so of any request from people who want this kind of programmability.
> > 
> > So Nack for now. Sorry.  
> 
> Thanks for your opinion, however it sounds a bit harsh to me.
> 
> RV is clearly not a popular tool, but I wouldn't say it doesn't have active use:
> we're actively using it to validate the deadline scheduler/server changes and it
> already found a number of issues (e.g. [1], [2]).
> The RTapp monitors are used to validate if userspace real-time applications
> interact with the kernel in the best way for predictability, as far as I'm aware
> that's regularly used [3].
> 
> In my eyes, the missing piece of RV is in fact this programmability. Adding a
> monitor now requires to write in-tree built-in kernel code.
> 
> There are strong use cases for domain-specific rules that aren't general enough
> to belong in the kernel tree:
> * stricter RTapp rules that may not apply to all use-cases
> * verify some exotic system within Linux (e.g. a co-kernel [4])
> * modelling for functional safety certification process (used for in-vehicle
> OSes like RHIVOS)
> 
> BPF seems the natural tool for the job to me. Mind that RV monitors would be
> simple users of BPF, requiring only modifications in the RV subsystem but using
> standard BPF facilities.
> 

As RV would simply be a user of BPF and not modifying BPF infrastructure,
I'm not so sure you have the jurisdiction to NAK it. It would be like me
NAKing how you create a trace event.

RV monitors are a very good way to make sure the system is preforming
properly. This looks exactly like a perfect use case for using BPF instead
of having to install modules.

Either BPF is a tool for the kernel or it isn't. You can't judge where BPF
gets used. You can only judge on modifications to BFP.

-- Steve


> I'm totally open to discuss on this though.
> 
> Thanks,
> Gabriele
> 
> [1] - https://lore.kernel.org/lkml/20260522125833.264145-1-gmonaco@redhat.com
> [2] - https://lore.kernel.org/lkml/20260113085159.114226-3-gmonaco@redhat.com
> [3] -
> https://lore.kernel.org/linux-rt-users/20260826132153.2476006-1-bigeasy@linutronix.de/T/#t
> [4] -
> https://lore.kernel.org/lkml/20260827072400.45734-1-tobias.schaffner@siemens.com

On Thu, Sep 3, 2026 at 6:01 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 03 Sep 2026 09:21:18 +0200
> Gabriele Monaco <gmonaco@redhat.com> wrote:
>
> > On Wed, 2026-09-02 at 18:57 -0700, Alexei Starovoitov wrote:
> > > I don't think bpf fits here. I haven't seen active use of RV and even less
> > > so of any request from people who want this kind of programmability.
> > >
> > > So Nack for now. Sorry.
> >
> > Thanks for your opinion, however it sounds a bit harsh to me.
> >
> > RV is clearly not a popular tool, but I wouldn't say it doesn't have active use:
> > we're actively using it to validate the deadline scheduler/server changes and it
> > already found a number of issues (e.g. [1], [2]).
> > The RTapp monitors are used to validate if userspace real-time applications
> > interact with the kernel in the best way for predictability, as far as I'm aware
> > that's regularly used [3].
> >
> > In my eyes, the missing piece of RV is in fact this programmability. Adding a
> > monitor now requires to write in-tree built-in kernel code.
> >
> > There are strong use cases for domain-specific rules that aren't general enough
> > to belong in the kernel tree:
> > * stricter RTapp rules that may not apply to all use-cases
> > * verify some exotic system within Linux (e.g. a co-kernel [4])
> > * modelling for functional safety certification process (used for in-vehicle
> > OSes like RHIVOS)
> >
> > BPF seems the natural tool for the job to me. Mind that RV monitors would be
> > simple users of BPF, requiring only modifications in the RV subsystem but using
> > standard BPF facilities.
> >
>
> As RV would simply be a user of BPF and not modifying BPF infrastructure,
> I'm not so sure you have the jurisdiction to NAK it. It would be like me
> NAKing how you create a trace event.
>
> RV monitors are a very good way to make sure the system is preforming
> properly. This looks exactly like a perfect use case for using BPF instead
> of having to install modules.
>
> Either BPF is a tool for the kernel or it isn't. You can't judge where BPF
> gets used. You can only judge on modifications to BFP.

Ok.
Please add
Nacked-by: Alexei Starovoitov <ast@kernel.org>

when you submit it to Linus.
Let him decide whether bpf maintainers have an authority
to say where bpf is used in the kernel.

On Thu, 3 Sep 2026 20:30:19 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> > > On Wed, 2026-09-02 at 18:57 -0700, Alexei Starovoitov wrote:  
> > > > I don't think bpf fits here. I haven't seen active use of RV and even less
> > > > so of any request from people who want this kind of programmability.
> > > >
> > > > So Nack for now. Sorry.  
> > >

> 
> Ok.
> Please add
> Nacked-by: Alexei Starovoitov <ast@kernel.org>
> 
> when you submit it to Linus.
> Let him decide whether bpf maintainers have an authority
> to say where bpf is used in the kernel.

When this is ready, I'll be happy to inform Linus on the pull request
that you NACKed it, if you are still against it by then. He's Cc'd on
this conversation now. But you left out any technical reason for the
NACK. You basically just said "I don't see how this is useful to me".
That's not a valid reason for a NACK.

Gabriele will be presenting this work at Linux Plumbers[1]. This will
not be going in before then. I would strongly recommend attending
Gabriele's session and we can then go into the technical arguments
about the use of BPF there. If you bring up valid technical reasons
against it, then we will either work with you to solve those technical
issues, and if they can't be solved then sure, it will not go in. But
if it's just your opinion about the work not being useful to you, then
I will push this forward, as it is very useful to others.

Cheers,

-- Steve

[1] https://lpc.events/event/20/contributions/2446/

On Fri Sep 4, 2026 at 4:43 AM PDT, Steven Rostedt wrote:
> On Thu, 3 Sep 2026 20:30:19 -0700
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
>> > > On Wed, 2026-09-02 at 18:57 -0700, Alexei Starovoitov wrote:  
>> > > > I don't think bpf fits here. I haven't seen active use of RV and even less
>> > > > so of any request from people who want this kind of programmability.
>> > > >
>> > > > So Nack for now. Sorry.  
>> > >
>
>> 
>> Ok.
>> Please add
>> Nacked-by: Alexei Starovoitov <ast@kernel.org>
>> 
>> when you submit it to Linus.
>> Let him decide whether bpf maintainers have an authority
>> to say where bpf is used in the kernel.
>
> When this is ready, I'll be happy to inform Linus on the pull request
> that you NACKed it, if you are still against it by then. He's Cc'd on
> this conversation now. But you left out any technical reason for the
> NACK. You basically just said "I don't see how this is useful to me".
> That's not a valid reason for a NACK.

You added a whole bunch of hardcoded "monitors". None of them were
necessary. The whole RV is imo a waste of kernel code.
All of that could have been done via existing bpf tracing functionality.
One can delete kernel/trace/rv and do the same thing with bpf.
So I'm strongly against bolting bpf to RV as yet another "monitor".
It's a wrong design. Keep adding hard coded monitors and don't mess
with bpf.

> Gabriele will be presenting this work at Linux Plumbers[1]. This will

I didn't vote for it. Other bpf maintainers did. Hence it's in schedule.
But my stance is still a nack.

On Fri, 04 Sep 2026 09:16:28 -0700
"Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:

> You added a whole bunch of hardcoded "monitors". None of them were
> necessary. The whole RV is imo a waste of kernel code.
> All of that could have been done via existing bpf tracing functionality.
> One can delete kernel/trace/rv and do the same thing with bpf.
> So I'm strongly against bolting bpf to RV as yet another "monitor".
> It's a wrong design. Keep adding hard coded monitors and don't mess
> with bpf.

In the beginning, Daniel Bristot looked into doing this with BPF and found
issues with it. I do not recall what they were and unfortunately Daniel is
no longer around to explain it. Maybe Gabriele knows what they were.

> 
> > Gabriele will be presenting this work at Linux Plumbers[1]. This will  
> 
> I didn't vote for it. Other bpf maintainers did. Hence it's in schedule.
> But my stance is still a nack.

So be it, it will likely be a lively discussion ;-)

-- Steve

On Fri, 4 Sep 2026 12:31:08 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> In the beginning, Daniel Bristot looked into doing this with BPF and found
> issues with it. I do not recall what they were and unfortunately Daniel is
> no longer around to explain it. Maybe Gabriele knows what they were.

IIRC, Daniel even told me there was a competing effort within Red Hat to
implement the same functionality using only BPF. That effort never
materialized and everything went toward the RV code Daniel was working on.

This was years ago. Gabriele do you know anything about that?

-- Steve

On Fri Sep 4, 2026 at 10:24 AM PDT, Steven Rostedt wrote:
> On Fri, 4 Sep 2026 12:31:08 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
>> In the beginning, Daniel Bristot looked into doing this with BPF and found
>> issues with it. I do not recall what they were and unfortunately Daniel is
>> no longer around to explain it. Maybe Gabriele knows what they were.
>
> IIRC, Daniel even told me there was a competing effort within Red Hat to
> implement the same functionality using only BPF. That effort never
> materialized and everything went toward the RV code Daniel was working on.

bpf in 2022 was surely less capable then it is today.
It took us 2 years of bpf core development to statisfy sched-ext demands
and we're still adding new features for sched-ext needs.
If you're trully willing to remove 90% of kernel/trace/rv/ and refactor
it into tiny shim where all of the core pieces are bpf driven we can
certainly work together (like we did with sched-ext) and add whatever
is missing on bpf side. Then all existing monitors will become bpf programs.
But adding bpf as another 'monitor', sorry but hard NO.
If hardcoded monitors was a mistake then admit it and fix it by deleting it,
if it's not a mistake then keep adding hardcoded monitors.

On Fri, 04 Sep 2026 16:34:00 -0700
"Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:

> bpf in 2022 was surely less capable then it is today.
> It took us 2 years of bpf core development to statisfy sched-ext demands
> and we're still adding new features for sched-ext needs.
> If you're trully willing to remove 90% of kernel/trace/rv/ and refactor
> it into tiny shim where all of the core pieces are bpf driven we can
> certainly work together (like we did with sched-ext) and add whatever
> is missing on bpf side. Then all existing monitors will become bpf programs.
> But adding bpf as another 'monitor', sorry but hard NO.
> If hardcoded monitors was a mistake then admit it and fix it by deleting it,
> if it's not a mistake then keep adding hardcoded monitors.

Regardless of whether or not BPF can replace "hardcoded monitors" today, it
wasn't a mistake back then if BPF wasn't able to do it when they were first
being added. sched_ext wanted to use BPF for scheduling as module plugins
were nack'd by the scheduler maintainers for a long time. BPF programs to
handle scheduling was the work-around to that, and basically the only way
forward. And it still required special hooks into the scheduler.

The rv monitors only needed to use tracepoints for hooks. A module was the
easiest way to get there, as BPF at the time wasn't an option.

If you want to use hostile language like "admit you made a mistake" then it
makes it harder to collaborate.

-- Steve

On Fri, Sep 4, 2026 at 4:45 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 04 Sep 2026 16:34:00 -0700
> "Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:
>
> > bpf in 2022 was surely less capable then it is today.
> > It took us 2 years of bpf core development to statisfy sched-ext demands
> > and we're still adding new features for sched-ext needs.
> > If you're trully willing to remove 90% of kernel/trace/rv/ and refactor
> > it into tiny shim where all of the core pieces are bpf driven we can
> > certainly work together (like we did with sched-ext) and add whatever
> > is missing on bpf side. Then all existing monitors will become bpf programs.
> > But adding bpf as another 'monitor', sorry but hard NO.
> > If hardcoded monitors was a mistake then admit it and fix it by deleting it,
> > if it's not a mistake then keep adding hardcoded monitors.
>
> Regardless of whether or not BPF can replace "hardcoded monitors" today, it
> wasn't a mistake back then if BPF wasn't able to do it when they were first
> being added. sched_ext wanted to use BPF for scheduling as module plugins
> were nack'd by the scheduler maintainers for a long time. BPF programs to
> handle scheduling was the work-around to that, and basically the only way
> forward. And it still required special hooks into the scheduler.
>
> The rv monitors only needed to use tracepoints for hooks. A module was the
> easiest way to get there, as BPF at the time wasn't an option.
>
> If you want to use hostile language like "admit you made a mistake" then it
> makes it harder to collaborate.

If you're offended by the word "mistake" then ok. There is no path forward.
Keep adding hardcoded stuff.

On Fri, 4 Sep 2026 16:48:55 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> On Fri, Sep 4, 2026 at 4:45 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Fri, 04 Sep 2026 16:34:00 -0700
> > "Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:
> >  
> > > bpf in 2022 was surely less capable then it is today.
> > > It took us 2 years of bpf core development to statisfy sched-ext demands
> > > and we're still adding new features for sched-ext needs.
> > > If you're trully willing to remove 90% of kernel/trace/rv/ and refactor
> > > it into tiny shim where all of the core pieces are bpf driven we can
> > > certainly work together (like we did with sched-ext) and add whatever
> > > is missing on bpf side. Then all existing monitors will become bpf programs.
> > > But adding bpf as another 'monitor', sorry but hard NO.
> > > If hardcoded monitors was a mistake then admit it and fix it by deleting it,
> > > if it's not a mistake then keep adding hardcoded monitors.  
> >
> > Regardless of whether or not BPF can replace "hardcoded monitors" today, it
> > wasn't a mistake back then if BPF wasn't able to do it when they were first
> > being added. sched_ext wanted to use BPF for scheduling as module plugins
> > were nack'd by the scheduler maintainers for a long time. BPF programs to
> > handle scheduling was the work-around to that, and basically the only way
> > forward. And it still required special hooks into the scheduler.
> >
> > The rv monitors only needed to use tracepoints for hooks. A module was the
> > easiest way to get there, as BPF at the time wasn't an option.
> >
> > If you want to use hostile language like "admit you made a mistake" then it
> > makes it harder to collaborate.  
> 
> If you're offended by the word "mistake" then ok. There is no path forward.
> Keep adding hardcoded stuff.

I'm not offended by the words, I'm offended by your tone. As BPF was
insufficient when RV started, it obviously wasn't a mistake that we didn't
use it. If you are just going to say it was a mistake because we didn't
modify BPF to suite our needs (which I highly doubt you would have wanted
to help us, as you already stated that you don't care about this project),
then you are obviously biased against us.

Your response was fine until you through in that last sentence. As making a
mistake means that the other non-mistake option was viable at the item. It
wasn't.

Look, if BPF has changed so much that it could be re-evaluated to see if it
suits the needs of RV, then sure. We could look into doing that. Perhaps it
can replace the hard-coded monitors. But what we have been doing has not
been a mistake. Saying so is a bit disrespectful to the project and a
direct insult to Daniel.

-- Steve


-- Steve

On Fri, Sep 4, 2026 at 5:16 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 4 Sep 2026 16:48:55 -0700
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> > On Fri, Sep 4, 2026 at 4:45 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > On Fri, 04 Sep 2026 16:34:00 -0700
> > > "Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:
> > >
> > > > bpf in 2022 was surely less capable then it is today.
> > > > It took us 2 years of bpf core development to statisfy sched-ext demands
> > > > and we're still adding new features for sched-ext needs.
> > > > If you're trully willing to remove 90% of kernel/trace/rv/ and refactor
> > > > it into tiny shim where all of the core pieces are bpf driven we can
> > > > certainly work together (like we did with sched-ext) and add whatever
> > > > is missing on bpf side. Then all existing monitors will become bpf programs.
> > > > But adding bpf as another 'monitor', sorry but hard NO.
> > > > If hardcoded monitors was a mistake then admit it and fix it by deleting it,
> > > > if it's not a mistake then keep adding hardcoded monitors.
> > >
> > > Regardless of whether or not BPF can replace "hardcoded monitors" today, it
> > > wasn't a mistake back then if BPF wasn't able to do it when they were first
> > > being added. sched_ext wanted to use BPF for scheduling as module plugins
> > > were nack'd by the scheduler maintainers for a long time. BPF programs to
> > > handle scheduling was the work-around to that, and basically the only way
> > > forward. And it still required special hooks into the scheduler.
> > >
> > > The rv monitors only needed to use tracepoints for hooks. A module was the
> > > easiest way to get there, as BPF at the time wasn't an option.
> > >
> > > If you want to use hostile language like "admit you made a mistake" then it
> > > makes it harder to collaborate.
> >
> > If you're offended by the word "mistake" then ok. There is no path forward.
> > Keep adding hardcoded stuff.
>
> I'm not offended by the words, I'm offended by your tone. As BPF was
> insufficient when RV started, it obviously wasn't a mistake that we didn't
> use it. If you are just going to say it was a mistake because we didn't
> modify BPF to suite our needs (which I highly doubt you would have wanted
> to help us, as you already stated that you don't care about this project),
> then you are obviously biased against us.
>
> Your response was fine until you through in that last sentence. As making a
> mistake means that the other non-mistake option was viable at the item. It
> wasn't.
>
> Look, if BPF has changed so much that it could be re-evaluated to see if it
> suits the needs of RV, then sure. We could look into doing that. Perhaps it
> can replace the hard-coded monitors. But what we have been doing has not
> been a mistake. Saying so is a bit disrespectful to the project and a
> direct insult to Daniel.

I respectfully disagree. Let's stop here.

Il 5 settembre 2026 00:22:01 UTC, Alexei Starovoitov <alexei.starovoitov@gmail.com> ha scritto:
>On Fri, Sep 4, 2026 at 5:16 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>>
>> On Fri, 4 Sep 2026 16:48:55 -0700
>> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>>
>> > On Fri, Sep 4, 2026 at 4:45 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>> > >
>> > > On Fri, 04 Sep 2026 16:34:00 -0700
>> > > "Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:
>> > >
>> > > > bpf in 2022 was surely less capable then it is today.
>> > > > It took us 2 years of bpf core development to statisfy sched-ext demands
>> > > > and we're still adding new features for sched-ext needs.
>> > > > If you're trully willing to remove 90% of kernel/trace/rv/ and refactor
>> > > > it into tiny shim where all of the core pieces are bpf driven we can
>> > > > certainly work together (like we did with sched-ext) and add whatever
>> > > > is missing on bpf side. Then all existing monitors will become bpf programs.
>> > > > But adding bpf as another 'monitor', sorry but hard NO.
>> > > > If hardcoded monitors was a mistake then admit it and fix it by deleting it,
>> > > > if it's not a mistake then keep adding hardcoded monitors.

Alexei, what I read from your opinion is: if you really want to do this RV thing, then you should just implement it all in BPF.

Please correct me if I'm wrong, but I don't see how this series isn't a step towards it. I am obviously open to all sorts of criticism on /how/ I'm doing it.
We are adding the possibility to write (existing or new) monitors in BPF, not adding another fat "bpf monitor".

And if you believe my attempt to have in-kernel and BPF monitors coexist is a waste of time, I get that too and I'm open to discuss.

As you mentioned, when the project started, BPF was just not ready. My intention with this series is to reassess this today.

RV was initially built with the functional safety use-case in mind. For various technical reasons this didn't take off, but that scenario drove towards implementing RV reactors like the panic() one. Something BPF just shouldn't do, as I get it.

The interoperability with other tools (via RV-generated tracepoints) was also a driving factor, together with Daniel probably being more proficient with those back then.

That's just from the top of my head: there are historical and technical reasons for why RV looks like it is today, whether those are valid or just "mistakes" is simply not for us to say.

I'd rather discuss on what is the best approach /today/. And that's precisely why I submitted the talk for LPC.

Gabriele

On Fri Sep 4, 2026 at 9:53 PM PDT, Gabriele Monaco wrote:
>

Sorry for the delay.

> Alexei, what I read from your opinion is: if you really want to do this RV thing, then you should just implement it all in BPF.

yes.

> like the panic() one. Something BPF just shouldn't do, as I get it.

There is a disconnect here.
we have one KF_DESTRUCTIVE kfunc already. bpf_panic() can be another one.
It will require CAP_SYS_BOOT.

> I'd rather discuss on what is the best approach /today/. And that's precisely why I submitted the talk for LPC.

Excellent. We can start this discussion over email and continue at LPC.
My understanding of RV is primitive, but from reading kernel/trace/rv/*.c
it seems to me that it's a thin glue between tracepoints and monitors,
a bit of boiler plate code via tracefs to enable monitors and seq file for visibility.
What you're proposing is "yet another monitor" that is reusing this glue code,
and that's my main objection. I don't see the value in kernel/trace/rv/*.c.
(kernel/trace/rv/monitors/* are useful, of course)

What stops you from attaching tracing bpf progs to all tracepoints that you need,
pinning few "bpf iterator" progs in bpffs that will provide text or binary
output via seq files, run a state machine inside the prog,
and do whatever "verification" logic inside them ?
You don't need the rv/*.c glue. I see no need to introduce new bpf struct-ops api
just to look-like-a-monitor from RV glue perspective.
"runtime verification" as a concept makes sense, so focus on that.

On Sat, 2026-09-12 at 18:26 -0700, Alexei Starovoitov wrote:
> On Fri Sep 4, 2026 at 9:53 PM PDT, Gabriele Monaco wrote:
> 
> > Alexei, what I read from your opinion is: if you really want to do this RV
> > thing, then you should just implement it all in BPF.
> 
> yes.
> 
> > like the panic() one. Something BPF just shouldn't do, as I get it.
> 
> There is a disconnect here.
> we have one KF_DESTRUCTIVE kfunc already. bpf_panic() can be another one.
> It will require CAP_SYS_BOOT.

Great, I have also been pointed to crash_kexec() [1] which seems to be doing
already the same thing.
Anyway this doesn't seem a blocker.

> > I'd rather discuss on what is the best approach /today/. And that's
> > precisely why I submitted the talk for LPC.
> 
> Excellent. We can start this discussion over email and continue at LPC.
> My understanding of RV is primitive, but from reading kernel/trace/rv/*.c
> it seems to me that it's a thin glue between tracepoints and monitors,
> a bit of boiler plate code via tracefs to enable monitors and seq file for
> visibility.
> What you're proposing is "yet another monitor" that is reusing this glue code,
> and that's my main objection. I don't see the value in kernel/trace/rv/*.c.
> (kernel/trace/rv/monitors/* are useful, of course)

I get it, essentially the point of contention here is that, besides loading
tracing BPF programs (the event handlers), I'm /also/ loading a BPF struct_ops
program to register the monitor to the existing in-kernel infrastructure.

It is indeed not fully necessary to have BPF monitors share the same sysfs API,
as they still need the userspace component to do useful monitoring.

> What stops you from attaching tracing bpf progs to all tracepoints that you
> need, pinning few "bpf iterator" progs in bpffs that will provide text or
> binary output via seq files, run a state machine inside the prog,
> and do whatever "verification" logic inside them ?
> You don't need the rv/*.c glue. I see no need to introduce new bpf struct-ops
> api just to look-like-a-monitor from RV glue perspective.
> "runtime verification" as a concept makes sense, so focus on that.

I actually started my POC without struct_ops, activation happened by just
triggering a dummy BPF program from userspace, but I'll look into these bpf
iterators.

Essentially sharing the API simplified how the userspace component handles some
things and allowed to share reactors.
But again, that's not necessarily the way.

Thanks,
Gabriele

[1] - https://docs.ebpf.io/linux/kfuncs/crash_kexec

On Tue Sep 15, 2026 at 7:15 AM UTC, Gabriele Monaco wrote:
>
> Essentially sharing the API simplified how the userspace component handles some
> things and allowed to share reactors.

It's surely tempting to reuse userspace, but unlike kernel the user
space is much easier to change, so let's not add kernel struct-ops,
since they're not necessary to achieve the goal.

Gabriele Monaco <gmonaco@redhat.com> writes:
> I should definitely have included it in the cover letter.. I'm writing it
> everywhere (will present at LPC) but forgot it here.
>
> Essentially BPF monitors can be pluggable, folks writing their own monitors
> won't need to submit a patch or maintain a separate tree, which is useful for
> domain-specific models.
> By being pluggable you also don't need to reboot to use a new/updated monitor.
>
> Think of being able to distribute a more granular set of rules for RTapp, I
> remember we had conversation along those lines, not all rules apply to all
> contexts and what you send upstream has to be general, what you keep for
> yourself doesn't.
>
> Having monitors in BPF brings also other perks over kernel modules: a whole
> bunch of readily available probe types (uprobes, fprobes, all unexported
> tracepoints that are cumbersome for modules), the map infrastructure for
> allocation is arguably easier and the code is verified when loaded against
> common issues (NULL pointer access, unbound loops, etc.).
>
> That said, I try to mimic as much as possible the in-kernel functionality, but
> some things are not the same (event/error tracepoints).
>
> These support DA only because BPF loading needs a userspace component and the RV
> tool doesn't support LTL and HA yet, there shouldn't be any technical reason not
> to extend to those in the future.

Cool. I also wanted to do something like this, for these exact same
reasons. Let me look at your patches..

Nam