We would like to have in production a way to record sched wakeups and
sched switch, and be able to save the information in a small file
with as much available as possible. Currently the wake up and sched switch
events are 36 and 64 bytes each (plus a 4 byte ring buffer event header).

By having a custom module tap into the sched switch and waking trace points
we can bring those events down to 16 and 14 bytes respectively.

Steven Rostedt (Google) (2):
      tracing: Allow custom events to be added to the tracefs directory
      tracing: Add sample code for custom trace events

----
 kernel/trace/trace_events.c               |   2 +
 samples/Kconfig                           |   8 +-
 samples/Makefile                          |   1 +
 samples/trace_events/Makefile             |   2 +
 samples/trace_events/trace_custom_sched.c | 280 ++++++++++++++++++++++++++++++
 5 files changed, 292 insertions(+), 1 deletion(-)
 create mode 100644 samples/trace_events/trace_custom_sched.c

On Tue, Mar 1, 2022 at 10:28 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> We would like to have in production a way to record sched wakeups and
> sched switch, and be able to save the information in a small file
> with as much available as possible. Currently the wake up and sched switch
> events are 36 and 64 bytes each (plus a 4 byte ring buffer event header).
>
> By having a custom module tap into the sched switch and waking trace points
> we can bring those events down to 16 and 14 bytes respectively.
>
> Steven Rostedt (Google) (2):
>       tracing: Allow custom events to be added to the tracefs directory
>       tracing: Add sample code for custom trace events

Great! I will test these out / review it in the coming week or so.

Thanks!

- Joel

>
> ----
>  kernel/trace/trace_events.c               |   2 +
>  samples/Kconfig                           |   8 +-
>  samples/Makefile                          |   1 +
>  samples/trace_events/Makefile             |   2 +
>  samples/trace_events/trace_custom_sched.c | 280 ++++++++++++++++++++++++++++++
>  5 files changed, 292 insertions(+), 1 deletion(-)
>  create mode 100644 samples/trace_events/trace_custom_sched.c

On Tue, 01 Mar 2022 22:24:14 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> We would like to have in production a way to record sched wakeups and
> sched switch, and be able to save the information in a small file
> with as much available as possible. Currently the wake up and sched switch
> events are 36 and 64 bytes each (plus a 4 byte ring buffer event header).
> 
> By having a custom module tap into the sched switch and waking trace points
> we can bring those events down to 16 and 14 bytes respectively.

OK, so we can use eprobe to shrink down the 'visible' log for the event,
but it still consumes the event buffer because eprobe will fetch the event
data from the event log. So to reduce the actual consumption of the
trace buffer, we have to define a new event format and callback.

Thank you,

> 
> Steven Rostedt (Google) (2):
>       tracing: Allow custom events to be added to the tracefs directory
>       tracing: Add sample code for custom trace events
> 
> ----
>  kernel/trace/trace_events.c               |   2 +
>  samples/Kconfig                           |   8 +-
>  samples/Makefile                          |   1 +
>  samples/trace_events/Makefile             |   2 +
>  samples/trace_events/trace_custom_sched.c | 280 ++++++++++++++++++++++++++++++
>  5 files changed, 292 insertions(+), 1 deletion(-)
>  create mode 100644 samples/trace_events/trace_custom_sched.c


-- 
Masami Hiramatsu <mhiramat@kernel.org>

On Thu, 3 Mar 2022 10:31:01 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> On Tue, 01 Mar 2022 22:24:14 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > We would like to have in production a way to record sched wakeups and
> > sched switch, and be able to save the information in a small file
> > with as much available as possible. Currently the wake up and sched switch
> > events are 36 and 64 bytes each (plus a 4 byte ring buffer event header).
> > 
> > By having a custom module tap into the sched switch and waking trace points
> > we can bring those events down to 16 and 14 bytes respectively.  
> 
> OK, so we can use eprobe to shrink down the 'visible' log for the event,
> but it still consumes the event buffer because eprobe will fetch the event
> data from the event log. So to reduce the actual consumption of the
> trace buffer, we have to define a new event format and callback.
>

Well, the buffer content itself is shrunk, and we were using eprobes to begin
with for this purpose. The issue is that eprobes still needs to record the
event into a temporary buffer (or the ring buffer then discard it) to
copy the data into the eprobe. This makes using eprobes slower than the
event it is taken from, as the event it is attached to must run first.

Since we have the ability to create a custom module, to do this
directly, and this is much smaller and even a bit faster than the
tracepoints we are attached to.

-- Steve