include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++ kernel/softirq.c | 9 ++++++-- 2 files changed, 54 insertions(+), 2 deletions(-)
commit f4bf3ca2e5cba655824b6e0893a98dfb33ed24e5 upstream.
Tasklets are supposed to finish their work quickly and should not block the
current running process, but it is not guaranteed that they do so.
Currently softirq_entry/exit can be used to analyse the total tasklets
execution time, but that's not helpful to track individual tasklets
execution time. That makes it hard to identify tasklet functions, which
take more time than expected.
Add tasklet_entry/exit trace point support to track individual tasklet
execution.
Trivial usage example:
# echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable
# echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable
# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 4/4 #P:4
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
<idle>-0 [003] ..s1. 314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
<idle>-0 [003] ..s1. 314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
<idle>-0 [003] ..s1. 314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
<idle>-0 [003] ..s1. 314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: J. Avila <elavila@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Link: https://lore.kernel.org/r/20230407230526.1685443-1-jstultz@google.com
[elavila: Port to android-mainline]
[jstultz: Rebased to upstream, cut unused trace points, added
comments for the tracepoints, reworded commit]
Signed-off-by: Sumanth Gavini <sumanth.gavini@yahoo.com>
---
include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++
kernel/softirq.c | 9 ++++++--
2 files changed, 54 insertions(+), 2 deletions(-)
diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h
index eeceafaaea4c..a07b4607b663 100644
--- a/include/trace/events/irq.h
+++ b/include/trace/events/irq.h
@@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise,
TP_ARGS(vec_nr)
);
+DECLARE_EVENT_CLASS(tasklet,
+
+ TP_PROTO(struct tasklet_struct *t, void *func),
+
+ TP_ARGS(t, func),
+
+ TP_STRUCT__entry(
+ __field( void *, tasklet)
+ __field( void *, func)
+ ),
+
+ TP_fast_assign(
+ __entry->tasklet = t;
+ __entry->func = func;
+ ),
+
+ TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func)
+);
+
+/**
+ * tasklet_entry - called immediately before the tasklet is run
+ * @t: tasklet pointer
+ * @func: tasklet callback or function being run
+ *
+ * Used to find individual tasklet execution time
+ */
+DEFINE_EVENT(tasklet, tasklet_entry,
+
+ TP_PROTO(struct tasklet_struct *t, void *func),
+
+ TP_ARGS(t, func)
+);
+
+/**
+ * tasklet_exit - called immediately after the tasklet is run
+ * @t: tasklet pointer
+ * @func: tasklet callback or function being run
+ *
+ * Used to find individual tasklet execution time
+ */
+DEFINE_EVENT(tasklet, tasklet_exit,
+
+ TP_PROTO(struct tasklet_struct *t, void *func),
+
+ TP_ARGS(t, func)
+);
+
#endif /* _TRACE_IRQ_H */
/* This part must be outside protection */
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 9ab5ca399a99..fadc6bbda27b 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -822,10 +822,15 @@ static void tasklet_action_common(struct softirq_action *a,
if (tasklet_trylock(t)) {
if (!atomic_read(&t->count)) {
if (tasklet_clear_sched(t)) {
- if (t->use_callback)
+ if (t->use_callback) {
+ trace_tasklet_entry(t, t->callback);
t->callback(t);
- else
+ trace_tasklet_exit(t, t->callback);
+ } else {
+ trace_tasklet_entry(t, t->func);
t->func(t->data);
+ trace_tasklet_exit(t, t->func);
+ }
}
tasklet_unlock(t);
continue;
--
2.43.0On Tue, Aug 12, 2025 at 11:17:54AM -0500, Sumanth Gavini wrote: > commit f4bf3ca2e5cba655824b6e0893a98dfb33ed24e5 upstream. > > Tasklets are supposed to finish their work quickly and should not block the > current running process, but it is not guaranteed that they do so. > > Currently softirq_entry/exit can be used to analyse the total tasklets > execution time, but that's not helpful to track individual tasklets > execution time. That makes it hard to identify tasklet functions, which > take more time than expected. > > Add tasklet_entry/exit trace point support to track individual tasklet > execution. > > Trivial usage example: > # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable > # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable > # cat /sys/kernel/debug/tracing/trace > # tracer: nop > # > # entries-in-buffer/entries-written: 4/4 #P:4 > # > # _-----=> irqs-off/BH-disabled > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / _-=> migrate-disable > # |||| / delay > # TASK-PID CPU# ||||| TIMESTAMP FUNCTION > # | | | ||||| | | > <idle>-0 [003] ..s1. 314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func > <idle>-0 [003] ..s1. 314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func > <idle>-0 [003] ..s1. 314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func > <idle>-0 [003] ..s1. 314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func > > Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org> > Signed-off-by: J. Avila <elavila@google.com> > Signed-off-by: John Stultz <jstultz@google.com> > Signed-off-by: Thomas Gleixner <tglx@linutronix.de> > Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org> > Link: https://lore.kernel.org/r/20230407230526.1685443-1-jstultz@google.com > > [elavila: Port to android-mainline] This is not android-mainline, this is the normal stable tree. And I'm with John, this makes no sense as to why you need/want these. I think that the syzbot report is bogus, sorry. Please prove me wrong :) thanks, greg k-h
On Fri, Aug 22 2025 at 15:07, Greg KH wrote:
> On Tue, Aug 12, 2025 at 11:17:54AM -0500, Sumanth Gavini wrote:
>
> And I'm with John, this makes no sense as to why you need/want these. I
> think that the syzbot report is bogus, sorry. Please prove me wrong :)
It was validated by AI (Absense of Intelligence) that adding tracepoints
makes the problem go away! So why do you want extra proof?
Thanks,
tglx
Hi Thomas, John, Thanks for the feedback. Just to clarify — my intention here was only to backport the already accepted upstream changes into this branch. I do not plan to introduce any additional modifications or syzbot-only patches. Thank you for your time and consideration. Regards, Sumanth Gavini
On Tue, Sep 23, 2025 at 6:13 AM Sumanth Gavini <sumanth.gavini@yahoo.com> wrote: > > Hi Thomas, John, > > Thanks for the feedback. Just to clarify — my intention here was only to backport the already accepted upstream changes into this branch. I do not plan to introduce any additional modifications or syzbot-only patches. > Might be good to revisit: https://docs.kernel.org/process/stable-kernel-rules.html And, assuming you feel like it still might be needed, you might add some annotation before your signed-off-by line to clarify your motivation for submitting it, ie: how it "fix[es] a real bug that bothers people". Adding tracepoints definitely feels more like a feature, not a bug fix. thanks -john
Hi All, Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit". Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/ Would you have any feedback on this change? I'd be happy to address any comments or concerns. This patch fixes this three bugs 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4 Thank you for your time and consideration. Regards, Sumanth Gavini
On Thu, Aug 21, 2025 at 11:41 AM Sumanth Gavini <sumanth.gavini@yahoo.com> wrote: > > Hi All, > > Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit". > > Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/ > > Would you have any feedback on this change? I'd be happy to address any comments or concerns. > > This patch fixes this three bugs > 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a > 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511 > 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4 How does a patch adding a tracepoint fix the bugs highlighted here? It seems maybe it would help in debugging those issues, but I'm not sure I see how it would fix them. thanks -john
>> >> Hi All, >> >> Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit". >> >> Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/ >> >> Would you have any feedback on this change? I'd be happy to address any comments or concerns. >> >> This patch fixes this three bugs >> 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a >> 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511 >> 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4 > How does a patch adding a tracepoint fix the bugs highlighted here? > It seems maybe it would help in debugging those issues, but I'm not > sure I see how it would fix them. This patch is related to linux 6.1/backports, the backports(https://syzkaller.appspot.com/linux-6.1/backports) I see this patch would fix these bugs. Let me know if my understand is wrong. Regards, Sumanth
On Thu, Aug 21, 2025 at 2:02 PM Sumanth Gavini <sumanth.gavini@yahoo.com> wrote: > >> > >> Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit". > >> > >> Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/ > >> > >> Would you have any feedback on this change? I'd be happy to address any comments or concerns. > >> > >> This patch fixes this three bugs > >> 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a > >> 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511 > >> 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4 > > > How does a patch adding a tracepoint fix the bugs highlighted here? > > It seems maybe it would help in debugging those issues, but I'm not > > sure I see how it would fix them. > > This patch is related to linux 6.1/backports, the backports(https://syzkaller.appspot.com/linux-6.1/backports) > I see this patch would fix these bugs. Let me know if my understand is wrong. But that doesn't explain why or how it fixes the bugs. I'm not opposed to stable taking this, but the reasoning should be clear, if that is the motivation for including this change. I fret there is something incidental in this patch that avoids the problem, and that those issues may need a deeper fix rather then to hide them with this change. thanks -john
© 2016 - 2025 Red Hat, Inc.