During kernel boot, the setup_boot_kprobe_events() function causes
significant delays, increasing overall startup time.
The root cause is a lock contention chain: its child function
enable_boot_kprobe_events() requires the event_mutex, which is
already held by early_event_add_tracer(). early_event_add_tracer()
itself is blocked waiting for the trace_event_sem read-write lock,
which is held for an extended period by trace_event_update_all().
To resolve this, we have moved the execution of
setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing
it to run asynchronously.
Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
---
kernel/trace/trace_kprobe.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 9953506370a5..4c6621f02696 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void)
}
core_initcall(init_kprobe_trace_early);
+static struct work_struct kprobe_trace_work __initdata;
+
+static void __init kprobe_trace_works_func(struct work_struct *work)
+{
+ setup_boot_kprobe_events();
+}
+
/* Make a tracefs interface for controlling probe points */
static __init int init_kprobe_trace(void)
{
@@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void)
trace_create_file("kprobe_profile", TRACE_MODE_READ,
NULL, NULL, &kprobe_profile_ops);
- setup_boot_kprobe_events();
+ if (trace_init_wq) {
+ INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func);
+ queue_work(trace_init_wq, &kprobe_trace_work);
+ } else {
+ setup_boot_kprobe_events();
+ }
return 0;
}
--
2.25.1On Tue, 27 Jan 2026 09:23:12 +0800
Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
> During kernel boot, the setup_boot_kprobe_events() function causes
> significant delays, increasing overall startup time.
>
> The root cause is a lock contention chain: its child function
> enable_boot_kprobe_events() requires the event_mutex, which is
> already held by early_event_add_tracer(). early_event_add_tracer()
> itself is blocked waiting for the trace_event_sem read-write lock,
> which is held for an extended period by trace_event_update_all().
>
> To resolve this, we have moved the execution of
> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing
> it to run asynchronously.
>
> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
> ---
> kernel/trace/trace_kprobe.c | 14 +++++++++++++-
> 1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
> index 9953506370a5..4c6621f02696 100644
> --- a/kernel/trace/trace_kprobe.c
> +++ b/kernel/trace/trace_kprobe.c
> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void)
> }
> core_initcall(init_kprobe_trace_early);
>
> +static struct work_struct kprobe_trace_work __initdata;
> +
> +static void __init kprobe_trace_works_func(struct work_struct *work)
> +{
> + setup_boot_kprobe_events();
> +}
> +
> /* Make a tracefs interface for controlling probe points */
> static __init int init_kprobe_trace(void)
> {
> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void)
> trace_create_file("kprobe_profile", TRACE_MODE_READ,
> NULL, NULL, &kprobe_profile_ops);
>
> - setup_boot_kprobe_events();
> + if (trace_init_wq) {
> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func);
> + queue_work(trace_init_wq, &kprobe_trace_work);
Hmm, this queue_work is not required if kprobe_boot_events_buf[] is
empty. We should check it because most of the time we don't need it.
Also, deferring initialization makes it indeterminate when this
tracing will begin.
For kprobe event use case, I think setup_boot_kprobe_events() should
check the kprobe_boot_events_buf is empty at first. But I think Yaxiong
use case happens when you are using kprobe events via cmdline, is that
correct?
I think introducing "async" cmdline option is more preferable.
BTW, I found that the kprobe events from kernel cmdline will be made
after boot-time tracing from bootconfig. Maybe it should be run this
earlier timing too.
Thank you,
> + } else {
> + setup_boot_kprobe_events();
> + }
>
> return 0;
> }
> --
> 2.25.1
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道:
> On Tue, 27 Jan 2026 09:23:12 +0800
> Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
>
>> During kernel boot, the setup_boot_kprobe_events() function causes
>> significant delays, increasing overall startup time.
>>
>> The root cause is a lock contention chain: its child function
>> enable_boot_kprobe_events() requires the event_mutex, which is
>> already held by early_event_add_tracer(). early_event_add_tracer()
>> itself is blocked waiting for the trace_event_sem read-write lock,
>> which is held for an extended period by trace_event_update_all().
>>
>> To resolve this, we have moved the execution of
>> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing
>> it to run asynchronously.
>>
>> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
>> ---
>> kernel/trace/trace_kprobe.c | 14 +++++++++++++-
>> 1 file changed, 13 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
>> index 9953506370a5..4c6621f02696 100644
>> --- a/kernel/trace/trace_kprobe.c
>> +++ b/kernel/trace/trace_kprobe.c
>> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void)
>> }
>> core_initcall(init_kprobe_trace_early);
>>
>> +static struct work_struct kprobe_trace_work __initdata;
>> +
>> +static void __init kprobe_trace_works_func(struct work_struct *work)
>> +{
>> + setup_boot_kprobe_events();
>> +}
>> +
>> /* Make a tracefs interface for controlling probe points */
>> static __init int init_kprobe_trace(void)
>> {
>> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void)
>> trace_create_file("kprobe_profile", TRACE_MODE_READ,
>> NULL, NULL, &kprobe_profile_ops);
>>
>> - setup_boot_kprobe_events();
>> + if (trace_init_wq) {
>> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func);
>> + queue_work(trace_init_wq, &kprobe_trace_work);
> Hmm, this queue_work is not required if kprobe_boot_events_buf[] is
> empty. We should check it because most of the time we don't need it.
Yes, I will improve it in the next version.
> Also, deferring initialization makes it indeterminate when this
> tracing will begin.
Indeed, While most scenarios don't need boot-time tracing, and users
prioritize boot speed, we must balance the need for deterministic traces
with faster startup.
>
> For kprobe event use case, I think setup_boot_kprobe_events() should
> check the kprobe_boot_events_buf is empty at first. But I think Yaxiong
> use case happens when you are using kprobe events via cmdline, is that
> correct?
The issue was identified without enabling kprobe events via the cmdline.
The core finding is that asynchronous initialization can drastically cut
boot time under different workloads. As blocking occurs elsewhere in the
tracing infrastructure beyond just kprobe events, adopting async is a
broadly applicable strategy for boot time optimization.
>
> I think introducing "async" cmdline option is more preferable.
Agreed, this works. Users focused on boot speed over early-boot tracing
can opt for this parameter to gain a startup performance boost.
>
> BTW, I found that the kprobe events from kernel cmdline will be made
> after boot-time tracing from bootconfig. Maybe it should be run this
> earlier timing too.
Yes. Additionally, this optimization does not conflict with the current
patch series at all.
I'll submit the updated patch for the next version promptly.
> Thank you,
>
>
>> + } else {
>> + setup_boot_kprobe_events();
>> + }
>>
>> return 0;
>> }
>> --
>> 2.25.1
>>
>
On Wed, 28 Jan 2026 15:24:15 +0800
Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
>
> 在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道:
> > On Tue, 27 Jan 2026 09:23:12 +0800
> > Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
> >
> >> During kernel boot, the setup_boot_kprobe_events() function causes
> >> significant delays, increasing overall startup time.
> >>
> >> The root cause is a lock contention chain: its child function
> >> enable_boot_kprobe_events() requires the event_mutex, which is
> >> already held by early_event_add_tracer(). early_event_add_tracer()
> >> itself is blocked waiting for the trace_event_sem read-write lock,
> >> which is held for an extended period by trace_event_update_all().
> >>
> >> To resolve this, we have moved the execution of
> >> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing
> >> it to run asynchronously.
> >>
> >> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
> >> ---
> >> kernel/trace/trace_kprobe.c | 14 +++++++++++++-
> >> 1 file changed, 13 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
> >> index 9953506370a5..4c6621f02696 100644
> >> --- a/kernel/trace/trace_kprobe.c
> >> +++ b/kernel/trace/trace_kprobe.c
> >> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void)
> >> }
> >> core_initcall(init_kprobe_trace_early);
> >>
> >> +static struct work_struct kprobe_trace_work __initdata;
> >> +
> >> +static void __init kprobe_trace_works_func(struct work_struct *work)
> >> +{
> >> + setup_boot_kprobe_events();
> >> +}
> >> +
> >> /* Make a tracefs interface for controlling probe points */
> >> static __init int init_kprobe_trace(void)
> >> {
> >> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void)
> >> trace_create_file("kprobe_profile", TRACE_MODE_READ,
> >> NULL, NULL, &kprobe_profile_ops);
> >>
> >> - setup_boot_kprobe_events();
> >> + if (trace_init_wq) {
> >> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func);
> >> + queue_work(trace_init_wq, &kprobe_trace_work);
> > Hmm, this queue_work is not required if kprobe_boot_events_buf[] is
> > empty. We should check it because most of the time we don't need it.
> Yes, I will improve it in the next version.
> > Also, deferring initialization makes it indeterminate when this
> > tracing will begin.
> Indeed, While most scenarios don't need boot-time tracing, and users
> prioritize boot speed, we must balance the need for deterministic traces
> with faster startup.
I just wonder why don't you define kprobe events after boot (e.g.
from init script) instead of kernel cmdline. Using cmdline means
it will be used for tracing kernel boot.
- tracing kernel boot -> use kernel cmdline (synchronous)
- tracing user boot -> use tracefs (asynchronous)
> > For kprobe event use case, I think setup_boot_kprobe_events() should
> > check the kprobe_boot_events_buf is empty at first. But I think Yaxiong
> > use case happens when you are using kprobe events via cmdline, is that
> > correct?
> The issue was identified without enabling kprobe events via the cmdline.
Interesting. So is it fixed by another patch [1]?
[1] https://lore.kernel.org/all/20260127053848.108473-1-sunliming@linux.dev/
> The core finding is that asynchronous initialization can drastically cut
> boot time under different workloads. As blocking occurs elsewhere in the
> tracing infrastructure beyond just kprobe events, adopting async is a
> broadly applicable strategy for boot time optimization.
Yes, but it is also possible to set it up from user space, because that
user process can work asynchronously.
We can make the ftrace initialization async to accelerate boot time, but
that means it is not useful for tracing kernel boot.
> >
> > I think introducing "async" cmdline option is more preferable.
>
> Agreed, this works. Users focused on boot speed over early-boot tracing
> can opt for this parameter to gain a startup performance boost.
Yeah, that is an option. Anyway, basically, users have another option to
setup ftrace after boot user space asynchronously. That is my
recommendation for such use case.
>
> >
> > BTW, I found that the kprobe events from kernel cmdline will be made
> > after boot-time tracing from bootconfig. Maybe it should be run this
> > earlier timing too.
>
> Yes. Additionally, this optimization does not conflict with the current
> patch series at all.
>
> I'll submit the updated patch for the next version promptly.
OK.
Thank you,
>
> > Thank you,
> >
> >
> >> + } else {
> >> + setup_boot_kprobe_events();
> >> + }
> >>
> >> return 0;
> >> }
> >> --
> >> 2.25.1
> >>
> >
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
在 2026/1/28 15:53, Masami Hiramatsu (Google) 写道:
> On Wed, 28 Jan 2026 15:24:15 +0800
> Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
>
>> 在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道:
>>> On Tue, 27 Jan 2026 09:23:12 +0800
>>> Yaxiong Tian <tianyaxiong@kylinos.cn> wrote:
>>>
>>>> During kernel boot, the setup_boot_kprobe_events() function causes
>>>> significant delays, increasing overall startup time.
>>>>
>>>> The root cause is a lock contention chain: its child function
>>>> enable_boot_kprobe_events() requires the event_mutex, which is
>>>> already held by early_event_add_tracer(). early_event_add_tracer()
>>>> itself is blocked waiting for the trace_event_sem read-write lock,
>>>> which is held for an extended period by trace_event_update_all().
>>>>
>>>> To resolve this, we have moved the execution of
>>>> setup_boot_kprobe_events() to the trace_init_wq workqueue, allowing
>>>> it to run asynchronously.
>>>>
>>>> Signed-off-by: Yaxiong Tian <tianyaxiong@kylinos.cn>
>>>> ---
>>>> kernel/trace/trace_kprobe.c | 14 +++++++++++++-
>>>> 1 file changed, 13 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
>>>> index 9953506370a5..4c6621f02696 100644
>>>> --- a/kernel/trace/trace_kprobe.c
>>>> +++ b/kernel/trace/trace_kprobe.c
>>>> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void)
>>>> }
>>>> core_initcall(init_kprobe_trace_early);
>>>>
>>>> +static struct work_struct kprobe_trace_work __initdata;
>>>> +
>>>> +static void __init kprobe_trace_works_func(struct work_struct *work)
>>>> +{
>>>> + setup_boot_kprobe_events();
>>>> +}
>>>> +
>>>> /* Make a tracefs interface for controlling probe points */
>>>> static __init int init_kprobe_trace(void)
>>>> {
>>>> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void)
>>>> trace_create_file("kprobe_profile", TRACE_MODE_READ,
>>>> NULL, NULL, &kprobe_profile_ops);
>>>>
>>>> - setup_boot_kprobe_events();
>>>> + if (trace_init_wq) {
>>>> + INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func);
>>>> + queue_work(trace_init_wq, &kprobe_trace_work);
>>> Hmm, this queue_work is not required if kprobe_boot_events_buf[] is
>>> empty. We should check it because most of the time we don't need it.
>> Yes, I will improve it in the next version.
>>> Also, deferring initialization makes it indeterminate when this
>>> tracing will begin.
>> Indeed, While most scenarios don't need boot-time tracing, and users
>> prioritize boot speed, we must balance the need for deterministic traces
>> with faster startup.
> I just wonder why don't you define kprobe events after boot (e.g.
> from init script) instead of kernel cmdline. Using cmdline means
> it will be used for tracing kernel boot.
>
> - tracing kernel boot -> use kernel cmdline (synchronous)
> - tracing user boot -> use tracefs (asynchronous)
I was actually working on a boot optimization task when I found that my
machine was getting blocked on trace-related operations.
>
>>> For kprobe event use case, I think setup_boot_kprobe_events() should
>>> check the kprobe_boot_events_buf is empty at first. But I think Yaxiong
>>> use case happens when you are using kprobe events via cmdline, is that
>>> correct?
>> The issue was identified without enabling kprobe events via the cmdline.
> Interesting. So is it fixed by another patch [1]?
>
> [1] https://lore.kernel.org/all/20260127053848.108473-1-sunliming@linux.dev/
>
Yes, specifically looking at trace_kprobe in isolation, that issue is
indeed resolved. However, the process still ends up blocking the
initialization of blktrace later on.
>> The core finding is that asynchronous initialization can drastically cut
>> boot time under different workloads. As blocking occurs elsewhere in the
>> tracing infrastructure beyond just kprobe events, adopting async is a
>> broadly applicable strategy for boot time optimization.
> Yes, but it is also possible to set it up from user space, because that
> user process can work asynchronously.
> We can make the ftrace initialization async to accelerate boot time, but
> that means it is not useful for tracing kernel boot.
>
>>> I think introducing "async" cmdline option is more preferable.
>> Agreed, this works. Users focused on boot speed over early-boot tracing
>> can opt for this parameter to gain a startup performance boost.
> Yeah, that is an option. Anyway, basically, users have another option to
> setup ftrace after boot user space asynchronously. That is my
> recommendation for such use case.
Yes, that's what I normally do.
>
>>> BTW, I found that the kprobe events from kernel cmdline will be made
>>> after boot-time tracing from bootconfig. Maybe it should be run this
>>> earlier timing too.
>> Yes. Additionally, this optimization does not conflict with the current
>> patch series at all.
>>
>> I'll submit the updated patch for the next version promptly.
> OK.
>
> Thank you,
>
>>> Thank you,
>>>
>>>
>>>> + } else {
>>>> + setup_boot_kprobe_events();
>>>> + }
>>>>
>>>> return 0;
>>>> }
>>>> --
>>>> 2.25.1
>>>>
>
© 2016 - 2026 Red Hat, Inc.