tracing: zero the pipe cpumask on alloc to avoid spurious -EBUSY

[PATCH] tracing: zero the pipe cpumask on alloc to avoid spurious -EBUSY

Posted by Brian Foster 2 years, 3 months ago

The pipe cpumask used to serialize opens between the main and percpu
trace pipes is not zeroed or initialized. This can result in
spurious -EBUSY returns if underlying memory is not fully zeroed.
This has been observed by immediate failure to read the main
trace_pipe file on an otherwise newly booted and idle system:

 # cat /sys/kernel/debug/tracing/trace_pipe
 cat: /sys/kernel/debug/tracing/trace_pipe: Device or resource busy

Zero the allocation of pipe_cpumask to avoid the problem.

Fixes: c2489bb7e6be ("tracing: Introduce pipe_cpumask to avoid race on trace_pipes")
Signed-off-by: Brian Foster <bfoster@redhat.com>
---

Hi,

I ran into this problem just recently on one of my test VMs immediately
after updating to a v6.5 base. A revert of the aforementioned commit
addressed the problem. I'm not terribly familiar with the tracing code,
but on further inspection I noticed the cpumask doesn't appear to be
initialized anywhere. I suppose this could alternatively do a
cpumask_clear() or whatever after allocation, but either way this
addresses the problem for me.

Please CC on replies as I'm not subscribed to the list. Thanks.

Brian

 kernel/trace/trace.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 8e64aaad5361..2656ca3b9b39 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -9486,7 +9486,7 @@ static struct trace_array *trace_array_create(const char *name)
 	if (!alloc_cpumask_var(&tr->tracing_cpumask, GFP_KERNEL))
 		goto out_free_tr;
 
-	if (!alloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL))
+	if (!zalloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL))
 		goto out_free_tr;
 
 	tr->trace_flags = global_trace.trace_flags & ~ZEROED_TRACE_FLAGS;
@@ -10431,7 +10431,7 @@ __init static int tracer_alloc_buffers(void)
 	if (trace_create_savedcmd() < 0)
 		goto out_free_temp_buffer;
 
-	if (!alloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL))
+	if (!zalloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL))
 		goto out_free_savedcmd;
 
 	/* TODO: make the number of buffers hot pluggable with CPUS */
-- 
2.41.0

Re: [PATCH] tracing: zero the pipe cpumask on alloc to avoid spurious -EBUSY

Posted by Zheng Yejian 2 years, 3 months ago

On 2023/8/31 20:55, Brian Foster wrote:
> The pipe cpumask used to serialize opens between the main and percpu
> trace pipes is not zeroed or initialized. This can result in
> spurious -EBUSY returns if underlying memory is not fully zeroed.
> This has been observed by immediate failure to read the main
> trace_pipe file on an otherwise newly booted and idle system:
> 
>   # cat /sys/kernel/debug/tracing/trace_pipe
>   cat: /sys/kernel/debug/tracing/trace_pipe: Device or resource busy
> 
> Zero the allocation of pipe_cpumask to avoid the problem.
> 
> Fixes: c2489bb7e6be ("tracing: Introduce pipe_cpumask to avoid race on trace_pipes")
> Signed-off-by: Brian Foster <bfoster@redhat.com>
> ---
> 
> Hi,
> 
> I ran into this problem just recently on one of my test VMs immediately
> after updating to a v6.5 base. A revert of the aforementioned commit
> addressed the problem. I'm not terribly familiar with the tracing code,
> but on further inspection I noticed the cpumask doesn't appear to be
> initialized anywhere. I suppose this could alternatively do a
> cpumask_clear() or whatever after allocation, but either way this
> addresses the problem for me.

Yes, pipe_cpumask must be initialized.

--

Thanks,
Zheng Yejian

> 
> Please CC on replies as I'm not subscribed to the list. Thanks.
> 
> Brian
> 
>   kernel/trace/trace.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 8e64aaad5361..2656ca3b9b39 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -9486,7 +9486,7 @@ static struct trace_array *trace_array_create(const char *name)
>   	if (!alloc_cpumask_var(&tr->tracing_cpumask, GFP_KERNEL))
>   		goto out_free_tr;
>   
> -	if (!alloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL))
> +	if (!zalloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL))
>   		goto out_free_tr;
>   
>   	tr->trace_flags = global_trace.trace_flags & ~ZEROED_TRACE_FLAGS;
> @@ -10431,7 +10431,7 @@ __init static int tracer_alloc_buffers(void)
>   	if (trace_create_savedcmd() < 0)
>   		goto out_free_temp_buffer;
>   
> -	if (!alloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL))
> +	if (!zalloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL))
>   		goto out_free_savedcmd;
>   
>   	/* TODO: make the number of buffers hot pluggable with CPUS */

Re: [PATCH] tracing: zero the pipe cpumask on alloc to avoid spurious -EBUSY

Posted by Steven Rostedt 2 years, 3 months ago

On Thu, 31 Aug 2023 21:51:18 +0800
Zheng Yejian <zhengyejian1@huawei.com> wrote:

> > Hi,
> > 
> > I ran into this problem just recently on one of my test VMs immediately
> > after updating to a v6.5 base. A revert of the aforementioned commit
> > addressed the problem. I'm not terribly familiar with the tracing code,
> > but on further inspection I noticed the cpumask doesn't appear to be
> > initialized anywhere. I suppose this could alternatively do a
> > cpumask_clear() or whatever after allocation, but either way this
> > addresses the problem for me.  
> 
> Yes, pipe_cpumask must be initialized.

Can I add a Reviewed-by tag from you?

> 
> > 
> > Please CC on replies as I'm not subscribed to the list. Thanks.

That's the default with Linux kernel lists.

-- Steve

Re: [PATCH] tracing: zero the pipe cpumask on alloc to avoid spurious -EBUSY

Posted by Zheng Yejian 2 years, 3 months ago

On 2023/9/1 04:33, Steven Rostedt wrote:
> On Thu, 31 Aug 2023 21:51:18 +0800
> Zheng Yejian <zhengyejian1@huawei.com> wrote:
> 
>>> Hi,
>>>
>>> I ran into this problem just recently on one of my test VMs immediately
>>> after updating to a v6.5 base. A revert of the aforementioned commit
>>> addressed the problem. I'm not terribly familiar with the tracing code,
>>> but on further inspection I noticed the cpumask doesn't appear to be
>>> initialized anywhere. I suppose this could alternatively do a
>>> cpumask_clear() or whatever after allocation, but either way this
>>> addresses the problem for me.
>>
>> Yes, pipe_cpumask must be initialized.
> 
> Can I add a Reviewed-by tag from you?

Of course :)
Reviewed-by: Zheng Yejian <zhengyejian1@huawei.com>

--

Thanks,
Zheng Yejian

> 
>>
>>>
>>> Please CC on replies as I'm not subscribed to the list. Thanks.
> 
> That's the default with Linux kernel lists.
> 
> -- Steve
>