From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A676212EFD for ; Fri, 4 Oct 2024 13:56:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=FBjLcs6Q2CZ1C3yt+1QkEZtPcFIMnEHZ7zOJYhVX0F0o62TF6ec+P7Th56J3I3arFCEOxrD0PhVmGYkCb6RwelVol4IYWQ9RYw4/MgG94f1iyjqHCA1uGqZt6d7cGJysJJERt2LSbcsK9nPc+dbzT2wEHErP///NZqaHsfJlJLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=gUyHsly5W4oj6s1eQx07m1wwA5qvoBZADocdlOMG69A=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=YoX44M2ALmZblJBV+SrlWZBsdjrVWj9F/J5gSb7ohrXRdU/zmRaacm7naCWaY3Ho4GXIrz03NXXY/7ApiXfYj2GAkqbdnWwhcIOqr1mr6x4/HjjhW5dA4dgmvoFW4x7wbKFx7Yuk4HflXOL8cf8JtlS7+eARWqmEcXNnwcV4PP0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id D0DFFC4CECE; Fri, 4 Oct 2024 13:56:26 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio7-00000005C86-1De2; Fri, 04 Oct 2024 09:57:23 -0400 Message-ID: <20241004135723.151356550@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:56:56 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Alison Schofield Subject: [for-linus][PATCH 1/8] tracing: Fix trace_check_vprintf() when tp_printk is used References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt When the tp_printk kernel command line is used, the trace events go directly to printk(). It is still checked via the trace_check_vprintf() function to make sure the pointers of the trace event are legit. The addition of reading buffers from previous boots required adding a delta between the addresses of the previous boot and the current boot so that the pointers in the old buffer can still be used. But this required adding a trace_array pointer to acquire the delta offsets. The tp_printk code does not provide a trace_array (tr) pointer, so when the offsets were examined, a NULL pointer dereference happened and the kernel crashed. If the trace_array does not exist, just default the delta offsets to zero, as that also means the trace event is not being read from a previous boot. Link: https://lore.kernel.org/all/Zv3z5UsG_jsO9_Tb@aschofie-mobl2.lan/ Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Link: https://lore.kernel.org/20241003104925.4e1b1fd9@gandalf.local.home Fixes: 07714b4bb3f98 ("tracing: Handle old buffer mappings for event string= s and functions") Reported-by: Alison Schofield Tested-by: Alison Schofield Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index c01375adc471..1c69ca1f1088 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -3697,8 +3697,8 @@ static void test_can_verify(void) void trace_check_vprintf(struct trace_iterator *iter, const char *fmt, va_list ap) { - long text_delta =3D iter->tr->text_delta; - long data_delta =3D iter->tr->data_delta; + long text_delta =3D 0; + long data_delta =3D 0; const char *p =3D fmt; const char *str; bool good; @@ -3710,6 +3710,17 @@ void trace_check_vprintf(struct trace_iterator *iter= , const char *fmt, if (static_branch_unlikely(&trace_no_verify)) goto print; =20 + /* + * When the kernel is booted with the tp_printk command line + * parameter, trace events go directly through to printk(). + * It also is checked by this function, but it does not + * have an associated trace_array (tr) for it. + */ + if (iter->tr) { + text_delta =3D iter->tr->text_delta; + data_delta =3D iter->tr->data_delta; + } + /* Don't bother checking when doing a ftrace_dump() */ if (iter->fmt =3D=3D static_fmt_buf) goto print; --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B75E212EFF for ; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=mJzTh0TuU/eesYulpG+AYBJBbpVNm2azzf1giVEwNTkcc3PrC5tyL/D1MeLfoj6hWgJoHIj+fpwIMINHLqLfHes2x58nv5LCZUQJk2pQZesAOQXS4tbunTz0u1RGMMZ2lK1CHcPAz5OvlF8omiDjIKAq4V0E5TaJvtkvo/yJiuE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=781C09YPnsjkINW3XIBNRX3vCRYJ04sGUh0a6jwoKjE=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=mE+b1yW87AKYaVbrXSnb7NbJjIQ6QIs5S2Kkwl6e2fez9wvRV5kyHHqxIPGYcbtzFiYUwR7n2rh9Y61A73/W/10ot+AZMBsQKiJvEL2+J0Le7rLTTy6tqZTcry7DqnKpwr2V7uGowHt0F67ta1HO2P0sz5PIpwYR2tDMK5+4d7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C16BC4CED1; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio7-00000005C8a-1t6m; Fri, 04 Oct 2024 09:57:23 -0400 Message-ID: <20241004135723.311401384@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:56:57 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Tomas Glozar , Ben Hutchings Subject: [for-linus][PATCH 2/8] tools/rtla: Fix installation from out-of-tree build References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ben Hutchings rtla now supports out-of-tree builds, but installation fails as it still tries to install the rtla binary from the source tree. Use the existing macro $(RTLA) to refer to the binary. Link: https://lore.kernel.org/ZudubuoU_JHjPZ7w@decadent.org.uk Fixes: 01474dc706ca ("tools/rtla: Use tools/build makefiles to build rtla") Reviewed-by: Tomas Glozar Tested-by: Tomas Glozar Signed-off-by: Ben Hutchings Signed-off-by: Steven Rostedt (Google) --- tools/tracing/rtla/Makefile.rtla | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/tracing/rtla/Makefile.rtla b/tools/tracing/rtla/Makefile= .rtla index 3ff0b8970896..cc1d6b615475 100644 --- a/tools/tracing/rtla/Makefile.rtla +++ b/tools/tracing/rtla/Makefile.rtla @@ -38,7 +38,7 @@ BINDIR :=3D /usr/bin .PHONY: install install: doc_install @$(MKDIR) -p $(DESTDIR)$(BINDIR) - $(call QUIET_INSTALL,rtla)$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) + $(call QUIET_INSTALL,rtla)$(INSTALL) $(RTLA) -m 755 $(DESTDIR)$(BINDIR) @$(STRIP) $(DESTDIR)$(BINDIR)/rtla @test ! -f $(DESTDIR)$(BINDIR)/osnoise || $(RM) $(DESTDIR)$(BINDIR)/osnoi= se @$(LN) rtla $(DESTDIR)$(BINDIR)/osnoise --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 828AE212F1E; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=Hs/pfo1xBP6cRi0YnW7tg9SZ5/bIax1iqt8nU57efVlaE0Srr8Gdl7PuoKgFEzkZesNEzY9g3bSECAT8lmr0Qkb+sHlbLUB3+B8RNv8n9q+LmcExCrp4LC/kcsoAoFmjnVGyy444Sfb6aDHsbrs2+7+39ld9gKLdwLZDx4imYV8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=kE5N9LMzZ5xDIVfqC2ZVNeflYoiHF/zPTxu7bKcTfKA=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=OVoZ48NddykV62lXnGe3Rqixi+71wIHpNRcLZs4Kq7THu/EH2FxeF8oFCtDrI7j7flPVGPbCuoT9johIOBclNmTLASKUGjINCd8r143jyEZu1s3r8HENDSnV9WghicpgHV+gOh0xsYjcR+OPlaxJew7R0BQ41XLSRavm4G0LLLg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 344F0C4CEC7; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio7-00000005C94-2aO5; Fri, 04 Oct 2024 09:57:23 -0400 Message-ID: <20241004135723.471955359@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:56:58 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , stable@vger.kernel.org, Tomas Glozar , Eder Zulian Subject: [for-linus][PATCH 3/8] rtla: Fix the help text in osnoise and timerlat top tools References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eder Zulian The help text in osnoise top and timerlat top had some minor errors and omissions. The -d option was missing the 's' (second) abbreviation and the error message for '-d' used '-D'. Cc: stable@vger.kernel.org Fixes: 1eceb2fc2ca54 ("rtla/osnoise: Add osnoise top mode") Fixes: a828cd18bc4ad ("rtla: Add timerlat tool and timelart top mode") Link: https://lore.kernel.org/20240813155831.384446-1-ezulian@redhat.com Suggested-by: Tomas Glozar Reviewed-by: Tomas Glozar Signed-off-by: Eder Zulian Signed-off-by: Steven Rostedt (Google) --- tools/tracing/rtla/src/osnoise_top.c | 2 +- tools/tracing/rtla/src/timerlat_top.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/= osnoise_top.c index 2f756628613d..30e3853076a0 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -442,7 +442,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int a= rgc, char **argv) case 'd': params->duration =3D parse_seconds_duration(optarg); if (!params->duration) - osnoise_top_usage(params, "Invalid -D duration\n"); + osnoise_top_usage(params, "Invalid -d duration\n"); break; case 'e': tevent =3D trace_event_alloc(optarg); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 8c16419fe22a..210b0f533534 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -459,7 +459,7 @@ static void timerlat_top_usage(char *usage) " -c/--cpus cpus: run the tracer only on the given cpus", " -H/--house-keeping cpus: run rtla control threads only on the given = cpus", " -C/--cgroup[=3Dcgroup_name]: set cgroup, if no cgroup_name is passed= , the rtla's cgroup will be inherited", - " -d/--duration time[m|h|d]: duration of the session in seconds", + " -d/--duration time[s|m|h|d]: duration of the session", " -D/--debug: print debug info", " --dump-tasks: prints the task running on all CPUs if stop conditi= ons are met (depends on !--no-aa)", " -t/--trace[file]: save the stopped trace to [file|timerlat_trace.txt= ]", @@ -613,7 +613,7 @@ static struct timerlat_top_params case 'd': params->duration =3D parse_seconds_duration(optarg); if (!params->duration) - timerlat_top_usage("Invalid -D duration\n"); + timerlat_top_usage("Invalid -d duration\n"); break; case 'e': tevent =3D trace_event_alloc(optarg); --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FBF12139A7 for ; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=rqsmiiOqTgEY3ITULHaE+a3tSiObP7Ncq71xCSuYqxG39D6dDpfaUV+qmOK75NtfaONKXXyxWncowQuEle61xSHM+PPNUxC+i+6pHyzY6RsPyiIwdgVHMGKvFu8+kkX44ldSEv1/+WYBbexkfhqMQmG0YO+q3JJ7LmJiR7Ott0w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=W5gg0bmYi+cHGHIRHDi7OUpL3uRLaizOzsk2Cu9nxag=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=neKGSdKBzSTtY46/IxZ+arlmHLlzaDWn8aIF2IeqenFbNtCheQ+NrfE/gCZ7yXRLw1b4Ifsb2UQ4PVmF+nc9NL2qb537S/O5O7qM0QWZR1fbWxGyljgWh/t2lIBNQp9B61NyevBJMEbvKv+pcz39kHbvIYlX4RUH0DhydMk2/NY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4D12DC4CED2; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio7-00000005C9Y-3Gia; Fri, 04 Oct 2024 09:57:23 -0400 Message-ID: <20241004135723.638567806@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:56:59 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Masahiro Yamada , Sami Tolvanen Subject: [for-linus][PATCH 4/8] x86/ftrace: Include References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Sami Tolvanen uses struct pt_regs in several places. Include to ensure it's visible. This is needed to make sure object files that only include compile. Cc: Mark Rutland Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Link: https://lore.kernel.org/20240916221557.846853-2-samitolvanen@google.c= om Suggested-by: Masahiro Yamada Signed-off-by: Sami Tolvanen Acked-by: Masami Hiramatsu (Google) Signed-off-by: Steven Rostedt (Google) --- arch/x86/include/asm/ftrace.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 0152a81d9b4a..b4d719de2c84 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -2,6 +2,8 @@ #ifndef _ASM_X86_FTRACE_H #define _ASM_X86_FTRACE_H =20 +#include + #ifdef CONFIG_FUNCTION_TRACER #ifndef CC_USING_FENTRY # error Compiler does not support fentry? --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B16C02139C8; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=fxhNu1IhYzk9U6sIV1D9SJtv4ckg943Zy+C0vLHhZaV1URXGdNuZPgAhGJCW9J3rp7hryyEGeJ6YrTUgGNm/5yCkFywtXNSWcpE7vKeywdogUJCu7xcc7DokY20fE9lGkT8yam1vyEU9mmFezkOqZ6eZE4uK56WNvvLZ9kXfulw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=k8cXzyqbg2s7qf7ZrUuCUbyXUbR4GVpJT/q3XIstq7o=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=IYb+jgtXxee+Ht2sMm6Y+rA9kvClxk5aFIh5Rk9MszB2ZUrPQqOh/6U39zgUaivsj5lOhbDdoL2afcLjpie2raqvdjSmuWG9RtbacwxNdhLOdtiSpZR/LxG7nDuzSA/tHeeRuuvVc8yq1Oq+pO3i++av4XJACnaO7T5PVIZL14Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8632DC4CECF; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio7-00000005CA2-3vqH; Fri, 04 Oct 2024 09:57:23 -0400 Message-ID: <20241004135723.800438615@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:57:00 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , stable@vger.kernel.org, Wei Li Subject: [for-linus][PATCH 5/8] tracing/timerlat: Fix duplicated kthread creation due to CPU online/offline References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Wei Li osnoise_hotplug_workfn() is the asynchronous online callback for "trace/osnoise:online". It may be congested when a CPU goes online and offline repeatedly and is invoked for multiple times after a certain online. This will lead to kthread leak and timer corruption. Add a check in start_kthread() to prevent this situation. Cc: stable@vger.kernel.org Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Link: https://lore.kernel.org/20240924094515.3561410-2-liwei391@huawei.com Fixes: c8895e271f79 ("trace/osnoise: Support hotplug operations") Signed-off-by: Wei Li Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_osnoise.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 1439064f65d6..d1a539913a5f 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -2007,6 +2007,10 @@ static int start_kthread(unsigned int cpu) void *main =3D osnoise_main; char comm[24]; =20 + /* Do not start a new thread if it is already running */ + if (per_cpu(per_cpu_osnoise_var, cpu).kthread) + return 0; + if (timerlat_enabled()) { snprintf(comm, 24, "timerlat/%d", cpu); main =3D timerlat_main; @@ -2061,11 +2065,10 @@ static int start_per_cpu_kthreads(void) if (cpumask_test_and_clear_cpu(cpu, &kthread_cpumask)) { struct task_struct *kthread; =20 - kthread =3D per_cpu(per_cpu_osnoise_var, cpu).kthread; + kthread =3D xchg_relaxed(&(per_cpu(per_cpu_osnoise_var, cpu).kthread), = NULL); if (!WARN_ON(!kthread)) kthread_stop(kthread); } - per_cpu(per_cpu_osnoise_var, cpu).kthread =3D NULL; } =20 for_each_cpu(cpu, current_mask) { --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4DCE2141D0; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=fAN+vIYdtUeS6XADzU3tarSemeFBtv/B74ASjp+zy2SSUb8gbWMKQUDV6eceITl+wWuSAlq95P/ojat1rcH3n1tMlcZBUySV1+jKJ3LJ5D9u8DjoZcWlh8hPJ83vEzzkdYNPwfY8qV7dEN00W4GRtDgdU7wH4oTGF5+rdsRaDC8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=Vlvy5l4fmOohbA20KkO/nfnBsMBTpT7pZhN5CpbYE9U=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=EumchhNVOl6QUuwzGveG39Ydxwhd5ao/bh1Y+1nU7VWxz4g+P7blxzJ8XYLB3oAEY2MlFQs32XzQppTUIzy1qmrWm3AJILq+R43BB2zUKVQsciCVh6Ol1WdQ2S0eTzV52xuVFcF/XZuEFBnF6HGuhkMfAes8k0QtdsUVuavHCS0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5710C4CED9; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio8-00000005CAW-0PsF; Fri, 04 Oct 2024 09:57:24 -0400 Message-ID: <20241004135723.958450678@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:57:01 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , stable@vger.kernel.org, Wei Li Subject: [for-linus][PATCH 6/8] tracing/timerlat: Drop interface_lock in stop_kthread() References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Wei Li stop_kthread() is the offline callback for "trace/osnoise:online", since commit 5bfbcd1ee57b ("tracing/timerlat: Add interface_lock around clearing of kthread in stop_kthread()"), the following ABBA deadlock scenario is introduced: T1 | T2 [BP] | T3 [AP] osnoise_hotplug_workfn() | work_for_cpu_fn() | cpuhp_thread_fun() | _cpu_down() | osnoise_cpu_die() mutex_lock(&interface_lock) | | stop_kthread() | cpus_write_lock() | mutex_lock(&i= nterface_lock) cpus_read_lock() | cpuhp_kick_ap() | As the interface_lock here in just for protecting the "kthread" field of the osn_var, use xchg() instead to fix this issue. Also use for_each_online_cpu() back in stop_per_cpu_kthreads() as it can take cpu_read_lock() again. Cc: stable@vger.kernel.org Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Link: https://lore.kernel.org/20240924094515.3561410-3-liwei391@huawei.com Fixes: 5bfbcd1ee57b ("tracing/timerlat: Add interface_lock around clearing = of kthread in stop_kthread()") Signed-off-by: Wei Li Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_osnoise.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index d1a539913a5f..e22567174dd3 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1953,12 +1953,8 @@ static void stop_kthread(unsigned int cpu) { struct task_struct *kthread; =20 - mutex_lock(&interface_lock); - kthread =3D per_cpu(per_cpu_osnoise_var, cpu).kthread; + kthread =3D xchg_relaxed(&(per_cpu(per_cpu_osnoise_var, cpu).kthread), NU= LL); if (kthread) { - per_cpu(per_cpu_osnoise_var, cpu).kthread =3D NULL; - mutex_unlock(&interface_lock); - if (cpumask_test_and_clear_cpu(cpu, &kthread_cpumask) && !WARN_ON(!test_bit(OSN_WORKLOAD, &osnoise_options))) { kthread_stop(kthread); @@ -1972,7 +1968,6 @@ static void stop_kthread(unsigned int cpu) put_task_struct(kthread); } } else { - mutex_unlock(&interface_lock); /* if no workload, just return */ if (!test_bit(OSN_WORKLOAD, &osnoise_options)) { /* @@ -1994,8 +1989,12 @@ static void stop_per_cpu_kthreads(void) { int cpu; =20 - for_each_possible_cpu(cpu) + cpus_read_lock(); + + for_each_online_cpu(cpu) stop_kthread(cpu); + + cpus_read_unlock(); } =20 /* --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE46C215F43; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; cv=none; b=VoFU4u4RDENJ5JPopbyrhxZFigTwH4aTxOsvsoS5TfbKGgryPrp9wSBLH4JfwyZyMDzXQUfBiAaZawjpulnnyRILSgv0VhzYY2CphQE0H3dEVdLGWJkahUNaiftiz3wLlinInsTzR4mXoeMD9BbTKwJQa/+jaR48BHDEUKIDDHQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050187; c=relaxed/simple; bh=c9y7dUf6LzrNr2Vv1tlqVvqf+vP4CIexsGEoNQB3WTQ=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=FjdoYEm1DcmMKaxx+EImKTFj/cgXVcPyZoxdvNJlnvtx122KYeZW/Epn6jJAg7bFB2Mr1CcgYDps63cv7FxTwD+9yqFQFn+ktC+cEKGzSeOG7U8NEQ/C9aQZcnZu0LM/4M1wB9SnFORkYJkFtJFytYZ/AkCo8eArXArNZgZwRPg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC0B3C4CECE; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio8-00000005CB0-1632; Fri, 04 Oct 2024 09:57:24 -0400 Message-ID: <20241004135724.120503644@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:57:02 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , stable@vger.kernel.org, Wei Li Subject: [for-linus][PATCH 7/8] tracing/timerlat: Fix a race during cpuhp processing References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Wei Li There is another found exception that the "timerlat/1" thread was scheduled on CPU0, and lead to timer corruption finally: ``` ODEBUG: init active (active state 0) object: ffff888237c2e108 object type: = hrtimer hint: timerlat_irq+0x0/0x220 WARNING: CPU: 0 PID: 426 at lib/debugobjects.c:518 debug_print_object+0x7d/= 0xb0 Modules linked in: CPU: 0 UID: 0 PID: 426 Comm: timerlat/1 Not tainted 6.11.0-rc7+ #45 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1= .1 04/01/2014 RIP: 0010:debug_print_object+0x7d/0xb0 ... Call Trace: ? __warn+0x7c/0x110 ? debug_print_object+0x7d/0xb0 ? report_bug+0xf1/0x1d0 ? prb_read_valid+0x17/0x20 ? handle_bug+0x3f/0x70 ? exc_invalid_op+0x13/0x60 ? asm_exc_invalid_op+0x16/0x20 ? debug_print_object+0x7d/0xb0 ? debug_print_object+0x7d/0xb0 ? __pfx_timerlat_irq+0x10/0x10 __debug_object_init+0x110/0x150 hrtimer_init+0x1d/0x60 timerlat_main+0xab/0x2d0 ? __pfx_timerlat_main+0x10/0x10 kthread+0xb7/0xe0 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2d/0x40 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 ``` After tracing the scheduling event, it was discovered that the migration of the "timerlat/1" thread was performed during thread creation. Further analysis confirmed that it is because the CPU online processing for osnoise is implemented through workers, which is asynchronous with the offline processing. When the worker was scheduled to create a thread, the CPU may has already been removed from the cpu_online_mask during the offline process, resulting in the inability to select the right CPU: T1 | T2 [CPUHP_ONLINE] | cpu_device_down() osnoise_hotplug_workfn() | | cpus_write_lock() | takedown_cpu(1) | cpus_write_unlock() [CPUHP_OFFLINE] | cpus_read_lock() | start_kthread(1) | cpus_read_unlock() | To fix this, skip online processing if the CPU is already offline. Cc: stable@vger.kernel.org Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Link: https://lore.kernel.org/20240924094515.3561410-4-liwei391@huawei.com Fixes: c8895e271f79 ("trace/osnoise: Support hotplug operations") Signed-off-by: Wei Li Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_osnoise.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index e22567174dd3..a50ed23bee77 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -2097,6 +2097,8 @@ static void osnoise_hotplug_workfn(struct work_struct= *dummy) mutex_lock(&interface_lock); cpus_read_lock(); =20 + if (!cpu_online(cpu)) + goto out_unlock; if (!cpumask_test_cpu(cpu, &osnoise_cpumask)) goto out_unlock; =20 --=20 2.45.2 From nobody Thu Nov 28 04:43:57 2024 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35169215F79; Fri, 4 Oct 2024 13:56:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050188; cv=none; b=LFtk5rkaSEIzJuCEev7QnX1+zvjUBI2vxQf0sJGWOJWgOUXPMfi7U0blrvi/FbzXM7dMurvWLRbPUvSjUNN67JsnyO8lvWjVQsR7hLgk8ugdXxb+RV0xVEDp9+UCx2Zarew7jhluG1fjgo3XoszV90JHum5Ih/652B5v0IKYfAA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728050188; c=relaxed/simple; bh=iJWUfcBOjvKfwKKrTHNMffbCW4LAcVM3302EPSIDozA=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=fa6Z8EviQXJgKs/gMYqXpxCFOvx8UHtWWEUqZKBRTPUXs7oWxlhkpHYYu8AeM64hWIY3k4UoqNQ+43PQWQ0jOGcVhSzVA0IgmqJzT9R7Hnh/41gyyxJGwkAEWQzb0gMM5niOqz1ShW5NT74hRpwvxYot5dLdvP8SL8wi6aGKads= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id E528CC4CED9; Fri, 4 Oct 2024 13:56:27 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.98) (envelope-from ) id 1swio8-00000005CBW-1kGh; Fri, 04 Oct 2024 09:57:24 -0400 Message-ID: <20241004135724.280243457@goodmis.org> User-Agent: quilt/0.68 Date: Fri, 04 Oct 2024 09:57:03 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , stable@vger.kernel.org, Wei Li Subject: [for-linus][PATCH 8/8] tracing/hwlat: Fix a race during cpuhp processing References: <20241004135655.993267242@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Wei Li The cpuhp online/offline processing race also exists in percpu-mode hwlat tracer in theory, apply the fix too. That is: T1 | T2 [CPUHP_ONLINE] | cpu_device_down() hwlat_hotplug_workfn() | | cpus_write_lock() | takedown_cpu(1) | cpus_write_unlock() [CPUHP_OFFLINE] | cpus_read_lock() | start_kthread(1) | cpus_read_unlock() | Cc: stable@vger.kernel.org Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Link: https://lore.kernel.org/20240924094515.3561410-5-liwei391@huawei.com Fixes: ba998f7d9531 ("trace/hwlat: Support hotplug operations") Signed-off-by: Wei Li Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_hwlat.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c index b791524a6536..3bd6071441ad 100644 --- a/kernel/trace/trace_hwlat.c +++ b/kernel/trace/trace_hwlat.c @@ -520,6 +520,8 @@ static void hwlat_hotplug_workfn(struct work_struct *du= mmy) if (!hwlat_busy || hwlat_data.thread_mode !=3D MODE_PER_CPU) goto out_unlock; =20 + if (!cpu_online(cpu)) + goto out_unlock; if (!cpumask_test_cpu(cpu, tr->tracing_cpumask)) goto out_unlock; =20 --=20 2.45.2