From nobody Fri Sep 19 08:29:24 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92252C4167B for ; Fri, 25 Nov 2022 21:20:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230036AbiKYVUk (ORCPT ); Fri, 25 Nov 2022 16:20:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229495AbiKYVUi (ORCPT ); Fri, 25 Nov 2022 16:20:38 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AB6F2F3A5; Fri, 25 Nov 2022 13:20:37 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DC609B82C23; Fri, 25 Nov 2022 21:20:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7CB5EC433D7; Fri, 25 Nov 2022 21:20:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669411234; bh=UofQ2Om5TKKPZw19SG+8wKH8TntN+ojXWrOiFhpkAis=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VuAbw/1nYj19+PgmBb5Y3Tlxzw3F5c8blP52lPX36dfvLIrNBuvnd6VvVcJnWuEjz eFFoRQPvsI4BTMUeG89I6iBWGnL32xONihx/6HnJM8GLhbYi6Flt0t0DL/NbprIc+q 1AvJtEaBq+hwwJElK64wfyqyTJRik+TSted6iGwgv2y9htPBpp6vMLJs2+IKf+ZL0Z uUs8o3E3BN+0ELn+eBBxe2xV6w9iaYDS4udWHqLC2sqeGVy7ERe+I8vMwMK7h9qJOg Jd/OMx8G89ciDSgpTpmeJg3JJmzRz0QcJe9KZutZtDw3k4greXC5SkGAWGxUy7K3xn u5lfMBvzHhApw== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V3 1/3] tracing/osnoise: Add PANIC_ON_STOP option Date: Fri, 25 Nov 2022 22:20:22 +0100 Message-Id: <31f67f8f5ff4bc4e1549b34411f918423abcaa78.1669409262.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Often the latency observed in a CPU is not caused by the work being done in the CPU itself, but by work done on another CPU that causes the hardware to stall all CPUs. In this case, it is interesting to know what is happening on ALL CPUs, and the best way to do this is via crash dump analysis. Add the PANIC_ON_STOP option to osnoise/timerlat tracers. The default behavior is having this option off. When enabled by the user, the system will panic after hitting a stop tracing condition. This option was motivated by a real scenario that Juri Lelli and I were debugging. Cc: Juri Lelli Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 3f10dd1f2f1c..801eba0b5cf8 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -54,10 +54,11 @@ enum osnoise_options_index { OSN_DEFAULTS =3D 0, OSN_WORKLOAD, + OSN_PANIC_ON_STOP, OSN_MAX }; =20 -static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD" }; +static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD", "PANIC_ON_STOP" }; =20 #define OSN_DEFAULT_OPTIONS 0x2 unsigned long osnoise_options =3D OSN_DEFAULT_OPTIONS; @@ -1270,6 +1271,9 @@ static __always_inline void osnoise_stop_tracing(void) trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, "stop tracing hit on cpu %d\n", smp_processor_id()); =20 + if (test_bit(OSN_PANIC_ON_STOP, &osnoise_options)) + panic("tracer hit stop condition on CPU %d\n", smp_processor_id()); + tracer_tracing_off(tr); } rcu_read_unlock(); --=20 2.32.0 From nobody Fri Sep 19 08:29:24 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60600C4332F for ; Fri, 25 Nov 2022 21:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230047AbiKYVUl (ORCPT ); Fri, 25 Nov 2022 16:20:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230022AbiKYVUj (ORCPT ); Fri, 25 Nov 2022 16:20:39 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17D4A2FC3E; Fri, 25 Nov 2022 13:20:38 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9923260E9C; Fri, 25 Nov 2022 21:20:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08EDDC433B5; Fri, 25 Nov 2022 21:20:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669411237; bh=572LKrFlwGTVNo7fZjQcd1sMaO3bjlC22vJvxTRspSc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jto3bk6IN5/drHQa14B8tArBHRwMStZzLveA5oWqJWutgqe6xLUZuzupgugqT6tO5 9ZoTbLku59KK4o/YnWBdJpkTCfkZWTDvJxX8ewRR7Fzr+hdPZRNAsu3GOx2XXpgON6 mHvhFYyi75q66IWXMASAb19/+66ygG2MXESy8QP45qzq/c7fQB7fE+rNllFM+T1Ct9 rTxMbpa20T9bMAiEUWCb/VJxuY8yuabyt6GcZuAd6s4BFOmPh3+X1Fcxl7xP2gIgMz FyGbSHei3CFPfcr5nT11T689Clf9T9YgNlpZkrlnXG4O9D3odwLhIRiVdcF7lnUHRQ 45J4/VdSva2Wg== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V3 2/3] tracing/osnoise: Add preempt/irq disable options Date: Fri, 25 Nov 2022 22:20:23 +0100 Message-Id: <03d4a8522792fa3a51920c79f8a5074933a2fcb3.1669409262.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The osnoise workload runs with preemption and IRQs enabled in such a way as to allow all sorts of noise to disturb osnoise's execution. hwlat tracer has a similar workload but works with irq disabled, allowing only NMIs and the hardware to generate noise. While thinking about adding an options file to hwlat tracer to allow the system to panic, and other features I was thinking to add, like having a tracepoint at each noise detection, it came to my mind that is easier to make osnoise and also do hardware latency detection than making hwlat "feature compatible" with osnoise. Other points are: - osnoise already has an independent cpu file. - osnoise has a more intuitive interface, e.g., runtime/period vs. window/width (and people often need help remembering what it is). - osnoise: tracepoints - osnoise stop options - osnoise options file itself Moreover, the user-space side (in rtla) is simplified by reusing the existing osnoise code. Finally, people have been asking me about using osnoise for hw latency detection, and I have to explain that it was sufficient but not necessary. These options make it sufficient and necessary. Adding a Suggested-by Clark, as he often asked me about this possibility. Cc: Suggested-by: Clark Williams Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 40 +++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 801eba0b5cf8..14b7f4092982 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -55,10 +55,17 @@ enum osnoise_options_index { OSN_DEFAULTS =3D 0, OSN_WORKLOAD, OSN_PANIC_ON_STOP, + OSN_PREEMPT_DISABLE, + OSN_IRQ_DISABLE, OSN_MAX }; =20 -static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD", "PANIC_ON_STOP" }; +static const char * const osnoise_options_str[OSN_MAX] =3D { + "DEFAULTS", + "OSNOISE_WORKLOAD", + "PANIC_ON_STOP", + "OSNOISE_PREEMPT_DISABLE", + "OSNOISE_IRQ_DISABLE" }; =20 #define OSN_DEFAULT_OPTIONS 0x2 unsigned long osnoise_options =3D OSN_DEFAULT_OPTIONS; @@ -1308,6 +1315,8 @@ static void notify_new_max_latency(u64 latency) */ static int run_osnoise(void) { + bool preempt_disable =3D test_bit(OSN_PREEMPT_DISABLE, &osnoise_options); + bool irq_disable =3D test_bit(OSN_IRQ_DISABLE, &osnoise_options); struct osnoise_variables *osn_var =3D this_cpu_osn_var(); u64 start, sample, last_sample; u64 last_int_count, int_count; @@ -1335,6 +1344,14 @@ static int run_osnoise(void) */ threshold =3D tracing_thresh ? : 5000; =20 + /* + * IRQ disable also implies in preempt disable. + */ + if (irq_disable) + local_irq_disable(); + else if (preempt_disable) + preempt_disable(); + /* * Make sure NMIs see sampling first */ @@ -1422,16 +1439,21 @@ static int run_osnoise(void) * cond_resched() */ if (IS_ENABLED(CONFIG_PREEMPT_RCU)) { - local_irq_disable(); + if (!irq_disable) + local_irq_disable(); + rcu_momentary_dyntick_idle(); - local_irq_enable(); + + if (!irq_disable) + local_irq_enable(); } =20 /* * For the non-preemptive kernel config: let threads runs, if - * they so wish. + * they so wish, unless set not do to so. */ - cond_resched(); + if (!irq_disable && !preempt_disable) + cond_resched(); =20 last_sample =3D sample; last_int_count =3D int_count; @@ -1450,6 +1472,14 @@ static int run_osnoise(void) */ barrier(); =20 + /* + * Return to the preemptive state. + */ + if (irq_disable) + local_irq_enable(); + else if (preempt_disable) + preempt_enable(); + /* * Save noise info. */ --=20 2.32.0 From nobody Fri Sep 19 08:29:24 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA5ECC4167D for ; Fri, 25 Nov 2022 21:20:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230071AbiKYVUs (ORCPT ); Fri, 25 Nov 2022 16:20:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230022AbiKYVUn (ORCPT ); Fri, 25 Nov 2022 16:20:43 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DC0C442C5; Fri, 25 Nov 2022 13:20:42 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B6A0BB82AEF; Fri, 25 Nov 2022 21:20:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86710C433D6; Fri, 25 Nov 2022 21:20:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669411239; bh=fM3Z9ts7iQFlYVNcl8/0c3e3g8FbZxuJ3byH62UaVoI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ruupuHhKnNKrd+Mso6lNPvh33EUuONTam/VAUFnQkL+O8tbAyzVsOp/1sDpk9OSEP DI6IiCSrg5aRM9cuT5Jfo4qgrKtadbGxOuGJHF35yIxpgIblKn+UbI7EV5eahKVKjo 6b9C5+nj8zH/3gp0l6WOOLzM+KFT8kv6qdPQxeghL3WGAoUY+W11p+OnK0yacnHMBM 0fwijyXUHh5Lu43mVsZ6eUxuv99hS79K1/Jwn8WZD0LSmJJW69fQl0Bro2saEA+Xnj BPIqiOMa55/h+4dTDF93fsVGbwfEHompw+JvRgxrN9kOBGgTNCCgd3wz7iWDXfL0VS jfywj/dhuQIWQ== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V3 3/3] Documentation/osnoise: Add osnoise/options documentation Date: Fri, 25 Nov 2022 22:20:24 +0100 Message-Id: <0a8ecf364af29b0c1d5007f4d71265127c114a1f.1669409262.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add the documentation about the osnoise/options file, the options, and some additional explanation about the OSNOISE_WORKLOAD option. Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- Documentation/trace/osnoise-tracer.rst | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/Documentation/trace/osnoise-tracer.rst b/Documentation/trace/o= snoise-tracer.rst index 3c675ed82b27..0641781b00f5 100644 --- a/Documentation/trace/osnoise-tracer.rst +++ b/Documentation/trace/osnoise-tracer.rst @@ -92,8 +92,8 @@ Note that the example above shows a high number of HW noi= se samples. The reason being is that this sample was taken on a virtual machine, and the host interference is detected as a hardware interference. =20 -Tracer options ---------------------- +Tracer Configuration +-------------------- =20 The tracer has a set of options inside the osnoise directory, they are: =20 @@ -115,6 +115,22 @@ The tracer has a set of options inside the osnoise dir= ectory, they are: NO_OSNOISE_WORKLOAD disables the OSNOISE_WORKLOAD option. The special DEAFAULTS option resets all options to the default value. =20 +Tracer Options +-------------- + +The osnoise/options file exposes a set of on/off configuration options for +the osnoise tracer. These options are: + + - DEFAULTS: reset the options to the default value. + - OSNOISE_WORKLOAD: do not dispatch osnoise workload (see dedicated + section below). + - PANIC_ON_STOP: call panic() if the tracer stops. This option serves to + capture a vmcore. + - OSNOISE_PREEMPT_DISABLE: disable preemption while running the osnoise + workload, allowing only IRQ and hardware-related noise. + - OSNOISE_IRQ_DISABLE: disable IRQs while running the osnoise workload, + allowing only NMIs and hardware-related noise, like hwlat tracer. + Additional Tracing ------------------ =20 --=20 2.32.0