From nobody Thu Apr 16 05:07:58 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA89AC433FE for ; Wed, 23 Nov 2022 17:06:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238902AbiKWRGn (ORCPT ); Wed, 23 Nov 2022 12:06:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238755AbiKWRGe (ORCPT ); Wed, 23 Nov 2022 12:06:34 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AAA8A9963; Wed, 23 Nov 2022 09:06:33 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1804361DF7; Wed, 23 Nov 2022 17:06:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A50CDC433D6; Wed, 23 Nov 2022 17:06:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223192; bh=Bv+gg9QsVueJO7E+n5Q2iX6/w77b8aeX5SBwgMpol3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ml5eeCLV3q9PB5lnsVbFjpEZNZfEthRtv6kmtwD/gl7fqVM+CnAhx1hkxr42fOJzC 6zGXYhy0ClkCfve+FJnXp98fdTdxa2gEOgayNaw+fk45fMoYP0/O2caP3yrlg4zLTL P8X/HB5GI8dn/cn872xr/itsAcLDUykhkDbzkChVyQLTgxMwkMRGmxv6k557emQG1O 2Pc7omffgZtUltUfee0ePvkQrKzdZfIOB8DzbXMR0cbkYcOXLgTQQRx1J7eMz0RlKl M5YbixpK9BwcPBt8J0w0BswMo3XwCZyGlx/fm06760eW/Dr0N9iJCx6XKnZ45DOFv5 Eu2kpvo2Zd/kA== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 1/6] tracing/osnoise: Fix duration type Date: Wed, 23 Nov 2022 18:06:19 +0100 Message-Id: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The duration type is a 64 long value, not an int. This was causing some long noise to report wrong values. Change the duration to a 64 bits value. Fixes: bce29ac9ce0b ("trace: Add osnoise tracer") Acked-by: Masami Hiramatsu (Google) Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 78d536d3ff3d..4300c5dc4e5d 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -917,7 +917,7 @@ void osnoise_trace_irq_entry(int id) void osnoise_trace_irq_exit(int id, const char *desc) { struct osnoise_variables *osn_var =3D this_cpu_osn_var(); - int duration; + s64 duration; =20 if (!osn_var->sampling) return; @@ -1048,7 +1048,7 @@ static void trace_softirq_entry_callback(void *data, = unsigned int vec_nr) static void trace_softirq_exit_callback(void *data, unsigned int vec_nr) { struct osnoise_variables *osn_var =3D this_cpu_osn_var(); - int duration; + s64 duration; =20 if (!osn_var->sampling) return; @@ -1144,7 +1144,7 @@ thread_entry(struct osnoise_variables *osn_var, struc= t task_struct *t) static void thread_exit(struct osnoise_variables *osn_var, struct task_struct *t) { - int duration; + s64 duration; =20 if (!osn_var->sampling) return; --=20 2.32.0 From nobody Thu Apr 16 05:07:58 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81318C433FE for ; Wed, 23 Nov 2022 17:06:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238875AbiKWRGq (ORCPT ); Wed, 23 Nov 2022 12:06:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238839AbiKWRGi (ORCPT ); Wed, 23 Nov 2022 12:06:38 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61CC9B972B; Wed, 23 Nov 2022 09:06:37 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0A7F5B821BE; Wed, 23 Nov 2022 17:06:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ECE1FC433D7; Wed, 23 Nov 2022 17:06:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223194; bh=Si6HR0F2oJEI23uPOBQgw/rGU8hXmlR8WuABPnN/muQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OOX8kvWPEvxfMxTT9VuNzRdr7TQ5Wetax4sPh4dvyQvENNbezwGEc86Bdlv7rkI9H C6R7j0APN3QUS2+V+XiA+m4bqVX0gpsi11edchGXWb4SUQ22iiNrdxxrhXGPRrnEBr kN585REmvFJ4UaPnjCItm91izXDl5byx1awb8plmnbvyQ+IWMd4IFLNb/rNsllyaoz DDxrwLSy23QHgS5U73g+hdyIOQz/C8j4EwlUscOplxSRfdjPC3dELE+bzmOu2gdErY TMjTKcT5hbNCXPsiSm7Yzi8Wh1XJFOYN6eU1zaSZ+MkDIZWIbKBMWpwvRVXLwF24Qc L23gK5BMOb5tQ== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 2/6] tracing/osnoise: Add osnoise/options file Date: Wed, 23 Nov 2022 18:06:20 +0100 Message-Id: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add the tracing/osnoise/options file to control osnoise/timerlat tracer features. It is a single file to contain multiple features, similar to the sched/features file. Reading the file displays a list of options. Writing the OPTION_NAME enables it, writing NO_OPTION_NAME disables it. The DEAFULTS is a particular option that resets the options to the default ones. It uses a bitmask to keep track of the status of the option. When needed, we can add a list of static keys, but for now it does not justify the memory increase. Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 170 +++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 4300c5dc4e5d..17b77fe3950b 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -48,6 +48,19 @@ #define DEFAULT_TIMERLAT_PERIOD 1000 /* 1ms */ #define DEFAULT_TIMERLAT_PRIO 95 /* FIFO 95 */ =20 +/* + * osnoise/options entries. + */ +enum osnoise_options_index { + OSN_DEFAULTS =3D 0, + OSN_MAX +}; + +static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS" }; + +#define OSN_DEFAULT_OPTIONS 0 +unsigned long osnoise_options =3D OSN_DEFAULT_OPTIONS; + /* * trace_array of the enabled osnoise/timerlat instances. */ @@ -1860,6 +1873,150 @@ static void osnoise_init_hotplug_support(void) } #endif /* CONFIG_HOTPLUG_CPU */ =20 +/* + * seq file functions for the osnoise/options file. + */ +static void *s_options_start(struct seq_file *s, loff_t *pos) +{ + int option =3D *pos; + + mutex_lock(&interface_lock); + + if (option >=3D OSN_MAX) + return NULL; + + return pos; +} + +static void *s_options_next(struct seq_file *s, void *v, loff_t *pos) +{ + int option =3D ++(*pos); + + if (option >=3D OSN_MAX) + return NULL; + + return pos; +} + +static int s_options_show(struct seq_file *s, void *v) +{ + loff_t *pos =3D v; + int option =3D *pos; + + if (option =3D=3D OSN_DEFAULTS) { + if (osnoise_options =3D=3D OSN_DEFAULT_OPTIONS) + seq_printf(s, "%s", osnoise_options_str[option]); + else + seq_printf(s, "NO_%s", osnoise_options_str[option]); + goto out; + } + + if (test_bit(option, &osnoise_options)) + seq_printf(s, "%s", osnoise_options_str[option]); + else + seq_printf(s, "NO_%s", osnoise_options_str[option]); + +out: + if (option !=3D OSN_MAX) + seq_puts(s, " "); + + return 0; +} + +static void s_options_stop(struct seq_file *s, void *v) +{ + seq_puts(s, "\n"); + mutex_unlock(&interface_lock); +} + +static const struct seq_operations osnoise_options_seq_ops =3D { + .start =3D s_options_start, + .next =3D s_options_next, + .show =3D s_options_show, + .stop =3D s_options_stop +}; + +static int osnoise_options_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &osnoise_options_seq_ops); +}; + +/** + * osnoise_options_write - Write function for "options" entry + * @filp: The active open file structure + * @ubuf: The user buffer that contains the value to write + * @cnt: The maximum number of bytes to write to "file" + * @ppos: The current position in @file + * + * Writing the option name sets the option, writing the "NO_" + * prefix in front of the option name disables it. + * + * Writing "DEFAULTS" resets the option values to the default ones. + */ +static ssize_t osnoise_options_write(struct file *filp, const char __user = *ubuf, + size_t cnt, loff_t *ppos) +{ + int running, option, enable, retval; + char buf[256], *option_str; + + if (cnt >=3D 256) + return -EINVAL; + + if (copy_from_user(buf, ubuf, cnt)) + return -EFAULT; + + buf[cnt] =3D 0; + + if (strncmp(buf, "NO_", 3)) { + option_str =3D strstrip(buf); + enable =3D true; + } else { + option_str =3D strstrip(&buf[3]); + enable =3D false; + } + + option =3D match_string(osnoise_options_str, OSN_MAX, option_str); + if (option < 0) + return -EINVAL; + + /* + * trace_types_lock is taken to avoid concurrency on start/stop. + */ + mutex_lock(&trace_types_lock); + running =3D osnoise_has_registered_instances(); + if (running) + stop_per_cpu_kthreads(); + + mutex_lock(&interface_lock); + /* + * avoid CPU hotplug operations that might read options. + */ + cpus_read_lock(); + + retval =3D cnt; + + if (enable) { + if (option =3D=3D OSN_DEFAULTS) + osnoise_options =3D OSN_DEFAULT_OPTIONS; + else + set_bit(option, &osnoise_options); + } else { + if (option =3D=3D OSN_DEFAULTS) + retval =3D -EINVAL; + else + clear_bit(option, &osnoise_options); + } + + cpus_read_unlock(); + mutex_unlock(&interface_lock); + + if (running) + start_per_cpu_kthreads(); + mutex_unlock(&trace_types_lock); + + return retval; +} + /* * osnoise_cpus_read - Read function for reading the "cpus" file * @filp: The active open file structure @@ -2042,6 +2199,14 @@ static const struct file_operations cpus_fops =3D { .llseek =3D generic_file_llseek, }; =20 +static const struct file_operations osnoise_options_fops =3D { + .open =3D osnoise_options_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D seq_release, + .write =3D osnoise_options_write +}; + #ifdef CONFIG_TIMERLAT_TRACER #ifdef CONFIG_STACKTRACE static int init_timerlat_stack_tracefs(struct dentry *top_dir) @@ -2128,6 +2293,11 @@ static int init_tracefs(void) if (!tmp) goto err; =20 + tmp =3D trace_create_file("options", TRACE_MODE_WRITE, top_dir, NULL, + &osnoise_options_fops); + if (!tmp) + goto err; + ret =3D init_timerlat_tracefs(top_dir); if (ret) goto err; --=20 2.32.0 From nobody Thu Apr 16 05:07:58 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F5FC4332F for ; Wed, 23 Nov 2022 17:07:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239205AbiKWRG7 (ORCPT ); Wed, 23 Nov 2022 12:06:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238942AbiKWRGo (ORCPT ); Wed, 23 Nov 2022 12:06:44 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C10DDC4951; Wed, 23 Nov 2022 09:06:40 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 1D521CE2498; Wed, 23 Nov 2022 17:06:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 405D1C43470; Wed, 23 Nov 2022 17:06:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223197; bh=ybjnmnbdgG5oznw+D3D3EoANpgY2rjuP8S4++LExstU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YDd2947JFcLM/g375aHLKLP6gISs3mYEY/D7MpYJC2M6csVX5pzcxeX65f2x9z+np Pebq3NWT7wVq2Al+SUCSefnbWsnCz12yCC4RRpKpS+n2FiWlxTSFcpJ6N9bXkRXW96 sElVlH0upIvI3CnPEmxBnA4ctUrVyvd+TNYH5wZXJEIfXJCyw5mC4h2DNPp24IPqdZ UNlzeKO0+QP2FG2E/Y61ZfoW65ACWfzWPw/khz5I47fAdF0XwOx+1krgtKRU0X8tsZ PcvvEVbijULFAqeGVo7vM2ac4yunMEQ+vXPgbnHNXT0P+3/RXa/eKbOujUtveD5uN+ uvc6vpepg2SNg== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 3/6] tracing/osnoise: Add OSNOISE_WORKLOAD option Date: Wed, 23 Nov 2022 18:06:21 +0100 Message-Id: <3b96fffdd10b4a1b1537235c57683cb75b3f3070.1669115208.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The osnoise tracer is not only a tracer, and a set of tracepoints, but also a workload dispatcher. In preparation for having other workloads, e.g., in user-space, add an option to avoid dispatching the workload. By not dispatching the workload, the osnoise: tracepoints become generic events to measure the execution time of *any* task on Linux. For example: # cd /sys/kernel/tracing/ # cat osnoise/options DEFAULTS OSNOISE_WORKLOAD # echo NO_OSNOISE_WORKLOAD > osnoise/options # cat osnoise/options NO_DEFAULTS NO_OSNOISE_WORKLOAD # echo osnoise > set_event # echo osnoise > current_tracer # tail -8 trace make-94722 [002] d..3. 1371.794507: thread_noise: make:94722 s= tart 1371.794302286 duration 200897 ns sh-121042 [020] d..3. 1371.794534: thread_noise: sh:121042 = start 1371.781610976 duration 8943683 ns make-121097 [005] d..3. 1371.794542: thread_noise: make:121097 = start 1371.794481522 duration 60444 ns <...>-40 [005] d..3. 1371.794550: thread_noise: migration/5:40 s= tart 1371.794542256 duration 7154 ns -0 [018] dNh2. 1371.794554: irq_noise: reschedule:253 star= t 1371.794553547 duration 40 ns -0 [018] dNh2. 1371.794561: irq_noise: local_timer:236 sta= rt 1371.794556222 duration 4890 ns -0 [018] .Ns2. 1371.794563: softirq_noise: SCHED:7 star= t 1371.794561803 duration 992 ns -0 [018] d..3. 1371.794566: thread_noise: swapper/18:0 sta= rt 1371.781368110 duration 13191798 ns In preparation for the rtla exec_time tracer/tool and rtla osnoise --user option. Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 17b77fe3950b..3f10dd1f2f1c 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -53,12 +53,13 @@ */ enum osnoise_options_index { OSN_DEFAULTS =3D 0, + OSN_WORKLOAD, OSN_MAX }; =20 -static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS" }; +static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD" }; =20 -#define OSN_DEFAULT_OPTIONS 0 +#define OSN_DEFAULT_OPTIONS 0x2 unsigned long osnoise_options =3D OSN_DEFAULT_OPTIONS; =20 /* @@ -1186,11 +1187,12 @@ trace_sched_switch_callback(void *data, bool preemp= t, unsigned int prev_state) { struct osnoise_variables *osn_var =3D this_cpu_osn_var(); + int workload =3D test_bit(OSN_WORKLOAD, &osnoise_options); =20 - if (p->pid !=3D osn_var->pid) + if ((p->pid !=3D osn_var->pid) || !workload) thread_exit(osn_var, p); =20 - if (n->pid !=3D osn_var->pid) + if ((n->pid !=3D osn_var->pid) || !workload) thread_entry(osn_var, n); } =20 @@ -1723,9 +1725,16 @@ static void stop_kthread(unsigned int cpu) struct task_struct *kthread; =20 kthread =3D per_cpu(per_cpu_osnoise_var, cpu).kthread; - if (kthread) + if (kthread) { kthread_stop(kthread); - per_cpu(per_cpu_osnoise_var, cpu).kthread =3D NULL; + per_cpu(per_cpu_osnoise_var, cpu).kthread =3D NULL; + } else { + if (!test_bit(OSN_WORKLOAD, &osnoise_options)) { + per_cpu(per_cpu_osnoise_var, cpu).sampling =3D false; + barrier(); + return; + } + } } =20 /* @@ -1759,6 +1768,13 @@ static int start_kthread(unsigned int cpu) snprintf(comm, 24, "timerlat/%d", cpu); main =3D timerlat_main; } else { + /* if no workload, just return */ + if (!test_bit(OSN_WORKLOAD, &osnoise_options)) { + per_cpu(per_cpu_osnoise_var, cpu).sampling =3D true; + barrier(); + return 0; + } + snprintf(comm, 24, "osnoise/%d", cpu); } =20 --=20 2.32.0 From nobody Thu Apr 16 05:07:58 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3259C433FE for ; Wed, 23 Nov 2022 17:07:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239039AbiKWRHk (ORCPT ); Wed, 23 Nov 2022 12:07:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239024AbiKWRGp (ORCPT ); Wed, 23 Nov 2022 12:06:45 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EE0BC4B5B; Wed, 23 Nov 2022 09:06:42 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ACCF5B821BC; Wed, 23 Nov 2022 17:06:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8ABA2C4347C; Wed, 23 Nov 2022 17:06:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223199; bh=UofQ2Om5TKKPZw19SG+8wKH8TntN+ojXWrOiFhpkAis=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zpuzjps4rlDgFJHDwYeS5J9U2UQqY/xuJGczyIdL1b18H1liRMAp3/nDzPFYH4cDs oFxeCTPeCgFLw55Ltd74fC9YCrughkpeeRZWM/N4IJ7R96hByx9/Js7nDcf1Med+Ce sMMRWUCUMjBoraelsC500d4UZFobIBm/d3X8gDXucwOjzjB8YmE4rfQoNwWugurjrZ oU0b9KP0EgNI6zaFVn/ghRM1IX3Lx3d8+55m7VYUQLhbjj9m+C7ceYiWMT0E6zdXP2 kdBmF+xl43/9hyTlGlfuC0cCGCBAHAniTPnIregkApe8xkw8+6VLq4/nHBR1DvJNTD 3S7nCir1b+MPQ== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 4/6] tracing/osnoise: Add PANIC_ON_STOP option Date: Wed, 23 Nov 2022 18:06:22 +0100 Message-Id: <31aa66b01dfc31af481bcadea04b4affed59430b.1669115208.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Often the latency observed in a CPU is not caused by the work being done in the CPU itself, but by work done on another CPU that causes the hardware to stall all CPUs. In this case, it is interesting to know what is happening on ALL CPUs, and the best way to do this is via crash dump analysis. Add the PANIC_ON_STOP option to osnoise/timerlat tracers. The default behavior is having this option off. When enabled by the user, the system will panic after hitting a stop tracing condition. This option was motivated by a real scenario that Juri Lelli and I were debugging. Cc: Juri Lelli Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 3f10dd1f2f1c..801eba0b5cf8 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -54,10 +54,11 @@ enum osnoise_options_index { OSN_DEFAULTS =3D 0, OSN_WORKLOAD, + OSN_PANIC_ON_STOP, OSN_MAX }; =20 -static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD" }; +static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD", "PANIC_ON_STOP" }; =20 #define OSN_DEFAULT_OPTIONS 0x2 unsigned long osnoise_options =3D OSN_DEFAULT_OPTIONS; @@ -1270,6 +1271,9 @@ static __always_inline void osnoise_stop_tracing(void) trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, "stop tracing hit on cpu %d\n", smp_processor_id()); =20 + if (test_bit(OSN_PANIC_ON_STOP, &osnoise_options)) + panic("tracer hit stop condition on CPU %d\n", smp_processor_id()); + tracer_tracing_off(tr); } rcu_read_unlock(); --=20 2.32.0 From nobody Thu Apr 16 05:07:58 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F384BC433FE for ; Wed, 23 Nov 2022 17:07:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238942AbiKWRHd (ORCPT ); Wed, 23 Nov 2022 12:07:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239020AbiKWRGp (ORCPT ); Wed, 23 Nov 2022 12:06:45 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A66EAC4B69; Wed, 23 Nov 2022 09:06:42 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 456EB61DF7; Wed, 23 Nov 2022 17:06:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2C2EC433D6; Wed, 23 Nov 2022 17:06:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223201; bh=572LKrFlwGTVNo7fZjQcd1sMaO3bjlC22vJvxTRspSc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Cy9qvLxY59+KxQyUyH+9sCX6HVEFqQVPPLH787k/5aUI4w1y1iVKXLv7gJ8lV/FjH JIKLuGm+rugW5eK2RiWgfAzaVnleYP75YhnvLYK4Wym80AmihlZsAyQbgA6dtIYXcl JAHvAmWGY9Iq+DHgsG5n3AQa1CIWCxbzDZbgvxwRyYp5uhtPxNSTY69EKgGEOQFwkB VtTM+spQyjkNlT6lhHs1lW/EGcpYIMt7mpRFiKRBG3niH0UdlQFjglDtcUUD7r5ROG 1dlZQwBRVpqHMsq/XWzc9g3LjLmBf2nnavqQp+Vj74LmaKa0UpKHtvpRdL9STbz6P2 Q9bM3V3r2BdAA== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 5/6] tracing/osnoise: Add preempt and/or irq disabled options Date: Wed, 23 Nov 2022 18:06:23 +0100 Message-Id: <78ab591cbda81b833578ee69e34c90f16438102a.1669115208.git.bristot@kernel.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The osnoise workload runs with preemption and IRQs enabled in such a way as to allow all sorts of noise to disturb osnoise's execution. hwlat tracer has a similar workload but works with irq disabled, allowing only NMIs and the hardware to generate noise. While thinking about adding an options file to hwlat tracer to allow the system to panic, and other features I was thinking to add, like having a tracepoint at each noise detection, it came to my mind that is easier to make osnoise and also do hardware latency detection than making hwlat "feature compatible" with osnoise. Other points are: - osnoise already has an independent cpu file. - osnoise has a more intuitive interface, e.g., runtime/period vs. window/width (and people often need help remembering what it is). - osnoise: tracepoints - osnoise stop options - osnoise options file itself Moreover, the user-space side (in rtla) is simplified by reusing the existing osnoise code. Finally, people have been asking me about using osnoise for hw latency detection, and I have to explain that it was sufficient but not necessary. These options make it sufficient and necessary. Adding a Suggested-by Clark, as he often asked me about this possibility. Cc: Suggested-by: Clark Williams Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 40 +++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 801eba0b5cf8..14b7f4092982 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -55,10 +55,17 @@ enum osnoise_options_index { OSN_DEFAULTS =3D 0, OSN_WORKLOAD, OSN_PANIC_ON_STOP, + OSN_PREEMPT_DISABLE, + OSN_IRQ_DISABLE, OSN_MAX }; =20 -static const char * const osnoise_options_str[OSN_MAX] =3D { "DEFAULTS", "= OSNOISE_WORKLOAD", "PANIC_ON_STOP" }; +static const char * const osnoise_options_str[OSN_MAX] =3D { + "DEFAULTS", + "OSNOISE_WORKLOAD", + "PANIC_ON_STOP", + "OSNOISE_PREEMPT_DISABLE", + "OSNOISE_IRQ_DISABLE" }; =20 #define OSN_DEFAULT_OPTIONS 0x2 unsigned long osnoise_options =3D OSN_DEFAULT_OPTIONS; @@ -1308,6 +1315,8 @@ static void notify_new_max_latency(u64 latency) */ static int run_osnoise(void) { + bool preempt_disable =3D test_bit(OSN_PREEMPT_DISABLE, &osnoise_options); + bool irq_disable =3D test_bit(OSN_IRQ_DISABLE, &osnoise_options); struct osnoise_variables *osn_var =3D this_cpu_osn_var(); u64 start, sample, last_sample; u64 last_int_count, int_count; @@ -1335,6 +1344,14 @@ static int run_osnoise(void) */ threshold =3D tracing_thresh ? : 5000; =20 + /* + * IRQ disable also implies in preempt disable. + */ + if (irq_disable) + local_irq_disable(); + else if (preempt_disable) + preempt_disable(); + /* * Make sure NMIs see sampling first */ @@ -1422,16 +1439,21 @@ static int run_osnoise(void) * cond_resched() */ if (IS_ENABLED(CONFIG_PREEMPT_RCU)) { - local_irq_disable(); + if (!irq_disable) + local_irq_disable(); + rcu_momentary_dyntick_idle(); - local_irq_enable(); + + if (!irq_disable) + local_irq_enable(); } =20 /* * For the non-preemptive kernel config: let threads runs, if - * they so wish. + * they so wish, unless set not do to so. */ - cond_resched(); + if (!irq_disable && !preempt_disable) + cond_resched(); =20 last_sample =3D sample; last_int_count =3D int_count; @@ -1450,6 +1472,14 @@ static int run_osnoise(void) */ barrier(); =20 + /* + * Return to the preemptive state. + */ + if (irq_disable) + local_irq_enable(); + else if (preempt_disable) + preempt_enable(); + /* * Save noise info. */ --=20 2.32.0 From nobody Thu Apr 16 05:07:58 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2E89C4332F for ; Wed, 23 Nov 2022 17:07:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239139AbiKWRHp (ORCPT ); Wed, 23 Nov 2022 12:07:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238904AbiKWRGx (ORCPT ); Wed, 23 Nov 2022 12:06:53 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28D69C5B51; Wed, 23 Nov 2022 09:06:44 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8FE3661DEC; Wed, 23 Nov 2022 17:06:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26521C43470; Wed, 23 Nov 2022 17:06:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669223204; bh=wx38S8oDI/3l7QO5/iOpwcyoWSlY3bOo8MNcfNr6WO8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b/fagKtWUhKJtbRJor0oTXVyLiMPuyF+ekrSh4ajY32KOJCqwkQZC/WYTFXOcdI3Q kN4IbX198yi8Tej6qJYMnQoiY5WOc4l961AtInYV0qsPonyLjEqb2FVTdEHk3U1n7O AXNQU+8tJU6SLtUalujKnx85OgapVKsO8T4Mf85sH0UA8hXv6F/zn/d1qP09i1a0fA VhlaDw0PhdPgJStJWwGhtZ58OZqHGCPgMkYp0a3+hUeeBJYW1NPDqq2f7MebxwdEI3 vdf7OGZB4WR3vLchauCrRU91A4J88OLtx4+p54t8+XAhdmyx0rWm6HZkKB5YBa88/7 WxwqidzJ38sbw== From: Daniel Bristot de Oliveira To: Daniel Bristot de Oliveira , Steven Rostedt Cc: Masami Hiramatsu , Jonathan Corbet , Juri Lelli , Clark Williams , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 6/6] Documentation/osnoise: Add osnoise/options documentation Date: Wed, 23 Nov 2022 18:06:24 +0100 Message-Id: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add the documentation about the osnoise/options file, the options, and some additional explanation about the OSNOISE_WORKLOAD option. Cc: Daniel Bristot de Oliveira Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- Documentation/trace/osnoise-tracer.rst | 32 ++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/Documentation/trace/osnoise-tracer.rst b/Documentation/trace/o= snoise-tracer.rst index 963def9f97c6..618bcdc90bb6 100644 --- a/Documentation/trace/osnoise-tracer.rst +++ b/Documentation/trace/osnoise-tracer.rst @@ -92,8 +92,8 @@ Note that the example above shows a high number of HW noi= se samples. The reason being is that this sample was taken on a virtual machine, and the host interference is detected as a hardware interference. =20 -Tracer options ---------------------- +Tracer Configuration +-------------------- =20 The tracer has a set of options inside the osnoise directory, they are: =20 @@ -109,6 +109,27 @@ The tracer has a set of options inside the osnoise dir= ectory, they are: - tracing_threshold: the minimum delta between two time() reads to be considered as noise, in us. When set to 0, the default value will be used, which is currently 5 us. + - osnoise/options: a set of on/off options that can be enabled by + writing the option name to the file or disabled by writing the option + name preceded with the 'NO_' prefix. For example, writing + NO_OSNOISE_WORKLOAD disables the OSNOISE_WORKLOAD option. The + special DEAFAULTS option resets all options to the default value. + +Tracer Options +-------------- + +The osnoise/options file exposes a set of on/off configuration options for +the osnoise tracer. These options are: + + - DEFAULTS: reset the options to the default value. + - OSNOISE_WORKLOAD: do not dispatch osnoise workload (see dedicated + section below). + - PANIC_ON_STOP: call panic() if the tracer stops. This option serves to + capture a vmcore. + - OSNOISE_PREEMPT_DISABLE: disable preemption while running the osnoise + workload, allowing only IRQ-related and hardware-related noise. + - OSNOISE_IRQ_DISABLE: disable IRQs while running the osnoise workload, + allowing only NMIs and hardware-related noise, like hwlat tracer. =20 Additional Tracing ------------------ @@ -150,3 +171,10 @@ tracepoints is smaller than eight us reported in the s= ample_threshold. The reason roots in the overhead of the entry and exit code that happens before and after any interference execution. This justifies the dual approach: measuring thread and tracing. + +Running osnoise tracer without workload +--------------------------------------- + +By enabling the osnoise tracer with the NO_OSNOISE_WORKLOAD option set, +the osnoise: tracepoints serves to measure the execution time of +any type of Linux task, free from the interference of other tasks. --=20 2.32.0