From nobody Fri Dec 19 17:14:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D2712E9EB9; Tue, 11 Nov 2025 23:24:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762903458; cv=none; b=U4ZyoJqTsJ9zS0zHqUl+kYQ0IcEmYsFb3bby43MkseIr4SMEHra4VcQuGmAmqvb46jwzZOEf4KFrRpKT4mt+a4U14GsheBAwOR3UWWB2zh5FwvnMdH5ipHyGKUXwdRbmNdK/ctivefOmQY3WGauHp4L3fROfPcw8UnIT3X5lmBk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762903458; c=relaxed/simple; bh=4sXzPLdZ5KAtiaOblquQvaOmj0BJ88Q3ycOf/MDgf3s=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=NASOBy51gces2dNEi91pMt8i+6FOKYYvyexpXll0YHOX1fPkn73I29mkUQgFnhsRMcbFBiNqXUWoOIJcxyp3LCp2ZY77Wr7dXK+LKfEj3wLDfo6Q8pKWfjihTN2h8VZ27Zxqz0Rafq0ALzG/8RUdNR5pAHoXFuOJm0PzuCCUfzc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=W09RoQ3V; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="W09RoQ3V" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E400C4CEFB; Tue, 11 Nov 2025 23:24:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762903457; bh=4sXzPLdZ5KAtiaOblquQvaOmj0BJ88Q3ycOf/MDgf3s=; h=Date:From:To:Cc:Subject:References:From; b=W09RoQ3VJ8nf7AVkZVs7mMZs+/LXGefkoWl0b9wilkefKuv630F498Q4JUpH9NUan wVMqTnt54wIQ44xEglIK3eFSa+dkzb/e+8owhTqBX5TzXq3ggXMFuga7RrwKxMCArv so5uqLPmuoX6pcbbkORn0eyGRFn5HbLllsXVI8KZmOVuedw4u+eJOvK8zkG+2aVaKp 8tLSPUNJYHkW+XcOhecTCd9CnMfGXTk83VuonjEkBrTUvVBfMKf3iD60C9MeGHU9J1 lKzePFF0JldXzE9FYUFehUDUS0rlVLoEozWkJXNPARUBs9aTXiC+SZQzFCBJdyJ0M6 V5QRxk7J/grDQ== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1vIxiv-00000000uRt-1tCt; Tue, 11 Nov 2025 18:24:29 -0500 Message-ID: <20251111232429.305317942@kernel.org> User-Agent: quilt/0.68 Date: Tue, 11 Nov 2025 18:24:07 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton Subject: [PATCH v4 1/3] tracing: Have tracer option be instance specific References: <20251111232406.183056289@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt Tracers can add specify options to modify them. This logic was added before instances were created and the tracer flags were global variables. After instances were created where a tracer may exist in more than one instance, the flags were not updated from being global into instance specific. This causes confusion with these options. For example, the function tracer has an option to enable function arguments: # cd /sys/kernel/tracing # mkdir instances/foo # echo function > instances/foo/current_tracer # echo 1 > options/func-args # echo function > current_tracer # cat trace [..] -0 [005] d..3. 1050.656187: rcu_needs_cpu() <-tick_nohz_next= _event -0 [005] d..3. 1050.656188: get_next_timer_interrupt(basej= =3D0x10002dbad, basem=3D0xf45fd7d300) <-tick_nohz_next_event -0 [005] d..3. 1050.656189: _raw_spin_lock(lock=3D0xffff8944= bdf5de80) <-__get_next_timer_interrupt -0 [005] d..4. 1050.656190: do_raw_spin_lock(lock=3D0xffff89= 44bdf5de80) <-__get_next_timer_interrupt -0 [005] d..4. 1050.656191: _raw_spin_lock_nested(lock=3D0xf= fff8944bdf5f140, subclass=3D1) <-__get_next_timer_interrupt # cat instances/foo/options/func-args 1 # cat instances/foo/trace [..] kworker/4:1-88 [004] ...1. 298.127735: next_zone <-refresh_cpu_vm_= stats kworker/4:1-88 [004] ...1. 298.127736: first_online_pgdat <-refres= h_cpu_vm_stats kworker/4:1-88 [004] ...1. 298.127738: next_online_pgdat <-refresh= _cpu_vm_stats kworker/4:1-88 [004] ...1. 298.127739: fold_diff <-refresh_cpu_vm_= stats kworker/4:1-88 [004] ...1. 298.127741: round_jiffies_relative <-vm= stat_update [..] The above shows that setting "func-args" in the top level instance also set it in the instance "foo", but since the interface of the trace flags are per instance, the update didn't take affect in the "foo" instance. Update the infrastructure to allow tracers to add a "default_flags" field in the tracer structure that can be set instead of "flags" which will make the flags per instance. If a tracer needs to keep the flags global (like blktrace), keeping the "flags" field set will keep the old behavior. This does not update function or the function graph tracers. That will be handled later. Fixes: f20a580627f43 ("ftrace: Allow instances to use function tracing") Signed-off-by: Steven Rostedt (Google) Reported-by: Andy Shevchenko --- Changes since v3: https://patch.msgid.link/20251110234341.767135255@kernel.= org - Fix registering of command line tracers. A command line tracer is registered before the global array attaches itself to the list of instances. This keeps the tracer from attaching to the global array and that keeps the tracer from adding itself to the global array, and this means it fails to start at boot. - Also add a warning if the starting of a boot tracer fails to start. kernel/trace/trace.c | 257 ++++++++++++++++++++++++++++++------------- kernel/trace/trace.h | 3 + 2 files changed, 186 insertions(+), 74 deletions(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 0aea9cb84276..9268489d2ce8 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -94,6 +94,13 @@ static bool tracepoint_printk_stop_on_boot __initdata; static bool traceoff_after_boot __initdata; static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key); =20 +/* Store tracers and their flags per instance */ +struct tracers { + struct list_head list; + struct tracer *tracer; + struct tracer_flags *flags; +}; + /* * To prevent the comm cache from being overwritten when no * tracing is active, only save the comm when a trace event @@ -2164,6 +2171,7 @@ static int save_selftest(struct tracer *type) static int run_tracer_selftest(struct tracer *type) { struct trace_array *tr =3D &global_trace; + struct tracer_flags *saved_flags =3D tr->current_trace_flags; struct tracer *saved_tracer =3D tr->current_trace; int ret; =20 @@ -2194,6 +2202,7 @@ static int run_tracer_selftest(struct tracer *type) tracing_reset_online_cpus(&tr->array_buffer); =20 tr->current_trace =3D type; + tr->current_trace_flags =3D type->flags ? : type->default_flags; =20 #ifdef CONFIG_TRACER_MAX_TRACE if (type->use_max_tr) { @@ -2210,6 +2219,7 @@ static int run_tracer_selftest(struct tracer *type) ret =3D type->selftest(type, tr); /* the test is responsible for resetting too */ tr->current_trace =3D saved_tracer; + tr->current_trace_flags =3D saved_flags; if (ret) { printk(KERN_CONT "FAILED!\n"); /* Add the warning after printing 'FAILED' */ @@ -2302,10 +2312,23 @@ static inline int do_run_tracer_selftest(struct tra= cer *type) } #endif /* CONFIG_FTRACE_STARTUP_TEST */ =20 -static int add_tracer_options(struct trace_array *tr, struct tracer *t); +static int add_tracer(struct trace_array *tr, struct tracer *t); =20 static void __init apply_trace_boot_options(void); =20 +static void free_tracers(struct trace_array *tr) +{ + struct tracers *t, *n; + + lockdep_assert_held(&trace_types_lock); + + list_for_each_entry_safe(t, n, &tr->tracers, list) { + list_del(&t->list); + kfree(t->flags); + kfree(t); + } +} + /** * register_tracer - register a tracer with the ftrace system. * @type: the plugin for the tracer @@ -2314,6 +2337,7 @@ static void __init apply_trace_boot_options(void); */ int __init register_tracer(struct tracer *type) { + struct trace_array *tr; struct tracer *t; int ret =3D 0; =20 @@ -2353,10 +2377,13 @@ int __init register_tracer(struct tracer *type) if (ret < 0) goto out; =20 - ret =3D add_tracer_options(&global_trace, type); - if (ret < 0) { - pr_warn("Failed to create tracer options for %s\n", type->name); - goto out; + list_for_each_entry(tr, &ftrace_trace_arrays, list) { + ret =3D add_tracer(tr, type); + if (ret < 0) { + /* The tracer will still exist but without options */ + pr_warn("Failed to create tracer options for %s\n", type->name); + break; + } } =20 type->next =3D trace_types; @@ -2373,7 +2400,7 @@ int __init register_tracer(struct tracer *type) =20 printk(KERN_INFO "Starting tracer '%s'\n", type->name); /* Do we want this tracer to start on bootup? */ - tracing_set_tracer(&global_trace, type->name); + WARN_ON(tracing_set_tracer(&global_trace, type->name) < 0); default_bootup_tracer =3D NULL; =20 apply_trace_boot_options(); @@ -5139,6 +5166,7 @@ static int tracing_trace_options_show(struct seq_file= *m, void *v) { struct tracer_opt *trace_opts; struct trace_array *tr =3D m->private; + struct tracer_flags *flags; struct tracer *trace; u32 tracer_flags; int i; @@ -5152,12 +5180,14 @@ static int tracing_trace_options_show(struct seq_fi= le *m, void *v) seq_printf(m, "no%s\n", trace_options[i]); } =20 - trace =3D tr->current_trace; - if (!trace->flags || !trace->flags->opts) + flags =3D tr->current_trace_flags; + if (!flags || !flags->opts) return 0; =20 - tracer_flags =3D tr->current_trace->flags->val; - trace_opts =3D tr->current_trace->flags->opts; + trace =3D tr->current_trace; + + tracer_flags =3D flags->val; + trace_opts =3D flags->opts; =20 for (i =3D 0; trace_opts[i].name; i++) { if (tracer_flags & trace_opts[i].bit) @@ -5191,8 +5221,7 @@ static int __set_tracer_option(struct trace_array *tr, /* Try to assign a tracer specific option */ static int set_tracer_option(struct trace_array *tr, char *cmp, int neg) { - struct tracer *trace =3D tr->current_trace; - struct tracer_flags *tracer_flags =3D trace->flags; + struct tracer_flags *tracer_flags =3D tr->current_trace_flags; struct tracer_opt *opts =3D NULL; int i; =20 @@ -5203,7 +5232,7 @@ static int set_tracer_option(struct trace_array *tr, = char *cmp, int neg) opts =3D &tracer_flags->opts[i]; =20 if (strcmp(cmp, opts->name) =3D=3D 0) - return __set_tracer_option(tr, trace->flags, opts, neg); + return __set_tracer_option(tr, tracer_flags, opts, neg); } =20 return -EINVAL; @@ -6235,11 +6264,6 @@ int tracing_update_buffers(struct trace_array *tr) return ret; } =20 -struct trace_option_dentry; - -static int -create_trace_option_files(struct trace_array *tr, struct tracer *tracer); - /* * Used to clear out the tracer before deletion of an instance. * Must have trace_types_lock held. @@ -6255,26 +6279,15 @@ static void tracing_set_nop(struct trace_array *tr) tr->current_trace->reset(tr); =20 tr->current_trace =3D &nop_trace; + tr->current_trace_flags =3D nop_trace.flags; } =20 static bool tracer_options_updated; =20 -static int add_tracer_options(struct trace_array *tr, struct tracer *t) -{ - /* Only enable if the directory has been created already. */ - if (!tr->dir && !(tr->flags & TRACE_ARRAY_FL_GLOBAL)) - return 0; - - /* Only create trace option files after update_tracer_options finish */ - if (!tracer_options_updated) - return 0; - - return create_trace_option_files(tr, t); -} - int tracing_set_tracer(struct trace_array *tr, const char *buf) { - struct tracer *t; + struct tracer *trace =3D NULL; + struct tracers *t; #ifdef CONFIG_TRACER_MAX_TRACE bool had_max_tr; #endif @@ -6292,18 +6305,20 @@ int tracing_set_tracer(struct trace_array *tr, cons= t char *buf) ret =3D 0; } =20 - for (t =3D trace_types; t; t =3D t->next) { - if (strcmp(t->name, buf) =3D=3D 0) + list_for_each_entry(t, &tr->tracers, list) { + if (strcmp(t->tracer->name, buf) =3D=3D 0) { + trace =3D t->tracer; break; + } } - if (!t) + if (!trace) return -EINVAL; =20 - if (t =3D=3D tr->current_trace) + if (trace =3D=3D tr->current_trace) return 0; =20 #ifdef CONFIG_TRACER_SNAPSHOT - if (t->use_max_tr) { + if (trace->use_max_tr) { local_irq_disable(); arch_spin_lock(&tr->max_lock); ret =3D tr->cond_snapshot ? -EBUSY : 0; @@ -6314,14 +6329,14 @@ int tracing_set_tracer(struct trace_array *tr, cons= t char *buf) } #endif /* Some tracers won't work on kernel command line */ - if (system_state < SYSTEM_RUNNING && t->noboot) { + if (system_state < SYSTEM_RUNNING && trace->noboot) { pr_warn("Tracer '%s' is not allowed on command line, ignored\n", - t->name); + trace->name); return -EINVAL; } =20 /* Some tracers are only allowed for the top level buffer */ - if (!trace_ok_for_array(t, tr)) + if (!trace_ok_for_array(trace, tr)) return -EINVAL; =20 /* If trace pipe files are being read, we can't change the tracer */ @@ -6340,8 +6355,9 @@ int tracing_set_tracer(struct trace_array *tr, const = char *buf) =20 /* Current trace needs to be nop_trace before synchronize_rcu */ tr->current_trace =3D &nop_trace; + tr->current_trace_flags =3D nop_trace.flags; =20 - if (had_max_tr && !t->use_max_tr) { + if (had_max_tr && !trace->use_max_tr) { /* * We need to make sure that the update_max_tr sees that * current_trace changed to nop_trace to keep it from @@ -6354,7 +6370,7 @@ int tracing_set_tracer(struct trace_array *tr, const = char *buf) tracing_disarm_snapshot(tr); } =20 - if (!had_max_tr && t->use_max_tr) { + if (!had_max_tr && trace->use_max_tr) { ret =3D tracing_arm_snapshot_locked(tr); if (ret) return ret; @@ -6363,18 +6379,21 @@ int tracing_set_tracer(struct trace_array *tr, cons= t char *buf) tr->current_trace =3D &nop_trace; #endif =20 - if (t->init) { - ret =3D tracer_init(t, tr); + tr->current_trace_flags =3D t->flags ? : t->tracer->flags; + + if (trace->init) { + ret =3D tracer_init(trace, tr); if (ret) { #ifdef CONFIG_TRACER_MAX_TRACE - if (t->use_max_tr) + if (trace->use_max_tr) tracing_disarm_snapshot(tr); #endif + tr->current_trace_flags =3D nop_trace.flags; return ret; } } =20 - tr->current_trace =3D t; + tr->current_trace =3D trace; tr->current_trace->enabled++; trace_branch_enable(tr); =20 @@ -9598,40 +9617,20 @@ create_trace_option_file(struct trace_array *tr, =20 topt->entry =3D trace_create_file(opt->name, TRACE_MODE_WRITE, t_options, topt, &trace_options_fops); - } =20 static int -create_trace_option_files(struct trace_array *tr, struct tracer *tracer) +create_trace_option_files(struct trace_array *tr, struct tracer *tracer, + struct tracer_flags *flags) { struct trace_option_dentry *topts; struct trace_options *tr_topts; - struct tracer_flags *flags; struct tracer_opt *opts; int cnt; - int i; - - if (!tracer) - return 0; - - flags =3D tracer->flags; =20 if (!flags || !flags->opts) return 0; =20 - /* - * If this is an instance, only create flags for tracers - * the instance may have. - */ - if (!trace_ok_for_array(tracer, tr)) - return 0; - - for (i =3D 0; i < tr->nr_topts; i++) { - /* Make sure there's no duplicate flags. */ - if (WARN_ON_ONCE(tr->topts[i].tracer->flags =3D=3D tracer->flags)) - return -EINVAL; - } - opts =3D flags->opts; =20 for (cnt =3D 0; opts[cnt].name; cnt++) @@ -9663,6 +9662,96 @@ create_trace_option_files(struct trace_array *tr, st= ruct tracer *tracer) return 0; } =20 +static int get_global_flags_val(struct tracer *tracer) +{ + struct tracers *t; + + list_for_each_entry(t, &global_trace.tracers, list) { + if (t->tracer !=3D tracer) + continue; + if (!t->flags) + return -1; + return t->flags->val; + } + return -1; +} + +static int add_tracer_options(struct trace_array *tr, struct tracers *t) +{ + struct tracer *tracer =3D t->tracer; + struct tracer_flags *flags =3D t->flags ?: tracer->flags; + + if (!flags) + return 0; + + /* Only add tracer options after update_tracer_options finish */ + if (!tracer_options_updated) + return 0; + + return create_trace_option_files(tr, tracer, flags); +} + +static int add_tracer(struct trace_array *tr, struct tracer *tracer) +{ + struct tracer_flags *flags; + struct tracers *t; + int ret; + + /* Only enable if the directory has been created already. */ + if (!tr->dir && !(tr->flags & TRACE_ARRAY_FL_GLOBAL)) + return 0; + + /* + * If this is an instance, only create flags for tracers + * the instance may have. + */ + if (!trace_ok_for_array(tracer, tr)) + return 0; + + t =3D kmalloc(sizeof(*t), GFP_KERNEL); + if (!t) + return -ENOMEM; + + t->tracer =3D tracer; + t->flags =3D NULL; + list_add(&t->list, &tr->tracers); + + flags =3D tracer->flags; + if (!flags) { + if (!tracer->default_flags) + return 0; + + /* + * If the tracer defines default flags, it means the flags are + * per trace instance. + */ + flags =3D kmalloc(sizeof(*flags), GFP_KERNEL); + if (!flags) + return -ENOMEM; + + *flags =3D *tracer->default_flags; + flags->trace =3D tracer; + + t->flags =3D flags; + + /* If this is an instance, inherit the global_trace flags */ + if (!(tr->flags & TRACE_ARRAY_FL_GLOBAL)) { + int val =3D get_global_flags_val(tracer); + if (!WARN_ON_ONCE(val < 0)) + flags->val =3D val; + } + } + + ret =3D add_tracer_options(tr, t); + if (ret < 0) { + list_del(&t->list); + kfree(t->flags); + kfree(t); + } + + return ret; +} + static struct dentry * create_trace_option_core_file(struct trace_array *tr, const char *option, long index) @@ -10111,13 +10200,27 @@ static void init_trace_flags_index(struct trace_a= rray *tr) tr->trace_flags_index[i] =3D i; } =20 -static int __update_tracer_options(struct trace_array *tr) +static int __update_tracer(struct trace_array *tr) { struct tracer *t; int ret =3D 0; =20 for (t =3D trace_types; t && !ret; t =3D t->next) + ret =3D add_tracer(tr, t); + + return ret; +} + +static __init int __update_tracer_options(struct trace_array *tr) +{ + struct tracers *t; + int ret =3D 0; + + list_for_each_entry(t, &tr->tracers, list) { ret =3D add_tracer_options(tr, t); + if (ret < 0) + break; + } =20 return ret; } @@ -10171,7 +10274,7 @@ static int trace_array_create_dir(struct trace_arra= y *tr) } =20 init_tracer_tracefs(tr, tr->dir); - ret =3D __update_tracer_options(tr); + ret =3D __update_tracer(tr); if (ret) { event_trace_del_tracer(tr); tracefs_remove(tr->dir); @@ -10226,11 +10329,13 @@ trace_array_create_systems(const char *name, cons= t char *systems, spin_lock_init(&tr->snapshot_trigger_lock); #endif tr->current_trace =3D &nop_trace; + tr->current_trace_flags =3D nop_trace.flags; =20 INIT_LIST_HEAD(&tr->systems); INIT_LIST_HEAD(&tr->events); INIT_LIST_HEAD(&tr->hist_vars); INIT_LIST_HEAD(&tr->err_log); + INIT_LIST_HEAD(&tr->tracers); INIT_LIST_HEAD(&tr->marker_list); =20 #ifdef CONFIG_MODULES @@ -10403,6 +10508,7 @@ static int __remove_instance(struct trace_array *tr) free_percpu(tr->last_func_repeats); free_trace_buffers(tr); clear_tracing_err_log(tr); + free_tracers(tr); =20 if (tr->range_name) { reserve_mem_release_by_name(tr->range_name); @@ -11437,6 +11543,7 @@ __init static int tracer_alloc_buffers(void) * just a bootstrap of current_trace anyway. */ global_trace.current_trace =3D &nop_trace; + global_trace.current_trace_flags =3D nop_trace.flags; =20 global_trace.max_lock =3D (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; #ifdef CONFIG_TRACER_MAX_TRACE @@ -11450,10 +11557,7 @@ __init static int tracer_alloc_buffers(void) =20 init_trace_flags_index(&global_trace); =20 - register_tracer(&nop_trace); - - /* Function tracing may start here (via kernel command line) */ - init_function_trace(); + INIT_LIST_HEAD(&global_trace.tracers); =20 /* All seems OK, enable tracing */ tracing_disabled =3D 0; @@ -11474,6 +11578,11 @@ __init static int tracer_alloc_buffers(void) list_add(&global_trace.marker_list, &marker_copies); list_add(&global_trace.list, &ftrace_trace_arrays); =20 + register_tracer(&nop_trace); + + /* Function tracing may start here (via kernel command line) */ + init_function_trace(); + apply_trace_boot_options(); =20 register_snapshot_cmd(); diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 8ecaf91ca823..299862aad66c 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -392,6 +392,7 @@ struct trace_array { int buffer_percent; unsigned int n_err_log_entries; struct tracer *current_trace; + struct tracer_flags *current_trace_flags; u64 trace_flags; unsigned char trace_flags_index[TRACE_FLAGS_MAX_SIZE]; unsigned int flags; @@ -406,6 +407,7 @@ struct trace_array { struct list_head systems; struct list_head events; struct list_head marker_list; + struct list_head tracers; struct trace_event_file *trace_marker_file; cpumask_var_t tracing_cpumask; /* only trace on set CPUs */ /* one per_cpu trace_pipe can be opened by only one user */ @@ -637,6 +639,7 @@ struct tracer { u64 mask, int set); struct tracer *next; struct tracer_flags *flags; + struct tracer_flags *default_flags; int enabled; bool print_max; bool allow_instances; --=20 2.51.0 From nobody Fri Dec 19 17:14:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CFAF2D6E5B; Tue, 11 Nov 2025 23:24:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762903458; cv=none; b=rQX5QVLIlOceP4Pr/jr/9HwHxdyGEkyvUwtzEp2+4sq4JI93XRYi73OvU6lQp5oDd9LSxZmVWsK62p2vthbs4vJlxLbGPVCb/hM0+ZiqN7h/sqnCghRbOhYopYoZYc7GPKFsg0aoHuYALU3FDpK1svUqValUE/YjanLs2sSc68U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762903458; c=relaxed/simple; bh=OUFFj9AZqHaaEQJANSjC5PFp63yqEirX5OV3P3nO7UQ=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=czugFtF0PPzN3On7e53ksHYXPzRjUCm0LvdLTuiRJMreV9rYWPnKynw4QO8nfMsHtdsPUYrMUIQIoeWUy8FhSA9Hq9Guk6EqE5/mf4FaHoO3Lypk65RTRaPuqLG1QhDvot+i1kI4e39Lrqlh4uys/G6nTo57NfmeZ/Mwz8SBNow= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jjSory1Y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jjSory1Y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C5CBCC116B1; Tue, 11 Nov 2025 23:24:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762903457; bh=OUFFj9AZqHaaEQJANSjC5PFp63yqEirX5OV3P3nO7UQ=; h=Date:From:To:Cc:Subject:References:From; b=jjSory1YK8IgJF6IFGDdgmvGBU9P2x5+yhlFoyZXih1wBQUlhf1EB3rhIjJ/gpI77 r3dGZtpJmw+8zPA10VqSBvSoFkPoNnzqQJRe3CQYhfKwwuelfpNhDHf5s8KRGUyPP6 8ApDjXCjXQh1A0P9BhhV6L7H64Y0bKjRQzo8CLjvP+bQkWj6P+OfVRPCKScZLBIOGY 4/RYryIUxBLiP29gNCelonM+HcJSmzCOsb3K0LMyEYqiGGRC8rKWxTZHdFRZIhXgO8 dmA9jMrz21XO5mgRiNvOL4CXH1utss1j6E9mPCw5aBsB3l23/ZbdcFMMlZDQCGqbkk fLpL5V8Ttxvzw== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1vIxiv-00000000uSN-2bbZ; Tue, 11 Nov 2025 18:24:29 -0500 Message-ID: <20251111232429.470883736@kernel.org> User-Agent: quilt/0.68 Date: Tue, 11 Nov 2025 18:24:08 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton Subject: [PATCH v4 2/3] tracing: Have function tracer define options per instance References: <20251111232406.183056289@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt Currently the function tracer's options are saved via a global mask when it should be per instance. Use the new infrastructure to define a "default_flags" field in the tracer structure that is used for the top level instance as well as new ones. Currently the global mask causes confusion: # cd /sys/kernel/tracing # mkdir instances/foo # echo function > instances/foo/current_tracer # echo 1 > options/func-args # echo function > current_tracer # cat trace [..] -0 [005] d..3. 1050.656187: rcu_needs_cpu() <-tick_nohz_next= _event -0 [005] d..3. 1050.656188: get_next_timer_interrupt(basej= =3D0x10002dbad, basem=3D0xf45fd7d300) <-tick_nohz_next_event -0 [005] d..3. 1050.656189: _raw_spin_lock(lock=3D0xffff8944= bdf5de80) <-__get_next_timer_interrupt -0 [005] d..4. 1050.656190: do_raw_spin_lock(lock=3D0xffff89= 44bdf5de80) <-__get_next_timer_interrupt -0 [005] d..4. 1050.656191: _raw_spin_lock_nested(lock=3D0xf= fff8944bdf5f140, subclass=3D1) <-__get_next_timer_interrupt # cat instances/foo/options/func-args 1 # cat instances/foo/trace [..] kworker/4:1-88 [004] ...1. 298.127735: next_zone <-refresh_cpu_vm_= stats kworker/4:1-88 [004] ...1. 298.127736: first_online_pgdat <-refres= h_cpu_vm_stats kworker/4:1-88 [004] ...1. 298.127738: next_online_pgdat <-refresh= _cpu_vm_stats kworker/4:1-88 [004] ...1. 298.127739: fold_diff <-refresh_cpu_vm_= stats kworker/4:1-88 [004] ...1. 298.127741: round_jiffies_relative <-vm= stat_update [..] The above shows that updating the "func-args" option at the top level instance also updates the "func-args" option in the instance but because the update is only done by the instance that gets changed (as it should), it's confusing to see that the option is already set in the other instance. Fixes: f20a580627f43 ("ftrace: Allow instances to use function tracing") Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_functions.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index d17c18934445..c12795c2fb39 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -154,11 +154,11 @@ static int function_trace_init(struct trace_array *tr) if (!tr->ops) return -ENOMEM; =20 - func =3D select_trace_function(func_flags.val); + func =3D select_trace_function(tr->current_trace_flags->val); if (!func) return -EINVAL; =20 - if (!handle_func_repeats(tr, func_flags.val)) + if (!handle_func_repeats(tr, tr->current_trace_flags->val)) return -ENOMEM; =20 ftrace_init_array_ops(tr, func); @@ -459,14 +459,14 @@ func_set_flag(struct trace_array *tr, u32 old_flags, = u32 bit, int set) u32 new_flags; =20 /* Do nothing if already set. */ - if (!!set =3D=3D !!(func_flags.val & bit)) + if (!!set =3D=3D !!(tr->current_trace_flags->val & bit)) return 0; =20 /* We can change this flag only when not running. */ if (tr->current_trace !=3D &function_trace) return 0; =20 - new_flags =3D (func_flags.val & ~bit) | (set ? bit : 0); + new_flags =3D (tr->current_trace_flags->val & ~bit) | (set ? bit : 0); func =3D select_trace_function(new_flags); if (!func) return -EINVAL; @@ -491,7 +491,7 @@ static struct tracer function_trace __tracer_data =3D .init =3D function_trace_init, .reset =3D function_trace_reset, .start =3D function_trace_start, - .flags =3D &func_flags, + .default_flags =3D &func_flags, .set_flag =3D func_set_flag, .allow_instances =3D true, #ifdef CONFIG_FTRACE_SELFTEST --=20 2.51.0 From nobody Fri Dec 19 17:14:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A0DE2F39D7; Tue, 11 Nov 2025 23:24:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762903458; cv=none; b=OvwsaqRYLh05xT959mzt23Oo+9F9oz8kazgf470dLwe5+enbXi/jw9Wmb56ikwvNIHxTwBVia7AtUYNKcJP6+e0aZzzo8FdlVk0p7JaTD5VwzVZahfnRT7uqvUNdn1sU0HoX3dErMvWBPRtxM6B+RQEgNbiO5Ff4mtSmFhzaWxI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762903458; c=relaxed/simple; bh=CF9vtFLqgxRgpJ1KV2wSkiIvgBr/keqs7YoOKJexPFM=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=kvVw3uQZcUnIIZT+ZqFCO0lJe3admaelrwXiX+Qeg1jE0jCfLRvLsaAXNUAJZXVML7upFZQHF6Mvm8/hHIRkkvatEU/S8bUA/OrnD15CUd/bXKWdnJrbNNtHQZrGvo/5K/CRm44x1Ye/+mD3kqrxi5Gm1K0MghQXXRCHIJhS43c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Y58MTPAs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Y58MTPAs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13BDBC113D0; Tue, 11 Nov 2025 23:24:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762903458; bh=CF9vtFLqgxRgpJ1KV2wSkiIvgBr/keqs7YoOKJexPFM=; h=Date:From:To:Cc:Subject:References:From; b=Y58MTPAsmxEUE+yB906hV/bQHDxR3wRsSkr/yoWQ/Kr5UI+2/n5nSwsXceZLrfSmF T4Vj2TPFGXA+mi7Hd4qjpZcu4oVIgIBEve3RpH0cgZzxsLTfEV/Qo51xxBfWJDvEBK 3/pm1CD40mEQzjlam7B2UoUOVUoWF7qZ2SleAvAIXXs+k2UwfFg8PNUJOqxUlMsOR/ VNtHJKmr1cjJdEh/G7iCpTc/G+y1zAXbUMQdRwkdci7v1UUf3nSJYKCJnqL463utev 8n4NtVr2Acl1lnBW79nP3u+gRp7xuVpUUL2Au9GaPTCJGU+ClDU4uiFGyeF2UPWMiU 4R1es0x/xG0Pw== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1vIxiv-00000000uSr-3IYl; Tue, 11 Nov 2025 18:24:29 -0500 Message-ID: <20251111232429.641030027@kernel.org> User-Agent: quilt/0.68 Date: Tue, 11 Nov 2025 18:24:09 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton Subject: [PATCH v4 3/3] tracing: Have function graph tracer define options per instance References: <20251111232406.183056289@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt Currently the function graph tracer's options are saved via a global mask when it should be per instance. Use the new infrastructure to define a "default_flags" field in the tracer structure that is used for the top level instance as well as new ones. Currently the global mask causes confusion: # cd /sys/kernel/tracing # mkdir instances/foo # echo function_graph > instances/foo/current_tracer # echo 1 > options/funcgraph-args # echo function_graph > current_tracer # cat trace [..] 2) | _raw_spin_lock_irq(lock=3D0xffff96b97dea16c0) { 2) 0.422 us | do_raw_spin_lock(lock=3D0xffff96b97dea16c0); 7) | rcu_sched_clock_irq(user=3D0) { 2) 1.478 us | } 7) 0.758 us | rcu_is_cpu_rrupt_from_idle(); 2) 0.647 us | enqueue_hrtimer(timer=3D0xffff96b97dea2058, ba= se=3D0xffff96b97dea1740, mode=3D0); # cat instances/foo/options/funcgraph-args 1 # cat instances/foo/trace [..] 4) | __x64_sys_read() { 4) | ksys_read() { 4) 0.755 us | fdget_pos(); 4) | vfs_read() { 4) | rw_verify_area() { 4) | security_file_permission() { 4) | apparmor_file_permission() { 4) | common_file_perm() { 4) | aa_file_perm() { 4) | rcu_read_lock_held() { [..] The above shows that updating the "funcgraph-args" option at the top level instance also updates the "funcgraph-args" option in the instance but because the update is only done by the instance that gets changed (as it should), it's confusing to see that the option is already set in the other instance. Link: https://patch.msgid.link/20251110234342.105763200@kernel.org Fixes: c132be2c4fcc1 ("function_graph: Have the instances use their own ftr= ace_ops for filtering") Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_functions_graph.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_func= tions_graph.c index fe9607edc8f9..4e86adf6dd4d 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -101,9 +101,9 @@ static struct tracer_flags tracer_flags =3D { .opts =3D trace_opts }; =20 -static bool tracer_flags_is_set(u32 flags) +static bool tracer_flags_is_set(struct trace_array *tr, u32 flags) { - return (tracer_flags.val & flags) =3D=3D flags; + return (tr->current_trace_flags->val & flags) =3D=3D flags; } =20 /* @@ -263,7 +263,7 @@ static int graph_entry(struct ftrace_graph_ent *trace, =20 trace_ctx =3D tracing_gen_ctx(); if (IS_ENABLED(CONFIG_FUNCTION_GRAPH_RETADDR) && - tracer_flags_is_set(TRACE_GRAPH_PRINT_RETADDR)) { + tracer_flags_is_set(tr, TRACE_GRAPH_PRINT_RETADDR)) { unsigned long retaddr =3D ftrace_graph_top_ret_addr(current); ret =3D __trace_graph_retaddr_entry(tr, trace, trace_ctx, retaddr); } else { @@ -441,7 +441,7 @@ static int graph_trace_init(struct trace_array *tr) { int ret; =20 - if (tracer_flags_is_set(TRACE_GRAPH_ARGS)) + if (tracer_flags_is_set(tr, TRACE_GRAPH_ARGS)) tr->gops->entryfunc =3D trace_graph_entry_args; else tr->gops->entryfunc =3D trace_graph_entry; @@ -1459,7 +1459,8 @@ print_graph_function_flags(struct trace_iterator *ite= r, u32 flags) static enum print_line_t print_graph_function(struct trace_iterator *iter) { - return print_graph_function_flags(iter, tracer_flags.val); + struct trace_array *tr =3D iter->tr; + return print_graph_function_flags(iter, tr->current_trace_flags->val); } =20 static enum print_line_t @@ -1535,7 +1536,10 @@ static void __print_graph_headers_flags(struct trace= _array *tr, =20 static void print_graph_headers(struct seq_file *s) { - print_graph_headers_flags(s, tracer_flags.val); + struct trace_iterator *iter =3D s->private; + struct trace_array *tr =3D iter->tr; + + print_graph_headers_flags(s, tr->current_trace_flags->val); } =20 void print_graph_headers_flags(struct seq_file *s, u32 flags) @@ -1660,7 +1664,7 @@ static struct tracer graph_trace __tracer_data =3D { .reset =3D graph_trace_reset, .print_line =3D print_graph_function, .print_header =3D print_graph_headers, - .flags =3D &tracer_flags, + .default_flags =3D &tracer_flags, .set_flag =3D func_graph_set_flag, .allow_instances =3D true, #ifdef CONFIG_FTRACE_SELFTEST --=20 2.51.0