From nobody Mon Dec 1 22:35:41 2025 Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4DE6342525; Wed, 26 Nov 2025 20:23:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=216.40.44.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764188603; cv=none; b=KqUmUDWm7pUJjNUIbcvHXjewjPO/qV3DuEWb9bpd6asctIzj22z8jA+JgF0RAYWSoO5Rel5GihUhzZpCd+5/+0VJVpl7aFROSgpiyAvVfakC5s4iax7XJjeCBNrrdz5Nwmv6ukgLDdb/VXHXEP3BWYGV4haeKjzmW9ZGUKLVUEU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764188603; c=relaxed/simple; bh=Q7PxFDH/V3y4b2jU6jA0143NeJJPCK+NnK+WQz2cE0Y=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type; b=S4Zg/8NV45JirrlBuhrfHkLtWveNnHGAmYvc3FELGin2HlNscddBRL/VKz2/Z2ZAlf8ou4h1nqeVELRCYB28TL9Ul9GcLXEPjlQGLjlQxdlgdDgmvVj4CofjVot74c6IeQzOafny6N++uSJQSS9K8C7OLoJbxERntcbGHBlRbFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=goodmis.org; spf=pass smtp.mailfrom=goodmis.org; arc=none smtp.client-ip=216.40.44.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=goodmis.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=goodmis.org Received: from omf18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 656DFC076D; Wed, 26 Nov 2025 20:23:19 +0000 (UTC) Received: from [HIDDEN] (Authenticated sender: rostedt@goodmis.org) by omf18.hostedemail.com (Postfix) with ESMTPA id B79962E; Wed, 26 Nov 2025 20:23:17 +0000 (UTC) Date: Wed, 26 Nov 2025 15:24:04 -0500 From: Steven Rostedt To: LKML , Linux Trace Kernel Cc: Masami Hiramatsu , Mathieu Desnoyers , Tom Zanussi Subject: [PATCH] tracing: Add system trigger file to enable triggers for all the system's events Message-ID: <20251126152404.21a8e3c9@gandalf.local.home> X-Mailer: Claws Mail 3.20.0git84 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspamout06 X-Rspamd-Queue-Id: B79962E X-Stat-Signature: cpdrio8wbs96k3gdip8tykw3ue66rawp X-Session-Marker: 726F737465647440676F6F646D69732E6F7267 X-Session-ID: U2FsdGVkX18oVdf3MazKohrpSzNPZJ5Tf85ntCeG1Ms= X-HE-Tag: 1764188597-272988 X-HE-Meta: U2FsdGVkX1/uyLhVuB9lpLRRKCFfDM5H/M62rD/BWH+i5scTRcCwK00cmSH+LpLVEkKXhEpbZQpfMS5zvhH567FEa1IvBfpetYDFnNKHKWi7w2e3sldxHjSZoHXHtEHVuC8VcKBUfnDJb6LUwQQy5mMRqzYSsP8Yb/AUlBHKsyYEuoLCSbas4RiKjqwghOwftUPQARiMrY0DiBQsjTIWgYpq2xXv1FyEV4cv+4pD05AwU/HkT0VMkNUs0Xh2P4sWofdh3tffCe3uiu/GNCdmceCtLAcYZLbL4bCg5+cmrpdszDTAPrYM5R+EKYE8UHsRJbyuha6yvw8KPWKGokVtYSEvJaARxZ5S9NPwo9fSNmq6LEf3wo526gz20PXAdiq+whOkXoCtD03mJdWpLeWXt3RZ2SfBWCII63uz16S+KKaaf4GKtGZbIR5VR1QemGJ10k41Wy5aGj1IrUbqXP8vRQ== Content-Type: text/plain; charset="utf-8" From: Steven Rostedt Currently a trigger can only be added to individual events. Some triggers (like stacktrace) can be useful to add as a bulk trigger for a set of system events (like interrupt or scheduling). Add a trigger file to the system directories: /sys/kernel/tracing/events/*/trigger And allow stacktrace trigger to be enabled for all those events. Writing into the system/trigger file acts the same as writing into each of the system event's trigger files individually. This also allows to remove a trigger from all events in a subsystem (even if it's not a subsystem trigger!). Signed-off-by: Steven Rostedt (Google) --- Changes since v2: https://patch.msgid.link/20251125200837.31aae207@gandalf.= local.home - Removed unneeded NULL initialization of tr (Masami Hiramatsu) Documentation/trace/events.rst | 25 ++++ kernel/trace/trace.c | 11 +- kernel/trace/trace.h | 15 ++- kernel/trace/trace_events.c | 70 +++++----- kernel/trace/trace_events_trigger.c | 199 +++++++++++++++++++++++++++- 5 files changed, 278 insertions(+), 42 deletions(-) diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst index 18d112963dec..caa4958af43a 100644 --- a/Documentation/trace/events.rst +++ b/Documentation/trace/events.rst @@ -416,6 +416,31 @@ way, so beware about making generalizations between th= e two. can also enable triggers that are written into /sys/kernel/tracing/events/ftrace/print/trigger =20 +The system directory also has a trigger file that allows some triggers to = be +set for all the system's events. This is limited to only a small subset of= the +triggers and does not allow for the count parameter. But it does allow for +filters. Writing into this file is the same as writing into each of the +system's event's trigger files individually. Although only a subset of +triggers may use this file for enabling, all triggers may use this file for +disabling:: + + cd /sys/kernel/tracing + cat events/sched/trigger + # Available system triggers: + # stacktrace + + echo stacktrace > events/sched/trigger + cat events/sched/sched_switch/trigger + stacktrace:unlimited + + echo snapshot > events/sched/sched_waking/trigger + cat events/sched/sched_waking/trigger + snapshot:unlimited + echo '!snapshot' > events/sched/trigger + cat events/sched/sched_waking/trigger + # Available triggers: + # traceon traceoff snapshot stacktrace enable_event disable_event enable_= hist disable_hist hist + 6.1 Expression syntax --------------------- =20 diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 73f8b79f1b0c..f59645ab5140 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -592,11 +592,12 @@ void trace_set_ring_buffer_expanded(struct trace_arra= y *tr) =20 LIST_HEAD(ftrace_trace_arrays); =20 -int trace_array_get(struct trace_array *this_tr) +int __trace_array_get(struct trace_array *this_tr) { struct trace_array *tr; =20 - guard(mutex)(&trace_types_lock); + lockdep_assert_held(&trace_types_lock); + list_for_each_entry(tr, &ftrace_trace_arrays, list) { if (tr =3D=3D this_tr) { tr->ref++; @@ -607,6 +608,12 @@ int trace_array_get(struct trace_array *this_tr) return -ENODEV; } =20 +int trace_array_get(struct trace_array *tr) +{ + guard(mutex)(&trace_types_lock); + return __trace_array_get(tr); +} + static void __trace_array_put(struct trace_array *this_tr) { WARN_ON(!this_tr->ref); diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index c2b61bcd912f..c4d6074e184c 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -471,10 +471,14 @@ extern struct list_head ftrace_trace_arrays; extern struct mutex trace_types_lock; =20 extern int trace_array_get(struct trace_array *tr); +extern int __trace_array_get(struct trace_array *tr); extern int tracing_check_open_get_tr(struct trace_array *tr); extern struct trace_array *trace_array_find(const char *instance); extern struct trace_array *trace_array_find_get(const char *instance); =20 +extern struct trace_subsystem_dir *trace_get_system_dir(struct inode *inod= e); +void trace_put_system_dir(struct trace_subsystem_dir *dir); + extern u64 tracing_event_time_stamp(struct trace_buffer *buffer, struct ri= ng_buffer_event *rbe); extern int tracing_set_filter_buffering(struct trace_array *tr, bool set); extern int tracing_set_clock(struct trace_array *tr, const char *clockstr); @@ -1777,6 +1781,7 @@ static inline struct trace_event_file *event_file_fil= e(struct file *filp) } =20 extern const struct file_operations event_trigger_fops; +extern const struct file_operations event_system_trigger_fops; extern const struct file_operations event_hist_fops; extern const struct file_operations event_hist_debug_fops; extern const struct file_operations event_inject_fops; @@ -2060,10 +2065,16 @@ struct event_command { * regardless of whether or not it has a filter associated with * it (filters make a trigger require access to the trace record * but are not always present). + * + * @SYSTEM: A flag that says whether or not this command can be used + * at the event system level. For example, can it be written into + * events/sched/trigger file where it will be enabled for all + * sched events? */ enum event_command_flags { - EVENT_CMD_FL_POST_TRIGGER =3D 1, - EVENT_CMD_FL_NEEDS_REC =3D 2, + EVENT_CMD_FL_POST_TRIGGER =3D BIT(1), + EVENT_CMD_FL_NEEDS_REC =3D BIT(2), + EVENT_CMD_FL_SYSTEM =3D BIT(3), }; =20 static inline bool event_command_post_trigger(struct event_command *cmd_op= s) diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c index 9b07ad9eb284..5cbbcd86cef0 100644 --- a/kernel/trace/trace_events.c +++ b/kernel/trace/trace_events.c @@ -2168,51 +2168,52 @@ event_filter_write(struct file *filp, const char __= user *ubuf, size_t cnt, =20 static LIST_HEAD(event_subsystems); =20 -static int subsystem_open(struct inode *inode, struct file *filp) +struct trace_subsystem_dir *trace_get_system_dir(struct inode *inode) { - struct trace_subsystem_dir *dir =3D NULL, *iter_dir; - struct trace_array *tr =3D NULL, *iter_tr; - struct event_subsystem *system =3D NULL; - int ret; + struct trace_subsystem_dir *dir; + struct trace_array *tr; =20 - if (tracing_is_disabled()) - return -ENODEV; + guard(mutex)(&event_mutex); + guard(mutex)(&trace_types_lock); =20 /* Make sure the system still exists */ - mutex_lock(&event_mutex); - mutex_lock(&trace_types_lock); - list_for_each_entry(iter_tr, &ftrace_trace_arrays, list) { - list_for_each_entry(iter_dir, &iter_tr->systems, list) { - if (iter_dir =3D=3D inode->i_private) { + list_for_each_entry(tr, &ftrace_trace_arrays, list) { + list_for_each_entry(dir, &tr->systems, list) { + if (dir =3D=3D inode->i_private) { /* Don't open systems with no events */ - tr =3D iter_tr; - dir =3D iter_dir; - if (dir->nr_events) { - __get_system_dir(dir); - system =3D dir->subsystem; - } - goto exit_loop; + if (!dir->nr_events) + return NULL; + if (__trace_array_get(tr) < 0) + return NULL; + __get_system_dir(dir); + return dir; } } } - exit_loop: - mutex_unlock(&trace_types_lock); - mutex_unlock(&event_mutex); + return NULL; +} =20 - if (!system) +void trace_put_system_dir(struct trace_subsystem_dir *dir) +{ + trace_array_put(dir->tr); + put_system(dir); +} + +static int subsystem_open(struct inode *inode, struct file *filp) +{ + struct trace_subsystem_dir *dir; + int ret; + + if (tracing_is_disabled()) return -ENODEV; =20 - /* Still need to increment the ref count of the system */ - if (trace_array_get(tr) < 0) { - put_system(dir); + dir =3D trace_get_system_dir(inode); + if (!dir) return -ENODEV; - } =20 ret =3D tracing_open_generic(inode, filp); - if (ret < 0) { - trace_array_put(tr); - put_system(dir); - } + if (ret < 0) + trace_put_system_dir(dir); =20 return ret; } @@ -2761,6 +2762,9 @@ static int system_callback(const char *name, umode_t = *mode, void **data, else if (strcmp(name, "enable") =3D=3D 0) *fops =3D &ftrace_system_enable_fops; =20 + else if (strcmp(name, "trigger") =3D=3D 0) + *fops =3D &event_system_trigger_fops; + else return 0; =20 @@ -2784,6 +2788,10 @@ event_subsystem_dir(struct trace_array *tr, const ch= ar *name, { .name =3D "enable", .callback =3D system_callback, + }, + { + .name =3D "trigger", + .callback =3D system_callback, } }; =20 diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_event= s_trigger.c index 96aad82b1628..b69b906fb620 100644 --- a/kernel/trace/trace_events_trigger.c +++ b/kernel/trace/trace_events_trigger.c @@ -328,21 +328,28 @@ int trigger_process_regex(struct trace_event_file *fi= le, char *buff) return -EINVAL; } =20 +static char *get_user_buf(const char __user *ubuf, size_t cnt) +{ + if (!cnt) + return NULL; + + if (cnt >=3D PAGE_SIZE) + return ERR_PTR(-EINVAL); + + return memdup_user_nul(ubuf, cnt); +} + static ssize_t event_trigger_regex_write(struct file *file, const char __user *ubuf, size_t cnt, loff_t *ppos) { struct trace_event_file *event_file; ssize_t ret; - char *buf __free(kfree) =3D NULL; + char *buf __free(kfree) =3D get_user_buf(ubuf, cnt); =20 - if (!cnt) + if (!buf) return 0; =20 - if (cnt >=3D PAGE_SIZE) - return -EINVAL; - - buf =3D memdup_user_nul(ubuf, cnt); if (IS_ERR(buf)) return PTR_ERR(buf); =20 @@ -396,6 +403,184 @@ const struct file_operations event_trigger_fops =3D { .release =3D event_trigger_release, }; =20 +static ssize_t +event_system_trigger_read(struct file *filp, char __user *ubuf, + size_t count, loff_t *ppos) +{ + char *buf __free(kfree) =3D kmalloc(SZ_4K, GFP_KERNEL); + struct event_command *p; + struct seq_buf s; + int len; + + if (!buf) + return -ENOMEM; + + seq_buf_init(&s, buf, SZ_4K); + + seq_buf_puts(&s, "# Available system triggers:\n"); + seq_buf_putc(&s, '#'); + + guard(mutex)(&trigger_cmd_mutex); + list_for_each_entry_reverse(p, &trigger_commands, list) { + if (p->flags & EVENT_CMD_FL_SYSTEM) + seq_buf_printf(&s, " %s", p->name); + } + seq_buf_putc(&s, '\n'); + + len =3D seq_buf_used(&s); + + if (*ppos >=3D len) + return 0; + + len -=3D *ppos; + + if (count > len) + count =3D len; + + if (copy_to_user(ubuf, buf + *ppos, count)) + return -EFAULT; + + *ppos +=3D count; + + return count; +} + +static int process_system_events(struct trace_subsystem_dir *dir, + struct event_command *p, char *buff, + char *command, char *next) +{ + struct event_subsystem *system =3D dir->subsystem; + struct trace_event_file *file; + struct trace_array *tr =3D dir->tr; + bool remove =3D false; + int ret =3D 0; + + if (buff[0] =3D=3D '!') + remove =3D true; + + lockdep_assert_held(&event_mutex); + + list_for_each_entry(file, &tr->events, list) { + + if (strcmp(system->name, file->event_call->class->system) !=3D 0) + continue; + + ret =3D p->parse(p, file, buff, command, next); + + /* Removals and existing events do not error */ + if (ret < 0 && ret !=3D -EEXIST && !remove) { + pr_warn("Failed adding trigger %s on %s\n", + command, trace_event_name(file->event_call)); + } + } + return 0; +} + +static ssize_t +event_system_trigger_write(struct file *filp, const char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + struct trace_subsystem_dir *dir =3D filp->private_data; + struct event_command *p; + char *command, *next; + char *buf __free(kfree) =3D get_user_buf(ubuf, cnt); + bool remove =3D false; + bool found =3D false; + ssize_t ret; + + if (!buf) + return 0; + + if (IS_ERR(buf)) + return PTR_ERR(buf); + + /* system triggers are not allowed to have counters */ + if (strchr(buf, ':')) + return -EINVAL; + + /* If opened for read too, dir is in the seq_file descriptor */ + if (filp->f_mode & FMODE_READ) { + struct seq_file *m =3D filp->private_data; + dir =3D m->private; + } + + /* Skip added space at beginning of buf */ + next =3D strim(buf); + + command =3D strsep(&next, " \t"); + if (next) { + next =3D skip_spaces(next); + if (!*next) + next =3D NULL; + } + if (command[0] =3D=3D '!') { + remove =3D true; + command++; + } + + guard(mutex)(&event_mutex); + guard(mutex)(&trigger_cmd_mutex); + + list_for_each_entry(p, &trigger_commands, list) { + /* Allow to remove any trigger */ + if (!remove && !(p->flags & EVENT_CMD_FL_SYSTEM)) + continue; + if (strcmp(p->name, command) =3D=3D 0) { + found =3D true; + ret =3D process_system_events(dir, p, buf, command, next); + break; + } + } + + if (!found) + ret =3D -ENODEV; + + if (!ret) + *ppos +=3D cnt; + + if (remove || ret < 0) + return ret ? : cnt; + + return cnt; +} + +static int +event_system_trigger_open(struct inode *inode, struct file *file) +{ + struct trace_subsystem_dir *dir; + int ret; + + ret =3D security_locked_down(LOCKDOWN_TRACEFS); + if (ret) + return ret; + + dir =3D trace_get_system_dir(inode); + if (!dir) + return -ENODEV; + + file->private_data =3D dir; + + return ret; +} + +static int +event_system_trigger_release(struct inode *inode, struct file *file) +{ + struct trace_subsystem_dir *dir =3D inode->i_private; + + trace_put_system_dir(dir); + + return 0; +} + +const struct file_operations event_system_trigger_fops =3D { + .open =3D event_system_trigger_open, + .read =3D event_system_trigger_read, + .write =3D event_system_trigger_write, + .llseek =3D tracing_lseek, + .release =3D event_system_trigger_release, +}; + /* * Currently we only register event commands from __init, so mark this * __init too. @@ -1586,7 +1771,7 @@ stacktrace_trigger_print(struct seq_file *m, struct e= vent_trigger_data *data) static struct event_command trigger_stacktrace_cmd =3D { .name =3D "stacktrace", .trigger_type =3D ETT_STACKTRACE, - .flags =3D EVENT_CMD_FL_POST_TRIGGER, + .flags =3D EVENT_CMD_FL_POST_TRIGGER | EVENT_CMD_FL_SYSTEM, .parse =3D event_trigger_parse, .reg =3D register_trigger, .unreg =3D unregister_trigger, --=20 2.51.0