From nobody Mon Oct 6 13:35:27 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABAAB230BFF for ; Mon, 21 Jul 2025 23:11:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753139511; cv=none; b=ja76KQUYlLhWd6k2oMBwORGNH0R5K9eMKRF8M7etiT0tY4O6aalDclaZ8dUT6a519amQEGurVOt5XG++Zz7ileZyn+BVem9xJpmYl0JhKffHwIPOaJ8e4xt56Q/RRKandZZ+WnwF7YKEmvB2MmeQtfpMp/6R3U+cac3msB3q7JY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753139511; c=relaxed/simple; bh=G2RJ51kWmJy8O6Pp/VgFTe/xKhVrmKlhuQsGwc6Z7dA=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=aD2HmrQlnXvwfTW3EOb/jcZiGCq7RIIFQB/wSwxYbf6OAg8iJmYlr4f8To5I/vuTmpq3/el96mJIWzRSzzVgQjTWpozCUO+7Te7z8n5Xj0igWtnEf2NV3eKK8j84++ULg6ki8pY7/yaNTF8znCmQYF0i7zHC0sb1goIHO1d/kw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fGXB9JR+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fGXB9JR+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A4D9C4CEF8; Mon, 21 Jul 2025 23:11:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753139511; bh=G2RJ51kWmJy8O6Pp/VgFTe/xKhVrmKlhuQsGwc6Z7dA=; h=Date:From:To:Cc:Subject:References:From; b=fGXB9JR+bsqlRZ3xURsEQEvYeoOruFwzlpD/XxgkLXGFCzIj+Ltidyl67legwGkd4 9z6DuaGN3LXkp0+Vgvc6DT446g8Xrbru0HFIdsv1FKPvlxyjChQDBsbRh1fykxmcjV 87pabuNG8A2eg2SHPX1BrXCVPndFaZ11WxmhEP7gKgs03SmQZZ8K6rIrrmQivWl0Ro ykzX1W5V1K8PcEZCaCH2G78rFZpE5hRRvi2dxfqUz0OfQhmazJnIcK8qD0kxZt5hZD UbD/Kfhob/jGh/dHao50lwEd9p7b3iRmD+rNFD6Pf48/xbc42lZztjezFIpFmi2zDi ndKxg8Djq8IMQ== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1udzgE-00000009vHu-2YSO; Mon, 21 Jul 2025 19:12:22 -0400 Message-ID: <20250721231222.462764521@kernel.org> User-Agent: quilt/0.68 Date: Mon, 21 Jul 2025 19:12:06 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org Cc: Tomas Glozar , John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco Subject: [for-next][PATCH 2/9] rtla/timerlat: Add action on threshold feature References: <20250721231204.100737734@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tomas Glozar Extend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support implementing arbitrary actions. A new option, --on-threshold, is added, taking an argument that further specifies the action. Actions added in this patch are: - trace[,file=3D]: Saves tracefs buffer, optionally taking a filename. - signal,num=3D,pid=3D: Sends signal to process. "parent" might be specified instead of number to send signal to parent process. - shell,command=3D: Execute shell command. Multiple actions may be specified and will be executed in order, including multiple actions of the same type. Trace output requested via -t and -a now adds a trace action to the end of the list. If an action fails, the following actions are not executed. For example, this command: $ rtla timerlat -T 20 --on-threshold trace \ --on-threshold shell,command=3D"grep ipi_send timerlat_trace.txt" \ --on-threshold signal,num=3D2,pid=3Dparent will send signal 2 (SIGINT) to parent process, but only if saved trace contains the text "ipi_send". This way, the feature can be used for flexible reactions on latency spikes, and allows combining rtla with other tooling like perf. Cc: John Kacur Cc: Luis Goncalves Cc: Arnaldo Carvalho de Melo Cc: Chang Yin Cc: Costa Shulyupin Cc: Crystal Wood Cc: Gabriele Monaco Link: https://lore.kernel.org/20250626123405.1496931-3-tglozar@redhat.com Signed-off-by: Tomas Glozar Signed-off-by: Steven Rostedt (Google) --- tools/tracing/rtla/src/Build | 1 + tools/tracing/rtla/src/actions.c | 235 +++++++++++++++++++++++++ tools/tracing/rtla/src/actions.h | 49 ++++++ tools/tracing/rtla/src/timerlat.h | 3 +- tools/tracing/rtla/src/timerlat_hist.c | 37 ++-- tools/tracing/rtla/src/timerlat_top.c | 38 ++-- 6 files changed, 341 insertions(+), 22 deletions(-) create mode 100644 tools/tracing/rtla/src/actions.c create mode 100644 tools/tracing/rtla/src/actions.h diff --git a/tools/tracing/rtla/src/Build b/tools/tracing/rtla/src/Build index 7bb7e39e391a..66631280b75b 100644 --- a/tools/tracing/rtla/src/Build +++ b/tools/tracing/rtla/src/Build @@ -1,5 +1,6 @@ rtla-y +=3D trace.o rtla-y +=3D utils.o +rtla-y +=3D actions.o rtla-y +=3D osnoise.o rtla-y +=3D osnoise_top.o rtla-y +=3D osnoise_hist.o diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/acti= ons.c new file mode 100644 index 000000000000..63bee5bdabfd --- /dev/null +++ b/tools/tracing/rtla/src/actions.c @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include "actions.h" +#include "trace.h" +#include "utils.h" + +/* + * actions_init - initialize struct actions + */ +void +actions_init(struct actions *self) +{ + self->size =3D action_default_size; + self->list =3D calloc(self->size, sizeof(struct action)); + self->len =3D 0; + + memset(&self->present, 0, sizeof(self->present)); + + /* This has to be set by the user */ + self->trace_output_inst =3D NULL; +} + +/* + * actions_destroy - destroy struct actions + */ +void +actions_destroy(struct actions *self) +{ + /* Free any action-specific data */ + for (struct action *action =3D self->list; action < self->list + self->le= n; action++) { + if (action->type =3D=3D ACTION_SHELL) + free(action->command); + if (action->type =3D=3D ACTION_TRACE_OUTPUT) + free(action->trace_output); + } + + /* Free action list */ + free(self->list); +} + +/* + * actions_new - Get pointer to new action + */ +static struct action * +actions_new(struct actions *self) +{ + if (self->size >=3D self->len) { + self->size *=3D 2; + self->list =3D realloc(self->list, self->size * sizeof(struct action)); + } + + return &self->list[self->len++]; +} + +/* + * actions_add_trace_output - add an action to output trace + */ +int +actions_add_trace_output(struct actions *self, const char *trace_output) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_TRACE_OUTPUT] =3D true; + action->type =3D ACTION_TRACE_OUTPUT; + action->trace_output =3D calloc(strlen(trace_output) + 1, sizeof(char)); + if (!action->trace_output) + return -1; + strcpy(action->trace_output, trace_output); + + return 0; +} + +/* + * actions_add_trace_output - add an action to send signal to a process + */ +int +actions_add_signal(struct actions *self, int signal, int pid) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_SIGNAL] =3D true; + action->type =3D ACTION_SIGNAL; + action->signal =3D signal; + action->pid =3D pid; + + return 0; +} + +/* + * actions_add_shell - add an action to execute a shell command + */ +int +actions_add_shell(struct actions *self, const char *command) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_SHELL] =3D true; + action->type =3D ACTION_SHELL; + action->command =3D calloc(strlen(command) + 1, sizeof(char)); + if (!action->command) + return -1; + strcpy(action->command, command); + + return 0; +} + +/* + * actions_parse - add an action based on text specification + */ +int +actions_parse(struct actions *self, const char *trigger) +{ + enum action_type type =3D ACTION_NONE; + char *token; + char trigger_c[strlen(trigger)]; + + /* For ACTION_SIGNAL */ + int signal =3D 0, pid =3D 0; + + /* For ACTION_TRACE_OUTPUT */ + char *trace_output; + + strcpy(trigger_c, trigger); + token =3D strtok(trigger_c, ","); + + if (strcmp(token, "trace") =3D=3D 0) + type =3D ACTION_TRACE_OUTPUT; + else if (strcmp(token, "signal") =3D=3D 0) + type =3D ACTION_SIGNAL; + else if (strcmp(token, "shell") =3D=3D 0) + type =3D ACTION_SHELL; + else + /* Invalid trigger type */ + return -1; + + token =3D strtok(NULL, ","); + + switch (type) { + case ACTION_TRACE_OUTPUT: + /* Takes no argument */ + if (token =3D=3D NULL) + trace_output =3D "timerlat_trace.txt"; + else { + if (strlen(token) > 5 && strncmp(token, "file=3D", 5) =3D=3D 0) { + trace_output =3D token + 5; + } else { + /* Invalid argument */ + return -1; + } + + token =3D strtok(NULL, ","); + if (token !=3D NULL) + /* Only one argument allowed */ + return -1; + } + return actions_add_trace_output(self, trace_output); + case ACTION_SIGNAL: + /* Takes two arguments, num (signal) and pid */ + while (token !=3D NULL) { + if (strlen(token) > 4 && strncmp(token, "num=3D", 4) =3D=3D 0) { + signal =3D atoi(token + 4); + } else if (strlen(token) > 4 && strncmp(token, "pid=3D", 4) =3D=3D 0) { + if (strncmp(token + 4, "parent", 7) =3D=3D 0) + pid =3D -1; + else + pid =3D atoi(token + 4); + } else { + /* Invalid argument */ + return -1; + } + + token =3D strtok(NULL, ","); + } + + if (!signal || !pid) + /* Missing argument */ + return -1; + + return actions_add_signal(self, signal, pid); + case ACTION_SHELL: + if (token =3D=3D NULL) + return -1; + if (strlen(token) > 8 && strncmp(token, "command=3D", 8) =3D=3D 0) + return actions_add_shell(self, token + 8); + return -1; + default: + return -1; + } +} + +/* + * actions_perform - perform all actions + */ +int +actions_perform(const struct actions *self) +{ + int pid, retval; + const struct action *action; + + for (action =3D self->list; action < self->list + self->len; action++) { + switch (action->type) { + case ACTION_TRACE_OUTPUT: + retval =3D save_trace_to_file(self->trace_output_inst, action->trace_ou= tput); + if (retval) { + err_msg("Error saving trace\n"); + return retval; + } + break; + case ACTION_SIGNAL: + if (action->pid =3D=3D -1) + pid =3D getppid(); + else + pid =3D action->pid; + retval =3D kill(pid, action->signal); + if (retval) { + err_msg("Error sending signal\n"); + return retval; + } + break; + case ACTION_SHELL: + retval =3D system(action->command); + if (retval) + return retval; + break; + default: + break; + } + } + + return 0; +} diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/acti= ons.h new file mode 100644 index 000000000000..076bbff8519e --- /dev/null +++ b/tools/tracing/rtla/src/actions.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include +#include + +enum action_type { + ACTION_NONE =3D 0, + ACTION_TRACE_OUTPUT, + ACTION_SIGNAL, + ACTION_SHELL, + ACTION_FIELD_N +}; + +struct action { + enum action_type type; + union { + struct { + /* For ACTION_TRACE_OUTPUT */ + char *trace_output; + }; + struct { + /* For ACTION_SIGNAL */ + int signal; + int pid; + }; + struct { + /* For ACTION_SHELL */ + char *command; + }; + }; +}; + +static const int action_default_size =3D 8; + +struct actions { + struct action *list; + int len, size; + bool present[ACTION_FIELD_N]; + + /* External dependencies */ + struct tracefs_instance *trace_output_inst; +}; + +void actions_init(struct actions *self); +void actions_destroy(struct actions *self); +int actions_add_trace_output(struct actions *self, const char *trace_outpu= t); +int actions_add_signal(struct actions *self, int signal, int pid); +int actions_add_shell(struct actions *self, const char *command); +int actions_parse(struct actions *self, const char *trigger); +int actions_perform(const struct actions *self); diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/tim= erlat.h index e0a553545d03..d1fcf9a97621 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include "actions.h" #include "osnoise.h" =20 /* @@ -22,7 +23,6 @@ struct timerlat_params { /* Common params */ char *cpus; cpu_set_t monitored_cpus; - char *trace_output; char *cgroup_name; unsigned long long runtime; long long stop_us; @@ -48,6 +48,7 @@ struct timerlat_params { struct sched_attr sched_param; struct trace_events *events; enum timerlat_tracing_mode mode; + struct actions actions; union { struct { /* top only */ diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 6cf260e8553b..d975d2cd6604 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -757,6 +757,7 @@ static void timerlat_hist_usage(char *usage) " --warm-up s: let the workload run for s seconds before collecting= data", " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", + " --on-threshold : define action to be executed at latency = threshold, multiple are allowed", NULL, }; =20 @@ -786,11 +787,14 @@ static struct timerlat_params int auto_thresh; int retval; int c; + char *trace_output =3D NULL; =20 params =3D calloc(1, sizeof(*params)); if (!params) exit(1); =20 + actions_init(¶ms->actions); + /* disabled by default */ params->dma_latency =3D -1; =20 @@ -841,6 +845,7 @@ static struct timerlat_params {"warm-up", required_argument, 0, '\2'}, {"trace-buffer-size", required_argument, 0, '\3'}, {"deepest-idle-state", required_argument, 0, '\4'}, + {"on-threshold", required_argument, 0, '\5'}, {0, 0, 0, 0} }; =20 @@ -866,7 +871,7 @@ static struct timerlat_params params->print_stack =3D auto_thresh; =20 /* set trace */ - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; =20 break; case 'c': @@ -956,13 +961,13 @@ static struct timerlat_params case 't': if (optarg) { if (optarg[0] =3D=3D '=3D') - params->trace_output =3D &optarg[1]; + trace_output =3D &optarg[1]; else - params->trace_output =3D &optarg[0]; + trace_output =3D &optarg[0]; } else if (optind < argc && argv[optind][0] !=3D '-') - params->trace_output =3D argv[optind]; + trace_output =3D argv[optind]; else - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; break; case 'u': params->user_workload =3D 1; @@ -1032,11 +1037,21 @@ static struct timerlat_params case '\4': params->deepest_idle_state =3D get_llong_from_str(optarg); break; + case '\5': + retval =3D actions_parse(¶ms->actions, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } + break; default: timerlat_hist_usage("Invalid option"); } } =20 + if (trace_output) + actions_add_trace_output(¶ms->actions, trace_output); + if (geteuid()) { err_msg("rtla needs root permission\n"); exit(EXIT_FAILURE); @@ -1061,7 +1076,8 @@ static struct timerlat_params * If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ - if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + if (params->mode =3D=3D TRACING_MODE_BPF && + (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1254,12 +1270,13 @@ int timerlat_hist_main(int argc, char *argv[]) } } =20 - if (params->trace_output) { + if (params->actions.present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); goto out_free; } + params->actions.trace_output_inst =3D record->trace.inst; =20 if (params->events) { retval =3D trace_events_enable(&record->trace, params->events); @@ -1325,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->trace_output) + if (params->actions.present[ACTION_TRACE_OUTPUT]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1395,8 +1412,7 @@ int timerlat_hist_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - save_trace_to_file(record ? record->trace.inst : NULL, - params->trace_output); + actions_perform(¶ms->actions); return_value =3D FAILED; } =20 @@ -1418,6 +1434,7 @@ int timerlat_hist_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(tool); + actions_destroy(¶ms->actions); if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); free(params); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 1644eeb60181..cdbfda009974 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -516,6 +516,7 @@ static void timerlat_top_usage(char *usage) " --warm-up s: let the workload run for s seconds before collecting= data", " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", + " --on-threshold : define action to be executed at latency = threshold, multiple are allowed", NULL, }; =20 @@ -545,11 +546,14 @@ static struct timerlat_params long long auto_thresh; int retval; int c; + char *trace_output =3D NULL; =20 params =3D calloc(1, sizeof(*params)); if (!params) exit(1); =20 + actions_init(¶ms->actions); + /* disabled by default */ params->dma_latency =3D -1; =20 @@ -592,6 +596,7 @@ static struct timerlat_params {"warm-up", required_argument, 0, '6'}, {"trace-buffer-size", required_argument, 0, '7'}, {"deepest-idle-state", required_argument, 0, '8'}, + {"on-threshold", required_argument, 0, '9'}, {0, 0, 0, 0} }; =20 @@ -617,7 +622,7 @@ static struct timerlat_params params->print_stack =3D auto_thresh; =20 /* set trace */ - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; break; case '5': /* it is here because it is similar to -a */ @@ -712,14 +717,13 @@ static struct timerlat_params case 't': if (optarg) { if (optarg[0] =3D=3D '=3D') - params->trace_output =3D &optarg[1]; + trace_output =3D &optarg[1]; else - params->trace_output =3D &optarg[0]; + trace_output =3D &optarg[0]; } else if (optind < argc && argv[optind][0] !=3D '-') - params->trace_output =3D argv[optind]; + trace_output =3D argv[optind]; else - params->trace_output =3D "timerlat_trace.txt"; - + trace_output =3D "timerlat_trace.txt"; break; case 'u': params->user_workload =3D true; @@ -771,11 +775,21 @@ static struct timerlat_params case '8': params->deepest_idle_state =3D get_llong_from_str(optarg); break; + case '9': + retval =3D actions_parse(¶ms->actions, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } + break; default: timerlat_top_usage("Invalid option"); } } =20 + if (trace_output) + actions_add_trace_output(¶ms->actions, trace_output); + if (geteuid()) { err_msg("rtla needs root permission\n"); exit(EXIT_FAILURE); @@ -797,7 +811,8 @@ static struct timerlat_params * If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ - if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + if (params->mode =3D=3D TRACING_MODE_BPF && + (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1099,12 +1114,13 @@ int timerlat_top_main(int argc, char *argv[]) } } =20 - if (params->trace_output) { + if (params->actions.present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); goto out_free; } + params->actions.trace_output_inst =3D record->trace.inst; =20 if (params->events) { retval =3D trace_events_enable(&record->trace, params->events); @@ -1171,7 +1187,7 @@ int timerlat_top_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->trace_output) + if (params->actions.present[ACTION_TRACE_OUTPUT]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1214,8 +1230,7 @@ int timerlat_top_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - save_trace_to_file(record ? record->trace.inst : NULL, - params->trace_output); + actions_perform(¶ms->actions); return_value =3D FAILED; } else if (params->aa_only) { /* @@ -1248,6 +1263,7 @@ int timerlat_top_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(top); + actions_destroy(¶ms->actions); if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); free(params); --=20 2.47.2