From nobody Sat Oct 11 08:29:14 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC98728C028 for ; Wed, 11 Jun 2025 13:57:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749650245; cv=none; b=Vm3vQDmEGc8rGFY+kxu30guiwRACAkswdSBDPS33tF4XyAzoSUrQXEzWdfWg3H/v1/MhOS3Sh5f2wPK44M/1FIL1MqepcSndA1gXd1KwSB9DpO1i16PscRd1865cd6b/c4sgQkd0jFfx16DaTb29vaAnyPNR4azHQtS7gygPK90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749650245; c=relaxed/simple; bh=BixDELRT6uG05RHpWncF2IbGIJnbavZUgZD5qePCxjE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PGRcL0myUWOgF34/GCCNVr6O7iTJAgYkGUAXB60+NVX1lML9L4RJQT7jdboTXqwOMHiHEpHdpkpDJqC0UMCMYJe5yjC8AtpfL47Hxl1QKwqC9aV8k/QXGKJmBluvVVoVfYMdtk7BVFQV0BZICzHQf49xazw90STUrNSfY5pyzlc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gYgt6jEU; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gYgt6jEU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1749650243; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ggVAEc+wazxmKfd8bKckrZJOLakgnftHGfBZZLcfyHs=; b=gYgt6jEUA+rAZKwlQuGy71g9j3F+P0Ggs5+GOFZpI/PeFMb0vc4A3+9qw4bswZIby9SThy sykbt0uPG16tX4HC6teklrMeaQlCt7evsGwDW7EVAFUCw8eZWK5vQPtOVyrbFCBQlCDRfB 8Ov+lFvmLLChPHSBRoZ/QezrfUY7Zuo= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-501-mutgKT0jOBK8hRWB_Frtag-1; Wed, 11 Jun 2025 09:57:17 -0400 X-MC-Unique: mutgKT0jOBK8hRWB_Frtag-1 X-Mimecast-MFC-AGG-ID: mutgKT0jOBK8hRWB_Frtag_1749650236 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id BE0D0180AD43; Wed, 11 Jun 2025 13:57:16 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.45.225.32]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A4D5C19560A3; Wed, 11 Jun 2025 13:57:13 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Tomas Glozar Subject: [PATCH 4/8] rtla/timerlat: Add continue action Date: Wed, 11 Jun 2025 15:56:40 +0200 Message-ID: <20250611135644.219127-5-tglozar@redhat.com> In-Reply-To: <20250611135644.219127-1-tglozar@redhat.com> References: <20250611135644.219127-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Introduce option to resume tracing after a latency threshold overflow. The option is implemented as an action, specified using "-A continue". Example: $ rtla timerlat top -q -T 200 -d 1s -A exec,command=3D"echo Threshold" \ -A continue Threshold Threshold Threshold Timer Latency ... The feature is supported for both hist and top. After the continue action is executed, processing of the list of actions is stopped and tracing is resumed. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/actions.c | 27 +++++++++++- tools/tracing/rtla/src/actions.h | 5 ++- tools/tracing/rtla/src/timerlat_hist.c | 40 +++++++++++++++--- tools/tracing/rtla/src/timerlat_top.c | 57 ++++++++++++++++---------- 4 files changed, 100 insertions(+), 29 deletions(-) diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/acti= ons.c index 63bee5bdabfd..aaf0808125d7 100644 --- a/tools/tracing/rtla/src/actions.c +++ b/tools/tracing/rtla/src/actions.c @@ -17,6 +17,7 @@ actions_init(struct actions *self) self->size =3D action_default_size; self->list =3D calloc(self->size, sizeof(struct action)); self->len =3D 0; + self->continue_flag =3D false; =20 memset(&self->present, 0, sizeof(self->present)); =20 @@ -108,6 +109,20 @@ actions_add_shell(struct actions *self, const char *co= mmand) return 0; } =20 +/* + * actions_add_continue - add an action to resume measurement + */ +int +actions_add_continue(struct actions *self) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_CONTINUE] =3D true; + action->type =3D ACTION_CONTINUE; + + return 0; +} + /* * actions_parse - add an action based on text specification */ @@ -133,6 +148,8 @@ actions_parse(struct actions *self, const char *trigger) type =3D ACTION_SIGNAL; else if (strcmp(token, "shell") =3D=3D 0) type =3D ACTION_SHELL; + else if (strcmp(token, "continue") =3D=3D 0) + type =3D ACTION_CONTINUE; else /* Invalid trigger type */ return -1; @@ -187,6 +204,11 @@ actions_parse(struct actions *self, const char *trigge= r) if (strlen(token) > 8 && strncmp(token, "command=3D", 8) =3D=3D 0) return actions_add_shell(self, token + 8); return -1; + case ACTION_CONTINUE: + /* Takes no argument */ + if (token !=3D NULL) + return -1; + return actions_add_continue(self); default: return -1; } @@ -196,7 +218,7 @@ actions_parse(struct actions *self, const char *trigger) * actions_perform - perform all actions */ int -actions_perform(const struct actions *self) +actions_perform(struct actions *self) { int pid, retval; const struct action *action; @@ -226,6 +248,9 @@ actions_perform(const struct actions *self) if (retval) return retval; break; + case ACTION_CONTINUE: + self->continue_flag =3D true; + return 0; default: break; } diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/acti= ons.h index 076bbff8519e..b10a19d55c49 100644 --- a/tools/tracing/rtla/src/actions.h +++ b/tools/tracing/rtla/src/actions.h @@ -7,6 +7,7 @@ enum action_type { ACTION_TRACE_OUTPUT, ACTION_SIGNAL, ACTION_SHELL, + ACTION_CONTINUE, ACTION_FIELD_N }; =20 @@ -35,6 +36,7 @@ struct actions { struct action *list; int len, size; bool present[ACTION_FIELD_N]; + bool continue_flag; =20 /* External dependencies */ struct tracefs_instance *trace_output_inst; @@ -45,5 +47,6 @@ void actions_destroy(struct actions *self); int actions_add_trace_output(struct actions *self, const char *trace_outpu= t); int actions_add_signal(struct actions *self, int signal, int pid); int actions_add_shell(struct actions *self, const char *command); +int actions_add_continue(struct actions *self); int actions_parse(struct actions *self, const char *trigger); -int actions_perform(const struct actions *self); +int actions_perform(struct actions *self); diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 06ee3d925ef9..1feae978344b 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -1375,8 +1375,20 @@ int timerlat_hist_main(int argc, char *argv[]) goto out_hist; } =20 - if (osnoise_trace_is_off(tool, record)) - break; + if (osnoise_trace_is_off(tool, record)) { + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + trace_instance_start(trace); + } =20 /* is there still any user-threads ? */ if (params->user_workload) { @@ -1386,8 +1398,27 @@ int timerlat_hist_main(int argc, char *argv[]) } } } - } else - timerlat_bpf_wait(-1); + } else { + while (!stop_tracing) { + timerlat_bpf_wait(-1); + + if (!stop_tracing) { + /* Threshold overflow, perform actions on threshold */ + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + timerlat_bpf_restart_tracing(); + } + } + } =20 if (params->mode !=3D TRACING_MODE_TRACEFS) { timerlat_bpf_detach(); @@ -1413,7 +1444,6 @@ int timerlat_hist_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - actions_perform(¶ms->actions); return_value =3D FAILED; } =20 diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index fa1072c0d126..ff0e6a7c3a1b 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -906,6 +906,7 @@ timerlat_top_set_signals(struct timerlat_params *params) static int timerlat_top_main_loop(struct osnoise_tool *top, struct osnoise_tool *record, + struct osnoise_tool *aa, struct timerlat_params *params, struct timerlat_u_params *params_u) { @@ -932,8 +933,20 @@ timerlat_top_main_loop(struct osnoise_tool *top, if (!params->quiet) timerlat_print_stats(params, top); =20 - if (osnoise_trace_is_off(top, record)) - break; + if (osnoise_trace_is_off(top, record)) { + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + trace_instance_start(trace); + } =20 /* is there still any user-threads ? */ if (params->user_workload) { @@ -953,6 +966,7 @@ timerlat_top_main_loop(struct osnoise_tool *top, static int timerlat_top_bpf_main_loop(struct osnoise_tool *top, struct osnoise_tool *record, + struct osnoise_tool *aa, struct timerlat_params *params, struct timerlat_u_params *params_u) { @@ -964,22 +978,9 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top, return 0; } =20 - if (params->quiet) { - /* Quiet mode: wait for stop and then, print results */ - timerlat_bpf_wait(-1); - - retval =3D timerlat_top_bpf_pull_data(top); - if (retval) { - err_msg("Error pulling BPF data\n"); - return retval; - } - - return 0; - } - /* Pull and display data in a loop */ while (!stop_tracing) { - wait_retval =3D timerlat_bpf_wait(params->sleep_time); + wait_retval =3D timerlat_bpf_wait(params->quiet ? -1 : params->sleep_tim= e); =20 retval =3D timerlat_top_bpf_pull_data(top); if (retval) { @@ -987,11 +988,24 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top, return retval; } =20 - timerlat_print_stats(params, top); + if (!params->quiet) + timerlat_print_stats(params, top); =20 - if (wait_retval =3D=3D 1) + if (wait_retval =3D=3D 1) { /* Stopping requested by tracer */ - break; + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + timerlat_bpf_restart_tracing(); + } =20 /* is there still any user-threads ? */ if (params->user_workload) { @@ -1205,9 +1219,9 @@ int timerlat_top_main(int argc, char *argv[]) timerlat_top_set_signals(params); =20 if (params->mode =3D=3D TRACING_MODE_TRACEFS) - retval =3D timerlat_top_main_loop(top, record, params, ¶ms_u); + retval =3D timerlat_top_main_loop(top, record, aa, params, ¶ms_u); else - retval =3D timerlat_top_bpf_main_loop(top, record, params, ¶ms_u); + retval =3D timerlat_top_bpf_main_loop(top, record, aa, params, ¶ms_u= ); =20 if (retval) goto out_top; @@ -1230,7 +1244,6 @@ int timerlat_top_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - actions_perform(¶ms->actions); return_value =3D FAILED; } else if (params->aa_only) { /* --=20 2.49.0