From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B99012E06E4 for ; Thu, 26 Jun 2025 12:36:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941378; cv=none; b=s+FCFIysbvlVq+IEHHb7h5MHdI2nFcprklkUiAFnXktsWQDG3C3XZYz3QI1M+UEeus7ieQ0nRa+ewDt/vW7b5W7komkcGHL0NaEQQ36sKvybo3siEI7tXceg/joCyvODckFPJXYmKV34Rlb+d9EYEO8t/MhsNsoy7EcMCx0jSyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941378; c=relaxed/simple; bh=e5G1LQaHGKEk+Uq+ZBbUmduHV4uJn9Y27OD86/nVQTc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qQVyQrUBXfz68TQYqlThW57G23dCFQwMK9IBcaGWzSQU95ptitKJk4ZTy1iy2wLhprAQcfNGXA4OUm2Te1hpO3kz5PdoXHIkLG0IVR2CJsrjK/eeKZJBvBRhVfrrVal1e6UfYVbFiWSgrwnUMUmgmtmgPMcxjjGwIza17zgN5v4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=HF79Risw; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HF79Risw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PsKlgJn34is2kOicYgqOhMcE5nDUrsqk6HaC7NVZhto=; b=HF79Risw3Q/ilAxX6WTZLG22XJz3dUktgHMis9yGoc5u4b2Lgfg/tgzUlmhMxSngpd+QEH Q5J/t5O/qq8098fSMJ7puebtat4SYxeYyNcefMPisesGwjANowgekHIkha6xHC8hYn7unN VmckcD1YFKv1JTGZI7m/HjqepoPOqH0= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-382-j9_UXHl_P7SCABDdSeod2g-1; Thu, 26 Jun 2025 08:36:09 -0400 X-MC-Unique: j9_UXHl_P7SCABDdSeod2g-1 X-Mimecast-MFC-AGG-ID: j9_UXHl_P7SCABDdSeod2g_1750941368 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 55ABF1978F63; Thu, 26 Jun 2025 12:36:08 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 5E801180045B; Thu, 26 Jun 2025 12:36:05 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 1/9] rtla/timerlat: Introduce enum timerlat_tracing_mode Date: Thu, 26 Jun 2025 14:33:57 +0200 Message-ID: <20250626123405.1496931-2-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" After the introduction of BPF-based sample collection, rtla-timerlat effectively runs in one of three modes: - Pure BPF mode, with tracefs only being used to set up the timerlat tracer. Sample processing and stop on threshold are handled by BPF. - tracefs mode. BPF is unsupported or kernel is lacking the necessary trace event (osnoise:timerlat_sample). Stop on theshold is handled by timerlat tracer stopping tracing in all instances. - BPF/tracefs mixed mode - BPF is used for sample collection for top or histogram, tracefs is used for trace output and/or auto-analysis. Stop on threshold is handled both through BPF program, which stops sample collection for top/histogram and wakes up rtla, and by timerlat tracer, which stops tracing for trace output/auto-analysis instances. Add enum timerlat_tracing_mode, with three values: - TRACING_MODE_BPF - TRACING_MODE_TRACEFS - TRACING_MODE_MIXED Those represent the modes described above. A field of this type is added to struct timerlat_params, named "mode", replacing the no_bpf variable. params->mode is set in timerlat_{top,hist}_parse_args to TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output and/or auto-analysis is requested. timerlat_{top,hist}_main then checks if BPF is not unavailable or disabled, in that case, it sets params->mode to TRACING_MODE_TRACEFS. A condition is added to timerlat_apply_config that skips setting timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those are unnecessary, since they only turn off tracing, which is already turned off in that case, since BPF is used to collect samples). Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/timerlat.c | 24 +++++++---- tools/tracing/rtla/src/timerlat.h | 18 ++++++++ tools/tracing/rtla/src/timerlat_hist.c | 51 +++++++++++++---------- tools/tracing/rtla/src/timerlat_top.c | 57 +++++++++++++++----------- 4 files changed, 97 insertions(+), 53 deletions(-) diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/tim= erlat.c index c29e2ba2d7d8..63d6d43eafff 100644 --- a/tools/tracing/rtla/src/timerlat.c +++ b/tools/tracing/rtla/src/timerlat.c @@ -40,16 +40,22 @@ timerlat_apply_config(struct osnoise_tool *tool, struct= timerlat_params *params) CPU_SET(i, ¶ms->monitored_cpus); } =20 - retval =3D osnoise_set_stop_us(tool->context, params->stop_us); - if (retval) { - err_msg("Failed to set stop us\n"); - goto out_err; - } + if (params->mode !=3D TRACING_MODE_BPF) { + /* + * In tracefs and mixed mode, timerlat tracer handles stopping + * on threshold + */ + retval =3D osnoise_set_stop_us(tool->context, params->stop_us); + if (retval) { + err_msg("Failed to set stop us\n"); + goto out_err; + } =20 - retval =3D osnoise_set_stop_total_us(tool->context, params->stop_total_us= ); - if (retval) { - err_msg("Failed to set stop total us\n"); - goto out_err; + retval =3D osnoise_set_stop_total_us(tool->context, params->stop_total_u= s); + if (retval) { + err_msg("Failed to set stop total us\n"); + goto out_err; + } } =20 =20 diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/tim= erlat.h index 73045aef23fa..e0a553545d03 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -1,6 +1,23 @@ // SPDX-License-Identifier: GPL-2.0 #include "osnoise.h" =20 +/* + * Define timerlat tracing mode. + * + * There are three tracing modes: + * - tracefs-only, used when BPF is unavailable. + * - BPF-only, used when BPF is available and neither trace saving nor + * auto-analysis are enabled. + * - mixed mode, used when BPF is available and either trace saving or + * auto-analysis is enabled (which rely on sample collection through + * tracefs). + */ +enum timerlat_tracing_mode { + TRACING_MODE_BPF, + TRACING_MODE_TRACEFS, + TRACING_MODE_MIXED, +}; + struct timerlat_params { /* Common params */ char *cpus; @@ -30,6 +47,7 @@ struct timerlat_params { cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; + enum timerlat_tracing_mode mode; union { struct { /* top only */ diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 36d2294c963d..6cf260e8553b 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -802,6 +802,9 @@ static struct timerlat_params params->bucket_size =3D 1; params->entries =3D 256; =20 + /* default to BPF mode */ + params->mode =3D TRACING_MODE_BPF; + while (1) { static struct option long_options[] =3D { {"auto", required_argument, 0, 'a'}, @@ -1054,6 +1057,13 @@ static struct timerlat_params if (params->kernel_workload && params->user_workload) timerlat_hist_usage("--kernel-threads and --user-threads are mutually ex= clusive!"); =20 + /* + * If auto-analysis or trace output is enabled, switch from BPF mode to + * mixed mode + */ + if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + params->mode =3D TRACING_MODE_MIXED; + return params; } =20 @@ -1149,7 +1159,6 @@ int timerlat_hist_main(int argc, char *argv[]) pthread_t timerlat_u; int retval; int nr_cpus, i; - bool no_bpf =3D false; =20 params =3D timerlat_hist_parse_args(argc, argv); if (!params) @@ -1161,12 +1170,6 @@ int timerlat_hist_main(int argc, char *argv[]) goto out_exit; } =20 - retval =3D timerlat_hist_apply_config(tool, params); - if (retval) { - err_msg("Could not apply config\n"); - goto out_free; - } - trace =3D &tool->trace; /* * Save trace instance into global variable so that SIGINT can stop @@ -1175,24 +1178,30 @@ int timerlat_hist_main(int argc, char *argv[]) */ hist_inst =3D trace; =20 + /* + * Try to enable BPF, unless disabled explicitly. + * If BPF enablement fails, fall back to tracefs mode. + */ if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) =3D= =3D 0) { debug_msg("RTLA_NO_BPF set, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_s= ample")) { + params->mode =3D TRACING_MODE_TRACEFS; + } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sampl= e")) { debug_msg("osnoise:timerlat_sample missing, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf) { + params->mode =3D TRACING_MODE_TRACEFS; + } else { retval =3D timerlat_bpf_init(params); if (retval) { debug_msg("Could not enable BPF\n"); - no_bpf =3D true; + params->mode =3D TRACING_MODE_TRACEFS; } } =20 + retval =3D timerlat_hist_apply_config(tool, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_free; + } + retval =3D enable_timerlat(trace); if (retval) { err_msg("Failed to enable timerlat tracer\n"); @@ -1320,7 +1329,7 @@ int timerlat_hist_main(int argc, char *argv[]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); - if (no_bpf) { + if (params->mode =3D=3D TRACING_MODE_TRACEFS) { trace_instance_start(trace); } else { retval =3D timerlat_bpf_attach(); @@ -1333,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[]) tool->start_time =3D time(NULL); timerlat_hist_set_signals(params); =20 - if (no_bpf) { + if (params->mode =3D=3D TRACING_MODE_TRACEFS) { while (!stop_tracing) { sleep(params->sleep_time); =20 @@ -1362,7 +1371,7 @@ int timerlat_hist_main(int argc, char *argv[]) } else timerlat_bpf_wait(-1); =20 - if (!no_bpf) { + if (params->mode !=3D TRACING_MODE_TRACEFS) { timerlat_bpf_detach(); retval =3D timerlat_hist_bpf_pull_data(tool); if (retval) { @@ -1409,10 +1418,10 @@ int timerlat_hist_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(tool); + if (params->mode !=3D TRACING_MODE_TRACEFS) + timerlat_bpf_destroy(); free(params); free_cpu_idle_disable_states(); - if (!no_bpf) - timerlat_bpf_destroy(); out_exit: exit(return_value); } diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 7365e08fe986..1644eeb60181 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -559,6 +559,9 @@ static struct timerlat_params /* display data in microseconds */ params->output_divisor =3D 1000; =20 + /* default to BPF mode */ + params->mode =3D TRACING_MODE_BPF; + while (1) { static struct option long_options[] =3D { {"auto", required_argument, 0, 'a'}, @@ -790,6 +793,13 @@ static struct timerlat_params if (params->kernel_workload && params->user_workload) timerlat_top_usage("--kernel-threads and --user-threads are mutually exc= lusive!"); =20 + /* + * If auto-analysis or trace output is enabled, switch from BPF mode to + * mixed mode + */ + if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + params->mode =3D TRACING_MODE_MIXED; + return params; } =20 @@ -994,7 +1004,6 @@ int timerlat_top_main(int argc, char *argv[]) char *max_lat; int retval; int nr_cpus, i; - bool no_bpf =3D false; =20 params =3D timerlat_top_parse_args(argc, argv); if (!params) @@ -1006,38 +1015,38 @@ int timerlat_top_main(int argc, char *argv[]) goto out_exit; } =20 - retval =3D timerlat_top_apply_config(top, params); - if (retval) { - err_msg("Could not apply config\n"); - goto out_free; - } - trace =3D &top->trace; /* - * Save trace instance into global variable so that SIGINT can stop - * the timerlat tracer. - * Otherwise, rtla could loop indefinitely when overloaded. - */ + * Save trace instance into global variable so that SIGINT can stop + * the timerlat tracer. + * Otherwise, rtla could loop indefinitely when overloaded. + */ top_inst =3D trace; =20 + /* + * Try to enable BPF, unless disabled explicitly. + * If BPF enablement fails, fall back to tracefs mode. + */ if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) =3D= =3D 0) { debug_msg("RTLA_NO_BPF set, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_s= ample")) { + params->mode =3D TRACING_MODE_TRACEFS; + } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sampl= e")) { debug_msg("osnoise:timerlat_sample missing, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf) { + params->mode =3D TRACING_MODE_TRACEFS; + } else { retval =3D timerlat_bpf_init(params); if (retval) { debug_msg("Could not enable BPF\n"); - no_bpf =3D true; + params->mode =3D TRACING_MODE_TRACEFS; } } =20 + retval =3D timerlat_top_apply_config(top, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_free; + } + retval =3D enable_timerlat(trace); if (retval) { err_msg("Failed to enable timerlat tracer\n"); @@ -1166,7 +1175,7 @@ int timerlat_top_main(int argc, char *argv[]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); - if (no_bpf) { + if (params->mode =3D=3D TRACING_MODE_TRACEFS) { trace_instance_start(trace); } else { retval =3D timerlat_bpf_attach(); @@ -1179,7 +1188,7 @@ int timerlat_top_main(int argc, char *argv[]) top->start_time =3D time(NULL); timerlat_top_set_signals(params); =20 - if (no_bpf) + if (params->mode =3D=3D TRACING_MODE_TRACEFS) retval =3D timerlat_top_main_loop(top, record, params, ¶ms_u); else retval =3D timerlat_top_bpf_main_loop(top, record, params, ¶ms_u); @@ -1187,7 +1196,7 @@ int timerlat_top_main(int argc, char *argv[]) if (retval) goto out_top; =20 - if (!no_bpf) + if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_detach(); =20 if (params->user_workload && !params_u.stopped_running) { @@ -1239,6 +1248,8 @@ int timerlat_top_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(top); + if (params->mode !=3D TRACING_MODE_TRACEFS) + timerlat_bpf_destroy(); free(params); free_cpu_idle_disable_states(); out_exit: --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 499E32E11AF for ; Thu, 26 Jun 2025 12:36:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941379; cv=none; b=oBjKUp3jw9QEwYz9jsa8b50lhSdcAxFBTTDlgsBUoEbOsY1Hgo9IWx4nLAXmd5nB+MTnM9NQDSbed68U2/HTWJIk9vEGglrNO5H2TUYobkVTTgZtrr+Q/njZCrLhWly8ouRWXVI0EdMyKXilpYqvhHgCT/LfeYm2ziXOMp33ka8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941379; c=relaxed/simple; bh=KJTcyfl718gsSFbGMLOb1+yq3qllHiHvKtOW8XHBmuY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h1LJDHRqhhzA95OcU8F/pEyrEqDrKBbHnx73bgYTJp7HUyxZvPfHAWiiRiXfe8q8/uuja+JcL+ScQiQmzOQzBonYB9p4CfEZts73Z7lTx6IpclRJJ8Ds/vbaHU+TzvEJme1EOpGQ4l65RnwTrcokvq4+p2tI8wBJPo00pcZP3wk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Cno84MUg; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Cno84MUg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KLuOzLW+2NVz+fGXi5b17YSiYie0t2mHvQ3Mk2GK9fc=; b=Cno84MUgnx2x5AAdGwnyPzeNGz5aUaz4n0NlNPTPuHgntduFCIZ2cr9OMpxkglD6ZGyRej XeyVsfwGC3uf8HklvZ80BsybFWhC8nBz8HYKNt1s2TfF1Qn6SP4ATqe43oWIizdSbndymz kEDuJ87DNuFQh+2kcrvi5EL0k7/QF08= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-642-c3f45pJEM5-84cOqBOwgQw-1; Thu, 26 Jun 2025 08:36:13 -0400 X-MC-Unique: c3f45pJEM5-84cOqBOwgQw-1 X-Mimecast-MFC-AGG-ID: c3f45pJEM5-84cOqBOwgQw_1750941372 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EA5E418011F9; Thu, 26 Jun 2025 12:36:11 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D323218003FC; Thu, 26 Jun 2025 12:36:08 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 2/9] rtla/timerlat: Add action on threshold feature Date: Thu, 26 Jun 2025 14:33:58 +0200 Message-ID: <20250626123405.1496931-3-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Extend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support implementing arbitrary actions. A new option, --on-threshold, is added, taking an argument that further specifies the action. Actions added in this patch are: - trace[,file=3D]: Saves tracefs buffer, optionally taking a filename. - signal,num=3D,pid=3D: Sends signal to process. "parent" might be specified instead of number to send signal to parent process. - shell,command=3D: Execute shell command. Multiple actions may be specified and will be executed in order, including multiple actions of the same type. Trace output requested via -t and -a now adds a trace action to the end of the list. If an action fails, the following actions are not executed. For example, this command: $ rtla timerlat -T 20 --on-threshold trace \ --on-threshold shell,command=3D"grep ipi_send timerlat_trace.txt" \ --on-threshold signal,num=3D2,pid=3Dparent will send signal 2 (SIGINT) to parent process, but only if saved trace contains the text "ipi_send". This way, the feature can be used for flexible reactions on latency spikes, and allows combining rtla with other tooling like perf. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/Build | 1 + tools/tracing/rtla/src/actions.c | 235 +++++++++++++++++++++++++ tools/tracing/rtla/src/actions.h | 49 ++++++ tools/tracing/rtla/src/timerlat.h | 3 +- tools/tracing/rtla/src/timerlat_hist.c | 37 ++-- tools/tracing/rtla/src/timerlat_top.c | 38 ++-- 6 files changed, 341 insertions(+), 22 deletions(-) create mode 100644 tools/tracing/rtla/src/actions.c create mode 100644 tools/tracing/rtla/src/actions.h diff --git a/tools/tracing/rtla/src/Build b/tools/tracing/rtla/src/Build index 7bb7e39e391a..66631280b75b 100644 --- a/tools/tracing/rtla/src/Build +++ b/tools/tracing/rtla/src/Build @@ -1,5 +1,6 @@ rtla-y +=3D trace.o rtla-y +=3D utils.o +rtla-y +=3D actions.o rtla-y +=3D osnoise.o rtla-y +=3D osnoise_top.o rtla-y +=3D osnoise_hist.o diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/acti= ons.c new file mode 100644 index 000000000000..63bee5bdabfd --- /dev/null +++ b/tools/tracing/rtla/src/actions.c @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include "actions.h" +#include "trace.h" +#include "utils.h" + +/* + * actions_init - initialize struct actions + */ +void +actions_init(struct actions *self) +{ + self->size =3D action_default_size; + self->list =3D calloc(self->size, sizeof(struct action)); + self->len =3D 0; + + memset(&self->present, 0, sizeof(self->present)); + + /* This has to be set by the user */ + self->trace_output_inst =3D NULL; +} + +/* + * actions_destroy - destroy struct actions + */ +void +actions_destroy(struct actions *self) +{ + /* Free any action-specific data */ + for (struct action *action =3D self->list; action < self->list + self->le= n; action++) { + if (action->type =3D=3D ACTION_SHELL) + free(action->command); + if (action->type =3D=3D ACTION_TRACE_OUTPUT) + free(action->trace_output); + } + + /* Free action list */ + free(self->list); +} + +/* + * actions_new - Get pointer to new action + */ +static struct action * +actions_new(struct actions *self) +{ + if (self->size >=3D self->len) { + self->size *=3D 2; + self->list =3D realloc(self->list, self->size * sizeof(struct action)); + } + + return &self->list[self->len++]; +} + +/* + * actions_add_trace_output - add an action to output trace + */ +int +actions_add_trace_output(struct actions *self, const char *trace_output) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_TRACE_OUTPUT] =3D true; + action->type =3D ACTION_TRACE_OUTPUT; + action->trace_output =3D calloc(strlen(trace_output) + 1, sizeof(char)); + if (!action->trace_output) + return -1; + strcpy(action->trace_output, trace_output); + + return 0; +} + +/* + * actions_add_trace_output - add an action to send signal to a process + */ +int +actions_add_signal(struct actions *self, int signal, int pid) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_SIGNAL] =3D true; + action->type =3D ACTION_SIGNAL; + action->signal =3D signal; + action->pid =3D pid; + + return 0; +} + +/* + * actions_add_shell - add an action to execute a shell command + */ +int +actions_add_shell(struct actions *self, const char *command) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_SHELL] =3D true; + action->type =3D ACTION_SHELL; + action->command =3D calloc(strlen(command) + 1, sizeof(char)); + if (!action->command) + return -1; + strcpy(action->command, command); + + return 0; +} + +/* + * actions_parse - add an action based on text specification + */ +int +actions_parse(struct actions *self, const char *trigger) +{ + enum action_type type =3D ACTION_NONE; + char *token; + char trigger_c[strlen(trigger)]; + + /* For ACTION_SIGNAL */ + int signal =3D 0, pid =3D 0; + + /* For ACTION_TRACE_OUTPUT */ + char *trace_output; + + strcpy(trigger_c, trigger); + token =3D strtok(trigger_c, ","); + + if (strcmp(token, "trace") =3D=3D 0) + type =3D ACTION_TRACE_OUTPUT; + else if (strcmp(token, "signal") =3D=3D 0) + type =3D ACTION_SIGNAL; + else if (strcmp(token, "shell") =3D=3D 0) + type =3D ACTION_SHELL; + else + /* Invalid trigger type */ + return -1; + + token =3D strtok(NULL, ","); + + switch (type) { + case ACTION_TRACE_OUTPUT: + /* Takes no argument */ + if (token =3D=3D NULL) + trace_output =3D "timerlat_trace.txt"; + else { + if (strlen(token) > 5 && strncmp(token, "file=3D", 5) =3D=3D 0) { + trace_output =3D token + 5; + } else { + /* Invalid argument */ + return -1; + } + + token =3D strtok(NULL, ","); + if (token !=3D NULL) + /* Only one argument allowed */ + return -1; + } + return actions_add_trace_output(self, trace_output); + case ACTION_SIGNAL: + /* Takes two arguments, num (signal) and pid */ + while (token !=3D NULL) { + if (strlen(token) > 4 && strncmp(token, "num=3D", 4) =3D=3D 0) { + signal =3D atoi(token + 4); + } else if (strlen(token) > 4 && strncmp(token, "pid=3D", 4) =3D=3D 0) { + if (strncmp(token + 4, "parent", 7) =3D=3D 0) + pid =3D -1; + else + pid =3D atoi(token + 4); + } else { + /* Invalid argument */ + return -1; + } + + token =3D strtok(NULL, ","); + } + + if (!signal || !pid) + /* Missing argument */ + return -1; + + return actions_add_signal(self, signal, pid); + case ACTION_SHELL: + if (token =3D=3D NULL) + return -1; + if (strlen(token) > 8 && strncmp(token, "command=3D", 8) =3D=3D 0) + return actions_add_shell(self, token + 8); + return -1; + default: + return -1; + } +} + +/* + * actions_perform - perform all actions + */ +int +actions_perform(const struct actions *self) +{ + int pid, retval; + const struct action *action; + + for (action =3D self->list; action < self->list + self->len; action++) { + switch (action->type) { + case ACTION_TRACE_OUTPUT: + retval =3D save_trace_to_file(self->trace_output_inst, action->trace_ou= tput); + if (retval) { + err_msg("Error saving trace\n"); + return retval; + } + break; + case ACTION_SIGNAL: + if (action->pid =3D=3D -1) + pid =3D getppid(); + else + pid =3D action->pid; + retval =3D kill(pid, action->signal); + if (retval) { + err_msg("Error sending signal\n"); + return retval; + } + break; + case ACTION_SHELL: + retval =3D system(action->command); + if (retval) + return retval; + break; + default: + break; + } + } + + return 0; +} diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/acti= ons.h new file mode 100644 index 000000000000..076bbff8519e --- /dev/null +++ b/tools/tracing/rtla/src/actions.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include +#include + +enum action_type { + ACTION_NONE =3D 0, + ACTION_TRACE_OUTPUT, + ACTION_SIGNAL, + ACTION_SHELL, + ACTION_FIELD_N +}; + +struct action { + enum action_type type; + union { + struct { + /* For ACTION_TRACE_OUTPUT */ + char *trace_output; + }; + struct { + /* For ACTION_SIGNAL */ + int signal; + int pid; + }; + struct { + /* For ACTION_SHELL */ + char *command; + }; + }; +}; + +static const int action_default_size =3D 8; + +struct actions { + struct action *list; + int len, size; + bool present[ACTION_FIELD_N]; + + /* External dependencies */ + struct tracefs_instance *trace_output_inst; +}; + +void actions_init(struct actions *self); +void actions_destroy(struct actions *self); +int actions_add_trace_output(struct actions *self, const char *trace_outpu= t); +int actions_add_signal(struct actions *self, int signal, int pid); +int actions_add_shell(struct actions *self, const char *command); +int actions_parse(struct actions *self, const char *trigger); +int actions_perform(const struct actions *self); diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/tim= erlat.h index e0a553545d03..d1fcf9a97621 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include "actions.h" #include "osnoise.h" =20 /* @@ -22,7 +23,6 @@ struct timerlat_params { /* Common params */ char *cpus; cpu_set_t monitored_cpus; - char *trace_output; char *cgroup_name; unsigned long long runtime; long long stop_us; @@ -48,6 +48,7 @@ struct timerlat_params { struct sched_attr sched_param; struct trace_events *events; enum timerlat_tracing_mode mode; + struct actions actions; union { struct { /* top only */ diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 6cf260e8553b..d975d2cd6604 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -757,6 +757,7 @@ static void timerlat_hist_usage(char *usage) " --warm-up s: let the workload run for s seconds before collecting= data", " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", + " --on-threshold : define action to be executed at latency = threshold, multiple are allowed", NULL, }; =20 @@ -786,11 +787,14 @@ static struct timerlat_params int auto_thresh; int retval; int c; + char *trace_output =3D NULL; =20 params =3D calloc(1, sizeof(*params)); if (!params) exit(1); =20 + actions_init(¶ms->actions); + /* disabled by default */ params->dma_latency =3D -1; =20 @@ -841,6 +845,7 @@ static struct timerlat_params {"warm-up", required_argument, 0, '\2'}, {"trace-buffer-size", required_argument, 0, '\3'}, {"deepest-idle-state", required_argument, 0, '\4'}, + {"on-threshold", required_argument, 0, '\5'}, {0, 0, 0, 0} }; =20 @@ -866,7 +871,7 @@ static struct timerlat_params params->print_stack =3D auto_thresh; =20 /* set trace */ - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; =20 break; case 'c': @@ -956,13 +961,13 @@ static struct timerlat_params case 't': if (optarg) { if (optarg[0] =3D=3D '=3D') - params->trace_output =3D &optarg[1]; + trace_output =3D &optarg[1]; else - params->trace_output =3D &optarg[0]; + trace_output =3D &optarg[0]; } else if (optind < argc && argv[optind][0] !=3D '-') - params->trace_output =3D argv[optind]; + trace_output =3D argv[optind]; else - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; break; case 'u': params->user_workload =3D 1; @@ -1032,11 +1037,21 @@ static struct timerlat_params case '\4': params->deepest_idle_state =3D get_llong_from_str(optarg); break; + case '\5': + retval =3D actions_parse(¶ms->actions, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } + break; default: timerlat_hist_usage("Invalid option"); } } =20 + if (trace_output) + actions_add_trace_output(¶ms->actions, trace_output); + if (geteuid()) { err_msg("rtla needs root permission\n"); exit(EXIT_FAILURE); @@ -1061,7 +1076,8 @@ static struct timerlat_params * If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ - if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + if (params->mode =3D=3D TRACING_MODE_BPF && + (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1254,12 +1270,13 @@ int timerlat_hist_main(int argc, char *argv[]) } } =20 - if (params->trace_output) { + if (params->actions.present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); goto out_free; } + params->actions.trace_output_inst =3D record->trace.inst; =20 if (params->events) { retval =3D trace_events_enable(&record->trace, params->events); @@ -1325,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->trace_output) + if (params->actions.present[ACTION_TRACE_OUTPUT]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1395,8 +1412,7 @@ int timerlat_hist_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - save_trace_to_file(record ? record->trace.inst : NULL, - params->trace_output); + actions_perform(¶ms->actions); return_value =3D FAILED; } =20 @@ -1418,6 +1434,7 @@ int timerlat_hist_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(tool); + actions_destroy(¶ms->actions); if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); free(params); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 1644eeb60181..cdbfda009974 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -516,6 +516,7 @@ static void timerlat_top_usage(char *usage) " --warm-up s: let the workload run for s seconds before collecting= data", " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", + " --on-threshold : define action to be executed at latency = threshold, multiple are allowed", NULL, }; =20 @@ -545,11 +546,14 @@ static struct timerlat_params long long auto_thresh; int retval; int c; + char *trace_output =3D NULL; =20 params =3D calloc(1, sizeof(*params)); if (!params) exit(1); =20 + actions_init(¶ms->actions); + /* disabled by default */ params->dma_latency =3D -1; =20 @@ -592,6 +596,7 @@ static struct timerlat_params {"warm-up", required_argument, 0, '6'}, {"trace-buffer-size", required_argument, 0, '7'}, {"deepest-idle-state", required_argument, 0, '8'}, + {"on-threshold", required_argument, 0, '9'}, {0, 0, 0, 0} }; =20 @@ -617,7 +622,7 @@ static struct timerlat_params params->print_stack =3D auto_thresh; =20 /* set trace */ - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; break; case '5': /* it is here because it is similar to -a */ @@ -712,14 +717,13 @@ static struct timerlat_params case 't': if (optarg) { if (optarg[0] =3D=3D '=3D') - params->trace_output =3D &optarg[1]; + trace_output =3D &optarg[1]; else - params->trace_output =3D &optarg[0]; + trace_output =3D &optarg[0]; } else if (optind < argc && argv[optind][0] !=3D '-') - params->trace_output =3D argv[optind]; + trace_output =3D argv[optind]; else - params->trace_output =3D "timerlat_trace.txt"; - + trace_output =3D "timerlat_trace.txt"; break; case 'u': params->user_workload =3D true; @@ -771,11 +775,21 @@ static struct timerlat_params case '8': params->deepest_idle_state =3D get_llong_from_str(optarg); break; + case '9': + retval =3D actions_parse(¶ms->actions, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } + break; default: timerlat_top_usage("Invalid option"); } } =20 + if (trace_output) + actions_add_trace_output(¶ms->actions, trace_output); + if (geteuid()) { err_msg("rtla needs root permission\n"); exit(EXIT_FAILURE); @@ -797,7 +811,8 @@ static struct timerlat_params * If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ - if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + if (params->mode =3D=3D TRACING_MODE_BPF && + (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1099,12 +1114,13 @@ int timerlat_top_main(int argc, char *argv[]) } } =20 - if (params->trace_output) { + if (params->actions.present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); goto out_free; } + params->actions.trace_output_inst =3D record->trace.inst; =20 if (params->events) { retval =3D trace_events_enable(&record->trace, params->events); @@ -1171,7 +1187,7 @@ int timerlat_top_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->trace_output) + if (params->actions.present[ACTION_TRACE_OUTPUT]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1214,8 +1230,7 @@ int timerlat_top_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - save_trace_to_file(record ? record->trace.inst : NULL, - params->trace_output); + actions_perform(¶ms->actions); return_value =3D FAILED; } else if (params->aa_only) { /* @@ -1248,6 +1263,7 @@ int timerlat_top_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(top); + actions_destroy(¶ms->actions); if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); free(params); --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F90D2DFF0D for ; Thu, 26 Jun 2025 12:36:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941380; cv=none; b=bgujWDvklvNwmYXri4P0zTMVIL4RPVQ+39hVP2EspG+o9WYPGFkpofvi1xKy/6R2phr7izUaHw+84NpoAZhv2HWiayoHS/yrI4TYj3S4J8vkuaRljLsCeUharCj2rl1TkgyWwSqm/kF8sluVUU/EZW0bUPMgiSeAdavHSFkONTs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941380; c=relaxed/simple; bh=cOpwCH8DVaBtcRF0XPiHH/CxI5nZlNIp5nt3wZOKeWs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tc1GCIWopzUU+2gEtESe1d5wHCib/B9F/4ktZ2MfSzSiabu0bCxnr4BTzxV/6uPZ5J8s2Lptk/PUjvpEJnofucpl4VzBENvro36eQvcn8TaifeOvZfH43vmPpovgudNFWupwiK4JRCB0zgeosOa9V48kzb9RjCmJDmD5d4ceAkY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=H9CPucYu; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="H9CPucYu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8wnuRWICz/3L1A5kr+hgYjZ7aQO+C6vPwfPXybDrYEg=; b=H9CPucYulMPrbxDTFima73+/Qs+YqmF7y8UVEAzh/p6AyU9mwPTnqPhEEqvnwJ/uwalqsN S8EFASm+5PqAz55BFOqZJd/ps7rzf55ClJYk0PODU/fOmKt0Mhyb0tEM+qWA28fN9mmCFP +0e6bXVtpvTC04c2YFcQtet3D+Gndpc= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-106-pd2-vGxGP-6TCkJ4sGP65A-1; Thu, 26 Jun 2025 08:36:16 -0400 X-MC-Unique: pd2-vGxGP-6TCkJ4sGP65A-1 X-Mimecast-MFC-AGG-ID: pd2-vGxGP-6TCkJ4sGP65A_1750941375 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4B52B1955EC3; Thu, 26 Jun 2025 12:36:15 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 683A9180045B; Thu, 26 Jun 2025 12:36:12 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 3/9] rtla/timerlat_bpf: Allow resuming tracing Date: Thu, 26 Jun 2025 14:33:59 +0200 Message-ID: <20250626123405.1496931-4-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Currently, rtla-timerlat BPF program uses a global variable stored in a .bss section to store whether tracing has been stopped. Move the information to a separate map, so that it is easily writable from userspace, and add a function that clears the value, resuming tracing after it has been stopped. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/timerlat.bpf.c | 13 +++++++++---- tools/tracing/rtla/src/timerlat_bpf.c | 13 +++++++++++++ tools/tracing/rtla/src/timerlat_bpf.h | 3 +++ 3 files changed, 25 insertions(+), 4 deletions(-) diff --git a/tools/tracing/rtla/src/timerlat.bpf.c b/tools/tracing/rtla/src= /timerlat.bpf.c index 96196d46e170..084cd10c21fc 100644 --- a/tools/tracing/rtla/src/timerlat.bpf.c +++ b/tools/tracing/rtla/src/timerlat.bpf.c @@ -28,6 +28,13 @@ struct { __type(value, unsigned long long); } summary_irq SEC(".maps"), summary_thread SEC(".maps"), summary_user SEC(= ".maps"); =20 +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, unsigned int); + __type(value, unsigned long long); +} stop_tracing SEC(".maps"); + struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 1); @@ -41,8 +48,6 @@ const volatile int irq_threshold; const volatile int thread_threshold; const volatile bool aa_only; =20 -int stop_tracing; - nosubprog unsigned long long map_get(void *map, unsigned int key) { @@ -109,7 +114,7 @@ nosubprog void set_stop_tracing(void) int value =3D 0; =20 /* Suppress further sample processing */ - stop_tracing =3D 1; + map_set(&stop_tracing, 0, 1); =20 /* Signal to userspace */ bpf_ringbuf_output(&signal_stop_tracing, &value, sizeof(value), 0); @@ -121,7 +126,7 @@ int handle_timerlat_sample(struct trace_event_raw_timer= lat_sample *tp_args) unsigned long long latency, latency_us; int bucket; =20 - if (stop_tracing) + if (map_get(&stop_tracing, 0)) return 0; =20 latency =3D tp_args->timer_latency / output_divisor; diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src= /timerlat_bpf.c index 0bc44ce5d69b..1666215dd687 100644 --- a/tools/tracing/rtla/src/timerlat_bpf.c +++ b/tools/tracing/rtla/src/timerlat_bpf.c @@ -106,6 +106,19 @@ int timerlat_bpf_wait(int timeout) return retval; } =20 +/* + * timerlat_bpf_restart_tracing - restart stopped tracing + */ +int timerlat_bpf_restart_tracing(void) +{ + unsigned int key =3D 0; + unsigned long long value =3D 0; + + return bpf_map__update_elem(bpf->maps.stop_tracing, + &key, sizeof(key), + &value, sizeof(value), BPF_ANY); +} + static int get_value(struct bpf_map *map_irq, struct bpf_map *map_thread, struct bpf_map *map_user, diff --git a/tools/tracing/rtla/src/timerlat_bpf.h b/tools/tracing/rtla/src= /timerlat_bpf.h index f1b54dbddb0e..118487436d30 100644 --- a/tools/tracing/rtla/src/timerlat_bpf.h +++ b/tools/tracing/rtla/src/timerlat_bpf.h @@ -18,6 +18,7 @@ int timerlat_bpf_attach(void); void timerlat_bpf_detach(void); void timerlat_bpf_destroy(void); int timerlat_bpf_wait(int timeout); +int timerlat_bpf_restart_tracing(void); int timerlat_bpf_get_hist_value(int key, long long *value_irq, long long *value_thread, @@ -28,6 +29,7 @@ int timerlat_bpf_get_summary_value(enum summary_field key, long long *value_thread, long long *value_user, int cpus); + static inline int have_libbpf_support(void) { return 1; } #else static inline int timerlat_bpf_init(struct timerlat_params *params) @@ -38,6 +40,7 @@ static inline int timerlat_bpf_attach(void) { return -1; } static inline void timerlat_bpf_detach(void) { }; static inline void timerlat_bpf_destroy(void) { }; static inline int timerlat_bpf_wait(int timeout) { return -1; } +static inline int timerlat_bpf_restart_tracing(void) { return -1; }; static inline int timerlat_bpf_get_hist_value(int key, long long *value_irq, long long *value_thread, --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4A572E1756 for ; Thu, 26 Jun 2025 12:36:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941385; cv=none; b=FF46TBxr+xgq4abMvRWZxGWtJX26R2103+aDPDLOcWivwrefJUZSWdeJM9xuj67Gz7Qrk/SoCrrwLibuAAgwEV6r/QDY53ZqpbloTeQkpzk0m6I5k6pJJEk9IYVasopeK51z7b3U8T9pdl+cHsyBhVZSSDAIykXg2vR6kxk2BmY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941385; c=relaxed/simple; bh=XBWN5KZHexp9ei5dl4X2qjbdQYssZL7zZfiVFNn2ZrE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HmgHBbOBVYMom5EuoI+U04BlKJ0ZAd4gHP4Mnbb/syJBVzpFUMIQO7OzwrmcIf1GnRwNWiWnVkHdGBvMKVz1PPSfRgz9HRLkREzdBISX9V7n4RotWlcuZ7WJ7DvGJTDBNF8lzeyW2bOz+b/MhdoQMln5ibkpzhXGN+OGxsiM9bo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=OI20YG7d; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OI20YG7d" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941382; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qJdek3Tmd1i6NiuOEw7/enh8kH7UPfbPx2bOIgJR26E=; b=OI20YG7dy9gQKipo33LVbm9ZURdpuuMMFHPWOir2VkKI9wIiMDBD6cwXQ5mxRU5i6mAUmr 4qZB7q81Xy9CzFjN1WWVKOxMF0P73ZFBhUtMjw2V0pNPCsBhouzC/CYsCtWvlccScLdFZo eWOS4uIYc5UjiM471SGF23YWwoJyKRM= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-263-MrmlD-aTOpKp-6Pe9XpCEQ-1; Thu, 26 Jun 2025 08:36:19 -0400 X-MC-Unique: MrmlD-aTOpKp-6Pe9XpCEQ-1 X-Mimecast-MFC-AGG-ID: MrmlD-aTOpKp-6Pe9XpCEQ_1750941378 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A3BDB1800368; Thu, 26 Jun 2025 12:36:18 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B2E4F18003FC; Thu, 26 Jun 2025 12:36:15 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 4/9] rtla/timerlat: Add continue action Date: Thu, 26 Jun 2025 14:34:00 +0200 Message-ID: <20250626123405.1496931-5-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Introduce option to resume tracing after a latency threshold overflow. The option is implemented as an action named "continue". Example: $ rtla timerlat top -q -T 200 -d 1s --on-threshold \ exec,command=3D"echo Threshold" --on-threshold continue Threshold Threshold Threshold Timer Latency ... The feature is supported for both hist and top. After the continue action is executed, processing of the list of actions is stopped and tracing is resumed. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/actions.c | 27 +++++++++++- tools/tracing/rtla/src/actions.h | 5 ++- tools/tracing/rtla/src/timerlat_hist.c | 40 +++++++++++++++--- tools/tracing/rtla/src/timerlat_top.c | 57 ++++++++++++++++---------- 4 files changed, 100 insertions(+), 29 deletions(-) diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/acti= ons.c index 63bee5bdabfd..aaf0808125d7 100644 --- a/tools/tracing/rtla/src/actions.c +++ b/tools/tracing/rtla/src/actions.c @@ -17,6 +17,7 @@ actions_init(struct actions *self) self->size =3D action_default_size; self->list =3D calloc(self->size, sizeof(struct action)); self->len =3D 0; + self->continue_flag =3D false; =20 memset(&self->present, 0, sizeof(self->present)); =20 @@ -108,6 +109,20 @@ actions_add_shell(struct actions *self, const char *co= mmand) return 0; } =20 +/* + * actions_add_continue - add an action to resume measurement + */ +int +actions_add_continue(struct actions *self) +{ + struct action *action =3D actions_new(self); + + self->present[ACTION_CONTINUE] =3D true; + action->type =3D ACTION_CONTINUE; + + return 0; +} + /* * actions_parse - add an action based on text specification */ @@ -133,6 +148,8 @@ actions_parse(struct actions *self, const char *trigger) type =3D ACTION_SIGNAL; else if (strcmp(token, "shell") =3D=3D 0) type =3D ACTION_SHELL; + else if (strcmp(token, "continue") =3D=3D 0) + type =3D ACTION_CONTINUE; else /* Invalid trigger type */ return -1; @@ -187,6 +204,11 @@ actions_parse(struct actions *self, const char *trigge= r) if (strlen(token) > 8 && strncmp(token, "command=3D", 8) =3D=3D 0) return actions_add_shell(self, token + 8); return -1; + case ACTION_CONTINUE: + /* Takes no argument */ + if (token !=3D NULL) + return -1; + return actions_add_continue(self); default: return -1; } @@ -196,7 +218,7 @@ actions_parse(struct actions *self, const char *trigger) * actions_perform - perform all actions */ int -actions_perform(const struct actions *self) +actions_perform(struct actions *self) { int pid, retval; const struct action *action; @@ -226,6 +248,9 @@ actions_perform(const struct actions *self) if (retval) return retval; break; + case ACTION_CONTINUE: + self->continue_flag =3D true; + return 0; default: break; } diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/acti= ons.h index 076bbff8519e..b10a19d55c49 100644 --- a/tools/tracing/rtla/src/actions.h +++ b/tools/tracing/rtla/src/actions.h @@ -7,6 +7,7 @@ enum action_type { ACTION_TRACE_OUTPUT, ACTION_SIGNAL, ACTION_SHELL, + ACTION_CONTINUE, ACTION_FIELD_N }; =20 @@ -35,6 +36,7 @@ struct actions { struct action *list; int len, size; bool present[ACTION_FIELD_N]; + bool continue_flag; =20 /* External dependencies */ struct tracefs_instance *trace_output_inst; @@ -45,5 +47,6 @@ void actions_destroy(struct actions *self); int actions_add_trace_output(struct actions *self, const char *trace_outpu= t); int actions_add_signal(struct actions *self, int signal, int pid); int actions_add_shell(struct actions *self, const char *command); +int actions_add_continue(struct actions *self); int actions_parse(struct actions *self, const char *trigger); -int actions_perform(const struct actions *self); +int actions_perform(struct actions *self); diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index d975d2cd6604..4f13a8f92711 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -1374,8 +1374,20 @@ int timerlat_hist_main(int argc, char *argv[]) goto out_hist; } =20 - if (osnoise_trace_is_off(tool, record)) - break; + if (osnoise_trace_is_off(tool, record)) { + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + trace_instance_start(trace); + } =20 /* is there still any user-threads ? */ if (params->user_workload) { @@ -1385,8 +1397,27 @@ int timerlat_hist_main(int argc, char *argv[]) } } } - } else - timerlat_bpf_wait(-1); + } else { + while (!stop_tracing) { + timerlat_bpf_wait(-1); + + if (!stop_tracing) { + /* Threshold overflow, perform actions on threshold */ + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + timerlat_bpf_restart_tracing(); + } + } + } =20 if (params->mode !=3D TRACING_MODE_TRACEFS) { timerlat_bpf_detach(); @@ -1412,7 +1443,6 @@ int timerlat_hist_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - actions_perform(¶ms->actions); return_value =3D FAILED; } =20 diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index cdbfda009974..60f9c78cb272 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -906,6 +906,7 @@ timerlat_top_set_signals(struct timerlat_params *params) static int timerlat_top_main_loop(struct osnoise_tool *top, struct osnoise_tool *record, + struct osnoise_tool *aa, struct timerlat_params *params, struct timerlat_u_params *params_u) { @@ -932,8 +933,20 @@ timerlat_top_main_loop(struct osnoise_tool *top, if (!params->quiet) timerlat_print_stats(params, top); =20 - if (osnoise_trace_is_off(top, record)) - break; + if (osnoise_trace_is_off(top, record)) { + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + trace_instance_start(trace); + } =20 /* is there still any user-threads ? */ if (params->user_workload) { @@ -953,6 +966,7 @@ timerlat_top_main_loop(struct osnoise_tool *top, static int timerlat_top_bpf_main_loop(struct osnoise_tool *top, struct osnoise_tool *record, + struct osnoise_tool *aa, struct timerlat_params *params, struct timerlat_u_params *params_u) { @@ -964,22 +978,9 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top, return 0; } =20 - if (params->quiet) { - /* Quiet mode: wait for stop and then, print results */ - timerlat_bpf_wait(-1); - - retval =3D timerlat_top_bpf_pull_data(top); - if (retval) { - err_msg("Error pulling BPF data\n"); - return retval; - } - - return 0; - } - /* Pull and display data in a loop */ while (!stop_tracing) { - wait_retval =3D timerlat_bpf_wait(params->sleep_time); + wait_retval =3D timerlat_bpf_wait(params->quiet ? -1 : params->sleep_tim= e); =20 retval =3D timerlat_top_bpf_pull_data(top); if (retval) { @@ -987,11 +988,24 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top, return retval; } =20 - timerlat_print_stats(params, top); + if (!params->quiet) + timerlat_print_stats(params, top); =20 - if (wait_retval =3D=3D 1) + if (wait_retval =3D=3D 1) { /* Stopping requested by tracer */ - break; + actions_perform(¶ms->actions); + + if (!params->actions.continue_flag) + /* continue flag not set, break */ + break; + + /* continue action reached, re-enable tracing */ + if (params->actions.present[ACTION_TRACE_OUTPUT]) + trace_instance_start(&record->trace); + if (!params->no_aa) + trace_instance_start(&aa->trace); + timerlat_bpf_restart_tracing(); + } =20 /* is there still any user-threads ? */ if (params->user_workload) { @@ -1205,9 +1219,9 @@ int timerlat_top_main(int argc, char *argv[]) timerlat_top_set_signals(params); =20 if (params->mode =3D=3D TRACING_MODE_TRACEFS) - retval =3D timerlat_top_main_loop(top, record, params, ¶ms_u); + retval =3D timerlat_top_main_loop(top, record, aa, params, ¶ms_u); else - retval =3D timerlat_top_bpf_main_loop(top, record, params, ¶ms_u); + retval =3D timerlat_top_bpf_main_loop(top, record, aa, params, ¶ms_u= ); =20 if (retval) goto out_top; @@ -1230,7 +1244,6 @@ int timerlat_top_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - actions_perform(¶ms->actions); return_value =3D FAILED; } else if (params->aa_only) { /* --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDF8D2E4263 for ; Thu, 26 Jun 2025 12:36:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941390; cv=none; b=Ia+yphJ3zNdO7sgArLtRnlj8tSabTQw4Ezclfpvi+hAWbhlK8zrp8OKmIPzfe62f0cdMW4KGBkYz6Kgs7Y0m02CgGMnhwiiG3B5PHz9EKU7L0xOmMktjkJsUSfNXAYR80osaWC7JdAfH+xO7rpwnLxBRt/YbvRbZ9/omymcNafk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941390; c=relaxed/simple; bh=72wucJGD61ASXRvziRF37YPvfvCkBbclhcdwDvqUTsI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HixNwX4EBryNLfaS3gBh2EkeB+6sjBES2lRHZmPCo9BCOZy0vUL3SiynwL0ne4S+3freDPuzK8yLy+2gIWwd3DfcnY5cjAI8TJC1imIb+ThBcOg4pDH5Yh3MdjbKYQYAtAiq4cq4xPv4TP7S0o9i0tGkuv0qe93YgFAMv5EAIm4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LwgVGQ+9; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LwgVGQ+9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941387; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wMJg0dH4UPkLa1Hf7lH56pyNZrxSWYtoTrC2yVPM9go=; b=LwgVGQ+9aepeLdQLetTOmj6eOstRA+C1uhpTNZfTDzDshENOEd5lklrvEI0jcI7mvc2MR2 iUvPUdLpj+SBpwMCqvn8x2ZyjZdya0ufZy3ZcKNe2bdikAxwixoHHSRJstP9itOB07c1Uc dgMHrw7Sfamx8zTibw7/+OsmCtuJXQw= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-7-3f56Zdm0M2OV4FIVE3J4eQ-1; Thu, 26 Jun 2025 08:36:24 -0400 X-MC-Unique: 3f56Zdm0M2OV4FIVE3J4eQ-1 X-Mimecast-MFC-AGG-ID: 3f56Zdm0M2OV4FIVE3J4eQ_1750941383 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4BA5419560B9; Thu, 26 Jun 2025 12:36:23 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 18B1B18003FC; Thu, 26 Jun 2025 12:36:18 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 5/9] rtla/timerlat: Add action on end feature Date: Thu, 26 Jun 2025 14:34:01 +0200 Message-ID: <20250626123405.1496931-6-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Implement actions on end next to actions on threshold. A new option, --on-end is added, parallel to --on-threshold. Instead of being executed whenever a latency threshold is reached, it is executed at the end of the measurement. For example: $ rtla timerlat hist -d 5s --on-end trace will save the trace output at the end. All actions supported by --on-threshold are also supported by --on-end, except for continue, which does nothing with --on-end. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/timerlat.h | 5 ++- tools/tracing/rtla/src/timerlat_hist.c | 44 +++++++++++++++++-------- tools/tracing/rtla/src/timerlat_top.c | 45 ++++++++++++++++++-------- 3 files changed, 65 insertions(+), 29 deletions(-) diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/tim= erlat.h index d1fcf9a97621..bc55ed04fc96 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -48,7 +48,10 @@ struct timerlat_params { struct sched_attr sched_param; struct trace_events *events; enum timerlat_tracing_mode mode; - struct actions actions; + + struct actions threshold_actions; + struct actions end_actions; + union { struct { /* top only */ diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 4f13a8f92711..9baea1b251ed 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -758,6 +758,7 @@ static void timerlat_hist_usage(char *usage) " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", " --on-threshold : define action to be executed at latency = threshold, multiple are allowed", + " --on-end : define action to be executed at measurement en= d, multiple are allowed", NULL, }; =20 @@ -793,7 +794,8 @@ static struct timerlat_params if (!params) exit(1); =20 - actions_init(¶ms->actions); + actions_init(¶ms->threshold_actions); + actions_init(¶ms->end_actions); =20 /* disabled by default */ params->dma_latency =3D -1; @@ -846,6 +848,7 @@ static struct timerlat_params {"trace-buffer-size", required_argument, 0, '\3'}, {"deepest-idle-state", required_argument, 0, '\4'}, {"on-threshold", required_argument, 0, '\5'}, + {"on-end", required_argument, 0, '\6'}, {0, 0, 0, 0} }; =20 @@ -1038,7 +1041,14 @@ static struct timerlat_params params->deepest_idle_state =3D get_llong_from_str(optarg); break; case '\5': - retval =3D actions_parse(¶ms->actions, optarg); + retval =3D actions_parse(¶ms->threshold_actions, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } + break; + case '\6': + retval =3D actions_parse(¶ms->end_actions, optarg); if (retval) { err_msg("Invalid action %s\n", optarg); exit(EXIT_FAILURE); @@ -1050,7 +1060,7 @@ static struct timerlat_params } =20 if (trace_output) - actions_add_trace_output(¶ms->actions, trace_output); + actions_add_trace_output(¶ms->threshold_actions, trace_output); =20 if (geteuid()) { err_msg("rtla needs root permission\n"); @@ -1077,7 +1087,8 @@ static struct timerlat_params * mixed mode */ if (params->mode =3D=3D TRACING_MODE_BPF && - (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) + (params->threshold_actions.present[ACTION_TRACE_OUTPUT] || + params->end_actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1270,13 +1281,15 @@ int timerlat_hist_main(int argc, char *argv[]) } } =20 - if (params->actions.present[ACTION_TRACE_OUTPUT]) { + if (params->threshold_actions.present[ACTION_TRACE_OUTPUT] || + params->end_actions.present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); goto out_free; } - params->actions.trace_output_inst =3D record->trace.inst; + params->threshold_actions.trace_output_inst =3D record->trace.inst; + params->end_actions.trace_output_inst =3D record->trace.inst; =20 if (params->events) { retval =3D trace_events_enable(&record->trace, params->events); @@ -1342,7 +1355,7 @@ int timerlat_hist_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->actions.present[ACTION_TRACE_OUTPUT]) + if (record) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1375,14 +1388,14 @@ int timerlat_hist_main(int argc, char *argv[]) } =20 if (osnoise_trace_is_off(tool, record)) { - actions_perform(¶ms->actions); + actions_perform(¶ms->threshold_actions); =20 - if (!params->actions.continue_flag) + if (!params->threshold_actions.continue_flag) /* continue flag not set, break */ break; =20 /* continue action reached, re-enable tracing */ - if (params->actions.present[ACTION_TRACE_OUTPUT]) + if (record) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1403,14 +1416,14 @@ int timerlat_hist_main(int argc, char *argv[]) =20 if (!stop_tracing) { /* Threshold overflow, perform actions on threshold */ - actions_perform(¶ms->actions); + actions_perform(¶ms->threshold_actions); =20 - if (!params->actions.continue_flag) + if (!params->threshold_actions.continue_flag) /* continue flag not set, break */ break; =20 /* continue action reached, re-enable tracing */ - if (params->actions.present[ACTION_TRACE_OUTPUT]) + if (record) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1435,6 +1448,8 @@ int timerlat_hist_main(int argc, char *argv[]) =20 timerlat_print_stats(params, tool); =20 + actions_perform(¶ms->end_actions); + return_value =3D PASSED; =20 if (osnoise_trace_is_off(tool, record) && !stop_tracing) { @@ -1464,7 +1479,8 @@ int timerlat_hist_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(tool); - actions_destroy(¶ms->actions); + actions_destroy(¶ms->threshold_actions); + actions_destroy(¶ms->end_actions); if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); free(params); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 60f9c78cb272..c80b81c0b4da 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -517,6 +517,7 @@ static void timerlat_top_usage(char *usage) " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", " --on-threshold : define action to be executed at latency = threshold, multiple are allowed", + " --on-end: define action to be executed at measurement end, multip= le are allowed", NULL, }; =20 @@ -552,7 +553,8 @@ static struct timerlat_params if (!params) exit(1); =20 - actions_init(¶ms->actions); + actions_init(¶ms->threshold_actions); + actions_init(¶ms->end_actions); =20 /* disabled by default */ params->dma_latency =3D -1; @@ -597,6 +599,7 @@ static struct timerlat_params {"trace-buffer-size", required_argument, 0, '7'}, {"deepest-idle-state", required_argument, 0, '8'}, {"on-threshold", required_argument, 0, '9'}, + {"on-end", required_argument, 0, '\1'}, {0, 0, 0, 0} }; =20 @@ -623,6 +626,7 @@ static struct timerlat_params =20 /* set trace */ trace_output =3D "timerlat_trace.txt"; + break; case '5': /* it is here because it is similar to -a */ @@ -776,7 +780,14 @@ static struct timerlat_params params->deepest_idle_state =3D get_llong_from_str(optarg); break; case '9': - retval =3D actions_parse(¶ms->actions, optarg); + retval =3D actions_parse(¶ms->threshold_actions, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } + break; + case '\1': + retval =3D actions_parse(¶ms->end_actions, optarg); if (retval) { err_msg("Invalid action %s\n", optarg); exit(EXIT_FAILURE); @@ -788,7 +799,7 @@ static struct timerlat_params } =20 if (trace_output) - actions_add_trace_output(¶ms->actions, trace_output); + actions_add_trace_output(¶ms->threshold_actions, trace_output); =20 if (geteuid()) { err_msg("rtla needs root permission\n"); @@ -812,7 +823,8 @@ static struct timerlat_params * mixed mode */ if (params->mode =3D=3D TRACING_MODE_BPF && - (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) + (params->threshold_actions.present[ACTION_TRACE_OUTPUT] || + params->end_actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -934,14 +946,14 @@ timerlat_top_main_loop(struct osnoise_tool *top, timerlat_print_stats(params, top); =20 if (osnoise_trace_is_off(top, record)) { - actions_perform(¶ms->actions); + actions_perform(¶ms->threshold_actions); =20 - if (!params->actions.continue_flag) + if (!params->threshold_actions.continue_flag) /* continue flag not set, break */ break; =20 /* continue action reached, re-enable tracing */ - if (params->actions.present[ACTION_TRACE_OUTPUT]) + if (record) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -993,14 +1005,14 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top, =20 if (wait_retval =3D=3D 1) { /* Stopping requested by tracer */ - actions_perform(¶ms->actions); + actions_perform(¶ms->threshold_actions); =20 - if (!params->actions.continue_flag) + if (!params->threshold_actions.continue_flag) /* continue flag not set, break */ break; =20 /* continue action reached, re-enable tracing */ - if (params->actions.present[ACTION_TRACE_OUTPUT]) + if (record) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1128,13 +1140,15 @@ int timerlat_top_main(int argc, char *argv[]) } } =20 - if (params->actions.present[ACTION_TRACE_OUTPUT]) { + if (params->threshold_actions.present[ACTION_TRACE_OUTPUT] || + params->end_actions.present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); goto out_free; } - params->actions.trace_output_inst =3D record->trace.inst; + params->threshold_actions.trace_output_inst =3D record->trace.inst; + params->end_actions.trace_output_inst =3D record->trace.inst; =20 if (params->events) { retval =3D trace_events_enable(&record->trace, params->events); @@ -1201,7 +1215,7 @@ int timerlat_top_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->actions.present[ACTION_TRACE_OUTPUT]) + if (record) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1236,6 +1250,8 @@ int timerlat_top_main(int argc, char *argv[]) =20 timerlat_print_stats(params, top); =20 + actions_perform(¶ms->end_actions); + return_value =3D PASSED; =20 if (osnoise_trace_is_off(top, record) && !stop_tracing) { @@ -1276,7 +1292,8 @@ int timerlat_top_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(top); - actions_destroy(¶ms->actions); + actions_destroy(¶ms->threshold_actions); + actions_destroy(¶ms->end_actions); if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); free(params); --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1E0D2E426C for ; Thu, 26 Jun 2025 12:36:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941391; cv=none; b=HTyhdq/UIjCrQabDkcMlUHtYedYDrPdQDc2JiqbbT8/sHSqPVWmTN6J1cWwjREw1zQHAhXWvW+D+AotrYmsPeIjOxsyYiv7Ksi+iFtGSQ90MoE7qurs6rBAgnmf+CYA2fAugucNbLIQ4Y0gyi3XwBm95IJoR0Jc/WEk6D35zX+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941391; c=relaxed/simple; bh=khl73MPmUq799HOT+UF9QQ+r9png3+sHIge1qd1A1WY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K9+bOk/I3B+h3AO4bdW06nJIO2H802sfoPkCrqxCRV+bxboLspNB5iISjRUD1ZKbGDEReijLiF+fO6P6NGaLVdvYb2CMGGTyMBefe9faCejAgwwjRdHoVbTdT1aZkjgCUZvnYgrOlPvVweROB/OtG7Fk7Stk/EcwqorZh6CQcoA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Bu2rOC2B; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Bu2rOC2B" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941389; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cEiqq7isWZiK/FNjFSdCqNp6cYFEr/QW0LyfClcFOf0=; b=Bu2rOC2BwlP79hlfQszcUEFqIiKcz9qsUfBTy42ZDnalz4G2SsfChs0XhYaM0BNLnYVUPz NkpYerYAYvW4vCoe4a4fyE/Lb2KrHdFDhiIdGBDH2j8PIZ58gQevkR4mEwYwDDYFvw7Ufa QoBtujMnjVJ30M5XYyrrj+H3XJ5dm60= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-21-v82t7H5_NreYnjcK-hxVxA-1; Thu, 26 Jun 2025 08:36:27 -0400 X-MC-Unique: v82t7H5_NreYnjcK-hxVxA-1 X-Mimecast-MFC-AGG-ID: v82t7H5_NreYnjcK-hxVxA_1750941386 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 96C761800289; Thu, 26 Jun 2025 12:36:26 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B48E918003FC; Thu, 26 Jun 2025 12:36:23 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 6/9] rtla/tests: Check rtla output with grep Date: Thu, 26 Jun 2025 14:34:02 +0200 Message-ID: <20250626123405.1496931-7-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Add argument to the check command in the test suite that takes a regular expression that the output of rtla command is checked against. This allows testing for specific information in rtla output in addition to checking the return value. Two minor improvements are included: running rtla with "eval" so that arguments with spaces can be passed to it via shell quotations, and the stdout of pushd and popd is suppressed to clean up the test output. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/tests/engine.sh | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/tools/tracing/rtla/tests/engine.sh b/tools/tracing/rtla/tests/= engine.sh index f2616a8e4179..64c5be4313de 100644 --- a/tools/tracing/rtla/tests/engine.sh +++ b/tools/tracing/rtla/tests/engine.sh @@ -11,7 +11,7 @@ test_begin() { reset_osnoise() { # Reset osnoise options to default and remove any dangling instances crea= ted # by improperly exited rtla runs. - pushd /sys/kernel/tracing || return 1 + pushd /sys/kernel/tracing >/dev/null || return 1 =20 # Remove dangling instances created by previous rtla run echo 0 > tracing_thresh @@ -35,11 +35,14 @@ reset_osnoise() { echo 0 > stop_tracing_us echo 1000 > timerlat_period_us =20 - popd + popd >/dev/null } =20 check() { + test_name=3D$0 + tested_command=3D$1 expected_exitcode=3D${3:-0} + expected_output=3D$4 # Simple check: run rtla with given arguments and test exit code. # If TEST_COUNT is set, run the test. Otherwise, just count. ctr=3D$(($ctr + 1)) @@ -49,8 +52,16 @@ check() { [ "$NO_RESET_OSNOISE" =3D=3D 1 ] || reset_osnoise # Run rtla; in case of failure, include its output as comment # in the test results. - result=3D$(stdbuf -oL $TIMEOUT "$RTLA" $2 2>&1); exitcode=3D$? - if [ $exitcode -eq $expected_exitcode ] + result=3D$(eval stdbuf -oL $TIMEOUT "$RTLA" $2 2>&1); exitcode=3D$? + # Test if the results matches if requested + if [ -n "$expected_output" ] + then + grep -E "$expected_output" <<< "$result" > /dev/null; grep_result=3D$? + else + grep_result=3D0 + fi + + if [ $exitcode -eq $expected_exitcode ] && [ $grep_result -eq 0 ] then echo "ok $ctr - $1" else @@ -58,6 +69,8 @@ check() { # Add rtla output and exit code as comments in case of failure echo "$result" | col -b | while read line; do echo "# $line"; done printf "#\n# exit code %s\n" $exitcode + [ -n "$expected_output" ] && \ + printf "# Output match failed: \"%s\"\n" "$expected_output" fi fi } --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46AAD2E5401 for ; Thu, 26 Jun 2025 12:36:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941394; cv=none; b=ageWDbrSqFk6jUieYIC4l/V0VCK3plFffOf2RpnwWzwGhMDnTv41z1lwy0u1RDQluPOeXJ9mMcrYFHDZViBY24nhWNR2r7dLe1sDZlZgVOKrA+ugIBP1CQBg+UBzsfuYEKFKnMY4hxNVBdb8ZVY2Lybu3O1Cf0WfuG660qR8pXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941394; c=relaxed/simple; bh=tbmjAPe9K8kZku7BODFUd9I5r2ppEDax4yKPfCbGKBI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z+HLEbzVdgQRI34wT9UJKjgpHKfzU+8Rkjxis9pO7viMcxunxevjFVJfFNli70w5sy3zvc7M2cjN1MKHceq4yqC9cp8OsmmH03a+zF+FxIVlncWgeGT8oxEZboau1zNg+9ZGAJxrSTY56E6Xid0lKH+kLVpMEyXqPzLk3QL550Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XA9KFSK9; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XA9KFSK9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941392; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h8T4xpbeiOqSe/DoRZSEQyk76AEAAvRMDQwoAZeiH0M=; b=XA9KFSK94LLyq8iCIgMju76KPqtodtJLN540aYsO1k8j0pTg8HmM0P+6S4G1j95DuZo5zw vDHiSIHcBhTuWsDm3ASEpsMUcZ+4Ti3Y7S4ZtgXSRlS9bgU15dT4o59xPa21/HILnAC8+Y tCsaou7A6MbmMEStFGxQHxahgjUHLfQ= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-278-IFnfb8ycMiSst3g2Kq6N2Q-1; Thu, 26 Jun 2025 08:36:30 -0400 X-MC-Unique: IFnfb8ycMiSst3g2Kq6N2Q-1 X-Mimecast-MFC-AGG-ID: IFnfb8ycMiSst3g2Kq6N2Q_1750941390 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DF1A01809C80; Thu, 26 Jun 2025 12:36:29 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0AB6918003FC; Thu, 26 Jun 2025 12:36:26 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 7/9] rtla/tests: Add tests for actions Date: Thu, 26 Jun 2025 14:34:03 +0200 Message-ID: <20250626123405.1496931-8-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Add a bunch of tests covering most of both --on-threshold and --on-end. Parts sensitive to implementation of hist/top are tested for both. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/tests/timerlat.t | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests= /timerlat.t index 579c12a85e8f..b354bacd78f8 100644 --- a/tools/tracing/rtla/tests/timerlat.t +++ b/tools/tracing/rtla/tests/timerlat.t @@ -18,6 +18,8 @@ fi for option in $no_bpf_options do export RTLA_NO_BPF=3D$option + +# Basic tests check "verify help page" \ "timerlat --help" check "verify -s/--stack" \ @@ -36,6 +38,32 @@ check "verify -c/--cpus" \ "timerlat hist -c 0 -d 30s" check "hist test in nanoseconds" \ "timerlat hist -i 2 -c 0 -n -d 30s" 2 + +# Actions tests +check "trace output through -t" \ + "timerlat hist -T 2 -t" 2 "^ Saving trace to timerlat_trace.txt$" +check "trace output through -t with custom filename" \ + "timerlat hist -T 2 -t custom_filename.txt" 2 "^ Saving trace to custom_= filename.txt$" +check "trace output through -A trace" \ + "timerlat hist -T 2 --on-threshold trace" 2 "^ Saving trace to timerlat_= trace.txt$" +check "trace output through -A trace with custom filename" \ + "timerlat hist -T 2 --on-threshold trace,file=3Dcustom_filename.txt" 2 "^= Saving trace to custom_filename.txt$" +check "exec command" \ + "timerlat hist -T 2 --on-threshold shell,command=3D'echo TestOutput'" 2 "= ^TestOutput$" +check "multiple actions" \ + "timerlat hist -T 2 --on-threshold shell,command=3D'echo -n 1' --on-thres= hold shell,command=3D'echo 2'" 2 "^12$" +check "hist stop at failed action" \ + "timerlat hist -T 2 --on-threshold shell,command=3D'echo -n 1; false' --o= n-threshold shell,command=3D'echo -n 2'" 2 "^1# RTLA timerlat histogram$" +check "top stop at failed action" \ + "timerlat top -T 2 --on-threshold shell,command=3D'echo -n 1; false' --on= -threshold shell,command=3D'echo -n 2'" 2 "^1ALL" +check "hist with continue" \ + "timerlat hist -T 2 -d 1s --on-threshold shell,command=3D'echo TestOutput= ' --on-threshold continue" 0 "^TestOutput$" +check "top with continue" \ + "timerlat top -q -T 2 -d 1s --on-threshold shell,command=3D'echo TestOutp= ut' --on-threshold continue" 0 "^TestOutput$" +check "hist with trace output at end" \ + "timerlat hist -d 1s --on-end trace" 0 "^ Saving trace to timerlat_trace= .txt$" +check "top with trace output at end" \ + "timerlat top -d 1s --on-end trace" 0 "^ Saving trace to timerlat_trace.= txt$" done =20 test_end --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E4AF2E5424 for ; Thu, 26 Jun 2025 12:36:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941398; cv=none; b=kYu5nP133g1amoPIKNJFTm9QX88OWYkeD2aYJPtw4DvzmB9l/4+21oGXzGJLMeoMzxLP3GrF45PT+z80XbNRSvl7DRKb/d70dq9/J/stT1SgQMDtb4KRFZ8mJu6f4kUQqwgGalc2L7vQ4z0NWdXxNdv+SJ2Kh7suHzDkd+OF3YE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941398; c=relaxed/simple; bh=7KTVanJ6wbwwpcVqvgg95UabiBY/hXBvFCMEZYdwtVA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P1RdBCuIgJHOpoZAIFi3DEP3hXzkyjtNc6L7pGMlzINd5Ssup0IOS3BQ6hG8qjC+sME07Efi82rmREG+LxFaJ7sEoC10se959gKD4gMpBqog3NRq1NrEeY7f3/l7NZGaRMTum20Vo3GqiEa2LmLgAKkZnB+ziteGuuFE0SMvrzE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=AFkl8Qon; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AFkl8Qon" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941395; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wM5d6m53RcugK0bflFiBhXo5+UqRE346sWjqDVNkzJI=; b=AFkl8QonU8wCQaMVwmaD749xjd4su1pEZjcIb/BfZtoy4WWNO2feeAfkv38+rqc5QeNpZT Eh1MOWeSBd018zyu3PSLplbdqxPeaLVciEqnDmC7wVMEr34UzQkmLn9+w8opkZCD3SiMWP A9OBNRDX/mzlmDIDz6wJxj2OFPdoyFs= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-224-xsulNSpfOQmx8lqeCc_uFw-1; Thu, 26 Jun 2025 08:36:34 -0400 X-MC-Unique: xsulNSpfOQmx8lqeCc_uFw-1 X-Mimecast-MFC-AGG-ID: xsulNSpfOQmx8lqeCc_uFw_1750941393 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 348641801210; Thu, 26 Jun 2025 12:36:33 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 533DD18003FC; Thu, 26 Jun 2025 12:36:30 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 8/9] rtla/tests: Limit duration to maximum of 10s Date: Thu, 26 Jun 2025 14:34:04 +0200 Message-ID: <20250626123405.1496931-9-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Many of the original rtla tests included durations of 1 minute and 30 seconds. Experience has shown this is unnecessary, since 10 seconds as waiting time for samples to appear. Change duration of all rtla tests to at most 10 seconds. This speeds up testing significantly. Before: $ make check All tests successful. Files=3D3, Tests=3D54, 536 wallclock secs ( 0.03 usr 0.00 sys + 20.31 cusr 22.02 csys =3D 42.36 CPU) Result: PASS After: $ make check ... All tests successful. Files=3D3, Tests=3D54, 196 wallclock secs ( 0.03 usr 0.01 sys + 20.28 cusr 20.68 csys =3D 41.00 CPU) Result: PASS Signed-off-by: Tomas Glozar --- tools/tracing/rtla/tests/hwnoise.t | 8 ++++---- tools/tracing/rtla/tests/osnoise.t | 4 ++-- tools/tracing/rtla/tests/timerlat.t | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/tools/tracing/rtla/tests/hwnoise.t b/tools/tracing/rtla/tests/= hwnoise.t index 5f71401a139e..448877564b8d 100644 --- a/tools/tracing/rtla/tests/hwnoise.t +++ b/tools/tracing/rtla/tests/hwnoise.t @@ -10,12 +10,12 @@ check "verify help page" \ check "detect noise higher than one microsecond" \ "hwnoise -c 0 -T 1 -d 5s -q" check "set the automatic trace mode" \ - "hwnoise -a 5 -d 30s" 2 + "hwnoise -a 5 -d 10s" 2 check "set scheduling param to the osnoise tracer threads" \ - "hwnoise -P F:1 -c 0 -r 900000 -d 1M -q" + "hwnoise -P F:1 -c 0 -r 900000 -d 10s -q" check "stop the trace if a single sample is higher than 1 us" \ - "hwnoise -s 1 -T 1 -t -d 30s" 2 + "hwnoise -s 1 -T 1 -t -d 10s" 2 check "enable a trace event trigger" \ - "hwnoise -t -e osnoise:irq_noise trigger=3D\"hist:key=3Ddesc,duration:sor= t=3Ddesc,duration:vals=3Dhitcount\" -d 1m" + "hwnoise -t -e osnoise:irq_noise trigger=3D\"hist:key=3Ddesc,duration:sor= t=3Ddesc,duration:vals=3Dhitcount\" -d 10s" =20 test_end diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/= osnoise.t index 44908fc01abf..6a4dfa31dc55 100644 --- a/tools/tracing/rtla/tests/osnoise.t +++ b/tools/tracing/rtla/tests/osnoise.t @@ -8,13 +8,13 @@ set_timeout 2m check "verify help page" \ "osnoise --help" check "verify the --priority/-P param" \ - "osnoise top -P F:1 -c 0 -r 900000 -d 1M -q" + "osnoise top -P F:1 -c 0 -r 900000 -d 10s -q" check "verify the --stop/-s param" \ "osnoise top -s 30 -T 1 -t" 2 check "verify the --trace param" \ "osnoise hist -s 30 -T 1 -t" 2 check "verify the --entries/-E param" \ - "osnoise hist -P F:1 -c 0 -r 900000 -d 1M -b 10 -E 25" + "osnoise hist -P F:1 -c 0 -r 900000 -d 10s -b 10 -E 25" =20 # Test setting default period by putting an absurdly high period # and stopping on threshold. diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests= /timerlat.t index b354bacd78f8..2d59ee199c4d 100644 --- a/tools/tracing/rtla/tests/timerlat.t +++ b/tools/tracing/rtla/tests/timerlat.t @@ -25,9 +25,9 @@ check "verify help page" \ check "verify -s/--stack" \ "timerlat top -s 3 -T 10 -t" 2 check "verify -P/--priority" \ - "timerlat top -P F:1 -c 0 -d 1M -q" + "timerlat top -P F:1 -c 0 -d 10s -q" check "test in nanoseconds" \ - "timerlat top -i 2 -c 0 -n -d 30s" 2 + "timerlat top -i 2 -c 0 -n -d 10s" 2 check "set the automatic trace mode" \ "timerlat top -a 5 --dump-tasks" 2 check "print the auto-analysis if hits the stop tracing condition" \ @@ -35,9 +35,9 @@ check "print the auto-analysis if hits the stop tracing c= ondition" \ check "disable auto-analysis" \ "timerlat top -s 3 -T 10 -t --no-aa" 2 check "verify -c/--cpus" \ - "timerlat hist -c 0 -d 30s" + "timerlat hist -c 0 -d 10s" check "hist test in nanoseconds" \ - "timerlat hist -i 2 -c 0 -n -d 30s" 2 + "timerlat hist -i 2 -c 0 -n -d 10s" 2 =20 # Actions tests check "trace output through -t" \ --=20 2.49.0 From nobody Wed Oct 8 16:04:24 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A67E42E542F for ; Thu, 26 Jun 2025 12:36:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941401; cv=none; b=PG17W4244mTeWHDpLZjVNxMTJp0qLuaQnGJx0M8fDFJuUDXBSvhHa/Q6WlFBr+QfvdsrKm0VKHJJJ/y2t8BAr1cFbV3F7j4fFfXPY+a8NXzgFZROxhElXLjuQfq57Uoxl5YNxXQwMMANyDIAbJ4wHslyydzJTOm0KewnQPvXmmo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750941401; c=relaxed/simple; bh=M3hdx2skcWoqfn5nBTo8ujRdHVpFUS3GEfGFgitw2iM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QlIIvncpDPt5ZTcxY+EtcgHA4lulQNpymA3rXtx4nn5Hw4TDz6ky3KEwb8EkYi97NOVazwCnWywhX80Cg5IOYWJbDXMolqwefKmvSplm6P/HHZ1ztevU8q6ypEE8tCRdWQ/r5M+CggmTaMPY+NsoV0BD6KlbIuDLw6OuTome3vM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=I8A+iu+2; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="I8A+iu+2" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750941398; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M92KV+ly5nxnIVd6n/jgRKgpirFRCSP8wIHHelv0nmQ=; b=I8A+iu+25ae6OZobViB2/82nClalf9+v2awl3HZqFW+znjAV98bCfqvPsj+vacO3JPUU0V ahF4kA+n7I6DOEjHErUgD/0i1iiREVWqYWWroe1M2j/DaO8L5FWjSosugh9t3YmDE1AHJQ HxK/yU3tZukLSQK6LdGdYSlBrS8yKCw= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-313-ZcDS_bLPN1iw_IT1Y43BWg-1; Thu, 26 Jun 2025 08:36:37 -0400 X-MC-Unique: ZcDS_bLPN1iw_IT1Y43BWg-1 X-Mimecast-MFC-AGG-ID: ZcDS_bLPN1iw_IT1Y43BWg_1750941396 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7B2241978C9A; Thu, 26 Jun 2025 12:36:36 +0000 (UTC) Received: from fedora.brq.redhat.com (unknown [10.43.17.241]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9B6E7180045C; Thu, 26 Jun 2025 12:36:33 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Arnaldo Carvalho de Melo , Chang Yin , Costa Shulyupin , Crystal Wood , Gabriele Monaco , Tomas Glozar Subject: [PATCH v2 9/9] Documentation/rtla: Add actions feature Date: Thu, 26 Jun 2025 14:34:05 +0200 Message-ID: <20250626123405.1496931-10-tglozar@redhat.com> In-Reply-To: <20250626123405.1496931-1-tglozar@redhat.com> References: <20250626123405.1496931-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Document both --on-threshold and --on-end, with examples. Signed-off-by: Tomas Glozar --- .../tools/rtla/common_timerlat_options.rst | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/Documentation/tools/rtla/common_timerlat_options.rst b/Documen= tation/tools/rtla/common_timerlat_options.rst index 10dc802f8d65..7854368f1827 100644 --- a/Documentation/tools/rtla/common_timerlat_options.rst +++ b/Documentation/tools/rtla/common_timerlat_options.rst @@ -55,3 +55,67 @@ Set timerlat to run without workload, waiting for the user to disp= atch a per-cpu task that waits for a new period on the tracing/osnoise/per_cpu/cp= u$ID/timerlat_fd. See linux/tools/rtla/sample/timerlat_load.py for an example of use= r-load code. + +**--on-threshold** *action* + + Defines an action to be executed when tracing is stopped on a late= ncy threshold + specified by **-i/--irq** or **-T/--thread**. + + Multiple --on-threshold actions may be specified, and they will be= executed in + the order they are provided. If any action fails, subsequent actio= ns in the list + will not be executed. + + Supported actions are: + + - *trace[,file=3D]* + + Saves trace output, optionally taking a filename. Alternative to= -t/--trace. + Note that nlike -t/--trace, specifying this multiple times will = result in + the trace being saved multiple times. + + - *signal,num=3D,pid=3D* + + Sends signal to process. "parent" might be specified in place of= pid to target + the parent process of rtla. + + - *shell,command=3D* + + Execute shell command. + + - *continue* + + Continue tracing after actions are executed instead of stopping. + + Example: + + $ rtla timerlat -T 20 --on-threshold trace + --on-threshold shell,command=3D"grep ipi_send timerlat_trace.txt" + --on-threshold signal,num=3D2,pid=3Dparent + + This will save a trace with the default filename "timerlat_trace.t= xt", print its + lines that contain the text "ipi_send" on standard output, and sen= d signal 2 + (SIGINT) to the parent process. + + Performance Considerations: + + For time-sensitive actions, it is recommended to run **rtla timerl= at** with BPF + support and RT priority. Note that due to implementational limitat= ions, actions + might be delayed up to one second after tracing is stopped if BPF = mode is not + available or disabled. + +**--on-end** *action* + + Defines an action to be executed at the end of **rtla timerlat** t= racing. + + Multiple --on-end actions can be specified, and they will be execu= ted in the order + they are provided. If any action fails, subsequent actions in the = list will not be + executed. + + See the documentation for **--on-threshold** for the list of suppo= rted actions, with + the exception that *continue* has no effect. + + Example: + + $ rtla timerlat -d 5s --on-end trace + + This runs rtla timerlat with default options and save trace output= at the end. --=20 2.49.0