From nobody Tue Feb 10 23:12:03 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89808548EE for ; Wed, 28 May 2025 14:32:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748442767; cv=none; b=SuB+dv1p5o427Pvw13lQ44CjF1qOhX8NrbSwuoKHPorv+L23HCC4B9/49VBZGbt/bsoYRjzHhmALxz3EtWl+DmwSWj/Pv3P61t/643VrCAZ0xbhdla6AjGkGpqAzzxEW9LViKWH+hJr8/yQCWLssBbv7APupnBBJXDKLdfLsvE4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748442767; c=relaxed/simple; bh=7pzEOFDkSk4OGTttMhg7+3fNiqnabN4HSGPccqE4KZs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PeK9BjvobtPZG6gWpzmTKExKzJj6VPBmq3HzWJX2PUjav2ad010A5lilXsNJkxDtzG2lQ76MLhTO9zBSxVl1XvQYW/Ct8ScW1UHZ2ad1CVcdlZAUO6kXJIcB9epRg0t43nmibxxPSAqCJRMfGOhGNUQdRhqPd/3UszK9xsQNuIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Q5jHZ/4r; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Q5jHZ/4r" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1748442764; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X9FghnowdvIfcJWFBcyPjWLjwFh8ZP/bVgyoxmiG59E=; b=Q5jHZ/4rsTZEzbRZT2UMrfi5Sgoqj2njTxSLqRg97TcJGOJsWnFRHJglidttmk8GGNcbXY AzO8uLPiWOlYi4gE5KT91WhHRw+sB3D0XctxSA2n8FDS3nORy8s6O7rZ6HWhFQ3WvUTs1P kzcrXwcKBXf+6nLjR5nbMQZnf5/M9ys= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-387-V5p-lMsaMT68GLKarubc0Q-1; Wed, 28 May 2025 10:32:40 -0400 X-MC-Unique: V5p-lMsaMT68GLKarubc0Q-1 X-Mimecast-MFC-AGG-ID: V5p-lMsaMT68GLKarubc0Q_1748442759 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9B8B6195608A; Wed, 28 May 2025 14:32:39 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.45.225.94]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 405A019560AA; Wed, 28 May 2025 14:32:36 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Tomas Glozar Subject: [RFC PATCH 1/2] rtla/timerlat: Introduce enum timerlat_tracing_mode Date: Wed, 28 May 2025 16:28:57 +0200 Message-ID: <20250528142858.185017-2-tglozar@redhat.com> In-Reply-To: <20250528142858.185017-1-tglozar@redhat.com> References: <20250528142858.185017-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 Content-Type: text/plain; charset="utf-8" After the introduction of BPF-based sample collection, rtla-timerlat effectively runs in one of three modes: - Pure BPF mode, with tracefs only being used to set up the timerlat tracer. Sample processing and stop on threshold are handled by BPF. - tracefs mode. BPF is unsupported or kernel is lacking the necessary trace event (osnoise:timerlat_sample). Stop on theshold is handled by timerlat tracer stopping tracing in all instances. - BPF/tracefs mixed mode - BPF is used for sample collection for top or histogram, tracefs is used for trace output and/or auto-analysis. Stop on threshold is handled both through BPF program, which stops sample collection for top/histogram and wakes up rtla, and by timerlat tracer, which stops tracing for trace output/auto-analysis instances. Add enum timerlat_tracing_mode, with three values: - TRACING_MODE_BPF - TRACING_MODE_TRACEFS - TRACING_MODE_MIXED Those represent the modes described above. A field of this type is added to struct timerlat_params, named "mode", replacing the no_bpf variable. params->mode is set in timerlat_{top,hist}_parse_args to TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output and/or auto-analysis is requested. timerlat_{top,hist}_main then checks if BPF is not unavailable or disabled, in that case, it sets params->mode to TRACING_MODE_TRACEFS. A condition is added to timerlat_apply_config that skips setting timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those are unnecessary, since they only turn off tracing, which is already turned off in that case, since BPF is used to collect samples). Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/timerlat.c | 24 ++++++----- tools/tracing/rtla/src/timerlat.h | 18 +++++++++ tools/tracing/rtla/src/timerlat_hist.c | 49 +++++++++++++---------- tools/tracing/rtla/src/timerlat_top.c | 55 +++++++++++++++----------- 4 files changed, 94 insertions(+), 52 deletions(-) diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/tim= erlat.c index c29e2ba2d7d8..63d6d43eafff 100644 --- a/tools/tracing/rtla/src/timerlat.c +++ b/tools/tracing/rtla/src/timerlat.c @@ -40,16 +40,22 @@ timerlat_apply_config(struct osnoise_tool *tool, struct= timerlat_params *params) CPU_SET(i, ¶ms->monitored_cpus); } =20 - retval =3D osnoise_set_stop_us(tool->context, params->stop_us); - if (retval) { - err_msg("Failed to set stop us\n"); - goto out_err; - } + if (params->mode !=3D TRACING_MODE_BPF) { + /* + * In tracefs and mixed mode, timerlat tracer handles stopping + * on threshold + */ + retval =3D osnoise_set_stop_us(tool->context, params->stop_us); + if (retval) { + err_msg("Failed to set stop us\n"); + goto out_err; + } =20 - retval =3D osnoise_set_stop_total_us(tool->context, params->stop_total_us= ); - if (retval) { - err_msg("Failed to set stop total us\n"); - goto out_err; + retval =3D osnoise_set_stop_total_us(tool->context, params->stop_total_u= s); + if (retval) { + err_msg("Failed to set stop total us\n"); + goto out_err; + } } =20 =20 diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/tim= erlat.h index 73045aef23fa..e0a553545d03 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -1,6 +1,23 @@ // SPDX-License-Identifier: GPL-2.0 #include "osnoise.h" =20 +/* + * Define timerlat tracing mode. + * + * There are three tracing modes: + * - tracefs-only, used when BPF is unavailable. + * - BPF-only, used when BPF is available and neither trace saving nor + * auto-analysis are enabled. + * - mixed mode, used when BPF is available and either trace saving or + * auto-analysis is enabled (which rely on sample collection through + * tracefs). + */ +enum timerlat_tracing_mode { + TRACING_MODE_BPF, + TRACING_MODE_TRACEFS, + TRACING_MODE_MIXED, +}; + struct timerlat_params { /* Common params */ char *cpus; @@ -30,6 +47,7 @@ struct timerlat_params { cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; + enum timerlat_tracing_mode mode; union { struct { /* top only */ diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 9d9efeedc4c2..70548beed812 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -802,6 +802,9 @@ static struct timerlat_params params->bucket_size =3D 1; params->entries =3D 256; =20 + /* default to BPF mode */ + params->mode =3D TRACING_MODE_BPF; + while (1) { static struct option long_options[] =3D { {"auto", required_argument, 0, 'a'}, @@ -1054,6 +1057,13 @@ static struct timerlat_params if (params->kernel_workload && params->user_workload) timerlat_hist_usage("--kernel-threads and --user-threads are mutually ex= clusive!"); =20 + /* + * If auto-analysis or trace output is enabled, switch from BPF mode to + * mixed mode + */ + if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + params->mode =3D TRACING_MODE_MIXED; + return params; } =20 @@ -1149,7 +1159,6 @@ int timerlat_hist_main(int argc, char *argv[]) pthread_t timerlat_u; int retval; int nr_cpus, i; - bool no_bpf =3D false; =20 params =3D timerlat_hist_parse_args(argc, argv); if (!params) @@ -1161,12 +1170,6 @@ int timerlat_hist_main(int argc, char *argv[]) goto out_exit; } =20 - retval =3D timerlat_hist_apply_config(tool, params); - if (retval) { - err_msg("Could not apply config\n"); - goto out_free; - } - trace =3D &tool->trace; /* * Save trace instance into global variable so that SIGINT can stop @@ -1175,24 +1178,30 @@ int timerlat_hist_main(int argc, char *argv[]) */ hist_inst =3D trace; =20 + /* + * Try to enable BPF, unless disabled explicitly. + * If BPF enablement fails, fall back to tracefs mode. + */ if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) =3D= =3D 0) { debug_msg("RTLA_NO_BPF set, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_s= ample")) { + params->mode =3D TRACING_MODE_TRACEFS; + } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sampl= e")) { debug_msg("osnoise:timerlat_sample missing, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf) { + params->mode =3D TRACING_MODE_TRACEFS; + } else { retval =3D timerlat_bpf_init(params); if (retval) { debug_msg("Could not enable BPF\n"); - no_bpf =3D true; + params->mode =3D TRACING_MODE_TRACEFS; } } =20 + retval =3D timerlat_hist_apply_config(tool, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_free; + } + retval =3D enable_timerlat(trace); if (retval) { err_msg("Failed to enable timerlat tracer\n"); @@ -1320,7 +1329,7 @@ int timerlat_hist_main(int argc, char *argv[]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); - if (no_bpf) { + if (params->mode =3D=3D TRACING_MODE_TRACEFS) { trace_instance_start(trace); } else { retval =3D timerlat_bpf_attach(); @@ -1333,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[]) tool->start_time =3D time(NULL); timerlat_hist_set_signals(params); =20 - if (no_bpf) { + if (params->mode =3D=3D TRACING_MODE_TRACEFS) { while (!stop_tracing) { sleep(params->sleep_time); =20 @@ -1362,7 +1371,7 @@ int timerlat_hist_main(int argc, char *argv[]) } else timerlat_bpf_wait(-1); =20 - if (!no_bpf) { + if (params->mode !=3D TRACING_MODE_TRACEFS) { timerlat_bpf_detach(); retval =3D timerlat_hist_bpf_pull_data(tool); if (retval) { @@ -1410,7 +1419,7 @@ int timerlat_hist_main(int argc, char *argv[]) osnoise_destroy_tool(tool); free(params); free_cpu_idle_disable_states(); - if (!no_bpf) + if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_destroy(); out_exit: exit(return_value); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 79cb6f28967f..30f1d6cbcad1 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -559,6 +559,9 @@ static struct timerlat_params /* display data in microseconds */ params->output_divisor =3D 1000; =20 + /* default to BPF mode */ + params->mode =3D TRACING_MODE_BPF; + while (1) { static struct option long_options[] =3D { {"auto", required_argument, 0, 'a'}, @@ -790,6 +793,13 @@ static struct timerlat_params if (params->kernel_workload && params->user_workload) timerlat_top_usage("--kernel-threads and --user-threads are mutually exc= lusive!"); =20 + /* + * If auto-analysis or trace output is enabled, switch from BPF mode to + * mixed mode + */ + if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + params->mode =3D TRACING_MODE_MIXED; + return params; } =20 @@ -994,7 +1004,6 @@ int timerlat_top_main(int argc, char *argv[]) char *max_lat; int retval; int nr_cpus, i; - bool no_bpf =3D false; =20 params =3D timerlat_top_parse_args(argc, argv); if (!params) @@ -1006,38 +1015,38 @@ int timerlat_top_main(int argc, char *argv[]) goto out_exit; } =20 - retval =3D timerlat_top_apply_config(top, params); - if (retval) { - err_msg("Could not apply config\n"); - goto out_free; - } - trace =3D &top->trace; /* - * Save trace instance into global variable so that SIGINT can stop - * the timerlat tracer. - * Otherwise, rtla could loop indefinitely when overloaded. - */ + * Save trace instance into global variable so that SIGINT can stop + * the timerlat tracer. + * Otherwise, rtla could loop indefinitely when overloaded. + */ top_inst =3D trace; =20 + /* + * Try to enable BPF, unless disabled explicitly. + * If BPF enablement fails, fall back to tracefs mode. + */ if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) =3D= =3D 0) { debug_msg("RTLA_NO_BPF set, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_s= ample")) { + params->mode =3D TRACING_MODE_TRACEFS; + } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sampl= e")) { debug_msg("osnoise:timerlat_sample missing, disabling BPF\n"); - no_bpf =3D true; - } - - if (!no_bpf) { + params->mode =3D TRACING_MODE_TRACEFS; + } else { retval =3D timerlat_bpf_init(params); if (retval) { debug_msg("Could not enable BPF\n"); - no_bpf =3D true; + params->mode =3D TRACING_MODE_TRACEFS; } } =20 + retval =3D timerlat_top_apply_config(top, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_free; + } + retval =3D enable_timerlat(trace); if (retval) { err_msg("Failed to enable timerlat tracer\n"); @@ -1166,7 +1175,7 @@ int timerlat_top_main(int argc, char *argv[]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); - if (no_bpf) { + if (params->mode =3D=3D TRACING_MODE_TRACEFS) { trace_instance_start(trace); } else { retval =3D timerlat_bpf_attach(); @@ -1179,7 +1188,7 @@ int timerlat_top_main(int argc, char *argv[]) top->start_time =3D time(NULL); timerlat_top_set_signals(params); =20 - if (no_bpf) + if (params->mode =3D=3D TRACING_MODE_TRACEFS) retval =3D timerlat_top_main_loop(top, record, params, ¶ms_u); else retval =3D timerlat_top_bpf_main_loop(top, record, params, ¶ms_u); @@ -1187,7 +1196,7 @@ int timerlat_top_main(int argc, char *argv[]) if (retval) goto out_top; =20 - if (!no_bpf) + if (params->mode !=3D TRACING_MODE_TRACEFS) timerlat_bpf_detach(); =20 if (params->user_workload && !params_u.stopped_running) { --=20 2.49.0 From nobody Tue Feb 10 23:12:03 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65F99289360 for ; Wed, 28 May 2025 14:32:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748442769; cv=none; b=kBntyCev25M/EHA2SkXIMNUeDIf018pPsoo09zHwbBc69fdwLbC5IPVUA8fKOTX7m9IH/Wjc4GpMssAcGLHse3h08WkbWqiMhSOvtnsGa5mG47pxI9evrxFTZ0SmmH6Wpgbrfu8+OrUZG+ebF7QfLRxxgxug7Z+GKPab1Fmvc3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748442769; c=relaxed/simple; bh=ISyB477vsWeF42Yz0SalyeZ2i4D75hkmWsj/iYuj4CI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BU1HnquGhesV5H9WzkdwH6eSIfRzA503lGxUTzsyvAz5BUFL1PyZYvgpfvmVtu/GHnLsQfgk8ImtrzyMIpRH5nQ0v20hiTbRNHDUzesgnqWn2EEYp1fSqnJ2OVO7cfL6UYlC+T21vwGeMlhuLr6hOghHagIg19+7VbU7nrEAA3o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=OO6CleXO; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OO6CleXO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1748442766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pvzNszbzHtfrqe4Ama9R3Qoz9uboq2bWIkQhK85os6U=; b=OO6CleXOBn4KY4LfiSjJI6vAFvbe66Ve9GDaA4of16xGzUHMAnSgY6V6OzWJ2+8A+WHpes dbhZDyINS3Dpb2JYxTcz5+itB9N9p8dIOemI31H3tWUIzyVQGQZZQ4iH3eUA53TAACSh2/ 7lvgSINv5/n/jcbkxhFpOt2+5Ms13Yg= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-623-93D2QbdFNZ2vz1SoMBxQgA-1; Wed, 28 May 2025 10:32:43 -0400 X-MC-Unique: 93D2QbdFNZ2vz1SoMBxQgA-1 X-Mimecast-MFC-AGG-ID: 93D2QbdFNZ2vz1SoMBxQgA_1748442762 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9450B1801A08; Wed, 28 May 2025 14:32:42 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.45.225.94]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 33E3819560AA; Wed, 28 May 2025 14:32:39 +0000 (UTC) From: Tomas Glozar To: Steven Rostedt Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org, John Kacur , Luis Goncalves , Tomas Glozar Subject: [RFC PATCH 2/2] rtla/timerlat: Add action on threshold Date: Wed, 28 May 2025 16:28:58 +0200 Message-ID: <20250528142858.185017-3-tglozar@redhat.com> In-Reply-To: <20250528142858.185017-1-tglozar@redhat.com> References: <20250528142858.185017-1-tglozar@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 Content-Type: text/plain; charset="utf-8" Extend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support implementing arbitrary actions. A new option, -A/--action-on-overflow, is added. Supported actions are: - -A trace: Saves tracefs buffer. Same as -t. - -A signal,num=3D,pid=3D: Sends signal to process. "parent" might be specified instead of number to send signal to parent process. - -A exec,command=3D: Execute shell command. The action is executed when tracing is stopped, at the same place as saving the trace output. Multiple actions might be specified and will be executed in order; however, only one action of each type is allowed. If the action fails, the following actions are not executed. For example, this command: $ rtla timerlat -T 20 -A trace -A command,"grep ipi_send timerlat_output.tx= t" -A signal,num=3D2,pid=3Dparent will send signal 2 (SIGINT) to parent process, but only if saved trace contains the text "ipi_send". This way, the feature can be used for flexible reactions on latency spikes, and allows combining rtla with other tooling like perf. Signed-off-by: Tomas Glozar --- tools/tracing/rtla/src/Build | 1 + tools/tracing/rtla/src/actions.c | 155 +++++++++++++++++++++++++ tools/tracing/rtla/src/actions.h | 33 ++++++ tools/tracing/rtla/src/timerlat.c | 42 +++++++ tools/tracing/rtla/src/timerlat.h | 4 +- tools/tracing/rtla/src/timerlat_hist.c | 36 ++++-- tools/tracing/rtla/src/timerlat_top.c | 36 ++++-- 7 files changed, 283 insertions(+), 24 deletions(-) create mode 100644 tools/tracing/rtla/src/actions.c create mode 100644 tools/tracing/rtla/src/actions.h diff --git a/tools/tracing/rtla/src/Build b/tools/tracing/rtla/src/Build index 7bb7e39e391a..66631280b75b 100644 --- a/tools/tracing/rtla/src/Build +++ b/tools/tracing/rtla/src/Build @@ -1,5 +1,6 @@ rtla-y +=3D trace.o rtla-y +=3D utils.o +rtla-y +=3D actions.o rtla-y +=3D osnoise.o rtla-y +=3D osnoise_top.o rtla-y +=3D osnoise_hist.o diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/acti= ons.c new file mode 100644 index 000000000000..de3a4a031699 --- /dev/null +++ b/tools/tracing/rtla/src/actions.c @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +#include "actions.h" + +/* + * action_options_init - initialize struct action_options + */ +void +action_options_init(struct action_options *opts) +{ + opts->actions_length =3D 0; + for (int i =3D 0; i < ACTION_FIELD_N; i++) { + opts->actions[i] =3D ACTION_NONE; + opts->action_present[i] =3D false; + } + opts->trace_output =3D NULL; +} + +/* + * action_options_destroy - destroy struct action_options + */ +void +action_options_destroy(struct action_options *opts) +{ + if (opts->command) + free(opts->command); +} + +/* + * action_add_trace_output - add an action to output trace + */ +int +action_add_trace_output(struct action_options *opts, char *trace_output) +{ + if (opts->action_present[ACTION_TRACE_OUTPUT]) + return 1; + opts->action_present[ACTION_TRACE_OUTPUT] =3D true; + + opts->actions[opts->actions_length++] =3D ACTION_TRACE_OUTPUT; + opts->trace_output =3D trace_output; + + return 0; +} + +/* + * action_add_trace_output - add an action to send signal to a process + */ +int +action_add_signal(struct action_options *opts, int signal, int pid) +{ + if (opts->action_present[ACTION_SIGNAL]) + return 1; + opts->action_present[ACTION_SIGNAL] =3D true; + + opts->actions[opts->actions_length++] =3D ACTION_SIGNAL; + opts->signal =3D signal; + opts->pid =3D pid; + + return 0; +} + +/* + * action_add_exec - add an action to execute a shell command + */ +int +action_add_exec(struct action_options *opts, char *command) +{ + if (opts->action_present[ACTION_EXEC]) + return 1; + opts->action_present[ACTION_EXEC] =3D true; + + opts->actions[opts->actions_length++] =3D ACTION_EXEC; + if (opts->command) + free(opts->command); + opts->command =3D calloc(sizeof(char), strlen(command) + 1); + if (!opts->command) + return -1; + strcpy(opts->command, command); + + return 0; +} + +/* + * action_parse - add an action based on text specification + */ +int +action_parse(struct action_options *opts, char *trigger) +{ + enum action_type type =3D ACTION_NONE; + char *token; + char trigger_c[strlen(trigger)]; + + /* For ACTION_SIGNAL */ + int signal =3D 0, pid =3D 0; + + if (opts->actions_length =3D=3D ACTION_FIELD_N) + return -1; + + strcpy(trigger_c, trigger); + token =3D strtok(trigger_c, ","); + + if (strcmp(token, "trace") =3D=3D 0) + type =3D ACTION_TRACE_OUTPUT; + else if (strcmp(token, "signal") =3D=3D 0) + type =3D ACTION_SIGNAL; + else if (strcmp(token, "exec") =3D=3D 0) + type =3D ACTION_EXEC; + else + /* Invalid trigger type */ + return -1; + + token =3D strtok(NULL, ","); + + switch (type) { + case ACTION_TRACE_OUTPUT: + /* Takes no argument */ + if (token !=3D NULL) + return -1; + return action_add_trace_output(opts, "timerlat_trace.txt"); + case ACTION_SIGNAL: + /* Takes two arguments, num (signal) and pid */ + while (token !=3D NULL) { + if (strlen(token) > 4 && strncmp(token, "num=3D", 4) =3D=3D 0) { + signal =3D atoi(token + 4); + } else if (strlen(token) > 4 && strncmp(token, "pid=3D", 4) =3D=3D 0) { + if (strncmp(token + 4, "parent", 7) =3D=3D 0) + pid =3D -1; + else + pid =3D atoi(token + 4); + } else { + /* Invalid argument */ + return -1; + } + + token =3D strtok(NULL, ","); + } + + if (!signal || !pid) + /* Missing argument */ + return -1; + + return action_add_signal(opts, signal, pid); + case ACTION_EXEC: + if (token =3D=3D NULL) + return -1; + if (strlen(token) > 8 && strncmp(token, "command=3D", 8) =3D=3D 0) { + return action_add_exec(opts, token + 8); + } + return -1; + default: + return -1; + } +} diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/acti= ons.h new file mode 100644 index 000000000000..70caeca9b85b --- /dev/null +++ b/tools/tracing/rtla/src/actions.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include + +enum action_type { + ACTION_NONE =3D 0, + ACTION_TRACE_OUTPUT, + ACTION_SIGNAL, + ACTION_EXEC, + ACTION_FIELD_N +}; + +struct action_options { + enum action_type actions[ACTION_FIELD_N]; + int actions_length; + bool action_present[ACTION_FIELD_N]; + + /* For ACTION_TRACE_OUTPUT */ + char *trace_output; + + /* For ACTION_SIGNAL */ + int signal; + int pid; + + /* For ACTION_COMMAND */ + char *command; +}; + +void action_options_init(struct action_options *opts); +void action_options_destroy(struct action_options *opts); +int action_add_trace_output(struct action_options *opts, char *trace_outpu= t); +int action_add_signal(struct action_options *opts, int signal, int pid); +int action_add_exec(struct action_options *opts, char *command); +int action_parse(struct action_options *opts, char *trigger); diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/tim= erlat.c index 63d6d43eafff..98d8c4950f03 100644 --- a/tools/tracing/rtla/src/timerlat.c +++ b/tools/tracing/rtla/src/timerlat.c @@ -13,6 +13,7 @@ #include #include #include +#include =20 #include "timerlat.h" =20 @@ -126,6 +127,47 @@ timerlat_apply_config(struct osnoise_tool *tool, struc= t timerlat_params *params) return -1; } =20 +/* + * timerlat_execute_actions - execute actions at threshold overflow + */ +int +timerlat_execute_actions(struct osnoise_tool *record, struct action_option= s *action_opts) +{ + int i, pid, retval; + + for (i =3D 0; i < action_opts->actions_length; i++) { + switch (action_opts->actions[i]) { + case ACTION_TRACE_OUTPUT: + retval =3D save_trace_to_file(record->trace.inst, action_opts->trace_ou= tput); + if (retval) { + err_msg("Error saving trace\n"); + return retval; + } + break; + case ACTION_SIGNAL: + if (action_opts->pid =3D=3D -1) + pid =3D getppid(); + else + pid =3D action_opts->pid; + retval =3D kill(pid, action_opts->signal); + if (retval) { + err_msg("Error sending signal\n"); + return retval; + } + break; + case ACTION_EXEC: + retval =3D system(action_opts->command); + if (retval) + return retval; + break; + default: + break; + } + } + + return 0; +} + static void timerlat_usage(int err) { int i; diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/tim= erlat.h index e0a553545d03..35d274fc2875 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include "actions.h" #include "osnoise.h" =20 /* @@ -22,7 +23,6 @@ struct timerlat_params { /* Common params */ char *cpus; cpu_set_t monitored_cpus; - char *trace_output; char *cgroup_name; unsigned long long runtime; long long stop_us; @@ -48,6 +48,7 @@ struct timerlat_params { struct sched_attr sched_param; struct trace_events *events; enum timerlat_tracing_mode mode; + struct action_options action_opts; union { struct { /* top only */ @@ -69,6 +70,7 @@ struct timerlat_params { }; }; =20 +int timerlat_execute_actions(struct osnoise_tool *tool, struct action_opti= ons *action_opts); int timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_param= s *params); =20 int timerlat_hist_main(int argc, char *argv[]); diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/sr= c/timerlat_hist.c index 70548beed812..072d445782ad 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -753,6 +753,7 @@ static void timerlat_hist_usage(char *usage) " in nanoseconds", " -u/--user-threads: use rtla user-space threads instead of kernel-spa= ce timerlat threads", " -k/--kernel-threads: use timerlat kernel-space threads instead of rt= la user-space threads", + " -A/--action-on-overflow : trigger action on threshold = overflow, multiple -A are allowed", " -U/--user-load: enable timerlat for user-defined user-space workload= ", " --warm-up s: let the workload run for s seconds before collecting= data", " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", @@ -786,11 +787,14 @@ static struct timerlat_params int auto_thresh; int retval; int c; + char *trace_output; =20 params =3D calloc(1, sizeof(*params)); if (!params) exit(1); =20 + action_options_init(¶ms->action_opts); + /* disabled by default */ params->dma_latency =3D -1; =20 @@ -807,6 +811,7 @@ static struct timerlat_params =20 while (1) { static struct option long_options[] =3D { + {"action-on-overflow", required_argument, 0, 'A'}, {"auto", required_argument, 0, 'a'}, {"cpus", required_argument, 0, 'c'}, {"cgroup", optional_argument, 0, 'C'}, @@ -847,7 +852,7 @@ static struct timerlat_params /* getopt_long stores the option index here. */ int option_index =3D 0; =20 - c =3D getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:knp:P:s:t::T:uU01234= 56:7:8:9\1\2:\3:", + c =3D getopt_long(argc, argv, "A:a:c:C::b:d:e:E:DhH:i:knp:P:s:t::T:uU012= 3456:7:8:9\1\2:\3:", long_options, &option_index); =20 /* detect the end of the options. */ @@ -866,8 +871,15 @@ static struct timerlat_params params->print_stack =3D auto_thresh; =20 /* set trace */ - params->trace_output =3D "timerlat_trace.txt"; + action_add_trace_output(¶ms->action_opts, "timerlat_trace.txt"); =20 + break; + case 'A': + retval =3D action_parse(¶ms->action_opts, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } break; case 'c': retval =3D parse_cpu_set(optarg, ¶ms->monitored_cpus); @@ -956,13 +968,14 @@ static struct timerlat_params case 't': if (optarg) { if (optarg[0] =3D=3D '=3D') - params->trace_output =3D &optarg[1]; + trace_output =3D &optarg[1]; else - params->trace_output =3D &optarg[0]; + trace_output =3D &optarg[0]; } else if (optind < argc && argv[optind][0] !=3D '-') - params->trace_output =3D argv[optind]; + trace_output =3D argv[optind]; else - params->trace_output =3D "timerlat_trace.txt"; + trace_output =3D "timerlat_trace.txt"; + action_add_trace_output(¶ms->action_opts, trace_output); break; case 'u': params->user_workload =3D 1; @@ -1061,7 +1074,8 @@ static struct timerlat_params * If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ - if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + if (params->mode =3D=3D TRACING_MODE_BPF && + (params->action_opts.action_present[ACTION_TRACE_OUTPUT] || !params->= no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1254,7 +1268,7 @@ int timerlat_hist_main(int argc, char *argv[]) } } =20 - if (params->trace_output) { + if (params->action_opts.action_present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); @@ -1325,7 +1339,7 @@ int timerlat_hist_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->trace_output) + if (params->action_opts.action_present[ACTION_TRACE_OUTPUT]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1395,8 +1409,7 @@ int timerlat_hist_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - save_trace_to_file(record ? record->trace.inst : NULL, - params->trace_output); + timerlat_execute_actions(record, ¶ms->action_opts); } =20 out_hist: @@ -1417,6 +1430,7 @@ int timerlat_hist_main(int argc, char *argv[]) osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(tool); + action_options_destroy(¶ms->action_opts); free(params); free_cpu_idle_disable_states(); if (params->mode !=3D TRACING_MODE_TRACEFS) diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src= /timerlat_top.c index 30f1d6cbcad1..ab89184387b8 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -513,6 +513,7 @@ static void timerlat_top_usage(char *usage) " -u/--user-threads: use rtla user-space threads instead of kernel-spa= ce timerlat threads", " -k/--kernel-threads: use timerlat kernel-space threads instead of rt= la user-space threads", " -U/--user-load: enable timerlat for user-defined user-space workload= ", + " -A/--action-on-overflow : trigger action on threshold = overflow, multiple -A are allowed", " --warm-up s: let the workload run for s seconds before collecting= data", " --trace-buffer-size kB: set the per-cpu trace buffer size in kB", " --deepest-idle-state n: only go down to idle state n on cpus used= by timerlat to reduce exit from idle latency", @@ -545,11 +546,14 @@ static struct timerlat_params long long auto_thresh; int retval; int c; + char *trace_output; =20 params =3D calloc(1, sizeof(*params)); if (!params) exit(1); =20 + action_options_init(¶ms->action_opts); + /* disabled by default */ params->dma_latency =3D -1; =20 @@ -564,6 +568,7 @@ static struct timerlat_params =20 while (1) { static struct option long_options[] =3D { + {"action-on-overflow", required_argument, 0, 'A'}, {"auto", required_argument, 0, 'a'}, {"cpus", required_argument, 0, 'c'}, {"cgroup", optional_argument, 0, 'C'}, @@ -598,7 +603,7 @@ static struct timerlat_params /* getopt_long stores the option index here. */ int option_index =3D 0; =20 - c =3D getopt_long(argc, argv, "a:c:C::d:De:hH:i:knp:P:qs:t::T:uU0:1:2:34= 5:6:7:", + c =3D getopt_long(argc, argv, "A:a:c:C::d:De:hH:i:knp:P:qs:t::T:uU0:1:2:= 345:6:7:", long_options, &option_index); =20 /* detect the end of the options. */ @@ -617,7 +622,14 @@ static struct timerlat_params params->print_stack =3D auto_thresh; =20 /* set trace */ - params->trace_output =3D "timerlat_trace.txt"; + action_add_trace_output(¶ms->action_opts, "timerlat_trace.txt"); + break; + case 'A': + retval =3D action_parse(¶ms->action_opts, optarg); + if (retval) { + err_msg("Invalid action %s\n", optarg); + exit(EXIT_FAILURE); + } break; case '5': /* it is here because it is similar to -a */ @@ -712,14 +724,14 @@ static struct timerlat_params case 't': if (optarg) { if (optarg[0] =3D=3D '=3D') - params->trace_output =3D &optarg[1]; + trace_output =3D &optarg[1]; else - params->trace_output =3D &optarg[0]; + trace_output =3D &optarg[0]; } else if (optind < argc && argv[optind][0] !=3D '-') - params->trace_output =3D argv[optind]; + trace_output =3D argv[optind]; else - params->trace_output =3D "timerlat_trace.txt"; - + trace_output =3D "timerlat_trace.txt"; + action_add_trace_output(¶ms->action_opts, trace_output); break; case 'u': params->user_workload =3D true; @@ -797,7 +809,8 @@ static struct timerlat_params * If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ - if (params->mode =3D=3D TRACING_MODE_BPF && params->trace_output && !para= ms->no_aa) + if (params->mode =3D=3D TRACING_MODE_BPF && + (params->action_opts.action_present[ACTION_TRACE_OUTPUT] || !params->= no_aa)) params->mode =3D TRACING_MODE_MIXED; =20 return params; @@ -1099,7 +1112,7 @@ int timerlat_top_main(int argc, char *argv[]) } } =20 - if (params->trace_output) { + if (params->action_opts.action_present[ACTION_TRACE_OUTPUT]) { record =3D osnoise_init_trace_tool("timerlat"); if (!record) { err_msg("Failed to enable the trace instance\n"); @@ -1171,7 +1184,7 @@ int timerlat_top_main(int argc, char *argv[]) * tracing while enabling other instances. The trace instance is the * one with most valuable information. */ - if (params->trace_output) + if (params->action_opts.action_present[ACTION_TRACE_OUTPUT]) trace_instance_start(&record->trace); if (!params->no_aa) trace_instance_start(&aa->trace); @@ -1214,8 +1227,7 @@ int timerlat_top_main(int argc, char *argv[]) if (!params->no_aa) timerlat_auto_analysis(params->stop_us, params->stop_total_us); =20 - save_trace_to_file(record ? record->trace.inst : NULL, - params->trace_output); + timerlat_execute_actions(record, ¶ms->action_opts); } else if (params->aa_only) { /* * If the trace did not stop with --aa-only, at least print the --=20 2.49.0