From nobody Mon Dec 1 23:33:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2DD91D61A3; Tue, 25 Nov 2025 21:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764106787; cv=none; b=BEAmweuv0+7Sl4SDFjyq5zmmDv6V69zl/+T2RHaYy0xUJry0pCSRcuRNbheI96ZcXlZkCoVzm5I9+Il3JnEVWL71aQa+ZnTXZ+6OMY34T9Ie0176EcPqdz4aQGilfEKgPzeDkQa5QLD5AiDAAKzKNpgAmZ3onMhOZfcegwgs8fE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764106787; c=relaxed/simple; bh=LBC2NLYIzg53Cvi5iggo757bpBm9xkAtIbRZJPAhd8g=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=O5WSKXNnAp5dfY+vg3tfbObL/mwrYR3kDAQJ31c2t7e+MWg0skh+irCk9edJyyrw+vSbt7H2UGDalkd0y4ShBIYeN5/KIUiMcxUvAluZG4UgetiUDmO4erVAA82Pf5n2RZx04YZVXqPrM6mSXs3CvacayD9yP7A1Hk31e1sAzak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fG6zBRAO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fG6zBRAO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B7FFC116D0; Tue, 25 Nov 2025 21:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764106787; bh=LBC2NLYIzg53Cvi5iggo757bpBm9xkAtIbRZJPAhd8g=; h=Date:From:To:Cc:Subject:References:From; b=fG6zBRAOsVg5Xht9c5/jVFLG83KQMBlhi5brKTan4TI85NC81+rP+AjM8c5JztinC Ua5vJSydhe6+9QNBYHSeG24TJl1CBv3xX8Ai4gaIZUFHVBO7gzEguMIvrAGQIcagGB 4JW1wSg7tNOuYyWSmwIDCSwH5KnVXlzgYVIhZ3Js93BYc0RVyM+NrEIVEZtIg1MhCZ CqmJh2ZDnP5WxxoqGLi9Rmn2AikdTcFMwchSgCp59CLKEHU2uKnLdXrFXpmisfmAjs Et+P8EHA9kvQn0Dhdt2/Joe/qbN9S0YZPtMvwfHnw5cN8v4ISYtvANCK5Oq6Ra97Xd yWtFgsNZEmZ+w== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1vO0m0-00000005rNv-0WSN; Tue, 25 Nov 2025 16:40:32 -0500 Message-ID: <20251125214031.975879283@kernel.org> User-Agent: quilt/0.68 Date: Tue, 25 Nov 2025 16:40:05 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Tom Zanussi Subject: [PATCH v2 1/3] tracing: Remove unneeded event_mutex lock in event_trigger_regex_release() References: <20251125214004.396482401@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt In event_trigger_regex_release(), the only code is: mutex_lock(&event_mutex); if (file->f_mode & FMODE_READ) seq_release(inode, file); mutex_unlock(&event_mutex); return 0; There's nothing special about the file->f_mode or the seq_release() that requires any locking. Remove the unnecessary locks. Acked-by: Masami Hiramatsu (Google) Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_events_trigger.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_event= s_trigger.c index 7795af600466..e5dcfcbb2cd5 100644 --- a/kernel/trace/trace_events_trigger.c +++ b/kernel/trace/trace_events_trigger.c @@ -314,13 +314,9 @@ static ssize_t event_trigger_regex_write(struct file *= file, =20 static int event_trigger_regex_release(struct inode *inode, struct file *f= ile) { - mutex_lock(&event_mutex); - if (file->f_mode & FMODE_READ) seq_release(inode, file); =20 - mutex_unlock(&event_mutex); - return 0; } =20 --=20 2.51.0 From nobody Mon Dec 1 23:33:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F10F02DF6E3; Tue, 25 Nov 2025 21:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764106788; cv=none; b=Irjno0ppioXpQfWWWjFghDzdlEYV5fDJD9mC95mui263sUliz/fEoPzYBj8UQ5XHE/2ue0Bl8xwWqKwMIXb3r9F1ll09z2tHP8yM2Bd5E7cVrsj1URbBO6SsO3YgyFmr+2BGRh4s6GKlZTIE+Q4dDnZcinrdhGTFEVRZ1QngXQE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764106788; c=relaxed/simple; bh=+SI/WIpvJonjM4T+po4ooc5wwIDV9ayQnk7qM0Lhgug=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=kNEi7nxtvAqYWxwTdhi6ktVWNqYCn4XyIOHMGqHpxcmrO4As6uhnL94mpS0xuJHB5F6LIvilJSkfS8s6aS3aAfMp2wRQ3znNsecarzQ97M1pzUFj+6ESzvplWJmQ4t7lQkWpe/UVOSqKo1PiftVx3TPvEoimoRRK1fuGLTRqaAs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VshoEn7t; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VshoEn7t" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A27ABC19423; Tue, 25 Nov 2025 21:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764106787; bh=+SI/WIpvJonjM4T+po4ooc5wwIDV9ayQnk7qM0Lhgug=; h=Date:From:To:Cc:Subject:References:From; b=VshoEn7tEbORXhs2WyRX2y2lIFw3FJvxy4G/hIF+oeFMgql2Z3SkBAC1wFH30sOgK v6AmRrbUQhoE0Ko/SMQTvzosRZ1xU/Qdsgjs03PRDXSOHXqiOM5mx64RN78UR27ZMJ ecHnicNOk8fewoGPZi/I+tqRJsLTP+1vpfDQvCGMzAfhYbKBu1E3Ohbias3GdDK/vD rf0x49RVepQ7BOiFLYumuZdYCSmuylZVHUtZBKWMuLbqr4D9HxUH2w0cKqZkXwVVnI 4gXDgpJKh9WtUXSMHW37cdp5qys2H3UpuONLL4yFjyHBXkiXxgXPHystiWCu/quT+p fmIWOTd9z+g5w== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1vO0m0-00000005rOP-1H0F; Tue, 25 Nov 2025 16:40:32 -0500 Message-ID: <20251125214032.151674992@kernel.org> User-Agent: quilt/0.68 Date: Tue, 25 Nov 2025 16:40:06 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Tom Zanussi Subject: [PATCH v2 2/3] tracing: Add bulk garbage collection of freeing event_trigger_data References: <20251125214004.396482401@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt The event trigger data requires a full tracepoint_synchronize_unregister() call before freeing. That call can take 100s of milliseconds to complete. In order to allow for bulk freeing of the trigger data, it can not call the tracepoint_synchronize_unregister() for every individual trigger data being free. Create a kthread that gets created the first time a trigger data is freed, and have it use the lockless llist to get the list of data to free, run the tracepoint_synchronize_unregister() then free everything in the list. By freeing hundreds of event_trigger_data elements together, it only requires two runs of the synchronization function, and not hundreds of runs. This speeds up the operation by orders of magnitude (milliseconds instead of several seconds). Acked-by: Masami Hiramatsu (Google) Signed-off-by: Steven Rostedt (Google) --- Changes since v1: https://patch.msgid.link/20251120205710.151041470@kernel.= org - Moved include of llist.h to trace.h as it is used there (Masami Hiramatsu) kernel/trace/trace.h | 2 ++ kernel/trace/trace_events_trigger.c | 55 +++++++++++++++++++++++++++-- 2 files changed, 54 insertions(+), 3 deletions(-) diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 5863800b1ab3..911fc75dc6c4 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -22,6 +22,7 @@ #include #include #include +#include =20 #include "pid_list.h" =20 @@ -1808,6 +1809,7 @@ struct event_trigger_data { char *name; struct list_head named_list; struct event_trigger_data *named_data; + struct llist_node llist; }; =20 /* Avoid typos */ diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_event= s_trigger.c index e5dcfcbb2cd5..3b97c242b795 100644 --- a/kernel/trace/trace_events_trigger.c +++ b/kernel/trace/trace_events_trigger.c @@ -6,6 +6,7 @@ */ =20 #include +#include #include #include #include @@ -17,15 +18,63 @@ static LIST_HEAD(trigger_commands); static DEFINE_MUTEX(trigger_cmd_mutex); =20 +static struct task_struct *trigger_kthread; +static struct llist_head trigger_data_free_list; +static DEFINE_MUTEX(trigger_data_kthread_mutex); + +/* Bulk garbage collection of event_trigger_data elements */ +static int trigger_kthread_fn(void *ignore) +{ + struct event_trigger_data *data, *tmp; + struct llist_node *llnodes; + + /* Once this task starts, it lives forever */ + for (;;) { + set_current_state(TASK_INTERRUPTIBLE); + if (llist_empty(&trigger_data_free_list)) + schedule(); + + __set_current_state(TASK_RUNNING); + + llnodes =3D llist_del_all(&trigger_data_free_list); + + /* make sure current triggers exit before free */ + tracepoint_synchronize_unregister(); + + llist_for_each_entry_safe(data, tmp, llnodes, llist) + kfree(data); + } + + return 0; +} + void trigger_data_free(struct event_trigger_data *data) { if (data->cmd_ops->set_filter) data->cmd_ops->set_filter(NULL, data, NULL); =20 - /* make sure current triggers exit before free */ - tracepoint_synchronize_unregister(); + if (unlikely(!trigger_kthread)) { + guard(mutex)(&trigger_data_kthread_mutex); + /* Check again after taking mutex */ + if (!trigger_kthread) { + struct task_struct *kthread; + + kthread =3D kthread_create(trigger_kthread_fn, NULL, + "trigger_data_free"); + if (!IS_ERR(kthread)) + WRITE_ONCE(trigger_kthread, kthread); + } + } + + if (!trigger_kthread) { + /* Do it the slow way */ + tracepoint_synchronize_unregister(); + kfree(data); + return; + } =20 - kfree(data); + llist_add(&data->llist, &trigger_data_free_list); + wake_up_process(trigger_kthread); } =20 static inline void data_ops_trigger(struct event_trigger_data *data, --=20 2.51.0 From nobody Mon Dec 1 23:33:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 563C1301481; Tue, 25 Nov 2025 21:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764106788; cv=none; b=D8ZMqekJbybrtSGBT9YcpSDNshvnyZ/ArcwEzBbJSm1Ran80yjGYptKAnmWK+0w7aOnKsAA7b7UXKbgGpQlcDN4yunCEh0NnZCHz8Ss/G/o2MJuLj1JDfeGlS6M4IuPxUh/ozL8p4Fs1Lu1Z2y0TKhuTccsaXZr+qwjIc12tUwU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764106788; c=relaxed/simple; bh=Xokl0hfcRa58KbExXeNKCxrDCiPFSBZOqgzvcMmCh6Q=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=OhgVL0RpD2HOxVaeRqrIDGYWS6uJbFSKmTstaHzxEbGqb9UWkKC1yaeK27v8Er7ZX2/GO28sOf/fUun+vSyuy1iLBgnA+klTe8MO04Aj/2BYh31ROFePBmiolFf6W1bFMDGAFZC6Se8dypoKFMeugJHMJubrNF3FPRErSXZhIHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VibIE6Rn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VibIE6Rn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9D7EC2BC86; Tue, 25 Nov 2025 21:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764106787; bh=Xokl0hfcRa58KbExXeNKCxrDCiPFSBZOqgzvcMmCh6Q=; h=Date:From:To:Cc:Subject:References:From; b=VibIE6RnYt8vuQwjSGR+jSbyFE21VPcNc5f+wWAteNauU2npfm3IEEqhS6bZx8/nh MHkqGLZce1dGx6aj1b6yOJVy1wsdtpOfSPzd2lI/ROeg04MSe2wP2zkMw64IeuHh+S 8Q5cQSzRPITZNm385rfpKGVaTWGDbODIgoJFrJk8+O36gUxFEyUW2USPIaoWiPVa5H OInM21WAEdKIBLzwF3w6+fTj/y3rRhC3Onzi47QNAkU2qyeQpaInEn7QXWCow1ymJi 07j6A/FCw3ucqf60+VcxwKXnIWrNGTEShI7soG1sRlvFYaZ7uznEGIo10kkbTB9l3I aNm/6OR9KxnlA== Received: from rostedt by gandalf with local (Exim 4.98.2) (envelope-from ) id 1vO0m0-00000005rOt-1xos; Tue, 25 Nov 2025 16:40:32 -0500 Message-ID: <20251125214032.323747707@kernel.org> User-Agent: quilt/0.68 Date: Tue, 25 Nov 2025 16:40:07 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Tom Zanussi Subject: [PATCH v2 3/3] tracing: Use strim() in trigger_process_regex() instead of skip_spaces() References: <20251125214004.396482401@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Steven Rostedt The function trigger_process_regex() is called by a few functions, where only one calls strim() on the buffer passed to it. That leaves the other functions not trimming the end of the buffer passed in and making it a little inconsistent. Remove the strim() from event_trigger_regex_write() and have trigger_process_regex() use strim() instead of skip_spaces(). The buff variable is not passed in as const, so it can be modified. Acked-by: Masami Hiramatsu (Google) Signed-off-by: Steven Rostedt (Google) --- kernel/trace/trace_events_trigger.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_event= s_trigger.c index 3b97c242b795..96aad82b1628 100644 --- a/kernel/trace/trace_events_trigger.c +++ b/kernel/trace/trace_events_trigger.c @@ -308,7 +308,8 @@ int trigger_process_regex(struct trace_event_file *file= , char *buff) char *command, *next; struct event_command *p; =20 - next =3D buff =3D skip_spaces(buff); + next =3D buff =3D strim(buff); + command =3D strsep(&next, ": \t"); if (next) { next =3D skip_spaces(next); @@ -345,8 +346,6 @@ static ssize_t event_trigger_regex_write(struct file *f= ile, if (IS_ERR(buf)) return PTR_ERR(buf); =20 - strim(buf); - guard(mutex)(&event_mutex); =20 event_file =3D event_file_file(file); --=20 2.51.0