From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62C4E288C16; Tue, 29 Jul 2025 10:28:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784909; cv=none; b=Zm3kZ0bIShMJ0PA5VOSPIdL5A1HHJX4hpDB6DamwFTkFkpSQCCp/LVu5tRFf/cmvWLOgqVoAk1o5j9kfa2W1XjAxvcUNR1athWvsRApdiciwckhomHS3ACb2chZztfNktYpbZJvbtVOQyAxHMWmBpf/aaPcdi9+7dqvGuPMf9Es= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784909; c=relaxed/simple; bh=Sm7jV+tRKsytGK5LOleOia6QLTmymRlF5GUTJcd927I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=otWeEcecTeeGcWtX9IxGoK+jzOu39HexKhUVrhH6fu99v0pryjokU2Kh/W1cQA2ePBT7cf0ytQ+ukpyoNO2WZeuHrTGMPauIYRuxXKjUSFd8aqDWg+1ojRGSCFuO1qKjas4tnvuwUOrK8j8X8XSPUuHfAJr7+6s1HSimAAldey0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Fa94HDat; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Fa94HDat" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2B207C4CEEF; Tue, 29 Jul 2025 10:28:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784909; bh=Sm7jV+tRKsytGK5LOleOia6QLTmymRlF5GUTJcd927I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Fa94HDatVkSI/VlhVI0V22z6g6CxlV/NoPqz+yc+zjKjJUkB+9QSzWB1X4RMqR+6s Kc0V7nU/Nx4KMQirEt6DddN50VwYea+QdtZTQcYRJY0zya2q7v5M6F41xdT5ci2E33 xaAbUYW8p/b6ucLEhs5YXr9CUn9woOz/6ta/GtW/Qz3MwKnAaBUeBxbeRNK5JM+pcT aRlY4mh7bvXmj3bAbFZwUxmVr8waN3MscHIRtxQJCfcGrV20eB+zjLCAk3GobxLkxF 5d01W12qAJdZsNr6RRUa6XvJ4TO3iN2Zw5+3TVtiLMiW4vELhmAlZeaSX2zNelYDLN nS4YoTPAtwL0Q== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 01/10] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Date: Tue, 29 Jul 2025 12:28:04 +0200 Message-ID: <20250729102813.1531457-2-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make alloc_and_copy_ftrace_hash to copy also direct address for each hash entry. Signed-off-by: Jiri Olsa --- kernel/trace/ftrace.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 4203fad56b6c..5b8f565a1258 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -1190,7 +1190,7 @@ static void __add_hash_entry(struct ftrace_hash *hash, } =20 static struct ftrace_func_entry * -add_hash_entry(struct ftrace_hash *hash, unsigned long ip) +add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned= long direct) { struct ftrace_func_entry *entry; =20 @@ -1199,11 +1199,18 @@ add_hash_entry(struct ftrace_hash *hash, unsigned l= ong ip) return NULL; =20 entry->ip =3D ip; + entry->direct =3D direct; __add_hash_entry(hash, entry); =20 return entry; } =20 +static struct ftrace_func_entry * +add_hash_entry(struct ftrace_hash *hash, unsigned long ip) +{ + return add_hash_entry_direct(hash, ip, 0); +} + static void free_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry) @@ -1376,7 +1383,7 @@ alloc_and_copy_ftrace_hash(int size_bits, struct ftra= ce_hash *hash) size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { - if (add_hash_entry(new_hash, entry->ip) =3D=3D NULL) + if (add_hash_entry_direct(new_hash, entry->ip, entry->direct) =3D=3D NU= LL) goto free_hash; } } --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 869E3254841; Tue, 29 Jul 2025 10:28:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784919; cv=none; b=SCVmdTGhZ7dX5Wr8uruRNkCbwwHM78rKJevcQlGGMbA4QyQ6tqOKx4K6M2li7JSTexfPd9kndpJUgaff4YiNBDRuZpDO7hn4l6x8y+o7BhhtUzJ3F6/LJobQHeyuMoUQ7t2rycIXa/5RiSFu1jEFyqA7J8Hj2p+Pj/x2wkKa2sY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784919; c=relaxed/simple; bh=PoVJYmha+F5Q+Shyh8Z6KZj91FS00Vyug/N39XeYgMM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=di9yNkO5Ja2NBSzHYqwzmrjjaNNKSeLDo/htf7gnIxzyBvbwaWAsl8kIlElSpcJde/mQQMdk7JNOXeNZCSgPPKxcgtMhPE5Ac6ingbGW2l3oaSOWcqgNgtIM6u5uBtfECGqvjGb69TYL2A7/6sLNFsFG/D0nKgpT+HQ5sJrGtwE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cvoNT2pk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cvoNT2pk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33CB7C4CEEF; Tue, 29 Jul 2025 10:28:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784919; bh=PoVJYmha+F5Q+Shyh8Z6KZj91FS00Vyug/N39XeYgMM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cvoNT2pkbNwrm9T3+/0Qy1iPrq8hPGDjfzb55rvDPU92Tp6ra268grQaNQUx/5YMG flGt4OoLgVr05TK/y0/MaDFjfeaCK9Nmax3CMWqKkBtEQsUomO5+nm4IinpJJAoxKo DiuxSJys3Fu7aNhh/s7CmnP1JP7ESYd+2zu8r4pwK+SC6W8TDAlos1TLXwn6yIJSw5 PLqtuv8SRR78GdCvlV+OXenW26soPrMNPSXYxcVCGLhK0yavus3sL3n35uoWAykTU2 3r2sseihRrqZPsOB0wjVMzNJ/ibme2TeBY6y8zW9RNF7faQPJ10WIm8FXw+7goiAEU yXy0ct+8CUksg== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 02/10] ftrace: Add register_ftrace_direct_hash function Date: Tue, 29 Jul 2025 12:28:05 +0200 Message-ID: <20250729102813.1531457-3-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding register_ftrace_direct_hash function that registers all entries (ip -> direct) provided in hash argument. The difference to current register_ftrace_direct is - hash argument that allows to register multiple ip -> direct entries at once - we can call register_ftrace_direct_hash multiple times on the same ftrace_ops object, becase after first registration with register_ftrace_function_nolock, it uses ftrace_update_ops to update the ftrace_ops object This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 7 +++ kernel/trace/ftrace.c | 123 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index b672ca15f265..e45bcc9de53b 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -526,6 +526,8 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, un= signed long addr, int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); =20 +int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash= *hash); + void ftrace_stub_direct_tramp(void); =20 #else @@ -552,6 +554,11 @@ static inline int modify_ftrace_direct_nolock(struct f= trace_ops *ops, unsigned l return -ENODEV; } =20 +int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash= *hash) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 5b8f565a1258..1dbb113f4e9c 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6219,6 +6219,129 @@ int modify_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) return err; } EXPORT_SYMBOL_GPL(modify_ftrace_direct); + +static unsigned long hash_count(struct ftrace_hash *hash) +{ + return hash ? hash->count : 0; +} + +/** + * hash_add - adds two struct ftrace_hash and returns the result + * @a: struct ftrace_hash object + * @b: struct ftrace_hash object + * + * Returns struct ftrace_hash object on success, NULL on error. + */ +static struct ftrace_hash *hash_add(struct ftrace_hash *a, struct ftrace_h= ash *b) +{ + struct ftrace_func_entry *entry; + struct ftrace_hash *add; + int size, i; + + size =3D hash_count(a) + hash_count(b); + if (size > 32) + size =3D 32; + + add =3D alloc_and_copy_ftrace_hash(fls(size), a); + if (!add) + goto error; + + size =3D 1 << b->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &b->buckets[i], hlist) { + if (add_hash_entry_direct(add, entry->ip, entry->direct) =3D=3D NULL) + goto error; + } + } + return add; + + error: + free_ftrace_hash(add); + return NULL; +} + +static void call_direct_funcs_hash(unsigned long ip, unsigned long pip, + struct ftrace_ops *ops, struct ftrace_regs *fregs) +{ + unsigned long addr; + + addr =3D ftrace_find_rec_direct(ip); + if (!addr) + return; + + arch_ftrace_set_direct_caller(fregs, addr); +} + +int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash= *hash) +{ + struct ftrace_hash *filter_hash =3D NULL, *new_hash =3D NULL, *free_hash = =3D NULL; + struct ftrace_func_entry *entry; + int i, size, err; + bool reg; + + if (!hash_count(hash)) + return 0; + + mutex_lock(&direct_mutex); + + /* Make sure requested entry is not already registered. */ + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + if (__ftrace_lookup_ip(direct_functions, entry->ip)) + goto out_unlock; + } + } + + filter_hash =3D ops->func_hash ? ops->func_hash->filter_hash : NULL; + + /* If there's nothing in filter_hash we need to register the ops. */ + reg =3D hash_count(filter_hash) =3D=3D 0; + if (reg) { + if (ops->func || ops->trampoline) + goto out_unlock; + if (ops->flags & FTRACE_OPS_FL_ENABLED) + goto out_unlock; + } + + filter_hash =3D hash_add(filter_hash, hash); + if (!filter_hash) + goto out_unlock; + + new_hash =3D hash_add(direct_functions, hash); + if (!new_hash) + goto out_unlock; + + free_hash =3D direct_functions; + rcu_assign_pointer(direct_functions, new_hash); + new_hash =3D NULL; + + if (reg) { + ops->func =3D call_direct_funcs_hash; + ops->flags =3D MULTI_FLAGS; + ops->trampoline =3D FTRACE_REGS_ADDR; + ops->local_hash.filter_hash =3D filter_hash; + + err =3D register_ftrace_function_nolock(ops); + if (!err) + filter_hash =3D NULL; + } else { + err =3D ftrace_update_ops(ops, filter_hash, EMPTY_HASH); + } + + out_unlock: + mutex_unlock(&direct_mutex); + + if (free_hash && free_hash !=3D EMPTY_HASH) + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb); + + if (filter_hash) + free_ftrace_hash(filter_hash); + + return err; +} +EXPORT_SYMBOL_GPL(register_ftrace_direct_hash); + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0AD57254841; Tue, 29 Jul 2025 10:28:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784931; cv=none; b=i2j4SIEG6Es/tMnarx9NQBlnGMFIBiC0x2Bc3E76vh3LR0V7P6l10trecYKUApeWCaG/NiPSnZ6AtmAi9Chp0l90+mDOV6oA1aUv1wtlNni41qe1ot8ytjD91d5RUA32ty2wnqckfjUIxeS0LUbCtjh5HZ3QufWscS2xvUQqCBI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784931; c=relaxed/simple; bh=h3k3dbcXrYPALSttNREnJ435oHCZsHpXZ8zQdyx2QEQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S27suGYM/EJIwRbU54JNLwy6cmP8aKPgWyf9enwtLI2W/b/4xDoI6oRX8qMEtuggMuvLyMWQ9kU5SatIvt4WupVSz+SlmbdrgS4jta57+6KKut106xSckj8Dp9e9p1vQXqX8E4nQCCRS8NHBFbH+fI85S2z5BIWkTQSuuVelvU8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OYpdw+Ph; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OYpdw+Ph" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A33DEC4CEEF; Tue, 29 Jul 2025 10:28:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784930; bh=h3k3dbcXrYPALSttNREnJ435oHCZsHpXZ8zQdyx2QEQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OYpdw+Ph67FkPffntdOi6PrnWlZxh4yCTGTMToqQPOHc4at6u6XbgCY+XOEiDkG2d IEW+R6cUKYMvkfs2s0C67Bjyfs73V6KFcr4s0IaAEQQJC54oELWuzApH3QeFdblp+w FRM+DxOAN9N+uxe9r9IauULWkUxkuNBygGWqwuoXipP58cqIbF4z8vhMEPq1NWnsn+ cPrR6M88PkrfcdeA3nZj4kEglWi9ueQq4jnUCf+206U9BBNq0Xl+biu4B88p9kWfIO pnx/SfOorAG02aZj6Ai1ZC+/aBcDQ+Mkvv2TckMf9tIHtrSH74Dec5u8egU4t/cmOS lMr0e+E57ic7A== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 03/10] ftrace: Add unregister_ftrace_direct_hash function Date: Tue, 29 Jul 2025 12:28:06 +0200 Message-ID: <20250729102813.1531457-4-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding unregister_ftrace_direct_hash function that unregisters all entries (ip -> direct) provided in hash argument. The difference to current unregister_ftrace_direct is - hash argument that allows to unregister multiple ip -> direct entries at once - we can call unregister_ftrace_direct_hash multiple times on the same ftrace_ops object, becase we do not need to unregister all entries at once, we can do it gradualy with the help of ftrace_update_ops function This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 +++ kernel/trace/ftrace.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index e45bcc9de53b..7ff6004498c0 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -527,6 +527,7 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr); int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); =20 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash= *hash); +int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_ha= sh *hash); =20 void ftrace_stub_direct_tramp(void); =20 @@ -559,6 +560,11 @@ int register_ftrace_direct_hash(struct ftrace_ops *ops= , struct ftrace_hash *hash return -ENODEV; } =20 +int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_ha= sh *hash) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 1dbb113f4e9c..d761237ec70f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6342,6 +6342,104 @@ int register_ftrace_direct_hash(struct ftrace_ops *= ops, struct ftrace_hash *hash } EXPORT_SYMBOL_GPL(register_ftrace_direct_hash); =20 +/** + * hash_sub - substracts @b from @a and returns the result + * @a: struct ftrace_hash object + * @b: struct ftrace_hash object + * + * Returns struct ftrace_hash object on success, NULL on error. + */ +static struct ftrace_hash *hash_sub(struct ftrace_hash *a, struct ftrace_h= ash *b) +{ + struct ftrace_func_entry *entry, *del; + struct ftrace_hash *sub; + int size, i; + + sub =3D alloc_and_copy_ftrace_hash(a->size_bits, a); + if (!sub) + goto error; + + size =3D 1 << b->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &b->buckets[i], hlist) { + del =3D __ftrace_lookup_ip(sub, entry->ip); + if (WARN_ON_ONCE(!del)) + goto error; + remove_hash_entry(sub, del); + kfree(del); + } + } + return sub; + + error: + free_ftrace_hash(sub); + return NULL; +} + +int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_ha= sh *hash) +{ + struct ftrace_hash *new_hash =3D NULL, *filter_hash =3D NULL, *free_hash = =3D NULL; + struct ftrace_func_entry *del, *entry; + unsigned long size, i; + int err =3D -EINVAL; + + if (!hash_count(hash)) + return 0; + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + if (direct_functions =3D=3D EMPTY_HASH) + return -EINVAL; + + mutex_lock(&direct_mutex); + + /* Make sure requested entries are already registered. */ + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + del =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (!del || del->direct !=3D entry->direct) + goto out_unlock; + } + } + + filter_hash =3D hash_sub(ops->func_hash->filter_hash, hash); + if (!filter_hash) + goto out_unlock; + + new_hash =3D hash_sub(direct_functions, hash); + if (!new_hash) + goto out_unlock; + + /* If there's nothing left, we need to unregister the ops. */ + if (ftrace_hash_empty(filter_hash)) { + err =3D unregister_ftrace_function(ops); + /* cleanup for possible another register call */ + ops->func =3D NULL; + ops->trampoline =3D 0; + ftrace_free_filter(ops); + ops->func_hash->filter_hash =3D NULL; + } else { + err =3D ftrace_update_ops(ops, filter_hash, EMPTY_HASH); + } + + free_hash =3D direct_functions; + rcu_assign_pointer(direct_functions, new_hash); + + out_unlock: + mutex_unlock(&direct_mutex); + + if (free_hash && free_hash !=3D EMPTY_HASH) + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb); + + if (filter_hash) + free_ftrace_hash(filter_hash); + + return err; +} +EXPORT_SYMBOL_GPL(unregister_ftrace_direct_hash); + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 546E3288C16; Tue, 29 Jul 2025 10:29:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784942; cv=none; b=mj1Xr9+x75Z2kVJxq1uCNDkVV1n1GpsKmBN8jm4vQQePzX0t+J3SlXpIXE2VDXeAhyASUhprV9+5TT6AZLcAx0qjtbe7Ai6kJi9a9N1R4M6Wl0yPYqgKNfEL5RONGPdDEIrSNJUVTq75AWO8IVEjuLJ3Z+7Sa6ZdpAqV+YrAdTA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784942; c=relaxed/simple; bh=oE3MjIoq7PdzWxxy/FqOc+ixaBdp6bBF/6oQYdRPSBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WIc5R8+sekMw0nc1M2Li/SgRKTwNKIb3fivWRCWX95khNtvRLreSwmM2TSdEi8sL9I51AUVlnv+JJdD4qe4N0uOLRkbOKPcExj3ffjRRqKvcq+EMeEUXUqX4ltSHhfXPSq+WT/of3WHzTSkClnT3Nqsp5m8TcffpM8D83zgB2aI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=a42U5aG5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="a42U5aG5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F78DC4CEEF; Tue, 29 Jul 2025 10:28:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784941; bh=oE3MjIoq7PdzWxxy/FqOc+ixaBdp6bBF/6oQYdRPSBU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a42U5aG5a3fObGpx09e13vzlH8lWFxPtU3jQ9W6k96sXIjw38o+IqjNYzqZtTENk6 yGK+6RAQOaU4nKzQ8pDHasG6tqGANgcWrLG+oRde7haZ38J9LxP9Z0HzJ3cMwiqLBQ y015CVcacFqd+j0OOXXZ71fgADmsb2NCF+wl/rwk2ezbaERYPFre5ILI3eHdmKE17r EaIfP/zvGSgNXsyfmJST6LXZcQnHaYceMqDsGo3yO/NT2qEm0LoMCHQ+XYVlmJAnHu 4KKz+3+HaC1aiuNdFnlZsGi3VIUzZ8j2Q+bdgQcMDkwRGweiyIkrjlbxRGyAZTPPVb kdInU15FKBnNw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 04/10] ftrace: Add modify_ftrace_direct_hash function Date: Tue, 29 Jul 2025 12:28:07 +0200 Message-ID: <20250729102813.1531457-5-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding modify_ftrace_direct_hash function that modifies all entries (ip -> direct) provided in hash argument. The difference to current unregister_ftrace_direct is - hash argument that allows to modify multiple ip -> direct entries at once This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 +++++ kernel/trace/ftrace.c | 58 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 7ff6004498c0..8761765d9abc 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -528,6 +528,7 @@ int modify_ftrace_direct_nolock(struct ftrace_ops *ops,= unsigned long addr); =20 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash= *hash); int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_ha= sh *hash); +int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *= hash, bool do_direct_lock); =20 void ftrace_stub_direct_tramp(void); =20 @@ -565,6 +566,11 @@ int unregister_ftrace_direct_hash(struct ftrace_ops *o= ps, struct ftrace_hash *ha return -ENODEV; } =20 +int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *= hash, bool do_direct_lock) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index d761237ec70f..755d5550ac44 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6440,6 +6440,64 @@ int unregister_ftrace_direct_hash(struct ftrace_ops = *ops, struct ftrace_hash *ha } EXPORT_SYMBOL_GPL(unregister_ftrace_direct_hash); =20 +int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *= hash, bool do_direct_lock) +{ + struct ftrace_func_entry *entry, *tmp; + static struct ftrace_ops tmp_ops =3D { + .func =3D ftrace_stub, + .flags =3D FTRACE_OPS_FL_STUB, + }; + unsigned long size, i; + int err; + + if (!hash_count(hash)) + return 0; + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + if (direct_functions =3D=3D EMPTY_HASH) + return -EINVAL; + + if (do_direct_lock) + mutex_lock(&direct_mutex); + + /* Enable the tmp_ops to have the same functions as the direct ops */ + ftrace_ops_init(&tmp_ops); + tmp_ops.func_hash =3D ops->func_hash; + + err =3D register_ftrace_function_nolock(&tmp_ops); + if (err) + goto unlock; + + /* + * Now the ftrace_ops_list_func() is called to do the direct callers. + * We can safely change the direct functions attached to each entry. + */ + mutex_lock(&ftrace_lock); + + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + tmp =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (!tmp) + continue; + tmp->direct =3D entry->direct; + } + } + + mutex_unlock(&ftrace_lock); + + /* Removing the tmp_ops will add the updated direct callers to the functi= ons */ + unregister_ftrace_function(&tmp_ops); + +unlock: + if (do_direct_lock) + mutex_unlock(&direct_mutex); + return err; +} +EXPORT_SYMBOL_GPL(modify_ftrace_direct_hash); + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BBFD289E1D; Tue, 29 Jul 2025 10:29:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784952; cv=none; b=LG8AjfDXriY+PZbpUTSOGjTJZV+C5ps/LDHF05Hlbwdews5OznWe0/st4dUSD9rpXTRKMmQL0d9zLoHNUf8RbVegLTAvpNh2MT3+N9geKGXmeIHiQI9+Oti/5h8/JMno03/G7MXCdaPyw+gHPgQwGQq6LHIdw3D6JsCq7eTD2wA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784952; c=relaxed/simple; bh=s36VlYHt7rzpyx/CrkVrQ1GGY7J1e5RhRFc94AGzl2c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i63VjITmokCN7gPdnE9sQ8SuV4LQWSK2tBGvsII4HkkWXAx/234DfFjdEmHfR3XMYb21uWdg8Q75+rWdewe1LeKZAm5euwd8THhoooVLXT0wnIDkSJXee9EyeYVt9EV3KjgbqqnjCQFsMFMMfmlfFY47rHI3LrufwWix1VWcsUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fY1iq88j; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fY1iq88j" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 45FCFC4CEEF; Tue, 29 Jul 2025 10:29:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784952; bh=s36VlYHt7rzpyx/CrkVrQ1GGY7J1e5RhRFc94AGzl2c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fY1iq88jHx1GrHviGUxlO4tWGmKap18d55ll7NauQ+Mcq2RPpapqthWT85hfQeDn3 7yqEphpl+ThJweK0D03HOUAkr8NU31Gqbssej/nk2Zsc82MCTXIv3d/deV/lfbuzoa Tz+xdweZ5lVOsbnekAAAYcsXeoCPp18xQ5bgVMVGFcWKX6M9mPNlJb3HIMXX/twinF aLhy7gMyHR31gfNdf1ktTYnSgBih1tAgdDvnQESrgyvoNfiKMECvxEpSOtGHgyA+Cz QbNFKR/Ue7Qx3CKCTikYs8xeLbPwg+kXXnlg9fSzmXNB5MV6AXVEp5/OFmFJ1cGTgu aJuPWlhNquGIQ== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 05/10] ftrace: Export some of hash related functions Date: Tue, 29 Jul 2025 12:28:08 +0200 Message-ID: <20250729102813.1531457-6-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We are going to use these functions in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 16 ++++++++++++++++ kernel/trace/ftrace.c | 7 +++---- kernel/trace/trace.h | 8 -------- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8761765d9abc..9a6fcdafeda2 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -397,6 +397,22 @@ enum ftrace_ops_cmd { typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cm= d cmd); =20 #ifdef CONFIG_DYNAMIC_FTRACE + +#define FTRACE_HASH_DEFAULT_BITS 10 + +struct ftrace_hash { + unsigned long size_bits; + struct hlist_head *buckets; + unsigned long count; + unsigned long flags; + struct rcu_head rcu; +}; + +struct ftrace_hash *alloc_ftrace_hash(int size_bits); +void free_ftrace_hash(struct ftrace_hash *hash); +struct ftrace_func_entry *add_hash_entry_direct(struct ftrace_hash *hash, + unsigned long ip, unsigned long direct); + /* The hash used to know what functions callbacks trace */ struct ftrace_ops_hash { struct ftrace_hash __rcu *notrace_hash; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 755d5550ac44..fcb8f2d3172b 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -68,7 +68,6 @@ }) =20 /* hash bits for specific function selection */ -#define FTRACE_HASH_DEFAULT_BITS 10 #define FTRACE_HASH_MAX_BITS 12 =20 #ifdef CONFIG_DYNAMIC_FTRACE @@ -1189,7 +1188,7 @@ static void __add_hash_entry(struct ftrace_hash *hash, hash->count++; } =20 -static struct ftrace_func_entry * +struct ftrace_func_entry * add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned= long direct) { struct ftrace_func_entry *entry; @@ -1269,7 +1268,7 @@ static void clear_ftrace_mod_list(struct list_head *h= ead) mutex_unlock(&ftrace_lock); } =20 -static void free_ftrace_hash(struct ftrace_hash *hash) +void free_ftrace_hash(struct ftrace_hash *hash) { if (!hash || hash =3D=3D EMPTY_HASH) return; @@ -1309,7 +1308,7 @@ void ftrace_free_filter(struct ftrace_ops *ops) } EXPORT_SYMBOL_GPL(ftrace_free_filter); =20 -static struct ftrace_hash *alloc_ftrace_hash(int size_bits) +struct ftrace_hash *alloc_ftrace_hash(int size_bits) { struct ftrace_hash *hash; int size; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index bd084953a98b..74ef7755f361 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -899,14 +899,6 @@ enum { FTRACE_HASH_FL_MOD =3D (1 << 0), }; =20 -struct ftrace_hash { - unsigned long size_bits; - struct hlist_head *buckets; - unsigned long count; - unsigned long flags; - struct rcu_head rcu; -}; - struct ftrace_func_entry * ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip); =20 --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB40A289E1D; Tue, 29 Jul 2025 10:29:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784963; cv=none; b=UO5sJ7uwMhWFws46j6fhGiujyN8MUHRPElY1/zQQc91iHDglBD39RGH7Tv8etPhkgBxI+xkRdg79iwtHkdXMB2mzjKh2osYwjhmtx2CfaZNooA4/ghV9mD/FVc9fdmgt0Sai4cC5sI96p65JQgAGXzYIU2yDkTV3Jfw6dh+8vtQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784963; c=relaxed/simple; bh=LkMiSu9rJHL78CHuuZGPPKsY5tkXMO6EbHSe73wtQp4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uCFAKkRjk5sOist45vBcKiIfXIYglFWvog8lGCpMa7/dsGEUxreEnU+Qwgf//tLF06Gb+vemJf9C94SJtTfpd1OnBrtZFoRLGpte81ChV5OViXx16o+MlkbSEHjzSE/6wXX4RFZe6+1Qn7dToTtciJOQ1RwtraKZ7yiZdL3pLjs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rVYSeU9r; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rVYSeU9r" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9DE94C4CEEF; Tue, 29 Jul 2025 10:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784962; bh=LkMiSu9rJHL78CHuuZGPPKsY5tkXMO6EbHSe73wtQp4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rVYSeU9r1+uoZGBIGoBMQ6R+h4oY3hJF2KI+FDxgk5+uR6qnN5Wb9kjh7u6IMih6i O5jfI1GiHFgpxpo+HuYXZuHthjNJ5ms74hedEX7TfJmXzkiqMU4TMDwErUdajRxIwQ O4pt4+0jtJrojvbSrwLxvmoUwewjfLJRbfn7zHT383JT3QbYk6+V18BIHn7B/kD2MP StEoMpXkVCDweHzbG1dt+yA7qia0tfLCmQ0MwB57/y5SoTn99bOjhYcKdX9R27adND zlglEGgArKhemOYfUc9dZf2QNGZkqHcblZvwB6M5cK5f9t/nLeFIkPUSeZ5gLNSfSI TJlplUdrP/sww== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 06/10] ftrace: Use direct hash interface in direct functions Date: Tue, 29 Jul 2025 12:28:09 +0200 Message-ID: <20250729102813.1531457-7-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement current *_ftrace_direct function with their *_hash function counterparts. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 17 +-- kernel/bpf/trampoline.c | 10 +- kernel/trace/ftrace.c | 242 +++++----------------------------- kernel/trace/trace_selftest.c | 5 +- 4 files changed, 45 insertions(+), 229 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 9a6fcdafeda2..85f4ab1a1e72 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -536,11 +536,10 @@ struct ftrace_func_entry { =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS unsigned long ftrace_find_rec_direct(unsigned long ip); -int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); -int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr, +int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsig= ned long addr); +int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, uns= igned long addr, bool free_filters); -int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); -int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); +int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigne= d long addr, bool lock_direct_mutex); =20 int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash= *hash); int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_ha= sh *hash); @@ -554,20 +553,16 @@ static inline unsigned long ftrace_find_rec_direct(un= signed long ip) { return 0; } -static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned = long addr) +static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned = long ip, unsigned long addr) { return -ENODEV; } -static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigne= d long addr, +static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigne= d long ip, unsigned long addr, bool free_filters) { return -ENODEV; } -static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned lo= ng addr) -{ - return -ENODEV; -} -static inline int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsi= gned long addr) +static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned lo= ng ip, unsigned long addr, bool lock_direct_mutex) { return -ENODEV; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 0e364614c3a2..6bf272715f0e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -181,7 +181,7 @@ static int unregister_fentry(struct bpf_trampoline *tr,= void *old_addr) int ret; =20 if (tr->func.ftrace_managed) - ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); + ret =3D unregister_ftrace_direct(tr->fops, (unsigned long) ip, (long)old= _addr, false); else ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); =20 @@ -195,10 +195,7 @@ static int modify_fentry(struct bpf_trampoline *tr, vo= id *old_addr, void *new_ad int ret; =20 if (tr->func.ftrace_managed) { - if (lock_direct_mutex) - ret =3D modify_ftrace_direct(tr->fops, (long)new_addr); - else - ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); + ret =3D modify_ftrace_direct(tr->fops, (unsigned long) ip, (long)new_add= r, lock_direct_mutex); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); } @@ -220,8 +217,7 @@ static int register_fentry(struct bpf_trampoline *tr, v= oid *new_addr) } =20 if (tr->func.ftrace_managed) { - ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); - ret =3D register_ftrace_direct(tr->fops, (long)new_addr); + ret =3D register_ftrace_direct(tr->fops, (unsigned long)ip, (long)new_ad= dr); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); } diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index fcb8f2d3172b..151ca94f496a 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2593,16 +2593,6 @@ unsigned long ftrace_find_rec_direct(unsigned long i= p) return entry->direct; } =20 -static void call_direct_funcs(unsigned long ip, unsigned long pip, - struct ftrace_ops *ops, struct ftrace_regs *fregs) -{ - unsigned long addr =3D READ_ONCE(ops->direct_call); - - if (!addr) - return; - - arch_ftrace_set_direct_caller(fregs, addr); -} #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** @@ -5935,28 +5925,24 @@ static int check_direct_multi(struct ftrace_ops *op= s) return 0; } =20 -static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigne= d long addr) +static void register_ftrace_direct_cb(struct rcu_head *rhp) { - struct ftrace_func_entry *entry, *del; - int size, i; + struct ftrace_hash *fhp =3D container_of(rhp, struct ftrace_hash, rcu); =20 - size =3D 1 << hash->size_bits; - for (i =3D 0; i < size; i++) { - hlist_for_each_entry(entry, &hash->buckets[i], hlist) { - del =3D __ftrace_lookup_ip(direct_functions, entry->ip); - if (del && del->direct =3D=3D addr) { - remove_hash_entry(direct_functions, del); - kfree(del); - } - } - } + free_ftrace_hash(fhp); } =20 -static void register_ftrace_direct_cb(struct rcu_head *rhp) +static struct ftrace_hash *hash_from_ip(unsigned long ip, unsigned long ad= dr) { - struct ftrace_hash *fhp =3D container_of(rhp, struct ftrace_hash, rcu); + struct ftrace_hash *hash; =20 - free_ftrace_hash(fhp); + ip =3D ftrace_location(ip); + if (!ip) + return NULL; + hash =3D alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); + if (!hash || !add_hash_entry_direct(hash, ip, addr)) + return NULL; + return hash; } =20 /** @@ -5981,89 +5967,17 @@ static void register_ftrace_direct_cb(struct rcu_he= ad *rhp) * -ENODEV - @ip does not point to a ftrace nop location (or not support= ed) * -ENOMEM - There was an allocation failure. */ -int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) +int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsig= ned long addr) { - struct ftrace_hash *hash, *new_hash =3D NULL, *free_hash =3D NULL; - struct ftrace_func_entry *entry, *new; - int err =3D -EBUSY, size, i; - - if (ops->func || ops->trampoline) - return -EINVAL; - if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) - return -EINVAL; - if (ops->flags & FTRACE_OPS_FL_ENABLED) - return -EINVAL; - - hash =3D ops->func_hash->filter_hash; - if (ftrace_hash_empty(hash)) - return -EINVAL; - - mutex_lock(&direct_mutex); - - /* Make sure requested entries are not already registered.. */ - size =3D 1 << hash->size_bits; - for (i =3D 0; i < size; i++) { - hlist_for_each_entry(entry, &hash->buckets[i], hlist) { - if (ftrace_find_rec_direct(entry->ip)) - goto out_unlock; - } - } - - err =3D -ENOMEM; - - /* Make a copy hash to place the new and the old entries in */ - size =3D hash->count + direct_functions->count; - size =3D fls(size); - if (size > FTRACE_HASH_MAX_BITS) - size =3D FTRACE_HASH_MAX_BITS; - new_hash =3D alloc_ftrace_hash(size); - if (!new_hash) - goto out_unlock; - - /* Now copy over the existing direct entries */ - size =3D 1 << direct_functions->size_bits; - for (i =3D 0; i < size; i++) { - hlist_for_each_entry(entry, &direct_functions->buckets[i], hlist) { - new =3D add_hash_entry(new_hash, entry->ip); - if (!new) - goto out_unlock; - new->direct =3D entry->direct; - } - } - - /* ... and add the new entries */ - size =3D 1 << hash->size_bits; - for (i =3D 0; i < size; i++) { - hlist_for_each_entry(entry, &hash->buckets[i], hlist) { - new =3D add_hash_entry(new_hash, entry->ip); - if (!new) - goto out_unlock; - /* Update both the copy and the hash entry */ - new->direct =3D addr; - entry->direct =3D addr; - } - } - - free_hash =3D direct_functions; - rcu_assign_pointer(direct_functions, new_hash); - new_hash =3D NULL; - - ops->func =3D call_direct_funcs; - ops->flags =3D MULTI_FLAGS; - ops->trampoline =3D FTRACE_REGS_ADDR; - ops->direct_call =3D addr; - - err =3D register_ftrace_function_nolock(ops); - - out_unlock: - mutex_unlock(&direct_mutex); - - if (free_hash && free_hash !=3D EMPTY_HASH) - call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb); + struct ftrace_hash *hash; + int err; =20 - if (new_hash) - free_ftrace_hash(new_hash); + hash =3D hash_from_ip(ip, addr); + if (!hash) + return -ENOMEM; =20 + err =3D register_ftrace_direct_hash(ops, hash); + free_ftrace_hash(hash); return err; } EXPORT_SYMBOL_GPL(register_ftrace_direct); @@ -6083,111 +5997,24 @@ EXPORT_SYMBOL_GPL(register_ftrace_direct); * 0 on success * -EINVAL - The @ops object was not properly registered. */ -int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr, +int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, uns= igned long addr, bool free_filters) { - struct ftrace_hash *hash =3D ops->func_hash->filter_hash; + struct ftrace_hash *hash; int err; =20 - if (check_direct_multi(ops)) - return -EINVAL; - if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) - return -EINVAL; - - mutex_lock(&direct_mutex); - err =3D unregister_ftrace_function(ops); - remove_direct_functions_hash(hash, addr); - mutex_unlock(&direct_mutex); - - /* cleanup for possible another register call */ - ops->func =3D NULL; - ops->trampoline =3D 0; + hash =3D hash_from_ip(ip, addr); + if (!hash) + return -ENOMEM; =20 + err =3D unregister_ftrace_direct_hash(ops, hash); + free_ftrace_hash(hash); if (free_filters) ftrace_free_filter(ops); return err; } EXPORT_SYMBOL_GPL(unregister_ftrace_direct); =20 -static int -__modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) -{ - struct ftrace_hash *hash; - struct ftrace_func_entry *entry, *iter; - static struct ftrace_ops tmp_ops =3D { - .func =3D ftrace_stub, - .flags =3D FTRACE_OPS_FL_STUB, - }; - int i, size; - int err; - - lockdep_assert_held_once(&direct_mutex); - - /* Enable the tmp_ops to have the same functions as the direct ops */ - ftrace_ops_init(&tmp_ops); - tmp_ops.func_hash =3D ops->func_hash; - tmp_ops.direct_call =3D addr; - - err =3D register_ftrace_function_nolock(&tmp_ops); - if (err) - return err; - - /* - * Now the ftrace_ops_list_func() is called to do the direct callers. - * We can safely change the direct functions attached to each entry. - */ - mutex_lock(&ftrace_lock); - - hash =3D ops->func_hash->filter_hash; - size =3D 1 << hash->size_bits; - for (i =3D 0; i < size; i++) { - hlist_for_each_entry(iter, &hash->buckets[i], hlist) { - entry =3D __ftrace_lookup_ip(direct_functions, iter->ip); - if (!entry) - continue; - entry->direct =3D addr; - } - } - /* Prevent store tearing if a trampoline concurrently accesses the value = */ - WRITE_ONCE(ops->direct_call, addr); - - mutex_unlock(&ftrace_lock); - - /* Removing the tmp_ops will add the updated direct callers to the functi= ons */ - unregister_ftrace_function(&tmp_ops); - - return err; -} - -/** - * modify_ftrace_direct_nolock - Modify an existing direct 'multi' call - * to call something else - * @ops: The address of the struct ftrace_ops object - * @addr: The address of the new trampoline to call at @ops functions - * - * This is used to unregister currently registered direct caller and - * register new one @addr on functions registered in @ops object. - * - * Note there's window between ftrace_shutdown and ftrace_startup calls - * where there will be no callbacks called. - * - * Caller should already have direct_mutex locked, so we don't lock - * direct_mutex here. - * - * Returns: zero on success. Non zero on error, which includes: - * -EINVAL - The @ops object was not properly registered. - */ -int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr) -{ - if (check_direct_multi(ops)) - return -EINVAL; - if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) - return -EINVAL; - - return __modify_ftrace_direct(ops, addr); -} -EXPORT_SYMBOL_GPL(modify_ftrace_direct_nolock); - /** * modify_ftrace_direct - Modify an existing direct 'multi' call * to call something else @@ -6203,18 +6030,17 @@ EXPORT_SYMBOL_GPL(modify_ftrace_direct_nolock); * Returns: zero on success. Non zero on error, which includes: * -EINVAL - The @ops object was not properly registered. */ -int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) +int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigne= d long addr, bool lock_direct_mutex) { + struct ftrace_hash *hash; int err; =20 - if (check_direct_multi(ops)) - return -EINVAL; - if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) - return -EINVAL; + hash =3D hash_from_ip(ip, addr); + if (!hash) + return -ENOMEM; =20 - mutex_lock(&direct_mutex); - err =3D __modify_ftrace_direct(ops, addr); - mutex_unlock(&direct_mutex); + err =3D modify_ftrace_direct_hash(ops, hash, lock_direct_mutex); + free_ftrace_hash(hash); return err; } EXPORT_SYMBOL_GPL(modify_ftrace_direct); diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index d88c44f1dfa5..37f5eb1f252b 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -1135,8 +1135,7 @@ trace_selftest_startup_function_graph(struct tracer *= trace, * Register direct function together with graph tracer * and make sure we get graph trace. */ - ftrace_set_filter_ip(&direct, (unsigned long)DYN_FTRACE_TEST_NAME, 0, 0); - ret =3D register_ftrace_direct(&direct, + ret =3D register_ftrace_direct(&direct, (unsigned long)DYN_FTRACE_TEST_NA= ME, (unsigned long)ftrace_stub_direct_tramp); if (ret) goto out; @@ -1159,7 +1158,7 @@ trace_selftest_startup_function_graph(struct tracer *= trace, =20 unregister_ftrace_graph(&fgraph_ops); =20 - ret =3D unregister_ftrace_direct(&direct, + ret =3D unregister_ftrace_direct(&direct, (unsigned long)DYN_FTRACE_TEST_= NAME, (unsigned long)ftrace_stub_direct_tramp, true); if (ret) --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CC33289837; Tue, 29 Jul 2025 10:29:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784974; cv=none; b=A11MDhJohwGkgPNqR8h13VQoZ87FDeUQNVHrPsmfC75rFY7FCg/GoR05D/TjxK1iojbmyQR160dyDO1kZrcAHtovf+5ZNS47Dh2F5GzQjsnYDCY4AVf/W9U5YwuOR/7fjQVRa4XiOTSt1GtPK21hY9qX7VqH6irBMhV9YRhBsgs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784974; c=relaxed/simple; bh=3QJB2DoO8ptl5Ma9rnI3pBV9VwWVKMiKtM9HSvTM0oA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X+oZ9XqLDUKvl9wCEsK1WzZzj3Tz9fGapmgar+LYRXSoDGohLkOukTvpck8txphozphsVoG7a6nrw/qA2KYajKlaaE2lHYE5iPzIcXxmjJ1O14xj0Dvkm8bNnEHFdMY14O9gtZt8+NjUgMc2ZCksgqo0S48S8dlQARuBeFZXUUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=B7tRE936; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="B7tRE936" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2952C4CEEF; Tue, 29 Jul 2025 10:29:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784974; bh=3QJB2DoO8ptl5Ma9rnI3pBV9VwWVKMiKtM9HSvTM0oA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B7tRE936F9PLGHsR3yRf74NWRSMDUMYnlqsKrB3JLRtIuQLKt6U1KCMnepQcqv9eM Gk2c3EELpaxfz5iC2RaNIIF9tFnyhCGYcfJLvy5gCuDHXujc6U6SoT+0/V1H+JSdgo yOH5ChbNpwNIjIbhlVaOaarKEAt7hQyoYuxlmkw7OvbYeK8pzbnlDRXOWlzQDbE1Ox 45l+FKH/fJ+6cBfM2gG7IFQN/2EWeihq3mN2kw6oDeQADqwY4ZsDI+1oDtwK/Pj/CL 13nZ+M1WslZcOIGHCtv2cYwdJlno5pwQA1dPV0DHaMwgZb1H6NrR4fJqcWAvYB+4oW oMMMfSjViCsRA== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 07/10] bpf: Add trampoline ip hash table Date: Tue, 29 Jul 2025 12:28:10 +0200 Message-ID: <20250729102813.1531457-8-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Following changes need to lookup trampoline based on its ip address, adding hash table for that. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 7 +++++-- kernel/bpf/trampoline.c | 30 +++++++++++++++++++----------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f9cd2164ed23..c14bde400d97 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1253,14 +1253,17 @@ struct bpf_tramp_image { }; =20 struct bpf_trampoline { - /* hlist for trampoline_table */ - struct hlist_node hlist; + /* hlist for trampoline_key_table */ + struct hlist_node hlist_key; + /* hlist for trampoline_ip_table */ + struct hlist_node hlist_ip; struct ftrace_ops *fops; /* serializes access to fields of this trampoline */ struct mutex mutex; refcount_t refcnt; u32 flags; u64 key; + unsigned long ip; struct { struct btf_func_model model; void *addr; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 6bf272715f0e..84bcd9f6bd74 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -24,9 +24,10 @@ const struct bpf_prog_ops bpf_extension_prog_ops =3D { #define TRAMPOLINE_HASH_BITS 10 #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS) =20 -static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE]; +static struct hlist_head trampoline_key_table[TRAMPOLINE_TABLE_SIZE]; +static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; =20 -/* serializes access to trampoline_table */ +/* serializes access to trampoline tables */ static DEFINE_MUTEX(trampoline_mutex); =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS @@ -135,15 +136,15 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } =20 -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long= ip) { struct bpf_trampoline *tr; struct hlist_head *head; int i; =20 mutex_lock(&trampoline_mutex); - head =3D &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; - hlist_for_each_entry(tr, head, hlist) { + head =3D &trampoline_key_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head, hlist_key) { if (tr->key =3D=3D key) { refcount_inc(&tr->refcnt); goto out; @@ -164,8 +165,12 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u6= 4 key) #endif =20 tr->key =3D key; - INIT_HLIST_NODE(&tr->hlist); - hlist_add_head(&tr->hlist, head); + tr->ip =3D ip; + INIT_HLIST_NODE(&tr->hlist_key); + INIT_HLIST_NODE(&tr->hlist_ip); + hlist_add_head(&tr->hlist_key, head); + head =3D &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)]; + hlist_add_head(&tr->hlist_ip, head); refcount_set(&tr->refcnt, 1); mutex_init(&tr->mutex); for (i =3D 0; i < BPF_TRAMP_MAX; i++) @@ -800,7 +805,7 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog = *prog) prog->aux->attach_btf_id); =20 bpf_lsm_find_cgroup_shim(prog, &bpf_func); - tr =3D bpf_trampoline_lookup(key); + tr =3D bpf_trampoline_lookup(key, 0); if (WARN_ON_ONCE(!tr)) return; =20 @@ -820,7 +825,7 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, { struct bpf_trampoline *tr; =20 - tr =3D bpf_trampoline_lookup(key); + tr =3D bpf_trampoline_lookup(key, tgt_info->tgt_addr); if (!tr) return NULL; =20 @@ -856,7 +861,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) * fexit progs. The fentry-only trampoline will be freed via * multiple rcu callbacks. */ - hlist_del(&tr->hlist); + hlist_del(&tr->hlist_key); + hlist_del(&tr->hlist_ip); if (tr->fops) { ftrace_free_filter(tr->fops); kfree(tr->fops); @@ -1135,7 +1141,9 @@ static int __init init_trampolines(void) int i; =20 for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) - INIT_HLIST_HEAD(&trampoline_table[i]); + INIT_HLIST_HEAD(&trampoline_key_table[i]); + for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) + INIT_HLIST_HEAD(&trampoline_ip_table[i]); return 0; } late_initcall(init_trampolines); --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6A80289837; Tue, 29 Jul 2025 10:29:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784985; cv=none; b=XNxCtvLgNFZn4IZE3eveRhgD6T0LpBU28PIqi7bKKaCC+ADOx4ogO9vrf4pAd8Q3BZlvitOcmhwLT0s49kd1X6GUcv3CMIqoi5xmQhKqf0DzQkCIzZjr6NYxsfL/icabtohg+3qC5Y8YCugY+pHOpAgbjfMdisUCC8/tc7ENeF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784985; c=relaxed/simple; bh=gkMMkcsvhGCheI1Y4239jYBPUdUxOQ7UslqUuwaf+NY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O24B3RbEZmr4Rrj8Y8mXlA72TibKr+Sb6KGdM61UIkuk7LWmlCEKffZDGyg/4rVBE6d1WK9YyjhkAN0dB8yell9AtVLu6pYH4EUbY+rb+KWmjsbUjUmZbbVq489j2Ksm1p4ZrFgzChVawaL+sLUZgJSnZ+1SoP1wK2maXY281dM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=T1lpVrWs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="T1lpVrWs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7055BC4CEEF; Tue, 29 Jul 2025 10:29:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784984; bh=gkMMkcsvhGCheI1Y4239jYBPUdUxOQ7UslqUuwaf+NY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T1lpVrWsiXKASxyTeBu3hnP8GrAosrNWPwwnx8nqmGQ/OYdrOweMpZfcPJDM+xl/N F4XD0bEzsQhoeUhI1IHdDkNny0ntOU3R1ORBxaQKAcCSho40rLU0TZIsP/caOl2j6b eUlT9Uk4O4stCse23RQeKCo1mb0Ci0/GK4CSfYkh3xGtgDuubYvBZ7csbKtFs8vc6h NpQj1c+ZyyuLgC2IMKhfCxvkk5aJPNLOPJkD7xbB8QB/tfxtMsifUOhIdprlmn3mwt yVfgNANCpMlJmOR9VgQh7NBbTPXT9aTogVoczcShubaCjQA195m5rR5a8Jdnuv0HmG Vhk5fbNUqOm4Q== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 08/10] ftrace: Factor ftrace_ops ops_func interface Date: Tue, 29 Jul 2025 12:28:11 +0200 Message-ID: <20250729102813.1531457-9-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We are going to remove "ftrace_ops->private =3D=3D bpf_trampoline" setup in following changes. Adding ip argument to ftrace_ops_func_t callback function, so we can use it to look up the trampoline. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 2 +- kernel/bpf/trampoline.c | 26 ++++++++++++++++++++++++-- kernel/trace/ftrace.c | 6 +++--- 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 85f4ab1a1e72..1a61f969550d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -394,7 +394,7 @@ enum ftrace_ops_cmd { * Negative on failure. The return value is dependent on the * callback. */ -typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cm= d cmd); +typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, unsigned long ip, = enum ftrace_ops_cmd cmd); =20 #ifdef CONFIG_DYNAMIC_FTRACE =20 diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 84bcd9f6bd74..398c1a722d83 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -33,11 +33,33 @@ static DEFINE_MUTEX(trampoline_mutex); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_dire= ct_mutex); =20 -static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, enum ftrace_o= ps_cmd cmd) +static struct bpf_trampoline *bpf_trampoline_ip_lookup(unsigned long ip) { - struct bpf_trampoline *tr =3D ops->private; + struct hlist_head *head_ip; + struct bpf_trampoline *tr; + + mutex_lock(&trampoline_mutex); + head_ip =3D &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head_ip, hlist_ip) { + if (tr->func.addr =3D=3D (void *) ip) + goto out; + } + tr =3D NULL; +out: + mutex_unlock(&trampoline_mutex); + return tr; +} + +static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long= ip, + enum ftrace_ops_cmd cmd) +{ + struct bpf_trampoline *tr; int ret =3D 0; =20 + tr =3D bpf_trampoline_ip_lookup(ip); + if (!tr) + return -EINVAL; + if (cmd =3D=3D FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { /* This is called inside register_ftrace_direct_multi(), so * tr->mutex is already locked. diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 151ca94f496a..943feabdd5e6 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2040,7 +2040,7 @@ static int __ftrace_hash_update_ipmodify(struct ftrac= e_ops *ops, */ if (!ops->ops_func) return -EBUSY; - ret =3D ops->ops_func(ops, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF); + ret =3D ops->ops_func(ops, rec->ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODI= FY_SELF); if (ret) return ret; } else if (is_ipmodify) { @@ -8746,7 +8746,7 @@ static int prepare_direct_functions_for_ipmodify(stru= ct ftrace_ops *ops) if (!op->ops_func) return -EBUSY; =20 - ret =3D op->ops_func(op, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER); + ret =3D op->ops_func(op, ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER= ); if (ret) return ret; } @@ -8793,7 +8793,7 @@ static void cleanup_direct_functions_after_ipmodify(s= truct ftrace_ops *ops) =20 /* The cleanup is optional, ignore any errors */ if (found_op && op->ops_func) - op->ops_func(op, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER); + op->ops_func(op, ip, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER); } } mutex_unlock(&direct_mutex); --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB01B28B3F8; Tue, 29 Jul 2025 10:29:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784994; cv=none; b=gjLu9JLTi4Sq/rl3Rt/alNOfoOeRfdSRxa/6tumtlcrkNyFgjtTkOE53WNVN0tg2lSOqy+TY8DsPeXp0kFoksW+EYXbyIN3d2BZNVetPe41bqaqjQfjTun/ighK7mcepc0u3gyHn/JVthUS9d4SQqyiyX9AP709Q6KQjeP9lptQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753784994; c=relaxed/simple; bh=UFgBnii1sdnMHWK81UnyXcALAP0f+Ry5dK/qMV1lLOk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fq93kWfM4EonUIwrRTe2I5228bIzrOGNvgCGVNsO8ML/TLJ3VhVS11CPbK+utnOAtFvWN5+tNlTT/cnZ6RU6NBObr91j20DZtBI4tMVoEVunHIHtlg+mu39EihRrnjhPQPT9Yl0IOPnB5yzF7U8BQttnyNNgNVe+E8pEj3BHnAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Qk9/9OIX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Qk9/9OIX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D41DC4CEEF; Tue, 29 Jul 2025 10:29:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753784994; bh=UFgBnii1sdnMHWK81UnyXcALAP0f+Ry5dK/qMV1lLOk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Qk9/9OIXssyCgo9MwjG36TdZkK76F6foHCZOoraayhJ+SrW6vtIs9CmDiHWXO6FMJ xQt8jMV/GonZ01c8wf/1TquBLjawrFzDnkBaDctTYiqSl2o/IqBb28MsxZpy/K05Ry bmXdUD+NMWK5LJ4R1tOYvTY+8evd7RVzA46HwRoNyKIFcihfGaI/MXYhhXeCrveZLN j6xIBINpzVSzMgg76g1A/9thAccmLeHh2ctr27T3yhmFGSRFTJp7TAySEqBbUBklGy GorVHY7RnIrLvc+xHbWqEX0dttAddWiaITK136lnlTj/Iied4dOe/vjwg5pk/OeNCB cBJxEpgz/ZcPA== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 09/10] bpf: Remove ftrace_ops from bpf_trampoline object Date: Tue, 29 Jul 2025 12:28:12 +0200 Message-ID: <20250729102813.1531457-10-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We no longer need ftrace_ops in each bpf_trampoline object, we can manage with just single ftrace_ops for all direct trampoline attachments. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 1 - kernel/bpf/trampoline.c | 34 ++++++++++------------------------ 2 files changed, 10 insertions(+), 25 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c14bde400d97..bad29fe38a12 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1257,7 +1257,6 @@ struct bpf_trampoline { struct hlist_node hlist_key; /* hlist for trampoline_ip_table */ struct hlist_node hlist_ip; - struct ftrace_ops *fops; /* serializes access to fields of this trampoline */ struct mutex mutex; refcount_t refcnt; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 398c1a722d83..e6a0e7b20bb6 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -175,16 +175,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u6= 4 key, unsigned long ip) tr =3D kzalloc(sizeof(*tr), GFP_KERNEL); if (!tr) goto out; -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS - tr->fops =3D kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL); - if (!tr->fops) { - kfree(tr); - tr =3D NULL; - goto out; - } - tr->fops->private =3D tr; - tr->fops->ops_func =3D bpf_tramp_ftrace_ops_func; -#endif =20 tr->key =3D key; tr->ip =3D ip; @@ -202,13 +192,19 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u= 64 key, unsigned long ip) return tr; } =20 +struct ftrace_ops direct_ops =3D { +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + .ops_func =3D bpf_tramp_ftrace_ops_func, +#endif +}; + static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) { void *ip =3D tr->func.addr; int ret; =20 if (tr->func.ftrace_managed) - ret =3D unregister_ftrace_direct(tr->fops, (unsigned long) ip, (long)old= _addr, false); + ret =3D unregister_ftrace_direct(&direct_ops, (unsigned long) ip, (long)= old_addr, false); else ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); =20 @@ -222,7 +218,7 @@ static int modify_fentry(struct bpf_trampoline *tr, voi= d *old_addr, void *new_ad int ret; =20 if (tr->func.ftrace_managed) { - ret =3D modify_ftrace_direct(tr->fops, (unsigned long) ip, (long)new_add= r, lock_direct_mutex); + ret =3D modify_ftrace_direct(&direct_ops, (unsigned long) ip, (long)new_= addr, lock_direct_mutex); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); } @@ -237,14 +233,11 @@ static int register_fentry(struct bpf_trampoline *tr,= void *new_addr) int ret; =20 faddr =3D ftrace_location((unsigned long)ip); - if (faddr) { - if (!tr->fops) - return -ENOTSUPP; + if (faddr) tr->func.ftrace_managed =3D true; - } =20 if (tr->func.ftrace_managed) { - ret =3D register_ftrace_direct(tr->fops, (unsigned long)ip, (long)new_ad= dr); + ret =3D register_ftrace_direct(&direct_ops, (unsigned long)ip, (long)new= _addr); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); } @@ -502,9 +495,6 @@ static int bpf_trampoline_update(struct bpf_trampoline = *tr, bool lock_direct_mut * BPF_TRAMP_F_SHARE_IPMODIFY is set, we can generate the * trampoline again, and retry register. */ - /* reset fops->func and fops->trampoline for re-register */ - tr->fops->func =3D NULL; - tr->fops->trampoline =3D 0; =20 /* free im memory and reallocate later */ bpf_tramp_image_free(im); @@ -885,10 +875,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) */ hlist_del(&tr->hlist_key); hlist_del(&tr->hlist_ip); - if (tr->fops) { - ftrace_free_filter(tr->fops); - kfree(tr->fops); - } kfree(tr); out: mutex_unlock(&trampoline_mutex); --=20 2.50.1 From nobody Sun Oct 5 21:59:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 186D528C03A; Tue, 29 Jul 2025 10:30:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753785005; cv=none; b=WZJpHdu/eFxfas0s9vwAWBV1I+smFEm5E+zPO3H3yBDkra6RgH28ydiM4qUCrzDaVZK5vgnQpwGD1gr87AZkU7rkj/b3MXemEaXJ+a9L7NR46RseZtbo/ZHm0R+CscO8NH3aHFLz2NJxayQdL4O2nfvSgwVwHQUvTXCyB+npEXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753785005; c=relaxed/simple; bh=FPo8YQqOadYn1SLIdyIUhOdC1QxNA1+UX5dVUQRyf18=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=egUAz9uJLKXN1ut0Y5rfVsNcU0Hf7Ib+ZsdBnuCbZRXoK+k48d8m+1sYoEgS9Q0lPELWgzJOi8pN84LmyapprSd4rXMxDLXHq3E87bjk5sbV4eVTxDR7Tl51zwg0C0rqBfsA3vUa38jf5jr/S/YAhbggA+3kLU/Cc3O4LvEa3/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BagLhifs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BagLhifs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C8683C4CEEF; Tue, 29 Jul 2025 10:30:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753785004; bh=FPo8YQqOadYn1SLIdyIUhOdC1QxNA1+UX5dVUQRyf18=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BagLhifsVfRtRYe693MOPXaH0MDHVEcCsj/9+B5e1df1R4YGBcU7puY5o8QVWMWlO bbEimocZrrSVVvnZtezyTDHAwm7PQq+T9WTbdDS5NUCirB8//xDfGdjfhxQIZ8iz9g Ugl1Oe4veyFL9z7jKbZUMBh7YkkzUaQ8pQGTlgeX59XzZUvnzDlv8QtGOgulO+zlaD oqRokKw6yTaXTq3qJSQcwr1VXf5uBCTPxlmpXRoJvdGshwIkYqZXjELI4pZXV+TEDp oPfbVMS86b+4iNDkX6wQO57LzBqr5cql2YZmytGFmGLzIwmHvaVIShvrZgD3X/kB0L +JubbS7ROZTRg== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong Subject: [RFC 10/10] Revert "ftrace: Store direct called addresses in their ops" Date: Tue, 29 Jul 2025 12:28:13 +0200 Message-ID: <20250729102813.1531457-11-jolsa@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250729102813.1531457-1-jolsa@kernel.org> References: <20250729102813.1531457-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit dbaccb618fabde8b8596e341f8d76da63a9b0c2f. Current code uses ip address to lookup the trampoline and we need the ops to point multiple trampolines, hence this is no longer needed. TODO this probably breaks arm. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 3 --- 1 file changed, 3 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 1a61f969550d..27b26a87231c 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -457,9 +457,6 @@ struct ftrace_ops { struct list_head subop_list; ftrace_ops_func_t ops_func; struct ftrace_ops *managed; -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS - unsigned long direct_call; -#endif #endif }; =20 --=20 2.50.1