From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D74552F999F; Thu, 20 Nov 2025 21:24:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673863; cv=none; b=F5PxrUjCHBBvkjnz9jERYSpmAY9pzRuDxJNcVSpwTVh8de25AQjNV/lHdIYjVnNaiwYFMWiUbd4RaGZB/Fy3rFGyZIdXBxwfaNTZ2UDrLkZ9NgB0NwZmHDTaL90R1FTDnEq79kDo46PLk0y0JCWCWrInFpJYUKZEKUkgGSh20KQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673863; c=relaxed/simple; bh=YnnW7zN+aKxuUBmWBuWqRRKoptp3OePRxKnEM6fAzmA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P7+LMOW504p5rZ1jlN7w8KU3xVdv+HXHd+I3WADUzOtNnO2Q/N7qBRcRJfljenmf0up/2z5aopAz/mo1Rw7u7LOxJWYAcl0NqyJzA0fqsmol7wJ/nS60dP7cfXnbnKxq371bzzyhvFVlfzSFpU4ar7Gfy1WrH8uRRKOE8ZFFrTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mJiID6Fz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mJiID6Fz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 368F1C116B1; Thu, 20 Nov 2025 21:24:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673863; bh=YnnW7zN+aKxuUBmWBuWqRRKoptp3OePRxKnEM6fAzmA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mJiID6FzDqCsD96MzGN7HnAhWBymD9bFoqZ3uvJ03Iikha8ymsq+M3bqkPa8q11Jf rE6bGC3VFHdgZT7GcIrhRmCK4XYjh7ADIdjRmoOVmINk0cSxtXJiigG5OT2bPXv1Z9 LeKJLqR7nE21vmZCtB/vpKEWZe0JxaPCUj+taClupYABypOSMUTWdOKUH2vWzAt52y lMeeAnEzZIujCfouxYnmlZjGszEQTNnY61kfqdFKwbftrg57VRR8U2Ntmpzpycpql9 6c+mmxa8d+2HeREjY6284t7c58cH8gLEnK3Czs9IRsZBm5WFkxHD5eI0unHcR/HCjs RVnQuREpk3S2w== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 1/8] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Date: Thu, 20 Nov 2025 22:23:55 +0100 Message-ID: <20251120212402.466524-2-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make alloc_and_copy_ftrace_hash to copy also direct address for each hash entry. Signed-off-by: Jiri Olsa --- kernel/trace/ftrace.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 59cfacb8a5bb..0e3714b796d9 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -1186,7 +1186,7 @@ static void __add_hash_entry(struct ftrace_hash *hash, } =20 static struct ftrace_func_entry * -add_hash_entry(struct ftrace_hash *hash, unsigned long ip) +add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned= long direct) { struct ftrace_func_entry *entry; =20 @@ -1195,11 +1195,18 @@ add_hash_entry(struct ftrace_hash *hash, unsigned l= ong ip) return NULL; =20 entry->ip =3D ip; + entry->direct =3D direct; __add_hash_entry(hash, entry); =20 return entry; } =20 +static struct ftrace_func_entry * +add_hash_entry(struct ftrace_hash *hash, unsigned long ip) +{ + return add_hash_entry_direct(hash, ip, 0); +} + static void free_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry) @@ -1372,7 +1379,7 @@ alloc_and_copy_ftrace_hash(int size_bits, struct ftra= ce_hash *hash) size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { - if (add_hash_entry(new_hash, entry->ip) =3D=3D NULL) + if (add_hash_entry_direct(new_hash, entry->ip, entry->direct) =3D=3D NU= LL) goto free_hash; } } --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D04C3074B1; Thu, 20 Nov 2025 21:24:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673874; cv=none; b=qP+9pKQcIBTU7bidoAuXFrS6rWOpYCppxlAYMaw43Q/eBPzdxU8tme4VAR7MtmCCXb1hE8P/MqWCrPv/vKvYA4jtiRvxVHzUo0VLGR1YwNKkR0IjMx9FxShJjBlX4uWSq2sRFRd9YLBe5/ij2iT4IQZOZQ2e/sP+/bIpWOtHuzk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673874; c=relaxed/simple; bh=oeNb+XAyf1viA+5+4m2jeQ8qke3jBIhnjww7lRhsWv8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cU+rnLNOeTCrGxjhC9BNCDk8l6JcS81W9dFM8GnL5I/yat3eLede0NDw8vs0gZUuKw5TGMWJDK+/rGR+VvLOVXhf/2Q+1xkzBNV+R4mfCyv3s558wQP7+HFe5IiGG74v7v0dSSqC+4n4w7f+7Cqx3vvq3KlVixhCjIH0BHEN/gg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HzgihrQW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HzgihrQW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DEB15C4CEF1; Thu, 20 Nov 2025 21:24:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673874; bh=oeNb+XAyf1viA+5+4m2jeQ8qke3jBIhnjww7lRhsWv8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HzgihrQWD4WHooWmm+FJzIEL3hm6gaK8lRcUGxAH0J8CuTrGQojVLXZD7fPAO2Y6+ /LSMQLhi6SHh1HwO/08tYGsdNQIYyb9uuuE/OU3+PCUFYGHQ8RshxAGxgmzc+rRpLc 3T+eT6rHvLU4M5Z/pHBoLr+LCQUdXDGmmijD4dTWAHQ4YosUiIaGQFVK6Y7cazPdXi 5TQ0mtlyuWGDdCocmw6xH9df1xI9yPLDhKfWjzuturwoJv3eqrp/1kg3XAv+3dJs7K U0M8xoSei1I9KZ+uaZ+tCnZE+CWntUV58ZpX3o06xXftmMKpKTHbuVpMaVvGKaXrmL d/V9nhYpeaJAw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 2/8] ftrace: Export some of hash related functions Date: Thu, 20 Nov 2025 22:23:56 +0100 Message-ID: <20251120212402.466524-3-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We are going to use these functions in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 16 ++++++++++++++++ kernel/trace/ftrace.c | 7 +++---- kernel/trace/trace.h | 8 -------- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 07f8c309e432..5752553bff60 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -405,6 +405,22 @@ enum ftrace_ops_cmd { typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cm= d cmd); =20 #ifdef CONFIG_DYNAMIC_FTRACE + +#define FTRACE_HASH_DEFAULT_BITS 10 + +struct ftrace_hash { + unsigned long size_bits; + struct hlist_head *buckets; + unsigned long count; + unsigned long flags; + struct rcu_head rcu; +}; + +struct ftrace_hash *alloc_ftrace_hash(int size_bits); +void free_ftrace_hash(struct ftrace_hash *hash); +struct ftrace_func_entry *add_hash_entry_direct(struct ftrace_hash *hash, + unsigned long ip, unsigned long direct); + /* The hash used to know what functions callbacks trace */ struct ftrace_ops_hash { struct ftrace_hash __rcu *notrace_hash; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 0e3714b796d9..e6ccf572c5f6 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -68,7 +68,6 @@ }) =20 /* hash bits for specific function selection */ -#define FTRACE_HASH_DEFAULT_BITS 10 #define FTRACE_HASH_MAX_BITS 12 =20 #ifdef CONFIG_DYNAMIC_FTRACE @@ -1185,7 +1184,7 @@ static void __add_hash_entry(struct ftrace_hash *hash, hash->count++; } =20 -static struct ftrace_func_entry * +struct ftrace_func_entry * add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned= long direct) { struct ftrace_func_entry *entry; @@ -1265,7 +1264,7 @@ static void clear_ftrace_mod_list(struct list_head *h= ead) mutex_unlock(&ftrace_lock); } =20 -static void free_ftrace_hash(struct ftrace_hash *hash) +void free_ftrace_hash(struct ftrace_hash *hash) { if (!hash || hash =3D=3D EMPTY_HASH) return; @@ -1305,7 +1304,7 @@ void ftrace_free_filter(struct ftrace_ops *ops) } EXPORT_SYMBOL_GPL(ftrace_free_filter); =20 -static struct ftrace_hash *alloc_ftrace_hash(int size_bits) +struct ftrace_hash *alloc_ftrace_hash(int size_bits) { struct ftrace_hash *hash; int size; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 85eabb454bee..62e0ac625f65 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -899,14 +899,6 @@ enum { FTRACE_HASH_FL_MOD =3D (1 << 0), }; =20 -struct ftrace_hash { - unsigned long size_bits; - struct hlist_head *buckets; - unsigned long count; - unsigned long flags; - struct rcu_head rcu; -}; - struct ftrace_func_entry * ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip); =20 --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAB273074B1; Thu, 20 Nov 2025 21:24:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673884; cv=none; b=BRcX+1D5AL19uVPKIdIdcDkJ+Hfh4dBw3gB+JiERInD0HFs6wjl28ynWpmkcuZEj3D2fmNc+tXXYYCBCd2Xmu5LBE1hPtwyciqTu0caiT21muGPitazwx0AjAgMCHqZGgLwejpnW6SjjrsKLk0xD6+rcj8pqGJDdzv4wXjQR5N4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673884; c=relaxed/simple; bh=RNLSBGOz/DOmnUQjRG3FQUkHTupM2083U1onRORQjXI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TEDbFuDxRoAMZqkmVM9cfief1Gw+wchL2k9iVc37BM/LwMZm7tMiMrxxPn7ChpajqZZ3SgWgru3DwSwtoNf9kQuHfblXo/FWXzRt90V+/a3774MsYplEdwcb5M1Lvn6AU6m71GP8/V4pMRgfaurf/jYphVEJRwWuEwivzb69ET4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SWEoZaeH; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SWEoZaeH" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 61BB2C4CEF1; Thu, 20 Nov 2025 21:24:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673884; bh=RNLSBGOz/DOmnUQjRG3FQUkHTupM2083U1onRORQjXI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SWEoZaeHrw5zFW7xdxxw9Ezq7RJXLLmZc32rJRlKB4kG04794aKukSETz38/gDSn4 +kaiW0dV69nvkWPoEbtRSsH96MSXYhQzZ/W6qzY6sE4xtGoh/s/dhsP9Ox9fRw9MFA vDcSrH+iqeq5ErAOsYH/RpuLRbEUHJuuJ+faLkLqaYzsYDyRxDsWq3FTGGlsYn88q0 Klb3r23xIiBWIh+DjBiPiAIH4yUssV8EUJ6xwJ7eg3YGwMa/ljVDh6HOMDTMzrccQM FgF2ow0uXhfIyvUAkML0uu1NsmopI4CrhjslX7Wr9gSnczo09AQP+PiOyljOnkRxz8 03BPjaBEae1UA== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 3/8] ftrace: Add update_ftrace_direct_add function Date: Thu, 20 Nov 2025 22:23:57 +0100 Message-ID: <20251120212402.466524-4-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding update_ftrace_direct_add function that adds all entries (ip -> addr) provided in hash argument to direct ftrace ops and updates its attachments. The difference to current register_ftrace_direct is - hash argument that allows to register multiple ip -> direct entries at once - we can call update_ftrace_direct_add multiple times on the same ftrace_ops object, becase after first registration with register_ftrace_function_nolock, it uses ftrace_update_ops to update the ftrace_ops object This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 7 +++ kernel/trace/ftrace.c | 128 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 5752553bff60..9cf4cd56d982 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -550,6 +550,8 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, un= signed long addr, int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); =20 +int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash); + void ftrace_stub_direct_tramp(void); =20 #else @@ -576,6 +578,11 @@ static inline int modify_ftrace_direct_nolock(struct f= trace_ops *ops, unsigned l return -ENODEV; } =20 +static inline int update_ftrace_direct_add(struct ftrace_ops *ops, struct = ftrace_hash *hash) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index e6ccf572c5f6..850ff55ff25e 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6248,6 +6248,134 @@ int modify_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) return err; } EXPORT_SYMBOL_GPL(modify_ftrace_direct); + +static unsigned long hash_count(struct ftrace_hash *hash) +{ + return hash ? hash->count : 0; +} + +/** + * hash_add - adds two struct ftrace_hash and returns the result + * @a: struct ftrace_hash object + * @b: struct ftrace_hash object + * + * Returns struct ftrace_hash object on success, NULL on error. + */ +static struct ftrace_hash *hash_add(struct ftrace_hash *a, struct ftrace_h= ash *b) +{ + struct ftrace_func_entry *entry; + struct ftrace_hash *add; + int size, i; + + size =3D hash_count(a) + hash_count(b); + if (size > 32) + size =3D 32; + + add =3D alloc_and_copy_ftrace_hash(fls(size), a); + if (!add) + goto error; + + size =3D 1 << b->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &b->buckets[i], hlist) { + if (add_hash_entry_direct(add, entry->ip, entry->direct) =3D=3D NULL) + goto error; + } + } + return add; + + error: + free_ftrace_hash(add); + return NULL; +} + +int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash) +{ + struct ftrace_hash *old_direct_functions =3D NULL, *new_direct_functions = =3D NULL; + struct ftrace_hash *old_filter_hash =3D NULL, *new_filter_hash =3D NULL; + struct ftrace_func_entry *entry; + int i, size, err =3D -EINVAL; + bool reg; + + if (!hash_count(hash)) + return -EINVAL; + + mutex_lock(&direct_mutex); + + /* Make sure requested entries are not already registered. */ + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + if (__ftrace_lookup_ip(direct_functions, entry->ip)) + goto out_unlock; + } + } + + old_filter_hash =3D ops->func_hash ? ops->func_hash->filter_hash : NULL; + old_direct_functions =3D direct_functions; + + /* If there's nothing in filter_hash we need to register the ops. */ + reg =3D hash_count(old_filter_hash) =3D=3D 0; + if (reg) { + if (ops->func || ops->trampoline) + goto out_unlock; + if (ops->flags & FTRACE_OPS_FL_ENABLED) + goto out_unlock; + } + + err =3D -ENOMEM; + new_filter_hash =3D hash_add(old_filter_hash, hash); + if (!new_filter_hash) + goto out_unlock; + + new_direct_functions =3D hash_add(old_direct_functions, hash); + if (!new_direct_functions) + goto out_unlock; + + rcu_assign_pointer(direct_functions, new_direct_functions); + + if (reg) { + ops->func =3D call_direct_funcs; + ops->flags |=3D MULTI_FLAGS; + ops->trampoline =3D FTRACE_REGS_ADDR; + ops->local_hash.filter_hash =3D new_filter_hash; + + err =3D register_ftrace_function_nolock(ops); + if (err) { + /* restore old filter on error */ + ops->local_hash.filter_hash =3D old_filter_hash; + old_filter_hash =3D new_filter_hash; + + /* cleanup for possible another register call */ + ops->func =3D NULL; + ops->trampoline =3D 0; + } + } else { + err =3D ftrace_update_ops(ops, new_filter_hash, EMPTY_HASH); + /* + * new_filter_hash is dup-ed, so we need to release it anyway, + * old_filter_hash either stays on error or is released already + */ + old_filter_hash =3D new_filter_hash; + } + + if (err) { + /* reset direct_functions and free the new one */ + rcu_assign_pointer(direct_functions, old_direct_functions); + old_direct_functions =3D new_direct_functions; + } + + out_unlock: + mutex_unlock(&direct_mutex); + + if (old_direct_functions && old_direct_functions !=3D EMPTY_HASH) + call_rcu_tasks(&old_direct_functions->rcu, register_ftrace_direct_cb); + if (old_filter_hash) + free_ftrace_hash(old_filter_hash); + + return err; +} + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4725B30BBBA; Thu, 20 Nov 2025 21:24:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673895; cv=none; b=rBUSna46DJrKvvp2kJ5ZpDiA35GOKHcayAz2h/Ti1JJiMUgc5ubVvZGJa1/Lv5v6kGM2Qb4kp16P5TVdxwHYffJCnH5KmbGrg+mjBYIjsY+8N26ylYgnSODpFYdxF1VwM2QZc4FxdVIxf4xsHglu0LIQtuzp/u5WFVzPpx4NJ08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673895; c=relaxed/simple; bh=lfhp1WdaWfC5KnIhI8D/pxbcrNx0gUfpW6nQZKPGORs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OktYYs2X+hIPNr+UUfRr6pMPI3KA15zJOH7d40QXIaeq96fuPYuln/02tS5wvECTMYLllEI+FhmNgsAjnjpodXtm0561aNzt4yF91PbgG+uBkmrkg8jrJ72JkNrVbPvdSYLILoUpI++s+o7m3U+bFV5Z9a7lgcY3t6NBgM4NfMk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fW/jD4vJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fW/jD4vJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CA1EFC4CEF1; Thu, 20 Nov 2025 21:24:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673894; bh=lfhp1WdaWfC5KnIhI8D/pxbcrNx0gUfpW6nQZKPGORs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fW/jD4vJ8E6utvZI+At/2/HbVruCbI82EKwo7RX4iGEMwyc2fFr657vlgozhSDoUO nxAR/iXSnpgkCW9hB3BTdwN9XNVLRzx6hIuAIjadj2NK6glL/ZXixaBeunYqj7DBK9 4YWKjfiAzlHncZC7IrNgQObR6dPcdvPVJAIld6caF6wukWWQMLaCJhYgavRdzt1fbp ltMZd3h4XUfu/N0/UX1ZQcnGMHvZlYReP30izOvg/2DaEO0QswJy0sE2A3d9ZommN+ ZyTN+Kc0jkOEcF86BWnNnuToWjHkXfxB6hy3HLFY50BR9bG2Os2gghjl6aD7xlcmbH ZUTsMxohPgyAw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 4/8] ftrace: Add update_ftrace_direct_del function Date: Thu, 20 Nov 2025 22:23:58 +0100 Message-ID: <20251120212402.466524-5-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding update_ftrace_direct_del function that removes all entries (ip -> addr) provided in hash argument to direct ftrace ops and updates its attachments. The difference to current unregister_ftrace_direct is - hash argument that allows to unregister multiple ip -> direct entries at once - we can call update_ftrace_direct_del multiple times on the same ftrace_ops object, becase we do not need to unregister all entries at once, we can do it gradualy with the help of ftrace_update_ops function This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 +++ kernel/trace/ftrace.c | 110 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 9cf4cd56d982..c571deeff840 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -551,6 +551,7 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr); int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); =20 int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash); +int update_ftrace_direct_del(struct ftrace_ops *ops, struct ftrace_hash *h= ash); =20 void ftrace_stub_direct_tramp(void); =20 @@ -583,6 +584,11 @@ static inline int update_ftrace_direct_add(struct ftra= ce_ops *ops, struct ftrace return -ENODEV; } =20 +static inline int update_ftrace_direct_del(struct ftrace_ops *ops, struct = ftrace_hash *hash) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 850ff55ff25e..cc730a8fd087 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6376,6 +6376,116 @@ int update_ftrace_direct_add(struct ftrace_ops *ops= , struct ftrace_hash *hash) return err; } =20 +/** + * hash_sub - substracts @b from @a and returns the result + * @a: struct ftrace_hash object + * @b: struct ftrace_hash object + * + * Returns struct ftrace_hash object on success, NULL on error. + */ +static struct ftrace_hash *hash_sub(struct ftrace_hash *a, struct ftrace_h= ash *b) +{ + struct ftrace_func_entry *entry, *del; + struct ftrace_hash *sub; + int size, i; + + sub =3D alloc_and_copy_ftrace_hash(a->size_bits, a); + if (!sub) + goto error; + + size =3D 1 << b->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &b->buckets[i], hlist) { + del =3D __ftrace_lookup_ip(sub, entry->ip); + if (WARN_ON_ONCE(!del)) + goto error; + remove_hash_entry(sub, del); + kfree(del); + } + } + return sub; + + error: + free_ftrace_hash(sub); + return NULL; +} + +int update_ftrace_direct_del(struct ftrace_ops *ops, struct ftrace_hash *h= ash) +{ + struct ftrace_hash *old_direct_functions =3D NULL, *new_direct_functions = =3D NULL; + struct ftrace_hash *old_filter_hash =3D NULL, *new_filter_hash =3D NULL; + struct ftrace_func_entry *del, *entry; + unsigned long size, i; + int err =3D -EINVAL; + + if (!hash_count(hash)) + return -EINVAL; + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + if (direct_functions =3D=3D EMPTY_HASH) + return -EINVAL; + + mutex_lock(&direct_mutex); + + old_filter_hash =3D ops->func_hash ? ops->func_hash->filter_hash : NULL; + old_direct_functions =3D direct_functions; + + /* Make sure requested entries are already registered. */ + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + del =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (!del || del->direct !=3D entry->direct) + goto out_unlock; + } + } + + err =3D -ENOMEM; + new_filter_hash =3D hash_sub(old_filter_hash, hash); + if (!new_filter_hash) + goto out_unlock; + + new_direct_functions =3D hash_sub(old_direct_functions, hash); + if (!new_direct_functions) + goto out_unlock; + + /* If there's nothing left, we need to unregister the ops. */ + if (ftrace_hash_empty(new_filter_hash)) { + err =3D unregister_ftrace_function(ops); + if (!err) { + /* cleanup for possible another register call */ + ops->func =3D NULL; + ops->trampoline =3D 0; + ftrace_free_filter(ops); + ops->func_hash->filter_hash =3D NULL; + } + } else { + err =3D ftrace_update_ops(ops, new_filter_hash, EMPTY_HASH); + /* + * new_filter_hash is dup-ed, so we need to release it anyway, + * old_filter_hash either stays on error or is released already + */ + } + + if (err) { + /* reset direct_functions and free the new one */ + old_direct_functions =3D new_direct_functions; + } else { + rcu_assign_pointer(direct_functions, new_direct_functions); + } + + out_unlock: + mutex_unlock(&direct_mutex); + + if (old_direct_functions && old_direct_functions !=3D EMPTY_HASH) + call_rcu_tasks(&old_direct_functions->rcu, register_ftrace_direct_cb); + free_ftrace_hash(new_filter_hash); + + return err; +} + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 952A92F12CD; Thu, 20 Nov 2025 21:25:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673905; cv=none; b=S8lKJZU3HwYIy8gZcMAsDaBd6pO7iXrS2xgLkvAR6LbfHRjtEzXWHlJV9KuB4ycArh8OQRtjJKDzA11nrW7SrvJULcjlhzyAk0C39HrVwpYr18d88wAl5ZgkkK206hXyZyx2+1hZw6J9NkYHdX5M6yc3ZmPkUiW63lF30kDJeb8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673905; c=relaxed/simple; bh=Xs4AEFpvDY6JzCUFXkOpJeETRBiY34vm/qu0XZbAVcM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sFOeCcV11DI/nm9inkO9vJmcqKKStra/mW0zy7A5/dAhPiPDQhe2lgPKIcvr7IingrpyQL6RXjgAlGhPSpDYK1lsJbarVfhMI3igJAYRty7LRu8q465JXhdHUm/Fqz7U9fxEVWirOPX5ZH4yceGI/DQqlj8FiebaPxjHPReAf5E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tGU9vA0u; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tGU9vA0u" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10B4DC4CEF1; Thu, 20 Nov 2025 21:25:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673905; bh=Xs4AEFpvDY6JzCUFXkOpJeETRBiY34vm/qu0XZbAVcM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tGU9vA0ucsFWp6B9XXnY7o7XV791mPu6UI5szkwx4hc5qI4Ar7mkhIRSUqfHcfWLw WnyL2dfOIy7ul01z8SglOlCdK0BGZRvpqJoU54BdFa0Q2PIPhuM1JR7EsvAI92A5av FxJkl5TpFwECRDlW1t8ARXeNNkjehxMCh2PeMj7P/dDSIKoztchSBJse5w2EN/ZLym j68/uoJcKgzVGhGO25iu2VlMeCZ7GpgNSCojS7IJO1eIhLIKtauyBFVJ4+g3qhh2D2 D36tKfXrrYv/bPlcVgn3C47WYTGeX0GrI5ngPKba7e5YnfftsvAMWNhbs93w8F452J neOuUbAWEmc5g== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 5/8] ftrace: Add update_ftrace_direct_mod function Date: Thu, 20 Nov 2025 22:23:59 +0100 Message-ID: <20251120212402.466524-6-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding update_ftrace_direct_mod function that modifies all entries (ip -> direct) provided in hash argument to direct ftrace ops and updates its attachments. The difference to current modify_ftrace_direct is: - hash argument that allows to modify multiple ip -> direct entries at once This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 ++++ kernel/trace/ftrace.c | 68 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index c571deeff840..4e2c0ed769fc 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -552,6 +552,7 @@ int modify_ftrace_direct_nolock(struct ftrace_ops *ops,= unsigned long addr); =20 int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash); int update_ftrace_direct_del(struct ftrace_ops *ops, struct ftrace_hash *h= ash); +int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *h= ash, bool do_direct_lock); =20 void ftrace_stub_direct_tramp(void); =20 @@ -589,6 +590,11 @@ static inline int update_ftrace_direct_del(struct ftra= ce_ops *ops, struct ftrace return -ENODEV; } =20 +static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct = ftrace_hash *hash, bool do_direct_lock) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index cc730a8fd087..5243aefb6096 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6486,6 +6486,74 @@ int update_ftrace_direct_del(struct ftrace_ops *ops,= struct ftrace_hash *hash) return err; } =20 +int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *h= ash, bool do_direct_lock) +{ + struct ftrace_hash *orig_hash =3D ops->func_hash->filter_hash; + struct ftrace_func_entry *entry, *tmp; + static struct ftrace_ops tmp_ops =3D { + .func =3D ftrace_stub, + .flags =3D FTRACE_OPS_FL_STUB, + }; + unsigned long size, i; + int err; + + if (!hash_count(hash)) + return -EINVAL; + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + if (direct_functions =3D=3D EMPTY_HASH) + return -EINVAL; + + if (do_direct_lock) + mutex_lock(&direct_mutex); + + /* Enable the tmp_ops to have the same functions as the direct ops */ + ftrace_ops_init(&tmp_ops); + tmp_ops.func_hash =3D ops->func_hash; + + err =3D register_ftrace_function_nolock(&tmp_ops); + if (err) + goto unlock; + + /* + * Call __ftrace_hash_update_ipmodify() here, so that we can call + * ops->ops_func for the ops. This is needed because the above + * register_ftrace_function_nolock() worked on tmp_ops. + */ + err =3D __ftrace_hash_update_ipmodify(ops, orig_hash, orig_hash, true); + if (err) + goto out; + + /* + * Now the ftrace_ops_list_func() is called to do the direct callers. + * We can safely change the direct functions attached to each entry. + */ + mutex_lock(&ftrace_lock); + + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + tmp =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (!tmp) + continue; + tmp->direct =3D entry->direct; + } + } + + mutex_unlock(&ftrace_lock); + +out: + /* Removing the tmp_ops will add the updated direct callers to the functi= ons */ + unregister_ftrace_function(&tmp_ops); + +unlock: + if (do_direct_lock) + mutex_unlock(&direct_mutex); + return err; +} + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB384309EE0; Thu, 20 Nov 2025 21:25:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673916; cv=none; b=obINJj6a5JrrmS43tfysst80Ufui4EfCo6gXix/go5Zw9sOHSdA7iOgQqGRvMzuL0pqNro4mYYKvehPCqsl4JsMcRY5rFSPyo8MG/AFa5iZ0qvaiuDQqoylDeWAXd1ZJnnuBQ480+nohKyHTFTyjD4EJ9nyA5Z0uB49eTRZydzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673916; c=relaxed/simple; bh=iykBhG4b2o2xbV0/ykgJE+6aevyvXReLas2ZH30xEX4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UxuLSHUymxYHs2y+f4dbUMmXwRx8tvhQumTYEtn8lqOsb8ZPh8yF7Odj074O07UqsYfbR80x72wCmTcPDUOcQf/0howBnJV+6vOlsvLlUDv4EfuCcp7Y11iBjdl16NKRO6ayi7NY37WzB6/NsyIheUtW0STTdEpS2BWzJxFH87g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=C/LXjUSJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="C/LXjUSJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 35E85C4CEF1; Thu, 20 Nov 2025 21:25:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673915; bh=iykBhG4b2o2xbV0/ykgJE+6aevyvXReLas2ZH30xEX4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C/LXjUSJDXrV3FKgcr7yr/lvJ7wQMpxSVnZfG2VfhpUDczFVtjgmFIQhWaHKJlLui wqeyBeLajzWJK926gPWVX14J97ZvKfL55wNtYIZ/iAkzS4VZH3QxdGf/d/J7fo3qFT p7yjX8xo8+5alfByJ+V0ilgrCQjbAxDdg6U0DU0Y5HAKr5N/U8saYygOBkExOBzX3q nmdZNlCWvmNOs9nyZcc8V1u+iTYNf/owlWILQLzxCQ32U2wPh5IoRAJysGZb7uRY8p xlja37iv4AnQeWYbyipsy1RiudlSs0wEqUks4XmyrY4527liBqYZ2kMo/cvENntCtS xbfafu2gPrsSQ== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 6/8] bpf: Add trampoline ip hash table Date: Thu, 20 Nov 2025 22:24:00 +0100 Message-ID: <20251120212402.466524-7-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Following changes need to lookup trampoline based on its ip address, adding hash table for that. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 7 +++++-- kernel/bpf/trampoline.c | 30 +++++++++++++++++++----------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 09d5dc541d1c..ad86857a8562 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1295,14 +1295,17 @@ struct bpf_tramp_image { }; =20 struct bpf_trampoline { - /* hlist for trampoline_table */ - struct hlist_node hlist; + /* hlist for trampoline_key_table */ + struct hlist_node hlist_key; + /* hlist for trampoline_ip_table */ + struct hlist_node hlist_ip; struct ftrace_ops *fops; /* serializes access to fields of this trampoline */ struct mutex mutex; refcount_t refcnt; u32 flags; u64 key; + unsigned long ip; struct { struct btf_func_model model; void *addr; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 04104397c432..f33693061c2e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -24,9 +24,10 @@ const struct bpf_prog_ops bpf_extension_prog_ops =3D { #define TRAMPOLINE_HASH_BITS 10 #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS) =20 -static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE]; +static struct hlist_head trampoline_key_table[TRAMPOLINE_TABLE_SIZE]; +static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; =20 -/* serializes access to trampoline_table */ +/* serializes access to trampoline tables */ static DEFINE_MUTEX(trampoline_mutex); =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS @@ -135,15 +136,15 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } =20 -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long= ip) { struct bpf_trampoline *tr; struct hlist_head *head; int i; =20 mutex_lock(&trampoline_mutex); - head =3D &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; - hlist_for_each_entry(tr, head, hlist) { + head =3D &trampoline_key_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head, hlist_key) { if (tr->key =3D=3D key) { refcount_inc(&tr->refcnt); goto out; @@ -164,8 +165,12 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u6= 4 key) #endif =20 tr->key =3D key; - INIT_HLIST_NODE(&tr->hlist); - hlist_add_head(&tr->hlist, head); + tr->ip =3D ftrace_location(ip); + INIT_HLIST_NODE(&tr->hlist_key); + INIT_HLIST_NODE(&tr->hlist_ip); + hlist_add_head(&tr->hlist_key, head); + head =3D &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)]; + hlist_add_head(&tr->hlist_ip, head); refcount_set(&tr->refcnt, 1); mutex_init(&tr->mutex); for (i =3D 0; i < BPF_TRAMP_MAX; i++) @@ -801,7 +806,7 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog = *prog) prog->aux->attach_btf_id); =20 bpf_lsm_find_cgroup_shim(prog, &bpf_func); - tr =3D bpf_trampoline_lookup(key); + tr =3D bpf_trampoline_lookup(key, 0); if (WARN_ON_ONCE(!tr)) return; =20 @@ -821,7 +826,7 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, { struct bpf_trampoline *tr; =20 - tr =3D bpf_trampoline_lookup(key); + tr =3D bpf_trampoline_lookup(key, tgt_info->tgt_addr); if (!tr) return NULL; =20 @@ -857,7 +862,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) * fexit progs. The fentry-only trampoline will be freed via * multiple rcu callbacks. */ - hlist_del(&tr->hlist); + hlist_del(&tr->hlist_key); + hlist_del(&tr->hlist_ip); if (tr->fops) { ftrace_free_filter(tr->fops); kfree(tr->fops); @@ -1130,7 +1136,9 @@ static int __init init_trampolines(void) int i; =20 for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) - INIT_HLIST_HEAD(&trampoline_table[i]); + INIT_HLIST_HEAD(&trampoline_key_table[i]); + for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) + INIT_HLIST_HEAD(&trampoline_ip_table[i]); return 0; } late_initcall(init_trampolines); --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 607922ED16B; Thu, 20 Nov 2025 21:25:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673926; cv=none; b=SfoUKpxOeq2LtUQNnn32JREGxJ2DRWbFFkK7muHbIRqi3WacmiA2L4s/c/ulBHp7DUvz4YsNEEdfKnIOTlBjjSXmmuvbLjGLgbr4rJXnwiZSCtOkDwqfGBq7DN3ViboAdKDJ7FJegW8qxCnWOIGiVSqLjkQsgg0E/RlYimzoxIQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673926; c=relaxed/simple; bh=Mqo7EJXWsx+FIdks6hO54RzctmDz89Zob5j/Foh8C5s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qZHXEG3wKEG0u87E7uqJzkUzAI0G1L9xJ5a4Isd3VjfrS1yfePESoK2vW5GeD3kmgRgnG0EaLvSLEUKMmOjgoR+F4hLdZWuwu7g+DS1XVLfPqidh0ioV0O7KP0EPrvsxQhGnkI1VZtgQRE09+TasIz5a8B72fxbBfDc1BT6wr3Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XRg5qdDt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XRg5qdDt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2804AC4CEF1; Thu, 20 Nov 2025 21:25:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673926; bh=Mqo7EJXWsx+FIdks6hO54RzctmDz89Zob5j/Foh8C5s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XRg5qdDtc1bbMIuHUS7oV7q4fVldJSAs/eOXXb7KurtnJxPv9/edGgaZ64jo9BdDF Qd9z8LAa63AqLrYxBJp+DilczhZFAtY8dQItBTOQs1fNHBF9cFXQaHJUnWGWr0v9CM izWVLofNcXUz4w5M01+Uwly/Xg6dT1RsyRb8Bx4rRhI1Za0tDaP+lMySoTKAafggbV 1wssL8k2PGjPO7L4FlyPCNz5Xn1h/y5rPJDFzuntWQJJivlsOffHRQqoW/VH9KoAR4 qVrd3Xn7neGlbOF8alonNIyewGVs3p+4zhB3GwTPMQ5AAzwEX2591hbqMSefZ3LeXy KXI/zGrCUKnGw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 7/8] ftrace: Factor ftrace_ops ops_func interface Date: Thu, 20 Nov 2025 22:24:01 +0100 Message-ID: <20251120212402.466524-8-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We are going to remove "ftrace_ops->private =3D=3D bpf_trampoline" setup in following changes. Adding ip argument to ftrace_ops_func_t callback function, so we can use it to look up the trampoline. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 2 +- kernel/bpf/trampoline.c | 3 ++- kernel/trace/ftrace.c | 6 +++--- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 4e2c0ed769fc..5aa37dec0081 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -402,7 +402,7 @@ enum ftrace_ops_cmd { * Negative on failure. The return value is dependent on the * callback. */ -typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cm= d cmd); +typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, unsigned long ip, = enum ftrace_ops_cmd cmd); =20 #ifdef CONFIG_DYNAMIC_FTRACE =20 diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index f33693061c2e..d8762f7990ed 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -33,7 +33,8 @@ static DEFINE_MUTEX(trampoline_mutex); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_dire= ct_mutex); =20 -static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, enum ftrace_o= ps_cmd cmd) +static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long= ip, + enum ftrace_ops_cmd cmd) { struct bpf_trampoline *tr =3D ops->private; int ret =3D 0; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 5243aefb6096..b0e4d7a5d47c 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2049,7 +2049,7 @@ static int __ftrace_hash_update_ipmodify(struct ftrac= e_ops *ops, */ if (!ops->ops_func) return -EBUSY; - ret =3D ops->ops_func(ops, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF); + ret =3D ops->ops_func(ops, rec->ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODI= FY_SELF); if (ret) return ret; } else if (is_ipmodify) { @@ -8976,7 +8976,7 @@ static int prepare_direct_functions_for_ipmodify(stru= ct ftrace_ops *ops) if (!op->ops_func) return -EBUSY; =20 - ret =3D op->ops_func(op, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER); + ret =3D op->ops_func(op, ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER= ); if (ret) return ret; } @@ -9023,7 +9023,7 @@ static void cleanup_direct_functions_after_ipmodify(s= truct ftrace_ops *ops) =20 /* The cleanup is optional, ignore any errors */ if (found_op && op->ops_func) - op->ops_func(op, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER); + op->ops_func(op, ip, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER); } } mutex_unlock(&direct_mutex); --=20 2.51.1 From nobody Tue Dec 2 01:51:40 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E16963242A1; Thu, 20 Nov 2025 21:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673937; cv=none; b=KvN1gsYXjuR3m3ZM6iorStMgmrrYz5gACJRnc1SUwjJVc1o/J39XGl0N0YfJ2Gp5l6ObTRyE2FncLQuJsyJ1onQdwRPsVPUmcRWSvDmMw+2p04jyx4ptgE8WVWnld7zH2jbqZcwrg07h/TlDtQpb1Jet543l9Vcu+sw53d5XaUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763673937; c=relaxed/simple; bh=tWsIX28eUU09Y+4XW8T9/1bbDivP+CjAgctxdhSf0nM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J4NgM5pkF+jWujHc9ATYpQcHDNrolWwu3/IAvOHPSrWmPLRJV4yyOX4XGdFUH6qbmHymBQ3/GvTU3NQjPM5d07YalMmwnfh2Uje56PZdKnY65582mSsP6Qfx62rqhHuROYr3E8tO2XX/AGAj/Te33hdq/vXQGkrdOSeQKhjWHLw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G5bdWYXb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G5bdWYXb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 940F6C4CEF1; Thu, 20 Nov 2025 21:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763673936; bh=tWsIX28eUU09Y+4XW8T9/1bbDivP+CjAgctxdhSf0nM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G5bdWYXbUX7EtiNCcmeVlhB9VsPW+pdJ9kuu2dZxAM15q+VCh9jqvzzI3YvzYjLV7 WyY5d3z1KexwMY14vA19/UaPPX15FsSDX6FpNXEsDEYCyG/sQDY3zicRRnWS1pLzhO aa2Prrs92VBCtpZ5ZYadyKGTB1kn2Kdq5FDNueFlsU3swjzCOtWaAMhcorraHx1kUa xqZnOLkg3uejk5h7l1ZbwWY7UqtphvcIemA1lUQRCHV7vCxoLeBbJpI5ROaqleSApi fr0/L4gigkYqmCUnAguD0B8uXObigT8WS7R/VGoIxChs3B0sKAU+g4gqY0YVpIl8y1 DJ9a8t03RKUGA== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv3 bpf-next 8/8] bpf, x86: Use single ftrace_ops for direct calls Date: Thu, 20 Nov 2025 22:24:02 +0100 Message-ID: <20251120212402.466524-9-jolsa@kernel.org> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251120212402.466524-1-jolsa@kernel.org> References: <20251120212402.466524-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Using single ftrace_ops for direct calls update instead of allocating ftrace_ops object for each trampoline. With single ftrace_ops object we can use update_ftrace_direct_* api that allows multiple ip sites updates on single ftrace_ops object. Adding HAVE_SINGLE_FTRACE_DIRECT_OPS config option to be enabled on each arch that supports this. At the moment we can enable this only on x86 arch, because arm relies on ftrace_ops object representing just single trampoline image (stored in ftrace_ops::direct_call). Ach that do not support this will continue to use *_ftrace_direct api. Signed-off-by: Jiri Olsa --- arch/x86/Kconfig | 1 + kernel/bpf/trampoline.c | 172 +++++++++++++++++++++++++++++++++++----- kernel/trace/Kconfig | 3 + kernel/trace/ftrace.c | 7 +- 4 files changed, 164 insertions(+), 19 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fa3b616af03a..65a2fc279b46 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -332,6 +332,7 @@ config X86 select SCHED_SMT if SMP select ARCH_SUPPORTS_SCHED_CLUSTER if SMP select ARCH_SUPPORTS_SCHED_MC if SMP + select HAVE_SINGLE_FTRACE_DIRECT_OPS if X86_64 && DYNAMIC_FTRACE_WITH_DIR= ECT_CALLS =20 config INSTRUCTION_DECODER def_bool y diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d8762f7990ed..9cc4de215df7 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -33,12 +33,40 @@ static DEFINE_MUTEX(trampoline_mutex); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_dire= ct_mutex); =20 +#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS +static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops,= unsigned long ip) +{ + struct hlist_head *head_ip; + struct bpf_trampoline *tr; + + mutex_lock(&trampoline_mutex); + head_ip =3D &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head_ip, hlist_ip) { + if (tr->ip =3D=3D ip) + goto out; + } + tr =3D NULL; +out: + mutex_unlock(&trampoline_mutex); + return tr; +} +#else +static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops,= unsigned long ip) +{ + return ops->private; +} +#endif /* CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */ + static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long= ip, enum ftrace_ops_cmd cmd) { - struct bpf_trampoline *tr =3D ops->private; + struct bpf_trampoline *tr; int ret =3D 0; =20 + tr =3D direct_ops_ip_lookup(ops, ip); + if (!tr) + return -EINVAL; + if (cmd =3D=3D FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { /* This is called inside register_ftrace_direct_multi(), so * tr->mutex is already locked. @@ -137,6 +165,126 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS +#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS +/* + * We have only single direct_ops which contains all the direct call + * sites and is the only global ftrace_ops for all trampolines. + * + * We use 'update_ftrace_direct_*' api for attachment. + */ +struct ftrace_ops direct_ops =3D { + .ops_func =3D bpf_tramp_ftrace_ops_func, +}; + +static int direct_ops_alloc(struct bpf_trampoline *tr) +{ + tr->fops =3D &direct_ops; + return 0; +} + +static void direct_ops_free(struct bpf_trampoline *tr) { } + +static struct ftrace_hash *hash_from(unsigned long ip, void *addr) +{ + struct ftrace_hash *hash; + + ip =3D ftrace_location(ip); + if (!ip) + return NULL; + hash =3D alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); + if (!hash) + return NULL; + if (!add_hash_entry_direct(hash, ip, (unsigned long) addr)) { + free_ftrace_hash(hash); + return NULL; + } + return hash; +} + +static int direct_ops_add(struct ftrace_ops *ops, unsigned long ip, void *= addr) +{ + struct ftrace_hash *hash =3D hash_from(ip, addr); + int err =3D -ENOMEM; + + if (hash) + err =3D update_ftrace_direct_add(ops, hash); + free_ftrace_hash(hash); + return err; +} + +static int direct_ops_del(struct ftrace_ops *ops, unsigned long ip, void *= addr) +{ + struct ftrace_hash *hash =3D hash_from(ip, addr); + int err =3D -ENOMEM; + + if (hash) + err =3D update_ftrace_direct_del(ops, hash); + free_ftrace_hash(hash); + return err; +} + +static int direct_ops_mod(struct ftrace_ops *ops, unsigned long ip, void *= addr, bool lock_direct_mutex) +{ + struct ftrace_hash *hash =3D hash_from(ip, addr); + int err =3D -ENOMEM; + + if (hash) + err =3D update_ftrace_direct_mod(ops, hash, lock_direct_mutex); + free_ftrace_hash(hash); + return err; +} +#else +/* + * We allocate ftrace_ops object for each trampoline and it contains + * call site specific for that trampoline. + * + * We use *_ftrace_direct api for attachment. + */ +static int direct_ops_alloc(struct bpf_trampoline *tr) +{ + tr->fops =3D kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL); + if (!tr->fops) + return -ENOMEM; + tr->fops->private =3D tr; + tr->fops->ops_func =3D bpf_tramp_ftrace_ops_func; + return 0; +} + +static void direct_ops_free(struct bpf_trampoline *tr) +{ + if (tr->fops) { + ftrace_free_filter(tr->fops); + kfree(tr->fops); + } +} + +static int direct_ops_add(struct ftrace_ops *ops, unsigned long ip, void *= addr) +{ + int ret; + + ret =3D ftrace_set_filter_ip(ops, (unsigned long)ip, 0, 1); + if (ret) + return ret; + return register_ftrace_direct(ops, (long)addr); +} + +static int direct_ops_del(struct ftrace_ops *ops, unsigned long ip, void *= addr) +{ + return unregister_ftrace_direct(ops, (long)addr, false); +} + +static int direct_ops_mod(struct ftrace_ops *ops, unsigned long ip, void *= addr, bool lock_direct_mutex) +{ + if (lock_direct_mutex) + return modify_ftrace_direct(ops, (long)addr); + return modify_ftrace_direct_nolock(ops, (long)addr); +} +#endif /* CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */ +#else +static void direct_ops_free(struct bpf_trampoline *tr) { } +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ + static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long= ip) { struct bpf_trampoline *tr; @@ -155,14 +303,11 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u= 64 key, unsigned long ip) if (!tr) goto out; #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS - tr->fops =3D kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL); - if (!tr->fops) { + if (direct_ops_alloc(tr)) { kfree(tr); tr =3D NULL; goto out; } - tr->fops->private =3D tr; - tr->fops->ops_func =3D bpf_tramp_ftrace_ops_func; #endif =20 tr->key =3D key; @@ -187,7 +332,7 @@ static int unregister_fentry(struct bpf_trampoline *tr,= void *old_addr) int ret; =20 if (tr->func.ftrace_managed) - ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); + ret =3D direct_ops_del(tr->fops, tr->ip, old_addr); else ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); =20 @@ -201,10 +346,7 @@ static int modify_fentry(struct bpf_trampoline *tr, vo= id *old_addr, void *new_ad int ret; =20 if (tr->func.ftrace_managed) { - if (lock_direct_mutex) - ret =3D modify_ftrace_direct(tr->fops, (long)new_addr); - else - ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); + ret =3D direct_ops_mod(tr->fops, tr->ip, new_addr, lock_direct_mutex); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); } @@ -226,10 +368,7 @@ static int register_fentry(struct bpf_trampoline *tr, = void *new_addr) } =20 if (tr->func.ftrace_managed) { - ret =3D ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); - if (ret) - return ret; - ret =3D register_ftrace_direct(tr->fops, (long)new_addr); + ret =3D direct_ops_add(tr->fops, tr->ip, new_addr); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); } @@ -865,10 +1004,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) */ hlist_del(&tr->hlist_key); hlist_del(&tr->hlist_ip); - if (tr->fops) { - ftrace_free_filter(tr->fops); - kfree(tr->fops); - } + direct_ops_free(tr); kfree(tr); out: mutex_unlock(&trampoline_mutex); diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index d2c79da81e4f..4bf5beb04a5b 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -50,6 +50,9 @@ config HAVE_DYNAMIC_FTRACE_WITH_REGS config HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS bool =20 +config HAVE_SINGLE_FTRACE_DIRECT_OPS + bool + config HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS bool =20 diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index b0e4d7a5d47c..4380839d2422 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2605,8 +2605,13 @@ unsigned long ftrace_find_rec_direct(unsigned long i= p) static void call_direct_funcs(unsigned long ip, unsigned long pip, struct ftrace_ops *ops, struct ftrace_regs *fregs) { - unsigned long addr =3D READ_ONCE(ops->direct_call); + unsigned long addr; =20 +#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS + addr =3D ftrace_find_rec_direct(ip); +#else + addr =3D READ_ONCE(ops->direct_call); +#endif if (!addr) return; =20 --=20 2.51.1