From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C9971AA7A6; Wed, 3 Dec 2025 08:24:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750264; cv=none; b=WW7VdbXEBV/48gvf2mEpQUNrdyW/nemBtr8mebP6mA20mvBHAD79R3rkRGC2B4g+1AW91bTwniElTjXNJ9c/4/19EroFPkChABBt8me7UELyLENQambKLmHm1FbmXzokPnB2U5eF0QC+19yllMgF8mr5EIZO1r9lVCpVLqCkaIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750264; c=relaxed/simple; bh=ex4gsm1cH3NpvQRTYPCxl+CnhL7/Pm9OyE43k+dDVsg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eHwxmmX3FxmGgZqSMrGeghH3E/QhDUpDXoPYibRg7lWKd4XV9YHqXAOe13VIYUFITge6gmd+8sRdvVnti39BwbCGw4Sjetd5JzMr8943WRXxZGMyUbyU2GMqN4IhvKCfIHIVlGmmJ2wN1+CnCSQABvPfLF1aZDwLw4FLWiJFtIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ilS3U8O3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ilS3U8O3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 73FC8C116B1; Wed, 3 Dec 2025 08:24:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750263; bh=ex4gsm1cH3NpvQRTYPCxl+CnhL7/Pm9OyE43k+dDVsg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ilS3U8O3RDoKXs6nlHrLlOF3bjXstn+qMXssRebXTtPw7kCZjfB+EdUpQre2sNmJS xIEWTUk3OYkHV9go/mUyRBqbUaFYuqaz8XEpBda7CSBLvwfdGpbrPNjRVeBEnwHEj2 qZgc4V3W19NfHaN3WKbxUKqu4fRlRwmJHKHkhuNBbZ6gANmo8Jm3wYzPcow8YZr/Ds gECmaU73VaXqJO2t8QymQ5BASdhtkUCEzFdq8rqUBYKXgibauDjywtF4XnpIhV8m9v m+zUOI+QSY/t3c/qcziF52NeX7MFq8Wo/0ChK5m/X0+lIuOKuwD30TiMoBpbRq/hWC 5Sn7eJOVyiiCw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 1/9] ftrace,bpf: Remove FTRACE_OPS_FL_JMP ftrace_ops flag Date: Wed, 3 Dec 2025 09:23:54 +0100 Message-ID: <20251203082402.78816-2-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable At the moment the we allow the jmp attach only for ftrace_ops that has FTRACE_OPS_FL_JMP set. This conflicts with following changes where we use single ftrace_ops object for all direct call sites, so all could be be attached via just call or jmp. We already limit the jmp attach support with config option and bit (LSB) set on the trampoline address. It turns out that's actually enough to limit the jmp attach for architecture and only for chosen addresses (with LSB bit set). Each user of register_ftrace_direct or modify_ftrace_direct can set the trampoline bit (LSB) to indicate it has to be attached by jmp. The bpf trampoline generation code uses trampoline flags to generate jmp-attach specific code and ftrace inner code uses the trampoline bit (LSB) to handle return from jmp attachment, so there's no harm to remove the FTRACE_OPS_FL_JMP bit. The fexit/fmodret performance stays the same (did not drop), current code: fentry : 77.904 =C2=B1 0.546M/s fexit : 62.430 =C2=B1 0.554M/s fmodret : 66.503 =C2=B1 0.902M/s with this change: fentry : 80.472 =C2=B1 0.061M/s fexit : 63.995 =C2=B1 0.127M/s fmodret : 67.362 =C2=B1 0.175M/s Fixes: 25e4e3565d45 ("ftrace: Introduce FTRACE_OPS_FL_JMP") Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 1 - kernel/bpf/trampoline.c | 32 ++++++++++++++------------------ kernel/trace/ftrace.c | 14 -------------- 3 files changed, 14 insertions(+), 33 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 015dd1049bea..505b7d3f5641 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -359,7 +359,6 @@ enum { FTRACE_OPS_FL_DIRECT =3D BIT(17), FTRACE_OPS_FL_SUBOP =3D BIT(18), FTRACE_OPS_FL_GRAPH =3D BIT(19), - FTRACE_OPS_FL_JMP =3D BIT(20), }; =20 #ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 976d89011b15..b9a358d7a78f 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -214,10 +214,15 @@ static int modify_fentry(struct bpf_trampoline *tr, u= 32 orig_flags, int ret; =20 if (tr->func.ftrace_managed) { + unsigned long addr =3D (unsigned long) new_addr; + + if (bpf_trampoline_use_jmp(tr->flags)) + addr =3D ftrace_jmp_set(addr); + if (lock_direct_mutex) - ret =3D modify_ftrace_direct(tr->fops, (long)new_addr); + ret =3D modify_ftrace_direct(tr->fops, addr); else - ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); + ret =3D modify_ftrace_direct_nolock(tr->fops, addr); } else { ret =3D bpf_trampoline_update_fentry(tr, orig_flags, old_addr, new_addr); @@ -240,10 +245,15 @@ static int register_fentry(struct bpf_trampoline *tr,= void *new_addr) } =20 if (tr->func.ftrace_managed) { + unsigned long addr =3D (unsigned long) new_addr; + + if (bpf_trampoline_use_jmp(tr->flags)) + addr =3D ftrace_jmp_set(addr); + ret =3D ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); if (ret) return ret; - ret =3D register_ftrace_direct(tr->fops, (long)new_addr); + ret =3D register_ftrace_direct(tr->fops, addr); } else { ret =3D bpf_trampoline_update_fentry(tr, 0, NULL, new_addr); } @@ -499,13 +509,6 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut if (err) goto out_free; =20 -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP - if (bpf_trampoline_use_jmp(tr->flags)) - tr->fops->flags |=3D FTRACE_OPS_FL_JMP; - else - tr->fops->flags &=3D ~FTRACE_OPS_FL_JMP; -#endif - WARN_ON(tr->cur_image && total =3D=3D 0); if (tr->cur_image) /* progs already running at this address */ @@ -533,15 +536,8 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut tr->cur_image =3D im; out: /* If any error happens, restore previous flags */ - if (err) { + if (err) tr->flags =3D orig_flags; -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP - if (bpf_trampoline_use_jmp(tr->flags)) - tr->fops->flags |=3D FTRACE_OPS_FL_JMP; - else - tr->fops->flags &=3D ~FTRACE_OPS_FL_JMP; -#endif - } kfree(tlinks); return err; =20 diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index bbb37c0f8c6c..b0dc911411f1 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6017,15 +6017,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, u= nsigned long addr) if (ftrace_hash_empty(hash)) return -EINVAL; =20 - /* This is a "raw" address, and this should never happen. */ - if (WARN_ON_ONCE(ftrace_is_jmp(addr))) - return -EINVAL; - mutex_lock(&direct_mutex); =20 - if (ops->flags & FTRACE_OPS_FL_JMP) - addr =3D ftrace_jmp_set(addr); - /* Make sure requested entries are not already registered.. */ size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { @@ -6146,13 +6139,6 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsig= ned long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 - /* This is a "raw" address, and this should never happen. */ - if (WARN_ON_ONCE(ftrace_is_jmp(addr))) - return -EINVAL; - - if (ops->flags & FTRACE_OPS_FL_JMP) - addr =3D ftrace_jmp_set(addr); - /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E76329B764; Wed, 3 Dec 2025 08:24:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750274; cv=none; b=BbR/pDHxOcHvuUry26MeDDKXrjpKBlLHQM1LMg21W3F3GBisdgko6VmMh/oUDNMVy0UI2A7xHJGGFbUagMl2vEWMZ793M5hSTE2ErK63xrd4rDCJh1VeKBSnI7MKlAx87F3hId37kJQA79T36TR40C9Uo8XetBaQfDaUyyJrilY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750274; c=relaxed/simple; bh=wjKeRmILuEv3XhFbvkHTofLX7c9w7rQEYLqUWe4Qoek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GnTPYhHsCE5VJ4xLpvl09gvHkf9UOpbcvvDGUFE0cB6skW4Fyj1w19xnwizoVSDK3R0NR+nASka7+R7coyBNjJXAZWQQfE4fdE6V3nopMERKCwbAKRWUCQuUED+hIdJY0pH7x0v8QF20ZS79hoAKZMmKzfLC4+ux/O9dyr9H5Q8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WxUJfuJP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WxUJfuJP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C10F9C4CEFB; Wed, 3 Dec 2025 08:24:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750273; bh=wjKeRmILuEv3XhFbvkHTofLX7c9w7rQEYLqUWe4Qoek=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WxUJfuJPyPpworw3v/WLpFY416v3SYtgNevvykumUYKWYbHWCIKX73mSlUDOEDAb0 TVdVlaXQjrSS4mRrg8WDIXOFL1y/7XVw30DvGZnWefyFAHWnS8ELvSOZqGjRjV9t0+ tqGMmqoQogZGf9+tjbvIT8TTzkxWTAyc82Q5nEQngMrJDhIYn5XvIjaUUdIMDd0si+ lz9vFDiW99tuVzl94LayvsW+YZLTy7kM2nEPz2PuXjeULezuWgw0eBL/N6bn+V3M6i rB7177AyPub7QnvbUc7IBKjXXyKpqtBIclScPrnP1jgXmU7PfQyhaRvu/VrYtyx1TQ pkntOn4XjWBvw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 2/9] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Date: Wed, 3 Dec 2025 09:23:55 +0100 Message-ID: <20251203082402.78816-3-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make alloc_and_copy_ftrace_hash to copy also direct address for each hash entry. Signed-off-by: Jiri Olsa --- kernel/trace/ftrace.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index b0dc911411f1..7e3a81bd6f1e 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -1186,7 +1186,7 @@ static void __add_hash_entry(struct ftrace_hash *hash, } =20 static struct ftrace_func_entry * -add_hash_entry(struct ftrace_hash *hash, unsigned long ip) +add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned= long direct) { struct ftrace_func_entry *entry; =20 @@ -1195,11 +1195,18 @@ add_hash_entry(struct ftrace_hash *hash, unsigned l= ong ip) return NULL; =20 entry->ip =3D ip; + entry->direct =3D direct; __add_hash_entry(hash, entry); =20 return entry; } =20 +static struct ftrace_func_entry * +add_hash_entry(struct ftrace_hash *hash, unsigned long ip) +{ + return add_hash_entry_direct(hash, ip, 0); +} + static void free_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry) @@ -1372,7 +1379,7 @@ alloc_and_copy_ftrace_hash(int size_bits, struct ftra= ce_hash *hash) size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { - if (add_hash_entry(new_hash, entry->ip) =3D=3D NULL) + if (add_hash_entry_direct(new_hash, entry->ip, entry->direct) =3D=3D NU= LL) goto free_hash; } } --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E58E22C21EB; Wed, 3 Dec 2025 08:24:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750285; cv=none; b=G5azKbkqP+EKgFgO6vu6D+igMwM8xD7ZJ3bYM5xRuAHmYyGlI2flN6S5csVX5uNVsuT5ze4QpFYQxgDWSD2IiRegoUNvouwDqgQhsWUsKWcJEc0Z1VVQRiimnWpOIUpQBnO/32pIAj5ayUUf73WQ0SIv3nzouU9wkLmDQzhPPqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750285; c=relaxed/simple; bh=qzSU7P/h2A+9c5oHsCS4glXg/cUYxo9v5c2ips0x2Yc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fCVyJa14xlyMPwwhGo15zAnIPKfCziaK2N2SA6uUJH2Eeezx9aJP1tl++sHPQ7p1fEFwmnyV8ZCSfj7TJ0mY729r4XRsV/pFqteF6xv+768Gmm8/i2SBtciOVnM/Toab2uHZyf0R4ddav8vCuypmVrDAN6l6oUmPwkf9pwUh18k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fNfZuhJN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fNfZuhJN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C68AC4CEFB; Wed, 3 Dec 2025 08:24:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750284; bh=qzSU7P/h2A+9c5oHsCS4glXg/cUYxo9v5c2ips0x2Yc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fNfZuhJNaBFBy/E3xge+z6kXPUGnKw3UNQehHnhpG7M2QMShfTlHyhCglDx+cpcrW RtkPiJokeMDrYSE2l4hodZYu7mh/XWpsBw61HoiujSO+hnYBgEeArklgDE6jcT9yBY a0GyoijzavtfZVwIMRyOdsxzkC6zRY3e25bZKgSgjkBMOp/8Y+TmrASnPCGBatfYg4 M17Q7wAEg/0UF/iqbqG8dKaSDWRCNMXZj6gsva6Dj7ILyEsuuUOZdS27O82jpU4Zt1 caJ6S3fWK3JeqUvfs+SFHOdr8UYC9+h+1S1YGJMWO4E9UuAcdEnOtTJO6vSQJdYGW6 TFFnPfA0KDa3w== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 3/9] ftrace: Export some of hash related functions Date: Wed, 3 Dec 2025 09:23:56 +0100 Message-ID: <20251203082402.78816-4-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We are going to use these functions in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 16 ++++++++++++++++ kernel/trace/ftrace.c | 7 +++---- kernel/trace/trace.h | 8 -------- 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 505b7d3f5641..8c034b028af4 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -405,6 +405,22 @@ enum ftrace_ops_cmd { typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cm= d cmd); =20 #ifdef CONFIG_DYNAMIC_FTRACE + +#define FTRACE_HASH_DEFAULT_BITS 10 + +struct ftrace_hash { + unsigned long size_bits; + struct hlist_head *buckets; + unsigned long count; + unsigned long flags; + struct rcu_head rcu; +}; + +struct ftrace_hash *alloc_ftrace_hash(int size_bits); +void free_ftrace_hash(struct ftrace_hash *hash); +struct ftrace_func_entry *add_hash_entry_direct(struct ftrace_hash *hash, + unsigned long ip, unsigned long direct); + /* The hash used to know what functions callbacks trace */ struct ftrace_ops_hash { struct ftrace_hash __rcu *notrace_hash; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7e3a81bd6f1e..84aee9096a9e 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -68,7 +68,6 @@ }) =20 /* hash bits for specific function selection */ -#define FTRACE_HASH_DEFAULT_BITS 10 #define FTRACE_HASH_MAX_BITS 12 =20 #ifdef CONFIG_DYNAMIC_FTRACE @@ -1185,7 +1184,7 @@ static void __add_hash_entry(struct ftrace_hash *hash, hash->count++; } =20 -static struct ftrace_func_entry * +struct ftrace_func_entry * add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned= long direct) { struct ftrace_func_entry *entry; @@ -1265,7 +1264,7 @@ static void clear_ftrace_mod_list(struct list_head *h= ead) mutex_unlock(&ftrace_lock); } =20 -static void free_ftrace_hash(struct ftrace_hash *hash) +void free_ftrace_hash(struct ftrace_hash *hash) { if (!hash || hash =3D=3D EMPTY_HASH) return; @@ -1305,7 +1304,7 @@ void ftrace_free_filter(struct ftrace_ops *ops) } EXPORT_SYMBOL_GPL(ftrace_free_filter); =20 -static struct ftrace_hash *alloc_ftrace_hash(int size_bits) +struct ftrace_hash *alloc_ftrace_hash(int size_bits) { struct ftrace_hash *hash; int size; diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 85eabb454bee..62e0ac625f65 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -899,14 +899,6 @@ enum { FTRACE_HASH_FL_MOD =3D (1 << 0), }; =20 -struct ftrace_hash { - unsigned long size_bits; - struct hlist_head *buckets; - unsigned long count; - unsigned long flags; - struct rcu_head rcu; -}; - struct ftrace_func_entry * ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip); =20 --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D1262C2ABF; Wed, 3 Dec 2025 08:24:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750295; cv=none; b=kQ36BgJScaZLN4GYlzBfdAig4r008ZZ831Q3mwm1rIUMR8n+1DvkHpA7CWC1dAKY45iZmakAdm2J5VCpIM9zOlI5YO/HhHKbx48E1qZewrKVG6lzOROI+cYAJOsRFUN9tn9pW5TUuMl7cxfTJW21HyoV5rX6tuUJjLrLzbXc5Ug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750295; c=relaxed/simple; bh=Emu7TZrB/ba81eue9Hl9bE7eY0YOVtdlGr12xVhmDKI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=clIjQ1rroqJvSbzJFs1DNqjlZS3fEcE6OdIRQeVYaBmeUcdGhNpSel4QmSwO2p12b8ZzjqEOnvPoLxRa1z2e/SlrHDs4FkD9TrZPmggFOQHZtA+r65h7f0qG6I0FTwdckBYycpuKK7d2ybAkPQX1mEJxZyftpDytbJhwd7Me5e0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sh8Fb7Im; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sh8Fb7Im" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BF41BC4CEFB; Wed, 3 Dec 2025 08:24:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750294; bh=Emu7TZrB/ba81eue9Hl9bE7eY0YOVtdlGr12xVhmDKI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sh8Fb7Im7xyYcTF2xrSNDJ2NMg2CvH8hTvagHrIIwdOmBBjhY+ouhMVsYfNhGk2Hc 8wlEN/thKFYSr/Qe8EpAl4Dkh7LEivEzKnbIB94aeD2tUNHmrzi80D65RiecIPrMbC RR/UoHEMP/6JttVuso8JnsC6m3y65TazU9ssVDErULtHVd196+ydU4+ZL+xH8rVHkT 5kE8+XWSG32xuaYsFCA3s3GLggL0XtV/LiUVQ3s6kK4uPvmwQ5bRuSNzlKzqMHZgIs UNNqEpevQ1nVyXVjMOx9s9Bfga0yPVYGerh8wk4Sue+ZGOrkR/6zSKPIuq+mtTfDWz ZZ3w7XutaBJoA== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 4/9] ftrace: Add update_ftrace_direct_add function Date: Wed, 3 Dec 2025 09:23:57 +0100 Message-ID: <20251203082402.78816-5-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding update_ftrace_direct_add function that adds all entries (ip -> addr) provided in hash argument to direct ftrace ops and updates its attachments. The difference to current register_ftrace_direct is - hash argument that allows to register multiple ip -> direct entries at once - we can call update_ftrace_direct_add multiple times on the same ftrace_ops object, becase after first registration with register_ftrace_function_nolock, it uses ftrace_update_ops to update the ftrace_ops object This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 7 +++ kernel/trace/ftrace.c | 128 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 8c034b028af4..eb60d971ec1d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -550,6 +550,8 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, un= signed long addr, int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); =20 +int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash); + void ftrace_stub_direct_tramp(void); =20 #else @@ -576,6 +578,11 @@ static inline int modify_ftrace_direct_nolock(struct f= trace_ops *ops, unsigned l return -ENODEV; } =20 +static inline int update_ftrace_direct_add(struct ftrace_ops *ops, struct = ftrace_hash *hash) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 84aee9096a9e..1660a87547dd 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6249,6 +6249,134 @@ int modify_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) return err; } EXPORT_SYMBOL_GPL(modify_ftrace_direct); + +static unsigned long hash_count(struct ftrace_hash *hash) +{ + return hash ? hash->count : 0; +} + +/** + * hash_add - adds two struct ftrace_hash and returns the result + * @a: struct ftrace_hash object + * @b: struct ftrace_hash object + * + * Returns struct ftrace_hash object on success, NULL on error. + */ +static struct ftrace_hash *hash_add(struct ftrace_hash *a, struct ftrace_h= ash *b) +{ + struct ftrace_func_entry *entry; + struct ftrace_hash *add; + int size, i; + + size =3D hash_count(a) + hash_count(b); + if (size > 32) + size =3D 32; + + add =3D alloc_and_copy_ftrace_hash(fls(size), a); + if (!add) + goto error; + + size =3D 1 << b->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &b->buckets[i], hlist) { + if (add_hash_entry_direct(add, entry->ip, entry->direct) =3D=3D NULL) + goto error; + } + } + return add; + + error: + free_ftrace_hash(add); + return NULL; +} + +int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash) +{ + struct ftrace_hash *old_direct_functions =3D NULL, *new_direct_functions = =3D NULL; + struct ftrace_hash *old_filter_hash =3D NULL, *new_filter_hash =3D NULL; + struct ftrace_func_entry *entry; + int i, size, err =3D -EINVAL; + bool reg; + + if (!hash_count(hash)) + return -EINVAL; + + mutex_lock(&direct_mutex); + + /* Make sure requested entries are not already registered. */ + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + if (__ftrace_lookup_ip(direct_functions, entry->ip)) + goto out_unlock; + } + } + + old_filter_hash =3D ops->func_hash ? ops->func_hash->filter_hash : NULL; + old_direct_functions =3D direct_functions; + + /* If there's nothing in filter_hash we need to register the ops. */ + reg =3D hash_count(old_filter_hash) =3D=3D 0; + if (reg) { + if (ops->func || ops->trampoline) + goto out_unlock; + if (ops->flags & FTRACE_OPS_FL_ENABLED) + goto out_unlock; + } + + err =3D -ENOMEM; + new_filter_hash =3D hash_add(old_filter_hash, hash); + if (!new_filter_hash) + goto out_unlock; + + new_direct_functions =3D hash_add(old_direct_functions, hash); + if (!new_direct_functions) + goto out_unlock; + + rcu_assign_pointer(direct_functions, new_direct_functions); + + if (reg) { + ops->func =3D call_direct_funcs; + ops->flags |=3D MULTI_FLAGS; + ops->trampoline =3D FTRACE_REGS_ADDR; + ops->local_hash.filter_hash =3D new_filter_hash; + + err =3D register_ftrace_function_nolock(ops); + if (err) { + /* restore old filter on error */ + ops->local_hash.filter_hash =3D old_filter_hash; + old_filter_hash =3D new_filter_hash; + + /* cleanup for possible another register call */ + ops->func =3D NULL; + ops->trampoline =3D 0; + } + } else { + err =3D ftrace_update_ops(ops, new_filter_hash, EMPTY_HASH); + /* + * new_filter_hash is dup-ed, so we need to release it anyway, + * old_filter_hash either stays on error or is released already + */ + old_filter_hash =3D new_filter_hash; + } + + if (err) { + /* reset direct_functions and free the new one */ + rcu_assign_pointer(direct_functions, old_direct_functions); + old_direct_functions =3D new_direct_functions; + } + + out_unlock: + mutex_unlock(&direct_mutex); + + if (old_direct_functions && old_direct_functions !=3D EMPTY_HASH) + call_rcu_tasks(&old_direct_functions->rcu, register_ftrace_direct_cb); + if (old_filter_hash) + free_ftrace_hash(old_filter_hash); + + return err; +} + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AF652C15BB; Wed, 3 Dec 2025 08:25:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750305; cv=none; b=dbMBxcDLjlKWmbcjyz3PvsLX7LjbOA4ECs7kxZ6Jsdt/MM3mLL5MeDbFQxNVPUYnIgOW6aLdOLl2fsoX0kn4GPgd32p7mXAWrh+/19rt50LLFp7I89p6BaHBlDbLReRNCpt5ewPRMWaZE+QZLBOlODKoRipjlz4y3urKLsQoBUU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750305; c=relaxed/simple; bh=sjHj3Ruu15dXTg7oJ7kSKcH775GCTI2sTuBpJwydfrc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VAQvJqWyoGu5jJwlnwrOqr6U1UmFdLsP7rcn93bfBBZs7dO+3nlDDvkfkBU7pes5mU0qDPWW9WSlJm+OXqmy3n2LH3uRUv6kfAd3i0UyalQ9JYpmpULXDh+JeCTFT2x3vI9xdXebjxVKcEWdqTHLJCRJNDBRLaWSVvfT1kFG2aA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=re5sBgVv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="re5sBgVv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2B739C116B1; Wed, 3 Dec 2025 08:25:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750305; bh=sjHj3Ruu15dXTg7oJ7kSKcH775GCTI2sTuBpJwydfrc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=re5sBgVv5u2FxUbViNNVr8qEsmV76iUDcVhnyxz22MjNNnyD5d+Lq0zVbw/S7iQZ3 4D8eYze6f14hAhcIL9I8Ahpv+DfNaU7MS7mKvL5fnBWGTd+M0/jMZ/w7Q7srMYkhNU InJisV06XiUpltFa9ekDYXItUweDJHmWtWnZE2sZ8FgPUx5lQaS3i9LEx1QBsIoGa0 oCBjBq/CCFBXy6D2FUMoULQrPfeVsu8qT8M3Rl7qfBkqx4EjLwXwvkRvvs2Kr86bbO UOf5KFBtBpnWVehQxggYO/PMXgGMCiY8Jsosqjk4KEQ/rqXKojKsIgIGj3ZplKGSPu O6Y1a9Bvn7XkA== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 5/9] ftrace: Add update_ftrace_direct_del function Date: Wed, 3 Dec 2025 09:23:58 +0100 Message-ID: <20251203082402.78816-6-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding update_ftrace_direct_del function that removes all entries (ip -> addr) provided in hash argument to direct ftrace ops and updates its attachments. The difference to current unregister_ftrace_direct is - hash argument that allows to unregister multiple ip -> direct entries at once - we can call update_ftrace_direct_del multiple times on the same ftrace_ops object, becase we do not need to unregister all entries at once, we can do it gradualy with the help of ftrace_update_ops function This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 +++ kernel/trace/ftrace.c | 113 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index eb60d971ec1d..bac9dd784826 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -551,6 +551,7 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr); int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr= ); =20 int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash); +int update_ftrace_direct_del(struct ftrace_ops *ops, struct ftrace_hash *h= ash); =20 void ftrace_stub_direct_tramp(void); =20 @@ -583,6 +584,11 @@ static inline int update_ftrace_direct_add(struct ftra= ce_ops *ops, struct ftrace return -ENODEV; } =20 +static inline int update_ftrace_direct_del(struct ftrace_ops *ops, struct = ftrace_hash *hash) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 1660a87547dd..01e830be20e3 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6377,6 +6377,119 @@ int update_ftrace_direct_add(struct ftrace_ops *ops= , struct ftrace_hash *hash) return err; } =20 +/** + * hash_sub - substracts @b from @a and returns the result + * @a: struct ftrace_hash object + * @b: struct ftrace_hash object + * + * Returns struct ftrace_hash object on success, NULL on error. + */ +static struct ftrace_hash *hash_sub(struct ftrace_hash *a, struct ftrace_h= ash *b) +{ + struct ftrace_func_entry *entry, *del; + struct ftrace_hash *sub; + int size, i; + + sub =3D alloc_and_copy_ftrace_hash(a->size_bits, a); + if (!sub) + goto error; + + size =3D 1 << b->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &b->buckets[i], hlist) { + del =3D __ftrace_lookup_ip(sub, entry->ip); + if (WARN_ON_ONCE(!del)) + goto error; + remove_hash_entry(sub, del); + kfree(del); + } + } + return sub; + + error: + free_ftrace_hash(sub); + return NULL; +} + +int update_ftrace_direct_del(struct ftrace_ops *ops, struct ftrace_hash *h= ash) +{ + struct ftrace_hash *old_direct_functions =3D NULL, *new_direct_functions = =3D NULL; + struct ftrace_hash *old_filter_hash =3D NULL, *new_filter_hash =3D NULL; + struct ftrace_func_entry *del, *entry; + unsigned long size, i; + int err =3D -EINVAL; + + if (!hash_count(hash)) + return -EINVAL; + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + if (direct_functions =3D=3D EMPTY_HASH) + return -EINVAL; + + mutex_lock(&direct_mutex); + + old_filter_hash =3D ops->func_hash ? ops->func_hash->filter_hash : NULL; + old_direct_functions =3D direct_functions; + + if (!hash_count(old_filter_hash)) + goto out_unlock; + + /* Make sure requested entries are already registered. */ + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + del =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (!del || del->direct !=3D entry->direct) + goto out_unlock; + } + } + + err =3D -ENOMEM; + new_filter_hash =3D hash_sub(old_filter_hash, hash); + if (!new_filter_hash) + goto out_unlock; + + new_direct_functions =3D hash_sub(old_direct_functions, hash); + if (!new_direct_functions) + goto out_unlock; + + /* If there's nothing left, we need to unregister the ops. */ + if (ftrace_hash_empty(new_filter_hash)) { + err =3D unregister_ftrace_function(ops); + if (!err) { + /* cleanup for possible another register call */ + ops->func =3D NULL; + ops->trampoline =3D 0; + ftrace_free_filter(ops); + ops->func_hash->filter_hash =3D NULL; + } + } else { + err =3D ftrace_update_ops(ops, new_filter_hash, EMPTY_HASH); + /* + * new_filter_hash is dup-ed, so we need to release it anyway, + * old_filter_hash either stays on error or is released already + */ + } + + if (err) { + /* reset direct_functions and free the new one */ + old_direct_functions =3D new_direct_functions; + } else { + rcu_assign_pointer(direct_functions, new_direct_functions); + } + + out_unlock: + mutex_unlock(&direct_mutex); + + if (old_direct_functions && old_direct_functions !=3D EMPTY_HASH) + call_rcu_tasks(&old_direct_functions->rcu, register_ftrace_direct_cb); + free_ftrace_hash(new_filter_hash); + + return err; +} + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0C852C21FF; Wed, 3 Dec 2025 08:25:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750315; cv=none; b=PE3ud2FSc+sU5K1us3HtInTRo8DdS/Busr4XfAqrVEmamSB2I1cP/wsuvT/yLzWUWVSEPS1LG8q0CFFtxdrbLBLPnV0VDJp4S2Zeo2DrLA8lSqIkG4T0HtBUy6oIAt0al0l2Ihg8JudmAvynoIpmckvg8M7P31F8p3CxK06GrTg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750315; c=relaxed/simple; bh=pLgdjqwxCmhdWC6Ct/H1SR3MVjIRGtL9kTHwK/vSQVg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mLGia+tHnAUsXPhFMPjf7gvJdSQN2XfVk90imN77FHirJAJv1q2GrHmPSc/H6cMwQoVEZkREoskjPG/7lAiK3kgfyWS7YqJoT1keMsmJapLRLj2mcERSZWaGyoBeiJe72Ka/GlN6khGzv5jCvdB+1hVZ/Bk8t7G+S3FfsIiOyb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Zn8G+9ZK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Zn8G+9ZK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6150DC4CEFB; Wed, 3 Dec 2025 08:25:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750315; bh=pLgdjqwxCmhdWC6Ct/H1SR3MVjIRGtL9kTHwK/vSQVg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zn8G+9ZKr+LfhPle/3GxYOBDUz5mKUF0Mk1s4FShEHpgZeVYHOL6VOMlWE22IG9G/ B/g7mAdFI2/PcqVDaurUoITlvirqRV7SiEoP1SR+d6MkLOegg97QTxRcjpP2cE2xKy 2ZUpSYz5ipWao2WmDEj54jLdc29c+5DGLib6QARHRvaRCIQKiVEyYBcSOsyWbRl8Fr MT/Mxa2X61YnLEk0mxRsa0cYLfSiR0VX5faFFUGlWukljC7DRZvtlK0tG/5iFhwOCw oGcSeLbidHqlFTcOXNiuXIqhpmuOez/zsQF5FjdF+slLyq7HdoX330uERIfaUywGU4 7t3UOHMKvV/Fw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 6/9] ftrace: Add update_ftrace_direct_mod function Date: Wed, 3 Dec 2025 09:23:59 +0100 Message-ID: <20251203082402.78816-7-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adding update_ftrace_direct_mod function that modifies all entries (ip -> direct) provided in hash argument to direct ftrace ops and updates its attachments. The difference to current modify_ftrace_direct is: - hash argument that allows to modify multiple ip -> direct entries at once This change will allow us to have simple ftrace_ops for all bpf direct interface users in following changes. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 ++++ kernel/trace/ftrace.c | 72 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index bac9dd784826..c27b7381c5f1 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -552,6 +552,7 @@ int modify_ftrace_direct_nolock(struct ftrace_ops *ops,= unsigned long addr); =20 int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *h= ash); int update_ftrace_direct_del(struct ftrace_ops *ops, struct ftrace_hash *h= ash); +int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *h= ash, bool do_direct_lock); =20 void ftrace_stub_direct_tramp(void); =20 @@ -589,6 +590,11 @@ static inline int update_ftrace_direct_del(struct ftra= ce_ops *ops, struct ftrace return -ENODEV; } =20 +static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct = ftrace_hash *hash, bool do_direct_lock) +{ + return -ENODEV; +} + /* * This must be implemented by the architecture. * It is the way the ftrace direct_ops helper, when called diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 01e830be20e3..c77f620b3eb3 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6490,6 +6490,78 @@ int update_ftrace_direct_del(struct ftrace_ops *ops,= struct ftrace_hash *hash) return err; } =20 +int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *h= ash, bool do_direct_lock) +{ + struct ftrace_func_entry *entry, *tmp; + static struct ftrace_ops tmp_ops =3D { + .func =3D ftrace_stub, + .flags =3D FTRACE_OPS_FL_STUB, + }; + struct ftrace_hash *orig_hash; + unsigned long size, i; + int err =3D -EINVAL; + + if (!hash_count(hash)) + return -EINVAL; + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + if (direct_functions =3D=3D EMPTY_HASH) + return -EINVAL; + + if (do_direct_lock) + mutex_lock(&direct_mutex); + + orig_hash =3D ops->func_hash ? ops->func_hash->filter_hash : NULL; + if (!orig_hash) + goto unlock; + + /* Enable the tmp_ops to have the same functions as the direct ops */ + ftrace_ops_init(&tmp_ops); + tmp_ops.func_hash =3D ops->func_hash; + + err =3D register_ftrace_function_nolock(&tmp_ops); + if (err) + goto unlock; + + /* + * Call __ftrace_hash_update_ipmodify() here, so that we can call + * ops->ops_func for the ops. This is needed because the above + * register_ftrace_function_nolock() worked on tmp_ops. + */ + err =3D __ftrace_hash_update_ipmodify(ops, orig_hash, orig_hash, true); + if (err) + goto out; + + /* + * Now the ftrace_ops_list_func() is called to do the direct callers. + * We can safely change the direct functions attached to each entry. + */ + mutex_lock(&ftrace_lock); + + size =3D 1 << hash->size_bits; + for (i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + tmp =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (!tmp) + continue; + tmp->direct =3D entry->direct; + } + } + + mutex_unlock(&ftrace_lock); + +out: + /* Removing the tmp_ops will add the updated direct callers to the functi= ons */ + unregister_ftrace_function(&tmp_ops); + +unlock: + if (do_direct_lock) + mutex_unlock(&direct_mutex); + return err; +} + #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B70C2C21FF; Wed, 3 Dec 2025 08:25:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750326; cv=none; b=EBtKsUzatOazScFnzFOEHA8GFElqcS0oLjpqY9w6dMZli3ntZ/jZfOt8+i2+GbgoJB/KBaQGiB/IRZRA+0aIFWV5F4FEB/HqKbqfgvwPvRcEtRJwtDe59lG6NNugclTzFcG3tRzhLdEGB7iWGVVoGeNAFZEZr49FCToc0Aqqb0s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750326; c=relaxed/simple; bh=tSOG4mEp4ZPM4CNTyzicNcAGJ8np0EHQKNw1pWrn5Sw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s8hishbr6DVmxw18iItZ6Nyl/TFjRjUXrrtk58Enu7vFU2kKa08kxUk464hsaF/mfe2oJYJWOzPL9JKyTxCdeSsYXth3i/sGvQZ3UAFer5eRO8yevqUaFn9XouRgNJRzx7WxTH1JinDGmqffP5RUjvNDZcn/Wz79CoPDsYNZYBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QV40A99a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QV40A99a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6F7EC116B1; Wed, 3 Dec 2025 08:25:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750326; bh=tSOG4mEp4ZPM4CNTyzicNcAGJ8np0EHQKNw1pWrn5Sw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QV40A99aqtdZftxZn8TC8rQBRa7u7h9M77a2Kv2NAIavwJ795lOCNWDbrHi2Hz6D6 3S05RI+WVF7rULr/WBINFG1kEfzUY+XrV4h2A9h3DE4GWF183VG2PhhOsNbfY4jcmu GvYknkaoweAGWVz4wFklVLSagq+G7B8SarJ9w5FMO12wvNOtP9MsmE3NPya5YbFQif 8XJnfVIkdKvXSZDHHNOHwg82nmdlwwhm6ZAhadgyW8IUum/1Jj7Tx0RikDkFwfUWwC A6zJ3BVTxloBxTC67phbWl7nv8ZDsJJwmxdlxqSvSyzxO5s9KKEOeUKOCidzhPoKtB E5hvBkv2T8wTw== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 7/9] bpf: Add trampoline ip hash table Date: Wed, 3 Dec 2025 09:24:00 +0100 Message-ID: <20251203082402.78816-8-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Following changes need to lookup trampoline based on its ip address, adding hash table for that. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 7 +++++-- kernel/bpf/trampoline.c | 30 +++++++++++++++++++----------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 6498be4c44f8..ef44653eae44 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1307,14 +1307,17 @@ struct bpf_tramp_image { }; =20 struct bpf_trampoline { - /* hlist for trampoline_table */ - struct hlist_node hlist; + /* hlist for trampoline_key_table */ + struct hlist_node hlist_key; + /* hlist for trampoline_ip_table */ + struct hlist_node hlist_ip; struct ftrace_ops *fops; /* serializes access to fields of this trampoline */ struct mutex mutex; refcount_t refcnt; u32 flags; u64 key; + unsigned long ip; struct { struct btf_func_model model; void *addr; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index b9a358d7a78f..f298bafab76e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -24,9 +24,10 @@ const struct bpf_prog_ops bpf_extension_prog_ops =3D { #define TRAMPOLINE_HASH_BITS 10 #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS) =20 -static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE]; +static struct hlist_head trampoline_key_table[TRAMPOLINE_TABLE_SIZE]; +static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; =20 -/* serializes access to trampoline_table */ +/* serializes access to trampoline tables */ static DEFINE_MUTEX(trampoline_mutex); =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS @@ -135,15 +136,15 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } =20 -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long= ip) { struct bpf_trampoline *tr; struct hlist_head *head; int i; =20 mutex_lock(&trampoline_mutex); - head =3D &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; - hlist_for_each_entry(tr, head, hlist) { + head =3D &trampoline_key_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head, hlist_key) { if (tr->key =3D=3D key) { refcount_inc(&tr->refcnt); goto out; @@ -164,8 +165,12 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u6= 4 key) #endif =20 tr->key =3D key; - INIT_HLIST_NODE(&tr->hlist); - hlist_add_head(&tr->hlist, head); + tr->ip =3D ftrace_location(ip); + INIT_HLIST_NODE(&tr->hlist_key); + INIT_HLIST_NODE(&tr->hlist_ip); + hlist_add_head(&tr->hlist_key, head); + head =3D &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)]; + hlist_add_head(&tr->hlist_ip, head); refcount_set(&tr->refcnt, 1); mutex_init(&tr->mutex); for (i =3D 0; i < BPF_TRAMP_MAX; i++) @@ -846,7 +851,7 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog = *prog) prog->aux->attach_btf_id); =20 bpf_lsm_find_cgroup_shim(prog, &bpf_func); - tr =3D bpf_trampoline_lookup(key); + tr =3D bpf_trampoline_lookup(key, 0); if (WARN_ON_ONCE(!tr)) return; =20 @@ -866,7 +871,7 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, { struct bpf_trampoline *tr; =20 - tr =3D bpf_trampoline_lookup(key); + tr =3D bpf_trampoline_lookup(key, tgt_info->tgt_addr); if (!tr) return NULL; =20 @@ -902,7 +907,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) * fexit progs. The fentry-only trampoline will be freed via * multiple rcu callbacks. */ - hlist_del(&tr->hlist); + hlist_del(&tr->hlist_key); + hlist_del(&tr->hlist_ip); if (tr->fops) { ftrace_free_filter(tr->fops); kfree(tr->fops); @@ -1175,7 +1181,9 @@ static int __init init_trampolines(void) int i; =20 for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) - INIT_HLIST_HEAD(&trampoline_table[i]); + INIT_HLIST_HEAD(&trampoline_key_table[i]); + for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) + INIT_HLIST_HEAD(&trampoline_ip_table[i]); return 0; } late_initcall(init_trampolines); --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1C522D063F; Wed, 3 Dec 2025 08:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750336; cv=none; b=NFS6+U6nHb0MmdnjJTi30wayh8FQMBSaJBQ/9jCutm0ikgpXT6RVnaaZEnbeRB35tm92Zo9C1SJ+INlFr15TovlqEbJ06VQf2PNkb5o7HrVSP/JxPvD3FQ4wpn/eLnKv9znzAIzuMEGF/UHd306e6goYcSYo8nM6KY2gyMAIZCs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750336; c=relaxed/simple; bh=WcyS/E/AX8t/ip8FwSSMLUMvyrZSSGqjazXVjPdQ0cI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iNrdwG/6o7EjfJrTJeq5bZ9ZNX87E2j4SBJ4GVMvDFcrDn49+uOgz9D+VUenCIZK1WMYEuri3mLiIDTEW1w/99WrnkuIZHNXKVJSRXR69ceW/hm+SvCSwHNnjYHZnEBS5ktEnvn41806r0VUKPURquMaIGzIOSR/Uzk1QeIIqgQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Sd68vHHk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Sd68vHHk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3EA40C4CEFB; Wed, 3 Dec 2025 08:25:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750336; bh=WcyS/E/AX8t/ip8FwSSMLUMvyrZSSGqjazXVjPdQ0cI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Sd68vHHk3FrgM/ndqM6F0KwcZkQ17/d6I64Ma6Sn6aw3PjUcVB+Glb4+9xPht1znb 7manH/faXGPZhP+Nl89i3mADzP+JlyXfMJPaopdeOzVvxqIk6eJFTbsvvGN+QmxEIf i3EI4p1bE+YfUK2Jgoid/UT+4MzVMxzNCXgQW1cOe6/37b4tejaishM2SFl7jQPq04 ocHFjvOzU2Cjfcfdb4K72/QUFEyJ5rTBE+yO4nPXpejD77bH4mV5FGgJM9YLbpQjta Tnh7fCNLkwR92UvjpimyZ9puGjXHi6kqxeBPiYDlqs9cl+umiMdF/J9nTX731kiFvj VNAh18hXUFk6g== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 8/9] ftrace: Factor ftrace_ops ops_func interface Date: Wed, 3 Dec 2025 09:24:01 +0100 Message-ID: <20251203082402.78816-9-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We are going to remove "ftrace_ops->private =3D=3D bpf_trampoline" setup in following changes. Adding ip argument to ftrace_ops_func_t callback function, so we can use it to look up the trampoline. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 2 +- kernel/bpf/trampoline.c | 3 ++- kernel/trace/ftrace.c | 6 +++--- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index c27b7381c5f1..515cf5b723dd 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -402,7 +402,7 @@ enum ftrace_ops_cmd { * Negative on failure. The return value is dependent on the * callback. */ -typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cm= d cmd); +typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, unsigned long ip, = enum ftrace_ops_cmd cmd); =20 #ifdef CONFIG_DYNAMIC_FTRACE =20 diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index f298bafab76e..17af2aad8382 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -33,7 +33,8 @@ static DEFINE_MUTEX(trampoline_mutex); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_dire= ct_mutex); =20 -static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, enum ftrace_o= ps_cmd cmd) +static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long= ip, + enum ftrace_ops_cmd cmd) { struct bpf_trampoline *tr =3D ops->private; int ret =3D 0; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index c77f620b3eb3..9582cb374d4f 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2049,7 +2049,7 @@ static int __ftrace_hash_update_ipmodify(struct ftrac= e_ops *ops, */ if (!ops->ops_func) return -EBUSY; - ret =3D ops->ops_func(ops, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF); + ret =3D ops->ops_func(ops, rec->ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODI= FY_SELF); if (ret) return ret; } else if (is_ipmodify) { @@ -8984,7 +8984,7 @@ static int prepare_direct_functions_for_ipmodify(stru= ct ftrace_ops *ops) if (!op->ops_func) return -EBUSY; =20 - ret =3D op->ops_func(op, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER); + ret =3D op->ops_func(op, ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER= ); if (ret) return ret; } @@ -9031,7 +9031,7 @@ static void cleanup_direct_functions_after_ipmodify(s= truct ftrace_ops *ops) =20 /* The cleanup is optional, ignore any errors */ if (found_op && op->ops_func) - op->ops_func(op, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER); + op->ops_func(op, ip, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER); } } mutex_unlock(&direct_mutex); --=20 2.52.0 From nobody Fri Dec 19 20:52:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 059ED2D190C; Wed, 3 Dec 2025 08:25:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750347; cv=none; b=ZUlL4FBGooZyLGajleB48Nrcr4ejMhJ9HID37J5az5lBSPnMy5fkGavJzn9rImoWvqiLk1hkVVXRIwnVP1C/q7ro0+QgZY6up0RbMV7XqIeir9DKv753tU+O7TmB5a/Atr5UeOlC2bhktHuqWI7eK+0D6ZJyQXqR2Dlh92fuBXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764750347; c=relaxed/simple; bh=++f6yMiTdCFKivbaWeEKJoZJDBxCoJ9zz2SvUyi+7IY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rs0iPilEkKMqgy+vTaQ+GPR/7V0rkGsNtvZnNwkIPCrFfECYUglZrdCxnMW9UoPWzLcJMJpqZdSkTW4kmWI1EqDPyCXtOPZ2m73sqG8wJk48l4kP1+VcfcnusAcHULKut7ffleGCy7KvPh3zjtOz5mHWD5wSd+tiPwhjAUetmvw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZAhBi1UU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZAhBi1UU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9231BC2BC86; Wed, 3 Dec 2025 08:25:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1764750346; bh=++f6yMiTdCFKivbaWeEKJoZJDBxCoJ9zz2SvUyi+7IY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZAhBi1UUmJik11A52rWsoA5W6YmvvEc/TvhJtPYDFmu9ph/98InowC+hokUTrhAgL i29Q1NqIbjUvH65rrHwbY0b0ej15AsZsU9KM6uvzPTE4PVOjYAXCJ1HFUpkHVZZFvY cLPXG/VCkZyj/cLpkurzYSxnBX72fPSY5WBzErAsyX9ri/4msBbCk0qZR1ol4ZXMKk t+nbw8H4zkYDMJ6BQClhc0y5ePI6UeEAwqxhWStNVLIn8GoGtP2qbmlfKSGPEmwAfC Sz8FWSD4mNzjAXYVJDsQVPMgUHIOby3hsSXvZ9ZqtD1yXdbldUAPyR+AWtTaHJxNjU tcnWh1/0HMkGg== From: Jiri Olsa To: Steven Rostedt , Florent Revest , Mark Rutland Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Menglong Dong , Song Liu Subject: [PATCHv4 bpf-next 9/9] bpf,x86: Use single ftrace_ops for direct calls Date: Wed, 3 Dec 2025 09:24:02 +0100 Message-ID: <20251203082402.78816-10-jolsa@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251203082402.78816-1-jolsa@kernel.org> References: <20251203082402.78816-1-jolsa@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Using single ftrace_ops for direct calls update instead of allocating ftrace_ops object for each trampoline. With single ftrace_ops object we can use update_ftrace_direct_* api that allows multiple ip sites updates on single ftrace_ops object. Adding HAVE_SINGLE_FTRACE_DIRECT_OPS config option to be enabled on each arch that supports this. At the moment we can enable this only on x86 arch, because arm relies on ftrace_ops object representing just single trampoline image (stored in ftrace_ops::direct_call). Ach that do not support this will continue to use *_ftrace_direct api. Signed-off-by: Jiri Olsa --- arch/x86/Kconfig | 1 + kernel/bpf/trampoline.c | 195 ++++++++++++++++++++++++++++++++++------ kernel/trace/Kconfig | 3 + kernel/trace/ftrace.c | 7 +- 4 files changed, 177 insertions(+), 29 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 462250a20311..737dff48342b 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -333,6 +333,7 @@ config X86 select SCHED_SMT if SMP select ARCH_SUPPORTS_SCHED_CLUSTER if SMP select ARCH_SUPPORTS_SCHED_MC if SMP + select HAVE_SINGLE_FTRACE_DIRECT_OPS if X86_64 && DYNAMIC_FTRACE_WITH_DIR= ECT_CALLS =20 config INSTRUCTION_DECODER def_bool y diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 17af2aad8382..02371db3db3e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -33,12 +33,40 @@ static DEFINE_MUTEX(trampoline_mutex); #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_dire= ct_mutex); =20 +#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS +static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops,= unsigned long ip) +{ + struct hlist_head *head_ip; + struct bpf_trampoline *tr; + + mutex_lock(&trampoline_mutex); + head_ip =3D &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head_ip, hlist_ip) { + if (tr->ip =3D=3D ip) + goto out; + } + tr =3D NULL; +out: + mutex_unlock(&trampoline_mutex); + return tr; +} +#else +static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops,= unsigned long ip) +{ + return ops->private; +} +#endif /* CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */ + static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long= ip, enum ftrace_ops_cmd cmd) { - struct bpf_trampoline *tr =3D ops->private; + struct bpf_trampoline *tr; int ret =3D 0; =20 + tr =3D direct_ops_ip_lookup(ops, ip); + if (!tr) + return -EINVAL; + if (cmd =3D=3D FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { /* This is called inside register_ftrace_direct_multi(), so * tr->mutex is already locked. @@ -137,6 +165,139 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS +#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS +/* + * We have only single direct_ops which contains all the direct call + * sites and is the only global ftrace_ops for all trampolines. + * + * We use 'update_ftrace_direct_*' api for attachment. + */ +struct ftrace_ops direct_ops =3D { + .ops_func =3D bpf_tramp_ftrace_ops_func, +}; + +static int direct_ops_alloc(struct bpf_trampoline *tr) +{ + tr->fops =3D &direct_ops; + return 0; +} + +static void direct_ops_free(struct bpf_trampoline *tr) { } + +static struct ftrace_hash *hash_from_ip(struct bpf_trampoline *tr, void *p= tr) +{ + unsigned long ip, addr =3D (unsigned long) ptr; + struct ftrace_hash *hash; + + ip =3D ftrace_location(tr->ip); + if (!ip) + return NULL; + hash =3D alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); + if (!hash) + return NULL; + if (bpf_trampoline_use_jmp(tr->flags)) + addr =3D ftrace_jmp_set(addr); + if (!add_hash_entry_direct(hash, ip, addr)) { + free_ftrace_hash(hash); + return NULL; + } + return hash; +} + +static int direct_ops_add(struct bpf_trampoline *tr, void *addr) +{ + struct ftrace_hash *hash =3D hash_from_ip(tr, addr); + int err =3D -ENOMEM; + + if (hash) + err =3D update_ftrace_direct_add(tr->fops, hash); + free_ftrace_hash(hash); + return err; +} + +static int direct_ops_del(struct bpf_trampoline *tr, void *addr) +{ + struct ftrace_hash *hash =3D hash_from_ip(tr, addr); + int err =3D -ENOMEM; + + if (hash) + err =3D update_ftrace_direct_del(tr->fops, hash); + free_ftrace_hash(hash); + return err; +} + +static int direct_ops_mod(struct bpf_trampoline *tr, void *addr, bool lock= _direct_mutex) +{ + struct ftrace_hash *hash =3D hash_from_ip(tr, addr); + int err =3D -ENOMEM; + + if (hash) + err =3D update_ftrace_direct_mod(tr->fops, hash, lock_direct_mutex); + free_ftrace_hash(hash); + return err; +} +#else +/* + * We allocate ftrace_ops object for each trampoline and it contains + * call site specific for that trampoline. + * + * We use *_ftrace_direct api for attachment. + */ +static int direct_ops_alloc(struct bpf_trampoline *tr) +{ + tr->fops =3D kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL); + if (!tr->fops) + return -ENOMEM; + tr->fops->private =3D tr; + tr->fops->ops_func =3D bpf_tramp_ftrace_ops_func; + return 0; +} + +static void direct_ops_free(struct bpf_trampoline *tr) +{ + if (tr->fops) { + ftrace_free_filter(tr->fops); + kfree(tr->fops); + } +} + +static int direct_ops_add(struct bpf_trampoline *tr, void *ptr) +{ + unsigned long addr =3D (unsigned long) ptr; + struct ftrace_ops *ops =3D tr->fops; + int ret; + + if (bpf_trampoline_use_jmp(tr->flags)) + addr =3D ftrace_jmp_set(addr); + + ret =3D ftrace_set_filter_ip(ops, tr->ip, 0, 1); + if (ret) + return ret; + return register_ftrace_direct(ops, addr); +} + +static int direct_ops_del(struct bpf_trampoline *tr, void *addr) +{ + return unregister_ftrace_direct(tr->fops, (long)addr, false); +} + +static int direct_ops_mod(struct bpf_trampoline *tr, void *ptr, bool lock_= direct_mutex) +{ + unsigned long addr =3D (unsigned long) ptr; + struct ftrace_ops *ops =3D tr->fops; + + if (bpf_trampoline_use_jmp(tr->flags)) + addr =3D ftrace_jmp_set(addr); + if (lock_direct_mutex) + return modify_ftrace_direct(ops, addr); + return modify_ftrace_direct_nolock(ops, addr); +} +#endif /* CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */ +#else +static void direct_ops_free(struct bpf_trampoline *tr) { } +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ + static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long= ip) { struct bpf_trampoline *tr; @@ -155,14 +316,11 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u= 64 key, unsigned long ip) if (!tr) goto out; #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS - tr->fops =3D kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL); - if (!tr->fops) { + if (direct_ops_alloc(tr)) { kfree(tr); tr =3D NULL; goto out; } - tr->fops->private =3D tr; - tr->fops->ops_func =3D bpf_tramp_ftrace_ops_func; #endif =20 tr->key =3D key; @@ -206,7 +364,7 @@ static int unregister_fentry(struct bpf_trampoline *tr,= u32 orig_flags, int ret; =20 if (tr->func.ftrace_managed) - ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); + ret =3D direct_ops_del(tr, old_addr); else ret =3D bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL); =20 @@ -220,15 +378,7 @@ static int modify_fentry(struct bpf_trampoline *tr, u3= 2 orig_flags, int ret; =20 if (tr->func.ftrace_managed) { - unsigned long addr =3D (unsigned long) new_addr; - - if (bpf_trampoline_use_jmp(tr->flags)) - addr =3D ftrace_jmp_set(addr); - - if (lock_direct_mutex) - ret =3D modify_ftrace_direct(tr->fops, addr); - else - ret =3D modify_ftrace_direct_nolock(tr->fops, addr); + ret =3D direct_ops_mod(tr, new_addr, lock_direct_mutex); } else { ret =3D bpf_trampoline_update_fentry(tr, orig_flags, old_addr, new_addr); @@ -251,15 +401,7 @@ static int register_fentry(struct bpf_trampoline *tr, = void *new_addr) } =20 if (tr->func.ftrace_managed) { - unsigned long addr =3D (unsigned long) new_addr; - - if (bpf_trampoline_use_jmp(tr->flags)) - addr =3D ftrace_jmp_set(addr); - - ret =3D ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); - if (ret) - return ret; - ret =3D register_ftrace_direct(tr->fops, addr); + ret =3D direct_ops_add(tr, new_addr); } else { ret =3D bpf_trampoline_update_fentry(tr, 0, NULL, new_addr); } @@ -910,10 +1052,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) */ hlist_del(&tr->hlist_key); hlist_del(&tr->hlist_ip); - if (tr->fops) { - ftrace_free_filter(tr->fops); - kfree(tr->fops); - } + direct_ops_free(tr); kfree(tr); out: mutex_unlock(&trampoline_mutex); diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 4661b9e606e0..1ad2e307c834 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -50,6 +50,9 @@ config HAVE_DYNAMIC_FTRACE_WITH_REGS config HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS bool =20 +config HAVE_SINGLE_FTRACE_DIRECT_OPS + bool + config HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS bool =20 diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 9582cb374d4f..0bdee1758fa1 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2605,8 +2605,13 @@ unsigned long ftrace_find_rec_direct(unsigned long i= p) static void call_direct_funcs(unsigned long ip, unsigned long pip, struct ftrace_ops *ops, struct ftrace_regs *fregs) { - unsigned long addr =3D READ_ONCE(ops->direct_call); + unsigned long addr; =20 +#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS + addr =3D ftrace_find_rec_direct(ip); +#else + addr =3D READ_ONCE(ops->direct_call); +#endif if (!addr) return; =20 --=20 2.52.0