include/linux/bpf.h | 3 + include/uapi/linux/bpf.h | 6 +- kernel/bpf/trampoline.c | 87 +++--- kernel/bpf/verifier.c | 22 +- kernel/trace/bpf_trace.c | 212 ++++++++++++--- tools/bpf/bpftool/link.c | 59 +++- tools/include/uapi/linux/bpf.h | 6 +- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/bpf.h | 2 + tools/lib/bpf/libbpf.c | 39 ++- tools/lib/bpf/libbpf.h | 4 +- tools/lib/bpf/libbpf_internal.h | 1 + .../selftests/bpf/prog_tests/fill_link_info.c | 170 ++++++++++-- .../selftests/bpf/prog_tests/tailcalls.c | 62 +++++ .../selftests/bpf/prog_tests/tracing_multi.c | 254 ++++++++++++++++++ .../selftests/bpf/progs/tracing_multi_bpf.c | 61 +++++ 16 files changed, 869 insertions(+), 120 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_bpf.c
Similar to the tracing_multi link support for kernel functions [1], add support for bpf progs. When attaching to bpf progs, it must attaches to the target by text poke way. [1] https://lore.kernel.org/bpf/20260606123955.345967-1-jolsa@kernel.org/ Leon Hwang (13): bpf: Initialize ftrace_managed in bpf_trampoline_get bpf: Factor out update_fentry_multi helper bpf: Drop unnecessary ftrace_location() in update_fentry_multi() bpf: Add tracing_multi link support for bpf progs libbpf: Add tracing_multi link support for bpf progs bpf: Add tracing_multi link fdinfo support for bpf progs bpf: Add tracing_multi link info support for bpf progs selftests/bpf: Add tracing_multi bpf prog attach test selftests/bpf: Add tracing_multi bpf prog attach failure tests selftests/bpf: Add tracing_multi bpf prog cookie test selftests/bpf: Add tracing_multi bpf prog rollback test selftests/bpf: Add tracing_multi bpf prog link info test selftests/bpf: Test tailcall with fentry.multi include/linux/bpf.h | 3 + include/uapi/linux/bpf.h | 6 +- kernel/bpf/trampoline.c | 87 +++--- kernel/bpf/verifier.c | 22 +- kernel/trace/bpf_trace.c | 212 ++++++++++++--- tools/bpf/bpftool/link.c | 59 +++- tools/include/uapi/linux/bpf.h | 6 +- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/bpf.h | 2 + tools/lib/bpf/libbpf.c | 39 ++- tools/lib/bpf/libbpf.h | 4 +- tools/lib/bpf/libbpf_internal.h | 1 + .../selftests/bpf/prog_tests/fill_link_info.c | 170 ++++++++++-- .../selftests/bpf/prog_tests/tailcalls.c | 62 +++++ .../selftests/bpf/prog_tests/tracing_multi.c | 254 ++++++++++++++++++ .../selftests/bpf/progs/tracing_multi_bpf.c | 61 +++++ 16 files changed, 869 insertions(+), 120 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_bpf.c -- 2.55.0
On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: > > Similar to the tracing_multi link support for kernel functions [1], add > support for bpf progs. > > When attaching to bpf progs, it must attaches to the target by text poke > way. > Please spend a bit more human effort on justification for the change and explaining your use case. In what case you'll be attaching to a large amount of BPF programs such that attachment speed-up (if there is any) matters. Please do your homework. Generating code with AI is easy, reviewing and making decisions about whether it's the right approach (and ultimately supporting it long term) is still a human process, so let's weigh that in. > [1] https://lore.kernel.org/bpf/20260606123955.345967-1-jolsa@kernel.org/ > > Leon Hwang (13): > bpf: Initialize ftrace_managed in bpf_trampoline_get > bpf: Factor out update_fentry_multi helper > bpf: Drop unnecessary ftrace_location() in update_fentry_multi() > bpf: Add tracing_multi link support for bpf progs > libbpf: Add tracing_multi link support for bpf progs > bpf: Add tracing_multi link fdinfo support for bpf progs > bpf: Add tracing_multi link info support for bpf progs > selftests/bpf: Add tracing_multi bpf prog attach test > selftests/bpf: Add tracing_multi bpf prog attach failure tests > selftests/bpf: Add tracing_multi bpf prog cookie test > selftests/bpf: Add tracing_multi bpf prog rollback test > selftests/bpf: Add tracing_multi bpf prog link info test > selftests/bpf: Test tailcall with fentry.multi > > include/linux/bpf.h | 3 + > include/uapi/linux/bpf.h | 6 +- > kernel/bpf/trampoline.c | 87 +++--- > kernel/bpf/verifier.c | 22 +- > kernel/trace/bpf_trace.c | 212 ++++++++++++--- > tools/bpf/bpftool/link.c | 59 +++- > tools/include/uapi/linux/bpf.h | 6 +- > tools/lib/bpf/bpf.c | 1 + > tools/lib/bpf/bpf.h | 2 + > tools/lib/bpf/libbpf.c | 39 ++- > tools/lib/bpf/libbpf.h | 4 +- > tools/lib/bpf/libbpf_internal.h | 1 + > .../selftests/bpf/prog_tests/fill_link_info.c | 170 ++++++++++-- > .../selftests/bpf/prog_tests/tailcalls.c | 62 +++++ > .../selftests/bpf/prog_tests/tracing_multi.c | 254 ++++++++++++++++++ > .../selftests/bpf/progs/tracing_multi_bpf.c | 61 +++++ > 16 files changed, 869 insertions(+), 120 deletions(-) > create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_bpf.c > > -- > 2.55.0
On 15/8/26 04:14, Andrii Nakryiko wrote: > On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: >> >> Similar to the tracing_multi link support for kernel functions [1], add >> support for bpf progs. >> >> When attaching to bpf progs, it must attaches to the target by text poke >> way. >> > > Please spend a bit more human effort on justification for the change > and explaining your use case. In what case you'll be attaching to a > large amount of BPF programs such that attachment speed-up (if there > is any) matters. Will update the cover letter with justification and my use case. Here's my use case: I'm planning to enhance the function-graph feature of bpfsnoop [1]. It will trace all bpf progs, including their subprogs, to draw a full function call graph including bpf prog call sites. The attachment could speed-up when there are hundreds of targets. For example, on high CPU core count servers, it probably runs 200 tc-bpf progs as Kubernetes CNI, and each tc-bpf prog might have 4 subprogs. Hence, there will be 1000 targets to trace for bpfsnoop's function-graph feature. With tracing_multi link support for bpf progs, bpfsnoop will be able to attach to all targets quickly. [1] https://github.com/bpfsnoop/bpfsnoop > > Please do your homework. Generating code with AI is easy, reviewing Sure, will do my homework. > and making decisions about whether it's the right approach (and > ultimately supporting it long term) is still a human process, so let's > weigh that in. Will keep it in mind. Thanks, Leon > >> [1] https://lore.kernel.org/bpf/20260606123955.345967-1-jolsa@kernel.org/ >> >> Leon Hwang (13): >> bpf: Initialize ftrace_managed in bpf_trampoline_get >> bpf: Factor out update_fentry_multi helper >> bpf: Drop unnecessary ftrace_location() in update_fentry_multi() >> bpf: Add tracing_multi link support for bpf progs >> libbpf: Add tracing_multi link support for bpf progs >> bpf: Add tracing_multi link fdinfo support for bpf progs >> bpf: Add tracing_multi link info support for bpf progs >> selftests/bpf: Add tracing_multi bpf prog attach test >> selftests/bpf: Add tracing_multi bpf prog attach failure tests >> selftests/bpf: Add tracing_multi bpf prog cookie test >> selftests/bpf: Add tracing_multi bpf prog rollback test >> selftests/bpf: Add tracing_multi bpf prog link info test >> selftests/bpf: Test tailcall with fentry.multi >> >> include/linux/bpf.h | 3 + >> include/uapi/linux/bpf.h | 6 +- >> kernel/bpf/trampoline.c | 87 +++--- >> kernel/bpf/verifier.c | 22 +- >> kernel/trace/bpf_trace.c | 212 ++++++++++++--- >> tools/bpf/bpftool/link.c | 59 +++- >> tools/include/uapi/linux/bpf.h | 6 +- >> tools/lib/bpf/bpf.c | 1 + >> tools/lib/bpf/bpf.h | 2 + >> tools/lib/bpf/libbpf.c | 39 ++- >> tools/lib/bpf/libbpf.h | 4 +- >> tools/lib/bpf/libbpf_internal.h | 1 + >> .../selftests/bpf/prog_tests/fill_link_info.c | 170 ++++++++++-- >> .../selftests/bpf/prog_tests/tailcalls.c | 62 +++++ >> .../selftests/bpf/prog_tests/tracing_multi.c | 254 ++++++++++++++++++ >> .../selftests/bpf/progs/tracing_multi_bpf.c | 61 +++++ >> 16 files changed, 869 insertions(+), 120 deletions(-) >> create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_bpf.c >> >> -- >> 2.55.0
On Sun, Aug 16, 2026 at 11:20 PM Leon Hwang <leon.hwang@linux.dev> wrote: > > On 15/8/26 04:14, Andrii Nakryiko wrote: > > On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: > >> > >> Similar to the tracing_multi link support for kernel functions [1], add > >> support for bpf progs. > >> > >> When attaching to bpf progs, it must attaches to the target by text poke > >> way. > >> > > > > Please spend a bit more human effort on justification for the change > > and explaining your use case. In what case you'll be attaching to a > > large amount of BPF programs such that attachment speed-up (if there > > is any) matters. > > Will update the cover letter with justification and my use case. > > Here's my use case: > > I'm planning to enhance the function-graph feature of bpfsnoop [1]. It > will trace all bpf progs, including their subprogs, to draw a full > function call graph including bpf prog call sites. > fair enough, interesting use case, definitely outline that in the next revision (and provide before/after attach time as well for such use case, please) > The attachment could speed-up when there are hundreds of targets. For > example, on high CPU core count servers, it probably runs 200 tc-bpf > progs as Kubernetes CNI, and each tc-bpf prog might have 4 subprogs. > Hence, there will be 1000 targets to trace for bpfsnoop's function-graph > feature. > > With tracing_multi link support for bpf progs, bpfsnoop will be able to > attach to all targets quickly. > > [1] https://github.com/bpfsnoop/bpfsnoop > > > > > Please do your homework. Generating code with AI is easy, reviewing > > Sure, will do my homework. > > > and making decisions about whether it's the right approach (and > > ultimately supporting it long term) is still a human process, so let's > > weigh that in. > > Will keep it in mind. > > Thanks, > Leon > > > > >> [1] https://lore.kernel.org/bpf/20260606123955.345967-1-jolsa@kernel.org/ > >> > >> Leon Hwang (13): > >> bpf: Initialize ftrace_managed in bpf_trampoline_get > >> bpf: Factor out update_fentry_multi helper > >> bpf: Drop unnecessary ftrace_location() in update_fentry_multi() > >> bpf: Add tracing_multi link support for bpf progs > >> libbpf: Add tracing_multi link support for bpf progs > >> bpf: Add tracing_multi link fdinfo support for bpf progs > >> bpf: Add tracing_multi link info support for bpf progs > >> selftests/bpf: Add tracing_multi bpf prog attach test > >> selftests/bpf: Add tracing_multi bpf prog attach failure tests > >> selftests/bpf: Add tracing_multi bpf prog cookie test > >> selftests/bpf: Add tracing_multi bpf prog rollback test > >> selftests/bpf: Add tracing_multi bpf prog link info test > >> selftests/bpf: Test tailcall with fentry.multi > >> > >> include/linux/bpf.h | 3 + > >> include/uapi/linux/bpf.h | 6 +- > >> kernel/bpf/trampoline.c | 87 +++--- > >> kernel/bpf/verifier.c | 22 +- > >> kernel/trace/bpf_trace.c | 212 ++++++++++++--- > >> tools/bpf/bpftool/link.c | 59 +++- > >> tools/include/uapi/linux/bpf.h | 6 +- > >> tools/lib/bpf/bpf.c | 1 + > >> tools/lib/bpf/bpf.h | 2 + > >> tools/lib/bpf/libbpf.c | 39 ++- > >> tools/lib/bpf/libbpf.h | 4 +- > >> tools/lib/bpf/libbpf_internal.h | 1 + > >> .../selftests/bpf/prog_tests/fill_link_info.c | 170 ++++++++++-- > >> .../selftests/bpf/prog_tests/tailcalls.c | 62 +++++ > >> .../selftests/bpf/prog_tests/tracing_multi.c | 254 ++++++++++++++++++ > >> .../selftests/bpf/progs/tracing_multi_bpf.c | 61 +++++ > >> 16 files changed, 869 insertions(+), 120 deletions(-) > >> create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_bpf.c > >> > >> -- > >> 2.55.0 >
On 22/8/26 01:46, Andrii Nakryiko wrote: > On Sun, Aug 16, 2026 at 11:20 PM Leon Hwang <leon.hwang@linux.dev> wrote: >> >> On 15/8/26 04:14, Andrii Nakryiko wrote: >>> On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: >>>> >>>> Similar to the tracing_multi link support for kernel functions [1], add >>>> support for bpf progs. >>>> >>>> When attaching to bpf progs, it must attaches to the target by text poke >>>> way. >>>> >>> >>> Please spend a bit more human effort on justification for the change >>> and explaining your use case. In what case you'll be attaching to a >>> large amount of BPF programs such that attachment speed-up (if there >>> is any) matters. >> >> Will update the cover letter with justification and my use case. >> >> Here's my use case: >> >> I'm planning to enhance the function-graph feature of bpfsnoop [1]. It >> will trace all bpf progs, including their subprogs, to draw a full >> function call graph including bpf prog call sites. >> > > fair enough, interesting use case, definitely outline that in the next > revision (and provide before/after attach time as well for such use > case, please) > Will include my use case. Implemented the attachment micro-benchmark. Here's the result on an x86_64 16c16g VM: ./bench tracing-multi-attach-progs Setting up benchmark 'tracing-multi-attach-progs'... tracing-multi-attach-progs: prepared 1000 identical BPF program targets tracing-multi-attach-progs: fentry created and attached 1000 programs/links in 679.324ms tracing-multi-attach-progs: fentry.multi created one program and attached one 1000-target link in 470.042ms tracing-multi-attach-progs: fentry.multi creation/attachment speedup is 1.45x The speedup 1.45x looks good. But attaching to 1000 progs via fentry costs less than 1 second, which is really faster than attaching to around 1000 kernel functions via fentry: bpfsnoop -k '*:(struct sock *)sk' -m entry -D Tracing 1096 tracees costs 11.597266736s Thanks, Leon
On Sun, Aug 23, 2026 at 10:00 PM Leon Hwang <leon.hwang@linux.dev> wrote: > > On 22/8/26 01:46, Andrii Nakryiko wrote: > > On Sun, Aug 16, 2026 at 11:20 PM Leon Hwang <leon.hwang@linux.dev> wrote: > >> > >> On 15/8/26 04:14, Andrii Nakryiko wrote: > >>> On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: > >>>> > >>>> Similar to the tracing_multi link support for kernel functions [1], add > >>>> support for bpf progs. > >>>> > >>>> When attaching to bpf progs, it must attaches to the target by text poke > >>>> way. > >>>> > >>> > >>> Please spend a bit more human effort on justification for the change > >>> and explaining your use case. In what case you'll be attaching to a > >>> large amount of BPF programs such that attachment speed-up (if there > >>> is any) matters. > >> > >> Will update the cover letter with justification and my use case. > >> > >> Here's my use case: > >> > >> I'm planning to enhance the function-graph feature of bpfsnoop [1]. It > >> will trace all bpf progs, including their subprogs, to draw a full > >> function call graph including bpf prog call sites. > >> > > > > fair enough, interesting use case, definitely outline that in the next > > revision (and provide before/after attach time as well for such use > > case, please) > > > Will include my use case. > > Implemented the attachment micro-benchmark. Here's the result on an > x86_64 16c16g VM: > > ./bench tracing-multi-attach-progs > Setting up benchmark 'tracing-multi-attach-progs'... > tracing-multi-attach-progs: prepared 1000 identical BPF program targets > tracing-multi-attach-progs: fentry created and attached 1000 > programs/links in 679.324ms > tracing-multi-attach-progs: fentry.multi created one program and > attached one 1000-target link in 470.042ms > tracing-multi-attach-progs: fentry.multi creation/attachment speedup is > 1.45x > > The speedup 1.45x looks good. But attaching to 1000 progs via fentry I'd say "meh". If 500ms is ok, 700ms is not that much more noticeable. > costs less than 1 second, which is really faster than attaching to > around 1000 kernel functions via fentry: > > bpfsnoop -k '*:(struct sock *)sk' -m entry -D > Tracing 1096 tracees costs 11.597266736s > > Thanks, > Leon >
On 25/8/26 04:02, Andrii Nakryiko wrote: > On Sun, Aug 23, 2026 at 10:00 PM Leon Hwang <leon.hwang@linux.dev> wrote: >> >> On 22/8/26 01:46, Andrii Nakryiko wrote: >>> On Sun, Aug 16, 2026 at 11:20 PM Leon Hwang <leon.hwang@linux.dev> wrote: >>>> >>>> On 15/8/26 04:14, Andrii Nakryiko wrote: >>>>> On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: >>>>>> >>>>>> Similar to the tracing_multi link support for kernel functions [1], add >>>>>> support for bpf progs. >>>>>> >>>>>> When attaching to bpf progs, it must attaches to the target by text poke >>>>>> way. >>>>>> >>>>> >>>>> Please spend a bit more human effort on justification for the change >>>>> and explaining your use case. In what case you'll be attaching to a >>>>> large amount of BPF programs such that attachment speed-up (if there >>>>> is any) matters. >>>> >>>> Will update the cover letter with justification and my use case. >>>> >>>> Here's my use case: >>>> >>>> I'm planning to enhance the function-graph feature of bpfsnoop [1]. It >>>> will trace all bpf progs, including their subprogs, to draw a full >>>> function call graph including bpf prog call sites. >>>> >>> >>> fair enough, interesting use case, definitely outline that in the next >>> revision (and provide before/after attach time as well for such use >>> case, please) >>> >> Will include my use case. >> >> Implemented the attachment micro-benchmark. Here's the result on an >> x86_64 16c16g VM: >> >> ./bench tracing-multi-attach-progs >> Setting up benchmark 'tracing-multi-attach-progs'... >> tracing-multi-attach-progs: prepared 1000 identical BPF program targets >> tracing-multi-attach-progs: fentry created and attached 1000 >> programs/links in 679.324ms >> tracing-multi-attach-progs: fentry.multi created one program and >> attached one 1000-target link in 470.042ms >> tracing-multi-attach-progs: fentry.multi creation/attachment speedup is >> 1.45x >> >> The speedup 1.45x looks good. But attaching to 1000 progs via fentry > > I'd say "meh". If 500ms is ok, 700ms is not that much more noticeable. > Besides the attachment speedup, this series provides a way to attach to multiple bpf progs via one link. The current revision reuses the ftrace-based attachment, which relies on CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS. In the next revision, it will get rid of CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS with its own attachment implementation. Hence, this series will be able to run on those arches that have trampoline support. Thanks, Leon
On Mon, Aug 24, 2026 at 10:15 PM Leon Hwang <leon.hwang@linux.dev> wrote: > > On 25/8/26 04:02, Andrii Nakryiko wrote: > > On Sun, Aug 23, 2026 at 10:00 PM Leon Hwang <leon.hwang@linux.dev> wrote: > >> > >> On 22/8/26 01:46, Andrii Nakryiko wrote: > >>> On Sun, Aug 16, 2026 at 11:20 PM Leon Hwang <leon.hwang@linux.dev> wrote: > >>>> > >>>> On 15/8/26 04:14, Andrii Nakryiko wrote: > >>>>> On Sun, Aug 9, 2026 at 8:01 AM Leon Hwang <leon.hwang@linux.dev> wrote: > >>>>>> > >>>>>> Similar to the tracing_multi link support for kernel functions [1], add > >>>>>> support for bpf progs. > >>>>>> > >>>>>> When attaching to bpf progs, it must attaches to the target by text poke > >>>>>> way. > >>>>>> > >>>>> > >>>>> Please spend a bit more human effort on justification for the change > >>>>> and explaining your use case. In what case you'll be attaching to a > >>>>> large amount of BPF programs such that attachment speed-up (if there > >>>>> is any) matters. > >>>> > >>>> Will update the cover letter with justification and my use case. > >>>> > >>>> Here's my use case: > >>>> > >>>> I'm planning to enhance the function-graph feature of bpfsnoop [1]. It > >>>> will trace all bpf progs, including their subprogs, to draw a full > >>>> function call graph including bpf prog call sites. > >>>> > >>> > >>> fair enough, interesting use case, definitely outline that in the next > >>> revision (and provide before/after attach time as well for such use > >>> case, please) > >>> > >> Will include my use case. > >> > >> Implemented the attachment micro-benchmark. Here's the result on an > >> x86_64 16c16g VM: > >> > >> ./bench tracing-multi-attach-progs > >> Setting up benchmark 'tracing-multi-attach-progs'... > >> tracing-multi-attach-progs: prepared 1000 identical BPF program targets > >> tracing-multi-attach-progs: fentry created and attached 1000 > >> programs/links in 679.324ms > >> tracing-multi-attach-progs: fentry.multi created one program and > >> attached one 1000-target link in 470.042ms > >> tracing-multi-attach-progs: fentry.multi creation/attachment speedup is > >> 1.45x > >> > >> The speedup 1.45x looks good. But attaching to 1000 progs via fentry > > > > I'd say "meh". If 500ms is ok, 700ms is not that much more noticeable. > > > Besides the attachment speedup, this series provides a way to attach to > multiple bpf progs via one link. The current revision reuses the that's mostly a convenience which is pretty trivial to abstract away in user space, not really a hard requirement for kernel to support this > ftrace-based attachment, which relies on > CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS. In the next revision, it will get > rid of CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS with its own attachment > implementation. Hence, this series will be able to run on those arches > that have trampoline support. > sure, post a new version, but it's a bit less motivating to add extra complexity to the kernel if the only outcome is slight speed up and user space convenience of having single FD let's look at new revision and see how this goes > Thanks, > Leon >
© 2016 - 2026 Red Hat, Inc.