A task running in a trampoline image can call a prog that was detached
and freed in the meantime, when it slept in a sleepable prog before
reaching the detached one or was preempted right before calling it.
Patch 1 handles the preempted case with an RCU tasks grace period,
patch 2 handles the sleeping case by patching detached progs out of the
images that still call them and patch 3 adds a selftest for the sleeping
case.
v3 patched all the nops of an image when it was put. bpf-ci pointed out
that this lets a task skip fmod_ret and LSM progs that are still
attached, for as long as it sleeps in an earlier prog, and Alexei asked
to go back to what v2 did: only the nop of the prog that is detached is
patched, in every image that isn't freed yet. Patch 2 explains why that
takes a list of images and not just the current one. ip_after_call is
gone in both versions, and the call to the original function is never
skipped.
sashiko noted that on riscv and loongarch a task can be preempted in
the middle of a multi-instruction patch site. ip_after_call has this too
and it isn't addressed here, pending input from the JIT maintainers on
whether a single instruction site is fine for these in-image jumps.
Tested on x86_64 under KVM and on arm64, s390x, powerpc64le, riscv64
and loongarch64 under qemu TCG, all with KASAN: the new selftest crashes
the unpatched kernel and passes with the series on every one of them,
along with trampoline_count, fentry/fexit, fentry_fexit, modify_return
and lsm_cgroup (and the full test_progs on x86_64). On x86_64, 18
fsession progs with cookies on an 11 argument function still fit in the
image. Same on bpf-next, where patch 2 has a trivial conflict in x86's
arch_bpf_trampoline_size().
Changes since v3
(https://lore.kernel.org/bpf/20260924170543.1017048-1-florent.revest@linux.dev/):
- Only patch the nop of the prog that is detached, in all the images
that aren't freed yet, like v2 did, and explain why a list of images
is needed (Alexei, bpf-ci)
- Lower BPF_MAX_TRAMP_LINKS to 36 on x86, the worst case no longer fit
in a page with the nops (bpf-ci, Alexei)
- selftest: wait for the detached progs to really be freed before
releasing the task, the grace period of patch 1 made the previous wait
too short (bpf-ci). Detach two progs one after the other, so that the
second detach has to reach an image that isn't the current one anymore
- Keep a single comment block in bpf_tramp_image_put() (bpf-ci)
Changes since v2
(https://lore.kernel.org/bpf/20260912095924.866254-1-florent.revest@linux.dev/):
- Patch all the nops in bpf_tramp_image_put() instead of the detached
prog's nop at detach time, drop the image list, the trampoline
backpointer and ip_after_call (Alexei)
- Wait for an RCU tasks grace period before freeing progs that were
linked to a trampoline, for tasks preempted right before the enter
helper (Junseo, sashiko)
- Initialize the jit ctx in loongarch's arch_bpf_trampoline_size() and
the dummy image in every arch_bpf_trampoline_size() (bpf-ci)
- selftest: move the userfaultfd helper to testing_helpers.c and share
it with bpf_mod_race, comment fixes (bpf-ci), add a subtest where the
original function runs between the sleeping prog and the detached one
Changes since v1
(https://lore.kernel.org/bpf/20260819122252.1782790-1-florent.revest@linux.dev/):
- Patch nops in front of detached progs instead of taking prog
references from the image (Alexei)
- Lower BPF_MAX_TRAMP_LINKS on arm64, loongarch and powerpc so the
image still fits in a page
- Explain that the sleepable case doesn't depend on CONFIG_PREEMPTION
(Kumar, Alexei)
- Add a selftest (Jiri, Alexei)
- Add Sechang's Reported-by (Junseo, Kumar)
- Drop Leon's and Kumar's acks since the code changed entirely
Florent Revest (Anthropic) (3):
bpf: Wait for an RCU tasks grace period before freeing trampoline
progs
bpf: Skip detached progs in trampoline images that are still in use
selftests/bpf: Detach a trampoline prog while a task sleeps before it
arch/arm64/net/bpf_jit_comp.c | 33 +--
arch/loongarch/net/bpf_jit.c | 45 +++--
arch/powerpc/net/bpf_jit_comp.c | 45 +++--
arch/riscv/net/bpf_jit_comp64.c | 39 ++--
arch/s390/net/bpf_jit_comp.c | 46 +++--
arch/x86/net/bpf_jit_comp.c | 26 ++-
include/linux/bpf.h | 46 ++++-
kernel/bpf/syscall.c | 19 +-
kernel/bpf/trampoline.c | 75 +++++--
.../selftests/bpf/prog_tests/bpf_mod_race.c | 34 +---
.../bpf/prog_tests/tramp_prog_detach.c | 188 ++++++++++++++++++
.../selftests/bpf/progs/tramp_prog_detach.c | 56 ++++++
tools/testing/selftests/bpf/testing_helpers.c | 28 +++
tools/testing/selftests/bpf/testing_helpers.h | 2 +
14 files changed, 526 insertions(+), 156 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/tramp_prog_detach.c
create mode 100644 tools/testing/selftests/bpf/progs/tramp_prog_detach.c
base-commit: 5fc5768c7ca92895ccd1de94dc521e5a55ae7896
--
2.55.0