From nobody Sat Jul 25 22:31:43 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D25C31F9B7; Mon, 13 Jul 2026 02:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910658; cv=none; b=oyOjQFu7x39EAkEtuzgPbzREU9vgZh42Lue73Uw8XHXxLamM4f0P+3Tkw+e8gsouqjXrFeSK0+TcuA+LCewINf18ea3UWrMF1/0++8AbakT6w3/IO9w9IQEND9SlJ6D8lQwhB3q61HA5NyaSvHff2fcOEJnWf/gRiYeGCQ+2qbg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910658; c=relaxed/simple; bh=undR9u0NMDBm8Fk/FfEdax7WV4xMh3NFoI1kOYBGmQw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z90jEPGmc9aQKK7JivYJaRYMHZgfQsdaMN878Wq2cnxPR9PDZXiSsyVN+EgIVo4ZHLWihYzkWrnvrI4r0KF3NlVjJptl+qdlKMC1LqE9PeupZke0lEYbm/57RekoFYJkwndqIOJuKRs5rQ3V2OJuBrPFH4cKmA9j/yWBkx3zHtw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=isMeg9vB; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="isMeg9vB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E04261F00A3A; Mon, 13 Jul 2026 02:44:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783910657; bh=gzyRii9mw/0NcKb7ElmsK5Shp+DDYGHTpEmZBaBolpg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=isMeg9vB81wSI6j8U2MuDx+r/Ogc/RHNT0ydK0uz8eddd2xqaN2ph2r+moWq9UXVP iqyGRqrVDY33BzrjqJrehNpm2/fn7sHfArhxdUIGmZU/h/ls2o7CkzprjvI9moo77P p8UA7XKlOTke2RjyZPUcOnTOTEMHtP7LB1CAEZ0QTlPCdda+nKOFSH2rJrASpZbhxe dGK13yvOA5zyF0IkCp3vuGm1LtE7diD17tg12mUKU5KCX8O/j80+yZbz46JrE24MFF rE/B8S4/DLVBNvWZBjpcGhpdpnOlPch3jT1QH26d02ZW2UtHMB/Xc0QtELEzdpqbgt sb3f6mUi8iD4w== From: Tejun Heo To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Emil Tsalapatis , David Vernet , Andrea Righi , Changwoo Min , bpf@vger.kernel.org, sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCHSET SLOP RFC 1/6] bpf: Support __arena suffix for kfunc arguments Date: Sun, 12 Jul 2026 16:44:09 -1000 Message-ID: <20260713024414.3759854-2-tj@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260713024414.3759854-1-tj@kernel.org> References: <20260713024414.3759854-1-tj@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation. Add the __arena argument suffix and both steps go away. The kfunc declares the parameter by its real pointer type and dereferences it directly. The verifier rebases each one at the call site, rN =3D kern_vm_start + (u32)rN with NULL preserved. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments. Signed-off-by: Tejun Heo --- Documentation/bpf/kfuncs.rst | 22 +++++++++ include/linux/bpf_verifier.h | 2 + kernel/bpf/verifier.c | 86 +++++++++++++++++++++++++++++++++--- 3 files changed, 105 insertions(+), 5 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index c801a330aece..bad2ac85b318 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -250,6 +250,26 @@ This annotation is used to indicate that the argument = is a constant string. ... } =20 +2.3.7 __arena Annotation +------------------------ + +This annotation is used to indicate that the pointer argument points into +the program's arena. The verifier translates it to a directly +dereferenceable kernel address before the call, subject to the access +rules described in :ref:`BPF_kfunc_arena_access`. NULL is preserved and the +kfunc is responsible for checking for NULL before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map. The program +can pass any value without compromising the kernel. A value that does not +point into the arena is a program bug. + .. _BPF_kfunc_nodef: =20 2.4 Using an existing kernel function @@ -487,6 +507,8 @@ In order to accommodate such requirements, the verifier= will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. =20 +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments -------------------------------------------------- =20 diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 317e99b9acc0..2153d546736a 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -690,6 +690,8 @@ struct bpf_insn_aux_data { */ u8 fastcall_spills_num:3; u8 arg_prog:4; + /* bitmask of R1-R5 kfunc args to rebase to arena kernel addresses */ + u8 arg_arena_regs; =20 /* below fields are initialized once */ unsigned int orig_idx; /* original instruction index */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 03e2202cca13..b2b00a612e58 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10856,6 +10856,11 @@ static bool is_kfunc_arg_irq_flag(const struct btf= *btf, const struct btf_param return btf_param_match_suffix(btf, arg, "__irq_flag"); } =20 +static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_par= am *arg) +{ + return btf_param_match_suffix(btf, arg, "__arena"); +} + static bool is_kfunc_arg_scalar_with_name(const struct btf *btf, const struct btf_param *arg, const char *name) @@ -12060,6 +12065,32 @@ static int check_kfunc_args(struct bpf_verifier_en= v *env, struct bpf_kfunc_call_ continue; } =20 + if (is_kfunc_arg_arena(btf, &args[i])) { + t =3D btf_type_skip_modifiers(btf, args[i].type, NULL); + if (verifier_bug_if(!btf_type_is_ptr(t), env, + "kfunc %s arg#%d has __arena tag on non-pointer", + func_name, i)) + return -EFAULT; + if (!env->prog->aux->arena) { + verbose(env, + "%s arena pointer requires a program with an associated arena\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + if (regno < 0) { + verbose(env, "%s arena pointer cannot be a stack argument\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + if (reg->type !=3D PTR_TO_ARENA && reg->type !=3D SCALAR_VALUE) { + verbose(env, "%s is not a pointer to arena or scalar\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + cur_aux(env)->arg_arena_regs |=3D BIT(regno - BPF_REG_1); + continue; + } + if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i)) continue; =20 @@ -19890,13 +19921,58 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env = *env, struct bpf_insn *insn, *cnt =3D 6; } =20 - if (env->insn_aux_data[insn_idx].arg_prog) { - u32 regno =3D env->insn_aux_data[insn_idx].arg_prog; - struct bpf_insn ld_addrs[2] =3D { BPF_LD_IMM64(regno, (long)env->prog->a= ux) }; + if (env->insn_aux_data[insn_idx].arg_prog || + env->insn_aux_data[insn_idx].arg_arena_regs) { + unsigned long arena_regs =3D env->insn_aux_data[insn_idx].arg_arena_regs; + u64 kern_vm_start =3D bpf_arena_get_kern_vm_start(env->prog->aux->arena); + u32 prog_regno =3D env->insn_aux_data[insn_idx].arg_prog; int idx =3D *cnt; + int bit; =20 - insn_buf[idx++] =3D ld_addrs[0]; - insn_buf[idx++] =3D ld_addrs[1]; + if (verifier_bug_if(idx, env, "kfunc call at %d already patched", insn_i= dx)) + return -EFAULT; + + /* + * A kfunc may take both an __arena arg and the __prog aux. Emit + * each present fixup, then append the call once. + */ + if (arena_regs) { + bool blinding =3D env->prog->blinding_requested; + int base_reg =3D blinding ? BPF_REG_0 : BPF_REG_AX; + struct bpf_insn ld[2] =3D { BPF_LD_IMM64(base_reg, kern_vm_start) }; + + /* + * rN =3D kern_vm_start + (u32)rN, NULL left as 0. + * Constant blinding rewrites immediate-carrying insns + * unless they use BPF_REG_AX. With hardening, the + * constant sits in R0 (an AX-destined LD_IMM64 would be + * corrupted) and NULL is tested against AX =3D 0. Without + * hardening, use the cheaper AX constant and immediate + * test. + */ + insn_buf[idx++] =3D ld[0]; + insn_buf[idx++] =3D ld[1]; + if (blinding) + insn_buf[idx++] =3D BPF_MOV64_IMM(BPF_REG_AX, 0); + + for_each_set_bit(bit, &arena_regs, MAX_BPF_FUNC_REG_ARGS) { + int regno =3D BPF_REG_1 + bit; + + insn_buf[idx++] =3D BPF_ZEXT_REG(regno); + insn_buf[idx++] =3D blinding ? + BPF_JMP_REG(BPF_JEQ, regno, BPF_REG_AX, 1) : + BPF_JMP_IMM(BPF_JEQ, regno, 0, 1); + insn_buf[idx++] =3D BPF_ALU64_REG(BPF_ADD, regno, base_reg); + } + } + if (prog_regno) { + struct bpf_insn ld_addrs[2] =3D { + BPF_LD_IMM64(prog_regno, (long)env->prog->aux) + }; + + insn_buf[idx++] =3D ld_addrs[0]; + insn_buf[idx++] =3D ld_addrs[1]; + } insn_buf[idx++] =3D *insn; *cnt =3D idx; } --=20 2.55.0 From nobody Sat Jul 25 22:31:43 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90E4D371CE9; Mon, 13 Jul 2026 02:44:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910660; cv=none; b=I5SyvQbentT/aoMldPk44CofOY41tf4j+09BtLECVUkNr0TCeg290T82MBIuK4n+2K7TNpVi5nxqrkCMxDT4I6csdCSQDnDPM7Y1HWD8+gvpL7wTcQM7tMsq6rG8KE+6/umO93kYA1OgJIp/raMECrPm3/NifCR5lW1KtCc+2wo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910660; c=relaxed/simple; bh=r6eZKf1Py+AjIeeMe5G+N4judbNG+2UPCdSbWq1M6EE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gSKbqb97/gjNvd0iSBt1DrrqFtMIANB8boYMNc4xq73/G2t7+08oM4dlbGacgYHc8GiKzFg2VFOg1/qkcclQ7ueHh4QAuBFASK8pjh1hn8BkDWVPwJ28PpjCzR0i2H/KwzSdSvwifJBWYiPzwXkZwhn+2tdF7OllWK4ZOfyY7UQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nQUOO+yC; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nQUOO+yC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 04F011F000E9; Mon, 13 Jul 2026 02:44:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783910658; bh=RVcQpqbO4up4g0Hy0J+YgHqp9iQS+fAn5yaVmEvP0So=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=nQUOO+yC3Pp/M8nP23+TslF1FOmsaYNqEBdIuyO/LlnvQmR0cnO5TH9k3dGzkm7aV OG0m+Iy6sD57PH9i+yr+GwR3xNreIVB2LDt6QDhG0WtHf4W5zK8w80Z9gWQrfH9IMt 6jmwsfjak9tmqBwdAgXckiUt6zpPcciP7YnOAGIhSOtpgulDDf6ie8iovnILMRBONe KuMki/NETSUGukuRaPgFd6PqHipR40wFbNz7tFKEP9wdxAAg9zUffvk8TjixVnQ3hd 31E95M8Esujf7N99NfuzPzOq82X/s52Hnj2c0OjJH87a++BTmX2Dq3G246dbwBips9 sPLHQB01jpgog== From: Tejun Heo To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Emil Tsalapatis , David Vernet , Andrea Righi , Changwoo Min , bpf@vger.kernel.org, sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCHSET SLOP RFC 2/6] selftests/bpf: Add kfunc __arena argument tests Date: Sun, 12 Jul 2026 16:44:10 -1000 Message-ID: <20260713024414.3759854-3-tj@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260713024414.3759854-1-tj@kernel.org> References: <20260713024414.3759854-1-tj@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar, NULL), five arena arguments in one call with NULLs mixed in, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type). Signed-off-by: Tejun Heo --- .../selftests/bpf/prog_tests/arena_kfunc.c | 14 ++ .../testing/selftests/bpf/progs/arena_kfunc.c | 135 ++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 33 +++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 4 + 4 files changed, 186 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c diff --git a/tools/testing/selftests/bpf/prog_tests/arena_kfunc.c b/tools/t= esting/selftests/bpf/prog_tests/arena_kfunc.c new file mode 100644 index 000000000000..8946184553ea --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/arena_kfunc.c @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo */ +#include + +#include "arena_kfunc.skel.h" + +/* The test kfuncs live in bpf_testmod. Resolving kfuncs against module + * BTFs needs CAP_SYS_ADMIN, so run with full capabilities instead of + * through the verifier tests' capability-restricted runner. + */ +void test_arena_kfunc(void) +{ + RUN_TESTS(arena_kfunc); +} diff --git a/tools/testing/selftests/bpf/progs/arena_kfunc.c b/tools/testin= g/selftests/bpf/progs/arena_kfunc.c new file mode 100644 index 000000000000..b5996944ed73 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_kfunc.c @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo */ + +#define BPF_NO_KFUNC_PROTOTYPES +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" +#include +#include "../test_kmods/bpf_testmod_kfunc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 1); +} arena SEC(".maps"); + +/* volatile to force the scalar reloads below */ +volatile u64 stash; + +SEC("syscall") +__success __retval(0) +int arena_arg_forms(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + u64 ret; + + val =3D bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + /* PTR_TO_ARENA argument */ + *val =3D 41; + ret =3D bpf_kfunc_arena_arg_test((u64 *)val); + if (ret !=3D 41 || *val !=3D 42) + return 2; + + /* the low 32 bits as a scalar */ + stash =3D (u64)val; + ret =3D bpf_kfunc_arena_arg_test((u64 *)stash); + if (ret !=3D 42 || *val !=3D 43) + return 3; + + /* the full user address as a scalar */ + stash =3D (u64)val; + bpf_addr_space_cast(stash, 1, 0); + ret =3D bpf_kfunc_arena_arg_test((u64 *)stash); + if (ret !=3D 43 || *val !=3D 44) + return 4; + + /* NULL is preserved */ + ret =3D bpf_kfunc_arena_arg_test(NULL); + if (ret !=3D 0xdeadbeef) + return 5; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +SEC("syscall") +__success __retval(0) +int arena_args5(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + + val =3D bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + val[0] =3D 1; + val[1] =3D 2; + val[2] =3D 4; + val[3] =3D 8; + val[4] =3D 16; + + if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1], + (u64 *)&val[2], (u64 *)&val[3], + (u64 *)&val[4]) !=3D 31) + return 2; + + /* mix in NULLs */ + if (bpf_kfunc_arena_args5_test((u64 *)&val[0], NULL, (u64 *)&val[2], + NULL, (u64 *)&val[4]) !=3D 21) + return 3; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +/* kernel-side faults on unpopulated pages recover via the scratch page */ +SEC("syscall") +__success __retval(0) +int arena_arg_unpopulated(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ + (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) + u64 __arena *val; + + val =3D bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + stash =3D (u64)val + PAGE_SIZE; + bpf_kfunc_arena_arg_test((u64 *)stash); + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +SEC("syscall") +__failure __msg("arena pointer requires a program with an associated arena= ") +int arena_arg_no_arena(void *ctx) +{ + bpf_kfunc_arena_arg_test((u64 *)1); + return 0; +} + +SEC("syscall") +__failure __msg("is not a pointer to arena or scalar") +int arena_arg_bad_reg(void *ctx) +{ + u64 buf =3D 0; + + /* use the arena so the program passes the arena presence check */ + bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + bpf_kfunc_arena_arg_test(&buf); + return 0; +} + +char _license[] SEC("license") =3D "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/t= esting/selftests/bpf/test_kmods/bpf_testmod.c index 30f1cd23093c..9cc10c48a2d6 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -210,6 +210,37 @@ __bpf_kfunc void bpf_kfunc_common_test(void) { } =20 +__bpf_kfunc u64 bpf_kfunc_arena_arg_test(u64 *val__arena) +{ + u64 old; + + if (!val__arena) + return 0xdeadbeef; + + old =3D *val__arena; + *val__arena =3D old + 1; + return old; +} + +__bpf_kfunc u64 bpf_kfunc_arena_args5_test(u64 *a__arena, u64 *b__arena, + u64 *c__arena, u64 *d__arena, + u64 *e__arena) +{ + u64 sum =3D 0; + + if (a__arena) + sum +=3D *a__arena; + if (b__arena) + sum +=3D *b__arena; + if (c__arena) + sum +=3D *c__arena; + if (d__arena) + sum +=3D *d__arena; + if (e__arena) + sum +=3D *e__arena; + return sum; +} + __bpf_kfunc void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr, struct bpf_dynptr *ptr__nullable) { @@ -723,6 +754,8 @@ BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_N= EXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_iter_testmod_seq_value) BTF_ID_FLAGS(func, bpf_kfunc_common_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_arg_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_args5_test) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_dynptr_test) BTF_ID_FLAGS(func, bpf_kfunc_nested_acquire_nonzero_offset_test, KF_ACQUIR= E) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/t= ools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index c36bb911defa..932acf86c73f 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -98,6 +98,10 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kf= unc *p) __ksym; void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym; =20 void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym; +__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym; +__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena, + __u64 *c__arena, __u64 *d__arena, + __u64 *e__arena) __ksym; int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const= int rdwr_buf_size) __ksym; int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, con= st int rdonly_buf_size) __ksym; int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, con= st int rdonly_buf_size) __ksym; --=20 2.55.0 From nobody Sat Jul 25 22:31:43 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B25C372EFA; Mon, 13 Jul 2026 02:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910662; cv=none; b=lgZNHC/MaDeZ3+WK+/4ed5CNWqWBirzsSxbkpXydH3Y2kVIIAfanQZXrnkJf4gdUHYdKzLwUAO8U/h406B28Xw4itW+3q8CUQUl6mafq6il+EAMCvrn8U8Cp442Yys24uRHJUWyWcWZBEB3er63zpdI/h0QrreOCwpPDg7c2yW8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910662; c=relaxed/simple; bh=Y9+cwQQeNxjog3uzK5caVDzDAsbLvZ9nr4LrJufZAxg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fT8lTxebpGLF0m/HVCRSsCUAkJOvQ/ZBmxqgjF3zgCN5R9hR4KrGz7vIA0R65udFeEYA5+iQAQuIosg+ZlNl+Slsg4zntYEX3Gx1urDdYUs5NX9p78ssmmB4V+UPoZKYAfiTn9PM1gb0SKhm1YgVeU7F+5i0VSJs1ZWAP0B6JPw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Fa+f1YRy; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Fa+f1YRy" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 201A81F00A3A; Mon, 13 Jul 2026 02:44:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783910659; bh=Ui0Ha97l+bh1ebfuqnlTP5ETnG66Xz70Sx4zFAQWnSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Fa+f1YRyWyj7gRC4+MiJrwHDvy/JagMrqKIOUk4goZnVNM/G+Dgo1XX7msBgJ4cf6 owUObGRgghpp1k/CeQL/aMebxrhe+SFbgQFQ7d7t7FMEae+ApT0n4uEof/wSawOVxZ ULuHCNceVBtTCybg5PVpE7wrlUqfSRWFkkVORYt/jc/Kgi4ZEi5ruR0PqnD0jyiT8B kVuPnUi6q/mJgx+1JKoozGU+eNbxRn4AXDlIf2VGR+LTPI6HlHfwG2f+BdroI5YNAe Y4/8HwFwv5lpVvnYs7vB9S5MsKwojDsBIavIUqTLv6HibNPbTXZAN9rLJr249R1ZPF OVF76Q7VLndnw== From: Tejun Heo To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Emil Tsalapatis , David Vernet , Andrea Righi , Changwoo Min , bpf@vger.kernel.org, sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCHSET SLOP RFC 3/6] bpf: Support __arena suffix on struct_ops stub arguments Date: Sun, 12 Jul 2026 16:44:11 -1000 Message-ID: <20260713024414.3759854-4-tj@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260713024414.3759854-1-tj@kernel.org> References: <20260713024414.3759854-1-tj@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in. Add the __arena stub argument suffix and both steps go away. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). An entry prologue injected into the program rebases each such ctx slot once, arena =3D (u32)(kaddr - kern_vm_start) with NULL preserved, keyed by argument position. The rebased value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address. Signed-off-by: Tejun Heo --- Documentation/bpf/kfuncs.rst | 5 +++ kernel/bpf/bpf_struct_ops.c | 17 ++++++--- kernel/bpf/btf.c | 10 ++++-- kernel/bpf/fixups.c | 70 ++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 17 ++++++--- 5 files changed, 106 insertions(+), 13 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index bad2ac85b318..84726f5830c5 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -270,6 +270,11 @@ Calling such a kfunc requires the program to use an ar= ena map. The program can pass any value without compromising the kernel. A value that does not point into the arena is a program bug. =20 +The suffix has the same meaning on the arguments of struct_ops stub +functions. The kernel caller passes the kernel arena address and the +argument arrives in the BPF program as an arena pointer that can be +dereferenced directly, with NULL preserved. + .. _BPF_kfunc_nodef: =20 2.4 Using an existing kernel function diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 51b16e5f5534..a702fb39a238 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -147,6 +147,7 @@ void bpf_struct_ops_image_free(void *image) =20 #define MAYBE_NULL_SUFFIX "__nullable" #define REFCOUNTED_SUFFIX "__ref" +#define ARENA_SUFFIX "__arena" =20 /* Prepare argument info for every nullable argument of a member of a * struct_ops type. @@ -159,7 +160,7 @@ void bpf_struct_ops_image_free(void *image) * to provide an array of struct bpf_ctx_arg_aux, which in turn provides * the information that used by the verifier to check the arguments of the * BPF struct_ops program assigned to the member. Here, we only care about - * the arguments that are marked as __nullable. + * the arguments that are marked as __nullable, __ref or __arena. * * The array of struct bpf_ctx_arg_aux is eventually assigned to * prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the @@ -175,7 +176,7 @@ static int prepare_arg_info(struct btf *btf, struct bpf_struct_ops_arg_info *arg_info) { const struct btf_type *stub_func_proto, *pointed_type; - bool is_nullable =3D false, is_refcounted =3D false; + bool is_nullable =3D false, is_refcounted =3D false, is_arena =3D false; const struct btf_param *stub_args, *args; struct bpf_ctx_arg_aux *info, *info_buf; u32 nargs, arg_no, info_cnt =3D 0; @@ -226,26 +227,30 @@ static int prepare_arg_info(struct btf *btf, info =3D info_buf; for (arg_no =3D 0; arg_no < nargs; arg_no++) { /* Skip arguments that is not suffixed with - * "__nullable or __ref". + * "__nullable", "__ref" or "__arena". */ is_nullable =3D btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX); is_refcounted =3D btf_param_match_suffix(btf, &stub_args[arg_no], REFCOUNTED_SUFFIX); + is_arena =3D btf_param_match_suffix(btf, &stub_args[arg_no], + ARENA_SUFFIX); =20 if (is_nullable) suffix =3D MAYBE_NULL_SUFFIX; else if (is_refcounted) suffix =3D REFCOUNTED_SUFFIX; + else if (is_arena) + suffix =3D ARENA_SUFFIX; else continue; =20 - /* Should be a pointer to struct */ + /* Should be a pointer to struct, or any pointer for __arena */ pointed_type =3D btf_type_resolve_ptr(btf, args[arg_no].type, &arg_btf_id); if (!pointed_type || - !btf_type_is_struct(pointed_type)) { + (!is_arena && !btf_type_is_struct(pointed_type))) { pr_warn("stub function %s has %s tagging to an unsupported type\n", stub_fname, suffix); goto err_out; @@ -273,6 +278,8 @@ static int prepare_arg_info(struct btf *btf, } else if (is_refcounted) { info->reg_type =3D PTR_TRUSTED | PTR_TO_BTF_ID; info->refcounted =3D true; + } else if (is_arena) { + info->reg_type =3D PTR_TO_ARENA; } =20 info++; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 8c04c340f499..c6349915dc1f 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6961,15 +6961,19 @@ bool btf_ctx_access(int off, int size, enum bpf_acc= ess_type type, return false; } =20 - /* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */ + /* + * Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or + * PTR_TO_ARENA. + */ for (i =3D 0; i < prog->aux->ctx_arg_info_size; i++) { const struct bpf_ctx_arg_aux *ctx_arg_info =3D &prog->aux->ctx_arg_info[= i]; u32 type, flag; =20 type =3D base_type(ctx_arg_info->reg_type); flag =3D type_flag(ctx_arg_info->reg_type); - if (ctx_arg_info->offset =3D=3D off && type =3D=3D PTR_TO_BUF && - (flag & PTR_MAYBE_NULL)) { + if (ctx_arg_info->offset =3D=3D off && + (type =3D=3D PTR_TO_ARENA || + (type =3D=3D PTR_TO_BUF && (flag & PTR_MAYBE_NULL)))) { info->reg_type =3D ctx_arg_info->reg_type; return true; } diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index d3be972714b2..fde39088e4d0 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -680,6 +680,64 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verif= ier_env *env, return 0; } =20 +/* + * Rebase every PTR_TO_ARENA struct_ops ctx argument to the arena pointer + * form once at entry: arena =3D (u32)(kaddr - kern_vm_start), NULL preser= ved. + * Keyed by argument position (ctx_arg_info.offset), so the source ctx slot + * is converted once and every later load sees it. + * + * Runs post-verification, so the ctx store and the kern_vm_start constant + * need no re-verification. Constant blinding (on under bpf_jit_harden) + * forces the safe form: the constant in R0 (an AX-destined LD_IMM64 would + * be corrupted), NULL tested against BPF_REG_AX =3D 0. Off, the cheaper AX + * constant and immediate test are used. + * + * Return the prologue length, INSN_BUF_SIZE if it would not fit, or 0 if + * there is nothing to do. + */ +static int gen_arena_arg_prologue(struct bpf_verifier_env *env, + struct bpf_insn *insn_buf) +{ + struct bpf_prog_aux *aux =3D env->prog->aux; + bool blinding =3D env->prog->blinding_requested; + int base_reg =3D blinding ? BPF_REG_0 : BPF_REG_AX; + struct bpf_insn ld[2] =3D { + BPF_LD_IMM64(base_reg, bpf_arena_get_kern_vm_start(aux->arena)) + }; + int i, cnt =3D 0, nslots =3D 0; + + for (i =3D 0; i < aux->ctx_arg_info_size; i++) + if (base_type(aux->ctx_arg_info[i].reg_type) =3D=3D PTR_TO_ARENA) + nslots++; + if (!nslots) + return 0; + /* LD_IMM64(2) + MOV(1) + 5 per slot + chained entry insn(1) */ + if (4 + 5 * nslots > INSN_BUF_SIZE) + return INSN_BUF_SIZE; + + insn_buf[cnt++] =3D ld[0]; + insn_buf[cnt++] =3D ld[1]; + if (blinding) + insn_buf[cnt++] =3D BPF_MOV64_IMM(BPF_REG_AX, 0); + + for (i =3D 0; i < aux->ctx_arg_info_size; i++) { + int off; + + if (base_type(aux->ctx_arg_info[i].reg_type) !=3D PTR_TO_ARENA) + continue; + off =3D aux->ctx_arg_info[i].offset; + insn_buf[cnt++] =3D BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, off); + insn_buf[cnt++] =3D blinding ? + BPF_JMP_REG(BPF_JEQ, BPF_REG_2, BPF_REG_AX, 2) : + BPF_JMP_IMM(BPF_JEQ, BPF_REG_2, 0, 2); + insn_buf[cnt++] =3D BPF_ALU64_REG(BPF_SUB, BPF_REG_2, base_reg); + insn_buf[cnt++] =3D BPF_ZEXT_REG(BPF_REG_2); + insn_buf[cnt++] =3D BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_2, off); + } + insn_buf[cnt++] =3D env->prog->insnsi[0]; + return cnt; +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -749,6 +807,18 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *= env) } } =20 + cnt =3D gen_arena_arg_prologue(env, insn_buf); + if (cnt >=3D INSN_BUF_SIZE) { + verifier_bug(env, "arena arg prologue is too long"); + return -EFAULT; + } else if (cnt) { + new_prog =3D bpf_patch_insn_data(env, 0, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + env->prog =3D new_prog; + delta +=3D cnt - 1; + } + if (delta) WARN_ON(adjust_jmp_off(env->prog, 0, delta)); =20 diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b2b00a612e58..a70f66ef227a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18789,6 +18789,7 @@ static int check_struct_ops_btf_id(struct bpf_verif= ier_env *env) { const struct btf_type *t, *func_proto; const struct bpf_struct_ops_desc *st_ops_desc; + const struct bpf_struct_ops_arg_info *arg_info; const struct bpf_struct_ops *st_ops; const struct btf_member *member; struct bpf_prog *prog =3D env->prog; @@ -18867,10 +18868,17 @@ static int check_struct_ops_btf_id(struct bpf_ver= ifier_env *env) return -EACCES; } =20 - for (i =3D 0; i < st_ops_desc->arg_info[member_idx].cnt; i++) { - if (st_ops_desc->arg_info[member_idx].info[i].refcounted) { + arg_info =3D &st_ops_desc->arg_info[member_idx]; + for (i =3D 0; i < arg_info->cnt; i++) { + const struct bpf_ctx_arg_aux *info =3D &arg_info->info[i]; + + if (info->refcounted) has_refcounted_arg =3D true; - break; + if (base_type(info->reg_type) =3D=3D PTR_TO_ARENA && !prog->aux->arena) { + verbose(env, + "arena argument of %s requires a program with an associated arena\n", + mname); + return -EINVAL; } } =20 @@ -18891,8 +18899,7 @@ static int check_struct_ops_btf_id(struct bpf_verif= ier_env *env) prog->aux->attach_func_name =3D mname; env->ops =3D st_ops->verifier_ops; =20 - return bpf_prog_ctx_arg_info_init(prog, st_ops_desc->arg_info[member_idx]= .info, - st_ops_desc->arg_info[member_idx].cnt); + return bpf_prog_ctx_arg_info_init(prog, arg_info->info, arg_info->cnt); } #define SECURITY_PREFIX "security_" =20 --=20 2.55.0 From nobody Sat Jul 25 22:31:43 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84E4436654C; Mon, 13 Jul 2026 02:44:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910662; cv=none; b=hp5g9e+3Brh9Cdiow3Zw3YkRfx5tAnm14KSbB8xMIkY9ZdOhox7sGwpLA+lEaBkAuXJvdXuIMg6vt7OvoNkoUWnKG7hCq3zHrSXDTGKBRYCYArzdBx1OfAXMj/hUOI7x9TIBMtQZziMsw5S2ARoKrLFukIXkIRQ+moy7yypJJuE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910662; c=relaxed/simple; bh=UVd+SUQL+iS7YX0Uk4vjcQu++9x2mwBuyZbeCHa3ysQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VO5tms7Z4cuEZNG747LXBNQW3WIog7h70ZZ+gTjwk8s7NjIzDMFY7oCQ2yuZA6lo0unHcbnHpzJYZFTJgOP3UJ2hQZKsKx0/rB8atIhPiaPm3x50G4nFIN8t/XXCqAObW92GVX3dJJ98aYxW7S8YeI5s2uO4Yj4AIc8YaN4cm2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KvrGkvhv; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KvrGkvhv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C6871F000E9; Mon, 13 Jul 2026 02:44:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783910660; bh=Tx7yR2phZnd2rp1sOS4g3o0yHZkztimrGxZ5dj5pMz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=KvrGkvhvMcEjm8iA+vMqPyVLIefN9h7W57L7nxtpS06o5nPlp38JbpYSwNsD/Thnd Bj/JYkNXwhv9vhrBFZJgOPfy/m/1BlXmvv9nT5l+Po4tVnK2ofuKJCtgPOWQtbwccH 7pzDonUErfMrdKZVl3Myq3jPpCXA793lvy7vNShtKWeGtbYCYDvqRNAMUtgNvGTJGo S2PvJ0hK/lqKCj5a9UqrZFyIJ2YSrgBXd8/i+qDozEtof+DR8XSfUQTpLWkf/3EsV5 2pbOWxX98vSWWjoYTsqrQhTdagHuIb0+TxTSWzfz6c678TqIXin52uhFfHsM1d24Uj LM3uq9twPIRfQ== From: Tejun Heo To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Emil Tsalapatis , David Vernet , Andrea Righi , Changwoo Min , bpf@vger.kernel.org, sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCHSET SLOP RFC 4/6] selftests/bpf: Add struct_ops __arena argument tests Date: Sun, 12 Jul 2026 16:44:12 -1000 Message-ID: <20260713024414.3759854-5-tj@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260713024414.3759854-1-tj@kernel.org> References: <20260713024414.3759854-1-tj@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a test_arena member to bpf_testmod_ops3 with an __arena stub argument and a kfunc that forwards a caller-provided pointer to it. The kfunc takes an __arena argument, so the verifier hands it a kernel arena address, which it passes to the callback. The struct_ops entry prologue rebases that address back to an arena pointer for the callback, so the test exercises both conversion directions end to end. The callback dereferences the argument directly and the trigger program verifies the write and NULL passing. A second program with no arena verifies the attach rejection. Signed-off-by: Tejun Heo --- .../bpf/prog_tests/test_struct_ops_arena.c | 61 +++++++++++++++++ .../selftests/bpf/progs/struct_ops_arena.c | 67 +++++++++++++++++++ .../bpf/progs/struct_ops_arena_fail.c | 20 ++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 12 ++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 2 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 1 + 6 files changed, 163 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_= arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail= .c diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c= b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file mode 100644 index 000000000000..75a3a649060c --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo */ +#include + +#include "struct_ops_arena.skel.h" +#include "struct_ops_arena_fail.skel.h" + +/* Attach a callback with an __arena argument and drive it through the + * bpf_testmod_ops3_call_test_arena() kfunc. + */ +static void arena_arg(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct struct_ops_arena *skel; + struct bpf_link *link =3D NULL; + int err; + + skel =3D struct_ops_arena__open_and_load(); + if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load")) + return; + + link =3D bpf_map__attach_struct_ops(skel->maps.testmod_arena); + if (!ASSERT_OK_PTR(link, "attach_struct_ops")) + goto out; + + err =3D bpf_prog_test_run_opts(bpf_program__fd(skel->progs.trigger), + &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "trigger_retval"); + +out: + bpf_link__destroy(link); + struct_ops_arena__destroy(skel); +} + +/* A program with no arena cannot attach to a member with an __arena + * argument. + */ +static void arena_arg_fail(void) +{ + struct struct_ops_arena_fail *skel; + + skel =3D struct_ops_arena_fail__open_and_load(); + if (ASSERT_ERR_PTR(skel, "struct_ops_arena_fail__open_and_load")) + return; + + struct_ops_arena_fail__destroy(skel); +} + +/* + * Serialized because it attaches the singleton bpf_testmod_ops3, which + * test_struct_ops_private_stack also attaches; registering it twice fails + * with -EEXIST. + */ +void serial_test_struct_ops_arena(void) +{ + if (test__start_subtest("arena_arg")) + arena_arg(); + if (test__start_subtest("arena_arg_fail")) + arena_arg_fail(); +} diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena.c b/tools/t= esting/selftests/bpf/progs/struct_ops_arena.c new file mode 100644 index 000000000000..b748009f94bc --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena.c @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo */ + +#define BPF_NO_KFUNC_PROTOTYPES +#include +#include +#include "bpf_experimental.h" +#include +#include "../test_kmods/bpf_testmod.h" +#include "../test_kmods/bpf_testmod_kfunc.h" + +char _license[] SEC("license") =3D "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + /* page 0 hosts the arena globals, page 1 is for allocations */ + __uint(max_entries, 2); +} arena SEC(".maps"); + +/* also associates the callback with the arena */ +u64 __arena arena_touch; + +SEC("struct_ops/test_arena") +int test_arena_cb(unsigned long long *ctx) +{ + u64 __arena *ptr =3D (u64 __arena *)ctx[0]; + + arena_touch++; + if (!ptr) + return 0xbee; + *ptr +=3D 1; + return 0; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops3 testmod_arena =3D { + .test_arena =3D (void *)test_arena_cb, +}; + +SEC("syscall") +int trigger(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + int ret; + + val =3D bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + *val =3D 41; + ret =3D bpf_testmod_ops3_call_test_arena((u64 *)val); + if (ret) + return 2; + if (*val !=3D 42) + return 3; + + /* NULL reaches the callback as NULL */ + ret =3D bpf_testmod_ops3_call_test_arena(NULL); + if (ret !=3D 0xbee) + return 4; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c b/to= ols/testing/selftests/bpf/progs/struct_ops_arena_fail.c new file mode 100644 index 000000000000..4487f025018e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Tejun Heo */ + +#include +#include +#include "../test_kmods/bpf_testmod.h" + +char _license[] SEC("license") =3D "GPL"; + +/* No arena in the program: attaching to test_arena must be rejected. */ +SEC("struct_ops/test_arena") +int test_arena_no_arena(unsigned long long *ctx) +{ + return 0; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops3 testmod_arena_fail =3D { + .test_arena =3D (void *)test_arena_no_arena, +}; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/t= esting/selftests/bpf/test_kmods/bpf_testmod.c index 9cc10c48a2d6..6930eb1d48dc 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -351,9 +351,15 @@ static int bpf_testmod_test_4(void) return 0; } =20 +static int bpf_testmod_ops3__test_arena(u64 *ptr__arena) +{ + return 0; +} + static struct bpf_testmod_ops3 __bpf_testmod_ops3 =3D { .test_1 =3D bpf_testmod_test_3, .test_2 =3D bpf_testmod_test_4, + .test_arena =3D bpf_testmod_ops3__test_arena, }; =20 static void bpf_testmod_test_struct_ops3(void) @@ -372,6 +378,11 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) st_ops3->test_2(); } =20 +__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena) +{ + return st_ops3->test_arena(ptr__arena); +} + struct bpf_testmod_btf_type_tag_1 { int a; }; @@ -771,6 +782,7 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE |= KF_RET_NULL) BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena) BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test); BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test); BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h b/tools/t= esting/selftests/bpf/test_kmods/bpf_testmod.h index 863fd10f1619..49f9993ec331 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h @@ -106,6 +106,8 @@ struct bpf_testmod_ops2 { struct bpf_testmod_ops3 { int (*test_1)(void); int (*test_2)(void); + /* Used to test arena pointer arguments. */ + int (*test_arena)(u64 *ptr); }; =20 struct st_ops_args { diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/t= ools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index 932acf86c73f..bc2bbe0c8168 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -115,6 +115,7 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 = unused) __ksym; #endif =20 void bpf_testmod_test_mod_kfunc(int i) __ksym; +int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; =20 __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym; --=20 2.55.0 From nobody Sat Jul 25 22:31:43 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A408335839E; Mon, 13 Jul 2026 02:44:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910667; cv=none; b=iMuZQ63J2ZoTbn+2UqWKeKKGFgtShB3I+L72aP6iRZ8vLMXYA7NYjfCr5OG9/rpptYq0DJ9q8P6+0fh3qRDewqDQ5u/P9YKJl3IyQrP1XPiOZMbcoE9ldmQmah8eIhBE+efQD8SsRPVkwthQzlJyiuGcpDZeVVOReYVCpbI73xM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910667; c=relaxed/simple; bh=FbpYoFh9G9VSzEc+Y/xpfoSwukry1AwKLR+QtxmshNY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qhqG1WxoqOd3Jk5Yhkx4+s8bhcD0XyaKPrINAL0QAzBknay7hge0REBvEYiMpw32R4vmMAGZLHtsIvRpz+EWn79YepIEvD2JIOAWZ/jmbByDgYMRzhrUY5rMFgInRRWP/9glDk/L0lFmWL48w3SCRd+52245kfQRJYgqjbeXtqk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZrVr7T9p; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZrVr7T9p" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 533BE1F00A3D; Mon, 13 Jul 2026 02:44:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783910661; bh=zRcmOIyp+566PtKrOBLOLXjFc150jF/pYMntl7ewiTs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ZrVr7T9pupfrrYmyFA2JPj5uWf1DsD427vZUnZGHURWpVz6MhEn4Tx9JtP9/QTC3n w3BZivFRXcZOsDFQ7RGElncGEpeOpM5Bovsx5l/kTyQCY7F0yTXbDuISbyNlcnrZQn 0s9fknnuhZqkbQzZ/yMFpZ2LWrGTYmvyn5JgE0heAECcsPJhGhTM0al6wqWrVZNQC9 IGz9pL/ATE2yRNdYjLErVpdJ5nB4v31r3raC+R8ZbG06c8T2HIFZgLacMX5QT/BkuJ U0Xwwx6x6amdtX7U1PWAD1AhgIU1DlYNg972MIymKWl0iaYcIUiDY2zxjLh3Rk40a1 8lt+RKGffSDiA== From: Tejun Heo To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Emil Tsalapatis , David Vernet , Andrea Righi , Changwoo Min , bpf@vger.kernel.org, sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCHSET SLOP RFC 5/6] sched_ext: Pass a kernel arena pointer to ops_cid.set_cmask() Date: Sun, 12 Jul 2026 16:44:13 -1000 Message-ID: <20260713024414.3759854-6-tj@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260713024414.3759854-1-tj@kernel.org> References: <20260713024414.3759854-1-tj@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The cid-form set_cmask() callback receives a per-CPU cmask that the kernel builds in the arena. Without an __arena tag the callback would receive the argument as a trusted scx_cmask BTF pointer, and the kernel had to convert the kernel address to the arena pointer form by hand before the call. Tag the stub argument __arena and pass the kernel arena address directly. The struct_ops entry prologue rebases it to the program's arena pointer, so the manual scx_kaddr_to_arena() conversion and its now-unused helper are removed. NOT_SIGNED_OFF: to be reworked after bpf-next is pulled into sched_ext --- kernel/sched/ext/ext.c | 11 ++++++----- kernel/sched/ext/internal.h | 16 +++------------- 2 files changed, 9 insertions(+), 18 deletions(-) diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index 691d53fe0f64..c38a0e62534a 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -508,12 +508,13 @@ static inline void scx_call_op_set_cpumask(struct scx= _sched *sch, struct rq *rq, if (scx_is_cid_type()) { struct scx_cmask *kern_va =3D *this_cpu_ptr(sch->set_cmask_scratch); /* - * Build the per-CPU arena cmask and hand BPF its arena address. - * Caller holds the rq lock with IRQs disabled, which makes us - * the sole user of the scratch area. + * Build the per-CPU arena cmask and pass its kernel address. + * The BPF side translates it to an arena pointer. Caller holds + * the rq lock with IRQs disabled, which makes us the sole user + * of the scratch area. */ scx_cpumask_to_cmask(cpumask, kern_va); - sch->ops_cid.set_cmask(task, scx_kaddr_to_arena(sch, kern_va)); + sch->ops_cid.set_cmask(task, kern_va); } else { sch->ops.set_cpumask(task, cpumask); } @@ -8130,7 +8131,7 @@ static struct bpf_struct_ops bpf_sched_ext_ops =3D { * set_cmask needs a fresh stub since the second argument type differs. */ static void sched_ext_ops_cid__set_cmask(struct task_struct *p, - const struct scx_cmask *cmask) {} + const struct scx_cmask *cmask__arena) {} =20 static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid =3D { .select_cid =3D sched_ext_ops__select_cpu, diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h index 145272cb4d8a..a2db2e1a78d5 100644 --- a/kernel/sched/ext/internal.h +++ b/kernel/sched/ext/internal.h @@ -1125,8 +1125,7 @@ struct scx_sched { * * @arena_pool sub-allocates @arena_map. Each gen_pool chunk is added * at the kernel-side mapping address. @arena_kern_base is the start - * of the arena's kern_vm range. See scx_arena_to_kaddr() and - * scx_kaddr_to_arena(). + * of the arena's kern_vm range. See scx_arena_to_kaddr(). */ struct bpf_map *arena_map; struct gen_pool *arena_pool; @@ -1135,7 +1134,8 @@ struct scx_sched { /* * Per-CPU arena cmask used by scx_call_op_set_cpumask() to hand a cmask * to ops_cid.set_cmask(). The kernel writes through the stored kern_va - * and hands BPF its arena pointer via scx_kaddr_to_arena(). + * and passes it as the callback argument; the struct_ops entry prologue + * rebases it to the program's arena pointer. */ struct scx_cmask * __percpu *set_cmask_scratch; =20 @@ -1227,16 +1227,6 @@ static inline void *scx_arena_to_kaddr(struct scx_sc= hed *sch, const void *bpf_pt return (void *)(sch->arena_kern_base + (u32)(uintptr_t)bpf_ptr); } =20 -/** - * scx_kaddr_to_arena - Translate a kernel arena address to its BPF form - * @sch: scheduler whose arena hosts @kaddr - * @kaddr: kernel-side arena address, supplied by trusted kernel code - */ -static inline void *scx_kaddr_to_arena(struct scx_sched *sch, const void *= kaddr) -{ - return (void *)((uintptr_t)kaddr - sch->arena_kern_base); -} - enum scx_wake_flags { /* expose select WF_* flags as enums */ SCX_WAKE_FORK =3D WF_FORK, --=20 2.55.0 From nobody Sat Jul 25 22:31:43 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B001B3749EE; Mon, 13 Jul 2026 02:44:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910668; cv=none; b=SPhtv1wGH7FrnLDnbgiHvPGbZ4VmjeDZ+1bd4/lpGF6auQCcIWbQPAmCM6omZuQkgJsqhvsRIKFPhkT7UAcK/+06jXXKF3WcHl34cOZ1FThdCL/LSi6doFlsXT1T4o1zdEyh91z4MGpM89LFumSas/H9hCGLiHEawB9RUxbh2ow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783910668; c=relaxed/simple; bh=CBKN/vIJkik/NCshPrweB5B3z/nZRf/LL7xyjVWJs8M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HLNm53zbs6CbE0MonZLMw4s+0buZLxw9d6dmMqMsrOQA9IrQjRQpqSIlOAoSczDhmY21rXKJGcemErQ98c7Cl18npI7lU0FvztFnE9felADGtkBS9ghG7+Qz+2/ItixYUJMh9Kiw3XpWDPQptVpQcfQeq49mcGL/WEgeZ8GpZOM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kgEznwUO; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kgEznwUO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 691FD1F00A3A; Mon, 13 Jul 2026 02:44:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783910662; bh=6gsSLeWdlBEt6VLVr/FElPNMEYeTgT09BD00BynCXE8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=kgEznwUOZHU9mL7wWtq33S1o9e1HDJecu4/vnoXoInof1U5qg04oVdjEsTA0YQDpU 6JXfOL+v+S7smcONVreOPkWsuBvzm14debSoRdVgCc/JxLQkJAskOxMMmS9uiS6eLt 5cUXtJpdRJSXeM32LeB3UOvg/YKnRtx42t7jAcJUHNP592ZBfhP58hYh/JGje4rPjZ 57Y8bpBrVjz1NSy70L8mva5wmQU5NywveaHz54OgmLTkJxyHYsNLYxhdmAXoVzkXvH ml/8zLgrACG0B81hfJmAhd2lux9VB2njOyuCVrHVsP/2pQlm+w3fdVwJh6HRXj1+iz wIG9o6zj0WwyA== From: Tejun Heo To: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Kumar Kartikeya Dwivedi Cc: Martin KaFai Lau , Emil Tsalapatis , David Vernet , Andrea Righi , Changwoo Min , bpf@vger.kernel.org, sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCHSET SLOP RFC 6/6] sched_ext: Convert scx_bpf_cid_override() to take an arena pointer Date: Sun, 12 Jul 2026 16:44:14 -1000 Message-ID: <20260713024414.3759854-7-tj@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260713024414.3759854-1-tj@kernel.org> References: <20260713024414.3759854-1-tj@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The mem+size signature requires a verifier-visible buffer, which forces scx_qmap to keep the cpu_to_cid array in writable bss even though nothing writes it after load. Take the array as an __arena argument instead, with the count passed in entries. The verifier hands the kfunc a dereferenceable kernel address and nr_cpu_ids * sizeof(s32) stays within the guard region covered by arena fault recovery, so the kfunc needs no bounds check beyond the entry count validation. scx_qmap moves the array into struct qmap_arena. As the arena is mmapped at load, the loader now populates it between load and attach instead of before load. NOT_SIGNED_OFF: to be reworked after bpf-next is pulled into sched_ext --- kernel/sched/ext/cid.c | 24 ++++++++++---- tools/sched_ext/include/scx/compat.bpf.h | 8 +++-- tools/sched_ext/scx_qmap.bpf.c | 13 ++------ tools/sched_ext/scx_qmap.c | 41 +++++++++++++++--------- tools/sched_ext/scx_qmap.h | 3 ++ 5 files changed, 53 insertions(+), 36 deletions(-) diff --git a/kernel/sched/ext/cid.c b/kernel/sched/ext/cid.c index af83084ec740..872f699cb1ac 100644 --- a/kernel/sched/ext/cid.c +++ b/kernel/sched/ext/cid.c @@ -278,8 +278,8 @@ __bpf_kfunc_start_defs(); =20 /** * scx_bpf_cid_override - Install an explicit cpu->cid mapping - * @cpu_to_cid: array of nr_cpu_ids s32 entries (cid for each cpu) - * @cpu_to_cid__sz: must be nr_cpu_ids * sizeof(s32) bytes + * @cpu_to_cid__arena: arena array of nr_cpu_ids s32 entries (cid for each= cpu) + * @cnt: number of entries, must be nr_cpu_ids * @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs * * May only be called from ops.init() of the root scheduler. Replace the @@ -287,7 +287,7 @@ __bpf_kfunc_start_defs(); * must map to a unique cid in [0, num_possible_cpus()). Topo info is clea= red. * On invalid input, trigger scx_error() to abort the scheduler. */ -__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_ci= d__sz, +__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid__arena, u32 cn= t, const struct bpf_prog_aux *aux) { cpumask_var_t seen __free(free_cpumask_var) =3D CPUMASK_VAR_NULL; @@ -314,14 +314,24 @@ __bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_= to_cid, u32 cpu_to_cid__sz, return; } =20 - if (cpu_to_cid__sz !=3D nr_cpu_ids * sizeof(s32)) { - scx_error(sch, "scx_bpf_cid_override: expected %zu bytes, got %u", - nr_cpu_ids * sizeof(s32), cpu_to_cid__sz); + if (!cpu_to_cid__arena) { + scx_error(sch, "scx_bpf_cid_override: NULL cpu_to_cid"); return; } =20 + if (cnt !=3D nr_cpu_ids) { + scx_error(sch, "scx_bpf_cid_override: expected %u entries, got %u", + nr_cpu_ids, cnt); + return; + } + + /* + * @cpu_to_cid__arena arrives rebased to the arena kernel mapping. + * nr_cpu_ids * sizeof(s32) stays within the guard region covered by + * arena fault recovery, so no explicit bounds check is needed. + */ for_each_possible_cpu(cpu) { - s32 c =3D cpu_to_cid[cpu]; + s32 c =3D cpu_to_cid__arena[cpu]; =20 if (!cid_valid(sch, c)) return; diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/inc= lude/scx/compat.bpf.h index 87f15f296234..f71f843f9ddb 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -7,6 +7,8 @@ #ifndef __SCX_COMPAT_BPF_H #define __SCX_COMPAT_BPF_H =20 +#include "bpf_arena_common.bpf.h" + #define __COMPAT_ENUM_OR_ZERO(__type, __ent) \ ({ \ __type __ret =3D 0; \ @@ -125,12 +127,12 @@ static inline bool scx_bpf_sub_dispatch(u64 cgroup_id) * v7.2: scx_bpf_cid_override() for explicit cpu->cid mapping. Ignore if * missing. */ -void scx_bpf_cid_override___compat(const s32 *cpu_to_cid, u32 cpu_to_cid__= sz) __ksym __weak; +void scx_bpf_cid_override___compat(const s32 __arena *cpu_to_cid, u32 cnt)= __ksym __weak; =20 -static inline void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_= cid__sz) +static inline void scx_bpf_cid_override(const s32 __arena *cpu_to_cid, u32= cnt) { if (bpf_ksym_exists(scx_bpf_cid_override___compat)) - return scx_bpf_cid_override___compat(cpu_to_cid, cpu_to_cid__sz); + return scx_bpf_cid_override___compat(cpu_to_cid, cnt); } =20 /** diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c index fd9a82a67627..2b4cc55e04ce 100644 --- a/tools/sched_ext/scx_qmap.bpf.c +++ b/tools/sched_ext/scx_qmap.bpf.c @@ -63,13 +63,6 @@ const volatile u32 max_tasks; * 3 =3D invalid: out-of-range cid */ const volatile u32 cid_override_mode; -/* - * Array lives in bss (writable) because scx_bpf_cid_override()'s BPF - * verifier signature treats its len-paired pointer as read/write - rodata - * fails verification with "write into map forbidden". Userspace populates - * it before SCX_OPS_LOAD, same as rodata, and nothing writes it after. - */ -s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS]; =20 UEI_DEFINE(uei); =20 @@ -1094,10 +1087,8 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init) * cid space (scx_bpf_nr_cids, cmask_init, etc.). On invalid input, * the kfunc calls scx_error() which aborts the scheduler. */ - if (cid_override_mode) { - scx_bpf_cid_override((const s32 *)cid_override_cpu_to_cid, - nr_cpu_ids * sizeof(s32)); - } + if (cid_override_mode) + scx_bpf_cid_override(qa.cid_override_cpu_to_cid, nr_cpu_ids); =20 /* * Allocate the task_ctx slab in arena and thread the entire slab onto diff --git a/tools/sched_ext/scx_qmap.c b/tools/sched_ext/scx_qmap.c index 67ddd483a4c7..c54f050d9991 100644 --- a/tools/sched_ext/scx_qmap.c +++ b/tools/sched_ext/scx_qmap.c @@ -153,8 +153,7 @@ int main(int argc, char **argv) skel->rodata->immed_stress_nth =3D strtoul(optarg, NULL, 0); break; case 'C': { - u32 nr_cpus =3D libbpf_num_possible_cpus(); - u32 mode, i; + u32 mode; =20 if (!strcmp(optarg, "shuffle")) mode =3D 1; @@ -167,18 +166,6 @@ int main(int argc, char **argv) return 1; } skel->rodata->cid_override_mode =3D mode; - - /* shuffle: reversed cpu_to_cid, bad-dup: dup cid 0, bad-range: identit= y */ - for (i =3D 0; i < nr_cpus; i++) { - if (mode =3D=3D 1) - skel->bss->cid_override_cpu_to_cid[i] =3D nr_cpus - 1 - i; - else - skel->bss->cid_override_cpu_to_cid[i] =3D i; - } - if (mode =3D=3D 2 && nr_cpus >=3D 2) - skel->bss->cid_override_cpu_to_cid[1] =3D 0; - if (mode =3D=3D 3) - skel->bss->cid_override_cpu_to_cid[0] =3D (s32)nr_cpus; break; } case 'v': @@ -191,9 +178,33 @@ int main(int argc, char **argv) } =20 SCX_OPS_LOAD(skel, qmap_ops, scx_qmap, uei); - link =3D SCX_OPS_ATTACH(skel, qmap_ops, scx_qmap); =20 qa =3D &skel->arena->qa; + + /* + * The cid-override array lives in the arena, which is mmapped at load. + * Populate it before qmap_init() consumes it at attach. + */ + if (skel->rodata->cid_override_mode) { + u32 mode =3D skel->rodata->cid_override_mode; + u32 nr_cpus =3D libbpf_num_possible_cpus(); + u32 i; + + /* shuffle: reversed cpu_to_cid, bad-dup: dup cid 0, bad-range: identity= */ + for (i =3D 0; i < nr_cpus; i++) { + if (mode =3D=3D 1) + qa->cid_override_cpu_to_cid[i] =3D nr_cpus - 1 - i; + else + qa->cid_override_cpu_to_cid[i] =3D i; + } + if (mode =3D=3D 2 && nr_cpus >=3D 2) + qa->cid_override_cpu_to_cid[1] =3D 0; + if (mode =3D=3D 3) + qa->cid_override_cpu_to_cid[0] =3D (s32)nr_cpus; + } + + link =3D SCX_OPS_ATTACH(skel, qmap_ops, scx_qmap); + qa->test_error_cnt =3D test_error_cnt; =20 while (!exit_req && !UEI_EXITED(skel, uei)) { diff --git a/tools/sched_ext/scx_qmap.h b/tools/sched_ext/scx_qmap.h index d15a705d5ac5..3b01570bbde9 100644 --- a/tools/sched_ext/scx_qmap.h +++ b/tools/sched_ext/scx_qmap.h @@ -62,6 +62,9 @@ struct qmap_arena { =20 struct cpu_ctx cpu_ctxs[SCX_QMAP_MAX_CPUS]; =20 + /* cid-override test input, populated by the loader before attach */ + __s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS]; + /* task_ctx slab; allocated and threaded by qmap_init() */ struct task_ctx __arena *task_ctxs; struct task_ctx __arena *task_free_head; --=20 2.55.0