From nobody Mon Feb 9 18:14:18 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E48437C0E0; Wed, 14 Jan 2026 09:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382381; cv=none; b=OixZxogYafg643UcGIMOEo1tI1V2Ms6hLKjYbyw7rTXw/e/pHmEHdGyBxnD9p8I/EZtfYlNVMZd/fj6KeC0yzFvVG0Tk5BdjnxBCdgTby72br32t2Ulz4U9n41Sr1WrmpPbJPOAgRcMfJuRhwyMsmcKQMJckSfXwKVA5PNbsDTE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382381; c=relaxed/simple; bh=oLsXdY04CoiXcBqZThcQMcc0g9wxqZYX1RhhefWKYMQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GNpgT8mjQEix1ja9fFSKJLVpBvQGmmVO7jBAXLsv31aFJfFC3upmY3yuns0N7GqXLGWS3KiaUWCfJ60BJ79Y9pbFPxtDKtOA6E1M+eu41mApioTEw2ZJ9n7OeB4/pZFPIuLua88VYqC1NnS5O9WLVHxkMH1HnkXKlb6uvIcAH+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.198]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4drgXG36N7zKHMg2; Wed, 14 Jan 2026 17:18:38 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 0DC8040577; Wed, 14 Jan 2026 17:19:30 +0800 (CST) Received: from k01.k01 (unknown [10.67.174.197]) by APP2 (Coremail) with SMTP id Syh0CgCXsYCfX2dpDhLdDg--.16789S3; Wed, 14 Jan 2026 17:19:29 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Yonghong Song , Puranjay Mohan , Anton Protopopov Subject: [PATCH bpf-next v4 1/4] bpf: Fix an off-by-one error in check_indirect_jump Date: Wed, 14 Jan 2026 17:39:11 +0800 Message-ID: <20260114093914.2403982-2-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260114093914.2403982-1-xukuohai@huaweicloud.com> References: <20260114093914.2403982-1-xukuohai@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgCXsYCfX2dpDhLdDg--.16789S3 X-Coremail-Antispam: 1UD129KBjvdXoWruFy3tw4UXry8tFyxAFy3twb_yoWfXrc_G3 yrXw4DWws3CF4fZF13Aa47Wa4j9rnYgFy0kr12gryDJryqq34v9r95GF95ZryDJFZrAFZx CrZxWrZ0qr1fZjkaLaAFLSUrUUUUbb8apTn2vfkv8UJUUUU8Yxn0WfASr-VFAUDa7-sFnT 9fnUUIcSsGvfJTRUUUbvkYFVCjjxCrM7AC8VAFwI0_Xr0_Wr1l1xkIjI8I6I8E6xAIw20E Y4v20xvaj40_Wr0E3s1l1IIY67AEw4v_Jr0_Jr4l82xGYIkIc2x26280x7IE14v26r18M2 8IrcIa0xkI8VCY1x0267AKxVW8JVW5JwA2ocxC64kIII0Yj41l84x0c7CEw4AK67xGY2AK 021l84ACjcxK6xIIjxv20xvE14v26F1j6w1UM28EF7xvwVC0I7IYx2IY6xkF7I0E14v26r 4UJVWxJr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx 0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkEbVWU JVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262kKe7AKxV WUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Y z7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zV AF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4l IxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCw CI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnI WIevJa73UjIFyTuYvjxUI2-eUUUUU X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai Fix an off-by-one error in check_indirect_jump() that skips the last element returned by copy_insn_array_uniq(). Fixes: 493d9e0d6083 ("bpf, x86: add support for indirect jumps") Signed-off-by: Xu Kuohai --- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index faa1ecc1fe9d..22605d9e0ffa 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20336,7 +20336,7 @@ static int check_indirect_jump(struct bpf_verifier_= env *env, struct bpf_insn *in return -EINVAL; } =20 - for (i =3D 0; i < n - 1; i++) { + for (i =3D 0; i < n; i++) { other_branch =3D push_stack(env, env->gotox_tmp_buf->items[i], env->insn_idx, env->cur_state->speculative); if (IS_ERR(other_branch)) --=20 2.47.3 From nobody Mon Feb 9 18:14:18 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E6FD387376; Wed, 14 Jan 2026 09:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382381; cv=none; b=cfWr8G/sZqjiQxYTr51e8u+76AjAd7KfTWmZfuFSYB05Y4tr1kdtVJLGorJLTmhzP7VuMdJHnl1AO6vhajzSc1vYTxuO/YoZlTgHSiBYP4zT1HMs7pMuPVDBsuLE5ykSw3PXCjFFYRvIWql28X1C3dPACHh+NEXnnEKq9efedeA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382381; c=relaxed/simple; bh=FfnYyZqbIuPhFnXISvjq7/1+uz7K6mlBSL0DqoCpePY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JkEwCKmMhe1eGs7qwUSvE+zi/Daxjf0k/GR1m+eyKzkpqA80/DcGdXZ6UcwYjwW/3I5BT33IamM0jDKtC1tD6SDqnwDTUPKLttTBCEtcCMP+FPC4NzsjqyustK9OM4N925ezPt9yIIg4fSKvVOGQIpALg86uCNvL4UDolWBWjsE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.170]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4drgXG2JbGzKHMSk; Wed, 14 Jan 2026 17:18:38 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 229B240539; Wed, 14 Jan 2026 17:19:30 +0800 (CST) Received: from k01.k01 (unknown [10.67.174.197]) by APP2 (Coremail) with SMTP id Syh0CgCXsYCfX2dpDhLdDg--.16789S4; Wed, 14 Jan 2026 17:19:29 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Yonghong Song , Puranjay Mohan , Anton Protopopov Subject: [PATCH bpf-next v4 2/4] bpf: Add helper to detect indirect jump targets Date: Wed, 14 Jan 2026 17:39:12 +0800 Message-ID: <20260114093914.2403982-3-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260114093914.2403982-1-xukuohai@huaweicloud.com> References: <20260114093914.2403982-1-xukuohai@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgCXsYCfX2dpDhLdDg--.16789S4 X-Coremail-Antispam: 1UD129KBjvJXoW3Cw47JF4rCF47XFWDKF1UGFg_yoWDWw43pF 4DW3s7Ar4UXrsFgwnrAa18Ary3ta1rWa4DKFW7W3y8Aw1YqrnYgF4F9FWavF98trWUCw1x ZF4j9rW7Wry7ZFJanT9S1TB71UUUUUDqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmab4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMc Ij6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_ Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxVAaw2AFwI 0_Jw0_GFylc7CjxVAKzI0EY4vE52x082I5MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCj c4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4 CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1x MIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF 4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsG vfC2KfnxnUUI43ZEXa7IU8go7tUUUUU== X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai Introduce helper bpf_insn_is_indirect_target to determine whether a BPF instruction is an indirect jump target. This helper will be used by follow-up patches to decide where to emit indirect landing pad instructions. Add a new flag to struct bpf_insn_aux_data to mark instructions that are indirect jump targets. The BPF verifier sets this flag, and the helper checks it to determine whether an instruction is an indirect jump target. Since bpf_insn_aux_data is only available before JIT stage, add a new field to struct bpf_prog_aux to store a pointer to the bpf_insn_aux_data array, making it accessible to the JIT. For programs with multiple subprogs, each subprog uses its own private copy of insn_aux_data, since subprogs may insert additional instructions during JIT and need to update the array. For non-subprog, the verifier's insn_aux_data array is used directly to avoid unnecessary copying. Signed-off-by: Xu Kuohai --- include/linux/bpf.h | 2 ++ include/linux/bpf_verifier.h | 10 ++++--- kernel/bpf/core.c | 51 +++++++++++++++++++++++++++++++++--- kernel/bpf/verifier.c | 51 +++++++++++++++++++++++++++++++++++- 4 files changed, 105 insertions(+), 9 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5936f8e2996f..e7d7e705327e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1533,6 +1533,7 @@ bool bpf_has_frame_pointer(unsigned long ip); int bpf_jit_charge_modmem(u32 size); void bpf_jit_uncharge_modmem(u32 size); bool bpf_prog_has_trampoline(const struct bpf_prog *prog); +bool bpf_insn_is_indirect_target(const struct bpf_prog *prog, int idx); #else static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr, @@ -1760,6 +1761,7 @@ struct bpf_prog_aux { struct bpf_stream stream[2]; struct mutex st_ops_assoc_mutex; struct bpf_map __rcu *st_ops_assoc; + struct bpf_insn_aux_data *insn_aux; }; =20 #define BPF_NR_CONTEXTS 4 /* normal, softirq, hardirq, NMI */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 130bcbd66f60..758086b384df 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -574,16 +574,18 @@ struct bpf_insn_aux_data { =20 /* below fields are initialized once */ unsigned int orig_idx; /* original instruction index */ - bool jmp_point; - bool prune_point; + u32 jmp_point:1; + u32 prune_point:1; /* ensure we check state equivalence and save state checkpoint and * this instruction, regardless of any heuristics */ - bool force_checkpoint; + u32 force_checkpoint:1; /* true if instruction is a call to a helper function that * accepts callback function as a parameter. */ - bool calls_callback; + u32 calls_callback:1; + /* true if the instruction is an indirect jump target */ + u32 indirect_target:1; /* * CFG strongly connected component this instruction belongs to, * zero if it is a singleton SCC. diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index e0b8a8a5aaa9..bb870936e74b 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1486,6 +1486,35 @@ static void adjust_insn_arrays(struct bpf_prog *prog= , u32 off, u32 len) #endif } =20 +static int adjust_insn_aux(struct bpf_prog *prog, int off, int cnt) +{ + size_t size; + struct bpf_insn_aux_data *new_aux; + + if (cnt =3D=3D 1) + return 0; + + /* prog->len already accounts for the cnt - 1 newly inserted instructions= */ + size =3D array_size(prog->len, sizeof(struct bpf_insn_aux_data)); + new_aux =3D vrealloc(prog->aux->insn_aux, size, GFP_KERNEL_ACCOUNT | __GF= P_ZERO); + if (!new_aux) + return -ENOMEM; + + /* follow the same behavior as adjust_insn_array(): leave [0, off] unchan= ged and shift + * [off + 1, end) to [off + cnt, end). Otherwise, the JIT would emit land= ing pads at + * wrong locations, as the actual indirect jump target remains at off. + */ + size =3D array_size(prog->len - off - cnt, sizeof(struct bpf_insn_aux_dat= a)); + memmove(new_aux + off + cnt, new_aux + off + 1, size); + + size =3D array_size(cnt - 1, sizeof(struct bpf_insn_aux_data)); + memset(new_aux + off + 1, 0, size); + + prog->aux->insn_aux =3D new_aux; + + return 0; +} + struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) { struct bpf_insn insn_buff[16], aux[2]; @@ -1541,6 +1570,11 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_= prog *prog) clone =3D tmp; insn_delta =3D rewritten - 1; =20 + if (adjust_insn_aux(clone, i, rewritten)) { + bpf_jit_prog_release_other(prog, clone); + return ERR_PTR(-ENOMEM); + } + /* Instructions arrays must be updated using absolute xlated offsets */ adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten); =20 @@ -1553,6 +1587,11 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_= prog *prog) clone->blinded =3D 1; return clone; } + +bool bpf_insn_is_indirect_target(const struct bpf_prog *prog, int idx) +{ + return prog->aux->insn_aux && prog->aux->insn_aux[idx].indirect_target; +} #endif /* CONFIG_BPF_JIT */ =20 /* Base function for offset calculation. Needs to go into .text section, @@ -2540,24 +2579,24 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf= _prog *fp, int *err) if (!bpf_prog_is_offloaded(fp->aux)) { *err =3D bpf_prog_alloc_jited_linfo(fp); if (*err) - return fp; + goto free_insn_aux; =20 fp =3D bpf_int_jit_compile(fp); bpf_prog_jit_attempt_done(fp); if (!fp->jited && jit_needed) { *err =3D -ENOTSUPP; - return fp; + goto free_insn_aux; } } else { *err =3D bpf_prog_offload_compile(fp); if (*err) - return fp; + goto free_insn_aux; } =20 finalize: *err =3D bpf_prog_lock_ro(fp); if (*err) - return fp; + goto free_insn_aux; =20 /* The tail call compatibility check can only be done at * this late stage as we need to determine, if we deal @@ -2566,6 +2605,10 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_= prog *fp, int *err) */ *err =3D bpf_check_tail_call(fp); =20 +free_insn_aux: + vfree(fp->aux->insn_aux); + fp->aux->insn_aux =3D NULL; + return fp; } EXPORT_SYMBOL_GPL(bpf_prog_select_runtime); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 22605d9e0ffa..f2fe6baeceb9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3852,6 +3852,11 @@ static bool is_jmp_point(struct bpf_verifier_env *en= v, int insn_idx) return env->insn_aux_data[insn_idx].jmp_point; } =20 +static void mark_indirect_target(struct bpf_verifier_env *env, int idx) +{ + env->insn_aux_data[idx].indirect_target =3D true; +} + #define LR_FRAMENO_BITS 3 #define LR_SPI_BITS 6 #define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1) @@ -20337,6 +20342,7 @@ static int check_indirect_jump(struct bpf_verifier_= env *env, struct bpf_insn *in } =20 for (i =3D 0; i < n; i++) { + mark_indirect_target(env, env->gotox_tmp_buf->items[i]); other_branch =3D push_stack(env, env->gotox_tmp_buf->items[i], env->insn_idx, env->cur_state->speculative); if (IS_ERR(other_branch)) @@ -21243,6 +21249,37 @@ static void convert_pseudo_ld_imm64(struct bpf_ver= ifier_env *env) } } =20 +static int clone_insn_aux_data(struct bpf_prog *prog, struct bpf_verifier_= env *env, u32 off) +{ + u32 i; + size_t size; + bool has_indirect_target =3D false; + struct bpf_insn_aux_data *insn_aux; + + for (i =3D 0; i < prog->len; i++) { + if (env->insn_aux_data[off + i].indirect_target) { + has_indirect_target =3D true; + break; + } + } + + /* insn_aux is copied into bpf_prog so the JIT can check whether an instr= uction is an + * indirect jump target. If no indirect jump targets exist, copying is un= necessary. + */ + if (!has_indirect_target) + return 0; + + size =3D array_size(sizeof(struct bpf_insn_aux_data), prog->len); + insn_aux =3D vzalloc(size); + if (!insn_aux) + return -ENOMEM; + + memcpy(insn_aux, env->insn_aux_data + off, size); + prog->aux->insn_aux =3D insn_aux; + + return 0; +} + /* single env->prog->insni[off] instruction was replaced with the range * insni[off, off + cnt). Adjust corresponding insn_aux_data by copying * [0, off) and [off, end) to new locations, so the patched range stays ze= ro @@ -22239,6 +22276,10 @@ static int jit_subprogs(struct bpf_verifier_env *e= nv) if (!i) func[i]->aux->exception_boundary =3D env->seen_exception; =20 + err =3D clone_insn_aux_data(func[i], env, subprog_start); + if (err < 0) + goto out_free; + /* * To properly pass the absolute subprog start to jit * all instruction adjustments should be accumulated @@ -22306,6 +22347,8 @@ static int jit_subprogs(struct bpf_verifier_env *en= v) for (i =3D 0; i < env->subprog_cnt; i++) { func[i]->aux->used_maps =3D NULL; func[i]->aux->used_map_cnt =3D 0; + vfree(func[i]->aux->insn_aux); + func[i]->aux->insn_aux =3D NULL; } =20 /* finally lock prog and jit images for all functions and @@ -22367,6 +22410,7 @@ static int jit_subprogs(struct bpf_verifier_env *en= v) for (i =3D 0; i < env->subprog_cnt; i++) { if (!func[i]) continue; + vfree(func[i]->aux->insn_aux); func[i]->aux->poke_tab =3D NULL; bpf_jit_free(func[i]); } @@ -25350,6 +25394,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_att= r *attr, bpfptr_t uattr, __u3 env->verification_time =3D ktime_get_ns() - start_time; print_verification_stats(env); env->prog->aux->verified_insns =3D env->insn_processed; + env->prog->aux->insn_aux =3D env->insn_aux_data; =20 /* preserve original error even if log finalization is successful */ err =3D bpf_vlog_finalize(&env->log, &log_true_size); @@ -25428,7 +25473,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_at= tr *attr, bpfptr_t uattr, __u3 if (!is_priv) mutex_unlock(&bpf_verifier_lock); clear_insn_aux_data(env, 0, env->prog->len); - vfree(env->insn_aux_data); + /* on success, insn_aux_data will be freed by bpf_prog_select_runtime */ + if (ret) { + vfree(env->insn_aux_data); + env->prog->aux->insn_aux =3D NULL; + } err_free_env: bpf_stack_liveness_free(env); kvfree(env->cfg.insn_postorder); --=20 2.47.3 From nobody Mon Feb 9 18:14:18 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E63937F732; Wed, 14 Jan 2026 09:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382382; cv=none; b=bpXiZJemrinaigmCa6Hm+LOA5QCZ1CUU9Dya2rPhUgrjYkZQW2eJt6t48wM0I+c1wm1y9BZsehv6NecklbtVo49+yGK6k4J/U8IREiRkugXJt7TwyVDtVqf4+GMHygg5KL6MbuF0hEW/lcOrQrsn9qmqQWrftEOXDvKwEgp5rfk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382382; c=relaxed/simple; bh=c8YoUE4ujbZcNATjKXA/sAyyBDxZVcJjqtu9JZ9t+O8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PYC3EtPMuOsJL90jWdV66gDbJKhhuxrFhQ+bBeh180hsiv7HgdFiLWFzXlaT3Yo42GqN9btbOpfawlykiT5U4Ervu6SbjcJFboXBpzML8uawV15lC78kSzc0aVvdh/YC/MB8vRA/vLZ4X+qEyA1po74+/siqT2m32pIW4okqP6s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.198]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4drgXG2Vl4zKHMSk; Wed, 14 Jan 2026 17:18:38 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 2B16E40576; Wed, 14 Jan 2026 17:19:30 +0800 (CST) Received: from k01.k01 (unknown [10.67.174.197]) by APP2 (Coremail) with SMTP id Syh0CgCXsYCfX2dpDhLdDg--.16789S5; Wed, 14 Jan 2026 17:19:30 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Yonghong Song , Puranjay Mohan , Anton Protopopov Subject: [PATCH bpf-next v4 3/4] bpf, x86: Emit ENDBR for indirect jump targets Date: Wed, 14 Jan 2026 17:39:13 +0800 Message-ID: <20260114093914.2403982-4-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260114093914.2403982-1-xukuohai@huaweicloud.com> References: <20260114093914.2403982-1-xukuohai@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgCXsYCfX2dpDhLdDg--.16789S5 X-Coremail-Antispam: 1UD129KBjvJXoWxZr4DXF1DAFW8JrWfur1Dtrb_yoW5CF1Upa 9xArySvrZ8Wr4qyrnrXF47Ary7AF4qgryxXF4ft3yfZwsxWryagF1aga4SqFy5JryfArs3 Xa4UAF1Du3WkuwUanT9S1TB71UUUUUDqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmab4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUWw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMc Ij6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_ Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxVAaw2AFwI 0_Jw0_GFylc7CjxVAKzI0EY4vE52x082I5MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCj c4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4 CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1x MIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF 4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsG vfC2KfnxnUUI43ZEXa7IU8D5r7UUUUU== X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai On CPUs that support CET/IBT, the indirect jump selftest triggers a kernel panic because the indirect jump targets lack ENDBR instructions. To fix it, emit an ENDBR instruction to each indirect jump target. Since the ENDBR instruction shifts the position of original jited instructions, fix the instruction address calculation wherever the addresses are used. For reference, below is a sample panic log. Missing ENDBR: bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x97/0xe1 ------------[ cut here ]------------ kernel BUG at arch/x86/kernel/cet.c:133! Oops: invalid opcode: 0000 [#1] SMP NOPTI ... ? 0xffffffffc00fb258 ? bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x97/0xe1 bpf_prog_test_run_syscall+0x110/0x2f0 ? fdget+0xba/0xe0 __sys_bpf+0xe4b/0x2590 ? __kmalloc_node_track_caller_noprof+0x1c7/0x680 ? bpf_prog_test_run_syscall+0x215/0x2f0 __x64_sys_bpf+0x21/0x30 do_syscall_64+0x85/0x620 ? bpf_prog_test_run_syscall+0x1e2/0x2f0 Fixes: 493d9e0d6083 ("bpf, x86: add support for indirect jumps") Signed-off-by: Xu Kuohai --- arch/x86/net/bpf_jit_comp.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e3b1c4b1d550..ef79baac42d7 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1733,6 +1733,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *add= rs, u8 *image, u8 *rw_image dst_reg =3D X86_REG_R9; } =20 + if (bpf_insn_is_indirect_target(bpf_prog, i - 1)) + EMIT_ENDBR(); + switch (insn->code) { /* ALU */ case BPF_ALU | BPF_ADD | BPF_X: @@ -2439,7 +2442,7 @@ st: if (is_imm8(insn->off)) =20 /* call */ case BPF_JMP | BPF_CALL: { - u8 *ip =3D image + addrs[i - 1]; + u8 *ip =3D image + addrs[i - 1] + (prog - temp); =20 func =3D (u8 *) __bpf_call_base + imm32; if (src_reg =3D=3D BPF_PSEUDO_CALL && tail_call_reachable) { @@ -2464,7 +2467,8 @@ st: if (is_imm8(insn->off)) if (imm32) emit_bpf_tail_call_direct(bpf_prog, &bpf_prog->aux->poke_tab[imm32 - 1], - &prog, image + addrs[i - 1], + &prog, + image + addrs[i - 1] + (prog - temp), callee_regs_used, stack_depth, ctx); @@ -2473,7 +2477,7 @@ st: if (is_imm8(insn->off)) &prog, callee_regs_used, stack_depth, - image + addrs[i - 1], + image + addrs[i - 1] + (prog - temp), ctx); break; =20 @@ -2638,7 +2642,8 @@ st: if (is_imm8(insn->off)) break; =20 case BPF_JMP | BPF_JA | BPF_X: - emit_indirect_jump(&prog, insn->dst_reg, image + addrs[i - 1]); + emit_indirect_jump(&prog, insn->dst_reg, + image + addrs[i - 1] + (prog - temp)); break; case BPF_JMP | BPF_JA: case BPF_JMP32 | BPF_JA: @@ -2728,7 +2733,7 @@ st: if (is_imm8(insn->off)) ctx->cleanup_addr =3D proglen; if (bpf_prog_was_classic(bpf_prog) && !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) { - u8 *ip =3D image + addrs[i - 1]; + u8 *ip =3D image + addrs[i - 1] + (prog - temp); =20 if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) return -EINVAL; --=20 2.47.3 From nobody Mon Feb 9 18:14:18 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E55F37E31A; Wed, 14 Jan 2026 09:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382382; cv=none; b=ehS0USRtACt35onuHg1t47fY8fDwxykaG7fVJlV20f9pEkC/Ipoqu09q2zmgnRXPCllJYVjLH8F5qCJCSrzu1eTfK6u/mlMzLYzJN/hvMBycgLPhIgA7TWmAYQI4x7ic4Oo8X/1fqNUPNi8exBUYdbO1vOT41TyDWMRuTxe1ztQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768382382; c=relaxed/simple; bh=CK135lTsC1hwdaKngXEy7RFxJdXglBDy8HrbdQGRik8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AeiMfe6UGrwrabxd2GLDR+xws/b4NkYzjDMUexKhSguu3mC7u4YQPbplmeXxB9Fn4X9VC/32hfDWLDXvn+gpH26Be6wmGXT5RmSKRL4sFdHonjWFVNVRX8/0Epi1zl8NqyJ2YICzEUzeja5hxban2OxI0zvfMo+rMr48QkU6uyk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.170]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4drgXG34sSzKHMf3; Wed, 14 Jan 2026 17:18:38 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 4293540539; Wed, 14 Jan 2026 17:19:30 +0800 (CST) Received: from k01.k01 (unknown [10.67.174.197]) by APP2 (Coremail) with SMTP id Syh0CgCXsYCfX2dpDhLdDg--.16789S6; Wed, 14 Jan 2026 17:19:30 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Yonghong Song , Puranjay Mohan , Anton Protopopov Subject: [PATCH bpf-next v4 4/4] bpf, arm64: Emit BTI for indirect jump target Date: Wed, 14 Jan 2026 17:39:14 +0800 Message-ID: <20260114093914.2403982-5-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260114093914.2403982-1-xukuohai@huaweicloud.com> References: <20260114093914.2403982-1-xukuohai@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgCXsYCfX2dpDhLdDg--.16789S6 X-Coremail-Antispam: 1UD129KBjvJXoWruFy3XrykAw4rCFy3ZF1xuFg_yoW8JrWrpa 1DCwnI9rZY9a109a1DX3ZrZr1akF4UKF4UArW5trWfG3WYgryagF15K3ZIkrs8ArWYvw4r XFWjkF1kCaykJw7anT9S1TB71UUUUUDqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmmb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxVAaw2 AFwI0_Jw0_GFylc7CjxVAKzI0EY4vE52x082I5MxAIw28IcxkI7VAKI48JMxC20s026xCa FVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_Jr Wlx4CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j 6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVWxJVW8Jr1lIxAIcVCF04k26cxKx2IYs7xG6r 1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr0_Gr1U YxBIdaVFxhVjvjDU0xZFpf9x07j5CzZUUUUU= X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai On CPUs that support BTI, the indirect jump selftest triggers a kernel panic because there is no BTI instructions at the indirect jump targets. Fix it by emitting a BTI instruction for each indirect jump target. For reference, below is a sample panic log. Internal error: Oops - BTI: 0000000036000003 [#1] SMP ... Call trace: bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x54/0xf8 (P) bpf_prog_run_pin_on_cpu+0x140/0x468 bpf_prog_test_run_syscall+0x280/0x3b8 bpf_prog_test_run+0x22c/0x2c0 Fixes: f4a66cf1cb14 ("bpf: arm64: Add support for indirect jumps") Signed-off-by: Xu Kuohai --- arch/arm64/net/bpf_jit_comp.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 0c4d44bcfbf4..370ae0751b9e 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1231,6 +1231,9 @@ static int build_insn(const struct bpf_insn *insn, st= ruct jit_ctx *ctx, int ret; bool sign_extend; =20 + if (bpf_insn_is_indirect_target(ctx->prog, i)) + emit_bti(A64_BTI_J, ctx); + switch (code) { /* dst =3D src */ case BPF_ALU | BPF_MOV | BPF_X: --=20 2.47.3