From nobody Sun Apr 5 16:33:20 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC5033D904F; Tue, 24 Mar 2026 11:54:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774353285; cv=none; b=AQ7azPgzU0cKDdg5hPyOufCa/JCDIcuXGt818vSmq8nm1PMi1gtkxW/2s0RILLHwnumM8mMKv/LeuNNDRiJT0xHbMDidNpIY5iyS7IjzdV3BZhkPiL39peMCfh9L5K11EzSiFTRRzMd3i8wh6XtEOWsnSF4OW9T1I/Sz3TVdi98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774353285; c=relaxed/simple; bh=l9An8UTwuha9FnbI46d+7kUVwVpCIzqotCCZEVArBso=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xkf0EOMpJXFd0seutta2PceYjLPDqrKc7qmPezfKPxfLlnK+V2csZAtp7sq9Aiw+iTwGj9p25xqYXklEmvHGvOiNfxXZAoeXn/iwo64Jgd2PQmKh9ZZ01WZZssBd/f1hxhWxVVBIPjXWS/X9WQUzWpnd4XrYEdYqve1dKW84ic8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.198]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4fg7jk3HV0zKHMcc; Tue, 24 Mar 2026 19:54:02 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id 7E22B40574; Tue, 24 Mar 2026 19:54:39 +0800 (CST) Received: from k01.k01 (unknown [10.67.174.197]) by APP1 (Coremail) with SMTP id cCh0CgAXGdl7e8Jpv2M5CA--.508S3; Tue, 24 Mar 2026 19:54:39 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Yonghong Song , Puranjay Mohan , Anton Protopopov , =?UTF-8?q?Alexis=20Lothor=C3=A9?= , Shahab Vahedi , Russell King , Tiezhu Yang , Hengqi Chen , Johan Almbladh , Paul Burton , Hari Bathini , Christophe Leroy , Naveen N Rao , Luke Nelson , Xi Wang , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Pu Lehui , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , "David S . Miller" , Wang YanQing Subject: [PATCH bpf-next v10 1/5] bpf: Move constants blinding out of arch-specific JITs Date: Tue, 24 Mar 2026 20:20:48 +0800 Message-ID: <20260324122052.342751-2-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260324122052.342751-1-xukuohai@huaweicloud.com> References: <20260324122052.342751-1-xukuohai@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: cCh0CgAXGdl7e8Jpv2M5CA--.508S3 X-Coremail-Antispam: 1UD129KBjvAXoWDCw4fAFW3AF4rJry8Gr1kuFg_yoW7WF1kWo Wa9w1DAF4rK34kGw4xKrn7GF13Zr18GrZrZr4fGFZ8Gayxt3yUKrZrXws8KaySva15GryD uF97Jay5ArZ0krnxn29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7v73VFW2AGmfu7bjvjm3 AaLaJ3UjIYCTnIWjp_UUUO87kC6x804xWl14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK 8VAvwI8IcIk0rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jr 4l82xGYIkIc2x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48v e4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Ar0_tr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI 0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AK xVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ew Av7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY 6r1j6r4UM4x0Y48IcxkI7VAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0En4kS14 v26r4a6rW5MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8C rVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVW8ZVWrXw CIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x02 67AKxVWxJVW8Jr1lIxAIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r 1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x07jx txhUUUUU= X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai During the JIT stage, constants blinding rewrites instructions but only rewrites the private instruction copy of the JITed subprog, leaving the global instructions and insn_aux_data unchanged. This causes a mismatch between subprog instructions and the global state, making it difficult to look up the global insn_aux_data in the JIT. To avoid this mismatch, and given that all arch-specific JITs already support constants blinding, move it to the generic verifier code, and switch to rewrite the global env->insnsi with the global states adjusted, as other rewrites in the verifier do. This removes the constants blinding calls in each JIT, which are largely duplicated code across architectures. Since constants blinding is only required for JIT, and there are two entry functions for JIT, jit_subprogs() and bpf_prog_select_runtime(), move the constants blinding invocation into the two functions. If constants blinding fails, or if it succeeds but the subsequent JIT compilation fails, kernel falls back to running the BPF program with interpreter. To ensure a correct rollback, the program cloning before instruction rewriting in the constants blinding is preserved. During the blinding process, only the cloned instructions are patched, leaving the original program untouched. Since bpf_patch_insn_data() is chosen for the constants blinding in the verifier path, and it adjusts the global auxiliary data in the verifier state, a key question is whether this auxiliary data should be restored when JIT fails? Besides instructions, bpf_patch_insn_data() adjusts env->insn_aux_data, env->subprog_info, prog->aux->poke_tab and env->insn_array_maps. env-> insn_aux_data and env->subprog_info are no longer used after JIT failure and are freed at the end of bpf_check(). prog->aux->poke_tab is only used by JIT. And when the JIT fails, programs using insn_array would be rejected by bpf_insn_array_ready() function since no JITed addresses available. This means env->insn_array_maps is only useful for JIT. Therefore, all the auxiliary data adjusted does not need to be restored. For classic BPF programs, constants blinding works as before since it is still invoked from bpf_prog_select_runtime(). Reviewed-by: Anton Protopopov # v8 Reviewed-by: Hari Bathini # powerpc jit Reviewed-by: Pu Lehui # riscv jit Signed-off-by: Xu Kuohai --- arch/arc/net/bpf_jit_core.c | 39 ++++++----------- arch/arm/net/bpf_jit_32.c | 41 +++--------------- arch/arm64/net/bpf_jit_comp.c | 72 +++++++++---------------------- arch/loongarch/net/bpf_jit.c | 59 ++++++++------------------ arch/mips/net/bpf_jit_comp.c | 20 +-------- arch/parisc/net/bpf_jit_core.c | 73 +++++++++++++------------------- arch/powerpc/net/bpf_jit_comp.c | 68 +++++++++++------------------ arch/riscv/net/bpf_jit_core.c | 61 ++++++++++---------------- arch/s390/net/bpf_jit_comp.c | 59 +++++++++----------------- arch/sparc/net/bpf_jit_comp_64.c | 61 +++++++++----------------- arch/x86/net/bpf_jit_comp.c | 43 +++---------------- arch/x86/net/bpf_jit_comp32.c | 33 ++------------- include/linux/filter.h | 23 +++++++++- kernel/bpf/core.c | 66 +++++++++++++++++++++++++---- kernel/bpf/verifier.c | 53 ++++++++++++++--------- 15 files changed, 301 insertions(+), 470 deletions(-) diff --git a/arch/arc/net/bpf_jit_core.c b/arch/arc/net/bpf_jit_core.c index 1421eeced0f5..973ceae48675 100644 --- a/arch/arc/net/bpf_jit_core.c +++ b/arch/arc/net/bpf_jit_core.c @@ -79,7 +79,6 @@ struct arc_jit_data { * The JIT pertinent context that is used by different functions. * * prog: The current eBPF program being handled. - * orig_prog: The original eBPF program before any possible change. * jit: The JIT buffer and its length. * bpf_header: The JITed program header. "jit.buf" points inside it. * emit: If set, opcodes are written to memory; else, a dry-run. @@ -94,12 +93,10 @@ struct arc_jit_data { * need_extra_pass: A forecast if an "extra_pass" will occur. * is_extra_pass: Indicates if the current pass is an extra pass. * user_bpf_prog: True, if VM opcodes come from a real program. - * blinded: True if "constant blinding" step returned a new "prog". * success: Indicates if the whole JIT went OK. */ struct jit_context { struct bpf_prog *prog; - struct bpf_prog *orig_prog; struct jit_buffer jit; struct bpf_binary_header *bpf_header; bool emit; @@ -114,7 +111,6 @@ struct jit_context { bool need_extra_pass; bool is_extra_pass; bool user_bpf_prog; - bool blinded; bool success; }; =20 @@ -161,13 +157,7 @@ static int jit_ctx_init(struct jit_context *ctx, struc= t bpf_prog *prog) { memset(ctx, 0, sizeof(*ctx)); =20 - ctx->orig_prog =3D prog; - - /* If constant blinding was requested but failed, scram. */ - ctx->prog =3D bpf_jit_blind_constants(prog); - if (IS_ERR(ctx->prog)) - return PTR_ERR(ctx->prog); - ctx->blinded =3D (ctx->prog !=3D ctx->orig_prog); + ctx->prog =3D prog; =20 /* If the verifier doesn't zero-extend, then we have to do it. */ ctx->do_zext =3D !ctx->prog->aux->verifier_zext; @@ -214,14 +204,6 @@ static inline void maybe_free(struct jit_context *ctx,= void **mem) */ static void jit_ctx_cleanup(struct jit_context *ctx) { - if (ctx->blinded) { - /* if all went well, release the orig_prog. */ - if (ctx->success) - bpf_jit_prog_release_other(ctx->prog, ctx->orig_prog); - else - bpf_jit_prog_release_other(ctx->orig_prog, ctx->prog); - } - maybe_free(ctx, (void **)&ctx->bpf2insn); maybe_free(ctx, (void **)&ctx->jit_data); =20 @@ -229,12 +211,19 @@ static void jit_ctx_cleanup(struct jit_context *ctx) ctx->bpf2insn_valid =3D false; =20 /* Freeing "bpf_header" is enough. "jit.buf" is a sub-array of it. */ - if (!ctx->success && ctx->bpf_header) { - bpf_jit_binary_free(ctx->bpf_header); - ctx->bpf_header =3D NULL; - ctx->jit.buf =3D NULL; - ctx->jit.index =3D 0; - ctx->jit.len =3D 0; + if (!ctx->success) { + if (ctx->bpf_header) { + bpf_jit_binary_free(ctx->bpf_header); + ctx->bpf_header =3D NULL; + ctx->jit.buf =3D NULL; + ctx->jit.index =3D 0; + ctx->jit.len =3D 0; + } + if (ctx->is_extra_pass) { + ctx->prog->bpf_func =3D NULL; + ctx->prog->jited =3D 0; + ctx->prog->jited_len =3D 0; + } } =20 ctx->emit =3D false; diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index deeb8f292454..e6b1bb2de627 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -2144,9 +2144,7 @@ bool bpf_jit_needs_zext(void) =20 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { - struct bpf_prog *tmp, *orig_prog =3D prog; struct bpf_binary_header *header; - bool tmp_blinded =3D false; struct jit_ctx ctx; unsigned int tmp_idx; unsigned int image_size; @@ -2156,20 +2154,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) * the interpreter. */ if (!prog->jit_requested) - return orig_prog; - - /* If constant blinding was enabled and we failed during blinding - * then we must fall back to the interpreter. Otherwise, we save - * the new JITed code. - */ - tmp =3D bpf_jit_blind_constants(prog); - - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 memset(&ctx, 0, sizeof(ctx)); ctx.prog =3D prog; @@ -2179,10 +2164,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) * we must fall back to the interpreter */ ctx.offsets =3D kcalloc(prog->len, sizeof(int), GFP_KERNEL); - if (ctx.offsets =3D=3D NULL) { - prog =3D orig_prog; - goto out; - } + if (ctx.offsets =3D=3D NULL) + return prog; =20 /* 1) fake pass to find in the length of the JITed code, * to compute ctx->offsets and other context variables @@ -2194,10 +2177,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) * being successful in the second pass, so just fall back * to the interpreter. */ - if (build_body(&ctx)) { - prog =3D orig_prog; + if (build_body(&ctx)) goto out_off; - } =20 tmp_idx =3D ctx.idx; build_prologue(&ctx); @@ -2213,10 +2194,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) ctx.idx +=3D ctx.imm_count; if (ctx.imm_count) { ctx.imms =3D kcalloc(ctx.imm_count, sizeof(u32), GFP_KERNEL); - if (ctx.imms =3D=3D NULL) { - prog =3D orig_prog; + if (ctx.imms =3D=3D NULL) goto out_off; - } } #else /* there's nothing about the epilogue on ARMv7 */ @@ -2238,10 +2217,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) /* Not able to allocate memory for the structure then * we must fall back to the interpretation */ - if (header =3D=3D NULL) { - prog =3D orig_prog; + if (header =3D=3D NULL) goto out_imms; - } =20 /* 2.) Actual pass to generate final JIT code */ ctx.target =3D (u32 *) image_ptr; @@ -2278,16 +2255,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) #endif out_off: kfree(ctx.offsets); -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); + return prog; =20 out_free: image_ptr =3D NULL; bpf_jit_binary_free(header); - prog =3D orig_prog; goto out_imms; } =20 diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index adf84962d579..cd5a72fff500 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2009,14 +2009,12 @@ struct arm64_jit_data { struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { int image_size, prog_size, extable_size, extable_align, extable_offset; - struct bpf_prog *tmp, *orig_prog =3D prog; struct bpf_binary_header *header; struct bpf_binary_header *ro_header =3D NULL; struct arm64_jit_data *jit_data; void __percpu *priv_stack_ptr =3D NULL; bool was_classic =3D bpf_prog_was_classic(prog); int priv_stack_alloc_sz; - bool tmp_blinded =3D false; bool extra_pass =3D false; struct jit_ctx ctx; u8 *image_ptr; @@ -2025,26 +2023,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) int exentry_idx; =20 if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - /* If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. - */ - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 jit_data =3D prog->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - prog =3D orig_prog; - goto out; - } + if (!jit_data) + return prog; prog->aux->jit_data =3D jit_data; } priv_stack_ptr =3D prog->aux->priv_stack_ptr; @@ -2056,10 +2041,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) priv_stack_alloc_sz =3D round_up(prog->aux->stack_depth, 16) + 2 * PRIV_STACK_GUARD_SZ; priv_stack_ptr =3D __alloc_percpu_gfp(priv_stack_alloc_sz, 16, GFP_KERNE= L); - if (!priv_stack_ptr) { - prog =3D orig_prog; + if (!priv_stack_ptr) goto out_priv_stack; - } =20 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz); prog->aux->priv_stack_ptr =3D priv_stack_ptr; @@ -2079,10 +2062,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) ctx.prog =3D prog; =20 ctx.offset =3D kvzalloc_objs(int, prog->len + 1); - if (ctx.offset =3D=3D NULL) { - prog =3D orig_prog; + if (ctx.offset =3D=3D NULL) goto out_off; - } =20 ctx.user_vm_start =3D bpf_arena_get_user_vm_start(prog->aux->arena); ctx.arena_vm_start =3D bpf_arena_get_kern_vm_start(prog->aux->arena); @@ -2095,15 +2076,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) * BPF line info needs ctx->offset[i] to be the offset of * instruction[i] in jited image, so build prologue first. */ - if (build_prologue(&ctx, was_classic)) { - prog =3D orig_prog; + if (build_prologue(&ctx, was_classic)) goto out_off; - } =20 - if (build_body(&ctx, extra_pass)) { - prog =3D orig_prog; + if (build_body(&ctx, extra_pass)) goto out_off; - } =20 ctx.epilogue_offset =3D ctx.idx; build_epilogue(&ctx, was_classic); @@ -2121,10 +2098,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) ro_header =3D bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof(u64), &header, &image_ptr, jit_fill_hole); - if (!ro_header) { - prog =3D orig_prog; + if (!ro_header) goto out_off; - } =20 /* Pass 2: Determine jited position and result for each instruction */ =20 @@ -2152,10 +2127,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) /* Dont write body instructions to memory for now */ ctx.write =3D false; =20 - if (build_body(&ctx, extra_pass)) { - prog =3D orig_prog; + if (build_body(&ctx, extra_pass)) goto out_free_hdr; - } =20 ctx.epilogue_offset =3D ctx.idx; ctx.exentry_idx =3D exentry_idx; @@ -2164,19 +2137,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) =20 /* Pass 3: Adjust jump offset and write final image */ if (build_body(&ctx, extra_pass) || - WARN_ON_ONCE(ctx.idx !=3D ctx.epilogue_offset)) { - prog =3D orig_prog; + WARN_ON_ONCE(ctx.idx !=3D ctx.epilogue_offset)) goto out_free_hdr; - } =20 build_epilogue(&ctx, was_classic); build_plt(&ctx); =20 /* Extra pass to validate JITed code. */ - if (validate_ctx(&ctx)) { - prog =3D orig_prog; + if (validate_ctx(&ctx)) goto out_free_hdr; - } =20 /* update the real prog size */ prog_size =3D sizeof(u32) * ctx.idx; @@ -2193,16 +2162,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) if (extra_pass && ctx.idx > jit_data->ctx.idx) { pr_err_once("multi-func JIT bug %d > %d\n", ctx.idx, jit_data->ctx.idx); - prog->bpf_func =3D NULL; - prog->jited =3D 0; - prog->jited_len =3D 0; goto out_free_hdr; } if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) { - /* ro_header has been freed */ + /* ro_header and header has been freed */ ro_header =3D NULL; - prog =3D orig_prog; - goto out_off; + header =3D NULL; + goto out_free_hdr; } /* * The instructions have now been copied to the ROX region from @@ -2245,13 +2211,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) kfree(jit_data); prog->aux->jit_data =3D NULL; } -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); + return prog; =20 out_free_hdr: + if (extra_pass) { + prog->bpf_func =3D NULL; + prog->jited =3D 0; + prog->jited_len =3D 0; + } if (header) { bpf_arch_text_copy(&ro_header->size, &header->size, sizeof(header->size)); diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c index 9cb796e16379..fcc8c0c29fb0 100644 --- a/arch/loongarch/net/bpf_jit.c +++ b/arch/loongarch/net/bpf_jit.c @@ -1922,43 +1922,26 @@ int arch_bpf_trampoline_size(const struct btf_func_= model *m, u32 flags, =20 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { - bool tmp_blinded =3D false, extra_pass =3D false; + bool extra_pass =3D false; u8 *image_ptr, *ro_image_ptr; int image_size, prog_size, extable_size; struct jit_ctx ctx; struct jit_data *jit_data; struct bpf_binary_header *header; struct bpf_binary_header *ro_header; - struct bpf_prog *tmp, *orig_prog =3D prog; =20 /* * If BPF JIT was not enabled then we must fall back to * the interpreter. */ if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - /* - * If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. Otherwise, we save - * the new JITed code. - */ - if (IS_ERR(tmp)) - return orig_prog; - - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 jit_data =3D prog->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - prog =3D orig_prog; - goto out; - } + if (!jit_data) + return prog; prog->aux->jit_data =3D jit_data; } if (jit_data->ctx.offset) { @@ -1978,17 +1961,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) ctx.user_vm_start =3D bpf_arena_get_user_vm_start(prog->aux->arena); =20 ctx.offset =3D kvcalloc(prog->len + 1, sizeof(u32), GFP_KERNEL); - if (ctx.offset =3D=3D NULL) { - prog =3D orig_prog; + if (ctx.offset =3D=3D NULL) goto out_offset; - } =20 /* 1. Initial fake pass to compute ctx->idx and set ctx->flags */ build_prologue(&ctx); - if (build_body(&ctx, extra_pass)) { - prog =3D orig_prog; + if (build_body(&ctx, extra_pass)) goto out_offset; - } ctx.epilogue_offset =3D ctx.idx; build_epilogue(&ctx); =20 @@ -2004,10 +1983,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) /* Now we know the size of the structure to make */ ro_header =3D bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof= (u32), &header, &image_ptr, jit_fill_hole); - if (!ro_header) { - prog =3D orig_prog; + if (!ro_header) goto out_offset; - } =20 /* 2. Now, the actual pass to generate final JIT code */ /* @@ -2027,17 +2004,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) ctx.num_exentries =3D 0; =20 build_prologue(&ctx); - if (build_body(&ctx, extra_pass)) { - prog =3D orig_prog; + if (build_body(&ctx, extra_pass)) goto out_free; - } build_epilogue(&ctx); =20 /* 3. Extra pass to validate JITed code */ - if (validate_ctx(&ctx)) { - prog =3D orig_prog; + if (validate_ctx(&ctx)) goto out_free; - } =20 /* And we're done */ if (bpf_jit_enable > 1) @@ -2050,9 +2023,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) goto out_free; } if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) { - /* ro_header has been freed */ + /* ro_header and header have been freed */ ro_header =3D NULL; - prog =3D orig_prog; + header =3D NULL; goto out_free; } /* @@ -2084,13 +2057,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) prog->aux->jit_data =3D NULL; } =20 -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? tmp : orig_prog= ); - return prog; =20 out_free: + if (extra_pass) { + prog->bpf_func =3D NULL; + prog->jited =3D 0; + prog->jited_len =3D 0; + } + if (header) { bpf_arch_text_copy(&ro_header->size, &header->size, sizeof(header->size)= ); bpf_jit_binary_pack_free(ro_header, header); diff --git a/arch/mips/net/bpf_jit_comp.c b/arch/mips/net/bpf_jit_comp.c index e355dfca4400..d2b6c955f18e 100644 --- a/arch/mips/net/bpf_jit_comp.c +++ b/arch/mips/net/bpf_jit_comp.c @@ -911,10 +911,8 @@ bool bpf_jit_needs_zext(void) =20 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { - struct bpf_prog *tmp, *orig_prog =3D prog; struct bpf_binary_header *header =3D NULL; struct jit_context ctx; - bool tmp_blinded =3D false; unsigned int tmp_idx; unsigned int image_size; u8 *image_ptr; @@ -925,19 +923,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= prog) * the interpreter. */ if (!prog->jit_requested) - return orig_prog; - /* - * If constant blinding was enabled and we failed during blinding - * then we must fall back to the interpreter. Otherwise, we save - * the new JITed code. - */ - tmp =3D bpf_jit_blind_constants(prog); - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 memset(&ctx, 0, sizeof(ctx)); ctx.program =3D prog; @@ -1025,14 +1011,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) prog->jited_len =3D image_size; =20 out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); kfree(ctx.descriptors); return prog; =20 out_err: - prog =3D orig_prog; if (header) bpf_jit_binary_free(header); goto out; diff --git a/arch/parisc/net/bpf_jit_core.c b/arch/parisc/net/bpf_jit_core.c index a5eb6b51e27a..35dca372b5df 100644 --- a/arch/parisc/net/bpf_jit_core.c +++ b/arch/parisc/net/bpf_jit_core.c @@ -44,30 +44,19 @@ bool bpf_jit_needs_zext(void) struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { unsigned int prog_size =3D 0, extable_size =3D 0; - bool tmp_blinded =3D false, extra_pass =3D false; - struct bpf_prog *tmp, *orig_prog =3D prog; + bool extra_pass =3D false; int pass =3D 0, prev_ninsns =3D 0, prologue_len, i; struct hppa_jit_data *jit_data; struct hppa_jit_context *ctx; =20 if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 jit_data =3D prog->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - prog =3D orig_prog; - goto out; - } + if (!jit_data) + return prog; prog->aux->jit_data =3D jit_data; } =20 @@ -81,10 +70,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *pr= og) =20 ctx->prog =3D prog; ctx->offset =3D kzalloc_objs(int, prog->len); - if (!ctx->offset) { - prog =3D orig_prog; - goto out_offset; - } + if (!ctx->offset) + goto out_err; for (i =3D 0; i < prog->len; i++) { prev_ninsns +=3D 20; ctx->offset[i] =3D prev_ninsns; @@ -93,10 +80,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *pr= og) for (i =3D 0; i < NR_JIT_ITERATIONS; i++) { pass++; ctx->ninsns =3D 0; - if (build_body(ctx, extra_pass, ctx->offset)) { - prog =3D orig_prog; - goto out_offset; - } + if (build_body(ctx, extra_pass, ctx->offset)) + goto out_err; ctx->body_len =3D ctx->ninsns; bpf_jit_build_prologue(ctx); ctx->prologue_len =3D ctx->ninsns - ctx->body_len; @@ -116,10 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= prog) &jit_data->image, sizeof(long), bpf_fill_ill_insns); - if (!jit_data->header) { - prog =3D orig_prog; - goto out_offset; - } + if (!jit_data->header) + goto out_err; =20 ctx->insns =3D (u32 *)jit_data->image; /* @@ -134,8 +117,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *p= rog) pr_err("bpf-jit: image did not converge in <%d passes!\n", i); if (jit_data->header) bpf_jit_binary_free(jit_data->header); - prog =3D orig_prog; - goto out_offset; + goto out_err; } =20 if (extable_size) @@ -148,8 +130,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *p= rog) bpf_jit_build_prologue(ctx); if (build_body(ctx, extra_pass, NULL)) { bpf_jit_binary_free(jit_data->header); - prog =3D orig_prog; - goto out_offset; + goto out_err; } bpf_jit_build_epilogue(ctx); =20 @@ -160,20 +141,19 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) { extern int machine_restart(char *); machine_restart(""); } } =20 + if (!prog->is_func || extra_pass) { + if (bpf_jit_binary_lock_ro(jit_data->header)) { + bpf_jit_binary_free(jit_data->header); + goto out_err; + } + bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); + } + prog->bpf_func =3D (void *)ctx->insns; prog->jited =3D 1; prog->jited_len =3D prog_size; =20 - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); - if (!prog->is_func || extra_pass) { - if (bpf_jit_binary_lock_ro(jit_data->header)) { - bpf_jit_binary_free(jit_data->header); - prog->bpf_func =3D NULL; - prog->jited =3D 0; - prog->jited_len =3D 0; - goto out_offset; - } prologue_len =3D ctx->epilogue_offset - ctx->body_len; for (i =3D 0; i < prog->len; i++) ctx->offset[i] +=3D prologue_len; @@ -183,14 +163,19 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) kfree(jit_data); prog->aux->jit_data =3D NULL; } -out: + if (HPPA_JIT_REBOOT) { extern int machine_restart(char *); machine_restart(""); } =20 - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); return prog; + +out_err: + if (extra_pass) { + prog->bpf_func =3D NULL; + prog->jited =3D 0; + prog->jited_len =3D 0; + } + goto out_offset; } =20 u64 hppa_div64(u64 div, u64 divisor) diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_com= p.c index a62a9a92b7b5..711028bebea3 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -142,9 +142,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *f= p) int flen; struct bpf_binary_header *fhdr =3D NULL; struct bpf_binary_header *hdr =3D NULL; - struct bpf_prog *org_fp =3D fp; - struct bpf_prog *tmp_fp; - bool bpf_blinded =3D false; bool extra_pass =3D false; u8 *fimage =3D NULL; u32 *fcode_base; @@ -152,24 +149,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *fp) u32 fixup_len; =20 if (!fp->jit_requested) - return org_fp; - - tmp_fp =3D bpf_jit_blind_constants(org_fp); - if (IS_ERR(tmp_fp)) - return org_fp; - - if (tmp_fp !=3D org_fp) { - bpf_blinded =3D true; - fp =3D tmp_fp; - } + return fp; =20 jit_data =3D fp->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - fp =3D org_fp; - goto out; - } + if (!jit_data) + return fp; fp->aux->jit_data =3D jit_data; } =20 @@ -194,10 +180,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= fp) } =20 addrs =3D kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL); - if (addrs =3D=3D NULL) { - fp =3D org_fp; - goto out_addrs; - } + if (addrs =3D=3D NULL) + goto out_err; =20 memset(&cgctx, 0, sizeof(struct codegen_context)); bpf_jit_init_reg_mapping(&cgctx); @@ -211,11 +195,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= fp) cgctx.exception_cb =3D fp->aux->exception_cb; =20 /* Scouting faux-generate pass 0 */ - if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { + if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) /* We hit something illegal or unsupported. */ - fp =3D org_fp; - goto out_addrs; - } + goto out_err; =20 /* * If we have seen a tail call, we need a second pass. @@ -226,10 +208,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= fp) */ if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.= idx * 4)) { cgctx.idx =3D 0; - if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { - fp =3D org_fp; - goto out_addrs; - } + if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) + goto out_err; } =20 bpf_jit_realloc_regs(&cgctx); @@ -250,10 +230,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= fp) =20 fhdr =3D bpf_jit_binary_pack_alloc(alloclen, &fimage, 4, &hdr, &image, bpf_jit_fill_ill_insns); - if (!fhdr) { - fp =3D org_fp; - goto out_addrs; - } + if (!fhdr) + goto out_err; =20 if (extable_len) fp->aux->extable =3D (void *)fimage + FUNCTION_DESCR_SIZE + proglen + fi= xup_len; @@ -272,8 +250,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *f= p) extra_pass)) { bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size)); bpf_jit_binary_pack_free(fhdr, hdr); - fp =3D org_fp; - goto out_addrs; + goto out_err; } bpf_jit_build_epilogue(code_base, &cgctx); =20 @@ -295,15 +272,16 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *fp) ((u64 *)image)[1] =3D local_paca->kernel_toc; #endif =20 + if (!fp->is_func || extra_pass) { + if (bpf_jit_binary_pack_finalize(fhdr, hdr)) + goto out_err; + } + fp->bpf_func =3D (void *)fimage; fp->jited =3D 1; fp->jited_len =3D cgctx.idx * 4 + FUNCTION_DESCR_SIZE; =20 if (!fp->is_func || extra_pass) { - if (bpf_jit_binary_pack_finalize(fhdr, hdr)) { - fp =3D org_fp; - goto out_addrs; - } bpf_prog_fill_jited_linfo(fp, addrs); out_addrs: kfree(addrs); @@ -318,11 +296,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *fp) jit_data->hdr =3D hdr; } =20 -out: - if (bpf_blinded) - bpf_jit_prog_release_other(fp, fp =3D=3D org_fp ? tmp_fp : org_fp); - return fp; + +out_err: + if (extra_pass) { + fp->bpf_func =3D NULL; + fp->jited =3D 0; + fp->jited_len =3D 0; + } + goto out_addrs; } =20 /* diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index b3581e926436..527baa50dc68 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -44,29 +44,19 @@ bool bpf_jit_needs_zext(void) struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { unsigned int prog_size =3D 0, extable_size =3D 0; - bool tmp_blinded =3D false, extra_pass =3D false; - struct bpf_prog *tmp, *orig_prog =3D prog; + bool extra_pass =3D false; int pass =3D 0, prev_ninsns =3D 0, i; struct rv_jit_data *jit_data; struct rv_jit_context *ctx; =20 if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 jit_data =3D prog->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); if (!jit_data) { - prog =3D orig_prog; - goto out; + return prog; } prog->aux->jit_data =3D jit_data; } @@ -83,15 +73,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *p= rog) ctx->user_vm_start =3D bpf_arena_get_user_vm_start(prog->aux->arena); ctx->prog =3D prog; ctx->offset =3D kzalloc_objs(int, prog->len); - if (!ctx->offset) { - prog =3D orig_prog; + if (!ctx->offset) goto out_offset; - } =20 - if (build_body(ctx, extra_pass, NULL)) { - prog =3D orig_prog; + if (build_body(ctx, extra_pass, NULL)) goto out_offset; - } =20 for (i =3D 0; i < prog->len; i++) { prev_ninsns +=3D 32; @@ -105,10 +91,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *p= rog) bpf_jit_build_prologue(ctx, bpf_is_subprog(prog)); ctx->prologue_len =3D ctx->ninsns; =20 - if (build_body(ctx, extra_pass, ctx->offset)) { - prog =3D orig_prog; + if (build_body(ctx, extra_pass, ctx->offset)) goto out_offset; - } =20 ctx->epilogue_offset =3D ctx->ninsns; bpf_jit_build_epilogue(ctx); @@ -126,10 +110,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *= prog) &jit_data->ro_image, sizeof(u32), &jit_data->header, &jit_data->image, bpf_fill_ill_insns); - if (!jit_data->ro_header) { - prog =3D orig_prog; + if (!jit_data->ro_header) goto out_offset; - } =20 /* * Use the image(RW) for writing the JITed instructions. But also save @@ -150,7 +132,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *p= rog) =20 if (i =3D=3D NR_JIT_ITERATIONS) { pr_err("bpf-jit: image did not converge in <%d passes!\n", i); - prog =3D orig_prog; goto out_free_hdr; } =20 @@ -163,26 +144,27 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) ctx->nexentries =3D 0; =20 bpf_jit_build_prologue(ctx, bpf_is_subprog(prog)); - if (build_body(ctx, extra_pass, NULL)) { - prog =3D orig_prog; + if (build_body(ctx, extra_pass, NULL)) goto out_free_hdr; - } bpf_jit_build_epilogue(ctx); =20 if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); =20 - prog->bpf_func =3D (void *)ctx->ro_insns + cfi_get_offset(); - prog->jited =3D 1; - prog->jited_len =3D prog_size - cfi_get_offset(); - if (!prog->is_func || extra_pass) { if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->= header))) { /* ro_header has been freed */ jit_data->ro_header =3D NULL; - prog =3D orig_prog; - goto out_offset; + jit_data->header =3D NULL; + goto out_free_hdr; } + } + + prog->bpf_func =3D (void *)ctx->ro_insns + cfi_get_offset(); + prog->jited =3D 1; + prog->jited_len =3D prog_size - cfi_get_offset(); + + if (!prog->is_func || extra_pass) { /* * The instructions have now been copied to the ROX region from * where they will execute. @@ -198,14 +180,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) kfree(jit_data); prog->aux->jit_data =3D NULL; } -out: =20 - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); return prog; =20 out_free_hdr: + if (extra_pass) { + prog->bpf_func =3D NULL; + prog->jited =3D 0; + prog->jited_len =3D 0; + } if (jit_data->header) { bpf_arch_text_copy(&jit_data->ro_header->size, &jit_data->header->size, sizeof(jit_data->header->size)); diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index d08d159b6319..2dfc279b1be2 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2314,36 +2314,20 @@ static struct bpf_binary_header *bpf_jit_alloc(stru= ct bpf_jit *jit, */ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) { - struct bpf_prog *tmp, *orig_fp =3D fp; struct bpf_binary_header *header; struct s390_jit_data *jit_data; - bool tmp_blinded =3D false; bool extra_pass =3D false; struct bpf_jit jit; int pass; =20 if (!fp->jit_requested) - return orig_fp; - - tmp =3D bpf_jit_blind_constants(fp); - /* - * If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. - */ - if (IS_ERR(tmp)) - return orig_fp; - if (tmp !=3D fp) { - tmp_blinded =3D true; - fp =3D tmp; - } + return fp; =20 jit_data =3D fp->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - fp =3D orig_fp; - goto out; - } + if (!jit_data) + return fp; fp->aux->jit_data =3D jit_data; } if (jit_data->ctx.addrs) { @@ -2356,34 +2340,27 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *fp) =20 memset(&jit, 0, sizeof(jit)); jit.addrs =3D kvcalloc(fp->len + 1, sizeof(*jit.addrs), GFP_KERNEL); - if (jit.addrs =3D=3D NULL) { - fp =3D orig_fp; - goto free_addrs; - } + if (jit.addrs =3D=3D NULL) + goto out_err; /* * Three initial passes: * - 1/2: Determine clobbered registers * - 3: Calculate program size and addrs array */ for (pass =3D 1; pass <=3D 3; pass++) { - if (bpf_jit_prog(&jit, fp, extra_pass)) { - fp =3D orig_fp; - goto free_addrs; - } + if (bpf_jit_prog(&jit, fp, extra_pass)) + goto out_err; } /* * Final pass: Allocate and generate program */ header =3D bpf_jit_alloc(&jit, fp); - if (!header) { - fp =3D orig_fp; - goto free_addrs; - } + if (!header) + goto out_err; skip_init_ctx: if (bpf_jit_prog(&jit, fp, extra_pass)) { bpf_jit_binary_free(header); - fp =3D orig_fp; - goto free_addrs; + goto out_err; } if (bpf_jit_enable > 1) { bpf_jit_dump(fp->len, jit.size, pass, jit.prg_buf); @@ -2392,8 +2369,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *fp) if (!fp->is_func || extra_pass) { if (bpf_jit_binary_lock_ro(header)) { bpf_jit_binary_free(header); - fp =3D orig_fp; - goto free_addrs; + goto out_err; } } else { jit_data->header =3D header; @@ -2411,11 +2387,16 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *fp) kfree(jit_data); fp->aux->jit_data =3D NULL; } -out: - if (tmp_blinded) - bpf_jit_prog_release_other(fp, fp =3D=3D orig_fp ? - tmp : orig_fp); + return fp; + +out_err: + if (extra_pass) { + fp->bpf_func =3D NULL; + fp->jited =3D 0; + fp->jited_len =3D 0; + } + goto free_addrs; } =20 bool bpf_jit_supports_kfunc_call(void) diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp= _64.c index b23d1c645ae5..e83e29137566 100644 --- a/arch/sparc/net/bpf_jit_comp_64.c +++ b/arch/sparc/net/bpf_jit_comp_64.c @@ -1479,37 +1479,22 @@ struct sparc64_jit_data { =20 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { - struct bpf_prog *tmp, *orig_prog =3D prog; struct sparc64_jit_data *jit_data; struct bpf_binary_header *header; u32 prev_image_size, image_size; - bool tmp_blinded =3D false; bool extra_pass =3D false; struct jit_ctx ctx; u8 *image_ptr; int pass, i; =20 if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - /* If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. - */ - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 jit_data =3D prog->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - prog =3D orig_prog; - goto out; - } + if (!jit_data) + return prog; prog->aux->jit_data =3D jit_data; } if (jit_data->ctx.offset) { @@ -1527,10 +1512,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) ctx.prog =3D prog; =20 ctx.offset =3D kmalloc_array(prog->len, sizeof(unsigned int), GFP_KERNEL); - if (ctx.offset =3D=3D NULL) { - prog =3D orig_prog; - goto out_off; - } + if (ctx.offset =3D=3D NULL) + goto out_err; =20 /* Longest sequence emitted is for bswap32, 12 instructions. Pre-cook * the offset array so that we converge faster. @@ -1543,10 +1526,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) ctx.idx =3D 0; =20 build_prologue(&ctx); - if (build_body(&ctx)) { - prog =3D orig_prog; - goto out_off; - } + if (build_body(&ctx)) + goto out_err; build_epilogue(&ctx); =20 if (bpf_jit_enable > 1) @@ -1569,10 +1550,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) image_size =3D sizeof(u32) * ctx.idx; header =3D bpf_jit_binary_alloc(image_size, &image_ptr, sizeof(u32), jit_fill_hole); - if (header =3D=3D NULL) { - prog =3D orig_prog; - goto out_off; - } + if (header =3D=3D NULL) + goto out_err; =20 ctx.image =3D (u32 *)image_ptr; skip_init_ctx: @@ -1582,8 +1561,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) =20 if (build_body(&ctx)) { bpf_jit_binary_free(header); - prog =3D orig_prog; - goto out_off; + goto out_err; } =20 build_epilogue(&ctx); @@ -1592,8 +1570,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) pr_err("bpf_jit: Failed to converge, prev_size=3D%u size=3D%d\n", prev_image_size, ctx.idx * 4); bpf_jit_binary_free(header); - prog =3D orig_prog; - goto out_off; + goto out_err; } =20 if (bpf_jit_enable > 1) @@ -1604,8 +1581,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) if (!prog->is_func || extra_pass) { if (bpf_jit_binary_lock_ro(header)) { bpf_jit_binary_free(header); - prog =3D orig_prog; - goto out_off; + goto out_err; } } else { jit_data->ctx =3D ctx; @@ -1624,9 +1600,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) kfree(jit_data); prog->aux->jit_data =3D NULL; } -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); + return prog; + +out_err: + if (extra_pass) { + prog->bpf_func =3D NULL; + prog->jited =3D 0; + prog->jited_len =3D 0; + } + goto out_off; } diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e9b78040d703..77d00a8dec87 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3717,13 +3717,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) { struct bpf_binary_header *rw_header =3D NULL; struct bpf_binary_header *header =3D NULL; - struct bpf_prog *tmp, *orig_prog =3D prog; void __percpu *priv_stack_ptr =3D NULL; struct x64_jit_data *jit_data; int priv_stack_alloc_sz; int proglen, oldproglen =3D 0; struct jit_context ctx =3D {}; - bool tmp_blinded =3D false; bool extra_pass =3D false; bool padding =3D false; u8 *rw_image =3D NULL; @@ -3733,27 +3731,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) int i; =20 if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - /* - * If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. - */ - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 jit_data =3D prog->aux->jit_data; if (!jit_data) { jit_data =3D kzalloc_obj(*jit_data); - if (!jit_data) { - prog =3D orig_prog; - goto out; - } + if (!jit_data) + return prog; prog->aux->jit_data =3D jit_data; } priv_stack_ptr =3D prog->aux->priv_stack_ptr; @@ -3765,10 +3749,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) priv_stack_alloc_sz =3D round_up(prog->aux->stack_depth, 8) + 2 * PRIV_STACK_GUARD_SZ; priv_stack_ptr =3D __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL= ); - if (!priv_stack_ptr) { - prog =3D orig_prog; + if (!priv_stack_ptr) goto out_priv_stack; - } =20 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz); prog->aux->priv_stack_ptr =3D priv_stack_ptr; @@ -3786,10 +3768,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) goto skip_init_addrs; } addrs =3D kvmalloc_objs(*addrs, prog->len + 1); - if (!addrs) { - prog =3D orig_prog; + if (!addrs) goto out_addrs; - } =20 /* * Before first pass, make a rough estimation of addrs[] @@ -3820,8 +3800,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) sizeof(rw_header->size)); bpf_jit_binary_pack_free(header, rw_header); } - /* Fall back to interpreter mode */ - prog =3D orig_prog; if (extra_pass) { prog->bpf_func =3D NULL; prog->jited =3D 0; @@ -3852,10 +3830,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) header =3D bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_= size, &image, align, &rw_header, &rw_image, jit_fill_hole); - if (!header) { - prog =3D orig_prog; + if (!header) goto out_addrs; - } prog->aux->extable =3D (void *) image + roundup(proglen, align); } oldproglen =3D proglen; @@ -3908,8 +3884,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) prog->bpf_func =3D (void *)image + cfi_get_offset(); prog->jited =3D 1; prog->jited_len =3D proglen - cfi_get_offset(); - } else { - prog =3D orig_prog; } =20 if (!image || !prog->is_func || extra_pass) { @@ -3925,10 +3899,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) kfree(jit_data); prog->aux->jit_data =3D NULL; } -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); + return prog; } =20 diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index dda423025c3d..5f259577614a 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -2521,35 +2521,19 @@ bool bpf_jit_needs_zext(void) struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { struct bpf_binary_header *header =3D NULL; - struct bpf_prog *tmp, *orig_prog =3D prog; int proglen, oldproglen =3D 0; struct jit_context ctx =3D {}; - bool tmp_blinded =3D false; u8 *image =3D NULL; int *addrs; int pass; int i; =20 if (!prog->jit_requested) - return orig_prog; - - tmp =3D bpf_jit_blind_constants(prog); - /* - * If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. - */ - if (IS_ERR(tmp)) - return orig_prog; - if (tmp !=3D prog) { - tmp_blinded =3D true; - prog =3D tmp; - } + return prog; =20 addrs =3D kmalloc_objs(*addrs, prog->len); - if (!addrs) { - prog =3D orig_prog; - goto out; - } + if (!addrs) + return prog; =20 /* * Before first pass, make a rough estimation of addrs[] @@ -2574,7 +2558,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog) image =3D NULL; if (header) bpf_jit_binary_free(header); - prog =3D orig_prog; goto out_addrs; } if (image) { @@ -2588,10 +2571,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog= *prog) if (proglen =3D=3D oldproglen) { header =3D bpf_jit_binary_alloc(proglen, &image, 1, jit_fill_hole); - if (!header) { - prog =3D orig_prog; + if (!header) goto out_addrs; - } } oldproglen =3D proglen; cond_resched(); @@ -2604,16 +2585,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_pro= g *prog) prog->bpf_func =3D (void *)image; prog->jited =3D 1; prog->jited_len =3D proglen; - } else { - prog =3D orig_prog; } =20 out_addrs: kfree(addrs); -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog =3D=3D orig_prog ? - tmp : orig_prog); return prog; } =20 diff --git a/include/linux/filter.h b/include/linux/filter.h index 44d7ae95ddbc..d8c97f463e0f 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1184,6 +1184,18 @@ static inline bool bpf_dump_raw_ok(const struct cred= *cred) =20 struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, const struct bpf_insn *patch, u32 len); + +#ifdef CONFIG_BPF_SYSCALL +struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, + const struct bpf_insn *patch, u32 len); +#else +static inline struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env= *env, u32 off, + const struct bpf_insn *patch, u32 len) +{ + return ERR_PTR(-ENOTSUPP); +} +#endif + int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt); =20 static inline bool xdp_return_frame_no_direct(void) @@ -1310,7 +1322,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, =20 const char *bpf_jit_get_prog_name(struct bpf_prog *prog); =20 -struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *fp); +struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, str= uct bpf_prog *prog); void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_o= ther); =20 static inline void bpf_jit_dump(unsigned int flen, unsigned int proglen, @@ -1451,6 +1463,15 @@ static inline void bpf_prog_kallsyms_del(struct bpf_= prog *fp) { } =20 +static inline +struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, str= uct bpf_prog *prog) +{ + return prog; +} + +static inline void bpf_jit_prog_release_other(struct bpf_prog *fp, struct = bpf_prog *fp_other) +{ +} #endif /* CONFIG_BPF_JIT */ =20 void bpf_prog_kallsyms_del_all(struct bpf_prog *fp); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 67eb12b637a5..8f52f4b6c3af 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1507,13 +1507,16 @@ static void adjust_insn_arrays(struct bpf_prog *pro= g, u32 off, u32 len) #endif } =20 -struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) +struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, str= uct bpf_prog *prog) { struct bpf_insn insn_buff[16], aux[2]; struct bpf_prog *clone, *tmp; int insn_delta, insn_cnt; struct bpf_insn *insn; - int i, rewritten; + int i, rewritten, subprog_start; + + if (env) + prog =3D env->prog; =20 if (!prog->blinding_requested || prog->blinded) return prog; @@ -1522,8 +1525,13 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_= prog *prog) if (!clone) return ERR_PTR(-ENOMEM); =20 + /* make sure bpf_patch_insn_data() patches the correct prog */ + if (env) + env->prog =3D clone; + insn_cnt =3D clone->len; insn =3D clone->insnsi; + subprog_start =3D prog->aux->subprog_start; =20 for (i =3D 0; i < insn_cnt; i++, insn++) { if (bpf_pseudo_func(insn)) { @@ -1549,21 +1557,34 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf= _prog *prog) if (!rewritten) continue; =20 - tmp =3D bpf_patch_insn_single(clone, i, insn_buff, rewritten); - if (IS_ERR(tmp)) { + if (env) + tmp =3D bpf_patch_insn_data(env, subprog_start + i, insn_buff, rewritte= n); + else + tmp =3D bpf_patch_insn_single(clone, i, insn_buff, rewritten); + + if (IS_ERR_OR_NULL(tmp)) { + /* restore the original prog */ + if (env) + env->prog =3D prog; /* Patching may have repointed aux->prog during * realloc from the original one, so we need to * fix it up here on error. */ bpf_jit_prog_release_other(prog, clone); - return tmp; + return IS_ERR(tmp) ? tmp : ERR_PTR(-ENOMEM); } =20 clone =3D tmp; insn_delta =3D rewritten - 1; =20 - /* Instructions arrays must be updated using absolute xlated offsets */ - adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten); + if (env) + env->prog =3D clone; + else + /* Instructions arrays must be updated using absolute xlated offsets. + * The arrays have already been adjusted by bpf_patch_insn_data() when + * env is not NULL. + */ + adjust_insn_arrays(clone, subprog_start + i, rewritten); =20 /* Walk new program and skip insns we just inserted. */ insn =3D clone->insnsi + i + insn_delta; @@ -2532,6 +2553,35 @@ static bool bpf_prog_select_interpreter(struct bpf_p= rog *fp) return select_interpreter; } =20 +static struct bpf_prog *bpf_prog_jit_compile(struct bpf_prog *prog) +{ +#ifdef CONFIG_BPF_JIT + bool blinded =3D false; + struct bpf_prog *orig_prog =3D prog; + + prog =3D bpf_jit_blind_constants(NULL, orig_prog); + /* If blinding was requested and we failed during blinding, we must fall + * back to the interpreter. + */ + if (IS_ERR(prog)) + return orig_prog; + + if (prog !=3D orig_prog) + blinded =3D true; + + prog =3D bpf_int_jit_compile(prog); + if (blinded) { + if (!prog->jited) { + bpf_jit_prog_release_other(orig_prog, prog); + prog =3D orig_prog; + } else { + bpf_jit_prog_release_other(prog, orig_prog); + } + } +#endif + return prog; +} + /** * bpf_prog_select_runtime - select exec runtime for BPF program * @fp: bpf_prog populated with BPF program @@ -2571,7 +2621,7 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_p= rog *fp, int *err) if (*err) return fp; =20 - fp =3D bpf_int_jit_compile(fp); + fp =3D bpf_prog_jit_compile(fp); bpf_prog_jit_attempt_done(fp); if (!fp->jited && jit_needed) { *err =3D -ENOTSUPP; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index cd008b146ee5..1b2b99f2d859 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -22105,8 +22105,8 @@ static void adjust_poke_descs(struct bpf_prog *prog= , u32 off, u32 len) } } =20 -static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, = u32 off, - const struct bpf_insn *patch, u32 len) +struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, + const struct bpf_insn *patch, u32 len) { struct bpf_prog *new_prog; struct bpf_insn_aux_data *new_data =3D NULL; @@ -22875,17 +22875,23 @@ static int convert_ctx_accesses(struct bpf_verifi= er_env *env) =20 static int jit_subprogs(struct bpf_verifier_env *env) { - struct bpf_prog *prog =3D env->prog, **func, *tmp; + struct bpf_prog *orig_prog =3D env->prog, *prog, **func, *tmp; int i, j, subprog_start, subprog_end =3D 0, len, subprog; struct bpf_map *map_ptr; struct bpf_insn *insn; void *old_bpf_func; int err, num_exentries; - int old_len, subprog_start_adjustment =3D 0; + bool blinded =3D false; =20 if (env->subprog_cnt <=3D 1) return 0; =20 + prog =3D bpf_jit_blind_constants(env, NULL); + if (IS_ERR(prog)) + return -ENOMEM; + if (prog !=3D orig_prog) + blinded =3D true; + for (i =3D 0, insn =3D prog->insnsi; i < prog->len; i++, insn++) { if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn)) continue; @@ -22896,8 +22902,13 @@ static int jit_subprogs(struct bpf_verifier_env *e= nv) */ subprog =3D find_subprog(env, i + insn->imm + 1); if (verifier_bug_if(subprog < 0, env, "No program to jit at insn %d", - i + insn->imm + 1)) + i + insn->imm + 1)) { + if (blinded) { + bpf_jit_prog_release_other(orig_prog, prog); + env->prog =3D orig_prog; + } return -EFAULT; + } /* temporarily remember subprog id inside insn instead of * aux_data, since next loop will split up all insns into funcs */ @@ -22953,10 +22964,11 @@ static int jit_subprogs(struct bpf_verifier_env *= env) goto out_free; func[i]->is_func =3D 1; func[i]->sleepable =3D prog->sleepable; + func[i]->blinded =3D prog->blinded; func[i]->aux->func_idx =3D i; /* Below members will be freed only at prog->aux */ func[i]->aux->btf =3D prog->aux->btf; - func[i]->aux->subprog_start =3D subprog_start + subprog_start_adjustment; + func[i]->aux->subprog_start =3D subprog_start; func[i]->aux->func_info =3D prog->aux->func_info; func[i]->aux->func_info_cnt =3D prog->aux->func_info_cnt; func[i]->aux->poke_tab =3D prog->aux->poke_tab; @@ -23012,15 +23024,7 @@ static int jit_subprogs(struct bpf_verifier_env *e= nv) func[i]->aux->might_sleep =3D env->subprog_info[i].might_sleep; if (!i) func[i]->aux->exception_boundary =3D env->seen_exception; - - /* - * To properly pass the absolute subprog start to jit - * all instruction adjustments should be accumulated - */ - old_len =3D func[i]->len; func[i] =3D bpf_int_jit_compile(func[i]); - subprog_start_adjustment +=3D func[i]->len - old_len; - if (!func[i]->jited) { err =3D -ENOTSUPP; goto out_free; @@ -23124,6 +23128,10 @@ static int jit_subprogs(struct bpf_verifier_env *e= nv) prog->aux->bpf_exception_cb =3D (void *)func[env->exception_callback_subp= rog]->bpf_func; prog->aux->exception_boundary =3D func[0]->aux->exception_boundary; bpf_prog_jit_attempt_done(prog); + + if (blinded) + bpf_jit_prog_release_other(prog, orig_prog); + return 0; out_free: /* We failed JIT'ing, so at this point we need to unregister poke @@ -23146,14 +23154,21 @@ static int jit_subprogs(struct bpf_verifier_env *= env) } kfree(func); out_undo_insn: + if (blinded) { + bpf_jit_prog_release_other(orig_prog, prog); + env->prog =3D prog =3D orig_prog; + } /* cleanup main prog to be interpreted */ prog->jit_requested =3D 0; prog->blinding_requested =3D 0; - for (i =3D 0, insn =3D prog->insnsi; i < prog->len; i++, insn++) { - if (!bpf_pseudo_call(insn)) - continue; - insn->off =3D 0; - insn->imm =3D env->insn_aux_data[i].call_imm; + /* we already rolled back to the clean orig_prog when blinded is true */ + if (!blinded) { + for (i =3D 0, insn =3D prog->insnsi; i < prog->len; i++, insn++) { + if (!bpf_pseudo_call(insn)) + continue; + insn->off =3D 0; + insn->imm =3D env->insn_aux_data[i].call_imm; + } } bpf_prog_jit_attempt_done(prog); return err; --=20 2.47.3