From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-3.rrze.uni-erlangen.de (mx-rz-3.rrze.uni-erlangen.de [131.188.11.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E46C754918; Tue, 3 Jun 2025 21:08:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.22 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748984937; cv=none; b=Qpdxl8t4Gi7MRvyjFpwQrMeGuMR7DuhiuyiodJ+ksCImtCieZ99AJFUTaej0zuBJQpPJvEvcB0G72oLt65AnjtcYOBFJDDYTJr+ohVxccPDvBLlP0ek5lXgEegZQi/2k54tjbqdb85Sh6AaNrz7GX+w5V3VxTIH9f6+YnsYuFvo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748984937; c=relaxed/simple; bh=aWBgCi3B1Scf1zMtzS8uMObJlL/HdGNSPVktksO3kbI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sfkukSIpLDrpIZCd3gimBDgYzGwv1HyGLEFQ+HstdY7mHyy0KDLXNU6g09E1KEhNJtCzHlgJijZsShMO3yp1lLb78LQS3iaKUBOTC/9MkrxmfQYcKFxyagy/jT5RgWcrx1EH9e9nkTtZWdnMafbu3lymmIN307C8N6YNheDMhC8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=nTEM85N1; arc=none smtp.client-ip=131.188.11.22 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="nTEM85N1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748984470; bh=2Lf9tfZYrOpzCpGiAeazFx+eYcO3ZXEjViLfHjoKxvc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=nTEM85N1SHQYBShr1GceFe4QgYoYiCtWda0eqtuwoWSdUj007/bn3nxjsDnPS7Zqg itwtev5rBKG3nXIQD1m8IOiYHFMtBqIhtlvPtLuwXLtS9xG9MeqeMYOz+X8inKIOoA XM2TpZJM1mxD6E9UF3nbrLJflQo1D2xoSKHkxegJY5jaq/Z7l7r+ymcwadTm0vKkIC fyYucJTtduSuE0d9hA5sLk0NFQ7z39iMA4z0tRLpT0HGjeDB6aG+eo8TFn8TjD7lEX H3/wlqGmWqxU8ss6IHu4WPNGwGxPJg1S8PWxLqBWNFB/Z3kKUqrd53LfLdVQV/LYvK GMU5GeGq5Q0UQ== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-3.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBjmk3hHVz1yNg; Tue, 3 Jun 2025 23:01:10 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck4.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX1/NbMnGaXYiRuu9sySXs39BAyREF6WAvO0=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBjmd5NPMz1xxq; Tue, 3 Jun 2025 23:01:05 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 1/9] bpf: Move insn if/else into do_check_insn() Date: Tue, 3 Jun 2025 22:57:52 +0200 Message-ID: <20250603205800.334980-2-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is required to catch the errors later and fall back to a nospec if on a speculative path. Eliminate the regs variable as it is only used once and insn_idx is not modified in-between the definition and usage. Do not pass insn but compute it in the function itself. As Eduard points out [1], insn is assumed to correspond to env->insn_idx in many places (e.g, __check_reg_arg()). Move code into do_check_insn(), replace * "continue" with "return 0" after modifying insn_idx * "goto process_bpf_exit" with "return PROCESS_BPF_EXIT" * "goto process_bpf_exit_full" with "return process_bpf_exit_full()" * "do_print_state =3D " with "*do_print_state =3D " [1] https://lore.kernel.org/all/293dbe3950a782b8eb3b87b71d7a967e120191fd.ca= mel@gmail.com/ Signed-off-by: Luis Gerhorst Acked-by: Kumar Kartikeya Dwivedi Acked-by: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan --- kernel/bpf/verifier.c | 428 ++++++++++++++++++++++-------------------- 1 file changed, 223 insertions(+), 205 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a7d6e0c5928b..2e70be0cad49 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19421,20 +19421,223 @@ static int save_aux_ptr_type(struct bpf_verifier= _env *env, enum bpf_reg_type typ return 0; } =20 +enum { + PROCESS_BPF_EXIT =3D 1 +}; + +static int process_bpf_exit_full(struct bpf_verifier_env *env, + bool *do_print_state, + bool exception_exit) +{ + /* We must do check_reference_leak here before + * prepare_func_exit to handle the case when + * state->curframe > 0, it may be a callback function, + * for which reference_state must match caller reference + * state when it exits. + */ + int err =3D check_resource_leak(env, exception_exit, + !env->cur_state->curframe, + "BPF_EXIT instruction in main prog"); + if (err) + return err; + + /* The side effect of the prepare_func_exit which is + * being skipped is that it frees bpf_func_state. + * Typically, process_bpf_exit will only be hit with + * outermost exit. copy_verifier_state in pop_stack will + * handle freeing of any extra bpf_func_state left over + * from not processing all nested function exits. We + * also skip return code checks as they are not needed + * for exceptional exits. + */ + if (exception_exit) + return PROCESS_BPF_EXIT; + + if (env->cur_state->curframe) { + /* exit from nested function */ + err =3D prepare_func_exit(env, &env->insn_idx); + if (err) + return err; + *do_print_state =3D true; + return 0; + } + + err =3D check_return_code(env, BPF_REG_0, "R0"); + if (err) + return err; + return PROCESS_BPF_EXIT; +} + +static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_stat= e) +{ + int err; + struct bpf_insn *insn =3D &env->prog->insnsi[env->insn_idx]; + u8 class =3D BPF_CLASS(insn->code); + + if (class =3D=3D BPF_ALU || class =3D=3D BPF_ALU64) { + err =3D check_alu_op(env, insn); + if (err) + return err; + + } else if (class =3D=3D BPF_LDX) { + bool is_ldsx =3D BPF_MODE(insn->code) =3D=3D BPF_MEMSX; + + /* Check for reserved fields is already done in + * resolve_pseudo_ldimm64(). + */ + err =3D check_load_mem(env, insn, false, is_ldsx, true, "ldx"); + if (err) + return err; + } else if (class =3D=3D BPF_STX) { + if (BPF_MODE(insn->code) =3D=3D BPF_ATOMIC) { + err =3D check_atomic(env, insn); + if (err) + return err; + env->insn_idx++; + return 0; + } + + if (BPF_MODE(insn->code) !=3D BPF_MEM || insn->imm !=3D 0) { + verbose(env, "BPF_STX uses reserved fields\n"); + return -EINVAL; + } + + err =3D check_store_reg(env, insn, false); + if (err) + return err; + } else if (class =3D=3D BPF_ST) { + enum bpf_reg_type dst_reg_type; + + if (BPF_MODE(insn->code) !=3D BPF_MEM || + insn->src_reg !=3D BPF_REG_0) { + verbose(env, "BPF_ST uses reserved fields\n"); + return -EINVAL; + } + /* check src operand */ + err =3D check_reg_arg(env, insn->dst_reg, SRC_OP); + if (err) + return err; + + dst_reg_type =3D cur_regs(env)[insn->dst_reg].type; + + /* check that memory (dst_reg + off) is writeable */ + err =3D check_mem_access(env, env->insn_idx, insn->dst_reg, + insn->off, BPF_SIZE(insn->code), + BPF_WRITE, -1, false, false); + if (err) + return err; + + err =3D save_aux_ptr_type(env, dst_reg_type, false); + if (err) + return err; + } else if (class =3D=3D BPF_JMP || class =3D=3D BPF_JMP32) { + u8 opcode =3D BPF_OP(insn->code); + + env->jmps_processed++; + if (opcode =3D=3D BPF_CALL) { + if (BPF_SRC(insn->code) !=3D BPF_K || + (insn->src_reg !=3D BPF_PSEUDO_KFUNC_CALL && + insn->off !=3D 0) || + (insn->src_reg !=3D BPF_REG_0 && + insn->src_reg !=3D BPF_PSEUDO_CALL && + insn->src_reg !=3D BPF_PSEUDO_KFUNC_CALL) || + insn->dst_reg !=3D BPF_REG_0 || class =3D=3D BPF_JMP32) { + verbose(env, "BPF_CALL uses reserved fields\n"); + return -EINVAL; + } + + if (env->cur_state->active_locks) { + if ((insn->src_reg =3D=3D BPF_REG_0 && + insn->imm !=3D BPF_FUNC_spin_unlock) || + (insn->src_reg =3D=3D BPF_PSEUDO_KFUNC_CALL && + (insn->off !=3D 0 || !kfunc_spin_allowed(insn->imm)))) { + verbose(env, + "function calls are not allowed while holding a lock\n"); + return -EINVAL; + } + } + if (insn->src_reg =3D=3D BPF_PSEUDO_CALL) { + err =3D check_func_call(env, insn, &env->insn_idx); + } else if (insn->src_reg =3D=3D BPF_PSEUDO_KFUNC_CALL) { + err =3D check_kfunc_call(env, insn, &env->insn_idx); + if (!err && is_bpf_throw_kfunc(insn)) + return process_bpf_exit_full(env, do_print_state, true); + } else { + err =3D check_helper_call(env, insn, &env->insn_idx); + } + if (err) + return err; + + mark_reg_scratched(env, BPF_REG_0); + } else if (opcode =3D=3D BPF_JA) { + if (BPF_SRC(insn->code) !=3D BPF_K || + insn->src_reg !=3D BPF_REG_0 || + insn->dst_reg !=3D BPF_REG_0 || + (class =3D=3D BPF_JMP && insn->imm !=3D 0) || + (class =3D=3D BPF_JMP32 && insn->off !=3D 0)) { + verbose(env, "BPF_JA uses reserved fields\n"); + return -EINVAL; + } + + if (class =3D=3D BPF_JMP) + env->insn_idx +=3D insn->off + 1; + else + env->insn_idx +=3D insn->imm + 1; + return 0; + } else if (opcode =3D=3D BPF_EXIT) { + if (BPF_SRC(insn->code) !=3D BPF_K || + insn->imm !=3D 0 || + insn->src_reg !=3D BPF_REG_0 || + insn->dst_reg !=3D BPF_REG_0 || + class =3D=3D BPF_JMP32) { + verbose(env, "BPF_EXIT uses reserved fields\n"); + return -EINVAL; + } + return process_bpf_exit_full(env, do_print_state, false); + } else { + err =3D check_cond_jmp_op(env, insn, &env->insn_idx); + if (err) + return err; + } + } else if (class =3D=3D BPF_LD) { + u8 mode =3D BPF_MODE(insn->code); + + if (mode =3D=3D BPF_ABS || mode =3D=3D BPF_IND) { + err =3D check_ld_abs(env, insn); + if (err) + return err; + + } else if (mode =3D=3D BPF_IMM) { + err =3D check_ld_imm(env, insn); + if (err) + return err; + + env->insn_idx++; + sanitize_mark_insn_seen(env); + } else { + verbose(env, "invalid BPF_LD mode\n"); + return -EINVAL; + } + } else { + verbose(env, "unknown insn class %d\n", class); + return -EINVAL; + } + + env->insn_idx++; + return 0; +} + static int do_check(struct bpf_verifier_env *env) { bool pop_log =3D !(env->log.level & BPF_LOG_LEVEL2); struct bpf_verifier_state *state =3D env->cur_state; struct bpf_insn *insns =3D env->prog->insnsi; - struct bpf_reg_state *regs; int insn_cnt =3D env->prog->len; bool do_print_state =3D false; int prev_insn_idx =3D -1; =20 for (;;) { - bool exception_exit =3D false; struct bpf_insn *insn; - u8 class; int err; =20 /* reset current history entry on each new instruction */ @@ -19448,7 +19651,6 @@ static int do_check(struct bpf_verifier_env *env) } =20 insn =3D &insns[env->insn_idx]; - class =3D BPF_CLASS(insn->code); =20 if (++env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) { verbose(env, @@ -19518,215 +19720,31 @@ static int do_check(struct bpf_verifier_env *env) return err; } =20 - regs =3D cur_regs(env); sanitize_mark_insn_seen(env); prev_insn_idx =3D env->insn_idx; =20 - if (class =3D=3D BPF_ALU || class =3D=3D BPF_ALU64) { - err =3D check_alu_op(env, insn); - if (err) - return err; - - } else if (class =3D=3D BPF_LDX) { - bool is_ldsx =3D BPF_MODE(insn->code) =3D=3D BPF_MEMSX; - - /* Check for reserved fields is already done in - * resolve_pseudo_ldimm64(). - */ - err =3D check_load_mem(env, insn, false, is_ldsx, true, - "ldx"); - if (err) - return err; - } else if (class =3D=3D BPF_STX) { - if (BPF_MODE(insn->code) =3D=3D BPF_ATOMIC) { - err =3D check_atomic(env, insn); - if (err) - return err; - env->insn_idx++; - continue; - } - - if (BPF_MODE(insn->code) !=3D BPF_MEM || insn->imm !=3D 0) { - verbose(env, "BPF_STX uses reserved fields\n"); - return -EINVAL; - } - - err =3D check_store_reg(env, insn, false); - if (err) - return err; - } else if (class =3D=3D BPF_ST) { - enum bpf_reg_type dst_reg_type; - - if (BPF_MODE(insn->code) !=3D BPF_MEM || - insn->src_reg !=3D BPF_REG_0) { - verbose(env, "BPF_ST uses reserved fields\n"); - return -EINVAL; - } - /* check src operand */ - err =3D check_reg_arg(env, insn->dst_reg, SRC_OP); - if (err) - return err; - - dst_reg_type =3D regs[insn->dst_reg].type; - - /* check that memory (dst_reg + off) is writeable */ - err =3D check_mem_access(env, env->insn_idx, insn->dst_reg, - insn->off, BPF_SIZE(insn->code), - BPF_WRITE, -1, false, false); - if (err) - return err; - - err =3D save_aux_ptr_type(env, dst_reg_type, false); - if (err) - return err; - } else if (class =3D=3D BPF_JMP || class =3D=3D BPF_JMP32) { - u8 opcode =3D BPF_OP(insn->code); - - env->jmps_processed++; - if (opcode =3D=3D BPF_CALL) { - if (BPF_SRC(insn->code) !=3D BPF_K || - (insn->src_reg !=3D BPF_PSEUDO_KFUNC_CALL - && insn->off !=3D 0) || - (insn->src_reg !=3D BPF_REG_0 && - insn->src_reg !=3D BPF_PSEUDO_CALL && - insn->src_reg !=3D BPF_PSEUDO_KFUNC_CALL) || - insn->dst_reg !=3D BPF_REG_0 || - class =3D=3D BPF_JMP32) { - verbose(env, "BPF_CALL uses reserved fields\n"); - return -EINVAL; - } - - if (env->cur_state->active_locks) { - if ((insn->src_reg =3D=3D BPF_REG_0 && insn->imm !=3D BPF_FUNC_spin_u= nlock) || - (insn->src_reg =3D=3D BPF_PSEUDO_KFUNC_CALL && - (insn->off !=3D 0 || !kfunc_spin_allowed(insn->imm)))) { - verbose(env, "function calls are not allowed while holding a lock\n"= ); - return -EINVAL; - } - } - if (insn->src_reg =3D=3D BPF_PSEUDO_CALL) { - err =3D check_func_call(env, insn, &env->insn_idx); - } else if (insn->src_reg =3D=3D BPF_PSEUDO_KFUNC_CALL) { - err =3D check_kfunc_call(env, insn, &env->insn_idx); - if (!err && is_bpf_throw_kfunc(insn)) { - exception_exit =3D true; - goto process_bpf_exit_full; - } - } else { - err =3D check_helper_call(env, insn, &env->insn_idx); - } - if (err) - return err; - - mark_reg_scratched(env, BPF_REG_0); - } else if (opcode =3D=3D BPF_JA) { - if (BPF_SRC(insn->code) !=3D BPF_K || - insn->src_reg !=3D BPF_REG_0 || - insn->dst_reg !=3D BPF_REG_0 || - (class =3D=3D BPF_JMP && insn->imm !=3D 0) || - (class =3D=3D BPF_JMP32 && insn->off !=3D 0)) { - verbose(env, "BPF_JA uses reserved fields\n"); - return -EINVAL; - } - - if (class =3D=3D BPF_JMP) - env->insn_idx +=3D insn->off + 1; - else - env->insn_idx +=3D insn->imm + 1; - continue; - - } else if (opcode =3D=3D BPF_EXIT) { - if (BPF_SRC(insn->code) !=3D BPF_K || - insn->imm !=3D 0 || - insn->src_reg !=3D BPF_REG_0 || - insn->dst_reg !=3D BPF_REG_0 || - class =3D=3D BPF_JMP32) { - verbose(env, "BPF_EXIT uses reserved fields\n"); - return -EINVAL; - } -process_bpf_exit_full: - /* We must do check_reference_leak here before - * prepare_func_exit to handle the case when - * state->curframe > 0, it may be a callback - * function, for which reference_state must - * match caller reference state when it exits. - */ - err =3D check_resource_leak(env, exception_exit, !env->cur_state->curf= rame, - "BPF_EXIT instruction in main prog"); - if (err) - return err; - - /* The side effect of the prepare_func_exit - * which is being skipped is that it frees - * bpf_func_state. Typically, process_bpf_exit - * will only be hit with outermost exit. - * copy_verifier_state in pop_stack will handle - * freeing of any extra bpf_func_state left over - * from not processing all nested function - * exits. We also skip return code checks as - * they are not needed for exceptional exits. - */ - if (exception_exit) - goto process_bpf_exit; - - if (state->curframe) { - /* exit from nested function */ - err =3D prepare_func_exit(env, &env->insn_idx); - if (err) - return err; - do_print_state =3D true; - continue; - } - - err =3D check_return_code(env, BPF_REG_0, "R0"); - if (err) - return err; + err =3D do_check_insn(env, &do_print_state); + if (err < 0) { + return err; + } else if (err =3D=3D PROCESS_BPF_EXIT) { process_bpf_exit: - mark_verifier_state_scratched(env); - update_branch_counts(env, env->cur_state); - err =3D pop_stack(env, &prev_insn_idx, - &env->insn_idx, pop_log); - if (err < 0) { - if (err !=3D -ENOENT) - return err; - break; - } else { - if (verifier_bug_if(env->cur_state->loop_entry, env, - "broken loop detection")) - return -EFAULT; - do_print_state =3D true; - continue; - } - } else { - err =3D check_cond_jmp_op(env, insn, &env->insn_idx); - if (err) - return err; - } - } else if (class =3D=3D BPF_LD) { - u8 mode =3D BPF_MODE(insn->code); - - if (mode =3D=3D BPF_ABS || mode =3D=3D BPF_IND) { - err =3D check_ld_abs(env, insn); - if (err) - return err; - - } else if (mode =3D=3D BPF_IMM) { - err =3D check_ld_imm(env, insn); - if (err) + mark_verifier_state_scratched(env); + update_branch_counts(env, env->cur_state); + err =3D pop_stack(env, &prev_insn_idx, &env->insn_idx, + pop_log); + if (err < 0) { + if (err !=3D -ENOENT) return err; - - env->insn_idx++; - sanitize_mark_insn_seen(env); + break; } else { - verbose(env, "invalid BPF_LD mode\n"); - return -EINVAL; + if (verifier_bug_if(env->cur_state->loop_entry, env, + "broken loop detection")) + return -EFAULT; + do_print_state =3D true; + continue; } - } else { - verbose(env, "unknown insn class %d\n", class); - return -EINVAL; } - - env->insn_idx++; + WARN_ON_ONCE(err); } =20 return 0; --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-3.rrze.uni-erlangen.de (mx-rz-3.rrze.uni-erlangen.de [131.188.11.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04A4F1F12F4; Tue, 3 Jun 2025 21:04:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.22 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748984660; cv=none; b=ckA6r6/1/jh9ammce36dnz4n0mJooWcbdQalao89jq2fGjqa2T24AWinRViBDSqeOWBEIrCqw/RX8qh2qIJ/BXhe14H/C/jrm6bjLjDU4UDWnmOap7Sl4Bcd/zlbcFiO6oZuFjsv/nyIBUUKuqJ8ax0hC9Qm+BCLfVso/P2oqc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748984660; c=relaxed/simple; bh=JA8VLAJVwnfb2S1nrJrM04QNaNui1/wkR/RKqV9Jvhc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RJnLPjfoJsnnfEvOWfGvPPSfnOz5GhRr6WqkQqdFRMxRfhIRmL7Os1Y9m0JifFr3Tzh8Eb+TVxCY+Ss7b0Eu4Ec+c2lFvjf1TfBCjf7wttfKk20Y7nrryHUydjivphSGl4lZcBz5JLaQchn5gs1IV8vVNwa2PZd1IBycYbpPt+c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=wbOymEgZ; arc=none smtp.client-ip=131.188.11.22 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="wbOymEgZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748984655; bh=2UwvqxyJ2K/g48t8U4HwZCTixs5aBO0jfSMiokXjqnU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=wbOymEgZJKXNIzpeaUKrrp5oi0BLy74WRh9F3bwtCLqd61ATLudI+i9TyAASaewVx 1KCMgLhcMmSyhK5V2Gp4mHHdUrNt3yssRKlmRoRWNe/1aX51cOV1u1X1L0S1zYCPxr E97Q6dNoPofYhPz6xe02s6xanKM1LKzcIbl6C03D+ylvz8wP7Z3Q4ybppxrGIeXM7r FyiGksjD6Y+m4BZ9Wywtm8yksqQthGPndCVETWx6/tAqG5EgN1ie1zc6LyaEDfo/uW RCyMLneCN9We0rW4+IVRP3rS1ONtTLecLrtsaQ7Ev9B+M84UW7h3/IIKvj0Z4rBCvr S4CCr13iAm4vA== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-3.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBjrH49Rpz1yZg; Tue, 3 Jun 2025 23:04:15 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck1.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX1+WzYk6/QiU80/dd/SlFMOAjg12F+WTVXc=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBjrB4xLxz1y5T; Tue, 3 Jun 2025 23:04:10 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 2/9] bpf: Return -EFAULT on misconfigurations Date: Tue, 3 Jun 2025 22:57:53 +0200 Message-ID: <20250603205800.334980-3-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mark these cases as non-recoverable to later prevent them from being caught when they occur during speculative path verification. Eduard writes [1]: The only pace I'm aware of that might act upon specific error code from verifier syscall is libbpf. Looking through libbpf code, it seems that this change does not interfere with libbpf. [1] https://lore.kernel.org/all/785b4531ce3b44a84059a4feb4ba458c68fce719.ca= mel@gmail.com/ Signed-off-by: Luis Gerhorst Reviewed-by: Eduard Zingerman Acked-by: Kumar Kartikeya Dwivedi Acked-by: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan --- kernel/bpf/verifier.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2e70be0cad49..c76fbf46a365 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8954,7 +8954,7 @@ static int resolve_map_arg_type(struct bpf_verifier_e= nv *env, if (!meta->map_ptr) { /* kernel subsystem misconfigured verifier */ verbose(env, "invalid map_ptr to access map->type\n"); - return -EACCES; + return -EFAULT; } =20 switch (meta->map_ptr->map_type) { @@ -9642,7 +9642,7 @@ static int check_func_arg(struct bpf_verifier_env *en= v, u32 arg, * that kernel subsystem misconfigured verifier */ verbose(env, "invalid map_ptr to access map->key\n"); - return -EACCES; + return -EFAULT; } key_size =3D meta->map_ptr->key_size; err =3D check_helper_mem_access(env, regno, key_size, BPF_READ, false, N= ULL); @@ -9669,7 +9669,7 @@ static int check_func_arg(struct bpf_verifier_env *en= v, u32 arg, if (!meta->map_ptr) { /* kernel subsystem misconfigured verifier */ verbose(env, "invalid map_ptr to access map->value\n"); - return -EACCES; + return -EFAULT; } meta->raw_mode =3D arg_type & MEM_UNINIT; err =3D check_helper_mem_access(env, regno, meta->map_ptr->value_size, @@ -10965,7 +10965,7 @@ record_func_map(struct bpf_verifier_env *env, struc= t bpf_call_arg_meta *meta, =20 if (map =3D=3D NULL) { verbose(env, "kernel subsystem misconfigured verifier\n"); - return -EINVAL; + return -EFAULT; } =20 /* In case of read-only, some additional restrictions @@ -11004,7 +11004,7 @@ record_func_key(struct bpf_verifier_env *env, struc= t bpf_call_arg_meta *meta, return 0; if (!map || map->map_type !=3D BPF_MAP_TYPE_PROG_ARRAY) { verbose(env, "kernel subsystem misconfigured verifier\n"); - return -EINVAL; + return -EFAULT; } =20 reg =3D ®s[BPF_REG_3]; @@ -11258,7 +11258,7 @@ static int check_helper_call(struct bpf_verifier_en= v *env, struct bpf_insn *insn if (changes_data && fn->arg1_type !=3D ARG_PTR_TO_CTX) { verbose(env, "kernel subsystem misconfigured func %s#%d: r1 !=3D ctx\n", func_id_name(func_id), func_id); - return -EINVAL; + return -EFAULT; } =20 memset(&meta, 0, sizeof(meta)); @@ -11560,7 +11560,7 @@ static int check_helper_call(struct bpf_verifier_en= v *env, struct bpf_insn *insn if (meta.map_ptr =3D=3D NULL) { verbose(env, "kernel subsystem misconfigured verifier\n"); - return -EINVAL; + return -EFAULT; } =20 if (func_id =3D=3D BPF_FUNC_map_lookup_elem && @@ -16729,7 +16729,7 @@ static int check_ld_imm(struct bpf_verifier_env *en= v, struct bpf_insn *insn) dst_reg->type =3D CONST_PTR_TO_MAP; } else { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 return 0; @@ -16776,7 +16776,7 @@ static int check_ld_abs(struct bpf_verifier_env *en= v, struct bpf_insn *insn) =20 if (!env->ops->gen_ld_abs) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 if (insn->dst_reg !=3D BPF_REG_0 || insn->off !=3D 0 || @@ -20816,7 +20816,7 @@ static int convert_ctx_accesses(struct bpf_verifier= _env *env) -(subprogs[0].stack_depth + 8)); if (epilogue_cnt >=3D INSN_BUF_SIZE) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } else if (epilogue_cnt) { /* Save the ARG_PTR_TO_CTX for the epilogue to use */ cnt =3D 0; @@ -20839,13 +20839,13 @@ static int convert_ctx_accesses(struct bpf_verifi= er_env *env) if (ops->gen_prologue || env->seen_direct_write) { if (!ops->gen_prologue) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } cnt =3D ops->gen_prologue(insn_buf, env->seen_direct_write, env->prog); if (cnt >=3D INSN_BUF_SIZE) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } else if (cnt) { new_prog =3D bpf_patch_insn_data(env, 0, insn_buf, cnt); if (!new_prog) @@ -21002,7 +21002,7 @@ static int convert_ctx_accesses(struct bpf_verifier= _env *env) =20 if (type =3D=3D BPF_WRITE) { verbose(env, "bpf verifier narrow ctx access misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 size_code =3D BPF_H; @@ -21021,7 +21021,7 @@ static int convert_ctx_accesses(struct bpf_verifier= _env *env) if (cnt =3D=3D 0 || cnt >=3D INSN_BUF_SIZE || (ctx_field_size && !target_size)) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 if (is_narrower_load && size < target_size) { @@ -21029,7 +21029,7 @@ static int convert_ctx_accesses(struct bpf_verifier= _env *env) off, size, size_default) * 8; if (shift && cnt + 1 >=3D INSN_BUF_SIZE) { verbose(env, "bpf verifier narrow ctx load misconfigured\n"); - return -EINVAL; + return -EFAULT; } if (ctx_field_size <=3D 4) { if (shift) @@ -21794,7 +21794,7 @@ static int do_misc_fixups(struct bpf_verifier_env *= env) cnt =3D env->ops->gen_ld_abs(insn, insn_buf); if (cnt =3D=3D 0 || cnt >=3D INSN_BUF_SIZE) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 new_prog =3D bpf_patch_insn_data(env, i + delta, insn_buf, cnt); @@ -22130,7 +22130,7 @@ static int do_misc_fixups(struct bpf_verifier_env *= env) goto patch_map_ops_generic; if (cnt <=3D 0 || cnt >=3D INSN_BUF_SIZE) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 new_prog =3D bpf_patch_insn_data(env, i + delta, @@ -22490,7 +22490,7 @@ static int do_misc_fixups(struct bpf_verifier_env *= env) !map_ptr->ops->map_poke_untrack || !map_ptr->ops->map_poke_run) { verbose(env, "bpf verifier is misconfigured\n"); - return -EINVAL; + return -EFAULT; } =20 ret =3D map_ptr->ops->map_poke_track(map_ptr, prog->aux); --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-3.rrze.uni-erlangen.de (mx-rz-3.rrze.uni-erlangen.de [131.188.11.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D32CF19ABC3; Tue, 3 Jun 2025 21:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.22 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748984843; cv=none; b=bJWgQCODcQh5gHzzQ5gSlrPlyGEeJquRaF2ggQ0xumUMFX2BlwpGCUfQ1otEOBwGYTHyXJeL8/yB9ryHakxFJja4OwQrNkkAtUV3mu2R8Vr0R9hDbGYBrmdZAotVxwxD25o1TwWFR2YLk9MYp+1ESy6LibiwHiBvfx4Q4kkJ5+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748984843; c=relaxed/simple; bh=J/k4gYkZjn289vq7cf6/VQwnVRCn685tu5A4lQ4ohUY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gdgBO4G0VtKWZqfmfKaiboo+bHbHXZKG5wxx1mXtZDuoYsp/fWVwm1rrgWNm1NvZcw+iQ6ZmBy7W9rMP81pEdxe4Q0M19MQad3j6BDvLsR3+30rQcYZq8y4ZINQ7BrD2TvltKYFChF28j4EZmQXZU+8lpVpOWiCcPKmlD+9TIVg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=BHXbgFqX; arc=none smtp.client-ip=131.188.11.22 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="BHXbgFqX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748984839; bh=YVxtmO4xn8cSl7FBJlgzMb9wooxDCpxLcAclWqMaFmg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=BHXbgFqXtexbXiv6sFCqsIvSTwIIN1FMHcHaPV/j0mKpceFyEZkaMimsfvHqxhHYn g7m2ycAwHH99GxDrX5JaRlXlOXQESK/D5kJc/V3Bi2OYgH2WlKfAXD/BImG1fLtZ8w L89Kbi3GXnubdsYpALrgw0T6MOgu7mni3knFqc/a5Qw9n38LE3Zy5Lp8pzQM9iVWHL Ur3kZpqYhF/2wCmmvjIdnxrNWyVejF32jtn3vg/LVTlvctYHJg71tQFLrIyYFTQjqW P7eyaobt3/S5GVDmOWhUpxQFqHxc4Xe+6u+jOHBz3n2BYVZrI10O8kJpfVWDOM+P2u IibavPDvr3hcg== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-3.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBjvq3ZCmz1yTV; Tue, 3 Jun 2025 23:07:19 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck5.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX18Ca/MPWWDtOYDvTUE1FQFkkdhcs5t0j4o=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBjvl5qDwz1y0m; Tue, 3 Jun 2025 23:07:15 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 3/9] bpf: Return -EFAULT on internal errors Date: Tue, 3 Jun 2025 22:57:54 +0200 Message-ID: <20250603205800.334980-4-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This prevents us from trying to recover from these on speculative paths in the future. Signed-off-by: Luis Gerhorst Reviewed-by: Eduard Zingerman Acked-by: Kumar Kartikeya Dwivedi Acked-by: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan --- kernel/bpf/verifier.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index c76fbf46a365..46cf737acad5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11653,7 +11653,7 @@ static int check_helper_call(struct bpf_verifier_en= v *env, struct bpf_insn *insn verbose(env, "verifier internal error:"); verbose(env, "func %s has non-overwritten BPF_PTR_POISON return type\n= ", func_id_name(func_id)); - return -EINVAL; + return -EFAULT; } ret_btf =3D btf_vmlinux; ret_btf_id =3D *fn->ret_btf_id; @@ -15278,12 +15278,12 @@ static int adjust_reg_min_max_vals(struct bpf_ver= ifier_env *env, if (WARN_ON_ONCE(ptr_reg)) { print_verifier_state(env, vstate, vstate->curframe, true); verbose(env, "verifier internal error: unexpected ptr_reg\n"); - return -EINVAL; + return -EFAULT; } if (WARN_ON(!src_reg)) { print_verifier_state(env, vstate, vstate->curframe, true); verbose(env, "verifier internal error: no src_reg\n"); - return -EINVAL; + return -EFAULT; } err =3D adjust_scalar_min_max_vals(env, insn, dst_reg, *src_reg); if (err) --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-1.rrze.uni-erlangen.de (mx-rz-1.rrze.uni-erlangen.de [131.188.11.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 287882248A6; Tue, 3 Jun 2025 21:13:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985230; cv=none; b=V/QuFffWRo8vo0ThGVSuBAJ2jCHfmdnJJhMYhJApfGDwJWZBuDx9XV6NL867wrD2Z5pVltJny4Q5lwsK1Kk0ncjknU5BQEogjX/EpuhsgedrDN0V+Lu47Ltkt3KIsESSMqcqDJsGXLLe+hR/Ok2H/nTKHxHgwKXZcidm31wnoXE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985230; c=relaxed/simple; bh=aMDRsr8BgEnv2VC/stL9/Fn9Dhd4Wg8D2PKZI6RbKyM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nQl/YXSpq5uJos2N6ZWQK4WFnXFJjNbpkyHRS9VVKgVySKwih+KYp1WyaP0BdhYdyMah2hspmnGzhlUC0uBF4AiJbDyYsxaayxexPPa7rCDj5a+cCWXinW1a1JyhlqJTcNi0QLHut38FqQzT8OFo/jhlKMGgcTZk7KuBFwxYInw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=u72N0HKY; arc=none smtp.client-ip=131.188.11.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="u72N0HKY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748985226; bh=uCHDWZ/YSKO/QVzz8XQFkdmZfL60SSwm4+MhipptReg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=u72N0HKYOGr0NvUlYLTRCuXIZlktKw1V8ooPxftf94M73oCqM7EFWG8D71veK0RBc BnK4N3BnOc6MEFioZ3WfRSijkhcJs3qdTIJrTZ721vBmR3QOhkq8Xp4UtAh+H6FHdj ntbm6Wvpvrbj2YdD8IqQMcWnvsEunG5HwKGJAKtPzLJkaJai9LXqhmIqDelZJqOfZx e6OmxJ332JUfiIKne1407EBQNEPCBe/K9gx7ylUo8xNhWaiJ09HIPCBIqC30uqgh0d CT1HUrQd7PR8ZCBswl4GUdD3vrfrJljMRctiERzpi4I6UJgrzR3cLlA5Rk11SM/BCJ q0l3P7c7WzCzw== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-1.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBk3F6FY9z8sbt; Tue, 3 Jun 2025 23:13:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck4.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX1+u239X7Kf1MUWjocRBQ3Jv/Tc/rDSzZ4g=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBk396YP9z8sgj; Tue, 3 Jun 2025 23:13:41 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 4/9] bpf, arm64, powerpc: Add bpf_jit_bypass_spec_v1/v4() Date: Tue, 3 Jun 2025 23:13:18 +0200 Message-ID: <20250603211318.337474-1-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable JITs can set bpf_jit_bypass_spec_v1/v4() if they want the verifier to skip analysis/patching for the respective vulnerability. For v4, this will reduce the number of barriers the verifier inserts. For v1, it allows more programs to be accepted. The primary motivation for this is to not regress unpriv BPF's performance on ARM64 in a future commit where BPF_NOSPEC is also used against Spectre v1. This has the user-visible change that v1-induced rejections on non-vulnerable PowerPC CPUs are avoided. For now, this does not change the semantics of BPF_NOSPEC. It is still a v4-only barrier and must not be implemented if bypass_spec_v4 is always true for the arch. Changing it to a v1 AND v4-barrier is done in a future commit. As an alternative to bypass_spec_v1/v4, one could introduce NOSPEC_V1 AND NOSPEC_V4 instructions and allow backends to skip their lowering as suggested by commit f5e81d111750 ("bpf: Introduce BPF nospec instruction for mitigating Spectre v4"). Adding bpf_jit_bypass_spec_v1/v4() was found to be preferable for the following reason: * bypass_spec_v1/v4 benefits non-vulnerable CPUs: Always performing the same analysis (not taking into account whether the current CPU is vulnerable), needlessly restricts users of CPUs that are not vulnerable. The only use case for this would be portability-testing, but this can later be added easily when needed by allowing users to force bypass_spec_v1/v4 to false. * Portability is still acceptable: Directly disabling the analysis instead of skipping the lowering of BPF_NOSPEC(_V1/V4) might allow programs on non-vulnerable CPUs to be accepted while the program will be rejected on vulnerable CPUs. With the fallback to speculation barriers for Spectre v1 implemented in a future commit, this will only affect programs that do variable stack-accesses or are very complex. For PowerPC, the SEC_FTR checking in bpf_jit_bypass_spec_v4() is based on the check that was previously located in the BPF_NOSPEC case. For LoongArch, it would likely be safe to set both bpf_jit_bypass_spec_v1() and _v4() according to commit=C2=A0a6f6a95f2580 ("LoongArch, bpf: Fix jit to skip speculation barrier opcode"). This is omitted here as I am unable to do any testing for LoongArch. Hari's ack concerns the PowerPC part only. Signed-off-by: Luis Gerhorst Acked-by: Hari Bathini Cc: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan Acked-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 21 ++++++++++++--------- arch/powerpc/net/bpf_jit_comp64.c | 21 +++++++++++++++++---- include/linux/bpf.h | 11 +++++++++-- kernel/bpf/core.c | 15 +++++++++++++++ 4 files changed, 53 insertions(+), 15 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index da8b89dd2910..2cab9063f563 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1632,15 +1632,7 @@ static int build_insn(const struct bpf_insn *insn, s= truct jit_ctx *ctx, =20 /* speculation barrier */ case BPF_ST | BPF_NOSPEC: - /* - * Nothing required here. - * - * In case of arm64, we rely on the firmware mitigation of - * Speculative Store Bypass as controlled via the ssbd kernel - * parameter. Whenever the mitigation is enabled, it works - * for all of the kernel code with no need to provide any - * additional instructions. - */ + /* See bpf_jit_bypass_spec_v4() */ break; =20 /* ST: *(size *)(dst + off) =3D imm */ @@ -2911,6 +2903,17 @@ bool bpf_jit_supports_percpu_insn(void) return true; } =20 +bool bpf_jit_bypass_spec_v4(void) +{ + /* In case of arm64, we rely on the firmware mitigation of Speculative + * Store Bypass as controlled via the ssbd kernel parameter. Whenever + * the mitigation is enabled, it works for all of the kernel code with + * no need to provide any additional instructions. Therefore, skip + * inserting nospec insns against Spectre v4. + */ + return true; +} + bool bpf_jit_inlines_helper_call(s32 imm) { switch (imm) { diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_c= omp64.c index 5daa77aee7f7..a4335761b7f9 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -370,6 +370,23 @@ static int bpf_jit_emit_tail_call(u32 *image, struct c= odegen_context *ctx, u32 o return 0; } =20 +bool bpf_jit_bypass_spec_v1(void) +{ +#if defined(CONFIG_PPC_E500) || defined(CONFIG_PPC_BOOK3S_64) + return !(security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) && + security_ftr_enabled(SEC_FTR_BNDS_CHK_SPEC_BAR)); +#else + return true; +#endif +} + +bool bpf_jit_bypass_spec_v4(void) +{ + return !(security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) && + security_ftr_enabled(SEC_FTR_STF_BARRIER) && + stf_barrier_type_get() !=3D STF_BARRIER_NONE); +} + /* * We spill into the redzone always, even if the bpf program has its own s= tackframe. * Offsets hardcoded based on BPF_PPC_STACK_SAVE -- see bpf_jit_stack_loca= l() @@ -791,10 +808,6 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image= , u32 *fimage, struct code * BPF_ST NOSPEC (speculation barrier) */ case BPF_ST | BPF_NOSPEC: - if (!security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) || - !security_ftr_enabled(SEC_FTR_STF_BARRIER)) - break; - switch (stf_barrier) { case STF_BARRIER_EIEIO: EMIT(PPC_RAW_EIEIO() | 0x02000000); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5b25d278409b..5dd556e89cce 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2288,6 +2288,9 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array= *array, return ret; } =20 +bool bpf_jit_bypass_spec_v1(void); +bool bpf_jit_bypass_spec_v4(void); + #ifdef CONFIG_BPF_SYSCALL DECLARE_PER_CPU(int, bpf_prog_active); extern struct mutex bpf_stats_enabled_mutex; @@ -2475,12 +2478,16 @@ static inline bool bpf_allow_uninit_stack(const str= uct bpf_token *token) =20 static inline bool bpf_bypass_spec_v1(const struct bpf_token *token) { - return cpu_mitigations_off() || bpf_token_capable(token, CAP_PERFMON); + return bpf_jit_bypass_spec_v1() || + cpu_mitigations_off() || + bpf_token_capable(token, CAP_PERFMON); } =20 static inline bool bpf_bypass_spec_v4(const struct bpf_token *token) { - return cpu_mitigations_off() || bpf_token_capable(token, CAP_PERFMON); + return bpf_jit_bypass_spec_v4() || + cpu_mitigations_off() || + bpf_token_capable(token, CAP_PERFMON); } =20 int bpf_map_new_fd(struct bpf_map *map, int flags); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index c20babbf998f..f9bd9625438b 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3034,6 +3034,21 @@ bool __weak bpf_jit_needs_zext(void) return false; } =20 +/* By default, enable the verifier's mitigations against Spectre v1 and v4= for + * all archs. The value returned must not change at runtime as there is + * currently no support for reloading programs that were loaded without + * mitigations. + */ +bool __weak bpf_jit_bypass_spec_v1(void) +{ + return false; +} + +bool __weak bpf_jit_bypass_spec_v4(void) +{ + return false; +} + /* Return true if the JIT inlines the call to the helper corresponding to * the imm. * --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-2.rrze.uni-erlangen.de (mx-rz-2.rrze.uni-erlangen.de [131.188.11.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2DFE3225792; Tue, 3 Jun 2025 21:17:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.21 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985439; cv=none; b=YVQrSpV0gabhYCm+Nsyh0eVhe2N57iFknrLcY9hrkcXjYZH/8U0dq6yX7poaRwlNSDwQ5U2MJ0ugj9WiUbGxvaONulEYg7cpW1/43hWst+SXMiStB7GJg92jU2oNRBYKh9QDP4JtKGJogjRDlLPDXCDW125XeIS7Ud+eaynLZoc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985439; c=relaxed/simple; bh=jV7gCpI03Am4DpiZtxSjBZzR00tXFBN9D2mswwkNkok=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dyXEsQ21NHqWiSY2pEyxi7w9ycCrc3snhm1fhGrTSp0ohJ9WS+WfMYQBAMDYBcNZsmOBheTS1Nd3Mma/kIb8sjjeh4l2YCb7RbrGrfQJE/rAhM1R8v0FxIVosayNk+ExYpmBUD0P//HeG/4UpTYFajR2m7n7Tqe2K1eTVNQ0Rgk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=VdwI+qCs; arc=none smtp.client-ip=131.188.11.21 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="VdwI+qCs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748985433; bh=6kulVUqZ3Sz9/D6ZyDu4KR/IeYIBjf/9FGGPysGsUN4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=VdwI+qCs6yEe6Vrm1ttgl5XymmYa5QI87StdbvDilgCpTNBB/TA45s/SSBmLNq73t gFpUMfu6iKx8tqs8QhLXvUwvs4Qdx3SjvSo0IVjc+kbnsn6trt2XTLoFG0L5+ZRf4S Qv22AS9UHue8YaSDin/TZSRbxk/kfGmWDBek+1W41qtxN7tA/+C9nZ1pquEiqpoSL3 bcBgLRV/hDSJYtTmBcLSnV464/0yVYMk/CWVCzoelJ8hhBDSpFGMc5NHUGuRYQuD/u +EZPigcjdzl1vJ1fvifGiYyn0Om8KF9d+X8LGbz1dEXGb9j3WHDOd0E6zjQcBcqjoW fjmZ2qSMBDADw== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-2.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBk7F4HDVzPk7d; Tue, 3 Jun 2025 23:17:13 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck1.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX1/crT903BhwJVO52aTy1iLoysW5QidJY9E=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBk791VHJzPk63; Tue, 3 Jun 2025 23:17:09 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 5/9] bpf, arm64, powerpc: Change nospec to include v1 barrier Date: Tue, 3 Jun 2025 23:17:03 +0200 Message-ID: <20250603211703.337860-1-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This changes the semantics of BPF_NOSPEC (previously a v4-only barrier) to always emit a speculation barrier that works against both Spectre v1 AND v4. If mitigation is not needed on an architecture, the backend should set bpf_jit_bypass_spec_v4/v1(). As of now, this commit only has the user-visible implication that unpriv BPF's performance on PowerPC is reduced. This is the case because we have to emit additional v1 barrier instructions for BPF_NOSPEC now. This commit is required for a future commit to allow us to rely on BPF_NOSPEC for Spectre v1 mitigation. As of this commit, the feature that nospec acts as a v1 barrier is unused. Commit f5e81d111750 ("bpf: Introduce BPF nospec instruction for mitigating Spectre v4") noted that mitigation instructions for v1 and v4 might be different on some archs. While this would potentially offer improved performance on PowerPC, it was dismissed after the following considerations: * Only having one barrier simplifies the verifier and allows us to easily rely on v4-induced barriers for reducing the complexity of v1-induced speculative path verification. * For the architectures that implemented BPF_NOSPEC, only PowerPC has distinct instructions for v1 and v4. Even there, some insns may be shared between the barriers for v1 and v4 (e.g., 'ori 31,31,0' and 'sync'). If this is still found to impact performance in an unacceptable way, BPF_NOSPEC can be split into BPF_NOSPEC_V1 and BPF_NOSPEC_V4 later. As an optimization, we can already skip v1/v4 insns from being emitted for PowerPC with this setup if bypass_spec_v1/v4 is set. Vulnerability-status for BPF_NOSPEC-based Spectre mitigations (v4 as of this commit, v1 in the future) is therefore: * x86 (32-bit and 64-bit), ARM64, and PowerPC (64-bit): Mitigated - This patch implements BPF_NOSPEC for these architectures. The previous v4-only version was supported since commit f5e81d111750 ("bpf: Introduce BPF nospec instruction for mitigating Spectre v4") and commit b7540d625094 ("powerpc/bpf: Emit stf barrier instruction sequences for BPF_NOSPEC"). * LoongArch: Not Vulnerable - Commit a6f6a95f2580 ("LoongArch, bpf: Fix jit to skip speculation barrier opcode") is the only other past commit related to BPF_NOSPEC and indicates that the insn is not required there. * MIPS: Vulnerable (if unprivileged BPF is enabled) - Commit=C2=A0a6f6a95f2580 ("LoongArch, bpf: Fix jit to skip speculation barrier opcode") indicates that it is not vulnerable, but this contradicts the kernel and Debian documentation. Therefore, I assume that there exist vulnerable MIPS CPUs (but maybe not from Loongson?). In the future, BPF_NOSPEC could be implemented for MIPS based on the GCC speculation_barrier [1]. For now, we rely on unprivileged BPF being disabled by default. * Other: Unknown - To the best of my knowledge there is no definitive information available that indicates that any other arch is vulnerable. They are therefore left untouched (BPF_NOSPEC is not implemented, but bypass_spec_v1/v4 is also not set). I did the following testing to ensure the insn encoding is correct: * ARM64: * 'dsb nsh; isb' was successfully tested with the BPF CI in [2] * 'sb' locally using QEMU v7.2.15 -cpu max (emitted sb insn is executed for example with './test_progs -t verifier_array_access') * PowerPC: The following configs were tested locally with ppc64le QEMU v8.2 '-machine pseries -cpu POWER9': * STF_BARRIER_EIEIO + CONFIG_PPC_BOOK32_64 * STF_BARRIER_SYNC_ORI (forced on) + CONFIG_PPC_BOOK32_64 * STF_BARRIER_FALLBACK (forced on) + CONFIG_PPC_BOOK32_64 * CONFIG_PPC_E500 (forced on) + STF_BARRIER_EIEIO * CONFIG_PPC_E500 (forced on) + STF_BARRIER_SYNC_ORI (forced on) * CONFIG_PPC_E500 (forced on) + STF_BARRIER_FALLBACK (forced on) * CONFIG_PPC_E500 (forced on) + STF_BARRIER_NONE (forced on) Most of those cobinations should not occur in practice, but I was not able to get an PPC e6500 rootfs (for testing PPC_E500 without forcing it on). In any case, this should ensure that there are no unexpected conflicts between the insns when combined like this. Individual v1/v4 barriers were already emitted elsewhere. Hari's ack is for the PowerPC changes only. [1] https://gcc.gnu.org/git/?p=3Dgcc.git;a=3Dcommit;h=3D29b74545531f6afbee9= fc38c267524326dbfbedf ("MIPS: Add speculation_barrier support") [2] https://github.com/kernel-patches/bpf/pull/8576 Signed-off-by: Luis Gerhorst Acked-by: Hari Bathini Cc: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan Acked-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit.h | 5 +++ arch/arm64/net/bpf_jit_comp.c | 9 +++-- arch/powerpc/net/bpf_jit_comp64.c | 59 ++++++++++++++++++++++--------- include/linux/filter.h | 2 +- kernel/bpf/core.c | 17 ++++----- 5 files changed, 65 insertions(+), 27 deletions(-) diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index a3b0e693a125..bbea4f36f9f2 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -325,4 +325,9 @@ #define A64_MRS_SP_EL0(Rt) \ aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_SP_EL0) =20 +/* Barriers */ +#define A64_SB aarch64_insn_get_sb_value() +#define A64_DSB_NSH (aarch64_insn_get_dsb_base_value() | 0x7 << 8) +#define A64_ISB aarch64_insn_get_isb_value() + #endif /* _BPF_JIT_H */ diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 2cab9063f563..b6c42b5c9668 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1630,9 +1630,14 @@ static int build_insn(const struct bpf_insn *insn, s= truct jit_ctx *ctx, return ret; break; =20 - /* speculation barrier */ + /* speculation barrier against v1 and v4 */ case BPF_ST | BPF_NOSPEC: - /* See bpf_jit_bypass_spec_v4() */ + if (alternative_has_cap_likely(ARM64_HAS_SB)) { + emit(A64_SB, ctx); + } else { + emit(A64_DSB_NSH, ctx); + emit(A64_ISB, ctx); + } break; =20 /* ST: *(size *)(dst + off) =3D imm */ diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_c= omp64.c index a4335761b7f9..3665ff8bb4bc 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -414,6 +414,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,= u32 *fimage, struct code u32 *addrs, int pass, bool extra_pass) { enum stf_barrier_type stf_barrier =3D stf_barrier_type_get(); + bool sync_emitted, ori31_emitted; const struct bpf_insn *insn =3D fp->insnsi; int flen =3D fp->len; int i, ret; @@ -806,26 +807,52 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *imag= e, u32 *fimage, struct code =20 /* * BPF_ST NOSPEC (speculation barrier) + * + * The following must act as a barrier against both Spectre v1 + * and v4 if we requested both mitigations. Therefore, also emit + * 'isync; sync' on E500 or 'ori31' on BOOK3S_64 in addition to + * the insns needed for a Spectre v4 barrier. + * + * If we requested only !bypass_spec_v1 OR only !bypass_spec_v4, + * we can skip the respective other barrier type as an + * optimization. */ case BPF_ST | BPF_NOSPEC: - switch (stf_barrier) { - case STF_BARRIER_EIEIO: - EMIT(PPC_RAW_EIEIO() | 0x02000000); - break; - case STF_BARRIER_SYNC_ORI: + sync_emitted =3D false; + ori31_emitted =3D false; +#ifdef CONFIG_PPC_E500 + if (!bpf_jit_bypass_spec_v1()) { + EMIT(PPC_RAW_ISYNC()); EMIT(PPC_RAW_SYNC()); - EMIT(PPC_RAW_LD(tmp1_reg, _R13, 0)); - EMIT(PPC_RAW_ORI(_R31, _R31, 0)); - break; - case STF_BARRIER_FALLBACK: - ctx->seen |=3D SEEN_FUNC; - PPC_LI64(_R12, dereference_kernel_function_descriptor(bpf_stf_barrier)= ); - EMIT(PPC_RAW_MTCTR(_R12)); - EMIT(PPC_RAW_BCTRL()); - break; - case STF_BARRIER_NONE: - break; + sync_emitted =3D true; + } +#endif + if (!bpf_jit_bypass_spec_v4()) { + switch (stf_barrier) { + case STF_BARRIER_EIEIO: + EMIT(PPC_RAW_EIEIO() | 0x02000000); + break; + case STF_BARRIER_SYNC_ORI: + if (!sync_emitted) + EMIT(PPC_RAW_SYNC()); + EMIT(PPC_RAW_LD(tmp1_reg, _R13, 0)); + EMIT(PPC_RAW_ORI(_R31, _R31, 0)); + ori31_emitted =3D true; + break; + case STF_BARRIER_FALLBACK: + ctx->seen |=3D SEEN_FUNC; + PPC_LI64(_R12, dereference_kernel_function_descriptor(bpf_stf_barrier= )); + EMIT(PPC_RAW_MTCTR(_R12)); + EMIT(PPC_RAW_BCTRL()); + break; + case STF_BARRIER_NONE: + break; + } } +#ifdef CONFIG_PPC_BOOK3S_64 + if (!bpf_jit_bypass_spec_v1() && !ori31_emitted) + EMIT(PPC_RAW_ORI(_R31, _R31, 0)); +#endif break; =20 /* diff --git a/include/linux/filter.h b/include/linux/filter.h index f5cf4d35d83e..eca229752cbe 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -82,7 +82,7 @@ struct ctl_table_header; #define BPF_CALL_ARGS 0xe0 =20 /* unused opcode to mark speculation barrier for mitigating - * Speculative Store Bypass + * Spectre v1 and v4 */ #define BPF_NOSPEC 0xc0 =20 diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index f9bd9625438b..e536a34a32c8 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2102,14 +2102,15 @@ static u64 ___bpf_prog_run(u64 *regs, const struct = bpf_insn *insn) #undef COND_JMP /* ST, STX and LDX*/ ST_NOSPEC: - /* Speculation barrier for mitigating Speculative Store Bypass. - * In case of arm64, we rely on the firmware mitigation as - * controlled via the ssbd kernel parameter. Whenever the - * mitigation is enabled, it works for all of the kernel code - * with no need to provide any additional instructions here. - * In case of x86, we use 'lfence' insn for mitigation. We - * reuse preexisting logic from Spectre v1 mitigation that - * happens to produce the required code on x86 for v4 as well. + /* Speculation barrier for mitigating Speculative Store Bypass, + * Bounds-Check Bypass and Type Confusion. In case of arm64, we + * rely on the firmware mitigation as controlled via the ssbd + * kernel parameter. Whenever the mitigation is enabled, it + * works for all of the kernel code with no need to provide any + * additional instructions here. In case of x86, we use 'lfence' + * insn for mitigation. We reuse preexisting logic from Spectre + * v1 mitigation that happens to produce the required code on + * x86 for v4 as well. */ barrier_nospec(); CONT; --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-1.rrze.uni-erlangen.de (mx-rz-1.rrze.uni-erlangen.de [131.188.11.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E4F354918; Tue, 3 Jun 2025 21:20:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985649; cv=none; b=g0BuiC9K4tsB6P4xVkvwhnXdnnuTJ0tcUWI+iyACXimOMd+jx6AdwyVJsmULRKumD8oHWwX8pedk5nhdS9CLKbBEBW/3Yc2kqZB06imvpYHGJdpLOEounf2dPoZDIMsjmGPoRfTBIIHQz7/QhMpWL8UdBPfrgQfjspdKv7xrJDo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985649; c=relaxed/simple; bh=GCgJpdGC8x1aplCDloOk8f68QuQClfLMbfW8gs00A04=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jpNt9+qV8bVkA5RRcDG9Ge07KlmKYrwkz8WqrXTFh7PlfY/U9c5+PUyvcGS2XAc5wdVXAajX/V4z558GKJHsNKEdKuuxjKB41N6JSIiBfZ0AADpq46w3+47Fh2N7xnokdcXo/axTs5CdbjDcLTJQamzZ3V6/m9Gy6vqVoWKepq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=ZxTobW4P; arc=none smtp.client-ip=131.188.11.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="ZxTobW4P" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748985645; bh=aLkGL0PRMc+wzevKsgum6EPqlwzuAjn0TCVg62AcMaU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=ZxTobW4PE3pM/tpTf8BNM3K8XxSmHxvkMXDoj4HnsgnDr/ZjCdBfbWjm5kXjFrfpb pLw0T6rthXXulNltdG3KRYrPicTsTJ0igkLqasWFt9iYyCW6awL2K77vBdO2+Y9ZoS Aw5uSovDn/6lqObO171M5TPSNe3fSQmeOvlsOvFw4zAwkXhWR4MTLTf07cpeEdUIDT m7uRwKk2rZRbOW0I6Rveb5kpvyD/VfnSyw74lWWYT09lFN+7gmTp9W+B1lFx3yZrAx BgfJR4w9xf6Lfxz2LMvcnAfycFPncdvY+yZLXtLw5kiEuQMw6QXUk3sB2bDxQc+id/ Yqt1Q0K+GUW8Q== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-1.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBkCK0kxtz8sqC; Tue, 3 Jun 2025 23:20:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck2.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX1+uyXrtNenjCYPjNclS8sO6XydtJohY3+E=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBkCF2WYTz8ssj; Tue, 3 Jun 2025 23:20:41 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 6/9] bpf: Rename sanitize_stack_spill to nospec_result Date: Tue, 3 Jun 2025 23:20:24 +0200 Message-ID: <20250603212024.338154-1-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is made to clarify that this flag will cause a nospec to be added after this insn and can therefore be relied upon to reduce speculative path analysis. Signed-off-by: Luis Gerhorst Acked-by: Kumar Kartikeya Dwivedi Cc: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan --- include/linux/bpf_verifier.h | 2 +- kernel/bpf/verifier.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 256274acb1d8..2b0954202226 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -580,7 +580,7 @@ struct bpf_insn_aux_data { u64 map_key_state; /* constant (32 bit) key tracking for maps */ int ctx_field_size; /* the ctx field size for load insn, maybe 0 */ u32 seen; /* this insn was processed by the verifier at env->pass_cnt */ - bool sanitize_stack_spill; /* subject to Spectre v4 sanitation */ + bool nospec_result; /* result is unsafe under speculation, nospec must fo= llow */ bool zext_dst; /* this insn zero extends dst reg */ bool needs_zext; /* alu op needs to clear upper bits */ bool storage_get_func_atomic; /* bpf_*_storage_get() with atomic memory a= lloc */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 46cf737acad5..af79f4d7692f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5026,7 +5026,7 @@ static int check_stack_write_fixed_off(struct bpf_ver= ifier_env *env, } =20 if (sanitize) - env->insn_aux_data[insn_idx].sanitize_stack_spill =3D true; + env->insn_aux_data[insn_idx].nospec_result =3D true; } =20 err =3D destroy_if_dynptr_stack_slot(env, state, spi); @@ -20921,7 +20921,7 @@ static int convert_ctx_accesses(struct bpf_verifier= _env *env) } =20 if (type =3D=3D BPF_WRITE && - env->insn_aux_data[i + delta].sanitize_stack_spill) { + env->insn_aux_data[i + delta].nospec_result) { struct bpf_insn patch[] =3D { *insn, BPF_ST_NOSPEC(), --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-3.rrze.uni-erlangen.de (mx-rz-3.rrze.uni-erlangen.de [131.188.11.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC3E6224AF2; Tue, 3 Jun 2025 21:24:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.22 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985887; cv=none; b=a3UoK3mE+hAgpf1EODBq3BuxmCWqhhV4GvsNojogqEvS1QX0YwY6s4YPg83xUe/+nNfBVZsfqS8ZcXAxfMZgEFEYiYQ1jo4dRPcFKziZ1toq6ItPHw3AC2BYQfZySm4fBwgQfL0S1Bua2gsNDrFg1eTyYzWgeCK+E0tAhg3KHyE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748985887; c=relaxed/simple; bh=Dltz9/trmK4HCj8x+kuG5yKXPCmm5sMbgIkEK1FXzzA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=C77zzhZd4+Q80MJr8FxIniPMP171aHuh+pBee7etzjJaxotA4ybg8gNSAgDqKcTHjPtx3c+ca+2G4yd4abUNHnOfA8XvR1ygoSo+3pCu9jRjcbd4liyaKNOijAZOdQ0Ik0oqtH45hEnh89NuOt1xZFSwiNSKAzMCeoz08xK3Pm8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=qc1L3Oof; arc=none smtp.client-ip=131.188.11.22 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="qc1L3Oof" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748985882; bh=9QR+Gw2YFKvfCN+9UfqoBJ54/vfk1Mg1SLVULYFWZjA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=qc1L3OofYUOXuP7YQ9PWxRdivFcn0+NmyEhCLOVz1krUOmyMTlM8CsoZs8RQM56e2 ZZp0kQSHZYJb8a2XNORHGs/0N4mvOVmu9EdZyWpnww4uk5OgNJQG/5uUiyAeGwwMS1 JD4gGhVQdBSrayMFyqBokxPxB4C+ocZuP1fwiye5qLFQ0iVv+6ElI/oBuvYGXEGKyP ++suEmBtIKNViWSITgfmk8i7fgOR1tHNSc3piZDXNPO1Aju7estUI0nwvceyGNAy7P bnD1bh5Wka6yH2/DWwuvO80ugjWxzw7H8h7tXpfeyVxltqn6h3cd3jro8P3xFSMUrs PyFLkfmTORC9w== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-3.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBkHt22knz1yVn; Tue, 3 Jun 2025 23:24:42 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck2.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX19zwYIHXMomVOBTfgNfbMKU9qN40jpbJJ8=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBkHm4ZB9z1yLd; Tue, 3 Jun 2025 23:24:36 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Dustin Nguyen , Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 7/9] bpf: Fall back to nospec for Spectre v1 Date: Tue, 3 Jun 2025 23:24:28 +0200 Message-ID: <20250603212428.338473-1-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This implements the core of the series and causes the verifier to fall back to mitigating Spectre v1 using speculation barriers. The approach was presented at LPC'24 [1] and RAID'24 [2]. If we find any forbidden behavior on a speculative path, we insert a nospec (e.g., lfence speculation barrier on x86) before the instruction and stop verifying the path. While verifying a speculative path, we can furthermore stop verification of that path whenever we encounter a nospec instruction. A minimal example program would look as follows: A =3D true B =3D true if A goto e f() if B goto e unsafe() e: exit There are the following speculative and non-speculative paths (`cur->speculative` and `speculative` referring to the value of the push_stack() parameters): - A =3D true - B =3D true - if A goto e - A && !cur->speculative && !speculative - exit - !A && !cur->speculative && speculative - f() - if B goto e - B && cur->speculative && !speculative - exit - !B && cur->speculative && speculative - unsafe() If f() contains any unsafe behavior under Spectre v1 and the unsafe behavior matches `state->speculative && error_recoverable_with_nospec(err)`, do_check() will now add a nospec before f() instead of rejecting the program: A =3D true B =3D true if A goto e nospec f() if B goto e unsafe() e: exit Alternatively, the algorithm also takes advantage of nospec instructions inserted for other reasons (e.g., Spectre v4). Taking the program above as an example, speculative path exploration can stop before f() if a nospec was inserted there because of Spectre v4 sanitization. In this example, all instructions after the nospec are dead code (and with the nospec they are also dead code speculatively). For this, it relies on the fact that speculation barriers generally prevent all later instructions from executing if the speculation was not correct: * On Intel x86_64, lfence acts as full speculation barrier, not only as a load fence [3]: An LFENCE instruction or a serializing instruction will ensure that no later instructions execute, even speculatively, until all prior instructions complete locally. [...] Inserting an LFENCE instruction after a bounds check prevents later operations from executing before the bound check completes. This was experimentally confirmed in [4]. * On AMD x86_64, lfence is dispatch-serializing [5] (requires MSR C001_1029[1] to be set if the MSR is supported, this happens in init_amd()). AMD further specifies "A dispatch serializing instruction forces the processor to retire the serializing instruction and all previous instructions before the next instruction is executed" [8]. As dispatch is not specific to memory loads or branches, lfence therefore also affects all instructions there. Also, if retiring a branch means it's PC change becomes architectural (should be), this means any "wrong" speculation is aborted as required for this series. * ARM's SB speculation barrier instruction also affects "any instruction that appears later in the program order than the barrier" [6]. * PowerPC's barrier also affects all subsequent instructions [7]: [...] executing an ori R31,R31,0 instruction ensures that all instructions preceding the ori R31,R31,0 instruction have completed before the ori R31,R31,0 instruction completes, and that no subsequent instructions are initiated, even out-of-order, until after the ori R31,R31,0 instruction completes. The ori R31,R31,0 instruction may complete before storage accesses associated with instructions preceding the ori R31,R31,0 instruction have been performed Regarding the example, this implies that `if B goto e` will not execute before `if A goto e` completes. Once `if A goto e` completes, the CPU should find that the speculation was wrong and continue with `exit`. If there is any other path that leads to `if B goto e` (and therefore `unsafe()`) without going through `if A goto e`, then a nospec will still be needed there. However, this patch assumes this other path will be explored separately and therefore be discovered by the verifier even if the exploration discussed here stops at the nospec. This patch furthermore has the unfortunate consequence that Spectre v1 mitigations now only support architectures which implement BPF_NOSPEC. Before this commit, Spectre v1 mitigations prevented exploits by rejecting the programs on all architectures. Because some JITs do not implement BPF_NOSPEC, this patch therefore may regress unpriv BPF's security to a limited extent: * The regression is limited to systems vulnerable to Spectre v1, have unprivileged BPF enabled, and do NOT emit insns for BPF_NOSPEC. The latter is not the case for x86 64- and 32-bit, arm64, and powerpc 64-bit and they are therefore not affected by the regression. According to commit a6f6a95f2580 ("LoongArch, bpf: Fix jit to skip speculation barrier opcode"), LoongArch is not vulnerable to Spectre v1 and therefore also not affected by the regression. * To the best of my knowledge this regression may therefore only affect MIPS. This is deemed acceptable because unpriv BPF is still disabled there by default. As stated in a previous commit, BPF_NOSPEC could be implemented for MIPS based on GCC's speculation_barrier implementation. * It is unclear which other architectures (besides x86 64- and 32-bit, ARM64, PowerPC 64-bit, LoongArch, and MIPS) supported by the kernel are vulnerable to Spectre v1. Also, it is not clear if barriers are available on these architectures. Implementing BPF_NOSPEC on these architectures therefore is non-trivial. Searching GCC and the kernel for speculation barrier implementations for these architectures yielded no result. * If any of those regressed systems is also vulnerable to Spectre v4, the system was already vulnerable to Spectre v4 attacks based on unpriv BPF before this patch and the impact is therefore further limited. As an alternative to regressing security, one could still reject programs if the architecture does not emit BPF_NOSPEC (e.g., by removing the empty BPF_NOSPEC-case from all JITs except for LoongArch where it appears justified). However, this will cause rejections on these archs that are likely unfounded in the vast majority of cases. In the tests, some are now successful where we previously had a false-positive (i.e., rejection). Change them to reflect where the nospec should be inserted (using __xlated_unpriv) and modify the error message if the nospec is able to mitigate a problem that previously shadowed another problem (in that case __xlated_unpriv does not work, therefore just add a comment). Define SPEC_V1 to avoid duplicating this ifdef whenever we check for nospec insns using __xlated_unpriv, define it here once. This also improves readability. PowerPC can probably also be added here. However, omit it for now because the BPF CI currently does not include a test. Limit it to EPERM, EACCES, and EINVAL (and not everything except for EFAULT and ENOMEM) as it already has the desired effect for most real-world programs. Briefly went through all the occurrences of EPERM, EINVAL, and EACCESS in verifier.c to validate that catching them like this makes sense. Thanks to Dustin for their help in checking the vendor documentation. [1] https://lpc.events/event/18/contributions/1954/ ("Mitigating Spectre-PHT using Speculation Barriers in Linux eBPF") [2] https://arxiv.org/pdf/2405.00078 ("VeriFence: Lightweight and Precise Spectre Defenses for Untrusted Linux Kernel Extensions") [3] https://www.intel.com/content/www/us/en/developer/articles/technical/so= ftware-security-guidance/technical-documentation/runtime-speculative-side-c= hannel-mitigations.html ("Managed Runtime Speculative Execution Side Channel Mitigations") [4] https://dl.acm.org/doi/pdf/10.1145/3359789.3359837 ("Speculator: a tool to analyze speculative execution attacks and mitigations" - Section 4.6 "Stopping Speculative Execution") [5] https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/pr= ogrammer-references/software-techniques-for-managing-speculation.pdf ("White Paper - SOFTWARE TECHNIQUES FOR MANAGING SPECULATION ON AMD PROCESSORS - REVISION 5.09.23") [6] https://developer.arm.com/documentation/ddi0597/2020-12/Base-Instructio= ns/SB--Speculation-Barrier- ("SB - Speculation Barrier - Arm Armv8-A A32/T32 Instruction Set Architecture (2020-12)") [7] https://wiki.raptorcs.com/w/images/5/5f/OPF_PowerISA_v3.1C.pdf ("Power ISA=E2=84=A2 - Version 3.1C - May 26, 2024 - Section 9.2.1 of B= ook III") [8] https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/pr= ogrammer-references/40332.pdf ("AMD64 Architecture Programmer=E2=80=99s Manual Volumes 1=E2=80=935 - = Revision 4.08 - April 2024 - 7.6.4 Serializing Instructions") Signed-off-by: Luis Gerhorst Acked-by: Kumar Kartikeya Dwivedi Acked-by: Henriette Herzog Cc: Dustin Nguyen Cc: Maximilian Ott Cc: Milan Stephan --- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 78 ++++++++++++++++++- tools/testing/selftests/bpf/progs/bpf_misc.h | 4 + .../selftests/bpf/progs/verifier_and.c | 8 +- .../selftests/bpf/progs/verifier_bounds.c | 61 ++++++++++++--- .../selftests/bpf/progs/verifier_movsx.c | 16 +++- .../selftests/bpf/progs/verifier_unpriv.c | 8 +- .../bpf/progs/verifier_value_ptr_arith.c | 16 +++- .../selftests/bpf/verifier/dead_code.c | 3 +- tools/testing/selftests/bpf/verifier/jmp32.c | 33 +++----- tools/testing/selftests/bpf/verifier/jset.c | 10 +-- 11 files changed, 184 insertions(+), 54 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 2b0954202226..e6c26393c029 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -580,6 +580,7 @@ struct bpf_insn_aux_data { u64 map_key_state; /* constant (32 bit) key tracking for maps */ int ctx_field_size; /* the ctx field size for load insn, maybe 0 */ u32 seen; /* this insn was processed by the verifier at env->pass_cnt */ + bool nospec; /* do not execute this instruction speculatively */ bool nospec_result; /* result is unsafe under speculation, nospec must fo= llow */ bool zext_dst; /* this insn zero extends dst reg */ bool needs_zext; /* alu op needs to clear upper bits */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index af79f4d7692f..2b887989f521 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2012,6 +2012,18 @@ static int pop_stack(struct bpf_verifier_env *env, i= nt *prev_insn_idx, return 0; } =20 +static bool error_recoverable_with_nospec(int err) +{ + /* Should only return true for non-fatal errors that are allowed to + * occur during speculative verification. For these we can insert a + * nospec and the program might still be accepted. Do not include + * something like ENOMEM because it is likely to re-occur for the next + * architectural path once it has been recovered-from in all speculative + * paths. + */ + return err =3D=3D -EPERM || err =3D=3D -EACCES || err =3D=3D -EINVAL; +} + static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx, bool speculative) @@ -11146,7 +11158,7 @@ static int check_get_func_ip(struct bpf_verifier_en= v *env) return -ENOTSUPP; } =20 -static struct bpf_insn_aux_data *cur_aux(struct bpf_verifier_env *env) +static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *en= v) { return &env->insn_aux_data[env->insn_idx]; } @@ -14014,7 +14026,9 @@ static int retrieve_ptr_limit(const struct bpf_reg_= state *ptr_reg, static bool can_skip_alu_sanitation(const struct bpf_verifier_env *env, const struct bpf_insn *insn) { - return env->bypass_spec_v1 || BPF_SRC(insn->code) =3D=3D BPF_K; + return env->bypass_spec_v1 || + BPF_SRC(insn->code) =3D=3D BPF_K || + cur_aux(env)->nospec; } =20 static int update_alu_sanitation_state(struct bpf_insn_aux_data *aux, @@ -19723,10 +19737,41 @@ static int do_check(struct bpf_verifier_env *env) sanitize_mark_insn_seen(env); prev_insn_idx =3D env->insn_idx; =20 + /* Reduce verification complexity by stopping speculative path + * verification when a nospec is encountered. + */ + if (state->speculative && cur_aux(env)->nospec) + goto process_bpf_exit; + err =3D do_check_insn(env, &do_print_state); - if (err < 0) { + if (state->speculative && error_recoverable_with_nospec(err)) { + /* Prevent this speculative path from ever reaching the + * insn that would have been unsafe to execute. + */ + cur_aux(env)->nospec =3D true; + /* If it was an ADD/SUB insn, potentially remove any + * markings for alu sanitization. + */ + cur_aux(env)->alu_state =3D 0; + goto process_bpf_exit; + } else if (err < 0) { return err; } else if (err =3D=3D PROCESS_BPF_EXIT) { + goto process_bpf_exit; + } + WARN_ON_ONCE(err); + + if (state->speculative && cur_aux(env)->nospec_result) { + /* If we are on a path that performed a jump-op, this + * may skip a nospec patched-in after the jump. This can + * currently never happen because nospec_result is only + * used for the write-ops + * `*(size*)(dst_reg+off)=3Dsrc_reg|imm32` which must + * never skip the following insn. Still, add a warning + * to document this in case nospec_result is used + * elsewhere in the future. + */ + WARN_ON_ONCE(env->insn_idx !=3D prev_insn_idx + 1); process_bpf_exit: mark_verifier_state_scratched(env); update_branch_counts(env, env->cur_state); @@ -19744,7 +19789,6 @@ static int do_check(struct bpf_verifier_env *env) continue; } } - WARN_ON_ONCE(err); } =20 return 0; @@ -20872,6 +20916,29 @@ static int convert_ctx_accesses(struct bpf_verifie= r_env *env) bpf_convert_ctx_access_t convert_ctx_access; u8 mode; =20 + if (env->insn_aux_data[i + delta].nospec) { + WARN_ON_ONCE(env->insn_aux_data[i + delta].alu_state); + struct bpf_insn patch[] =3D { + BPF_ST_NOSPEC(), + *insn, + }; + + cnt =3D ARRAY_SIZE(patch); + new_prog =3D bpf_patch_insn_data(env, i + delta, patch, cnt); + if (!new_prog) + return -ENOMEM; + + delta +=3D cnt - 1; + env->prog =3D new_prog; + insn =3D new_prog->insnsi + i + delta; + /* This can not be easily merged with the + * nospec_result-case, because an insn may require a + * nospec before and after itself. Therefore also do not + * 'continue' here but potentially apply further + * patching to insn. *insn should equal patch[1] now. + */ + } + if (insn->code =3D=3D (BPF_LDX | BPF_MEM | BPF_B) || insn->code =3D=3D (BPF_LDX | BPF_MEM | BPF_H) || insn->code =3D=3D (BPF_LDX | BPF_MEM | BPF_W) || @@ -20922,6 +20989,9 @@ static int convert_ctx_accesses(struct bpf_verifier= _env *env) =20 if (type =3D=3D BPF_WRITE && env->insn_aux_data[i + delta].nospec_result) { + /* nospec_result is only used to mitigate Spectre v4 and + * to limit verification-time for Spectre v1. + */ struct bpf_insn patch[] =3D { *insn, BPF_ST_NOSPEC(), diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/s= elftests/bpf/progs/bpf_misc.h index 6e208e24ba3b..a678463e972c 100644 --- a/tools/testing/selftests/bpf/progs/bpf_misc.h +++ b/tools/testing/selftests/bpf/progs/bpf_misc.h @@ -231,4 +231,8 @@ #define CAN_USE_LOAD_ACQ_STORE_REL #endif =20 +#if defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) +#define SPEC_V1 +#endif + #endif diff --git a/tools/testing/selftests/bpf/progs/verifier_and.c b/tools/testi= ng/selftests/bpf/progs/verifier_and.c index e97e518516b6..2b4fdca162be 100644 --- a/tools/testing/selftests/bpf/progs/verifier_and.c +++ b/tools/testing/selftests/bpf/progs/verifier_and.c @@ -85,8 +85,14 @@ l0_%=3D: r0 =3D r0; \ =20 SEC("socket") __description("check known subreg with unknown reg") -__success __failure_unpriv __msg_unpriv("R1 !read_ok") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if w0 < 0x1 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R1 !read_ok'` */ +__xlated_unpriv("goto pc-1") /* `r1 =3D *(u32*)(r1 + 512)`, sanitized dead= code */ +__xlated_unpriv("r0 =3D 0") +#endif __naked void known_subreg_with_unknown_reg(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/te= sting/selftests/bpf/progs/verifier_bounds.c index 0eb33bb801b5..30e16153fdf1 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bounds.c +++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c @@ -620,8 +620,14 @@ l1_%=3D: exit; \ =20 SEC("socket") __description("bounds check mixed 32bit and 64bit arithmetic. test1") -__success __failure_unpriv __msg_unpriv("R0 invalid mem access 'scalar'") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 invalid mem access 's= calar'` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("exit") +#endif __naked void _32bit_and_64bit_arithmetic_test1(void) { asm volatile (" \ @@ -643,8 +649,14 @@ l1_%=3D: exit; \ =20 SEC("socket") __description("bounds check mixed 32bit and 64bit arithmetic. test2") -__success __failure_unpriv __msg_unpriv("R0 invalid mem access 'scalar'") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 invalid mem access 's= calar'` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("exit") +#endif __naked void _32bit_and_64bit_arithmetic_test2(void) { asm volatile (" \ @@ -691,9 +703,14 @@ l0_%=3D: r0 =3D 0; \ =20 SEC("socket") __description("bounds check for reg =3D 0, reg xor 1") -__success __failure_unpriv -__msg_unpriv("R0 min value is outside of the allowed memory range") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if r1 !=3D 0x0 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside = of the allowed memory range` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("r0 =3D 0") +#endif __naked void reg_0_reg_xor_1(void) { asm volatile (" \ @@ -719,9 +736,14 @@ l1_%=3D: r0 =3D 0; \ =20 SEC("socket") __description("bounds check for reg32 =3D 0, reg32 xor 1") -__success __failure_unpriv -__msg_unpriv("R0 min value is outside of the allowed memory range") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if w1 !=3D 0x0 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside = of the allowed memory range` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("r0 =3D 0") +#endif __naked void reg32_0_reg32_xor_1(void) { asm volatile (" \ @@ -747,9 +769,14 @@ l1_%=3D: r0 =3D 0; \ =20 SEC("socket") __description("bounds check for reg =3D 2, reg xor 3") -__success __failure_unpriv -__msg_unpriv("R0 min value is outside of the allowed memory range") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if r1 > 0x0 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside = of the allowed memory range` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("r0 =3D 0") +#endif __naked void reg_2_reg_xor_3(void) { asm volatile (" \ @@ -829,9 +856,14 @@ l1_%=3D: r0 =3D 0; \ =20 SEC("socket") __description("bounds check for reg > 0, reg xor 3") -__success __failure_unpriv -__msg_unpriv("R0 min value is outside of the allowed memory range") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if r1 >=3D 0x0 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside = of the allowed memory range` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("r0 =3D 0") +#endif __naked void reg_0_reg_xor_3(void) { asm volatile (" \ @@ -858,9 +890,14 @@ l1_%=3D: r0 =3D 0; \ =20 SEC("socket") __description("bounds check for reg32 > 0, reg32 xor 3") -__success __failure_unpriv -__msg_unpriv("R0 min value is outside of the allowed memory range") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if w1 >=3D 0x0 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R0 min value is outside = of the allowed memory range` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("r0 =3D 0") +#endif __naked void reg32_0_reg32_xor_3(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/progs/verifier_movsx.c b/tools/tes= ting/selftests/bpf/progs/verifier_movsx.c index 994bbc346d25..a4d8814eb5ed 100644 --- a/tools/testing/selftests/bpf/progs/verifier_movsx.c +++ b/tools/testing/selftests/bpf/progs/verifier_movsx.c @@ -245,7 +245,13 @@ l0_%=3D: \ SEC("socket") __description("MOV32SX, S8, var_off not u32_max, positive after s8 extensi= on") __success __retval(0) -__failure_unpriv __msg_unpriv("frame pointer is read only") +__success_unpriv +#ifdef SPEC_V1 +__xlated_unpriv("w0 =3D 0") +__xlated_unpriv("exit") +__xlated_unpriv("nospec") /* inserted to prevent `frame pointer is read on= ly` */ +__xlated_unpriv("goto pc-1") +#endif __naked void mov64sx_s32_varoff_2(void) { asm volatile (" \ @@ -267,7 +273,13 @@ l0_%=3D: \ SEC("socket") __description("MOV32SX, S8, var_off not u32_max, negative after s8 extensi= on") __success __retval(0) -__failure_unpriv __msg_unpriv("frame pointer is read only") +__success_unpriv +#ifdef SPEC_V1 +__xlated_unpriv("w0 =3D 0") +__xlated_unpriv("exit") +__xlated_unpriv("nospec") /* inserted to prevent `frame pointer is read on= ly` */ +__xlated_unpriv("goto pc-1") +#endif __naked void mov64sx_s32_varoff_3(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/te= sting/selftests/bpf/progs/verifier_unpriv.c index a4a5e2071604..c42c3839b30c 100644 --- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c +++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c @@ -572,8 +572,14 @@ l0_%=3D: exit; \ =20 SEC("socket") __description("alu32: mov u32 const") -__success __failure_unpriv __msg_unpriv("R7 invalid mem access 'scalar'") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if r0 =3D=3D 0x0 goto pc+2") +__xlated_unpriv("nospec") /* inserted to prevent `R7 invalid mem access 's= calar'` */ +__xlated_unpriv("goto pc-1") /* sanitized dead code */ +__xlated_unpriv("exit") +#endif __naked void alu32_mov_u32_const(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b= /tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c index 5ba6e53571c8..fcea9819e359 100644 --- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c +++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c @@ -398,7 +398,8 @@ l2_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: mixing value pointer and scalar, 1") -__success __failure_unpriv __msg_unpriv("R2 pointer comparison prohibited") +__success __failure_unpriv +__msg_unpriv("R2 tried to add from different maps, paths or scalars, point= er arithmetic with it prohibited for !root") __retval(0) __naked void value_pointer_and_scalar_1(void) { @@ -433,6 +434,7 @@ l2_%=3D: /* common instruction */ \ l3_%=3D: /* branch B */ \ r0 =3D 0x13371337; \ /* verifier follows fall-through */ \ + /* unpriv: nospec (inserted to prevent `R2 pointer comparison prohibited`= ) */\ if r2 !=3D 0x100000 goto l4_%=3D; \ r0 =3D 0; \ exit; \ @@ -450,7 +452,8 @@ l4_%=3D: /* fake-dead code; targeted from branch A to \ =20 SEC("socket") __description("map access: mixing value pointer and scalar, 2") -__success __failure_unpriv __msg_unpriv("R0 invalid mem access 'scalar'") +__success __failure_unpriv +__msg_unpriv("R2 tried to add from different maps, paths or scalars, point= er arithmetic with it prohibited for !root") __retval(0) __naked void value_pointer_and_scalar_2(void) { @@ -492,6 +495,7 @@ l4_%=3D: /* fake-dead code; targeted from branch A to \ * prevent dead code sanitization, rejected \ * via branch B however \ */ \ + /* unpriv: nospec (inserted to prevent `R0 invalid mem access 'scalar'`) = */\ r0 =3D *(u8*)(r0 + 0); \ r0 =3D 0; \ exit; \ @@ -1296,9 +1300,13 @@ l0_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: value_ptr -=3D unknown scalar, 2") -__success __failure_unpriv -__msg_unpriv("R0 pointer arithmetic of map value goes out of range") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("r1 &=3D 7") +__xlated_unpriv("nospec") /* inserted to prevent `R0 pointer arithmetic of= map value goes out of range` */ +__xlated_unpriv("r0 -=3D r1") +#endif __naked void value_ptr_unknown_scalar_2_2(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/verifier/dead_code.c b/tools/testi= ng/selftests/bpf/verifier/dead_code.c index ee454327e5c6..77207b498c6f 100644 --- a/tools/testing/selftests/bpf/verifier/dead_code.c +++ b/tools/testing/selftests/bpf/verifier/dead_code.c @@ -2,14 +2,13 @@ "dead code: start", .insns =3D { BPF_JMP_IMM(BPF_JA, 0, 0, 2), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_JMP_IMM(BPF_JA, 0, 0, 2), BPF_MOV64_IMM(BPF_REG_0, 7), BPF_JMP_IMM(BPF_JGE, BPF_REG_0, 10, -4), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 7, }, diff --git a/tools/testing/selftests/bpf/verifier/jmp32.c b/tools/testing/s= elftests/bpf/verifier/jmp32.c index 43776f6f92f4..91d83e9cb148 100644 --- a/tools/testing/selftests/bpf/verifier/jmp32.c +++ b/tools/testing/selftests/bpf/verifier/jmp32.c @@ -84,11 +84,10 @@ BPF_JMP32_IMM(BPF_JSET, BPF_REG_7, 0x10, 1), BPF_EXIT_INSN(), BPF_JMP32_IMM(BPF_JGE, BPF_REG_7, 0x10, 1), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .result =3D ACCEPT, }, { @@ -149,11 +148,10 @@ BPF_JMP32_IMM(BPF_JEQ, BPF_REG_7, 0x10, 1), BPF_EXIT_INSN(), BPF_JMP32_IMM(BPF_JSGE, BPF_REG_7, 0xf, 1), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .result =3D ACCEPT, }, { @@ -214,11 +212,10 @@ BPF_JMP32_IMM(BPF_JNE, BPF_REG_7, 0x10, 1), BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 0x10, 1), BPF_EXIT_INSN(), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .result =3D ACCEPT, }, { @@ -283,11 +280,10 @@ BPF_JMP32_REG(BPF_JGE, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP32_IMM(BPF_JGE, BPF_REG_7, 0x7ffffff0, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -354,11 +350,10 @@ BPF_JMP32_REG(BPF_JGT, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JGT, BPF_REG_7, 0x7ffffff0, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -425,11 +420,10 @@ BPF_JMP32_REG(BPF_JLE, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP32_IMM(BPF_JLE, BPF_REG_7, 0x7ffffff0, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -496,11 +490,10 @@ BPF_JMP32_REG(BPF_JLT, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0x7ffffff0, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -567,11 +560,10 @@ BPF_JMP32_REG(BPF_JSGE, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JSGE, BPF_REG_7, 0x7ffffff0, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -638,11 +630,10 @@ BPF_JMP32_REG(BPF_JSGT, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JSGT, BPF_REG_7, -2, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -709,11 +700,10 @@ BPF_JMP32_REG(BPF_JSLE, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JSLE, BPF_REG_7, 0x7ffffff0, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, @@ -780,11 +770,10 @@ BPF_JMP32_REG(BPF_JSLT, BPF_REG_7, BPF_REG_8, 1), BPF_EXIT_INSN(), BPF_JMP32_IMM(BPF_JSLT, BPF_REG_7, -1, 1), + /* unpriv: nospec (inserted to prevent "R0 invalid mem access 'scalar'") = */ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), BPF_EXIT_INSN(), }, - .errstr_unpriv =3D "R0 invalid mem access 'scalar'", - .result_unpriv =3D REJECT, .result =3D ACCEPT, .retval =3D 2, .flags =3D F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, diff --git a/tools/testing/selftests/bpf/verifier/jset.c b/tools/testing/se= lftests/bpf/verifier/jset.c index 11fc68da735e..e901eefd774a 100644 --- a/tools/testing/selftests/bpf/verifier/jset.c +++ b/tools/testing/selftests/bpf/verifier/jset.c @@ -78,12 +78,11 @@ .insns =3D { BPF_MOV64_IMM(BPF_REG_0, 1), BPF_JMP_IMM(BPF_JSET, BPF_REG_0, 1, 1), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_EXIT_INSN(), }, .prog_type =3D BPF_PROG_TYPE_SOCKET_FILTER, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .retval =3D 1, .result =3D ACCEPT, }, @@ -136,13 +135,12 @@ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_prandom_u32), BPF_ALU64_IMM(BPF_OR, BPF_REG_0, 2), BPF_JMP_IMM(BPF_JSET, BPF_REG_0, 3, 1), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_MOV64_IMM(BPF_REG_0, 0), BPF_EXIT_INSN(), }, .prog_type =3D BPF_PROG_TYPE_SOCKET_FILTER, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .result =3D ACCEPT, }, { @@ -154,16 +152,16 @@ BPF_ALU64_IMM(BPF_AND, BPF_REG_1, 0xff), BPF_JMP_IMM(BPF_JSET, BPF_REG_1, 0xf0, 3), BPF_JMP_IMM(BPF_JLT, BPF_REG_1, 0x10, 1), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JSET, BPF_REG_1, 0x10, 1), BPF_EXIT_INSN(), BPF_JMP_IMM(BPF_JGE, BPF_REG_1, 0x10, 1), + /* unpriv: nospec (inserted to prevent "R9 !read_ok") */ BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0), BPF_EXIT_INSN(), }, .prog_type =3D BPF_PROG_TYPE_SOCKET_FILTER, - .errstr_unpriv =3D "R9 !read_ok", - .result_unpriv =3D REJECT, .result =3D ACCEPT, }, --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-2.rrze.uni-erlangen.de (mx-rz-2.rrze.uni-erlangen.de [131.188.11.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EBB722579E; Tue, 3 Jun 2025 21:28:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.21 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748986111; cv=none; b=UkxS8CTdCjPpYJCz4X3Dgf5WBn6kydciJHm6j9CCsoxBWe/RAgIlEi6Iecu2aFEFEEk2HXfVj1UvBY/XXijQwbvt+hnDRTS+rKbxabQjJT+51agtdYXR5dXFm60sgFMWPKymSacXTc0KB83N2jR73WuyyBs2vnttFUuBr8cmFcQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748986111; c=relaxed/simple; bh=91X80LvGHfH90x/qxrnpS8bkmAvt6UUlA5rRQzLZg/I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fAGDMLJYjxnaHK1y80D5bTUufSvegHgv5Td4LNrFP708rglaNu1xDwJBy9xGsRbmiTSZoLBF/WHJGRD7/RgYk3yzeyUFqmN3txRVSrHAqqzSEkik+vRUFSKBWCwDMs02WtXgGuCGhOJB1TcevLWwR50FnB4uDOmPvIoHE7RQq8I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=R3pPMiSf; arc=none smtp.client-ip=131.188.11.21 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="R3pPMiSf" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748986107; bh=ARpytHsyfivjxGjflz99PobPaQ/Mtr9Wv+8PNmUUgKc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=R3pPMiSfvWBr9q+phUTH/uIaiJTRngnHfV38FDKrEu0rWW+NROPwI97cQI7vvjF9b QlyUwWpcjrB479J15KVsC/l25vUlnmyiCTosRvJ2hJOtB7Ds9aJObDxXllxWOhaPh8 YZre8UFpOFSwEEB9f0BZtu3V6yc+qQt878BdQqa2kxcqpCbIfrOkEDgXcvJd0cBB47 QfCCl0tiF4jeF8x+TUt+UUU8xvyOgWNbL+Xc3T65ZjnzZWH5jnWlhreysMVud+aA/U vKr1XiZC/4YJUdN/IdteEihDFKFaeCRI48HciCVZ9uM7rWMkpiKMlLhVpI1pBrIE3j OAviOnKmghToA== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-2.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBkNB6ZfkzPkZc; Tue, 3 Jun 2025 23:28:26 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck4.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX18910nUSGoVfudKBySRew8N0fTZeQak0HM=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBkN73X7dzPkPR; Tue, 3 Jun 2025 23:28:23 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi Subject: [PATCH bpf-next v4 8/9] selftests/bpf: Add test for Spectre v1 mitigation Date: Tue, 3 Jun 2025 23:28:14 +0200 Message-ID: <20250603212814.338867-1-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This is based on the gadget from the description of commit=C2=A09183671af6db ("bpf: Fix leakage under speculation on mispredicted branches"). Signed-off-by: Luis Gerhorst Acked-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/verifier_unpriv.c | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/te= sting/selftests/bpf/progs/verifier_unpriv.c index c42c3839b30c..43236b93ebb5 100644 --- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c +++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c @@ -729,4 +729,61 @@ l0_%=3D: r0 =3D 0; \ " ::: __clobber_all); } =20 +SEC("socket") +__description("unpriv: Spectre v1 path-based type confusion of scalar as s= tack-ptr") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("if r0 !=3D 0x1 goto pc+2") +/* This nospec prevents the exploit because it forces the mispredicted (not + * taken) `if r0 !=3D 0x0 goto l0_%=3D` to resolve before using r6 as a po= inter. + * This causes the CPU to realize that `r6 =3D r9` should have never execu= ted. It + * ensures that r6 always contains a readable stack slot ptr when the insn= after + * the nospec executes. + */ +__xlated_unpriv("nospec") +__xlated_unpriv("r9 =3D *(u8 *)(r6 +0)") +#endif +__naked void unpriv_spec_v1_type_confusion(void) +{ + asm volatile (" \ + r1 =3D 0; \ + *(u64*)(r10 - 8) =3D r1; \ + r2 =3D r10; \ + r2 +=3D -8; \ + r1 =3D %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 =3D=3D 0 goto l2_%=3D; \ + /* r0: pointer to a map array entry */ \ + r2 =3D r10; \ + r2 +=3D -8; \ + r1 =3D %[map_hash_8b] ll; \ + /* r1, r2: prepared call args */ \ + r6 =3D r10; \ + r6 +=3D -8; \ + /* r6: pointer to readable stack slot */ \ + r9 =3D 0xffffc900; \ + r9 <<=3D 32; \ + /* r9: scalar controlled by attacker */ \ + r0 =3D *(u64 *)(r0 + 0); /* cache miss */ \ + if r0 !=3D 0x0 goto l0_%=3D; \ + r6 =3D r9; \ +l0_%=3D: if r0 !=3D 0x1 goto l1_%=3D; \ + r9 =3D *(u8 *)(r6 + 0); \ +l1_%=3D: /* leak r9 */ \ + r9 &=3D 1; \ + r9 <<=3D 9; \ + *(u64*)(r10 - 8) =3D r9; \ + call %[bpf_map_lookup_elem]; \ + if r0 =3D=3D 0 goto l2_%=3D; \ + /* leak secret into is_cached(map[0|512]): */ \ + r0 =3D *(u64 *)(r0 + 0); \ +l2_%=3D: \ + r0 =3D 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + char _license[] SEC("license") =3D "GPL"; --=20 2.49.0 From nobody Wed Dec 17 22:45:43 2025 Received: from mx-rz-2.rrze.uni-erlangen.de (mx-rz-2.rrze.uni-erlangen.de [131.188.11.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BD0D223DFD; Tue, 3 Jun 2025 21:32:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=131.188.11.21 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748986367; cv=none; b=QQjyx4u8LE1Vu+ZxesJeANkSWacf9pnEi/v8L7xgvjmrT9HlpfR0cllkfcKvlryDc2xx+BOpsMsL6lRrAvnPQT1PKgZ+FtPcQihH8bm+JFku6wm960aAqjiLyTd7Iflsj4BUO9Dhb5ekVXUxMkK2sQICtOByNP6dOY7RFetctnk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748986367; c=relaxed/simple; bh=HeHmYbHaH59/oVsDL/E/aLhF1NoGDRqsPc9/VD+MHuk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K5hgUzmYpY+ttqQ4VkqvzeL5Nb65vkDQ5YfQG6FR+w5ARHcYtrx8UHiZish/laxp+OWhLw8H0MIlcDk2mJJt1R7aAYVd6DKrMeXPhX+7W1ARsF4mtFOaEOV3EYXAoOOKtyWvrmZv0j3vxHchqav0ZmB6YFDyPRCWTXSDKPY0X2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de; spf=pass smtp.mailfrom=fau.de; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b=EMFRlFeQ; arc=none smtp.client-ip=131.188.11.21 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fau.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fau.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fau.de header.i=@fau.de header.b="EMFRlFeQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fau.de; s=fau-2021; t=1748986361; bh=r0ROO6JQw8k4B1xCpzchdZ8NB8JXx7aNds4odXlAKyg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From:To:CC: Subject; b=EMFRlFeQdI9Emx7TLFKbg3oFiaFIZU7LOXY8tE4UtHQITvp2twd4qT9iL1zrbp38a Ip7kZcOtKwQrKTkC1hrLwPbwDHh3RzMCYR/j+LSKUVU+xACq7B77NIe99XTl8dt8wT ZKTUdx4y+4gkTYqMaXBZHoQbys5TQcBA8sYkbXLqG3H/EAEG9E9h5U+Rz7zEXTZ+kS Xe1GLpspBT75FME6dTJdi2LWcvilVYX1tbVYMNxao0e2IB5+t4bysLkVFDMo1QS2Hh 25eiaj3/yz9nVpiHPZUSi+NdppQV1E3vw7L7UoeJLTH64dCfPk47Bf/1Im5tzDljpx pzRs+361SH/Ew== Received: from mx-rz-smart.rrze.uni-erlangen.de (mx-rz-smart.rrze.uni-erlangen.de [IPv6:2001:638:a000:1025::1e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-rz-2.rrze.uni-erlangen.de (Postfix) with ESMTPS id 4bBkT54FsyzPk0P; Tue, 3 Jun 2025 23:32:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at boeck5.rrze.uni-erlangen.de (RRZE) X-RRZE-Flag: Not-Spam X-RRZE-Submit-IP: 2001:9e8:3639:fe00:a21f:4ce4:8495:5578 Received: from luis-tp.fritz.box (unknown [IPv6:2001:9e8:3639:fe00:a21f:4ce4:8495:5578]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: U2FsdGVkX19I3k1Uhpbt0kY6hKxDfPeI5WWla0ELut8=) by smtp-auth.uni-erlangen.de (Postfix) with ESMTPSA id 4bBkT02bNTzPkZc; Tue, 3 Jun 2025 23:32:36 +0200 (CEST) From: Luis Gerhorst To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Hari Bathini , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , Mykola Lysenko , Shuah Khan , Luis Gerhorst , Henriette Herzog , Saket Kumar Bhaskar , Cupertino Miranda , Jiayuan Chen , Matan Shachnai , Dimitar Kanaliev , Shung-Hsi Yu , Daniel Xu , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Maximilian Ott , Milan Stephan Subject: [PATCH bpf-next v4 9/9] bpf: Fall back to nospec for sanitization-failures Date: Tue, 3 Jun 2025 23:32:32 +0200 Message-ID: <20250603213232.339242-1-luis.gerhorst@fau.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250603205800.334980-1-luis.gerhorst@fau.de> References: <20250603205800.334980-1-luis.gerhorst@fau.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ALU sanitization was introduced to ensure that a subsequent ptr access can never go OOB, even under speculation. This is required because we currently allow speculative scalar confusion. Spec. scalar confusion is possible because Spectre v4 sanitization only adds a nospec after critical stores (e.g., scalar overwritten with a pointer). If we add a nospec before the ALU op, none of the operands can be subject to scalar confusion. As an ADD/SUB can not introduce scalar confusion itself, the result will also not be subject to scalar confusion. Therefore, the subsequent ptr access is always safe. We directly fall back to nospec for the sanitization errors REASON_BOUNDS, _TYPE, _PATHS, and _LIMIT, even if we are not on a speculative path. For REASON_STACK, we return the error -ENOMEM directly now. Previously, sanitize_err() returned -EACCES for this case but we change it to -ENOMEM because doing so prevents do_check() from falling back to a nospec if we are on a speculative path. This would not be a serious issue (the verifier would probably run into the -ENOMEM again shortly on the next non-speculative path and still abort verification), but -ENOMEM is more fitting here anyway. An alternative would be -EFAULT, which is also returned for some of the other cases where push_stack() fails, but this is more frequently used for verifier-internal bugs. Signed-off-by: Luis Gerhorst Acked-by: Kumar Kartikeya Dwivedi Acked-by: Henriette Herzog Cc: Maximilian Ott Cc: Milan Stephan --- kernel/bpf/verifier.c | 89 ++++++----------- .../selftests/bpf/progs/verifier_bounds.c | 5 +- .../bpf/progs/verifier_bounds_deduction.c | 45 ++++++--- .../selftests/bpf/progs/verifier_map_ptr.c | 20 +++- .../bpf/progs/verifier_value_ptr_arith.c | 97 ++++++++++++++++--- 5 files changed, 160 insertions(+), 96 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2b887989f521..62c0fe3dfdd4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13984,14 +13984,6 @@ static bool check_reg_sane_offset(struct bpf_verif= ier_env *env, return true; } =20 -enum { - REASON_BOUNDS =3D -1, - REASON_TYPE =3D -2, - REASON_PATHS =3D -3, - REASON_LIMIT =3D -4, - REASON_STACK =3D -5, -}; - static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg, u32 *alu_limit, bool mask_to_left) { @@ -14014,11 +14006,13 @@ static int retrieve_ptr_limit(const struct bpf_re= g_state *ptr_reg, ptr_reg->umax_value) + ptr_reg->off; break; default: - return REASON_TYPE; + /* Register has pointer with unsupported alu operation. */ + return -EOPNOTSUPP; } =20 + /* Register tried access beyond pointer bounds. */ if (ptr_limit >=3D max) - return REASON_LIMIT; + return -EOPNOTSUPP; *alu_limit =3D ptr_limit; return 0; } @@ -14039,8 +14033,14 @@ static int update_alu_sanitation_state(struct bpf_= insn_aux_data *aux, */ if (aux->alu_state && (aux->alu_state !=3D alu_state || - aux->alu_limit !=3D alu_limit)) - return REASON_PATHS; + aux->alu_limit !=3D alu_limit)) { + /* Tried to perform alu op from different maps, paths or + * scalars. + */ + aux->nospec =3D true; + aux->alu_state =3D 0; + return 0; + } =20 /* Corresponding fixup done in do_misc_fixups(). */ aux->alu_state =3D alu_state; @@ -14121,16 +14121,26 @@ static int sanitize_ptr_alu(struct bpf_verifier_e= nv *env, =20 if (!commit_window) { if (!tnum_is_const(off_reg->var_off) && - (off_reg->smin_value < 0) !=3D (off_reg->smax_value < 0)) - return REASON_BOUNDS; + (off_reg->smin_value < 0) !=3D (off_reg->smax_value < 0)) { + /* Register has unknown scalar with mixed signed + * bounds. + */ + aux->nospec =3D true; + aux->alu_state =3D 0; + return 0; + } =20 info->mask_to_left =3D (opcode =3D=3D BPF_ADD && off_is_neg) || (opcode =3D=3D BPF_SUB && !off_is_neg); } =20 err =3D retrieve_ptr_limit(ptr_reg, &alu_limit, info->mask_to_left); - if (err < 0) - return err; + if (err) { + WARN_ON_ONCE(err !=3D -EOPNOTSUPP); + aux->nospec =3D true; + aux->alu_state =3D 0; + return 0; + } =20 if (commit_window) { /* In commit phase we narrow the masking window based on @@ -14183,7 +14193,7 @@ static int sanitize_ptr_alu(struct bpf_verifier_env= *env, env->insn_idx); if (!ptr_is_dst_reg && ret) *dst_reg =3D tmp; - return !ret ? REASON_STACK : 0; + return !ret ? -ENOMEM : 0; } =20 static void sanitize_mark_insn_seen(struct bpf_verifier_env *env) @@ -14199,45 +14209,6 @@ static void sanitize_mark_insn_seen(struct bpf_ver= ifier_env *env) env->insn_aux_data[env->insn_idx].seen =3D env->pass_cnt; } =20 -static int sanitize_err(struct bpf_verifier_env *env, - const struct bpf_insn *insn, int reason, - const struct bpf_reg_state *off_reg, - const struct bpf_reg_state *dst_reg) -{ - static const char *err =3D "pointer arithmetic with it prohibited for !ro= ot"; - const char *op =3D BPF_OP(insn->code) =3D=3D BPF_ADD ? "add" : "sub"; - u32 dst =3D insn->dst_reg, src =3D insn->src_reg; - - switch (reason) { - case REASON_BOUNDS: - verbose(env, "R%d has unknown scalar with mixed signed bounds, %s\n", - off_reg =3D=3D dst_reg ? dst : src, err); - break; - case REASON_TYPE: - verbose(env, "R%d has pointer with unsupported alu operation, %s\n", - off_reg =3D=3D dst_reg ? src : dst, err); - break; - case REASON_PATHS: - verbose(env, "R%d tried to %s from different maps, paths or scalars, %s\= n", - dst, op, err); - break; - case REASON_LIMIT: - verbose(env, "R%d tried to %s beyond pointer bounds, %s\n", - dst, op, err); - break; - case REASON_STACK: - verbose(env, "R%d could not be pushed for speculative verification, %s\n= ", - dst, err); - break; - default: - verbose(env, "verifier internal error: unknown reason (%d)\n", - reason); - break; - } - - return -EACCES; -} - /* check that stack access falls within stack limits and that 'reg' doesn't * have a variable offset. * @@ -14403,7 +14374,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verif= ier_env *env, ret =3D sanitize_ptr_alu(env, insn, ptr_reg, off_reg, dst_reg, &info, false); if (ret < 0) - return sanitize_err(env, insn, ret, off_reg, dst_reg); + return ret; } =20 switch (opcode) { @@ -14531,7 +14502,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verif= ier_env *env, ret =3D sanitize_ptr_alu(env, insn, dst_reg, off_reg, dst_reg, &info, true); if (ret < 0) - return sanitize_err(env, insn, ret, off_reg, dst_reg); + return ret; } =20 return 0; @@ -15125,7 +15096,7 @@ static int adjust_scalar_min_max_vals(struct bpf_ve= rifier_env *env, if (sanitize_needed(opcode)) { ret =3D sanitize_val_alu(env, insn); if (ret < 0) - return sanitize_err(env, insn, ret, NULL, NULL); + return ret; } =20 /* Calculate sign/unsigned bounds and tnum for alu32 and alu64 bit ops. diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/te= sting/selftests/bpf/progs/verifier_bounds.c index 30e16153fdf1..f2ee6d7febda 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bounds.c +++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c @@ -47,9 +47,12 @@ SEC("socket") __description("subtraction bounds (map value) variant 2") __failure __msg("R0 min value is negative, either use unsigned index or do a if (ind= ex >=3D0) check.") -__msg_unpriv("R1 has unknown scalar with mixed signed bounds") +__msg_unpriv("R0 pointer arithmetic of map value goes out of range, prohib= ited for !root") __naked void bounds_map_value_variant_2(void) { + /* unpriv: nospec inserted to prevent "R1 has unknown scalar with mixed + * signed bounds". + */ asm volatile (" \ r1 =3D 0; \ *(u64*)(r10 - 8) =3D r1; \ diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c = b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c index c506afbdd936..24ecaf89004e 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c +++ b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c @@ -8,22 +8,26 @@ SEC("socket") __description("check deducing bounds from const, 1") __failure __msg("R0 tried to subtract pointer from scalar") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __naked void deducing_bounds_from_const_1(void) { asm volatile (" \ r0 =3D 1; \ if r0 s>=3D 1 goto l0_%=3D; \ -l0_%=3D: r0 -=3D r1; \ +l0_%=3D: /* unpriv: nospec (inserted to prevent `R1 has pointer with unsup= ported alu operation`) */\ + r0 -=3D r1; \ exit; \ " ::: __clobber_all); } =20 SEC("socket") __description("check deducing bounds from const, 2") -__success __failure_unpriv -__msg_unpriv("R1 has pointer with unsupported alu operation") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("nospec") /* inserted to prevent `R1 has pointer with unsu= pported alu operation` */ +__xlated_unpriv("r1 -=3D r0") +#endif __naked void deducing_bounds_from_const_2(void) { asm volatile (" \ @@ -40,22 +44,26 @@ l1_%=3D: r1 -=3D r0; \ SEC("socket") __description("check deducing bounds from const, 3") __failure __msg("R0 tried to subtract pointer from scalar") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __naked void deducing_bounds_from_const_3(void) { asm volatile (" \ r0 =3D 0; \ if r0 s<=3D 0 goto l0_%=3D; \ -l0_%=3D: r0 -=3D r1; \ +l0_%=3D: /* unpriv: nospec (inserted to prevent `R1 has pointer with unsup= ported alu operation`) */\ + r0 -=3D r1; \ exit; \ " ::: __clobber_all); } =20 SEC("socket") __description("check deducing bounds from const, 4") -__success __failure_unpriv -__msg_unpriv("R6 has pointer with unsupported alu operation") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("nospec") /* inserted to prevent `R6 has pointer with unsu= pported alu operation` */ +__xlated_unpriv("r6 -=3D r0") +#endif __naked void deducing_bounds_from_const_4(void) { asm volatile (" \ @@ -73,12 +81,13 @@ l1_%=3D: r6 -=3D r0; \ SEC("socket") __description("check deducing bounds from const, 5") __failure __msg("R0 tried to subtract pointer from scalar") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __naked void deducing_bounds_from_const_5(void) { asm volatile (" \ r0 =3D 0; \ if r0 s>=3D 1 goto l0_%=3D; \ + /* unpriv: nospec (inserted to prevent `R1 has pointer with unsupported a= lu operation`) */\ r0 -=3D r1; \ l0_%=3D: exit; \ " ::: __clobber_all); @@ -87,14 +96,15 @@ l0_%=3D: exit; \ SEC("socket") __description("check deducing bounds from const, 6") __failure __msg("R0 tried to subtract pointer from scalar") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __naked void deducing_bounds_from_const_6(void) { asm volatile (" \ r0 =3D 0; \ if r0 s>=3D 0 goto l0_%=3D; \ exit; \ -l0_%=3D: r0 -=3D r1; \ +l0_%=3D: /* unpriv: nospec (inserted to prevent `R1 has pointer with unsup= ported alu operation`) */\ + r0 -=3D r1; \ exit; \ " ::: __clobber_all); } @@ -102,14 +112,15 @@ l0_%=3D: r0 -=3D r1; \ SEC("socket") __description("check deducing bounds from const, 7") __failure __msg("dereference of modified ctx ptr") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __flag(BPF_F_ANY_ALIGNMENT) __naked void deducing_bounds_from_const_7(void) { asm volatile (" \ r0 =3D %[__imm_0]; \ if r0 s>=3D 0 goto l0_%=3D; \ -l0_%=3D: r1 -=3D r0; \ +l0_%=3D: /* unpriv: nospec (inserted to prevent `R1 has pointer with unsup= ported alu operation`) */\ + r1 -=3D r0; \ r0 =3D *(u32*)(r1 + %[__sk_buff_mark]); \ exit; \ " : @@ -121,13 +132,14 @@ l0_%=3D: r1 -=3D r0; \ SEC("socket") __description("check deducing bounds from const, 8") __failure __msg("negative offset ctx ptr R1 off=3D-1 disallowed") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __flag(BPF_F_ANY_ALIGNMENT) __naked void deducing_bounds_from_const_8(void) { asm volatile (" \ r0 =3D %[__imm_0]; \ if r0 s>=3D 0 goto l0_%=3D; \ + /* unpriv: nospec (inserted to prevent `R1 has pointer with unsupported a= lu operation`) */\ r1 +=3D r0; \ l0_%=3D: r0 =3D *(u32*)(r1 + %[__sk_buff_mark]); \ exit; \ @@ -140,13 +152,14 @@ l0_%=3D: r0 =3D *(u32*)(r1 + %[__sk_buff_mark]); \ SEC("socket") __description("check deducing bounds from const, 9") __failure __msg("R0 tried to subtract pointer from scalar") -__msg_unpriv("R1 has pointer with unsupported alu operation") +__failure_unpriv __naked void deducing_bounds_from_const_9(void) { asm volatile (" \ r0 =3D 0; \ if r0 s>=3D 0 goto l0_%=3D; \ -l0_%=3D: r0 -=3D r1; \ +l0_%=3D: /* unpriv: nospec (inserted to prevent `R1 has pointer with unsup= ported alu operation`) */\ + r0 -=3D r1; \ exit; \ " ::: __clobber_all); } diff --git a/tools/testing/selftests/bpf/progs/verifier_map_ptr.c b/tools/t= esting/selftests/bpf/progs/verifier_map_ptr.c index 11a079145966..a540f29b7532 100644 --- a/tools/testing/selftests/bpf/progs/verifier_map_ptr.c +++ b/tools/testing/selftests/bpf/progs/verifier_map_ptr.c @@ -110,9 +110,17 @@ __naked void ptr_read_ops_field_accepted(void) =20 SEC("socket") __description("bpf_map_ptr: r =3D 0, map_ptr =3D map_ptr + r") -__success __failure_unpriv -__msg_unpriv("R1 has pointer with unsupported alu operation") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("r1 =3D 0x") /* r0 =3D %[map_hash_16b] ll */ +/* This nospec is not really needed here, because there is no bypassable + * branch/store that could lead to r0 not containing 0 when `r1 +=3D r0` e= xecutes. + * We still keep it like this to not complicate the verifier. + */ +__xlated_unpriv("nospec") /* inserted to prevent `R1 has pointer with unsu= pported alu operation` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void map_ptr_map_ptr_r(void) { asm volatile (" \ @@ -134,9 +142,13 @@ __naked void map_ptr_map_ptr_r(void) =20 SEC("socket") __description("bpf_map_ptr: r =3D 0, r =3D r + map_ptr") -__success __failure_unpriv -__msg_unpriv("R0 has pointer with unsupported alu operation") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("r0 =3D 0x") /* r0 =3D %[map_hash_16b] ll */ +__xlated_unpriv("nospec") /* inserted to prevent `R0 has pointer with unsu= pported alu operation` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void _0_r_r_map_ptr(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b= /tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c index fcea9819e359..a35090e3de2b 100644 --- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c +++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c @@ -41,9 +41,17 @@ struct { =20 SEC("socket") __description("map access: known scalar +=3D value_ptr unknown vs const") -__success __failure_unpriv -__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("r1 &=3D 7") +__xlated_unpriv("goto pc+1") +/* l3_%=3D: */ +__xlated_unpriv("r1 =3D 3") +/* l4_%=3D: */ +__xlated_unpriv("nospec") /* inserted to prevent `R1 tried to add from dif= ferent maps, paths or scalars` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void value_ptr_unknown_vs_const(void) { asm volatile (" \ @@ -79,9 +87,14 @@ l2_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: known scalar +=3D value_ptr const vs unknown") -__success __failure_unpriv -__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("r1 &=3D 7") +/* l4_%=3D: */ +__xlated_unpriv("nospec") /* inserted to prevent `R1 tried to add from dif= ferent maps, paths or scalars` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void value_ptr_const_vs_unknown(void) { asm volatile (" \ @@ -117,9 +130,16 @@ l2_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: known scalar +=3D value_ptr const vs const (ne)= ") -__success __failure_unpriv -__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("goto pc+1") /* to l4, must not be pc+2 as this would skip= nospec */ +/* l3_%=3D: */ +__xlated_unpriv("r1 =3D 5") +/* l4_%=3D: */ +__xlated_unpriv("nospec") /* inserted to prevent `R1 tried to add from dif= ferent maps, paths or scalars` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void ptr_const_vs_const_ne(void) { asm volatile (" \ @@ -225,9 +245,18 @@ l2_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: known scalar +=3D value_ptr unknown vs unknown = (lt)") -__success __failure_unpriv -__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("r1 &=3D 3") +__xlated_unpriv("goto pc+3") /* must go to l4 (nospec) */ +__xlated_unpriv("r1 =3D 6") +__xlated_unpriv("r1 =3D -r1") +__xlated_unpriv("r1 &=3D 7") +/* l4_%=3D: */ +__xlated_unpriv("nospec") /* inserted to prevent `R1 tried to add from dif= ferent maps, paths or scalars` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void ptr_unknown_vs_unknown_lt(void) { asm volatile (" \ @@ -265,9 +294,14 @@ l2_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: known scalar +=3D value_ptr unknown vs unknown = (gt)") -__success __failure_unpriv -__msg_unpriv("R1 tried to add from different maps, paths or scalars") +__success __success_unpriv __retval(1) +#ifdef SPEC_V1 +__xlated_unpriv("r1 &=3D 3") +/* l4_%=3D: */ +__xlated_unpriv("nospec") /* inserted to prevent `R1 tried to add from dif= ferent maps, paths or scalars` */ +__xlated_unpriv("r1 +=3D r0") +#endif __naked void ptr_unknown_vs_unknown_gt(void) { asm volatile (" \ @@ -398,9 +432,27 @@ l2_%=3D: r0 =3D 1; \ =20 SEC("socket") __description("map access: mixing value pointer and scalar, 1") -__success __failure_unpriv -__msg_unpriv("R2 tried to add from different maps, paths or scalars, point= er arithmetic with it prohibited for !root") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("goto pc+2") /* to l2, must not be pc+3 as this would skip= nospec */ +__xlated_unpriv("r2 =3D 0") +__xlated_unpriv("r3 =3D 1048576") +/* l2_%=3D: */ +__xlated_unpriv("nospec") /* inserted to prevent `R2 tried to add from dif= ferent maps, paths or scalars, pointer arithmetic with it prohibited for !r= oot` */ +__xlated_unpriv("r2 +=3D r3") +__xlated_unpriv("if r1 !=3D 0x0 goto pc+1") +__xlated_unpriv("goto pc+5") /* to l4 */ +/* l3_%=3D: */ +__xlated_unpriv("r0 =3D 322376503") +__xlated_unpriv("nospec") /* inserted to prevent `R2 pointer comparison pr= ohibited` */ +__xlated_unpriv("if r2 !=3D 0x100000 goto pc+2") /* to l4 */ +__xlated_unpriv("r0 =3D 0") +__xlated_unpriv("exit") +/* l4_%=3D: */ +__xlated_unpriv("nospec") +__xlated_unpriv("r0 =3D *(u8 *)(r0 +0)") +#endif __naked void value_pointer_and_scalar_1(void) { asm volatile (" \ @@ -434,7 +486,6 @@ l2_%=3D: /* common instruction */ \ l3_%=3D: /* branch B */ \ r0 =3D 0x13371337; \ /* verifier follows fall-through */ \ - /* unpriv: nospec (inserted to prevent `R2 pointer comparison prohibited`= ) */\ if r2 !=3D 0x100000 goto l4_%=3D; \ r0 =3D 0; \ exit; \ @@ -452,9 +503,24 @@ l4_%=3D: /* fake-dead code; targeted from branch A to \ =20 SEC("socket") __description("map access: mixing value pointer and scalar, 2") -__success __failure_unpriv -__msg_unpriv("R2 tried to add from different maps, paths or scalars, point= er arithmetic with it prohibited for !root") +__success __success_unpriv __retval(0) +#ifdef SPEC_V1 +__xlated_unpriv("goto pc+2") +__xlated_unpriv("r2 =3D r0") +__xlated_unpriv("r3 =3D 0") +__xlated_unpriv("nospec") +__xlated_unpriv("r2 +=3D r3") +__xlated_unpriv("if r1 !=3D 0x0 goto pc+1") +__xlated_unpriv("goto pc+5") +__xlated_unpriv("r0 =3D 322376503") +__xlated_unpriv("nospec") +__xlated_unpriv("if r2 !=3D 0x100000 goto pc+2") +__xlated_unpriv("r0 =3D 0") +__xlated_unpriv("exit") +__xlated_unpriv("nospec") /* inserted to prevent `R0 invalid mem access 's= calar'` */ +__xlated_unpriv("r0 =3D *(u8 *)(r0 +0)") +#endif __naked void value_pointer_and_scalar_2(void) { asm volatile (" \ @@ -495,7 +561,6 @@ l4_%=3D: /* fake-dead code; targeted from branch A to \ * prevent dead code sanitization, rejected \ * via branch B however \ */ \ - /* unpriv: nospec (inserted to prevent `R0 invalid mem access 'scalar'`) = */\ r0 =3D *(u8*)(r0 + 0); \ r0 =3D 0; \ exit; \ --=20 2.49.0