From nobody Sat Jul 25 23:41:46 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46F2E209F43; Sat, 11 Jul 2026 05:24:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783747483; cv=none; b=DFRo0B6f6Eicx6iEFspmZ87cbotpZVyQJvS1KL5ID4fhf50wSsuRFBJe2HwE9khz5v4hRoikL824ipI2UUOaoZS2xF+/ygZNmDCjne+NagZWupl+2EFWBp54VoyLdVjJvDQDnYQ2boJnHUWx33n1RmPBTQdZCtV6jTS6fZ+dMfU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783747483; c=relaxed/simple; bh=cfv46CjdLZIW2w16XmE9uZr663U1grMRQ6XojQ6IbtA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pPtVe7SararyT0Ae0YGC1iD414Xjwua7VC1hqAPSqTUHJHj1yeaPmPsSWbmlc2toJFHz86BtdMpeiwE7BksQEBOzZzSPqIPbaz43wtp4uDoF+9ZnUezo3ZKDQkLiK91bLBU4tRxHRU28oOv2UxG5adI7vEhywJwCYVFxnO4Kl8Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.170]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4gxxvZ34VnzKHMJl; Sat, 11 Jul 2026 13:24:10 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id BF71E4056D; Sat, 11 Jul 2026 13:24:35 +0800 (CST) Received: from localhost.huawei.com (unknown [10.67.174.243]) by APP1 (Coremail) with UTF8SMTPA id cCh0CgBndnKS01FqF1tCAw--.37295S3; Sat, 11 Jul 2026 13:24:35 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Emil Tsalapatis , Yonghong Song , Anton Protopopov Subject: [PATCH bpf-next 1/2] bpf: Eliminate dup/restore of insn_aux_data Date: Sat, 11 Jul 2026 13:23:11 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: cCh0CgBndnKS01FqF1tCAw--.37295S3 X-Coremail-Antispam: 1UD129KBjvJXoW3JFWUGF15Gr1rur4kuFW7urg_yoWfWFWfpF 93Xwn7AF4kJws0gwnrAFWUArnxtr4jgw4DGFykZ3y8XF1jgrn5XFyj9ay0vasYyrW0kw1S vr4j9rW7Ww17WrJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmvb4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2IxkI6r1a6r45M2 8IrcIa0xkI8VA2jI8067AKxVWUGwA2048vs2IY020Ec7CjxVAFwI0_JFI_Gr1l8cAvFVAK 0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWUJVWUCwA2z4 x0Y4vE2Ix0cI8IcVCY1x0267AKxVWxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW8Jr0_Cr1U M28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc 02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAF wI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JM4IIrI8v6xkF7I 0E8cxan2IY04v7MxkF7I0En4kS14v26r1q6r43MxAIw28IcxkI7VAKI48JMxC20s026xCa FVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_Jr Wlx4CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j 6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr 0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUv cSsGvfC2KfnxnUUI43ZEXa7IU0l4iUUUUUU== X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai The dup/restore of insn_aux_data was introduced to resolve the inconsistency between insnsi and insn_aux_data arrays, which occurs on the failure path where insnsi was rolled back to the original state before constants blinding, while insn_aux_data was not. After JIT failure, there is only one user, bpf_clear_insn_aux_data(), that requires insnsi and insn_aux_data to be synchronized. It accesses both insnsi and insn_aux_data using the same array size and index. However, the access to insnsi in bpf_clear_insn_aux_data() is not necessary. It is checked to skip the second slot of an ldimm64 instruction, whose jt is never set and can be absorbed into the jt check itself. So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a specific length field for insn_aux_data to allow it to have a different length from the insnsi array. Then remove dup/restore of insn_aux_data. Signed-off-by: Xu Kuohai --- include/linux/bpf_verifier.h | 1 + include/linux/filter.h | 13 ------------- kernel/bpf/core.c | 16 ---------------- kernel/bpf/fixups.c | 36 ++---------------------------------- kernel/bpf/verifier.c | 4 ++-- 5 files changed, 5 insertions(+), 65 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 317e99b9acc0..c98ec9de5ba6 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -948,6 +948,7 @@ struct bpf_verifier_env { bool seen_direct_write; bool seen_exception; bool signature; + u32 insn_aux_data_len; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; diff --git a/include/linux/filter.h b/include/linux/filter.h index 14acb2455746..25148865f9c7 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1211,25 +1211,12 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_p= rog *prog, u32 off, #ifdef CONFIG_BPF_SYSCALL struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, const struct bpf_insn *patch, u32 len); -struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *e= nv); -void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, - struct bpf_insn_aux_data *orig_insn_aux); #else static inline struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env= *env, u32 off, const struct bpf_insn *patch, u32 len) { return ERR_PTR(-ENOTSUPP); } - -static inline struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_v= erifier_env *env) -{ - return NULL; -} - -static inline void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, - struct bpf_insn_aux_data *orig_insn_aux) -{ -} #endif /* CONFIG_BPF_SYSCALL */ =20 int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 47fe047ad30b..b7f17f89d992 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2622,22 +2622,10 @@ static struct bpf_prog *bpf_prog_jit_compile(struct= bpf_verifier_env *env, struc { #ifdef CONFIG_BPF_JIT struct bpf_prog *orig_prog; - struct bpf_insn_aux_data *orig_insn_aux; =20 if (!bpf_prog_need_blind(prog)) return bpf_int_jit_compile(env, prog); =20 - if (env) { - /* - * If env is not NULL, we are called from the end of bpf_check(), at this - * point, only insn_aux_data is used after failure, so it should be rest= ored - * on failure. - */ - orig_insn_aux =3D bpf_dup_insn_aux_data(env); - if (!orig_insn_aux) - return prog; - } - orig_prog =3D prog; prog =3D bpf_jit_blind_constants(env, prog); /* @@ -2650,8 +2638,6 @@ static struct bpf_prog *bpf_prog_jit_compile(struct b= pf_verifier_env *env, struc prog =3D bpf_int_jit_compile(env, prog); if (prog->jited) { bpf_jit_prog_release_other(prog, orig_prog); - if (env) - vfree(orig_insn_aux); return prog; } =20 @@ -2659,8 +2645,6 @@ static struct bpf_prog *bpf_prog_jit_compile(struct b= pf_verifier_env *env, struc =20 out_restore: prog =3D orig_prog; - if (env) - bpf_restore_insn_aux_data(env, orig_insn_aux); #endif return prog; } diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index d3be972714b2..37f22eb41854 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -174,6 +174,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_en= v *env, if (cnt =3D=3D 1) return; prog_len =3D new_prog->len; + env->insn_aux_data_len =3D prog_len; =20 memmove(data + off + cnt - 1, data + off, sizeof(struct bpf_insn_aux_data) * (prog_len - off - cnt + 1)); @@ -440,7 +441,6 @@ static int bpf_adj_linfo_after_remove(struct bpf_verifi= er_env *env, u32 off, void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int = len) { struct bpf_insn_aux_data *aux_data =3D env->insn_aux_data; - struct bpf_insn *insns =3D env->prog->insnsi; int end =3D start + len; int i; =20 @@ -449,9 +449,6 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *e= nv, int start, int len) kvfree(aux_data[i].jt); aux_data[i].jt =3D NULL; } - - if (bpf_is_ldimm64(&insns[i])) - i++; } } =20 @@ -464,7 +461,6 @@ static int verifier_remove_insns(struct bpf_verifier_en= v *env, u32 off, u32 cnt) if (bpf_prog_is_offloaded(env->prog->aux)) bpf_prog_offload_remove_insns(env, off, cnt); =20 - /* Should be called before bpf_remove_insns, as it uses prog->insnsi */ bpf_clear_insn_aux_data(env, off, cnt); =20 err =3D bpf_remove_insns(env->prog, off, cnt); @@ -483,6 +479,7 @@ static int verifier_remove_insns(struct bpf_verifier_en= v *env, u32 off, u32 cnt) =20 memmove(aux_data + off, aux_data + off + cnt, sizeof(*aux_data) * (orig_prog_len - off - cnt)); + env->insn_aux_data_len -=3D cnt; =20 return 0; } @@ -1005,26 +1002,6 @@ static void bpf_restore_subprog_starts(struct bpf_ve= rifier_env *env, u32 *orig_s env->subprog_info[env->subprog_cnt].start =3D env->prog->len; } =20 -struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *e= nv) -{ - size_t size; - void *new_aux; - - size =3D array_size(sizeof(struct bpf_insn_aux_data), env->prog->len); - new_aux =3D __vmalloc(size, GFP_KERNEL_ACCOUNT); - if (new_aux) - memcpy(new_aux, env->insn_aux_data, size); - return new_aux; -} - -void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, - struct bpf_insn_aux_data *orig_insn_aux) -{ - /* the expanded elements are zero-filled, so no special handling is requi= red */ - vfree(env->insn_aux_data); - env->insn_aux_data =3D orig_insn_aux; -} - static int jit_subprogs(struct bpf_verifier_env *env) { struct bpf_prog *prog =3D env->prog, **func, *tmp; @@ -1299,7 +1276,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) bool blinded =3D false; struct bpf_insn *insn; struct bpf_prog *prog, *orig_prog; - struct bpf_insn_aux_data *orig_insn_aux; u32 *orig_subprog_starts; =20 if (env->subprog_cnt <=3D 1) @@ -1307,14 +1283,8 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) =20 prog =3D orig_prog =3D env->prog; if (bpf_prog_need_blind(prog)) { - orig_insn_aux =3D bpf_dup_insn_aux_data(env); - if (!orig_insn_aux) { - err =3D -ENOMEM; - goto out_cleanup; - } orig_subprog_starts =3D bpf_dup_subprog_starts(env); if (!orig_subprog_starts) { - vfree(orig_insn_aux); err =3D -ENOMEM; goto out_cleanup; } @@ -1334,7 +1304,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) if (blinded) { bpf_jit_prog_release_other(prog, orig_prog); kvfree(orig_subprog_starts); - vfree(orig_insn_aux); } =20 return 0; @@ -1364,7 +1333,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) =20 out_restore: bpf_restore_subprog_starts(env, orig_subprog_starts); - bpf_restore_insn_aux_data(env, orig_insn_aux); kvfree(orig_subprog_starts); out_cleanup: /* cleanup main prog to be interpreted */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 03e2202cca13..85b05529f107 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20108,7 +20108,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_att= r *attr, bpfptr_t uattr, if (!is_priv) mutex_lock(&bpf_verifier_lock); =20 - len =3D env->prog->len; + len =3D env->insn_aux_data_len =3D env->prog->len; env->insn_aux_data =3D __vmalloc(array_size(sizeof(struct bpf_insn_aux_data), len), GFP_KERNEL_ACCOUNT | __GFP_ZERO); @@ -20354,7 +20354,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_att= r *attr, bpfptr_t uattr, release_btfs(env); err_free_env: if (env->insn_aux_data) - bpf_clear_insn_aux_data(env, 0, env->prog->len); + bpf_clear_insn_aux_data(env, 0, env->insn_aux_data_len); vfree(env->insn_aux_data); kvfree(env->fd_array); bpf_stack_liveness_free(env); --=20 2.43.0 From nobody Sat Jul 25 23:41:46 2026 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46E75495E5; Sat, 11 Jul 2026 05:24:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783747483; cv=none; b=SF/D7HpQYJI11JXJa9wpMY9WJB+qkgG6Td0w2fRKKGaZ0Bhc47uU99GlYS9SEHUAwf8NrXa6u8G3yLhi4ZpeikFS5O5IMAUGUKDKDNDebqW039er2Xu7f22oSUE2i3UCJT9cvkHIOwfBOC8ddK6FX/gMPYJJ9aAkzIN+dKrEBq0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783747483; c=relaxed/simple; bh=PmT9LorRstnkYjoQl9yybf/yBRDIaFXEmC5RSS0XpE0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y1LNR9LfI0a6z4ISk3jfvUea2hNLJ2yC6vBKonNJPpbNpxTQc5zATu1y8D4JAIiGpSHnko2UZ2IHy4psLrdpV5+ywFQcAKGvFGvU9IkD1Ei5r8a0BOOiHFLrcrDezKeWH/Xir/7nsguDH6zMlBP2X9WiQ01hbu+36VodeFEUOHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.177]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTPS id 4gxxvZ3cX2zKHMKH; Sat, 11 Jul 2026 13:24:10 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id D254040592; Sat, 11 Jul 2026 13:24:35 +0800 (CST) Received: from localhost.huawei.com (unknown [10.67.174.243]) by APP1 (Coremail) with UTF8SMTPA id cCh0CgBndnKS01FqF1tCAw--.37295S4; Sat, 11 Jul 2026 13:24:35 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Emil Tsalapatis , Yonghong Song , Anton Protopopov Subject: [PATCH bpf-next 2/2] bpf: Remove unnecessary dup/restore subprog_starts and prog clone Date: Sat, 11 Jul 2026 13:23:12 +0000 Message-ID: <329e83120f0d88c2e5dc9d7bb2c0d38988959d32.1783775928.git.xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: cCh0CgBndnKS01FqF1tCAw--.37295S4 X-Coremail-Antispam: 1UD129KBjvJXoWxuFWkZFWDXFy8WFW8ur18uFg_yoWfAFy7pF 95t34DCr4qqw40g3srJa18Ary5Ja18Ww15CrW8G34Iqa1jqrn5Wa13Kw4vqFZ3Cry8Ww1x Zr4q9r9rW3y8ZrJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmvb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2IxkI6r1a6r45M2 8IrcIa0xkI8VA2jI8067AKxVWUXwA2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK 0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWUCVW8JwA2z4 x0Y4vE2Ix0cI8IcVCY1x0267AKxVWxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW8Jr0_Cr1U M28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc 02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAF wI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JM4IIrI8v6xkF7I 0E8cxan2IY04v7MxkF7I0En4kS14v26r1q6r43MxAIw28IcxkI7VAKI48JMxC20s026xCa FVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_Jr Wlx4CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j 6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr 0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUv cSsGvfC2KfnxnUUI43ZEXa7IUnoEEUUUUUU== X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ Content-Type: text/plain; charset="utf-8" From: Xu Kuohai When JIT is required by the bpf program, the kernel rejects the program on JIT failure rather than falling back to the interpreter. In this case, the subprog_starts dup/restore and bpf prog clone are not necessary, since they are only used to restore state for the interpreter. Signed-off-by: Xu Kuohai --- include/linux/filter.h | 9 ++-- kernel/bpf/core.c | 104 +++++++++++++++++++++++------------------ kernel/bpf/fixups.c | 32 ++++++------- 3 files changed, 81 insertions(+), 64 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 25148865f9c7..232c02031fec 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1355,7 +1355,8 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, =20 const char *bpf_jit_get_prog_name(struct bpf_prog *prog); =20 -struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, str= uct bpf_prog *prog); +int bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog = **pprog, + bool clone_needed, bool *cloned); void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_o= ther); =20 static inline bool bpf_prog_need_blind(const struct bpf_prog *prog) @@ -1507,9 +1508,11 @@ static inline bool bpf_prog_need_blind(const struct = bpf_prog *prog) } =20 static inline -struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, str= uct bpf_prog *prog) +int bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog = **pprog, + bool clone_needed, bool *cloned) { - return prog; + *cloned =3D false; + return 0; } =20 static inline void bpf_jit_prog_release_other(struct bpf_prog *fp, struct = bpf_prog *fp_other) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index b7f17f89d992..0a8072f59789 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1552,27 +1552,34 @@ void bpf_jit_prog_release_other(struct bpf_prog *fp= , struct bpf_prog *fp_other) * Now this function is used only to blind the main prog and must be invok= ed only when * bpf_prog_need_blind() returns true. */ -struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, str= uct bpf_prog *prog) +int bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog = **pprog, + bool clone_needed, bool *cloned) { struct bpf_insn insn_buff[16], aux[2]; - struct bpf_prog *clone, *tmp; + struct bpf_prog *prog, *orig_prog, *tmp; int insn_delta, insn_cnt; struct bpf_insn *insn; int i, rewritten; =20 - if (WARN_ON_ONCE(env && env->prog !=3D prog)) - return ERR_PTR(-EINVAL); + *cloned =3D false; + if (WARN_ON_ONCE(env && env->prog !=3D *pprog)) + return -EINVAL; =20 - clone =3D bpf_prog_clone_create(prog, GFP_USER); - if (!clone) - return ERR_PTR(-ENOMEM); + prog =3D orig_prog =3D *pprog; + /* only clone the prog when we can fall back to the interpreter */ + if (clone_needed) { + prog =3D bpf_prog_clone_create(orig_prog, GFP_USER); + if (!prog) + return -ENOMEM; =20 - /* make sure bpf_patch_insn_data() patches the correct prog */ - if (env) - env->prog =3D clone; + *cloned =3D true; + /* make sure bpf_patch_insn_data() patches the correct prog */ + if (env) + env->prog =3D prog; + } =20 - insn_cnt =3D clone->len; - insn =3D clone->insnsi; + insn_cnt =3D prog->len; + insn =3D prog->insnsi; =20 for (i =3D 0; i < insn_cnt; i++, insn++) { if (bpf_pseudo_func(insn)) { @@ -1593,42 +1600,49 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf= _verifier_env *env, struct bp insn[1].code =3D=3D 0) memcpy(aux, insn, sizeof(aux)); =20 - rewritten =3D bpf_jit_blind_insn(insn, aux, insn_buff, - clone->aux->verifier_zext); + rewritten =3D bpf_jit_blind_insn(insn, aux, insn_buff, prog->aux->verifi= er_zext); if (!rewritten) continue; =20 if (env) tmp =3D bpf_patch_insn_data(env, i, insn_buff, rewritten); else - tmp =3D bpf_patch_insn_single(clone, i, insn_buff, rewritten); + tmp =3D bpf_patch_insn_single(prog, i, insn_buff, rewritten); =20 if (IS_ERR_OR_NULL(tmp)) { - if (env) - /* restore the original prog */ - env->prog =3D prog; - /* Patching may have repointed aux->prog during - * realloc from the original one, so we need to - * fix it up here on error. - */ - bpf_jit_prog_release_other(prog, clone); - return IS_ERR(tmp) ? tmp : ERR_PTR(-ENOMEM); + if (*cloned) { + /* roll back to the original prog */ + *pprog =3D orig_prog; + if (env) + env->prog =3D orig_prog; + /* Patching may have repointed aux->prog during + * realloc from the original one, so we need to + * fix it up here on error. + */ + bpf_jit_prog_release_other(orig_prog, prog); + } else { + /* just keep the latest successfully patched prog */ + *pprog =3D prog; + } + return IS_ERR(tmp) ? PTR_ERR(tmp) : -ENOMEM; } =20 - clone =3D tmp; + prog =3D tmp; insn_delta =3D rewritten - 1; =20 if (env) - env->prog =3D clone; + env->prog =3D prog; =20 /* Walk new program and skip insns we just inserted. */ - insn =3D clone->insnsi + i + insn_delta; + insn =3D prog->insnsi + i + insn_delta; insn_cnt +=3D insn_delta; i +=3D insn_delta; } =20 - clone->blinded =3D 1; - return clone; + prog->blinded =3D 1; + *pprog =3D prog; + + return 0; } =20 bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const= struct bpf_prog *prog, @@ -2618,33 +2632,33 @@ static bool bpf_prog_select_interpreter(struct bpf_= prog *fp) return select_interpreter; } =20 -static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env,= struct bpf_prog *prog) +static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env,= struct bpf_prog *prog, + bool jit_needed) { #ifdef CONFIG_BPF_JIT + int err; + bool cloned =3D false; struct bpf_prog *orig_prog; =20 if (!bpf_prog_need_blind(prog)) return bpf_int_jit_compile(env, prog); =20 orig_prog =3D prog; - prog =3D bpf_jit_blind_constants(env, prog); - /* - * If blinding was requested and we failed during blinding, we must fall - * back to the interpreter. - */ - if (IS_ERR(prog)) - goto out_restore; + err =3D bpf_jit_blind_constants(env, &prog, !jit_needed, &cloned); + if (err) + goto out; =20 prog =3D bpf_int_jit_compile(env, prog); - if (prog->jited) { - bpf_jit_prog_release_other(prog, orig_prog); - return prog; + if (cloned) { + if (prog->jited) { + bpf_jit_prog_release_other(prog, orig_prog); + } else { + bpf_jit_prog_release_other(orig_prog, prog); + prog =3D orig_prog; + } } =20 - bpf_jit_prog_release_other(orig_prog, prog); - -out_restore: - prog =3D orig_prog; +out: #endif return prog; } @@ -2674,7 +2688,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf= _verifier_env *env, struct if (*err) return fp; =20 - fp =3D bpf_prog_jit_compile(env, fp); + fp =3D bpf_prog_jit_compile(env, fp, jit_needed); bpf_prog_jit_attempt_done(fp); if (!fp->jited && jit_needed) { *err =3D -ENOTSUPP; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 37f22eb41854..80c8beb45ea0 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -996,6 +996,8 @@ static u32 *bpf_dup_subprog_starts(struct bpf_verifier_= env *env) =20 static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *= orig_starts) { + if (!orig_starts) + return; for (int i =3D 0; i < env->subprog_cnt; i++) env->subprog_info[i].start =3D orig_starts[i]; /* restore the start of fake 'exit' subprog as well */ @@ -1273,35 +1275,33 @@ static int jit_subprogs(struct bpf_verifier_env *en= v) int bpf_jit_subprogs(struct bpf_verifier_env *env) { int err, i; - bool blinded =3D false; struct bpf_insn *insn; struct bpf_prog *prog, *orig_prog; - u32 *orig_subprog_starts; + u32 *orig_subprog_starts =3D NULL; + bool cloned =3D false; =20 if (env->subprog_cnt <=3D 1) return 0; =20 prog =3D orig_prog =3D env->prog; if (bpf_prog_need_blind(prog)) { - orig_subprog_starts =3D bpf_dup_subprog_starts(env); - if (!orig_subprog_starts) { - err =3D -ENOMEM; - goto out_cleanup; + if (!prog->jit_required) { + orig_subprog_starts =3D bpf_dup_subprog_starts(env); + if (!orig_subprog_starts) { + err =3D -ENOMEM; + goto out_cleanup; + } } - prog =3D bpf_jit_blind_constants(env, prog); - if (IS_ERR(prog)) { - err =3D -ENOMEM; - prog =3D orig_prog; + err =3D bpf_jit_blind_constants(env, &prog, !prog->jit_required, &cloned= ); + if (err) goto out_restore; - } - blinded =3D true; } =20 err =3D jit_subprogs(env); if (err) goto out_jit_err; =20 - if (blinded) { + if (cloned) { bpf_jit_prog_release_other(prog, orig_prog); kvfree(orig_subprog_starts); } @@ -1309,13 +1309,13 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) return 0; =20 out_jit_err: - if (blinded) { + if (cloned) { bpf_jit_prog_release_other(orig_prog, prog); - /* roll back to the clean original prog */ + /* roll back to the clean original prog */ prog =3D env->prog =3D orig_prog; goto out_restore; } else { - if (err !=3D -EFAULT) { + if (err !=3D -EFAULT && !prog->jit_required) { /* * We will fall back to interpreter mode when err is not -EFAULT, before * that, insn->off and insn->imm should be restored to their original --=20 2.43.0