From nobody Sun Apr 5 13:11:11 2026 Received: from out-170.mta1.migadu.com (out-170.mta1.migadu.com [95.215.58.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02DE433F8D4 for ; Thu, 19 Feb 2026 14:30:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511431; cv=none; b=X4GTUZ4aYst+ej7qpxqn4ahLWyGUhc5OlDkfEZxV3OzXqzmHe/XW0bni2mo5UwkY/TNIq4ohgQJU/ociz70A1Kpg5uypzyceMT6li1mIcoWA5Lv6hUQ5pvOxcoif3rrNnKb0+qoKIQZre70g2QYFaH7iguT6XKoURl4YC6xYT+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511431; c=relaxed/simple; bh=NV9xlLs/FXCt8nqvw59dK19Va73ZVQVrRXiraIES+Vk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kKwTmXI98KnUa+DtqOUIQxq/EhmYV+7ezmpJ8C1Uo8Gx5QqxJd54AlhulI3PM66WfwHjD/78pxaRqfnmnkC1OlpwnirKZxLa+XJlYZ5Ys8pYb4UbVNjgYxrNPIfMJxHBSQmjTecU5inWaPR5d7KzTMiWk71O7dUziN44v8shqdE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=LdLGVj3J; arc=none smtp.client-ip=95.215.58.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="LdLGVj3J" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1771511428; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9+sGQUoFknPrYwnhrHGJoAgCpCs1zVsEElKhl0yi99I=; b=LdLGVj3JZVjoaC4PviPKhiaOGm4q8cPa83+pWAmoAjKnYhX5qUqu+FxPsdRrQXpeF0GQLE Zwqhen5nJJovrxh5oVLCDsPl+TM+8VTaL8zh3tlcVVGXorFEutTkIBqXrR4uvHnjdmooml pogdAz+4IAVZMGWa44hnUB7tulcT+UM= From: Leon Hwang To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Shuah Khan , Leon Hwang , Peilin Ye , Luis Gerhorst , Viktor Malik , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-patches-bot@fb.com Subject: [PATCH bpf-next v2 1/6] bpf: Introduce 64-bit bitops kfuncs Date: Thu, 19 Feb 2026 22:29:23 +0800 Message-ID: <20260219142933.13904-2-leon.hwang@linux.dev> In-Reply-To: <20260219142933.13904-1-leon.hwang@linux.dev> References: <20260219142933.13904-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add the following generic 64-bit bitops kfuncs: * bpf_clz64(): Count leading zeros. * bpf_ctz64(): Count trailing zeros. * bpf_ffs64(): Find first set bit, 1-based index, returns 0 when input is 0. * bpf_fls64(): Find last set bit, 1-based index. * bpf_bitrev64(): Reverse bits. * bpf_popcnt64(): Population count. * bpf_rol64(): Rotate left. * bpf_ror64(): Rotate right. Defined zero-input behavior: * bpf_clz64(0) =3D 64 * bpf_ctz64(0) =3D 64 * bpf_ffs64(0) =3D 0 * bpf_fls64(0) =3D 0 These kfuncs are inlined by JIT backends when the required CPU features are available. Otherwise, they fall back to regular function calls. Signed-off-by: Leon Hwang --- include/linux/filter.h | 10 ++++++++ kernel/bpf/core.c | 6 +++++ kernel/bpf/helpers.c | 50 +++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 53 +++++++++++++++++++++++++++++++++++++++++- 4 files changed, 118 insertions(+), 1 deletion(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 44d7ae95ddbc..b8a538bec5c6 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1157,6 +1157,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog = *prog); void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); bool bpf_jit_inlines_helper_call(s32 imm); +bool bpf_jit_inlines_kfunc_call(void *func_addr); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); @@ -1837,4 +1838,13 @@ static inline void *bpf_skb_meta_pointer(struct sk_b= uff *skb, u32 offset) } #endif /* CONFIG_NET */ =20 +u64 bpf_clz64(u64 x); +u64 bpf_ctz64(u64 x); +u64 bpf_ffs64(u64 x); +u64 bpf_fls64(u64 x); +u64 bpf_popcnt64(u64 x); +u64 bpf_bitrev64(u64 x); +u64 bpf_rol64(u64 x, u64 s); +u64 bpf_ror64(u64 x, u64 s); + #endif /* __LINUX_FILTER_H__ */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 5ab6bace7d0d..5f37309d83fc 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3114,6 +3114,12 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm) return false; } =20 +/* Return TRUE if the JIT backend inlines the kfunc. */ +bool __weak bpf_jit_inlines_kfunc_call(void *func_addr) +{ + return false; +} + /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls(void) { diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 7ac32798eb04..6bf73c46af72 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -29,6 +29,8 @@ #include #include #include +#include +#include =20 #include "../../lib/kstrtox.h" =20 @@ -4501,6 +4503,46 @@ __bpf_kfunc int bpf_timer_cancel_async(struct bpf_ti= mer *timer) } } =20 +__bpf_kfunc u64 bpf_clz64(u64 x) +{ + return x ? 64 - fls64(x) : 64; +} + +__bpf_kfunc u64 bpf_ctz64(u64 x) +{ + return x ? __ffs64(x) : 64; +} + +__bpf_kfunc u64 bpf_ffs64(u64 x) +{ + return x ? __ffs64(x) + 1 : 0; +} + +__bpf_kfunc u64 bpf_fls64(u64 x) +{ + return fls64(x); +} + +__bpf_kfunc u64 bpf_popcnt64(u64 x) +{ + return hweight64(x); +} + +__bpf_kfunc u64 bpf_bitrev64(u64 x) +{ + return ((u64)bitrev32(x & 0xFFFFFFFF) << 32) | bitrev32(x >> 32); +} + +__bpf_kfunc u64 bpf_rol64(u64 x, u64 s) +{ + return rol64(x, s); +} + +__bpf_kfunc u64 bpf_ror64(u64 x, u64 s) +{ + return ror64(x, s); +} + __bpf_kfunc_end_defs(); =20 static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work) @@ -4578,6 +4620,14 @@ BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE) BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE) #endif #endif +BTF_ID_FLAGS(func, bpf_clz64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_ctz64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_ffs64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_fls64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_popcnt64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_bitrev64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_rol64, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_ror64, KF_FASTCALL) BTF_KFUNCS_END(generic_btf_ids) =20 static const struct btf_kfunc_id_set generic_kfunc_set =3D { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0162f946032f..2cb29bc1b3c3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12461,6 +12461,14 @@ enum special_kfunc_type { KF_bpf_session_is_return, KF_bpf_stream_vprintk, KF_bpf_stream_print_stack, + KF_bpf_clz64, + KF_bpf_ctz64, + KF_bpf_ffs64, + KF_bpf_fls64, + KF_bpf_bitrev64, + KF_bpf_popcnt64, + KF_bpf_rol64, + KF_bpf_ror64, }; =20 BTF_ID_LIST(special_kfunc_list) @@ -12541,6 +12549,14 @@ BTF_ID(func, bpf_arena_reserve_pages) BTF_ID(func, bpf_session_is_return) BTF_ID(func, bpf_stream_vprintk) BTF_ID(func, bpf_stream_print_stack) +BTF_ID(func, bpf_clz64) +BTF_ID(func, bpf_ctz64) +BTF_ID(func, bpf_ffs64) +BTF_ID(func, bpf_fls64) +BTF_ID(func, bpf_bitrev64) +BTF_ID(func, bpf_popcnt64) +BTF_ID(func, bpf_rol64) +BTF_ID(func, bpf_ror64) =20 static bool is_task_work_add_kfunc(u32 func_id) { @@ -18204,6 +18220,34 @@ static bool verifier_inlines_helper_call(struct bp= f_verifier_env *env, s32 imm) } } =20 +static bool bpf_kfunc_is_fastcall(struct bpf_verifier_env *env, u32 func_i= d, u32 flags) +{ + if (!(flags & KF_FASTCALL)) + return false; + + if (!env->prog->jit_requested) + return true; + + if (func_id =3D=3D special_kfunc_list[KF_bpf_clz64]) + return bpf_jit_inlines_kfunc_call(bpf_clz64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_ctz64]) + return bpf_jit_inlines_kfunc_call(bpf_ctz64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_ffs64]) + return bpf_jit_inlines_kfunc_call(bpf_ffs64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_fls64]) + return bpf_jit_inlines_kfunc_call(bpf_fls64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_bitrev64]) + return bpf_jit_inlines_kfunc_call(bpf_bitrev64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_popcnt64]) + return bpf_jit_inlines_kfunc_call(bpf_popcnt64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_rol64]) + return bpf_jit_inlines_kfunc_call(bpf_rol64); + if (func_id =3D=3D special_kfunc_list[KF_bpf_ror64]) + return bpf_jit_inlines_kfunc_call(bpf_ror64); + + return true; +} + struct call_summary { u8 num_params; bool is_void; @@ -18246,7 +18290,7 @@ static bool get_call_summary(struct bpf_verifier_en= v *env, struct bpf_insn *call /* error would be reported later */ return false; cs->num_params =3D btf_type_vlen(meta.func_proto); - cs->fastcall =3D meta.kfunc_flags & KF_FASTCALL; + cs->fastcall =3D bpf_kfunc_is_fastcall(env, meta.func_id, meta.kfunc_fla= gs); cs->is_void =3D btf_type_is_void(btf_type_by_id(meta.btf, meta.func_prot= o->type)); return true; } @@ -23186,6 +23230,13 @@ static int fixup_kfunc_call(struct bpf_verifier_en= v *env, struct bpf_insn *insn, insn_buf[4] =3D BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1); insn_buf[5] =3D BPF_ALU64_IMM(BPF_NEG, BPF_REG_0, 0); *cnt =3D 6; + } else if (desc->func_id =3D=3D special_kfunc_list[KF_bpf_ffs64] && + bpf_jit_inlines_kfunc_call(bpf_ffs64)) { + insn_buf[0] =3D BPF_MOV64_IMM(BPF_REG_0, 0); + insn_buf[1] =3D BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2); + insn_buf[2] =3D *insn; + insn_buf[3] =3D BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1); + *cnt =3D 4; } =20 if (env->insn_aux_data[insn_idx].arg_prog) { --=20 2.52.0 From nobody Sun Apr 5 13:11:11 2026 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7649A33F8A8 for ; Thu, 19 Feb 2026 14:30:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511444; cv=none; b=e3m38uvvDPS1fFlbScE6ivwLEEEaeRp3zpAShk5RUJXXwFmM+eQvBITtyj+0d4RzrYdd5490xMPmlU9O8C8b6l29sWQYTgsb4fCOwMUknHXzpoxkzCdEQ99dxoJTct95sqJtdgYjE46wNlNKQFJUEJf4PMh1aC3zzZeq6lOCcLU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511444; c=relaxed/simple; bh=4ca/FLms+iViJ1tTVUEuHvLAF1+L3KEPpXg3rugt9ZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=owev8AkohoYT+AZcwtQhcndYxVrz7amjkwB0YbGff+bXDOEFycMxK/EzQHOrHnz5SwZ1J1zOcn06cT1b5vx6SAEzVrHvTB/A4Pbm+TQKnTZpXy1azzgjLGmNYS1SU7EAYFoDw8AJwKpb0u40FNyXoFI8c/ops35hKa2+JFynL6A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=WATtM/3Q; arc=none smtp.client-ip=95.215.58.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="WATtM/3Q" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1771511440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LbyH7r/+43Qi6pmkbYmF9oCsizXUoRzTF9X7Dq2Fsj8=; b=WATtM/3QzNXG5tUxfI9ZvXAHFtiYkE/toigRhrNoHvE1xC4pC3wH0zs+iR3XZEr9AyDElM sRqdRfkYLJJPNVpFvSttasDvnKplXkgPHcqyGcpy5M70F7J13M/is1d3qRiI9issMtdkeM xqMk/r3eiR5vHe5oCQOYbIj8/SO8e7E= From: Leon Hwang To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Shuah Khan , Leon Hwang , Peilin Ye , Luis Gerhorst , Viktor Malik , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-patches-bot@fb.com Subject: [PATCH bpf-next v2 2/6] bpf, x86: Add 64-bit bitops kfuncs support for x86_64 Date: Thu, 19 Feb 2026 22:29:24 +0800 Message-ID: <20260219142933.13904-3-leon.hwang@linux.dev> In-Reply-To: <20260219142933.13904-1-leon.hwang@linux.dev> References: <20260219142933.13904-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Implement JIT inlining of the 64-bit bitops kfuncs on x86_64. bpf_rol64() and bpf_ror64() are always supported via ROL/ROR. bpf_ctz64() and bpf_ffs64() are supported when the CPU has X86_FEATURE_BMI1 (TZCNT). bpf_clz64() and bpf_fls64() are supported when the CPU has X86_FEATURE_ABM (LZCNT). bpf_popcnt64() is supported when the CPU has X86_FEATURE_POPCNT. bpf_bitrev64() is not inlined as x86_64 has no native bit-reverse instruction, so it falls back to a regular function call. Signed-off-by: Leon Hwang --- arch/x86/net/bpf_jit_comp.c | 141 ++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 070ba80e39d7..193e1e2d7aa8 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 static bool all_callee_regs_used[4] =3D {true, true, true, true}; =20 @@ -1604,6 +1605,127 @@ static void emit_priv_frame_ptr(u8 **pprog, void __= percpu *priv_frame_ptr) *pprog =3D prog; } =20 +static bool bpf_inlines_func_call(u8 **pprog, void *func) +{ + bool has_popcnt =3D boot_cpu_has(X86_FEATURE_POPCNT); + bool has_bmi1 =3D boot_cpu_has(X86_FEATURE_BMI1); + bool has_abm =3D boot_cpu_has(X86_FEATURE_ABM); + bool inlined =3D true; + u8 *prog =3D *pprog; + + /* + * x86 Bit manipulation instruction set + * https://en.wikipedia.org/wiki/X86_Bit_manipulation_instruction_set + */ + + if (func =3D=3D bpf_clz64 && has_abm) { + /* + * Intel=C2=AE 64 and IA-32 Architectures Software Developer's Manual (J= une 2023) + * + * LZCNT - Count the Number of Leading Zero Bits + * + * Opcode/Instruction + * F3 REX.W 0F BD /r + * LZCNT r64, r/m64 + * + * Op/En + * RVM + * + * 64/32-bit Mode + * V/N.E. + * + * CPUID Feature Flag + * LZCNT + * + * Description + * Count the number of leading zero bits in r/m64, return + * result in r64. + */ + /* emit: x ? 64 - fls64(x) : 64 */ + /* lzcnt rax, rdi */ + EMIT5(0xF3, 0x48, 0x0F, 0xBD, 0xC7); + } else if (func =3D=3D bpf_ctz64 && has_bmi1) { + /* + * Intel=C2=AE 64 and IA-32 Architectures Software Developer's Manual (J= une 2023) + * + * TZCNT - Count the Number of Trailing Zero Bits + * + * Opcode/Instruction + * F3 REX.W 0F BC /r + * TZCNT r64, r/m64 + * + * Op/En + * RVM + * + * 64/32-bit Mode + * V/N.E. + * + * CPUID Feature Flag + * BMI1 + * + * Description + * Count the number of trailing zero bits in r/m64, return + * result in r64. + */ + /* emit: x ? __ffs64(x) : 64 */ + /* tzcnt rax, rdi */ + EMIT5(0xF3, 0x48, 0x0F, 0xBC, 0xC7); + } else if (func =3D=3D bpf_ffs64 && has_bmi1) { + /* emit: __ffs64(x); x =3D=3D 0 has been handled in verifier */ + /* tzcnt rax, rdi */ + EMIT5(0xF3, 0x48, 0x0F, 0xBC, 0xC7); + } else if (func =3D=3D bpf_fls64 && has_abm) { + /* emit: fls64(x) */ + /* lzcnt rax, rdi */ + EMIT5(0xF3, 0x48, 0x0F, 0xBD, 0xC7); + EMIT3(0x48, 0xF7, 0xD8); /* neg rax */ + EMIT4(0x48, 0x83, 0xC0, 0x40); /* add rax, 64 */ + } else if (func =3D=3D bpf_popcnt64 && has_popcnt) { + /* + * Intel=C2=AE 64 and IA-32 Architectures Software Developer's Manual (J= une 2023) + * + * POPCNT - Return the Count of Number of Bits Set to 1 + * + * Opcode/Instruction + * F3 REX.W 0F B8 /r + * POPCNT r64, r/m64 + * + * Op/En + * RM + * + * 64 Mode + * Valid + * + * Compat/Leg Mode + * N.E. + * + * Description + * POPCNT on r/m64 + */ + /* popcnt rax, rdi */ + EMIT5(0xF3, 0x48, 0x0F, 0xB8, 0xC7); + } else if (func =3D=3D bpf_rol64) { + EMIT1(0x51); /* push rcx */ + /* emit: rol64(x, s) */ + EMIT3(0x48, 0x89, 0xF1); /* mov rcx, rsi */ + EMIT3(0x48, 0x89, 0xF8); /* mov rax, rdi */ + EMIT3(0x48, 0xD3, 0xC0); /* rol rax, cl */ + EMIT1(0x59); /* pop rcx */ + } else if (func =3D=3D bpf_ror64) { + EMIT1(0x51); /* push rcx */ + /* emit: ror64(x, s) */ + EMIT3(0x48, 0x89, 0xF1); /* mov rcx, rsi */ + EMIT3(0x48, 0x89, 0xF8); /* mov rax, rdi */ + EMIT3(0x48, 0xD3, 0xC8); /* ror rax, cl */ + EMIT1(0x59); /* pop rcx */ + } else { + inlined =3D false; + } + + *pprog =3D prog; + return inlined; +} + #define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp))) =20 #define __LOAD_TCC_PTR(off) \ @@ -2452,6 +2574,8 @@ st: if (is_imm8(insn->off)) u8 *ip =3D image + addrs[i - 1]; =20 func =3D (u8 *) __bpf_call_base + imm32; + if (bpf_inlines_func_call(&prog, func)) + break; if (src_reg =3D=3D BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); ip +=3D 7; @@ -4117,3 +4241,20 @@ bool bpf_jit_supports_fsession(void) { return true; } + +bool bpf_jit_inlines_kfunc_call(void *func_addr) +{ + if (func_addr =3D=3D bpf_ctz64 || func_addr =3D=3D bpf_ffs64) + return boot_cpu_has(X86_FEATURE_BMI1); + + if (func_addr =3D=3D bpf_clz64 || func_addr =3D=3D bpf_fls64) + return boot_cpu_has(X86_FEATURE_ABM); + + if (func_addr =3D=3D bpf_popcnt64) + return boot_cpu_has(X86_FEATURE_POPCNT); + + if (func_addr =3D=3D bpf_rol64 || func_addr =3D=3D bpf_ror64) + return true; + + return false; +} --=20 2.52.0 From nobody Sun Apr 5 13:11:11 2026 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D26FB253B58 for ; Thu, 19 Feb 2026 14:31:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511491; cv=none; b=HI99wz3f7JgZg0HAhzTNLJLMy4/nz7jjEDBjFBhKeKI0pUssV1NpGDJl+wm0jWipqeSjCpAmLFSj+lWUzy+Q6SPGdvcuFYlayhw33TS7jDpb9fKQgZui0Akjyw1rku6gyUKHFZulJpEtUsrPXhq6i+PPhQaI/eH5NLrG1GLjHmU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511491; c=relaxed/simple; bh=okXF8hqTMNtslj4tawfqT6rUWjNbqU+Szo2bMXHuVtY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fyhtXKmFcr+tnI3m60pPh0Wpj2svkte2AmwdfYBPlCOosksOuusnqWXzmid7NmOVKSzZUBdYzHioR73brboCF10PPpcNTsfmMc7bnCNVjgLKxxlqLL1iznw5AIWS2cb7pRUAFgs8vtzDQp6juUfIFDxFehfwUD3A51hkpLwyQDM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=XDu1ztxs; arc=none smtp.client-ip=95.215.58.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="XDu1ztxs" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1771511487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W9I9TydWVo15yKzkAGFLtCWQjvdOPQocpTW8F8p/JRI=; b=XDu1ztxsOwv2MmylQD0qb4UCW5fqcEoF4pFpbhXOIVlzj8Js4KnUzKViS3VL1nbB5t0aI6 C0VmxmzJKZmbudyuazDQ2XpWcujZ2AsRVwShzv0kR39Hbv8mwH8uz8qS4aJawcAPwkLhxh vTPPAFLMR1eemfo9bIwXvWuSpLfUEVs= From: Leon Hwang To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Shuah Khan , Leon Hwang , Peilin Ye , Luis Gerhorst , Viktor Malik , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-patches-bot@fb.com Subject: [PATCH bpf-next v2 3/6] bpf, arm64: Add 64-bit bitops kfuncs support Date: Thu, 19 Feb 2026 22:29:25 +0800 Message-ID: <20260219142933.13904-4-leon.hwang@linux.dev> In-Reply-To: <20260219142933.13904-1-leon.hwang@linux.dev> References: <20260219142933.13904-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Implement JIT inlining of the 64-bit bitops kfuncs on arm64. bpf_clz64(), bpf_ffs64(), bpf_fls64(), and bpf_bitrev64() are always inlined using mandatory ARMv8 CLZ/RBIT instructions. bpf_ctz64() is inlined via RBIT + CLZ, or via the native CTZ instruction when FEAT_CSSC is available. bpf_rol64() and bpf_ror64() are always inlined via RORV. bpf_popcnt64() is not inlined as the native population count instruction requires NEON/SIMD registers, which should not be touched from BPF programs. It therefore falls back to a regular function call. Signed-off-by: Leon Hwang --- arch/arm64/net/bpf_jit_comp.c | 123 ++++++++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 7a530ea4f5ae..f03f732063d9 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1192,6 +1192,127 @@ static int add_exception_handler(const struct bpf_i= nsn *insn, return 0; } =20 +static inline u32 a64_clz64(u8 rd, u8 rn) +{ + /* + * Arm Architecture Reference Manual for A-profile architecture + * (Document number: ARM DDI 0487) + * + * A64 Base Instruction Descriptions + * C6.2 Alphabetical list of A64 base instructions + * + * C6.2.91 CLZ + * + * Count leading zeros + * + * This instruction counts the number of consecutive binary zero bits, + * starting from the most significant bit in the source register, + * and places the count in the destination register. + */ + /* CLZ Xd, Xn */ + return 0xdac01000 | (rn << 5) | rd; +} + +static inline u32 a64_ctz64(u8 rd, u8 rn) +{ + /* + * Arm Architecture Reference Manual for A-profile architecture + * (Document number: ARM DDI 0487) + * + * A64 Base Instruction Descriptions + * C6.2 Alphabetical list of A64 base instructions + * + * C6.2.144 CTZ + * + * Count trailing zeros + * + * This instruction counts the number of consecutive binary zero bits, + * starting from the least significant bit in the source register, + * and places the count in the destination register. + * + * This instruction requires FEAT_CSSC. + */ + /* CTZ Xd, Xn */ + return 0xdac01800 | (rn << 5) | rd; +} + +static inline u32 a64_rbit64(u8 rd, u8 rn) +{ + /* + * Arm Architecture Reference Manual for A-profile architecture + * (Document number: ARM DDI 0487) + * + * A64 Base Instruction Descriptions + * C6.2 Alphabetical list of A64 base instructions + * + * C6.2.320 RBIT + * + * Reverse bits + * + * This instruction reverses the bit order in a register. + */ + /* RBIT Xd, Xn */ + return 0xdac00000 | (rn << 5) | rd; +} + +static inline bool boot_cpu_supports_cssc(void) +{ + /* + * Documentation/arch/arm64/cpu-feature-registers.rst + * + * ID_AA64ISAR2_EL1 - Instruction set attribute register 2 + * + * CSSC + */ + return cpuid_feature_extract_unsigned_field(read_sanitised_ftr_reg(SYS_ID= _AA64ISAR2_EL1), + ID_AA64ISAR2_EL1_CSSC_SHIFT); +} + +static bool bpf_inlines_func_call(struct jit_ctx *ctx, void *func_addr) +{ + const u8 tmp =3D bpf2a64[TMP_REG_1]; + const u8 r0 =3D bpf2a64[BPF_REG_0]; + const u8 r1 =3D bpf2a64[BPF_REG_1]; + const u8 r2 =3D bpf2a64[BPF_REG_2]; + bool inlined =3D true; + + if (func_addr =3D=3D bpf_clz64) { + emit(a64_clz64(r0, r1), ctx); + } else if (func_addr =3D=3D bpf_ctz64 || func_addr =3D=3D bpf_ffs64) { + if (boot_cpu_supports_cssc()) { + emit(a64_ctz64(r0, r1), ctx); + } else { + emit(a64_rbit64(tmp, r1), ctx); + emit(a64_clz64(r0, tmp), ctx); + } + } else if (func_addr =3D=3D bpf_fls64) { + emit(a64_clz64(tmp, r1), ctx); + emit(A64_NEG(1, tmp, tmp), ctx); + emit(A64_ADD_I(1, r0, tmp, 64), ctx); + } else if (func_addr =3D=3D bpf_bitrev64) { + emit(a64_rbit64(r0, r1), ctx); + } else if (func_addr =3D=3D bpf_rol64) { + emit(A64_NEG(1, tmp, r2), ctx); + emit(A64_DATA2(1, r0, r1, tmp, RORV), ctx); + } else if (func_addr =3D=3D bpf_ror64) { + emit(A64_DATA2(1, r0, r1, r2, RORV), ctx); + } else { + inlined =3D false; + } + + return inlined; +} + +bool bpf_jit_inlines_kfunc_call(void *func_addr) +{ + if (func_addr =3D=3D bpf_clz64 || func_addr =3D=3D bpf_ctz64 || + func_addr =3D=3D bpf_ffs64 || func_addr =3D=3D bpf_fls64 || + func_addr =3D=3D bpf_rol64 || func_addr =3D=3D bpf_ror64 || + func_addr =3D=3D bpf_bitrev64) + return true; + return false; +} + /* JITs an eBPF instruction. * Returns: * 0 - successfully JITed an 8-byte eBPF instruction. @@ -1598,6 +1719,8 @@ static int build_insn(const struct bpf_insn *insn, st= ruct jit_ctx *ctx, &func_addr, &func_addr_fixed); if (ret < 0) return ret; + if (bpf_inlines_func_call(ctx, (void *) func_addr)) + break; emit_call(func_addr, ctx); /* * Call to arch_bpf_timed_may_goto() is emitted by the --=20 2.52.0 From nobody Sun Apr 5 13:11:11 2026 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 445ED2FE57B for ; Thu, 19 Feb 2026 14:31:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511506; cv=none; b=Hzu6c5FV2lOt7Nc7si+WYILn6G7v+ZfRQgDaSUrMuQDfvYAPzGolH3p8WGyoXaeQSO8XRk81yCAm4Lzj/bj+VhU3LGzI89nQLoeo23ddhol3ZlvMXLC7XnwRWpd5Mg9R0Q7BiXdYfm/yrTCKSB5IpA6pUdIfVqNwuaXqOLnqBFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511506; c=relaxed/simple; bh=ypZOtjUaRaMCXC1q6gBRjeCXf2W7vKM3BCik4l5EdmI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BnUajEfvPdvVkMeiZtMSyDCrtCDTHjNJ45OcsaEu8Wccm4/ph+af3o3Jwfo7GuZCM32h1wzv6yaT98HSGGUyuMdy5hxM14l1XAgNVZK9rsUKDX/RbBUoFIoV+0JRPDCC6UJGIP24cp0p+RzZ6jOE37ZLKXNpzFLgevHycKFJP7A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=pmq3iPTU; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="pmq3iPTU" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1771511502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jNWdJdK58m69LHIGW7l7DEM1Iok2ekKHAFCQkjcDuSU=; b=pmq3iPTU63sGP1ZRWI1plSYlIte7DRbyd44PXvjrbFkYECiRO2VgIlPGgGYL84uz3EB28a l66sqiSOUYeMXIAIyJkZkrwGIRzeuQWlmkkdDPEuthjbgQ+eODaXueBFgt4EY9LOVQREtV jYWNkODtdCcUubHfy1Q5R3NBRGfFRHA= From: Leon Hwang To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Shuah Khan , Leon Hwang , Peilin Ye , Luis Gerhorst , Viktor Malik , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-patches-bot@fb.com Subject: [PATCH bpf-next v2 4/6] selftests/bpf: Add tests for 64-bit bitops kfuncs Date: Thu, 19 Feb 2026 22:29:26 +0800 Message-ID: <20260219142933.13904-5-leon.hwang@linux.dev> In-Reply-To: <20260219142933.13904-1-leon.hwang@linux.dev> References: <20260219142933.13904-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add selftests for bpf_clz64(), bpf_ctz64(), bpf_ffs64(), bpf_fls64(), bpf_bitrev64(), bpf_popcnt64(), bpf_rol64(), and bpf_ror64(). Each subtest compares kfunc results against a userspace reference implementation over a set of test vectors. Signed-off-by: Leon Hwang --- .../testing/selftests/bpf/bpf_experimental.h | 9 + .../testing/selftests/bpf/prog_tests/bitops.c | 182 ++++++++++++++++++ tools/testing/selftests/bpf/progs/bitops.c | 68 +++++++ 3 files changed, 259 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/bitops.c create mode 100644 tools/testing/selftests/bpf/progs/bitops.c diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing= /selftests/bpf/bpf_experimental.h index 9df77e59d4f5..02a985ef71cc 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -594,6 +594,15 @@ extern void bpf_iter_dmabuf_destroy(struct bpf_iter_dm= abuf *it) __weak __ksym; extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__= str, struct bpf_dynptr *value_p) __weak __ksym; =20 +extern __u64 bpf_clz64(__u64 x) __weak __ksym; +extern __u64 bpf_ctz64(__u64 x) __weak __ksym; +extern __u64 bpf_ffs64(__u64 x) __weak __ksym; +extern __u64 bpf_fls64(__u64 x) __weak __ksym; +extern __u64 bpf_bitrev64(__u64 x) __weak __ksym; +extern __u64 bpf_popcnt64(__u64 x) __weak __ksym; +extern __u64 bpf_rol64(__u64 x, __u64 s) __weak __ksym; +extern __u64 bpf_ror64(__u64 x, __u64 s) __weak __ksym; + #define PREEMPT_BITS 8 #define SOFTIRQ_BITS 8 #define HARDIRQ_BITS 4 diff --git a/tools/testing/selftests/bpf/prog_tests/bitops.c b/tools/testin= g/selftests/bpf/prog_tests/bitops.c new file mode 100644 index 000000000000..9acc3cb1908c --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/bitops.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include "bitops.skel.h" + +struct bitops_case { + __u64 x; + __u64 s; + __u64 exp; +}; + +static struct bitops_case cases[] =3D { + { 0x0ULL, 0, 0 }, + { 0x1ULL, 1, 0 }, + { 0x8000000000000000ULL, 63, 0 }, + { 0xffffffffffffffffULL, 64, 0 }, + { 0x0123456789abcdefULL, 65, 0 }, + { 0x0000000100000000ULL, 127, 0 }, +}; + +static __u64 clz64(__u64 x, __u64 s) +{ + (void)s; + return x ? __builtin_clzll(x) : 64; +} + +static __u64 ctz64(__u64 x, __u64 s) +{ + (void)s; + return x ? __builtin_ctzll(x) : 64; +} + +static __u64 ffs64(__u64 x, __u64 s) +{ + (void)s; + return x ? (__u64)__builtin_ctzll(x) + 1 : 0; +} + +static __u64 fls64(__u64 x, __u64 s) +{ + (void)s; + return x ? 64 - __builtin_clzll(x) : 0; +} + +static __u64 popcnt64(__u64 x, __u64 s) +{ + (void)s; + return __builtin_popcountll(x); +} + +static __u64 bitrev64(__u64 x, __u64 s) +{ + __u64 y =3D 0; + int i; + + (void)s; + + for (i =3D 0; i < 64; i++) { + y <<=3D 1; + y |=3D x & 1; + x >>=3D 1; + } + return y; +} + +static __u64 rol64(__u64 x, __u64 s) +{ + s &=3D 63; + return (x << s) | (x >> ((-s) & 63)); +} + +static __u64 ror64(__u64 x, __u64 s) +{ + s &=3D 63; + return (x >> s) | (x << ((-s) & 63)); +} + +static void test_bitops_case(const char *prog_name) +{ + struct bpf_program *prog; + struct bitops *skel; + size_t i; + int err; + LIBBPF_OPTS(bpf_test_run_opts, topts); + + skel =3D bitops__open(); + if (!ASSERT_OK_PTR(skel, "bitops__open")) + return; + + prog =3D bpf_object__find_program_by_name(skel->obj, prog_name); + if (!ASSERT_OK_PTR(prog, "bpf_object__find_program_by_name")) + goto cleanup; + + bpf_program__set_autoload(prog, true); + + err =3D bitops__load(skel); + if (!ASSERT_OK(err, "bitops__load")) + goto cleanup; + + for (i =3D 0; i < ARRAY_SIZE(cases); i++) { + skel->bss->in_x =3D cases[i].x; + skel->bss->in_s =3D cases[i].s; + err =3D bpf_prog_test_run_opts(bpf_program__fd(prog), &topts); + if (!ASSERT_OK(err, "bpf_prog_test_run_opts")) + goto cleanup; + + if (!ASSERT_OK(topts.retval, "retval")) + goto cleanup; + + ASSERT_EQ(skel->bss->out, cases[i].exp, "out"); + } + +cleanup: + bitops__destroy(skel); +} + +#define RUN_BITOPS_CASE(_bitops, _prog) \ + do { \ + for (size_t i =3D 0; i < ARRAY_SIZE(cases); i++) \ + cases[i].exp =3D _bitops(cases[i].x, cases[i].s); \ + test_bitops_case(_prog); \ + } while (0) + +static void test_clz64(void) +{ + RUN_BITOPS_CASE(clz64, "bitops_clz64"); +} + +static void test_ctz64(void) +{ + RUN_BITOPS_CASE(ctz64, "bitops_ctz64"); +} + +static void test_ffs64(void) +{ + RUN_BITOPS_CASE(ffs64, "bitops_ffs64"); +} + +static void test_fls64(void) +{ + RUN_BITOPS_CASE(fls64, "bitops_fls64"); +} + +static void test_bitrev64(void) +{ + RUN_BITOPS_CASE(bitrev64, "bitops_bitrev"); +} + +static void test_popcnt64(void) +{ + RUN_BITOPS_CASE(popcnt64, "bitops_popcnt"); +} + +static void test_rol64(void) +{ + RUN_BITOPS_CASE(rol64, "bitops_rol64"); +} + +static void test_ror64(void) +{ + RUN_BITOPS_CASE(ror64, "bitops_ror64"); +} + +void test_bitops(void) +{ + if (test__start_subtest("clz64")) + test_clz64(); + if (test__start_subtest("ctz64")) + test_ctz64(); + if (test__start_subtest("ffs64")) + test_ffs64(); + if (test__start_subtest("fls64")) + test_fls64(); + if (test__start_subtest("bitrev64")) + test_bitrev64(); + if (test__start_subtest("popcnt64")) + test_popcnt64(); + if (test__start_subtest("rol64")) + test_rol64(); + if (test__start_subtest("ror64")) + test_ror64(); +} diff --git a/tools/testing/selftests/bpf/progs/bitops.c b/tools/testing/sel= ftests/bpf/progs/bitops.c new file mode 100644 index 000000000000..deac09bc8683 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/bitops.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" +#include +#include "bpf_experimental.h" + +__u64 in_x; +__u64 in_s; + +__u64 out; + +SEC("?syscall") +int bitops_clz64(void *ctx) +{ + out =3D bpf_clz64(in_x); + return 0; +} + +SEC("?syscall") +int bitops_ctz64(void *ctx) +{ + out =3D bpf_ctz64(in_x); + return 0; +} + +SEC("?syscall") +int bitops_ffs64(void *ctx) +{ + out =3D bpf_ffs64(in_x); + return 0; +} + +SEC("?syscall") +int bitops_fls64(void *ctx) +{ + out =3D bpf_fls64(in_x); + return 0; +} + +SEC("?syscall") +int bitops_bitrev(void *ctx) +{ + out =3D bpf_bitrev64(in_x); + return 0; +} + +SEC("?syscall") +int bitops_popcnt(void *ctx) +{ + out =3D bpf_popcnt64(in_x); + return 0; +} + +SEC("?syscall") +int bitops_rol64(void *ctx) +{ + out =3D bpf_rol64(in_x, in_s); + return 0; +} + +SEC("?syscall") +int bitops_ror64(void *ctx) +{ + out =3D bpf_ror64(in_x, in_s); + return 0; +} + +char _license[] SEC("license") =3D "GPL"; --=20 2.52.0 From nobody Sun Apr 5 13:11:11 2026 Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDF90253B58 for ; Thu, 19 Feb 2026 14:31:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511520; cv=none; b=OngC+wIxSII+/RqBmIX/XHM7Rr78FyLRQ5rjrxoFZKfkLi4stXiQ0C4LnYj6QyyXmM/Xay+do6j9bRGNBGdvLXR0mztRsvE7swjw2vvw8rp0rV2y8KCuIzF3uStG1VBAbW+55XoqVmCO0hagwRQNfkWmUmQCp77uVF4Cj3PHibE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511520; c=relaxed/simple; bh=4nZ/Hxz+9njS2oPp6AwC1Vd/k0sBQjMCSloOzzXzEmE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ub88jA8EwYMnG2V39x7wUJ6yYTnWQ3IcomL1rboYcmqWeOhjwXpOuClW6EiTajAWD+AlPKOXfcBh88tFfJB9p1cPAdRtNbOscA7cnf8MD3DZPPec9lCA7YxG52rgydx+qTXMZJmjOERccnWgw9JXWGtf5q7I2s/FYauYIS6lC6o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=eSagwInd; arc=none smtp.client-ip=95.215.58.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="eSagwInd" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1771511516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TDysQwXc7CzQOBmT/kFdKL3V02JclHexDCmLhDyq0UI=; b=eSagwIndy6NTov4HnTmGveVQ7Qn3mncFyVvmyz6/94sXxh9PvILdu/4ZsjM/jt0WTU9S2a BLdhXtWb30gxjHiVKtHZ/zt/WN0NAILwChi84PC8tvqHMBEEPCQsRcj3+ZmZh4gq04SNWw ALGgmG0EwvspQPRPbgdPHikpTLSCxls= From: Leon Hwang To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Shuah Khan , Leon Hwang , Peilin Ye , Luis Gerhorst , Viktor Malik , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-patches-bot@fb.com Subject: [PATCH bpf-next v2 5/6] selftests/bpf: Add __cpu_feature annotation for CPU-feature-gated tests Date: Thu, 19 Feb 2026 22:29:27 +0800 Message-ID: <20260219142933.13904-6-leon.hwang@linux.dev> In-Reply-To: <20260219142933.13904-1-leon.hwang@linux.dev> References: <20260219142933.13904-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add a new __cpu_feature("...") test annotation and parse it in selftests/bpf test_loader. Behavior: - Annotation value is matched against CPU feature tokens from /proc/cpuinfo (case-insensitive). - Multiple __cpu_feature annotations can be specified for one test; all required features must be present. - If any required feature is missing, the test is skipped. Limitation: - __cpu_feature is evaluated per test function and is not scoped per __arch_* block. A single test that combines multiple architectures cannot express different per-arch feature requirements. This lets JIT/disassembly-sensitive tests declare explicit CPU feature requirements and avoid false failures on unsupported systems. Signed-off-by: Leon Hwang --- tools/testing/selftests/bpf/progs/bpf_misc.h | 7 + tools/testing/selftests/bpf/test_loader.c | 150 +++++++++++++++++++ 2 files changed, 157 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/s= elftests/bpf/progs/bpf_misc.h index c9bfbe1bafc1..75e66373a64d 100644 --- a/tools/testing/selftests/bpf/progs/bpf_misc.h +++ b/tools/testing/selftests/bpf/progs/bpf_misc.h @@ -126,6 +126,12 @@ * Several __arch_* annotations could be specified at on= ce. * When test case is not run on current arch it is marke= d as skipped. * __caps_unpriv Specify the capabilities that should be set when runn= ing the test. + * __cpu_feature Specify required CPU feature for test execution. + * Multiple __cpu_feature annotations could be specified. + * Value must match a CPU feature token exposed by + * /proc/cpuinfo (case-insensitive). + * Can't be used together with multiple __arch_* tags. + * If any required feature is not present, test case is = skipped. * * __linear_size Specify the size of the linear area of non-linear skb= s, or * 0 for linear skbs. @@ -156,6 +162,7 @@ #define __arch_riscv64 __arch("RISCV64") #define __arch_s390x __arch("s390x") #define __caps_unpriv(caps) __attribute__((btf_decl_tag("comment:test_caps= _unpriv=3D" EXPAND_QUOTE(caps)))) +#define __cpu_feature(feat) __attribute__((btf_decl_tag("comment:test_cpu_= feature=3D" feat))) #define __load_if_JITed() __attribute__((btf_decl_tag("comment:load_mode= =3Djited"))) #define __load_if_no_JITed() __attribute__((btf_decl_tag("comment:load_mod= e=3Dno_jited"))) #define __stderr(msg) __attribute__((btf_decl_tag("comment:test_expect_st= derr=3D" XSTR(__COUNTER__) "=3D" msg))) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/self= tests/bpf/test_loader.c index 338c035c3688..3729d1572589 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -4,6 +4,7 @@ #include #include #include +#include =20 #include "autoconf_helper.h" #include "disasm_helpers.h" @@ -44,6 +45,7 @@ #define TEST_TAG_EXPECT_STDOUT_PFX "comment:test_expect_stdout=3D" #define TEST_TAG_EXPECT_STDOUT_PFX_UNPRIV "comment:test_expect_stdout_unpr= iv=3D" #define TEST_TAG_LINEAR_SIZE "comment:test_linear_size=3D" +#define TEST_TAG_CPU_FEATURE_PFX "comment:test_cpu_feature=3D" =20 /* Warning: duplicated in bpf_misc.h */ #define POINTER_VALUE 0xbadcafe @@ -67,6 +69,11 @@ enum load_mode { NO_JITED =3D 1 << 1, }; =20 +struct cpu_feature_set { + char **names; + size_t cnt; +}; + struct test_subspec { char *name; bool expect_failure; @@ -93,6 +100,7 @@ struct test_spec { int linear_sz; bool auxiliary; bool valid; + struct cpu_feature_set cpu_features; }; =20 static int tester_init(struct test_loader *tester) @@ -145,6 +153,16 @@ static void free_test_spec(struct test_spec *spec) free(spec->unpriv.name); spec->priv.name =3D NULL; spec->unpriv.name =3D NULL; + + if (spec->cpu_features.names) { + size_t i; + + for (i =3D 0; i < spec->cpu_features.cnt; i++) + free(spec->cpu_features.names[i]); + free(spec->cpu_features.names); + spec->cpu_features.names =3D NULL; + spec->cpu_features.cnt =3D 0; + } } =20 /* Compiles regular expression matching pattern. @@ -394,6 +412,122 @@ static int get_current_arch(void) return ARCH_UNKNOWN; } =20 +static int cpu_feature_set_add(struct cpu_feature_set *set, const char *na= me) +{ + char **tmp, *norm; + size_t i, len; + + if (!name || !name[0]) { + PRINT_FAIL("bad cpu feature spec: empty string"); + return -EINVAL; + } + + len =3D strlen(name); + norm =3D malloc(len + 1); + if (!norm) + return -ENOMEM; + + for (i =3D 0; i < len; i++) { + if (isspace(name[i])) { + free(norm); + PRINT_FAIL("bad cpu feature spec: whitespace is not allowed in '%s'", n= ame); + return -EINVAL; + } + norm[i] =3D tolower((unsigned char)name[i]); + } + norm[len] =3D '\0'; + + for (i =3D 0; i < set->cnt; i++) { + if (strcmp(set->names[i], norm) =3D=3D 0) { + free(norm); + return 0; + } + } + + tmp =3D realloc(set->names, (set->cnt + 1) * sizeof(*set->names)); + if (!tmp) { + free(norm); + return -ENOMEM; + } + set->names =3D tmp; + set->names[set->cnt++] =3D norm; + return 0; +} + +static bool cpu_feature_set_has(const struct cpu_feature_set *set, const c= har *name) +{ + size_t i; + + for (i =3D 0; i < set->cnt; i++) { + if (strcmp(set->names[i], name) =3D=3D 0) + return true; + } + return false; +} + +static bool cpu_feature_set_includes(const struct cpu_feature_set *have, + const struct cpu_feature_set *need) +{ + size_t i; + + for (i =3D 0; i < need->cnt; i++) { + if (!cpu_feature_set_has(have, need->names[i])) + return false; + } + return true; +} + +static const struct cpu_feature_set *get_current_cpu_features(void) +{ + static struct cpu_feature_set set; + static bool initialized; + char *line =3D NULL; + size_t len =3D 0; + FILE *fp; + int err; + + if (initialized) + return &set; + + initialized =3D true; + fp =3D fopen("/proc/cpuinfo", "r"); + if (!fp) + return &set; + + while (getline(&line, &len, fp) !=3D -1) { + char *p =3D line, *colon, *tok; + + while (*p && isspace(*p)) + p++; + if (!str_has_pfx(p, "flags") && + !str_has_pfx(p, "Features") && + !str_has_pfx(p, "features")) + continue; + + colon =3D strchr(p, ':'); + if (!colon) + continue; + + for (tok =3D strtok(colon + 1, " \t\n"); tok; tok =3D strtok(NULL, " \t\= n")) { + err =3D cpu_feature_set_add(&set, tok); + if (err) { + PRINT_FAIL("failed to parse cpu feature from '/proc/cpuinfo': '%s'", + tok); + break; + } + } + } + + free(line); + fclose(fp); + return &set; +} + +static int parse_cpu_feature(const char *name, struct cpu_feature_set *set) +{ + return cpu_feature_set_add(set, name); +} + /* Uses btf_decl_tag attributes to describe the expected test * behavior, see bpf_misc.h for detailed description of each attribute * and attribute combinations. @@ -650,9 +784,20 @@ static int parse_test_spec(struct test_loader *tester, err =3D -EINVAL; goto cleanup; } + } else if (str_has_pfx(s, TEST_TAG_CPU_FEATURE_PFX)) { + val =3D s + sizeof(TEST_TAG_CPU_FEATURE_PFX) - 1; + err =3D parse_cpu_feature(val, &spec->cpu_features); + if (err) + goto cleanup; } } =20 + if (spec->cpu_features.cnt && __builtin_popcount(arch_mask) !=3D 1) { + PRINT_FAIL("__cpu_feature requires exactly one __arch_* tag"); + err =3D -EINVAL; + goto cleanup; + } + spec->arch_mask =3D arch_mask ?: -1; spec->load_mask =3D load_mask ?: (JITED | NO_JITED); =20 @@ -1161,6 +1306,11 @@ void run_subtest(struct test_loader *tester, return; } =20 + if (!cpu_feature_set_includes(get_current_cpu_features(), &spec->cpu_feat= ures)) { + test__skip(); + return; + } + if (unpriv) { if (!can_execute_unpriv(tester, spec)) { test__skip(); --=20 2.52.0 From nobody Sun Apr 5 13:11:11 2026 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C913333FE26 for ; Thu, 19 Feb 2026 14:32:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511551; cv=none; b=ThCx06j8mHU75lW4Fa+751oBmmoniVOlmGXLmN0fZ4LpyG0Jdg1eUysSIDL4ta2VB/UycbQncRTkU0rfv1qh9zP7ug6ylBzwnzCwjycLbqbjsOSWEoYN7RJYsqUo8ChSQZOEckWZqK5qvEMm5GgzQwrIn3veKlM6XMq2xnM4o3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771511551; c=relaxed/simple; bh=Rwu0/Q5O6sDify4vtHE7EgDx/HdSGFfyoKuhTksF7Ao=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pqhQGvMAqMbYasecfzcxTy7Aql+XG8CgCxoGO0RWZZEWXb9W7m64AfHvlDwk3I6Dll4XHPRcKPJH+USrUV0eRYvEdp6xA+sPmN0Qx0EhZVsGdCHI5wBCrkxwrZxOeVSuLzhXUSO1im22NIRYqcO3xCrZnJSGLZuMFAYfxZaj8Lc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=sr5M0+63; arc=none smtp.client-ip=95.215.58.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="sr5M0+63" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1771511547; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q2bRCV7yDeBf8GuOUBEvMnX2Y0xlNieUH03bOqMBKOs=; b=sr5M0+63Wsz/ZZW1ycof/LgoMgshho1gsH8aRzB8qzGyjBs7rbmGy5s3WQDYQs8NGiSTHH mq0TRXvTdlSZkV6BWgnSjsF3wTVpux4bX3ZZf61jgHFh7IQrelrkdZ8gq8t86l0TpmxOZm hR78dUiXmERnYMyYvwn87pHpaqddEYo= From: Leon Hwang To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Puranjay Mohan , Xu Kuohai , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Shuah Khan , Leon Hwang , Peilin Ye , Luis Gerhorst , Viktor Malik , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-patches-bot@fb.com Subject: [PATCH bpf-next v2 6/6] selftests/bpf: Add JIT disassembly tests for 64-bit bitops kfuncs Date: Thu, 19 Feb 2026 22:29:28 +0800 Message-ID: <20260219142933.13904-7-leon.hwang@linux.dev> In-Reply-To: <20260219142933.13904-1-leon.hwang@linux.dev> References: <20260219142933.13904-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Content-Type: text/plain; charset="utf-8" Add bitops_jit selftests that verify JITed instruction sequences for supported 64-bit bitops kfuncs on x86_64 and arm64, including CPU-feature-gated coverage on x86 where required. Signed-off-by: Leon Hwang --- .../testing/selftests/bpf/prog_tests/bitops.c | 6 + .../testing/selftests/bpf/progs/bitops_jit.c | 153 ++++++++++++++++++ 2 files changed, 159 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/bitops_jit.c diff --git a/tools/testing/selftests/bpf/prog_tests/bitops.c b/tools/testin= g/selftests/bpf/prog_tests/bitops.c index 9acc3cb1908c..2c203904880d 100644 --- a/tools/testing/selftests/bpf/prog_tests/bitops.c +++ b/tools/testing/selftests/bpf/prog_tests/bitops.c @@ -2,6 +2,7 @@ =20 #include #include "bitops.skel.h" +#include "bitops_jit.skel.h" =20 struct bitops_case { __u64 x; @@ -180,3 +181,8 @@ void test_bitops(void) if (test__start_subtest("ror64")) test_ror64(); } + +void test_bitops_jit(void) +{ + RUN_TESTS(bitops_jit); +} diff --git a/tools/testing/selftests/bpf/progs/bitops_jit.c b/tools/testing= /selftests/bpf/progs/bitops_jit.c new file mode 100644 index 000000000000..9f414e56b1e8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/bitops_jit.c @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_experimental.h" +#include "bpf_misc.h" + +SEC("syscall") +__description("bitops jit: clz64 uses lzcnt on x86 with abm") +__success __retval(63) +__arch_x86_64 +__cpu_feature("abm") +__jited(" lzcnt{{.*}}") +int bitops_jit_clz64_x86(void *ctx) +{ + return bpf_clz64(1); +} + +SEC("syscall") +__description("bitops jit: ctz64 uses tzcnt on x86 with bmi1") +__success __retval(4) +__arch_x86_64 +__cpu_feature("bmi1") +__jited(" tzcnt{{.*}}") +int bitops_jit_ctz64_x86(void *ctx) +{ + return bpf_ctz64(0x10); +} + +SEC("syscall") +__description("bitops jit: ffs64 uses tzcnt on x86 with bmi1") +__success __retval(5) +__arch_x86_64 +__cpu_feature("bmi1") +__jited(" tzcnt{{.*}}") +int bitops_jit_ffs64_x86(void *ctx) +{ + return bpf_ffs64(0x10); +} + +SEC("syscall") +__description("bitops jit: fls64 uses lzcnt on x86 with abm") +__success __retval(5) +__arch_x86_64 +__cpu_feature("abm") +__jited(" lzcnt{{.*}}") +int bitops_jit_fls64_x86(void *ctx) +{ + return bpf_fls64(0x10); +} + +SEC("syscall") +__description("bitops jit: popcnt64 uses popcnt on x86") +__success __retval(3) +__arch_x86_64 +__cpu_feature("popcnt") +__jited(" popcnt{{.*}}") +int bitops_jit_popcnt64_x86(void *ctx) +{ + return bpf_popcnt64(0x1011); +} + +SEC("syscall") +__description("bitops jit: rol64 uses rol on x86") +__success __retval(6) +__arch_x86_64 +__jited(" rol{{.*}}") +int bitops_jit_rol64_x86(void *ctx) +{ + return bpf_rol64(3, 1); +} + +SEC("syscall") +__description("bitops jit: ror64 uses ror on x86") +__success __retval(3) +__arch_x86_64 +__jited(" ror{{.*}}") +int bitops_jit_ror64_x86(void *ctx) +{ + return bpf_ror64(6, 1); +} + +SEC("syscall") +__description("bitops jit: clz64 uses clz on arm64") +__success __retval(63) +__arch_arm64 +__jited(" clz {{.*}}") +int bitops_jit_clz64_arm64(void *ctx) +{ + return bpf_clz64(1); +} + +SEC("syscall") +__description("bitops jit: ctz64 uses ctz or rbit+clz on arm64") +__success __retval(4) +__arch_arm64 +__jited(" {{(ctz|rbit)}} {{.*}}") +int bitops_jit_ctz64_arm64(void *ctx) +{ + return bpf_ctz64(0x10); +} + +SEC("syscall") +__description("bitops jit: ffs64 uses ctz or rbit+clz on arm64") +__success __retval(5) +__arch_arm64 +__jited(" {{(ctz|rbit)}} {{.*}}") +int bitops_jit_ffs64_arm64(void *ctx) +{ + return bpf_ffs64(0x10); +} + +SEC("syscall") +__description("bitops jit: fls64 uses clz on arm64") +__success __retval(5) +__arch_arm64 +__jited(" clz {{.*}}") +int bitops_jit_fls64_arm64(void *ctx) +{ + return bpf_fls64(0x10); +} + +SEC("syscall") +__description("bitops jit: bitrev64 uses rbit on arm64") +__success __retval(1) +__arch_arm64 +__jited(" rbit {{.*}}") +int bitops_jit_bitrev64_arm64(void *ctx) +{ + return bpf_bitrev64(0x8000000000000000ULL); +} + +SEC("syscall") +__description("bitops jit: rol64 uses rorv on arm64") +__success __retval(6) +__arch_arm64 +__jited(" ror {{.*}}") +int bitops_jit_rol64_arm64(void *ctx) +{ + return bpf_rol64(3, 1); +} + +SEC("syscall") +__description("bitops jit: ror64 uses rorv on arm64") +__success __retval(3) +__arch_arm64 +__jited(" ror {{.*}}") +int bitops_jit_ror64_arm64(void *ctx) +{ + return bpf_ror64(6, 1); +} + +char _license[] SEC("license") =3D "GPL"; --=20 2.52.0