From nobody Fri Oct 3 14:34:23 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4E27334712; Fri, 29 Aug 2025 16:52:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486372; cv=none; b=qP5M/cQ1683fntrLBE9jNpOridQGg6Dlx/xQXUQpswnEf5uu6szB+80LBsPLPQq/06P23FyTBpb31mpZv9yARvcl17DhI08ei7Io44x90HOTL/7Ue3hS6sTK4u74ZYXXbygcAv2y9tUzK9y0Lri40wuCXFEQKNA8+SVCh8a5Ieo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486372; c=relaxed/simple; bh=YgWTc9RaRISYPKcUGGhk5+COXT2/F7WsMRbU5T5/LfY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aG9tbHi+RB+MJGALMYs/UBbxZfCBeUsOGOVrqG+EQPBFEgEYiutP/8hb73nMMv2J9Walg+U0BaGZQmZAi4lD2nTge7TKoftl1S/RWQp6qKdWHEZnAwbNCZwHmGTRsWRmCY8y6FN9Msf6bviJEOwwSeoYPXXCiJ1No6yHpw1IxUc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=YnNkMfKF; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="YnNkMfKF" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57TG5UT0012769; Fri, 29 Aug 2025 16:52:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=sog56zDGLDmOUB1u4 YLxO3l9gHEU+E7YRqDDNZnW0mc=; b=YnNkMfKFNmvPDdUBk5YIXSZLR1Qwq9AfN zHRMnzdk2fnnXGPscPV2/hJn2lVlhQbwP0pGWaG7WWw5bfOKxOyce0XaCXdOJbWR 1kMQo6JkSH6xORzkM85WsX1aj5L0/NmMPGwIJkptlgFyzLD/+sCGMOkj6bTlSWbj vtps+bWIEGpQ2AOx1tniEBgaRZamKd5F/jDK5VsFfe3bhCABXW2rU/ejmW9DAzIY hvzssAWdubq7yzdRroSc7JvwtKk0Y2CRIgDdwEULYncqzcP0o32jnFV2KNMtEcx/ FKZ0McRqNSkP9/RUeK9+0J98DeWnubRsKBhoulhdFeK7yvTiDgqJQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5aw0qj0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:14 +0000 (GMT) Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57TGmj8K028213; Fri, 29 Aug 2025 16:52:13 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5aw0qhp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:13 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57TGLoO1029982; Fri, 29 Aug 2025 16:52:12 GMT Received: from smtprelay07.fra02v.mail.ibm.com ([9.218.2.229]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 48qsfn2m1h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:11 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay07.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57TGq8kQ39584250 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Aug 2025 16:52:08 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E01432004D; Fri, 29 Aug 2025 16:52:07 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1BC5120067; Fri, 29 Aug 2025 16:51:56 +0000 (GMT) Received: from li-621bac4c-27c7-11b2-a85c-c2bf7c4b3c07.ibm.com.com (unknown [9.43.115.92]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 29 Aug 2025 16:51:55 +0000 (GMT) From: Saket Kumar Bhaskar To: bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Cc: hbathini@linux.ibm.com, sachinpb@linux.ibm.com, venkat88@linux.ibm.com, andrii@kernel.org, eddyz87@gmail.com, mykolal@fb.com, ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, christophe.leroy@csgroup.eu, naveen@kernel.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, memxor@gmail.com, iii@linux.ibm.com, shuah@kernel.org Subject: [PATCH bpf-next v2 1/5] powerpc64/bpf: Implement PROBE_MEM32 pseudo instructions Date: Fri, 29 Aug 2025 22:21:31 +0530 Message-ID: <20250829165135.1273071-2-skb99@linux.ibm.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250829165135.1273071-1-skb99@linux.ibm.com> References: <20250829165135.1273071-1-skb99@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: N_Xe4opC0_p-Si8z8jZ05S0_L5bjNHsq X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODIzMDAyMSBTYWx0ZWRfXyH87Mp+rqttH z5MMvV7nfp835ci4iWMVRhBB3PylFh3GxPqWTo3IgujL5ZmIPPB9w6Ws0XCD3Pr+Vc9sryf16sp kKbe3eCQP9n0Y7hTmPVp+cW7rUxUWmJqZfrq6urlGwv5jKkBFfjnjcxnGzoa/EUS/j9K8AiF8Sz PkYxJIOeC2d2T/GQPRtLbgMoQSXFlWn3LNR9pTANOy46r6i+qDJ1VoYn1OZAWMet9ZFYM3ZdK71 lqfZcOuUiE3YAHhQOoi5hFka5X2JXH5ahUTglLKzslCDgbIIL0yzqyClTdqx9/JvvexJOg3uOen Z1PrS0sHge6kfsuKUnth5nTRnuylE+AtHVEOk6s8ofAsModiEC7SCgnVSS0CiNGMjHsD7llWc3f LvJqFrI+ X-Proofpoint-ORIG-GUID: 5xilHA6ckvZDHplRH7G1EM-VmscAQgxw X-Authority-Analysis: v=2.4 cv=SNNCVPvH c=1 sm=1 tr=0 ts=68b1dabe cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=2OwXVqhp2XgA:10 a=VnNF1IyMAAAA:8 a=CBn9msP3JWrqCmtcpI4A:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-29_06,2025-08-28_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 bulkscore=0 phishscore=0 clxscore=1015 impostorscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508230021 Content-Type: text/plain; charset="utf-8" Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences: - PROBE_MEM32 supports store. - PROBE_MEM32 relies on the verifier to clear upper 32-bit of the src/dst register - PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in _R26 in the prologue). Due to bpf_arena constructions such _R26 + reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time. - PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed. To support these on powerpc, we do tmp1 =3D _R26 + src/dst reg and then use tmp1 as the new src/dst register. This allows us to reuse most of the code for normal [LDX | STX | ST]. Additionally, bpf_jit_emit_probe_mem_store() is introduced to emit instructions for storing memory values depending on the size (byte, halfword, word, doubleword). Stack layout is adjusted to introduce a new NVR (_R26) and to make BPF_PPC_STACKFRAME quadword aligned (local_tmp_var is increased by 8 bytes). Signed-off-by: Saket Kumar Bhaskar Reviewed-by: Hari Bathini Tested-by: Venkat Rao Bagalkote --- arch/powerpc/net/bpf_jit.h | 5 +- arch/powerpc/net/bpf_jit_comp.c | 10 +- arch/powerpc/net/bpf_jit_comp32.c | 2 +- arch/powerpc/net/bpf_jit_comp64.c | 162 ++++++++++++++++++++++++++---- 4 files changed, 155 insertions(+), 24 deletions(-) diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h index 4c26912c2e3c..2d095a873305 100644 --- a/arch/powerpc/net/bpf_jit.h +++ b/arch/powerpc/net/bpf_jit.h @@ -161,9 +161,10 @@ struct codegen_context { unsigned int seen; unsigned int idx; unsigned int stack_size; - int b2p[MAX_BPF_JIT_REG + 2]; + int b2p[MAX_BPF_JIT_REG + 3]; unsigned int exentry_idx; unsigned int alt_exit_addr; + u64 arena_vm_start; }; =20 #define bpf_to_ppc(r) (ctx->b2p[r]) @@ -201,7 +202,7 @@ int bpf_jit_emit_exit_insn(u32 *image, struct codegen_c= ontext *ctx, int tmp_reg, =20 int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, in= t pass, struct codegen_context *ctx, int insn_idx, - int jmp_off, int dst_reg); + int jmp_off, int dst_reg, u32 code); =20 #endif =20 diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_com= p.c index c0684733e9d6..7d070232159f 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -204,6 +204,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *f= p) =20 /* Make sure that the stack is quadword aligned. */ cgctx.stack_size =3D round_up(fp->aux->stack_depth, 16); + cgctx.arena_vm_start =3D bpf_arena_get_kern_vm_start(fp->aux->arena); =20 /* Scouting faux-generate pass 0 */ if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { @@ -326,7 +327,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *f= p) */ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, in= t pass, struct codegen_context *ctx, int insn_idx, int jmp_off, - int dst_reg) + int dst_reg, u32 code) { off_t offset; unsigned long pc; @@ -355,6 +356,9 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *ima= ge, u32 *fimage, int pass (ctx->exentry_idx * BPF_FIXUP_LEN * 4); =20 fixup[0] =3D PPC_RAW_LI(dst_reg, 0); + if (BPF_CLASS(code) =3D=3D BPF_ST || BPF_CLASS(code) =3D=3D BPF_STX) + fixup[0] =3D PPC_RAW_NOP(); + if (IS_ENABLED(CONFIG_PPC32)) fixup[1] =3D PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register= too */ =20 @@ -579,7 +583,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *ima= ge, struct codegen_contex { if (IS_ENABLED(CONFIG_PPC64)) { /* See bpf_jit_stack_tailcallcnt() */ - int tailcallcnt_offset =3D 6 * 8; + int tailcallcnt_offset =3D 7 * 8; =20 EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset)); EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset)); @@ -594,7 +598,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *i= mage, struct codegen_cont { if (IS_ENABLED(CONFIG_PPC64)) { /* See bpf_jit_stack_tailcallcnt() */ - int tailcallcnt_offset =3D 6 * 8; + int tailcallcnt_offset =3D 7 * 8; =20 EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset)); EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset)); diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_c= omp32.c index 0aace304dfe1..3087e744fb25 100644 --- a/arch/powerpc/net/bpf_jit_comp32.c +++ b/arch/powerpc/net/bpf_jit_comp32.c @@ -1087,7 +1087,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *imag= e, u32 *fimage, struct code } =20 ret =3D bpf_add_extable_entry(fp, image, fimage, pass, ctx, insn_idx, - jmp_off, dst_reg); + jmp_off, dst_reg, code); if (ret) return ret; } diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_c= omp64.c index 025524378443..569619f1b31c 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -25,18 +25,18 @@ * with our redzone usage. * * [ prev sp ] <------------- - * [ nv gpr save area ] 5*8 | + * [ nv gpr save area ] 6*8 | * [ tail_call_cnt ] 8 | - * [ local_tmp_var ] 16 | + * [ local_tmp_var ] 24 | * fp (r31) --> [ ebpf stack space ] upto 512 | * [ frame header ] 32/112 | * sp (r1) ---> [ stack pointer ] -------------- */ =20 /* for gpr non volatile registers BPG_REG_6 to 10 */ -#define BPF_PPC_STACK_SAVE (5*8) +#define BPF_PPC_STACK_SAVE (6*8) /* for bpf JIT code internal usage */ -#define BPF_PPC_STACK_LOCALS 24 +#define BPF_PPC_STACK_LOCALS 32 /* stack frame excluding BPF stack, ensure this is quadword aligned */ #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE) @@ -44,6 +44,7 @@ /* BPF register usage */ #define TMP_REG_1 (MAX_BPF_JIT_REG + 0) #define TMP_REG_2 (MAX_BPF_JIT_REG + 1) +#define ARENA_VM_START (MAX_BPF_JIT_REG + 2) =20 /* BPF to ppc register mappings */ void bpf_jit_init_reg_mapping(struct codegen_context *ctx) @@ -67,10 +68,12 @@ void bpf_jit_init_reg_mapping(struct codegen_context *c= tx) ctx->b2p[BPF_REG_AX] =3D _R12; ctx->b2p[TMP_REG_1] =3D _R9; ctx->b2p[TMP_REG_2] =3D _R10; + /* non volatile register for kern_vm_start address */ + ctx->b2p[ARENA_VM_START] =3D _R26; } =20 -/* PPC NVR range -- update this if we ever use NVRs below r27 */ -#define BPF_PPC_NVR_MIN _R27 +/* PPC NVR range -- update this if we ever use NVRs below r26 */ +#define BPF_PPC_NVR_MIN _R26 =20 static inline bool bpf_has_stack_frame(struct codegen_context *ctx) { @@ -89,9 +92,9 @@ static inline bool bpf_has_stack_frame(struct codegen_con= text *ctx) * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- - * [ nv gpr save area ] 5*8 + * [ nv gpr save area ] 6*8 * [ tail_call_cnt ] 8 - * [ local_tmp_var ] 16 + * [ local_tmp_var ] 24 * [ unused red zone ] 224 */ static int bpf_jit_stack_local(struct codegen_context *ctx) @@ -99,12 +102,12 @@ static int bpf_jit_stack_local(struct codegen_context = *ctx) if (bpf_has_stack_frame(ctx)) return STACK_FRAME_MIN_SIZE + ctx->stack_size; else - return -(BPF_PPC_STACK_SAVE + 24); + return -(BPF_PPC_STACK_SAVE + 32); } =20 static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx) { - return bpf_jit_stack_local(ctx) + 16; + return bpf_jit_stack_local(ctx) + 24; } =20 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg) @@ -170,10 +173,17 @@ void bpf_jit_build_prologue(u32 *image, struct codege= n_context *ctx) if (bpf_is_seen_register(ctx, bpf_to_ppc(i))) EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to= _ppc(i)))); =20 + if (ctx->arena_vm_start) + EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1, + bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START)))); + /* Setup frame pointer to point to the bpf stack area */ if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP))) EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1, STACK_FRAME_MIN_SIZE + ctx->stack_size)); + + if (ctx->arena_vm_start) + PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start); } =20 static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_contex= t *ctx) @@ -185,6 +195,10 @@ static void bpf_jit_emit_common_epilogue(u32 *image, s= truct codegen_context *ctx if (bpf_is_seen_register(ctx, bpf_to_ppc(i))) EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_= ppc(i)))); =20 + if (ctx->arena_vm_start) + EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1, + bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START)))); + /* Tear down our stack frame */ if (bpf_has_stack_frame(ctx)) { EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size)); @@ -396,11 +410,11 @@ void bpf_stf_barrier(void); asm ( " .global bpf_stf_barrier ;" " bpf_stf_barrier: ;" -" std 21,-64(1) ;" -" std 22,-56(1) ;" +" std 21,-80(1) ;" +" std 22,-72(1) ;" " sync ;" -" ld 21,-64(1) ;" -" ld 22,-56(1) ;" +" ld 21,-80(1) ;" +" ld 22,-72(1) ;" " ori 31,31,0 ;" " .rept 14 ;" " b 1f ;" @@ -409,6 +423,36 @@ asm ( " blr ;" ); =20 +static int bpf_jit_emit_probe_mem_store(struct codegen_context *ctx, u32 s= rc_reg, s16 off, + u32 code, u32 *image) +{ + u32 tmp1_reg =3D bpf_to_ppc(TMP_REG_1); + u32 tmp2_reg =3D bpf_to_ppc(TMP_REG_2); + + switch (BPF_SIZE(code)) { + case BPF_B: + EMIT(PPC_RAW_STB(src_reg, tmp1_reg, off)); + break; + case BPF_H: + EMIT(PPC_RAW_STH(src_reg, tmp1_reg, off)); + break; + case BPF_W: + EMIT(PPC_RAW_STW(src_reg, tmp1_reg, off)); + break; + case BPF_DW: + if (off % 4) { + EMIT(PPC_RAW_LI(tmp2_reg, off)); + EMIT(PPC_RAW_STDX(src_reg, tmp1_reg, tmp2_reg)); + } else { + EMIT(PPC_RAW_STD(src_reg, tmp1_reg, off)); + } + break; + default: + return -EINVAL; + } + return 0; +} + static int emit_atomic_ld_st(const struct bpf_insn insn, struct codegen_co= ntext *ctx, u32 *image) { u32 code =3D insn.code; @@ -960,6 +1004,50 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *imag= e, u32 *fimage, struct code } break; =20 + case BPF_STX | BPF_PROBE_MEM32 | BPF_B: + case BPF_STX | BPF_PROBE_MEM32 | BPF_H: + case BPF_STX | BPF_PROBE_MEM32 | BPF_W: + case BPF_STX | BPF_PROBE_MEM32 | BPF_DW: + + EMIT(PPC_RAW_ADD(tmp1_reg, dst_reg, bpf_to_ppc(ARENA_VM_START))); + + ret =3D bpf_jit_emit_probe_mem_store(ctx, src_reg, off, code, image); + if (ret) + return ret; + + ret =3D bpf_add_extable_entry(fp, image, fimage, pass, ctx, + ctx->idx - 1, 4, -1, code); + if (ret) + return ret; + + break; + + case BPF_ST | BPF_PROBE_MEM32 | BPF_B: + case BPF_ST | BPF_PROBE_MEM32 | BPF_H: + case BPF_ST | BPF_PROBE_MEM32 | BPF_W: + case BPF_ST | BPF_PROBE_MEM32 | BPF_DW: + + EMIT(PPC_RAW_ADD(tmp1_reg, dst_reg, bpf_to_ppc(ARENA_VM_START))); + + if (BPF_SIZE(code) =3D=3D BPF_W || BPF_SIZE(code) =3D=3D BPF_DW) { + PPC_LI32(tmp2_reg, imm); + src_reg =3D tmp2_reg; + } else { + EMIT(PPC_RAW_LI(tmp2_reg, imm)); + src_reg =3D tmp2_reg; + } + + ret =3D bpf_jit_emit_probe_mem_store(ctx, src_reg, off, code, image); + if (ret) + return ret; + + ret =3D bpf_add_extable_entry(fp, image, fimage, pass, ctx, + ctx->idx - 1, 4, -1, code); + if (ret) + return ret; + + break; + /* * BPF_STX ATOMIC (atomic ops) */ @@ -1112,9 +1200,10 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *ima= ge, u32 *fimage, struct code * Check if 'off' is word aligned for BPF_DW, because * we might generate two instructions. */ - if ((BPF_SIZE(code) =3D=3D BPF_DW || - (BPF_SIZE(code) =3D=3D BPF_B && BPF_MODE(code) =3D=3D BPF_PROBE_ME= MSX)) && - (off & 3)) + if ((BPF_SIZE(code) =3D=3D BPF_DW && (off & 3)) || + (BPF_SIZE(code) =3D=3D BPF_B && + BPF_MODE(code) =3D=3D BPF_PROBE_MEMSX) || + (BPF_SIZE(code) =3D=3D BPF_B && BPF_MODE(code) =3D=3D BPF_MEMSX)) PPC_JMP((ctx->idx + 3) * 4); else PPC_JMP((ctx->idx + 2) * 4); @@ -1160,12 +1249,49 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *im= age, u32 *fimage, struct code =20 if (BPF_MODE(code) =3D=3D BPF_PROBE_MEM) { ret =3D bpf_add_extable_entry(fp, image, fimage, pass, ctx, - ctx->idx - 1, 4, dst_reg); + ctx->idx - 1, 4, dst_reg, code); if (ret) return ret; } break; =20 + /* dst =3D *(u64 *)(ul) (src + ARENA_VM_START + off) */ + case BPF_LDX | BPF_PROBE_MEM32 | BPF_B: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_H: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_W: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW: + + EMIT(PPC_RAW_ADD(tmp1_reg, src_reg, bpf_to_ppc(ARENA_VM_START))); + + switch (size) { + case BPF_B: + EMIT(PPC_RAW_LBZ(dst_reg, tmp1_reg, off)); + break; + case BPF_H: + EMIT(PPC_RAW_LHZ(dst_reg, tmp1_reg, off)); + break; + case BPF_W: + EMIT(PPC_RAW_LWZ(dst_reg, tmp1_reg, off)); + break; + case BPF_DW: + if (off % 4) { + EMIT(PPC_RAW_LI(tmp2_reg, off)); + EMIT(PPC_RAW_LDX(dst_reg, tmp1_reg, tmp2_reg)); + } else { + EMIT(PPC_RAW_LD(dst_reg, tmp1_reg, off)); + } + break; + } + + if (size !=3D BPF_DW && insn_is_zext(&insn[i + 1])) + addrs[++i] =3D ctx->idx * 4; + + ret =3D bpf_add_extable_entry(fp, image, fimage, pass, ctx, + ctx->idx - 1, 4, dst_reg, code); + if (ret) + return ret; + break; + /* * Doubleword load * 16 byte instruction that uses two 'struct bpf_insn' --=20 2.43.5 From nobody Fri Oct 3 14:34:23 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB3BA33CEBB; Fri, 29 Aug 2025 16:53:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486386; cv=none; b=RkJZ0Z4HIm/EjXmUwPpXNZYe+OKhhTfqxrvUND79iP/1kl5uJjxYp+9nvI2uk5pMdFMI1NuuKjj3bFV6PIL0sUkw6st/VmlLTLxFv28oZR8im2fYWd6b6ANT0OhaaJiowvX5Al08B3KAD8ytif55k6GsxW9DnMt+pOeTyvcDoQA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486386; c=relaxed/simple; bh=aymrPgRy2dwNi45kVDhPv9bbAqpLh7m2A8vs4rcVLzQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MjTm1SXFHL4xjK7wqmvfqy5IViyeudEZZxvwCB3w5o3wxZRBJBL0sGCjkSv3xqno0qZi0X6CP+WbaXkm8LOrEekKKnW3HP2b2Jp/nYW6K1fy/PKKlJSyQlJTTySJBCdDY9rHhS40HRD3i81KyZ+vAuI2mJWCxHfsTiWuLXKWu6A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=SLXSBEW8; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="SLXSBEW8" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57T8HOic012917; Fri, 29 Aug 2025 16:52:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=wnsvaFFQ+aUnebRO0 K1Uqp8CvmGlNhUbsRzS3VrExeI=; b=SLXSBEW8QRzawnjKz7cQ8SgL2p9ngnQ2W 8oQP1c3GG2HinKTrmR+GE0dEAaHD6bQ1DJ/FHGNyVtAzwyI2MfUzuV5eozaeKP1C m7s2HhVpBS9I1TnPqH5RKMSFP444fI7RKZ9+WoNWv3ei6aS7ptDw3qlFSHusu/pc QleTpCMP+HZB8UW4xXponAAwqyxYDiQKDNTjb4ClIy70zAJNngE3eduHtdAucRSr Ztg4MhvJN7xmOGT4mGmkDiikVM/2I1f0+RRKdx+HJf5pvsYOvaRkn5JxLU7lyo8a q+yRcGJ0fgO4LSpS7gxqpgFv6sR8AQshf/5gcgbrZrPtX29bIqpKw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5aw0qjy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:26 +0000 (GMT) Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57TGmj8M028213; Fri, 29 Aug 2025 16:52:25 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5aw0qjw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:25 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57TG9FeP002512; Fri, 29 Aug 2025 16:52:24 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 48qt6mtgep-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:24 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57TGqKFu52298104 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Aug 2025 16:52:20 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 737402004F; Fri, 29 Aug 2025 16:52:20 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AAF6F2004E; Fri, 29 Aug 2025 16:52:08 +0000 (GMT) Received: from li-621bac4c-27c7-11b2-a85c-c2bf7c4b3c07.ibm.com.com (unknown [9.43.115.92]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 29 Aug 2025 16:52:08 +0000 (GMT) From: Saket Kumar Bhaskar To: bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Cc: hbathini@linux.ibm.com, sachinpb@linux.ibm.com, venkat88@linux.ibm.com, andrii@kernel.org, eddyz87@gmail.com, mykolal@fb.com, ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, christophe.leroy@csgroup.eu, naveen@kernel.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, memxor@gmail.com, iii@linux.ibm.com, shuah@kernel.org Subject: [PATCH bpf-next v2 2/5] powerpc64/bpf: Implement bpf_addr_space_cast instruction Date: Fri, 29 Aug 2025 22:21:32 +0530 Message-ID: <20250829165135.1273071-3-skb99@linux.ibm.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250829165135.1273071-1-skb99@linux.ibm.com> References: <20250829165135.1273071-1-skb99@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: dzZZdhqd2D8mE_aD_yJfrmHuiLou2pmQ X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODIzMDAyMSBTYWx0ZWRfX7fbYf2N605rw qRnzikqdP+kwjKFts0oNCrWC92z1xyPo4V+smSLhrG5FGRTPfir9fwSWfZ4QFH61iPEFCi2ygXV YUatwfhenBbCY3SQdwiSYKdtO+FEcygwcVqoxMLLPYsADYmmDV3+ZPD9W+HdJrzisWeIwblJRhX WAeqGMV2WcVzspuWW70mxFcde18yXUezNhQdmHu8X/gNQ4YiXsrWtjqKb1yfqGDlmToOAoZ1k8K 1+Uso/DZmsWqwZ1H3lBRsC+dfArerj81uRTyPCYMk4iuKUqL4tFe/z7j62XHwVCd2Of/Gwzgi3g oPK75BL8EDAcfc7V9/Y7eZSkLr80wWySTqCqMOtn7sPCE6s0XxgarzzvJYHHzS3D860L8VMUZkH l8IC+YYb X-Proofpoint-ORIG-GUID: 7gZXhimECLSHnbRWpBX8gVIyQXG2ljBB X-Authority-Analysis: v=2.4 cv=SNNCVPvH c=1 sm=1 tr=0 ts=68b1daca cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=2OwXVqhp2XgA:10 a=VnNF1IyMAAAA:8 a=xX9FRuLrfezasQXAf7QA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-29_06,2025-08-28_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 bulkscore=0 phishscore=0 clxscore=1015 impostorscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508230021 Content-Type: text/plain; charset="utf-8" LLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=3D0 is reserved as bpf_arena address space. rY =3D addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX =3D wY. rY =3D addr_space_cast(rX, 1, 0) : used to convert a bpf arena pointer to a pointer in the userspace vma. This has to be converted by the JIT. PPC_RAW_RLDICL_DOT, a variant of PPC_RAW_RLDICL is introduced to set condition register as well. Signed-off-by: Saket Kumar Bhaskar Tested-by: Venkat Rao Bagalkote --- arch/powerpc/include/asm/ppc-opcode.h | 1 + arch/powerpc/net/bpf_jit.h | 1 + arch/powerpc/net/bpf_jit_comp.c | 6 ++++++ arch/powerpc/net/bpf_jit_comp64.c | 10 ++++++++++ 4 files changed, 18 insertions(+) diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/a= sm/ppc-opcode.h index 8053b24afc39..55ca49d18319 100644 --- a/arch/powerpc/include/asm/ppc-opcode.h +++ b/arch/powerpc/include/asm/ppc-opcode.h @@ -571,6 +571,7 @@ (0x54000001 | ___PPC_RA(d) | ___PPC_RS(a) | __PPC_SH(i) | __PPC_MB(mb= ) | __PPC_ME(me)) #define PPC_RAW_RLWIMI(d, a, i, mb, me) (0x50000000 | ___PPC_RA(d) | ___PP= C_RS(a) | __PPC_SH(i) | __PPC_MB(mb) | __PPC_ME(me)) #define PPC_RAW_RLDICL(d, a, i, mb) (0x78000000 | ___PPC_RA(d) | ___PP= C_RS(a) | __PPC_SH64(i) | __PPC_MB64(mb)) +#define PPC_RAW_RLDICL_DOT(d, a, i, mb) (0x78000000 | ___PPC_RA(d) | ___PP= C_RS(a) | __PPC_SH64(i) | __PPC_MB64(mb) | 0x1) #define PPC_RAW_RLDICR(d, a, i, me) (0x78000004 | ___PPC_RA(d) | ___PP= C_RS(a) | __PPC_SH64(i) | __PPC_ME64(me)) =20 /* slwi =3D rlwinm Rx, Ry, n, 0, 31-n */ diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h index 2d095a873305..748e30e8b5b4 100644 --- a/arch/powerpc/net/bpf_jit.h +++ b/arch/powerpc/net/bpf_jit.h @@ -165,6 +165,7 @@ struct codegen_context { unsigned int exentry_idx; unsigned int alt_exit_addr; u64 arena_vm_start; + u64 user_vm_start; }; =20 #define bpf_to_ppc(r) (ctx->b2p[r]) diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_com= p.c index 7d070232159f..cfa84cab0a18 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -205,6 +205,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *f= p) /* Make sure that the stack is quadword aligned. */ cgctx.stack_size =3D round_up(fp->aux->stack_depth, 16); cgctx.arena_vm_start =3D bpf_arena_get_kern_vm_start(fp->aux->arena); + cgctx.user_vm_start =3D bpf_arena_get_user_vm_start(fp->aux->arena); =20 /* Scouting faux-generate pass 0 */ if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { @@ -439,6 +440,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } =20 +bool bpf_jit_supports_arena(void) +{ + return IS_ENABLED(CONFIG_PPC64); +} + bool bpf_jit_supports_far_kfunc_call(void) { return IS_ENABLED(CONFIG_PPC64); diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_c= omp64.c index 569619f1b31c..76efb47f02a6 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -812,6 +812,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image= , u32 *fimage, struct code */ case BPF_ALU | BPF_MOV | BPF_X: /* (u32) dst =3D src */ case BPF_ALU64 | BPF_MOV | BPF_X: /* dst =3D src */ + + if (insn_is_cast_user(&insn[i])) { + EMIT(PPC_RAW_RLDICL_DOT(tmp1_reg, src_reg, 0, 32)); + PPC_LI64(dst_reg, (ctx->user_vm_start & 0xffffffff00000000UL)); + PPC_BCC_SHORT(COND_EQ, (ctx->idx + 2) * 4); + EMIT(PPC_RAW_OR(tmp1_reg, dst_reg, tmp1_reg)); + EMIT(PPC_RAW_MR(dst_reg, tmp1_reg)); + break; + } + if (imm =3D=3D 1) { /* special mov32 for zext */ EMIT(PPC_RAW_RLWINM(dst_reg, dst_reg, 0, 0, 31)); --=20 2.43.5 From nobody Fri Oct 3 14:34:23 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5138433471C; Fri, 29 Aug 2025 16:53:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486395; cv=none; b=FlKjRHnq+2UqFV6rH3xYl4zXKLHwYMgKyY0lqZmOLC9iIcim2PJnl47lneHIaq5/W+i+WVj+JfhsiEque1u9pESPRfJ2BUQvUjor+wwum55Pk3MRzCOtxd2ld1L0XOBLuvNJ+zL7LvgRm02h89xKipiZigZe/uVSewEIyci1d80= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486395; c=relaxed/simple; bh=NPJGyq8ju68f2m2dwsnGzuCyGAgZoAPJUEL27kiZBIE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=stA4XFUjrRfdYY68WT0XqwURvsX09hKUHIZCNmdt84swPO/fz1WfZ36XDlr+V+Bj8BPxzaFTvMgRMs6XEknlVZlsHRNt8zQPk7mpU3+RPYWpbBQdFX0rds1f2Tgq58MYxtxj6Kc2qZylpiswHwF97rk0G71IJy5bCCywexPBpgs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=oOFOxHW6; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="oOFOxHW6" Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57T9IMAb020783; Fri, 29 Aug 2025 16:52:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=US1ykTnydS37YdpxL H/NRYLU0J9OvOWyeqm75mdVxss=; b=oOFOxHW6UQ1VOS/5PxqDQUXlzrM6uGLHW L0Lb9lo4Ow5DENHPcOt63f0C+NJONb4NvmNk+dsFzalohuyGkDIS15w+kx+CDeKP Uaxc66r5WmJdBfxWL35xhHlZrtgxBpZjkj6YehpTIQ9/359jT/t3RSPe70fgv+Se xeTpqJAunt5H7DiF5rDCllsec+LdexgHIgO6ZvZ5jEzHspe8MbhPsDg4XdNyJ3tt 9VcfmBBMOwJCh1BB/SlGGwiqhrY/qzDf+zmbF7IBA0d5cpo6zOIQzEBXuBoY7nMN idxEUKnOTHJsflzPXKzLmYHkSX8lX2uZe8mNDWOwl2hiHmD0ABVQQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5hqge97-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:38 +0000 (GMT) Received: from m0360072.ppops.net (m0360072.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57TGm3kc007296; Fri, 29 Aug 2025 16:52:38 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5hqge93-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:38 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57TDA6Hc017993; Fri, 29 Aug 2025 16:52:37 GMT Received: from smtprelay04.fra02v.mail.ibm.com ([9.218.2.228]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48qtp3tbuc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:37 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay04.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57TGqXAt19267918 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Aug 2025 16:52:33 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 34BBF2004B; Fri, 29 Aug 2025 16:52:33 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4331A20040; Fri, 29 Aug 2025 16:52:21 +0000 (GMT) Received: from li-621bac4c-27c7-11b2-a85c-c2bf7c4b3c07.ibm.com.com (unknown [9.43.115.92]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 29 Aug 2025 16:52:20 +0000 (GMT) From: Saket Kumar Bhaskar To: bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Cc: hbathini@linux.ibm.com, sachinpb@linux.ibm.com, venkat88@linux.ibm.com, andrii@kernel.org, eddyz87@gmail.com, mykolal@fb.com, ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, christophe.leroy@csgroup.eu, naveen@kernel.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, memxor@gmail.com, iii@linux.ibm.com, shuah@kernel.org Subject: [PATCH bpf-next v2 3/5] powerpc64/bpf: Introduce bpf_jit_emit_atomic_ops() to emit atomic instructions Date: Fri, 29 Aug 2025 22:21:33 +0530 Message-ID: <20250829165135.1273071-4-skb99@linux.ibm.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250829165135.1273071-1-skb99@linux.ibm.com> References: <20250829165135.1273071-1-skb99@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODIzMDAyMSBTYWx0ZWRfX2ILk7wBD7g+Y GslsxRI+rLEozL5ucqE3avuo3YRYQJHulr2KjmsyFrUZlK1ta2LvRvMBiepZAosQMObKNZi3ly4 MeD8Ii9j7hl5Ya8LgiUzXNzlQ4jB9SBw9arGihNsPGgnEWhZc56h0HQNRR5varzw1dLwgS9iMcy MExRYjQHkCV9twhPyvmLGzHEfUV49mp5SdgaFTnno/Cqj06DsnLD5erVVyDMD8HgckHV9CS9bs0 BzK+z2TDB9K9d8aXxOVWx+6cYLSAcyx7ynOXBmq+fuj4GGLesNilky5cZkT7OFamM4Dq++zMzvw 9gqajM9xyrK4eWqJD7Dqn+az3UDf+At/rWCqmY8O4afCOp25wPrFdAD0Xznne+268lQPg4B/Mrl St2p5CB5 X-Proofpoint-ORIG-GUID: BKV-v_1sfawB60GC5GYuAb0XnX0OfXwN X-Proofpoint-GUID: 3hMKP4GSNXj3in_mbOPTI-Jg1D73uiw9 X-Authority-Analysis: v=2.4 cv=Ndbm13D4 c=1 sm=1 tr=0 ts=68b1dad7 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=VnNF1IyMAAAA:8 a=mLi99PhJ1kaOFlELNpEA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-29_06,2025-08-28_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 impostorscore=0 clxscore=1015 phishscore=0 malwarescore=0 suspectscore=0 bulkscore=0 spamscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508230021 Content-Type: text/plain; charset="utf-8" The existing code for emitting bpf atomic instruction sequences for atomic operations such as XCHG, CMPXCHG, ADD, AND, OR, and XOR has been refactored into a reusable function, bpf_jit_emit_ppc_atomic_op(). It also computes the jump offset and tracks the instruction index for jited LDARX/LWARX to be used in case it causes a fault. Signed-off-by: Saket Kumar Bhaskar Tested-by: Venkat Rao Bagalkote --- arch/powerpc/net/bpf_jit_comp64.c | 203 +++++++++++++++++------------- 1 file changed, 115 insertions(+), 88 deletions(-) diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_c= omp64.c index 76efb47f02a6..cb4d1e954961 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -423,6 +423,111 @@ asm ( " blr ;" ); =20 +static int bpf_jit_emit_atomic_ops(u32 *image, struct codegen_context *ctx, + const struct bpf_insn *insn, u32 *jmp_off, + u32 *tmp_idx, u32 *addrp) +{ + u32 tmp1_reg =3D bpf_to_ppc(TMP_REG_1); + u32 tmp2_reg =3D bpf_to_ppc(TMP_REG_2); + u32 size =3D BPF_SIZE(insn->code); + u32 src_reg =3D bpf_to_ppc(insn->src_reg); + u32 dst_reg =3D bpf_to_ppc(insn->dst_reg); + s32 imm =3D insn->imm; + + u32 save_reg =3D tmp2_reg; + u32 ret_reg =3D src_reg; + u32 fixup_idx; + + /* Get offset into TMP_REG_1 */ + EMIT(PPC_RAW_LI(tmp1_reg, insn->off)); + /* + * Enforce full ordering for operations with BPF_FETCH by emitting a 'sync' + * before and after the operation. + * + * This is a requirement in the Linux Kernel Memory Model. + * See __cmpxchg_u64() in asm/cmpxchg.h as an example. + */ + if ((imm & BPF_FETCH) && IS_ENABLED(CONFIG_SMP)) + EMIT(PPC_RAW_SYNC()); + + *tmp_idx =3D ctx->idx; + + /* load value from memory into TMP_REG_2 */ + if (size =3D=3D BPF_DW) + EMIT(PPC_RAW_LDARX(tmp2_reg, tmp1_reg, dst_reg, 0)); + else + EMIT(PPC_RAW_LWARX(tmp2_reg, tmp1_reg, dst_reg, 0)); + /* Save old value in _R0 */ + if (imm & BPF_FETCH) + EMIT(PPC_RAW_MR(_R0, tmp2_reg)); + + switch (imm) { + case BPF_ADD: + case BPF_ADD | BPF_FETCH: + EMIT(PPC_RAW_ADD(tmp2_reg, tmp2_reg, src_reg)); + break; + case BPF_AND: + case BPF_AND | BPF_FETCH: + EMIT(PPC_RAW_AND(tmp2_reg, tmp2_reg, src_reg)); + break; + case BPF_OR: + case BPF_OR | BPF_FETCH: + EMIT(PPC_RAW_OR(tmp2_reg, tmp2_reg, src_reg)); + break; + case BPF_XOR: + case BPF_XOR | BPF_FETCH: + EMIT(PPC_RAW_XOR(tmp2_reg, tmp2_reg, src_reg)); + break; + case BPF_CMPXCHG: + /* + * Return old value in BPF_REG_0 for BPF_CMPXCHG & + * in src_reg for other cases. + */ + ret_reg =3D bpf_to_ppc(BPF_REG_0); + + /* Compare with old value in BPF_R0 */ + if (size =3D=3D BPF_DW) + EMIT(PPC_RAW_CMPD(bpf_to_ppc(BPF_REG_0), tmp2_reg)); + else + EMIT(PPC_RAW_CMPW(bpf_to_ppc(BPF_REG_0), tmp2_reg)); + /* Don't set if different from old value */ + PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) * 4); + fallthrough; + case BPF_XCHG: + save_reg =3D src_reg; + break; + default: + return -EOPNOTSUPP; + } + + /* store new value */ + if (size =3D=3D BPF_DW) + EMIT(PPC_RAW_STDCX(save_reg, tmp1_reg, dst_reg)); + else + EMIT(PPC_RAW_STWCX(save_reg, tmp1_reg, dst_reg)); + /* we're done if this succeeded */ + PPC_BCC_SHORT(COND_NE, *tmp_idx * 4); + fixup_idx =3D ctx->idx; + + if (imm & BPF_FETCH) { + /* Emit 'sync' to enforce full ordering */ + if (IS_ENABLED(CONFIG_SMP)) + EMIT(PPC_RAW_SYNC()); + EMIT(PPC_RAW_MR(ret_reg, _R0)); + /* + * Skip unnecessary zero-extension for 32-bit cmpxchg. + * For context, see commit 39491867ace5. + */ + if (size !=3D BPF_DW && imm =3D=3D BPF_CMPXCHG && + insn_is_zext(insn + 1)) + *addrp =3D ctx->idx * 4; + } + + *jmp_off =3D (fixup_idx - *tmp_idx) * 4; + + return 0; +} + static int bpf_jit_emit_probe_mem_store(struct codegen_context *ctx, u32 s= rc_reg, s16 off, u32 code, u32 *image) { @@ -538,7 +643,6 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,= u32 *fimage, struct code u32 size =3D BPF_SIZE(code); u32 tmp1_reg =3D bpf_to_ppc(TMP_REG_1); u32 tmp2_reg =3D bpf_to_ppc(TMP_REG_2); - u32 save_reg, ret_reg; s16 off =3D insn[i].off; s32 imm =3D insn[i].imm; bool func_addr_fixed; @@ -546,6 +650,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,= u32 *fimage, struct code u64 imm64; u32 true_cond; u32 tmp_idx; + u32 jmp_off; =20 /* * addrs[] maps a BPF bytecode address into a real offset from @@ -1080,93 +1185,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *im= age, u32 *fimage, struct code return -EOPNOTSUPP; } =20 - save_reg =3D tmp2_reg; - ret_reg =3D src_reg; - - /* Get offset into TMP_REG_1 */ - EMIT(PPC_RAW_LI(tmp1_reg, off)); - /* - * Enforce full ordering for operations with BPF_FETCH by emitting a 's= ync' - * before and after the operation. - * - * This is a requirement in the Linux Kernel Memory Model. - * See __cmpxchg_u64() in asm/cmpxchg.h as an example. - */ - if ((imm & BPF_FETCH) && IS_ENABLED(CONFIG_SMP)) - EMIT(PPC_RAW_SYNC()); - tmp_idx =3D ctx->idx * 4; - /* load value from memory into TMP_REG_2 */ - if (size =3D=3D BPF_DW) - EMIT(PPC_RAW_LDARX(tmp2_reg, tmp1_reg, dst_reg, 0)); - else - EMIT(PPC_RAW_LWARX(tmp2_reg, tmp1_reg, dst_reg, 0)); - - /* Save old value in _R0 */ - if (imm & BPF_FETCH) - EMIT(PPC_RAW_MR(_R0, tmp2_reg)); - - switch (imm) { - case BPF_ADD: - case BPF_ADD | BPF_FETCH: - EMIT(PPC_RAW_ADD(tmp2_reg, tmp2_reg, src_reg)); - break; - case BPF_AND: - case BPF_AND | BPF_FETCH: - EMIT(PPC_RAW_AND(tmp2_reg, tmp2_reg, src_reg)); - break; - case BPF_OR: - case BPF_OR | BPF_FETCH: - EMIT(PPC_RAW_OR(tmp2_reg, tmp2_reg, src_reg)); - break; - case BPF_XOR: - case BPF_XOR | BPF_FETCH: - EMIT(PPC_RAW_XOR(tmp2_reg, tmp2_reg, src_reg)); - break; - case BPF_CMPXCHG: - /* - * Return old value in BPF_REG_0 for BPF_CMPXCHG & - * in src_reg for other cases. - */ - ret_reg =3D bpf_to_ppc(BPF_REG_0); - - /* Compare with old value in BPF_R0 */ - if (size =3D=3D BPF_DW) - EMIT(PPC_RAW_CMPD(bpf_to_ppc(BPF_REG_0), tmp2_reg)); - else - EMIT(PPC_RAW_CMPW(bpf_to_ppc(BPF_REG_0), tmp2_reg)); - /* Don't set if different from old value */ - PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) * 4); - fallthrough; - case BPF_XCHG: - save_reg =3D src_reg; - break; - default: - pr_err_ratelimited( - "eBPF filter atomic op code %02x (@%d) unsupported\n", - code, i); - return -EOPNOTSUPP; - } - - /* store new value */ - if (size =3D=3D BPF_DW) - EMIT(PPC_RAW_STDCX(save_reg, tmp1_reg, dst_reg)); - else - EMIT(PPC_RAW_STWCX(save_reg, tmp1_reg, dst_reg)); - /* we're done if this succeeded */ - PPC_BCC_SHORT(COND_NE, tmp_idx); - - if (imm & BPF_FETCH) { - /* Emit 'sync' to enforce full ordering */ - if (IS_ENABLED(CONFIG_SMP)) - EMIT(PPC_RAW_SYNC()); - EMIT(PPC_RAW_MR(ret_reg, _R0)); - /* - * Skip unnecessary zero-extension for 32-bit cmpxchg. - * For context, see commit 39491867ace5. - */ - if (size !=3D BPF_DW && imm =3D=3D BPF_CMPXCHG && - insn_is_zext(&insn[i + 1])) - addrs[++i] =3D ctx->idx * 4; + ret =3D bpf_jit_emit_atomic_ops(image, ctx, &insn[i], + &jmp_off, &tmp_idx, &addrs[i + 1]); + if (ret) { + if (ret =3D=3D -EOPNOTSUPP) { + pr_err_ratelimited( + "eBPF filter atomic op code %02x (@%d) unsupported\n", + code, i); + } + return ret; } break; =20 --=20 2.43.5 From nobody Fri Oct 3 14:34:23 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0739879CD; Fri, 29 Aug 2025 16:53:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486406; cv=none; b=JyXpvdbOj1A4Hy60zKqn7SLPMT7U0f6KhGP0cnwGBj2xAfP5uLS3/MJ7luUA7WrbLf4CkREKlqCe/z+C6E3pA7Vgihp4CU/WiPCJuzdLlY6tPmwB5nI059hO625ZUffH9rHFCKU9qEaRTCrmHrqcLju/ubs+cVgvGxqdUtamAlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486406; c=relaxed/simple; bh=Apq2EUTqIm9H2LVUovIpzZr3gu+jmOhkPwR4lUaqO+w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qi8wy5HygsqaCNPyAGmqUMVxkqHe37DH0JJ5qvv4XEUnL6Jx5rV5uGdwpha6KOeLBiFeZGrL/HOEfICJoRV9SG4LZqmp+Jnnu0Cc3tDoqt5yiqjnISo0oKGV8YyUXAu2dJ/MR6ADu/OeNWZtd4nAbGnUSyqqWinPpP1Mzx04o+s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=LEBuRd14; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="LEBuRd14" Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57T9IRrC020847; Fri, 29 Aug 2025 16:52:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=x/jB12xFw5RluKVHq 9ZunPozyBFjcsDnMzg9TwLzyws=; b=LEBuRd14SLWfpwUYLdIeRrjxmb5t9mhiY vMNzihNCcPb8PNsv7bC+xLJodUuaiJTVO/rSbN4pGvex3uWLbsvkZYUh6JLO4Uv4 ms4lVwayx1kEQT9fe3iEsF4ygAnjgDIbK9EV5G5nX820DhONuUpu68mQ+Tn8Vk+1 RLL7f2h/uzdbG2alDYoRiMLbFYlkwJZCNemqHnRWMHZik7DkFFlhj9URwuU5Uirn gekQdF79I6X9Lys5d977JG0KA64F+SFY5sCk8YUyPDfkgcpqQT3YC1gziT42FoMw Ymzz4Xooy6Y5YXeiSzABkineFXQPVtOksIkdOfrs51n5k3RWMcZ+w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5hqgeb8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:51 +0000 (GMT) Received: from m0360072.ppops.net (m0360072.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57TGqoO4017235; Fri, 29 Aug 2025 16:52:51 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48q5hqgeb2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:50 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57TFBaAb002520; Fri, 29 Aug 2025 16:52:50 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 48qryq2q6w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:52:49 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57TGqkO044761472 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Aug 2025 16:52:46 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E308C2004B; Fri, 29 Aug 2025 16:52:45 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 006EB20040; Fri, 29 Aug 2025 16:52:34 +0000 (GMT) Received: from li-621bac4c-27c7-11b2-a85c-c2bf7c4b3c07.ibm.com.com (unknown [9.43.115.92]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 29 Aug 2025 16:52:33 +0000 (GMT) From: Saket Kumar Bhaskar To: bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Cc: hbathini@linux.ibm.com, sachinpb@linux.ibm.com, venkat88@linux.ibm.com, andrii@kernel.org, eddyz87@gmail.com, mykolal@fb.com, ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, christophe.leroy@csgroup.eu, naveen@kernel.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, memxor@gmail.com, iii@linux.ibm.com, shuah@kernel.org Subject: [PATCH bpf-next v2 4/5] powerpc64/bpf: Implement PROBE_ATOMIC instructions Date: Fri, 29 Aug 2025 22:21:34 +0530 Message-ID: <20250829165135.1273071-5-skb99@linux.ibm.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250829165135.1273071-1-skb99@linux.ibm.com> References: <20250829165135.1273071-1-skb99@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODIzMDAyMSBTYWx0ZWRfX30oyvMQx4kR9 4/kjfuXKqzrAgx3E6pCsqZhDNqP97j6Bsrxmqk06TuDbVvueaOXV//W94RUVbZ09pHFuJyZijd0 CT/oW8gN8lxnlJp6zVclUk0dz59sIr/qiYu5/X3FU6D81QQO8MBchuvYp7q4fLIJBVuJ/kY7gXc 59STubPrhOtWsZ2iWdFo7yHhfCQJ0LRkRIUVGhnWuDrRkJwI9egFAL78P6Odj3YNTK6MFWLRRlt QMRgsoVDf1G2/MMcorGTc0vWlwBGdP7PrbqFmaOUHZC6PGqbVtyy8rPh+Fl/ETi6NimhGqOZaEr uQirV3ai4DJk7SiI6Y8i/8H8rDDvnRsOjko6RBturRotoFvOuvSLSJv+ZiMqhzwolKiJDJfM0oc Y2ZuoCXC X-Proofpoint-ORIG-GUID: IBO8xZK8AzIwnId52vpmyAULqXYAFSpr X-Proofpoint-GUID: KlE1hyIQaQDV0pSgVNEV3JuNC_ULirB_ X-Authority-Analysis: v=2.4 cv=Ndbm13D4 c=1 sm=1 tr=0 ts=68b1dae3 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=2OwXVqhp2XgA:10 a=VnNF1IyMAAAA:8 a=I9DJoMOgBESJXZyRRMQA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-29_06,2025-08-28_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 impostorscore=0 clxscore=1015 phishscore=0 malwarescore=0 suspectscore=0 bulkscore=0 spamscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508230021 Content-Type: text/plain; charset="utf-8" powerpc supports BPF atomic operations using a loop around Load-And-Reserve(LDARX/LWARX) and Store-Conditional(STDCX/STWCX) instructions gated by sync instructions to enforce full ordering. To implement arena_atomics, arena vm start address is added to the dst_reg to be used for both the LDARX/LWARX and STDCX/STWCX instructions. Further, an exception table entry is added for LDARX/LWARX instruction to land after the loop on fault. At the end of sequence, dst_reg is restored by subtracting arena vm start address. bpf_jit_supports_insn() is introduced to selectively enable instruction support as in other architectures like x86 and arm64. Signed-off-by: Saket Kumar Bhaskar Tested-by: Venkat Rao Bagalkote --- arch/powerpc/net/bpf_jit_comp.c | 16 ++++++++++++++++ arch/powerpc/net/bpf_jit_comp64.c | 26 ++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_com= p.c index cfa84cab0a18..88ad5ba7b87f 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -450,6 +450,22 @@ bool bpf_jit_supports_far_kfunc_call(void) return IS_ENABLED(CONFIG_PPC64); } =20 +bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) +{ + if (!in_arena) + return true; + switch (insn->code) { + case BPF_STX | BPF_ATOMIC | BPF_H: + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_W: + case BPF_STX | BPF_ATOMIC | BPF_DW: + if (bpf_atomic_is_load_store(insn)) + return false; + return IS_ENABLED(CONFIG_PPC64); + } + return true; +} + void *arch_alloc_bpf_trampoline(unsigned int size) { return bpf_prog_pack_alloc(size, bpf_jit_fill_ill_insns); diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_c= omp64.c index cb4d1e954961..1fe37128c876 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -1163,6 +1163,32 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *ima= ge, u32 *fimage, struct code =20 break; =20 + /* + * BPF_STX PROBE_ATOMIC (arena atomic ops) + */ + case BPF_STX | BPF_PROBE_ATOMIC | BPF_W: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW: + EMIT(PPC_RAW_ADD(dst_reg, dst_reg, bpf_to_ppc(ARENA_VM_START))); + ret =3D bpf_jit_emit_atomic_ops(image, ctx, &insn[i], + &jmp_off, &tmp_idx, &addrs[i + 1]); + if (ret) { + if (ret =3D=3D -EOPNOTSUPP) { + pr_err_ratelimited( + "eBPF filter atomic op code %02x (@%d) unsupported\n", + code, i); + } + return ret; + } + /* LDARX/LWARX should land here on exception. */ + ret =3D bpf_add_extable_entry(fp, image, fimage, pass, ctx, + tmp_idx, jmp_off, dst_reg, code); + if (ret) + return ret; + + /* Retrieve the dst_reg */ + EMIT(PPC_RAW_SUB(dst_reg, dst_reg, bpf_to_ppc(ARENA_VM_START))); + break; + /* * BPF_STX ATOMIC (atomic ops) */ --=20 2.43.5 From nobody Fri Oct 3 14:34:23 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 199B1340D86; Fri, 29 Aug 2025 16:53:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486411; cv=none; b=H62oTxJVroXiJ9U5emVNi9WDFsd1NLw2VNeF8JURLM7DnUe0vMJf0lfAzFLr0JWv091/MoJ3juIp7sf6JJ9uIYCoPBHbMV/sjFCkCUj3OYT3YAqvMToqaiPK9OhoEij7HC86f/1WOh+Ltul8cvztbo+XQm6etDaniiEjbhcy0Yw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756486411; c=relaxed/simple; bh=ulRFzuagZFEOnjiOpDqhKWXsZK1C/pxXB+6+UfqLhE4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FR7ziTAsXqMyD93oWJLZZBcGKVbCIYXmO/lPFxN65JLz24wGhbOhVjvfV/xgEkyWvFoRNnzs4A7jN33Gb/S3+yIA8ficVMNk/e75Jl6kk7cVZbpAZyHXoTLdq9yuK0qcCLjDEGGwR6BJpvFuQyuWa/Wz+ZBmpIvTNB9+ChFgKXs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=bV8oV+uj; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="bV8oV+uj" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57T8HQTC006066; Fri, 29 Aug 2025 16:53:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=wlQ/7c3rr2aJ6RJOQ VKI6Viuq+AoqlPLvHQiVunbhEM=; b=bV8oV+ujkRy7ajVWehnsSa6bmca92t1lh D7aUk8lXfP6+ZvMQydOu5RCGFNT+VpciYnbQIQvtMLbhwvlJlqz3LObzzPLnvqV6 yM67jDWVPePVww3dyKrmrB1F9KIztHWIq1jINMmLb58Rkvqd/TubvTO/NZRfkvqb E90kltUNQkoVT8bqDON33bBDX98yvZ9qvuyzH/gmroHrJ8+3+x/QG5nMn+06bwCZ 00cuAKy1jiYMRmBGGcEaZ/LiBy4dK6/bEUmGN9PSnFo4Z0QANAzd5JdPNtgYW0Gx VmnHQGW0ifZkDL/97YuiU5/NNYnGIX4t0VlyeUTpcnq4jnTY95bJg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48tuaj5dfa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:53:03 +0000 (GMT) Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57TGfafe007210; Fri, 29 Aug 2025 16:53:03 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48tuaj5df5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:53:03 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57TD5eBL017963; Fri, 29 Aug 2025 16:53:02 GMT Received: from smtprelay01.fra02v.mail.ibm.com ([9.218.2.227]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48qtp3tbxg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Aug 2025 16:53:02 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay01.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57TGqwYl50463098 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Aug 2025 16:52:58 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78CE32004F; Fri, 29 Aug 2025 16:52:58 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AD5CB20043; Fri, 29 Aug 2025 16:52:46 +0000 (GMT) Received: from li-621bac4c-27c7-11b2-a85c-c2bf7c4b3c07.ibm.com.com (unknown [9.43.115.92]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 29 Aug 2025 16:52:46 +0000 (GMT) From: Saket Kumar Bhaskar To: bpf@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Cc: hbathini@linux.ibm.com, sachinpb@linux.ibm.com, venkat88@linux.ibm.com, andrii@kernel.org, eddyz87@gmail.com, mykolal@fb.com, ast@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, christophe.leroy@csgroup.eu, naveen@kernel.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, memxor@gmail.com, iii@linux.ibm.com, shuah@kernel.org Subject: [PATCH bpf-next v2 5/5] selftests/bpf: Fix arena_spin_lock selftest failure Date: Fri, 29 Aug 2025 22:21:35 +0530 Message-ID: <20250829165135.1273071-6-skb99@linux.ibm.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250829165135.1273071-1-skb99@linux.ibm.com> References: <20250829165135.1273071-1-skb99@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: lJPbpMxxCuoUV2wEYR8WIRlW-ihiAIKb X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODI4MDE0MCBTYWx0ZWRfX3iokaRGCKrjx v+MT8ILcp1yH1C7IUl0vluo9d9b6WyKiSU2mhtAXLlSr6PmppMNLhUCLceO+gjz2pvw7t2i1b0j 7KUz3ZI2HiXJbek1VP2tyc3ZeAIzqJjmk367RuXavsN/JqzpY1H3HPT+6Sr4d5o1k2h4Ix3T9yl fnDc+chQ2/AC4LfYBow6zqeFvTHW+ZYXpyFAg7Up99+Bn8GHs7Czm36UVZbWA59eDVHjUKG3OL9 iOdPK0GjKAmKsnTIKKhsNckqov82hD/sydcqtTYxXe2GZcr2GfuKL1TuscrHBKJ/YYiqvIjkj20 sJnBzXXu8zZ/B4A0+oK4vJ69GpPnLW7+w3WR1fdEEuCTSyLvgW+mCrfikm53pTgyoxODucKC58u d/Ou1h5N X-Authority-Analysis: v=2.4 cv=YfW95xRf c=1 sm=1 tr=0 ts=68b1daf0 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=VnNF1IyMAAAA:8 a=dOPeAGjUFLo6ntbE7XwA:9 X-Proofpoint-ORIG-GUID: ljkqhl2cCsTRUbADYNfMguYbVHlJ4kQl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-29_06,2025-08-28_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 suspectscore=0 bulkscore=0 impostorscore=0 clxscore=1015 priorityscore=1501 malwarescore=0 phishscore=0 spamscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508280140 Content-Type: text/plain; charset="utf-8" For systems having CONFIG_NR_CPUS set to > 1024 in kernel config the selftest fails as arena_spin_lock_irqsave() returns EOPNOTSUPP. The selftest is skipped incase bpf program returns EOPNOTSUPP, with a descriptive message logged. Signed-off-by: Saket Kumar Bhaskar Tested-by: Venkat Rao Bagalkote --- .../selftests/bpf/prog_tests/arena_spin_lock.c | 13 +++++++++++++ tools/testing/selftests/bpf/progs/arena_spin_lock.c | 5 ++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c b/too= ls/testing/selftests/bpf/prog_tests/arena_spin_lock.c index 0223fce4db2b..1ec1ca987893 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c @@ -40,8 +40,13 @@ static void *spin_lock_thread(void *arg) =20 err =3D bpf_prog_test_run_opts(prog_fd, &topts); ASSERT_OK(err, "test_run err"); + + if (topts.retval =3D=3D -EOPNOTSUPP) + goto end; + ASSERT_EQ((int)topts.retval, 0, "test_run retval"); =20 +end: pthread_exit(arg); } =20 @@ -63,6 +68,7 @@ static void test_arena_spin_lock_size(int size) skel =3D arena_spin_lock__open_and_load(); if (!ASSERT_OK_PTR(skel, "arena_spin_lock__open_and_load")) return; + if (skel->data->test_skip =3D=3D 2) { test__skip(); goto end; @@ -86,6 +92,13 @@ static void test_arena_spin_lock_size(int size) goto end_barrier; } =20 + if (skel->data->test_skip =3D=3D 2) { + printf("%s:SKIP: %d CPUs exceed the maximum supported by arena spinlock\= n", + __func__, get_nprocs()); + test__skip(); + goto end_barrier; + } + ASSERT_EQ(skel->bss->counter, repeat * nthreads, "check counter value"); =20 end_barrier: diff --git a/tools/testing/selftests/bpf/progs/arena_spin_lock.c b/tools/te= sting/selftests/bpf/progs/arena_spin_lock.c index c4500c37f85e..a475b974438e 100644 --- a/tools/testing/selftests/bpf/progs/arena_spin_lock.c +++ b/tools/testing/selftests/bpf/progs/arena_spin_lock.c @@ -37,8 +37,11 @@ int prog(void *ctx) #if defined(ENABLE_ATOMICS_TESTS) && defined(__BPF_FEATURE_ADDR_SPACE_CAST) unsigned long flags; =20 - if ((ret =3D arena_spin_lock_irqsave(&lock, flags))) + if ((ret =3D arena_spin_lock_irqsave(&lock, flags))) { + if (ret =3D=3D -EOPNOTSUPP) + test_skip =3D 2; return ret; + } if (counter !=3D limit) counter++; bpf_repeat(cs_count); --=20 2.43.5