From nobody Mon Feb 9 12:24:17 2026 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B85F30B515 for ; Fri, 14 Nov 2025 09:25:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112337; cv=none; b=t4Ovxe/v2Pk7iANtcutRirHIx2lrlMIFGTprWcIOSCEI11MA34zkU+Q3aSMicKm3g3BpqKfpA4JyRm+2cmvWkDwnzx40owiciW40AHlpe68dFwQ38NdQZu2buBAhsmjMxkVmtfPQC/RjU+81w+wlG/KfEo55xYcJv9NEOt2r/Hw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112337; c=relaxed/simple; bh=TIjsGgErGjhnsODM1Ph/ITY27XoQ96fhmRIiGKpLPac=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mcQlcjoGR9zAtKWPLE7aqX839g46w32kkuOEjh0DgzkDbehLRSBcRKh2+I77IDpgbjSQ7PnBRiR+p2+FzyyN5QdgjH8PZiDa20B/HFjY2yLV+SXldeXmS1wNpL2YSxiejtiDPKRlu5RftZdtQ5Uc8PpheweTIbnxk5CRoRNZ640= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yzd8SLfN; arc=none smtp.client-ip=209.85.214.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yzd8SLfN" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-2956d816c10so19131125ad.1 for ; Fri, 14 Nov 2025 01:25:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112334; x=1763717134; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0GH1XgjhxGvH7eOvtS78BjGizHMW/kgqOUJowiQXUBA=; b=Yzd8SLfNYUGICSEwn2XJUh+BrNTaUlFEeE2/2izq4OQlev4UEVBlmPG9MumotdLadi 6VfjXjDSQtHlJpcq7X9DfD6RQt4PBKeNr5+xBQmZEVu+GADo1eM3CdM89Rxk9tZsWo5m vIndcC9wXByLWG44c9AUDHCCOOXcQ2QDcXCN17zaEzR0mz3IYugeTpppNXpL+Hn2vdyF 84Si5QUrCjnhR5Tumgz2lSCWvv/6vMXr3e5OQwqcTGQ7MzN7aDbjkWMQhphMMVawxged 8gb6dUxrsi7tKKyvjaMvSK3SUV+W6jASh/7AzvpcqCyx2dbUBzfOZvWdCJBeTWEwqTgi PMxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112334; x=1763717134; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=0GH1XgjhxGvH7eOvtS78BjGizHMW/kgqOUJowiQXUBA=; b=WQfcYl+lyEW0r75zJp0DL5nCfMi08DYARQVbyJUMjCgWifwaznVi2ZsOUcN5IyNOny kduDKsFPOoBiYFVmSUjGGxm2oXwb3z2DhOftvzB8HKlgHsfKzSnk3dbS50Je1B4S6CzA lXs7qiJH8ombmAsvDLY3AuUOc4YxXrU7P5zGIDpnz0eJzXGg90UBtE8AFrQHFZbPP1x8 0lTYazPgQ7uIlaLgFeoAmE37Hq9+xxAZJijFuHbf9UOe7hc/BVMLuVBY1LpJEH29PvMR /68R5ysZathvtzw/CEo9dAqS7HVG31WTUFXuJu+MshBltLeIvVVKO0+15gGuBEgAYQMz JG7w== X-Forwarded-Encrypted: i=1; AJvYcCV+m7xoszdwczUDuStV0idMWvtYlmsG+dIV9Wkev7kOJtwpOws5goCJlppHR35PDr6w7J+e7j/AcJDwiQk=@vger.kernel.org X-Gm-Message-State: AOJu0YyTIbiOR4LnKILrUCt4XThrFJTDt1rt69wa9GWD/NkJOBgJH2We EHOMGG+8WRXtBtn32XmZ9dxg3sbx6rosi2402wGsJVJx+bTC7RED5T4n X-Gm-Gg: ASbGncvCGOchkg9ik/UnbvLTKRMC5g/lmyyby7twQbWvwtio0RSyjy45vaN08Leaffs IO8f1FF6Cw6Ykzu5io+zd53lsG8PS0+Ct7qlj1GUrtCDaDUl8djN1EzTjR+QINbxtLC6TWAW7FZ I6oONKgYGbU+lPMTRC1xkj+yG/shP+DngDJCpfOEMThERtMvlpbQUL44finND38NcvYkqCGvrvV nURaqZgvO35Yb7oq5Kdu7j3quOAf3AN1dhLKgR5f1Zw+RDgzhjA1fhz1LeDYPWiulQngP5U1tJR otkJQTmDutntFjqu5m550BiCBwPhuBpmtoE1pLpF2uKrubgUfNyTpEQ4C7XnD0uDQTG5pNcdGnT Ipu7hZUf1jE+pb0GY74p+EaldEoVdZV6cse78KK+4v1Ys6+s/z9SGcifAxYa232s7naacQrBU1u EE X-Google-Smtp-Source: AGHT+IH8JdBL4kgG2wlYs/vtfYVs3OXl9lPqSBwJDQ1zGrYP08Gk8O6hyA6D59Y+JlLxX9WZiLPJ4w== X-Received: by 2002:a17:903:2f0e:b0:295:8c51:64ff with SMTP id d9443c01a7336-2986a7420ebmr27102945ad.29.1763112334321; Fri, 14 Nov 2025 01:25:34 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:33 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 7/7] bpf: implement "jmp" mode for trampoline Date: Fri, 14 Nov 2025 17:24:50 +0800 Message-ID: <20251114092450.172024-8-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the "jmp" mode for the bpf trampoline. For the ftrace_managed case, we need only to set the FTRACE_OPS_FL_JMP on the tr->fops if "jmp" is needed. For the bpf poke case, the new flag BPF_TRAMP_F_JMPED is introduced to store and check if the trampoline is in the "jmp" mode. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 6 +++++ kernel/bpf/trampoline.c | 53 ++++++++++++++++++++++++++++++++++------- 2 files changed, 50 insertions(+), 9 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index aec7c65539f5..3598785ac8d1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1201,6 +1201,12 @@ struct btf_func_model { */ #define BPF_TRAMP_F_INDIRECT BIT(8) =20 +/* + * Indicate that the trampoline is using "jmp" instead of "call". This flag + * is only used in the !ftrace_managed case. + */ +#define BPF_TRAMP_F_JMPED BIT(9) + /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. */ diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 5949095e51c3..02a9f33d8f6c 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -175,15 +175,37 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u= 64 key) return tr; } =20 -static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) +static int bpf_text_poke(struct bpf_trampoline *tr, void *old_addr, + void *new_addr) { + enum bpf_text_poke_type new_t =3D BPF_MOD_CALL, old_t =3D BPF_MOD_CALL; void *ip =3D tr->func.addr; int ret; =20 + if (bpf_trampoline_need_jmp(tr->flags)) + new_t =3D BPF_MOD_JUMP; + if (tr->flags & BPF_TRAMP_F_JMPED) + old_t =3D BPF_MOD_JUMP; + + ret =3D bpf_arch_text_poke_type(ip, old_t, new_t, old_addr, new_addr); + if (!ret) { + if (new_t =3D=3D BPF_MOD_JUMP) + tr->flags |=3D BPF_TRAMP_F_JMPED; + else + tr->flags &=3D ~BPF_TRAMP_F_JMPED; + } + + return ret; +} + +static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) +{ + int ret; + if (tr->func.ftrace_managed) ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); else - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); + ret =3D bpf_text_poke(tr, old_addr, NULL); =20 return ret; } @@ -191,7 +213,6 @@ static int unregister_fentry(struct bpf_trampoline *tr,= void *old_addr) static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *= new_addr, bool lock_direct_mutex) { - void *ip =3D tr->func.addr; int ret; =20 if (tr->func.ftrace_managed) { @@ -200,7 +221,7 @@ static int modify_fentry(struct bpf_trampoline *tr, voi= d *old_addr, void *new_ad else ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); + ret =3D bpf_text_poke(tr, old_addr, new_addr); } return ret; } @@ -223,7 +244,7 @@ static int register_fentry(struct bpf_trampoline *tr, v= oid *new_addr) ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); ret =3D register_ftrace_direct(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); + ret =3D bpf_text_poke(tr, NULL, new_addr); } =20 return ret; @@ -415,7 +436,8 @@ static int bpf_trampoline_update(struct bpf_trampoline = *tr, bool lock_direct_mut } =20 /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ - tr->flags &=3D (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX); + tr->flags &=3D (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX | + BPF_TRAMP_F_JMPED); =20 if (tlinks[BPF_TRAMP_FEXIT].nr_links || tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) { @@ -432,9 +454,17 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS again: - if ((tr->flags & BPF_TRAMP_F_SHARE_IPMODIFY) && - (tr->flags & BPF_TRAMP_F_CALL_ORIG)) - tr->flags |=3D BPF_TRAMP_F_ORIG_STACK; + if (tr->flags & BPF_TRAMP_F_CALL_ORIG) { + if (tr->flags & BPF_TRAMP_F_SHARE_IPMODIFY) { + tr->flags |=3D BPF_TRAMP_F_ORIG_STACK; + } else if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_JMP)) { + /* Use "jmp" instead of "call" for the trampoline + * in the origin call case, and we don't need to + * skip the frame. + */ + tr->flags &=3D ~BPF_TRAMP_F_SKIP_FRAME; + } + } #endif =20 size =3D arch_bpf_trampoline_size(&tr->func.model, tr->flags, @@ -465,6 +495,11 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut if (err) goto out_free; =20 + if (bpf_trampoline_need_jmp(tr->flags)) + tr->fops->flags |=3D FTRACE_OPS_FL_JMP; + else + tr->fops->flags &=3D ~FTRACE_OPS_FL_JMP; + WARN_ON(tr->cur_image && total =3D=3D 0); if (tr->cur_image) /* progs already running at this address */ --=20 2.51.2