From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 656F03054FD for ; Fri, 14 Nov 2025 09:25:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112308; cv=none; b=A13eeIbW6VPRGB74Fu1JyQicC6flTcPdoemACjc3AQ5TmIPZDs1bGg1dI9vUHDWsxV9Cm4839b2D0+9wkFHL0p27KBC1sNexrWbU1en43LenxhRFhDlGXPvWH4j/2QuYoQXomwLDnC8pHSFsi0zLDO/IVJUekKWf9HoAo/iz1pU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112308; c=relaxed/simple; bh=S0ECY3J4eY0S5MNW8rLP5hJohbui7zvgamXvzag6zbQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DEblW1KalBzDhYrtzQ+ZPuueE8Kyt1gqPObt5/epGFjSA+PBnWwnrVJvvN27aSY2hQu6nUPtyh7B/s/QU7kDTVdEYCgAZfHCmCeazzOWyI1EQi1HN1AFez2FDkmaMtxNR78t0s2ENnmAR53H5msah1yli4ix2YeK/fXELq5ROTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aEROZlaW; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aEROZlaW" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-297e982506fso19955055ad.2 for ; Fri, 14 Nov 2025 01:25:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112304; x=1763717104; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y5AEyxkBUzWQX1qPT1Lf5tl2SWOdn4lIjFoqvWNFFWs=; b=aEROZlaWHJLaj7IaklCRtTDlcCI2NHdeeEeSwRlV20jKz0x88Vf/sALXcUw88b/7Py aA0gSKPXMWQ2sFFOb+KdE65/HQit0w/OV6AreHSXgmebytivLE/VcYlUnee831kZ8BR4 89XjWAKeVCj092zA7Vi4HBebQtopfI46yeTVgXc6Oh+8Xx8tiHkLKBAu+HgavvRlTZ9q 5Ud7RGX038yfwfkLvnOcH06ofFEhZmjRt9hvYSbaVRTLYCV1JkvUxegULl/npM/UdYRZ Mfo1JcXgYFhcUMZOyRej/9LxhQpJoMYkYkG30FjhmWIbPkDGr4sS/0wbA9YimTo2OrGi otlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112304; x=1763717104; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=y5AEyxkBUzWQX1qPT1Lf5tl2SWOdn4lIjFoqvWNFFWs=; b=JzoCpwa6YxXumKVclHomUd/GPsmCCc1i3Ntc5MFKCV5OxcKqAH67PoUTym6kglMy2S xR48Ul8oN7hPgqH6AoJ8pm1HWNUnRCyS2NYqE9/hyZPhubkj1HHEBFTOK+9lK/lXzhm+ YEIJFO07GOBNV6XLKWBdLD4iTkJJ0kTKg5rNfegUF/3Xj/DK4gQYQK6oXnmISYuT7vsF WpDj69XruNAXQn2GS6PFCDsqqPDWQbAKdweLiHHclritu9QYqoEHA2lobpbfX8IKlXdp SSZ0OswGOCtBS57Im5gDhBCmUWMNnbnDBcO8w4VCxaz2rOopOxrMPqB5z3GX+aU1/DpO VPUw== X-Forwarded-Encrypted: i=1; AJvYcCXITiAMj2HAEOY1cDo84tMPTRBVJiKjgygg9Vg7gkCfPn6Xi/FaKzLdBSqh68gJrTH2+fgd4qSNaPrDVec=@vger.kernel.org X-Gm-Message-State: AOJu0YwkQ02/2cQl9IEHv7Lei2xBEOvHbRxL1DupqY1M0m16y+DAL21H nDfXvMZ0QOahmAqhEUmh3rEVABjWuHKO/CqeQfELlQ214L9UjLU6ftco X-Gm-Gg: ASbGncs2jZDCW2TShOszgWa4K3rckNrOGWzZ5FKzowY7LQvwiYdo4Y5qPJ+cUJY1g90 3lveeMIa/0IB1F4syKSSJiQPr/oofaapX/ddIHPNotzq438CNE0+BCmSmJINXqOFJmRCWsHV7RL ibzPht3lv43EEGDT1JeAn+r+c0vE9E0qpI4AZN1XqPeUsVQM4t56qcJC9y9q+Jkolw2CbMKGyYM RznxWizcQzP+on/16o7LTIejAyxp52qvykCmWjCWtKl0ZfpTToFR7ewd4tEwLQ1m2jLFexZ7p+E j6LTOwBSYfmO2loz2acMorre47miF4YPtAVRptuhjbH1Jdo6iw2rfI2fxIuJV50aosxNn+iWhr3 Y9tCoLDgYrUlA56dPNJgL0M0lpvAKdIdliqTMxXvOeBwNgxRz0/CZbyWgthzJL9afhPY1K5bnaB vK1lgIAOjCL3E= X-Google-Smtp-Source: AGHT+IFnh52crzYkS3TAz7VEtpDOKOIs0IFpocAnXLP18gr0pPhgLCo1zXXKhdHJbPjGce+rZZ8OBQ== X-Received: by 2002:a17:902:da83:b0:298:90f:5b01 with SMTP id d9443c01a7336-2986a76f3e0mr26574065ad.52.1763112303652; Fri, 14 Nov 2025 01:25:03 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.24.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:03 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 1/7] ftrace: introduce FTRACE_OPS_FL_JMP Date: Fri, 14 Nov 2025 17:24:44 +0800 Message-ID: <20251114092450.172024-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, the "nop" will be replaced with a "call" instruction when a function is hooked by the ftrace. However, sometimes the "call" can break the RSB and introduce extra overhead. Therefore, introduce the flag FTRACE_OPS_FL_JMP, which indicate that the ftrace_ops should be called with a "jmp" instead of "call". For now, it is only used by the direct call case. When a direct ftrace_ops is marked with FTRACE_OPS_FL_JMP, the last bit of the ops->direct_call will be set to 1. Therefore, we can tell if we should use "jmp" for the callback in ftrace_call_replace(). Signed-off-by: Menglong Dong --- include/linux/ftrace.h | 33 +++++++++++++++++++++++++++++++++ kernel/trace/Kconfig | 12 ++++++++++++ kernel/trace/ftrace.c | 9 ++++++++- 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 7ded7df6e9b5..14705dec1b08 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -351,6 +351,7 @@ enum { FTRACE_OPS_FL_DIRECT =3D BIT(17), FTRACE_OPS_FL_SUBOP =3D BIT(18), FTRACE_OPS_FL_GRAPH =3D BIT(19), + FTRACE_OPS_FL_JMP =3D BIT(20), }; =20 #ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS @@ -569,6 +570,38 @@ static inline void arch_ftrace_set_direct_caller(struc= t ftrace_regs *fregs, unsigned long addr) { } #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return addr & 1; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr | 1UL; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr & ~1UL; +} +#else +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return false; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr; +} +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_JMP */ + #ifdef CONFIG_STACK_TRACER =20 int stack_trace_sysctl(const struct ctl_table *table, int write, void *buf= fer, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index d2c79da81e4f..4661b9e606e0 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -80,6 +80,12 @@ config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE If the architecture generates __patchable_function_entries sections but does not want them included in the ftrace locations. =20 +config HAVE_DYNAMIC_FTRACE_WITH_JMP + bool + help + If the architecture supports to replace the __fentry__ with a + "jmp" instruction. + config HAVE_SYSCALL_TRACEPOINTS bool help @@ -330,6 +336,12 @@ config DYNAMIC_FTRACE_WITH_ARGS depends on DYNAMIC_FTRACE depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS =20 +config DYNAMIC_FTRACE_WITH_JMP + def_bool y + depends on DYNAMIC_FTRACE + depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS + depends on HAVE_DYNAMIC_FTRACE_WITH_JMP + config FPROBE bool "Kernel Function Probe (fprobe)" depends on HAVE_FUNCTION_GRAPH_FREGS && HAVE_FTRACE_GRAPH_FUNC diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index efb5ce32298f..8d7b2a7f4b15 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5938,7 +5938,8 @@ static void remove_direct_functions_hash(struct ftrac= e_hash *hash, unsigned long for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { del =3D __ftrace_lookup_ip(direct_functions, entry->ip); - if (del && del->direct =3D=3D addr) { + if (del && ftrace_jmp_get(del->direct) =3D=3D + ftrace_jmp_get(addr)) { remove_hash_entry(direct_functions, del); kfree(del); } @@ -5994,6 +5995,9 @@ int register_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) =20 mutex_lock(&direct_mutex); =20 + if (ops->flags & FTRACE_OPS_FL_JMP) + addr =3D ftrace_jmp_set(addr); + /* Make sure requested entries are not already registered.. */ size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { @@ -6117,6 +6121,9 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 + if (ops->flags & FTRACE_OPS_FL_JMP) + addr =3D ftrace_jmp_set(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; --=20 2.51.2 From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86B89307492 for ; Fri, 14 Nov 2025 09:25:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112311; cv=none; b=SJ8/5o+HPt9Frxjd8qlRIgq4TIBpqbm9rMPrEzUlhHuV5eWj6VWhGrvfpCMsadTGPXCZe6++cy7+p6KFjZwmrw1DYzdwrAomhUsaOp8DEg19iM+W8wWxNtmCdelT7dKOrBwDIWnzacxg6a+W0NTZvRHEXSDDXK3HogOjA9Y8GkA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112311; c=relaxed/simple; bh=gokrqToajl3qLeodPuzcgSTB9cabw/WpaV1bSGQlVLA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=id/elyztWrbU9k+uBm8eX7011Lk4iSyjlZQGgHxThwisYrXsCPUTQEkx8igAonQevR6BsLzCaNqHMzGa67WsMOi4hyTHlT34/7OBydd4d5BCn8VD+59xeKujgqmh/KPrxLpvyP8jenqsi2hDxZUQw5QlBPCZHqTJG3iAvQGsb8Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PjmZXHsv; arc=none smtp.client-ip=209.85.214.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PjmZXHsv" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-29555415c5fso20484405ad.1 for ; Fri, 14 Nov 2025 01:25:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112309; x=1763717109; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c2wIn/kARIOhJ0xgPL8i1YkD6/QkhYtCyZijlaHlovw=; b=PjmZXHsvrHkZX1BfPdMHG1LHO+98ercxeRnouDfeem7fxEpj+PUhudvllYXeEuQ4/U FuT6ehPHuJspLMumi9AINBr1vPA2JVl8S2YQLy4FKYSZ6aSZF48ZLZofhFbvOxWJ/02/ +LF+cz/cia+PdpH0TvPIjwx1CV4uiTfIsCQFIq2IyxzpsujoFwm6UWx/v2m/bqbzoJFD ef8XIhouSlk/N3xz6iEFvQuI/U3MjK0ieSITtEbufSq0P8QmdA9sZXuT4iTfOrlzLUnf IRfnShm2o5/zgstgFCRkgWEoI2RGObF7GMJB/d10XZX/TZdxprnqQrmgqzF73LpIXbOr +ldw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112309; x=1763717109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=c2wIn/kARIOhJ0xgPL8i1YkD6/QkhYtCyZijlaHlovw=; b=qHeyH9Yz4FHfo/VufKTOP+WL48/Jm4d6ZztYRJNwWy1Yj5Z0Xj6EhvBYEw6QXBBtp/ seJJVDnP8YiZWGieXjsdG5X+xC44ykOf2bT1vOmPwJMflXlz9BewG90/3iXyocLwqlZy cVM+mILVClMc7/aWv0BAQ+CK9yHMU4KDH6BsOzYEAOJVHq+uKKI8Udg6sDABjQ3g4pIX Ulyquocir/GfJ45xTeU8jvzFT4j3ZQyjOvE46c+D0z0kfPli/kocjONN70okAddE7fya 933dFldllzc5HAqzgwBbBYsbPJdB30G38ZU7l8rxBVZlyEkGRjfCPP2+cytsO5WwmDPJ Afgw== X-Forwarded-Encrypted: i=1; AJvYcCX8jNkD0MsItkHstlS1wlG8r10ITKyphM84WU/K2EHt3mVm5mp2tQCt+23n18joWYUX8OOR3IfpZAku6hU=@vger.kernel.org X-Gm-Message-State: AOJu0Yy8mgY0nqj3FXMXk/JQtZGna6edgYyVkyLj60p5AbTf199fwoIR LzmksgRa4R7ygSE+jEIcnnHkmA08d0C+Fr64pDhB+JkNEz5gR9v5bP31 X-Gm-Gg: ASbGncvtgwX+b1vh1w3puVkd0X5ziuTyEpbMQOlrqG6217gy7CJO+sXkUPRkN7GSQPJ Ywo2OJCOgdqbL/nsmxv4cFbW4mg6gcbPOhvNNenW429+EURvvv1I8CxjzKPfowy2689sIA9Tc91 dzKKQ8qp8mmOK8nqV6KYSauNQbdghM0sZsgMYrJDdseuGGlRQz1y96ldz9R+lbrWTA3Goc+LRaV 1ksQK2QO5hyi/AscX+ckd6zRcGJ8kG2D3p4BudSlSlwBp4nQhqUyJaBlH4yJXBTRFXo7jxxuMms fZh1jrL5YBeFIwazSL//0PUODnB5rJAA5ebboiqDUsqnfX9aRY6sTbpnKVom/LE6B7rwm9DZexZ ZaT8DLtbQnOt2w6kmRT63iUo9Ht0KznhjhZdXCbm36udQpir6Uj+/2+VfqBrwjINIbRuutTdYyW Rk X-Google-Smtp-Source: AGHT+IG5VnwBl+1lxPrp6qdos/kEJNYbsFAuD4akbBy67+ZoqlT1tPAQmEIH9GoRP72WFoHDwsjAMw== X-Received: by 2002:a17:902:f60e:b0:290:c0ed:de42 with SMTP id d9443c01a7336-2986a7417b8mr26050145ad.36.1763112308699; Fri, 14 Nov 2025 01:25:08 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:08 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 2/7] x86/ftrace: implement DYNAMIC_FTRACE_WITH_JMP Date: Fri, 14 Nov 2025 17:24:45 +0800 Message-ID: <20251114092450.172024-3-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the DYNAMIC_FTRACE_WITH_JMP for x86_64. In ftrace_call_replace, we will use JMP32_INSN_OPCODE instead of CALL_INSN_OPCODE if the address should use "jmp". Meanwhile, adjust the direct call in the ftrace_regs_caller. Signed-off-by: Menglong Dong --- arch/x86/Kconfig | 1 + arch/x86/kernel/ftrace.c | 7 ++++++- arch/x86/kernel/ftrace_64.S | 12 +++++++++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fa3b616af03a..462250a20311 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -230,6 +230,7 @@ config X86 select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 select HAVE_FTRACE_REGS_HAVING_PT_REGS if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + select HAVE_DYNAMIC_FTRACE_WITH_JMP if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64 select HAVE_EBPF_JIT diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 4450acec9390..0543b57f54ee 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -74,7 +74,12 @@ static const char *ftrace_call_replace(unsigned long ip,= unsigned long addr) * No need to translate into a callthunk. The trampoline does * the depth accounting itself. */ - return text_gen_insn(CALL_INSN_OPCODE, (void *)ip, (void *)addr); + if (ftrace_is_jmp(addr)) { + addr =3D ftrace_jmp_get(addr); + return text_gen_insn(JMP32_INSN_OPCODE, (void *)ip, (void *)addr); + } else { + return text_gen_insn(CALL_INSN_OPCODE, (void *)ip, (void *)addr); + } } =20 static int ftrace_verify_code(unsigned long ip, const char *old_code) diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 367da3638167..068242e9c857 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -285,8 +285,18 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR RET =20 +1: + testb $1, %al + jz 2f + andq $0xfffffffffffffffe, %rax + movq %rax, MCOUNT_REG_SIZE+8(%rsp) + restore_mcount_regs + /* Restore flags */ + popfq + RET + /* Swap the flags with orig_rax */ -1: movq MCOUNT_REG_SIZE(%rsp), %rdi +2: movq MCOUNT_REG_SIZE(%rsp), %rdi movq %rdi, MCOUNT_REG_SIZE-8(%rsp) movq %rax, MCOUNT_REG_SIZE(%rsp) =20 --=20 2.51.2 From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F2F13090CB for ; Fri, 14 Nov 2025 09:25:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112317; cv=none; b=kNXD6p7/f70eg9DNcF2sRG8EEtRDYSrclA5w53Jfxfba6Hclg6hzGByb0MgvSBZbyLhgkDfqxoVzaT+5WsmvbQnlpRGY3bDq6UtCDXjx53k4m96Eicf0Maalgn+pO3rJIa/1S6bVY93FP095K6hE8CTzji9HBSlFbp/+zgxhTcY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112317; c=relaxed/simple; bh=13rTaiSsL0NTZ1YOaRxf6oSc1Dy8agBuYumvqKAnvQk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fyXsEJfxMomUKwGk8T5FF4xXB1Wb4md2F/ZZO5uN3uPsZ8/1CAbbZihDZAtjDlS8ha6drm5oJAlt2yvm95zvW3UuDHHPjRSuLQKRsMfQua5EgGCcOOJMJwL5TC/QB+zNxJQtNWVUhXvcuXE5//yDDOYshmRWbw0wV2vdUeMTedE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KrHL9VWt; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KrHL9VWt" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-297ef378069so16911735ad.3 for ; Fri, 14 Nov 2025 01:25:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112314; x=1763717114; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CR4G2sTlGki2siCN1jOZFPTFr3TbudN4PF6c7ryHzjk=; b=KrHL9VWtE4O82KxpyXvhAlHSQMbpyLBZwKUcaF4AbRQCceK+D0jWXXwEcDe9XZa/o9 wyE8FYRpIctdMJ7CAHP+l3qIBfo1b5cHCqcRWDzELndSENhLcw0ooW7g6Og4T6roba4r CaX90a2FMRbyAVPFARRBcCC4rrTFqE6SSBJYk88oUK1Fb5MHM8rGR66D8kUNsIyRIfkZ xQvX/uUFXgjuYlLocgrzDZKyzj4aoC4pg4yYkE3h/+lRVy4GZjpmNwsM7bYVTPbJN3wq PY1FDC2TnuVuNIEfteqFI9zr5zHORNi1i+y+IeQMf0yV5Hss7tyqxGcwlsQHGLud3r0s wSSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112314; x=1763717114; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=CR4G2sTlGki2siCN1jOZFPTFr3TbudN4PF6c7ryHzjk=; b=P3GhVI0wUpdmiqc1YEWGi9jGpBwuW2B9U9zVxHmJ88JLF7iG4spg/n1h3jkYH+3JlG GH9EREsYWwwD6ng6s6+dtGhzzv462PMKhZU3WZiaw5a+3QwkUhgDyCQZuxkCVuV4Hmv1 IA3nAz24t1yVD8pUO9ufkPKLuycjXL9NXMwKxFZ+OyOalZmtEs8QvbSKxSmvs5Jhshyn 4fWmzYau9cOixzkV8c4yQu2kKrwIt22Rv2TDjDd5e0+lzumNiXsjRiwZbwGQtFACd311 L7WJxesndCPEPsgD0jTjvPcJGpzrhsiEIaKjzRQMfkZwMJFEyLeG6ybO73O8ezJNJ9jy 1gfQ== X-Forwarded-Encrypted: i=1; AJvYcCWn+P60LL0Qm4R2PkFx/Zo0IJ5X+849COhDHTbZRxlEVAuCBb4/QrNXpYX2AcRulWPU/HXcWiwT7Sz+pMU=@vger.kernel.org X-Gm-Message-State: AOJu0YxVMFSSYBCvLR+W1vgx93pR+qmQejBOsyx0SsXI0WOdYUaBnhRL Y4ynATblRu1OPFNEq9CB79MAjAJ7KG4lWlOYnadVQpWKRiQoOhzqujv4 X-Gm-Gg: ASbGncv6JQUFhUEHf/kIilIVA+ViC3TyRApk7BBtoE2flMKZBwUUIHsbQ1ZM232PmOh lfN5QEjCx+ifKfXwJAGHpZKOH4uXgIapPPFhr8k5mtrYw/v8X3aBd8rm2q8ma0Viuxl83fYXgL4 bPJTEEYKVRb/qquuceN6/WjMX9lXlRCSAImXvfLTahEIvW1x4RlOrAdT6q0n8tOWKlMPUbdD714 EjooCsDMI3HYz11dsp2R03u4UxvCD/du4074Jh2zwROwzZhsggeTJzYuwZYLTXfrzDDNdRFztBw A3+UyapP1JJkqrSgK7YTWu+iOG119SWL2zxi10wl0gR5dUUye7+paPIarh0NVIBgMN10uWzTk8e xdZPDjb0PL/D20awETVV8DE44hu5jNKQ5bKEl8cvM0M9iN00U1Gg1e72Wlk0PCg23nqkig1frA6 e6oyN+HPD6jZI= X-Google-Smtp-Source: AGHT+IHcncu3fCVBTRgs6cuHPXbsXwtrJ/13HcgbxbxqCj94e5Leoiq1IYCMd5ZzNeVgfN5DkjRaag== X-Received: by 2002:a17:903:22c5:b0:295:2c8e:8e44 with SMTP id d9443c01a7336-2986a76bc1dmr29990775ad.59.1763112313782; Fri, 14 Nov 2025 01:25:13 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:13 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 3/7] bpf: fix the usage of BPF_TRAMP_F_SKIP_FRAME Date: Fri, 14 Nov 2025 17:24:46 +0800 Message-ID: <20251114092450.172024-4-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some places calculate the origin_call by checking if BPF_TRAMP_F_SKIP_FRAME is set. However, it should use BPF_TRAMP_F_ORIG_STACK for this propose. Just fix them. Signed-off-by: Menglong Dong --- arch/riscv/net/bpf_jit_comp64.c | 2 +- arch/x86/net/bpf_jit_comp.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp6= 4.c index 45cbc7c6fe49..21c70ae3296b 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1131,7 +1131,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, store_args(nr_arg_slots, args_off, ctx); =20 /* skip to actual body of traced function */ - if (flags & BPF_TRAMP_F_SKIP_FRAME) + if (flags & BPF_TRAMP_F_ORIG_STACK) orig_call +=3D RV_FENTRY_NINSNS * 4; =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index de5083cb1d37..2d300ab37cdd 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3272,7 +3272,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im =20 arg_stack_off =3D stack_size; =20 - if (flags & BPF_TRAMP_F_SKIP_FRAME) { + if (flags & BPF_TRAMP_F_CALL_ORIG) { /* skip patched call instruction and point orig_call to actual * body of the kernel function. */ --=20 2.51.2 From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D760309EEE for ; Fri, 14 Nov 2025 09:25:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112321; cv=none; b=EL8F4XFI3anPEDryJTev7AAfBMYHym9xxmrKt7k+KWw0Kf0/diQUsJ6CCqvcTr3aPEGi560Nzg+BNBeQGi14B919dGQa8IzbxI0mvzNuVVKEGJZpf6gYxuCYN2uMuRq+QZel6dP3So56NgvsC38Q75q2Oi2owSnxr+yc0jQpfLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112321; c=relaxed/simple; bh=FAoA8SSkUxlpnxysz5ifBOU2ouZ7qvBl4IFvJKPZUXM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=breKni/wrJ60dcPlyWEVH15o4CUs05JO/MZw3ke45IBnw7UBSVy/hp0T1OV5/oA1UPhnliyqb04sOT0GAz2xpW0+J3LEdSXZ+4zX1pOYGE0bM86+HgMAQyqe/d/+GidpzvMHVL3EYmMHzaVyCdQInA9YIUkuxpwAFwU2YoVizfc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=l8OeoAnq; arc=none smtp.client-ip=209.85.214.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l8OeoAnq" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-2964d616df7so23284735ad.3 for ; Fri, 14 Nov 2025 01:25:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112319; x=1763717119; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Lo0oHQehABD95AGfmd72YLS+a/OuJSMG1VdEJWGhMs=; b=l8OeoAnqOdyCTK4j5BnEaaAafBCerTiyukOiSHdMZmwNPCBWUPVhee8WqSzaHcT3B3 Li9ZlW0YnFc+pmxIrKaxEkWkO4vEDRoaXDmqgWD4jU9dflKPUwmnEvnYyVJLrCawAk6S SywDAy0zlsvNHse0h1HJ3pVqBoDY0fQEX3Vk1zQ5mof5hDbMW2Wcef15cXcTBKOpLLpZ KUN0D73KYMCRk93EVx+oAMR4acsQoo/E7QwU2XgSxw875oHeleyDscgUcIEVVtoKOHCo FaaVoubNbPZ6b53Mbe4j9qgTfUn/hkO+jbtb/qMWeq/WqVuZ4TCgZzQiEtnDjGcWzIyj oT3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112319; x=1763717119; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3Lo0oHQehABD95AGfmd72YLS+a/OuJSMG1VdEJWGhMs=; b=JeBe+ijMVa2uThMy9/zrDUQvMeJpMDbJbjG91lohLcGjpnQaTipR9rJqDRjNuGAmLd Tzb99oQfQAnLq5fYK/KsJIW8OqOEvAa4pOp0phbBwdBmIWSouE7y/LjitjtjeEvFCD1p EhRN5HjEZdtcKVPLDWs/wounjiPn/IAoE5DDlzhaBgS/iQfiq2sOZ8WoIbyPsyWlm/b1 DJjltlng0kKR749/8C5tkQ8n0Fypfv4oQNsQOkQB7teEiioQ871GKgfKBbqWc9pMzMhN QjSyUdem2/O5gOpnqdVNC8/D7zRkvjYXo9FIJv6yui0tUINe1Ugof6FrMLGkCuf8Ssst DhmA== X-Forwarded-Encrypted: i=1; AJvYcCVsKWZVht1eB/2zl3YaEo/831sjp/7arT75p+Vj97sfIYk/rJZujWYYDl0gY1l9etiO+qLnk5NNDXsfQAQ=@vger.kernel.org X-Gm-Message-State: AOJu0Ywl0CHo8uvlMyaqsYA1P3Jo5j8Qq2moVlFMSZ39Bk/dTYgzbscp VOPh6eDDNfHzawzMbdw/mSIL5Pc/LQcRBmmDoJPocRRG9gG5Qdv+xdzB X-Gm-Gg: ASbGncvHyXPjOL+oQx4BFc2xoRadQOvEv7tSPjrvLB4DE1/yirK/fgXZ0EX8w4ilSTh JjNMGV1wjXhNv2oZDeeayzJ/SVw03rFDpPmKvzNnSduwLiySbdWT3urHwfFL7X8GAE8UMAUCcDm gikvwPHZUDLEZHf4huNhnblhnS32J0ruSdChfyfPo4w11L2z+/SXlmAyCiO7HJ1YijTFgBg/671 l62QWhbjELLluUQrQFoVW8GoNmBkjfSQS70i8X6ARpdaHdvjncH5DxpeySftpJi/VdZHuBYIN/F rBrTPz2/cGBY/5wXto8yWq5GJ+dKsU52BC7RmDL96cAeCxyVJa2/CpkcXUNbquKxE3HHAxpyDMM 7YLh+IS78UPBwftHlEzerwB11WcomNfyC6vtQ7UhFcWysZ5cDRwTOaW3AEFR5GQf9ijCJHFnvKA zCj0byqOSRf8I= X-Google-Smtp-Source: AGHT+IH+Ak1j0xCi7Rm3y3incs6i/Rgb5Az/FlZugKLvJkcy7tgkhMk2eLmjZEILmQE742wMiBUzQw== X-Received: by 2002:a17:903:2411:b0:297:e59c:63cc with SMTP id d9443c01a7336-2986a73ace5mr27684325ad.35.1763112319229; Fri, 14 Nov 2025 01:25:19 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:18 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 4/7] bpf,x86: adjust the "jmp" mode for bpf trampoline Date: Fri, 14 Nov 2025 17:24:47 +0800 Message-ID: <20251114092450.172024-5-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the origin call case, if BPF_TRAMP_F_SKIP_FRAME is not set, it means that the trampoline is not called, but "jmp". Introduce the function bpf_trampoline_need_jmp() to check if the trampoline is in "jmp" mode. Do some adjustment on the "jmp" mode for the x86_64. The main adjustment that we make is for the stack parameter passing case, as the stack alignment logic changes in the "jmp" mode without the "rip". What's more, the location of the parameters on the stack also changes. Signed-off-by: Menglong Dong --- arch/x86/net/bpf_jit_comp.c | 15 ++++++++++----- include/linux/bpf.h | 12 ++++++++++++ 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2d300ab37cdd..21ce2b8457ec 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2830,7 +2830,7 @@ static int get_nr_used_regs(const struct btf_func_mod= el *m) } =20 static void save_args(const struct btf_func_model *m, u8 **prog, - int stack_size, bool for_call_origin) + int stack_size, bool for_call_origin, bool jmp) { int arg_regs, first_off =3D 0, nr_regs =3D 0, nr_stack_slots =3D 0; int i, j; @@ -2873,7 +2873,7 @@ static void save_args(const struct btf_func_model *m,= u8 **prog, */ for (j =3D 0; j < arg_regs; j++) { emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, - nr_stack_slots * 8 + 0x18); + nr_stack_slots * 8 + 16 + (!jmp) * 8); emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); =20 @@ -3267,7 +3267,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im * should be 16-byte aligned. Following code depend on * that stack_size is already 8-byte aligned. */ - stack_size +=3D (stack_size % 16) ? 0 : 8; + if (bpf_trampoline_need_jmp(flags)) { + /* no rip in the "jmp" case */ + stack_size +=3D (stack_size % 16) ? 8 : 0; + } else { + stack_size +=3D (stack_size % 16) ? 0 : 8; + } } =20 arg_stack_off =3D stack_size; @@ -3327,7 +3332,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -ip_off); } =20 - save_args(m, &prog, regs_off, false); + save_args(m, &prog, regs_off, false, bpf_trampoline_need_jmp(flags)); =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { /* arg1: mov rdi, im */ @@ -3360,7 +3365,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { restore_regs(m, &prog, regs_off); - save_args(m, &prog, arg_stack_off, true); + save_args(m, &prog, arg_stack_off, true, bpf_trampoline_need_jmp(flags)); =20 if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { /* Before calling the original function, load the diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a47d67db3be5..d65a71042aa3 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1263,6 +1263,18 @@ typedef void (*bpf_trampoline_exit_t)(struct bpf_pro= g *prog, u64 start, bpf_trampoline_enter_t bpf_trampoline_enter(const struct bpf_prog *prog); bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog); =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP +static inline bool bpf_trampoline_need_jmp(u64 flags) +{ + return flags & BPF_TRAMP_F_CALL_ORIG && !(flags & BPF_TRAMP_F_SKIP_FRAME); +} +#else +static inline bool bpf_trampoline_need_jmp(u64 flags) +{ + return false; +} +#endif + struct bpf_ksym { unsigned long start; unsigned long end; --=20 2.51.2 From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFCC130AAD7 for ; Fri, 14 Nov 2025 09:25:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112326; cv=none; b=OCPHQ33lOKq9Qi15tk+7G5jIfCmLcv/p0zdTp3xP8zANWqFS0oSz6EoTnUo/JY//RrtXkst1tOAFE4iJib7OQFY4myu5zCRWhANtfqRJgyE/HBIfieBS6eIZ+tJLC2jHrnzEC6y7rpvGLJrFRnKuwUwA4Qs+O9ha3Q+GDh5n8eU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112326; c=relaxed/simple; bh=4ETTO9+rYsbYHmKk114m2zMiu+9FB8O2MF688r6MCdA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iAzkeeTpGt6YV0dFjwYdQkRQQIffYmxfEchk5RXQKoyYHeo9SNxSH/5UoUgfbEgTcCV/MPILuf30sZJYIkIfVWqPoScFIwoelfLLWjg3GgBp6iQcKCruv+VGHRDlF823iibC1tzpWfzA7a6044TZgU9XUADhIruk4zqyzjflHeY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Il+n4sCY; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Il+n4sCY" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-295548467c7so20136145ad.2 for ; Fri, 14 Nov 2025 01:25:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112324; x=1763717124; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dLWaDKWIVOGQHPd1CPktpd+aufU0XSiNZY9F4JwbLEQ=; b=Il+n4sCYzUga5HWfg84pzU6Uz2bquLvqAxCwvQKnXbX14qwYtRStknLg6pmXpIWf3u thMJcUdDtgn6o3namLNRa+t1aabSGPnA4SXx035YjtEMbNUM1ph1e3jzRfuWYp8lmUht zERO9y6LzmlVdr2aMFrxXNAGZg2CF8LAxb5PnvVuypiUFkDWPyOF970FoGGHJl1RpY3O 0YLCA7eRYroQ5sztn3MSiNs/Hgm+liDGcOBY4fO7Lm9fxiGMAZQA2UL0GeRK8Muq2ka8 XBAcbr57zr9OsyXnKdmoPLlGp1vpKmiSyn3Oc5TLEbYe7wiBaO/VCY2jlnZILA52iebC KF9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112324; x=1763717124; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=dLWaDKWIVOGQHPd1CPktpd+aufU0XSiNZY9F4JwbLEQ=; b=EVxJYZAEpQQWJ6/4TWdo3E2EVZq+uMkF8y1yongKwwewUYj47D2ShubTayOOXrdv4e Zk6JuPo50ooZcsP0RpprcBJ3yKuH3HjQxMJkBZTZK1f0nKOTNm7Csg4YEIQuCI+pjo+1 PpTRvrYi4UJryraFuHAa0k5rnk+rMBDF8TuXLW5DeBqxEeaGSNnKhcOHElhOe3K2DysV LyohgfIx1Ly3pgosHZHlDRogdzMiaNdp4hIYjv2BSeg6DF0NbcYAPgqqOfCjyGRFDItv Wz9qIWmAR9Hx2dOGx/j2ICMrM7EboKucSd3EwOeZRcvwpzUNDG13SdvY9h5Kf5soNQB5 14Hg== X-Forwarded-Encrypted: i=1; AJvYcCX276QMHXo3n5d2JC1dFwNrWgjfyYb/1Kmmu9B72OyiTePu/LPKMBx242agzG35BzIZ8h7nKWr+4EJ8T8c=@vger.kernel.org X-Gm-Message-State: AOJu0YwsPjjSxNhdMXdi+LHYCXCbKtR8hjVB2n6RhPIOJLsQNAs9+4yr vzpO5VheP68PQ8TSt4oHOghOxlYqV85jg28l8nII398oIErOuZ1s226d X-Gm-Gg: ASbGnctlosmPkvdRHPEdA6cfceTvhD5X0soPDB3lnBDvPW7uZMVsBfZdckmzRccH7u+ 2SD9VjL/3aspDSPkE3gBoI9yzZv3A9K3REQa4YXoun3s+kABNcDTu8JmHaSVOknASxluOEHsTZP XBqmaLpAZgkPlcGLxS4v43Qrz4ThaIfGzEIgtVnoaq/K4oD+Fxsrz3vGY3CwmLnNepPtdMOnk4m TDrTj3x5zuAxmrd1zZTUJ57/g2Ayz3/WyCYw7NVIN2ekOrlWh4Uu6R8pRPVrfackxyTjhWW/JN6 GlQsJsemQCA8RyNrVUSlJDjeTHrWM+0jBRTCa0KA5A/Hoak5anhszPtdCXdPHkGbrTYVYeKVC82 0SbJDaXz5ksHQENDhnmuEt851xpvQgvK64xduvkSyD/uiCn/0U0gBILoxazO0Sa5SeGXpHwSqcY TwzB/OAIl3dKk= X-Google-Smtp-Source: AGHT+IGD1QDIS4ynyjBaIAAdvUjVM3PwRMzMtqFE9pfIu5HzKqw4/Q93kRj/ILiTts1EGOTBvn+WJA== X-Received: by 2002:a17:902:ebcd:b0:297:f527:a38f with SMTP id d9443c01a7336-2986a6bf92amr26828965ad.18.1763112324220; Fri, 14 Nov 2025 01:25:24 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:23 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 5/7] bpf: introduce bpf_arch_text_poke_type Date: Fri, 14 Nov 2025 17:24:48 +0800 Message-ID: <20251114092450.172024-6-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce the function bpf_arch_text_poke_type(), which is able to specify both the current and new opcode. If it is not implemented by the arch, bpf_arch_text_poke() will be called directly if the current opcode is the same as the new one. Otherwise, -EOPNOTSUPP will be returned. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 4 ++++ kernel/bpf/core.c | 10 ++++++++++ 2 files changed, 14 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index d65a71042aa3..aec7c65539f5 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3711,6 +3711,10 @@ enum bpf_text_poke_type { BPF_MOD_JUMP, }; =20 +int bpf_arch_text_poke_type(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *addr1, + void *addr2); + int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *addr1, void *addr2); =20 diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index d595fe512498..608c636e6cf0 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3135,6 +3135,16 @@ int __weak bpf_arch_text_poke(void *ip, enum bpf_tex= t_poke_type t, return -ENOTSUPP; } =20 +int __weak bpf_arch_text_poke_type(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) +{ + if (old_t =3D=3D new_t) + return bpf_arch_text_poke(ip, old_t, old_addr, new_addr); + + return -EOPNOTSUPP; +} + void * __weak bpf_arch_text_copy(void *dst, void *src, size_t len) { return ERR_PTR(-ENOTSUPP); --=20 2.51.2 From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D79B30B515 for ; Fri, 14 Nov 2025 09:25:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112331; cv=none; b=TLDT5L7LbVWA7zOe8932p+jqhfTm2/BdETZtIleJr4aEll3lL7HU53qpxGWBektW7QzTWfQA7U/8C0OinR4ZxNaFOD0ZN6XUwLMUbOqXT3pgouLWvPz3pj9giEa0GtoL+MsZqfec+UrSOtBdMUK49rvlN3Def+26yRcfkbZzSjY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112331; c=relaxed/simple; bh=nSRVg1kRiVYCpNkt7AXhxAnMSyGK6wFljHMdK0qKX7o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C7QXIDPRGr6UhmmxBaXHqi952hNiP6eNuXrD2py0tKT+m/0OmEM/R3P/6IvleqCcIiJYzlY1ZHEtapT0jTyPbt4jtvpfqfR3tO7arL8XKLO76M7mUiiFJXZ6jHuotqJ6l0Bl2cJf7jeqc74S6wY6aSbIZnHCxmOU6ESMCCOGUvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EM8dzaOr; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EM8dzaOr" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-29586626fbeso18099795ad.0 for ; Fri, 14 Nov 2025 01:25:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112329; x=1763717129; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IngssO9K5rvAy09b3O/itHOX87BH55Ede2DbNxyMyq0=; b=EM8dzaOrE2KU4KVDZpSb72WcdCv/AxpKiBYWFzAxm6e1ZGMcnYG04z02lmPtKd92GH /DGsu2EOzanwdcEIHYRROlaWHdz/4og+LfcHGEJW35hxyoRm36wS2jGE4jCiUTdgCVs2 VNafytOecBKpqdj1o/jEE3/qyXWcchSN00fc94hgnE8mUlymbqZWkuPPHwfCg1xeCX8M M4ocTxZPgU8FdSNUjKY6VmuohLKBHuPfRE51kS3HPpHCFH73Wyb/S6sSUjZjboWWkQAo pug/KGzgS2oh8cx7a/2WNjA7dGqxYVnjm4C8s+3iXJ0n+2SqLXNiDLGb+AFA5ccoZEC7 9yrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112329; x=1763717129; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=IngssO9K5rvAy09b3O/itHOX87BH55Ede2DbNxyMyq0=; b=VP+rCjvR83gpsb1zC7yrwMUbLQbpYjCw7y3TystwqLlVhOpwMp/vdET7a8OhzZd7/+ Muo15NZmoP7gP29JHFT9aQVKNmdxKEogOMWIZgXnF9zRgLsP7FEIIwjW3flxIxuvxjhJ PWHTqxGUfO6RDlIhWjYqZvDQDDhfHtwo5sSfMGIJIGuywIMiS0Tb6dl3JfbMHw3y9BWb +H9cMxBSM4ZhUBx8sMn/ylDSFv5jbg7LcNWHhtjzmwwVZIDesDeJJB07OjT8ErhJAfUf jZTA4GaGdW3E/WLJ9WGeG2/kxiTVvfg7p/P1YadMWlGIZHph5OVRKNChUIrYSMZr8Gic 8LGQ== X-Forwarded-Encrypted: i=1; AJvYcCVlJbVOfEAlIzdPWSBba9ZUVT6Tiz2PSY1ExV5S2dYcctldtqQoGdWUhRANMHhi36Pn/cZUAAo35Xl/1cI=@vger.kernel.org X-Gm-Message-State: AOJu0YysdHsuyP/9tlqSN2B5Hrrt2kWdAwQM+LuRzigpjqRKve6IVavU bZm4/z+AJlb2FIZu0YOQ6XFD3PAq2YnTK68zS/gjHOMViKEcZgFqeVh8 X-Gm-Gg: ASbGncsl/ZIuBU8EOkaxoN4/C/4+SYOtxdiZaM2lQ6KLJW/92Nyr66aSK1SDEVK8T/j hb+zYOYgbrgfSmtC8yLl4IOlD1zjN4q7I/TK+M4dBD7LlXlgpjZHlhnyXFbORK9Vu8VoFddxuSo 30EyOFdsrqJXiUKZXglC2E/yjgNxqLs52aaI+UM2fImwqBSLCWqarCNkG9yhM0JMbGbx10Lmx6B plDsc6Q2QBMsrsI0MpOOjWauq3B8D27XPXmrXFHOmj3Sqp7dHVamFmjgv/noFu7u/2awAqDMIQG 2IgRjMoa/YHgKhfqITU/69tVTXmZ8okG/bnclbkiZvOTaky9GjPI2ZOPmtcmDq7gGPgE42icT9c feua6bir923O6wGBYmWPR4PnSvXJRyPd2Ydi6Ux40CXNH/w1nfKvwKXo6eFYunUBMki/iuJIzGu 7RoH7PysRmjLI= X-Google-Smtp-Source: AGHT+IHlbzklfDa0d/deVykG+Yd1NG+0UeBxaD2Sb0CDDCd79clZdzvDiQVWnGMoXC2ne+kl8gS7QQ== X-Received: by 2002:a17:902:f691:b0:295:6a9:cb62 with SMTP id d9443c01a7336-2986a73b4a7mr29073255ad.35.1763112329230; Fri, 14 Nov 2025 01:25:29 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:28 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 6/7] bpf,x86: implement bpf_arch_text_poke_type for x86_64 Date: Fri, 14 Nov 2025 17:24:49 +0800 Message-ID: <20251114092450.172024-7-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the bpf_arch_text_poke_type() for x86_64. Signed-off-by: Menglong Dong --- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 21ce2b8457ec..c82bd282988f 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -597,8 +597,9 @@ static int emit_jump(u8 **pprog, void *func, void *ip) return emit_patch(pprog, func, ip, 0xE9); } =20 -static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, - void *old_addr, void *new_addr) +static int ___bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, + void *old_addr, void *new_addr) { const u8 *nop_insn =3D x86_nops[5]; u8 old_insn[X86_PATCH_SIZE]; @@ -609,7 +610,7 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text= _poke_type t, memcpy(old_insn, nop_insn, X86_PATCH_SIZE); if (old_addr) { prog =3D old_insn; - ret =3D t =3D=3D BPF_MOD_CALL ? + ret =3D old_t =3D=3D BPF_MOD_CALL ? emit_call(&prog, old_addr, ip) : emit_jump(&prog, old_addr, ip); if (ret) @@ -619,7 +620,7 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text= _poke_type t, memcpy(new_insn, nop_insn, X86_PATCH_SIZE); if (new_addr) { prog =3D new_insn; - ret =3D t =3D=3D BPF_MOD_CALL ? + ret =3D new_t =3D=3D BPF_MOD_CALL ? emit_call(&prog, new_addr, ip) : emit_jump(&prog, new_addr, ip); if (ret) @@ -640,8 +641,15 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_tex= t_poke_type t, return ret; } =20 -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, - void *old_addr, void *new_addr) +static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, + void *old_addr, void *new_addr) +{ + return ___bpf_arch_text_poke(ip, t, t, old_addr, new_addr); +} + +int bpf_arch_text_poke_type(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { if (!is_kernel_text((long)ip) && !is_bpf_text_address((long)ip)) @@ -655,7 +663,13 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_ty= pe t, if (is_endbr(ip)) ip +=3D ENDBR_INSN_SIZE; =20 - return __bpf_arch_text_poke(ip, t, old_addr, new_addr); + return ___bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr); +} + +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, + void *old_addr, void *new_addr) +{ + return bpf_arch_text_poke_type(ip, t, t, old_addr, new_addr); } =20 #define EMIT_LFENCE() EMIT3(0x0F, 0xAE, 0xE8) --=20 2.51.2 From nobody Sun Feb 8 10:05:23 2026 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B85F30B515 for ; Fri, 14 Nov 2025 09:25:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112337; cv=none; b=t4Ovxe/v2Pk7iANtcutRirHIx2lrlMIFGTprWcIOSCEI11MA34zkU+Q3aSMicKm3g3BpqKfpA4JyRm+2cmvWkDwnzx40owiciW40AHlpe68dFwQ38NdQZu2buBAhsmjMxkVmtfPQC/RjU+81w+wlG/KfEo55xYcJv9NEOt2r/Hw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763112337; c=relaxed/simple; bh=TIjsGgErGjhnsODM1Ph/ITY27XoQ96fhmRIiGKpLPac=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mcQlcjoGR9zAtKWPLE7aqX839g46w32kkuOEjh0DgzkDbehLRSBcRKh2+I77IDpgbjSQ7PnBRiR+p2+FzyyN5QdgjH8PZiDa20B/HFjY2yLV+SXldeXmS1wNpL2YSxiejtiDPKRlu5RftZdtQ5Uc8PpheweTIbnxk5CRoRNZ640= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yzd8SLfN; arc=none smtp.client-ip=209.85.214.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yzd8SLfN" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-2956d816c10so19131125ad.1 for ; Fri, 14 Nov 2025 01:25:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763112334; x=1763717134; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0GH1XgjhxGvH7eOvtS78BjGizHMW/kgqOUJowiQXUBA=; b=Yzd8SLfNYUGICSEwn2XJUh+BrNTaUlFEeE2/2izq4OQlev4UEVBlmPG9MumotdLadi 6VfjXjDSQtHlJpcq7X9DfD6RQt4PBKeNr5+xBQmZEVu+GADo1eM3CdM89Rxk9tZsWo5m vIndcC9wXByLWG44c9AUDHCCOOXcQ2QDcXCN17zaEzR0mz3IYugeTpppNXpL+Hn2vdyF 84Si5QUrCjnhR5Tumgz2lSCWvv/6vMXr3e5OQwqcTGQ7MzN7aDbjkWMQhphMMVawxged 8gb6dUxrsi7tKKyvjaMvSK3SUV+W6jASh/7AzvpcqCyx2dbUBzfOZvWdCJBeTWEwqTgi PMxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763112334; x=1763717134; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=0GH1XgjhxGvH7eOvtS78BjGizHMW/kgqOUJowiQXUBA=; b=WQfcYl+lyEW0r75zJp0DL5nCfMi08DYARQVbyJUMjCgWifwaznVi2ZsOUcN5IyNOny kduDKsFPOoBiYFVmSUjGGxm2oXwb3z2DhOftvzB8HKlgHsfKzSnk3dbS50Je1B4S6CzA lXs7qiJH8ombmAsvDLY3AuUOc4YxXrU7P5zGIDpnz0eJzXGg90UBtE8AFrQHFZbPP1x8 0lTYazPgQ7uIlaLgFeoAmE37Hq9+xxAZJijFuHbf9UOe7hc/BVMLuVBY1LpJEH29PvMR /68R5ysZathvtzw/CEo9dAqS7HVG31WTUFXuJu+MshBltLeIvVVKO0+15gGuBEgAYQMz JG7w== X-Forwarded-Encrypted: i=1; AJvYcCV+m7xoszdwczUDuStV0idMWvtYlmsG+dIV9Wkev7kOJtwpOws5goCJlppHR35PDr6w7J+e7j/AcJDwiQk=@vger.kernel.org X-Gm-Message-State: AOJu0YyTIbiOR4LnKILrUCt4XThrFJTDt1rt69wa9GWD/NkJOBgJH2We EHOMGG+8WRXtBtn32XmZ9dxg3sbx6rosi2402wGsJVJx+bTC7RED5T4n X-Gm-Gg: ASbGncvCGOchkg9ik/UnbvLTKRMC5g/lmyyby7twQbWvwtio0RSyjy45vaN08Leaffs IO8f1FF6Cw6Ykzu5io+zd53lsG8PS0+Ct7qlj1GUrtCDaDUl8djN1EzTjR+QINbxtLC6TWAW7FZ I6oONKgYGbU+lPMTRC1xkj+yG/shP+DngDJCpfOEMThERtMvlpbQUL44finND38NcvYkqCGvrvV nURaqZgvO35Yb7oq5Kdu7j3quOAf3AN1dhLKgR5f1Zw+RDgzhjA1fhz1LeDYPWiulQngP5U1tJR otkJQTmDutntFjqu5m550BiCBwPhuBpmtoE1pLpF2uKrubgUfNyTpEQ4C7XnD0uDQTG5pNcdGnT Ipu7hZUf1jE+pb0GY74p+EaldEoVdZV6cse78KK+4v1Ys6+s/z9SGcifAxYa232s7naacQrBU1u EE X-Google-Smtp-Source: AGHT+IH8JdBL4kgG2wlYs/vtfYVs3OXl9lPqSBwJDQ1zGrYP08Gk8O6hyA6D59Y+JlLxX9WZiLPJ4w== X-Received: by 2002:a17:903:2f0e:b0:295:8c51:64ff with SMTP id d9443c01a7336-2986a7420ebmr27102945ad.29.1763112334321; Fri, 14 Nov 2025 01:25:34 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2985c2346dasm50451525ad.7.2025.11.14.01.25.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Nov 2025 01:25:33 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC bpf-next 7/7] bpf: implement "jmp" mode for trampoline Date: Fri, 14 Nov 2025 17:24:50 +0800 Message-ID: <20251114092450.172024-8-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251114092450.172024-1-dongml2@chinatelecom.cn> References: <20251114092450.172024-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the "jmp" mode for the bpf trampoline. For the ftrace_managed case, we need only to set the FTRACE_OPS_FL_JMP on the tr->fops if "jmp" is needed. For the bpf poke case, the new flag BPF_TRAMP_F_JMPED is introduced to store and check if the trampoline is in the "jmp" mode. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 6 +++++ kernel/bpf/trampoline.c | 53 ++++++++++++++++++++++++++++++++++------- 2 files changed, 50 insertions(+), 9 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index aec7c65539f5..3598785ac8d1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1201,6 +1201,12 @@ struct btf_func_model { */ #define BPF_TRAMP_F_INDIRECT BIT(8) =20 +/* + * Indicate that the trampoline is using "jmp" instead of "call". This flag + * is only used in the !ftrace_managed case. + */ +#define BPF_TRAMP_F_JMPED BIT(9) + /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. */ diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 5949095e51c3..02a9f33d8f6c 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -175,15 +175,37 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u= 64 key) return tr; } =20 -static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) +static int bpf_text_poke(struct bpf_trampoline *tr, void *old_addr, + void *new_addr) { + enum bpf_text_poke_type new_t =3D BPF_MOD_CALL, old_t =3D BPF_MOD_CALL; void *ip =3D tr->func.addr; int ret; =20 + if (bpf_trampoline_need_jmp(tr->flags)) + new_t =3D BPF_MOD_JUMP; + if (tr->flags & BPF_TRAMP_F_JMPED) + old_t =3D BPF_MOD_JUMP; + + ret =3D bpf_arch_text_poke_type(ip, old_t, new_t, old_addr, new_addr); + if (!ret) { + if (new_t =3D=3D BPF_MOD_JUMP) + tr->flags |=3D BPF_TRAMP_F_JMPED; + else + tr->flags &=3D ~BPF_TRAMP_F_JMPED; + } + + return ret; +} + +static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) +{ + int ret; + if (tr->func.ftrace_managed) ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); else - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); + ret =3D bpf_text_poke(tr, old_addr, NULL); =20 return ret; } @@ -191,7 +213,6 @@ static int unregister_fentry(struct bpf_trampoline *tr,= void *old_addr) static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *= new_addr, bool lock_direct_mutex) { - void *ip =3D tr->func.addr; int ret; =20 if (tr->func.ftrace_managed) { @@ -200,7 +221,7 @@ static int modify_fentry(struct bpf_trampoline *tr, voi= d *old_addr, void *new_ad else ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); + ret =3D bpf_text_poke(tr, old_addr, new_addr); } return ret; } @@ -223,7 +244,7 @@ static int register_fentry(struct bpf_trampoline *tr, v= oid *new_addr) ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); ret =3D register_ftrace_direct(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); + ret =3D bpf_text_poke(tr, NULL, new_addr); } =20 return ret; @@ -415,7 +436,8 @@ static int bpf_trampoline_update(struct bpf_trampoline = *tr, bool lock_direct_mut } =20 /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ - tr->flags &=3D (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX); + tr->flags &=3D (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX | + BPF_TRAMP_F_JMPED); =20 if (tlinks[BPF_TRAMP_FEXIT].nr_links || tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) { @@ -432,9 +454,17 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS again: - if ((tr->flags & BPF_TRAMP_F_SHARE_IPMODIFY) && - (tr->flags & BPF_TRAMP_F_CALL_ORIG)) - tr->flags |=3D BPF_TRAMP_F_ORIG_STACK; + if (tr->flags & BPF_TRAMP_F_CALL_ORIG) { + if (tr->flags & BPF_TRAMP_F_SHARE_IPMODIFY) { + tr->flags |=3D BPF_TRAMP_F_ORIG_STACK; + } else if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_JMP)) { + /* Use "jmp" instead of "call" for the trampoline + * in the origin call case, and we don't need to + * skip the frame. + */ + tr->flags &=3D ~BPF_TRAMP_F_SKIP_FRAME; + } + } #endif =20 size =3D arch_bpf_trampoline_size(&tr->func.model, tr->flags, @@ -465,6 +495,11 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut if (err) goto out_free; =20 + if (bpf_trampoline_need_jmp(tr->flags)) + tr->fops->flags |=3D FTRACE_OPS_FL_JMP; + else + tr->fops->flags &=3D ~FTRACE_OPS_FL_JMP; + WARN_ON(tr->cur_image && total =3D=3D 0); if (tr->cur_image) /* progs already running at this address */ --=20 2.51.2