From nobody Tue Dec 2 02:58:13 2025 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 151763546E7 for ; Tue, 18 Nov 2025 12:36:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469417; cv=none; b=nUq++g6IlHbBZmSl5T8gT95T3YVZyuhG9+p8lHh0lkwzH4ZXbePBSOhI4MYb53C46IDxpZSkLgrm+WVN4+/ADAW13SHcH1J/KKgkNvoB+VH6wuRVolbQJ7uVQ+ZKuRzWltjwKv6YxcZv9P0EsB4cmPJmglpnfFrorm6owqGJxlM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469417; c=relaxed/simple; bh=HfdqWCvCXsXMTJZ0UXs4yz02O0KD4yDebfnbUfc9UBI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FRhuTpB+MNLvElAWMVzNQyO1SruET3R4YflLDWe1PZJZXDSzEIoyHXNNR40Gdeg0mkFJ+trLtThoAfnfyQfljL24MTkilIifB7X1rMfqzbTe+IREhi7gxBsg+HbYJre1z+2b2bRVWwx5rnZuOaP+ot409ekK6VD+xgFfYjvaH2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LBSSvhsI; arc=none smtp.client-ip=209.85.210.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LBSSvhsI" Received: by mail-pf1-f194.google.com with SMTP id d2e1a72fcca58-7b22ffa2a88so4465395b3a.1 for ; Tue, 18 Nov 2025 04:36:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469415; x=1764074215; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cGjFhDeMxWrGazpljFY/Z83jYWiFR4RMAA/Pm2GV6ZI=; b=LBSSvhsI1sqOJPY7QAQUtWmbtl419gPdfOjJSFQDG6LzPdUeCnZqt0Mf4xYV//cBD+ puymsRS1DcZlug9FKhG2mm+2fGhbW7Gb4Jh9k5lFzhjv5ROnlPyhcjOviVE3aPW/Cci7 C3lMqFQ1tzBX0hy/1668QvsDO1Zq+D/WvtsFlO1Tpe9NUDgBEnAUEmaHCgGqzfTndGTq N3uAOPb4hsRbuHGFo2hNwj1ooD7aBMMK/kxhWGsURSO0NpyAecmstap2hNjRWpxhIlBF UBtd51TxmKFBDpMwsGD2D+QGb19nxsUFGv/9RcbaJKZPRilhTgRFoRE0zzQ8ZToW7EzM /9eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469415; x=1764074215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=cGjFhDeMxWrGazpljFY/Z83jYWiFR4RMAA/Pm2GV6ZI=; b=pwF1Vmyd0JgXhTqFVJG36yxNQE5k5SFOscbhWslQYOheKz7DT9bqiXFzOdFfItU6ML XqRyXq6GrLo1ZipIGOECVyOih1siDGrhYynZM0cdv6VBoxKKt8gkcFPyYAtq6oNHHH86 EBX8jb7hNfUz94LkwYz1hqPg1Xq1fkEcPBao2AwfhNwJwdwvsI0Y+4R33qRv4IWnURpA AMYFTfCO8ZN5vOJtfS1RCX4AHE5fT2lotJDwIwhzOVhPBG42+30RJgrIYfJ1rP30X0Bm TrrAC8rep1cEhISXzlGdCP/aZI/tRiViS+Tz5i3rIWGEvC2JOPfxHuqnQi4AjF9crh2p cGnQ== X-Forwarded-Encrypted: i=1; AJvYcCUd10rLJWgpxFp71gzkfxLYxNGhLQt5vP+RG7eMuctwsfmH6lNRjPIyRBdf66m8ziHP3nDrTdqlBV3wwgs=@vger.kernel.org X-Gm-Message-State: AOJu0YwAWC26+NhwVkc6av4I+jawf0Scs8RL9piQKfGvFARP3vOBqp09 nAh2npenUMLa3XvZ7/NNC0u97sjTK8MSErnnjPffDoEjjd4pwVW4Wg0I X-Gm-Gg: ASbGncuHRC7QYqt895O+zWcUZQgRo0VrLxtx1CoHOJBxKGFwsdbFJv5rNhM4BFssOJf Ojupo7DQitQuwOtsczjv86JhRXMgViqrOUj08HvQ0MOzwvYA9H4ZBqKVUTAGKKZfdm9ONDNz1+l twTqf6qzbt604SJhpsGGazqdyaT0zUyCWYE1yifG1SRSxhgyfPZ1XLQxHVexOTYLpv20m3sKU1F dpKO+uQ8mGPP8oojz3vpKBJ/B7/CRhhk3Idj3OdG/T0ziiXQGDJZrYV28C98OXPZuodPOQBfvXw T8Mp4qtRYUSeG7h5yF1otcHyeKnIZbHK2fE9jfqYtdPJ9QgOh5PhhK2FQK4MvFUu4AINi8kg8kF 9Dok8cPPGUHGW4hKLu7P2tITo5uzUzP7y57XdlKsaWKcUrIbBDJIJ/B8oDnlt1mWHlL7JUmyoF7 hh9pYleCswQPg= X-Google-Smtp-Source: AGHT+IHzC6fTh7q+8Nly65oI1W6f1opZTTs/1Rfi00Y9lq9IK/uPekC1SfgGQME1iV9wmsttAbLwug== X-Received: by 2002:a05:6a00:21c5:b0:7ad:f6e8:d013 with SMTP id d2e1a72fcca58-7ba3cd668d4mr21966720b3a.32.1763469415307; Tue, 18 Nov 2025 04:36:55 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:36:55 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 1/6] ftrace: introduce FTRACE_OPS_FL_JMP Date: Tue, 18 Nov 2025 20:36:29 +0800 Message-ID: <20251118123639.688444-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, the "nop" will be replaced with a "call" instruction when a function is hooked by the ftrace. However, sometimes the "call" can break the RSB and introduce extra overhead. Therefore, introduce the flag FTRACE_OPS_FL_JMP, which indicate that the ftrace_ops should be called with a "jmp" instead of "call". For now, it is only used by the direct call case. When a direct ftrace_ops is marked with FTRACE_OPS_FL_JMP, the last bit of the ops->direct_call will be set to 1. Therefore, we can tell if we should use "jmp" for the callback in ftrace_call_replace(). Signed-off-by: Menglong Dong --- v3: - reject if the addr is already "jmp" in register_ftrace_direct() and __modify_ftrace_direct() --- include/linux/ftrace.h | 33 +++++++++++++++++++++++++++++++++ kernel/trace/Kconfig | 12 ++++++++++++ kernel/trace/ftrace.c | 17 ++++++++++++++++- 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 07f8c309e432..015dd1049bea 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -359,6 +359,7 @@ enum { FTRACE_OPS_FL_DIRECT =3D BIT(17), FTRACE_OPS_FL_SUBOP =3D BIT(18), FTRACE_OPS_FL_GRAPH =3D BIT(19), + FTRACE_OPS_FL_JMP =3D BIT(20), }; =20 #ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS @@ -577,6 +578,38 @@ static inline void arch_ftrace_set_direct_caller(struc= t ftrace_regs *fregs, unsigned long addr) { } #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return addr & 1; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr | 1UL; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr & ~1UL; +} +#else +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return false; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr; +} +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_JMP */ + #ifdef CONFIG_STACK_TRACER =20 int stack_trace_sysctl(const struct ctl_table *table, int write, void *buf= fer, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index d2c79da81e4f..4661b9e606e0 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -80,6 +80,12 @@ config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE If the architecture generates __patchable_function_entries sections but does not want them included in the ftrace locations. =20 +config HAVE_DYNAMIC_FTRACE_WITH_JMP + bool + help + If the architecture supports to replace the __fentry__ with a + "jmp" instruction. + config HAVE_SYSCALL_TRACEPOINTS bool help @@ -330,6 +336,12 @@ config DYNAMIC_FTRACE_WITH_ARGS depends on DYNAMIC_FTRACE depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS =20 +config DYNAMIC_FTRACE_WITH_JMP + def_bool y + depends on DYNAMIC_FTRACE + depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS + depends on HAVE_DYNAMIC_FTRACE_WITH_JMP + config FPROBE bool "Kernel Function Probe (fprobe)" depends on HAVE_FUNCTION_GRAPH_FREGS && HAVE_FTRACE_GRAPH_FUNC diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 59cfacb8a5bb..bbb37c0f8c6c 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5951,7 +5951,8 @@ static void remove_direct_functions_hash(struct ftrac= e_hash *hash, unsigned long for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { del =3D __ftrace_lookup_ip(direct_functions, entry->ip); - if (del && del->direct =3D=3D addr) { + if (del && ftrace_jmp_get(del->direct) =3D=3D + ftrace_jmp_get(addr)) { remove_hash_entry(direct_functions, del); kfree(del); } @@ -6016,8 +6017,15 @@ int register_ftrace_direct(struct ftrace_ops *ops, u= nsigned long addr) if (ftrace_hash_empty(hash)) return -EINVAL; =20 + /* This is a "raw" address, and this should never happen. */ + if (WARN_ON_ONCE(ftrace_is_jmp(addr))) + return -EINVAL; + mutex_lock(&direct_mutex); =20 + if (ops->flags & FTRACE_OPS_FL_JMP) + addr =3D ftrace_jmp_set(addr); + /* Make sure requested entries are not already registered.. */ size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { @@ -6138,6 +6146,13 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsig= ned long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 + /* This is a "raw" address, and this should never happen. */ + if (WARN_ON_ONCE(ftrace_is_jmp(addr))) + return -EINVAL; + + if (ops->flags & FTRACE_OPS_FL_JMP) + addr =3D ftrace_jmp_set(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; --=20 2.51.2