From nobody Tue Dec 2 02:49:47 2025 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 151763546E7 for ; Tue, 18 Nov 2025 12:36:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469417; cv=none; b=nUq++g6IlHbBZmSl5T8gT95T3YVZyuhG9+p8lHh0lkwzH4ZXbePBSOhI4MYb53C46IDxpZSkLgrm+WVN4+/ADAW13SHcH1J/KKgkNvoB+VH6wuRVolbQJ7uVQ+ZKuRzWltjwKv6YxcZv9P0EsB4cmPJmglpnfFrorm6owqGJxlM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469417; c=relaxed/simple; bh=HfdqWCvCXsXMTJZ0UXs4yz02O0KD4yDebfnbUfc9UBI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FRhuTpB+MNLvElAWMVzNQyO1SruET3R4YflLDWe1PZJZXDSzEIoyHXNNR40Gdeg0mkFJ+trLtThoAfnfyQfljL24MTkilIifB7X1rMfqzbTe+IREhi7gxBsg+HbYJre1z+2b2bRVWwx5rnZuOaP+ot409ekK6VD+xgFfYjvaH2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LBSSvhsI; arc=none smtp.client-ip=209.85.210.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LBSSvhsI" Received: by mail-pf1-f194.google.com with SMTP id d2e1a72fcca58-7b22ffa2a88so4465395b3a.1 for ; Tue, 18 Nov 2025 04:36:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469415; x=1764074215; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cGjFhDeMxWrGazpljFY/Z83jYWiFR4RMAA/Pm2GV6ZI=; b=LBSSvhsI1sqOJPY7QAQUtWmbtl419gPdfOjJSFQDG6LzPdUeCnZqt0Mf4xYV//cBD+ puymsRS1DcZlug9FKhG2mm+2fGhbW7Gb4Jh9k5lFzhjv5ROnlPyhcjOviVE3aPW/Cci7 C3lMqFQ1tzBX0hy/1668QvsDO1Zq+D/WvtsFlO1Tpe9NUDgBEnAUEmaHCgGqzfTndGTq N3uAOPb4hsRbuHGFo2hNwj1ooD7aBMMK/kxhWGsURSO0NpyAecmstap2hNjRWpxhIlBF UBtd51TxmKFBDpMwsGD2D+QGb19nxsUFGv/9RcbaJKZPRilhTgRFoRE0zzQ8ZToW7EzM /9eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469415; x=1764074215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=cGjFhDeMxWrGazpljFY/Z83jYWiFR4RMAA/Pm2GV6ZI=; b=pwF1Vmyd0JgXhTqFVJG36yxNQE5k5SFOscbhWslQYOheKz7DT9bqiXFzOdFfItU6ML XqRyXq6GrLo1ZipIGOECVyOih1siDGrhYynZM0cdv6VBoxKKt8gkcFPyYAtq6oNHHH86 EBX8jb7hNfUz94LkwYz1hqPg1Xq1fkEcPBao2AwfhNwJwdwvsI0Y+4R33qRv4IWnURpA AMYFTfCO8ZN5vOJtfS1RCX4AHE5fT2lotJDwIwhzOVhPBG42+30RJgrIYfJ1rP30X0Bm TrrAC8rep1cEhISXzlGdCP/aZI/tRiViS+Tz5i3rIWGEvC2JOPfxHuqnQi4AjF9crh2p cGnQ== X-Forwarded-Encrypted: i=1; AJvYcCUd10rLJWgpxFp71gzkfxLYxNGhLQt5vP+RG7eMuctwsfmH6lNRjPIyRBdf66m8ziHP3nDrTdqlBV3wwgs=@vger.kernel.org X-Gm-Message-State: AOJu0YwAWC26+NhwVkc6av4I+jawf0Scs8RL9piQKfGvFARP3vOBqp09 nAh2npenUMLa3XvZ7/NNC0u97sjTK8MSErnnjPffDoEjjd4pwVW4Wg0I X-Gm-Gg: ASbGncuHRC7QYqt895O+zWcUZQgRo0VrLxtx1CoHOJBxKGFwsdbFJv5rNhM4BFssOJf Ojupo7DQitQuwOtsczjv86JhRXMgViqrOUj08HvQ0MOzwvYA9H4ZBqKVUTAGKKZfdm9ONDNz1+l twTqf6qzbt604SJhpsGGazqdyaT0zUyCWYE1yifG1SRSxhgyfPZ1XLQxHVexOTYLpv20m3sKU1F dpKO+uQ8mGPP8oojz3vpKBJ/B7/CRhhk3Idj3OdG/T0ziiXQGDJZrYV28C98OXPZuodPOQBfvXw T8Mp4qtRYUSeG7h5yF1otcHyeKnIZbHK2fE9jfqYtdPJ9QgOh5PhhK2FQK4MvFUu4AINi8kg8kF 9Dok8cPPGUHGW4hKLu7P2tITo5uzUzP7y57XdlKsaWKcUrIbBDJIJ/B8oDnlt1mWHlL7JUmyoF7 hh9pYleCswQPg= X-Google-Smtp-Source: AGHT+IHzC6fTh7q+8Nly65oI1W6f1opZTTs/1Rfi00Y9lq9IK/uPekC1SfgGQME1iV9wmsttAbLwug== X-Received: by 2002:a05:6a00:21c5:b0:7ad:f6e8:d013 with SMTP id d2e1a72fcca58-7ba3cd668d4mr21966720b3a.32.1763469415307; Tue, 18 Nov 2025 04:36:55 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:36:55 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 1/6] ftrace: introduce FTRACE_OPS_FL_JMP Date: Tue, 18 Nov 2025 20:36:29 +0800 Message-ID: <20251118123639.688444-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, the "nop" will be replaced with a "call" instruction when a function is hooked by the ftrace. However, sometimes the "call" can break the RSB and introduce extra overhead. Therefore, introduce the flag FTRACE_OPS_FL_JMP, which indicate that the ftrace_ops should be called with a "jmp" instead of "call". For now, it is only used by the direct call case. When a direct ftrace_ops is marked with FTRACE_OPS_FL_JMP, the last bit of the ops->direct_call will be set to 1. Therefore, we can tell if we should use "jmp" for the callback in ftrace_call_replace(). Signed-off-by: Menglong Dong --- v3: - reject if the addr is already "jmp" in register_ftrace_direct() and __modify_ftrace_direct() --- include/linux/ftrace.h | 33 +++++++++++++++++++++++++++++++++ kernel/trace/Kconfig | 12 ++++++++++++ kernel/trace/ftrace.c | 17 ++++++++++++++++- 3 files changed, 61 insertions(+), 1 deletion(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 07f8c309e432..015dd1049bea 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -359,6 +359,7 @@ enum { FTRACE_OPS_FL_DIRECT =3D BIT(17), FTRACE_OPS_FL_SUBOP =3D BIT(18), FTRACE_OPS_FL_GRAPH =3D BIT(19), + FTRACE_OPS_FL_JMP =3D BIT(20), }; =20 #ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS @@ -577,6 +578,38 @@ static inline void arch_ftrace_set_direct_caller(struc= t ftrace_regs *fregs, unsigned long addr) { } #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return addr & 1; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr | 1UL; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr & ~1UL; +} +#else +static inline bool ftrace_is_jmp(unsigned long addr) +{ + return false; +} + +static inline unsigned long ftrace_jmp_set(unsigned long addr) +{ + return addr; +} + +static inline unsigned long ftrace_jmp_get(unsigned long addr) +{ + return addr; +} +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_JMP */ + #ifdef CONFIG_STACK_TRACER =20 int stack_trace_sysctl(const struct ctl_table *table, int write, void *buf= fer, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index d2c79da81e4f..4661b9e606e0 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -80,6 +80,12 @@ config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE If the architecture generates __patchable_function_entries sections but does not want them included in the ftrace locations. =20 +config HAVE_DYNAMIC_FTRACE_WITH_JMP + bool + help + If the architecture supports to replace the __fentry__ with a + "jmp" instruction. + config HAVE_SYSCALL_TRACEPOINTS bool help @@ -330,6 +336,12 @@ config DYNAMIC_FTRACE_WITH_ARGS depends on DYNAMIC_FTRACE depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS =20 +config DYNAMIC_FTRACE_WITH_JMP + def_bool y + depends on DYNAMIC_FTRACE + depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS + depends on HAVE_DYNAMIC_FTRACE_WITH_JMP + config FPROBE bool "Kernel Function Probe (fprobe)" depends on HAVE_FUNCTION_GRAPH_FREGS && HAVE_FTRACE_GRAPH_FUNC diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 59cfacb8a5bb..bbb37c0f8c6c 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5951,7 +5951,8 @@ static void remove_direct_functions_hash(struct ftrac= e_hash *hash, unsigned long for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { del =3D __ftrace_lookup_ip(direct_functions, entry->ip); - if (del && del->direct =3D=3D addr) { + if (del && ftrace_jmp_get(del->direct) =3D=3D + ftrace_jmp_get(addr)) { remove_hash_entry(direct_functions, del); kfree(del); } @@ -6016,8 +6017,15 @@ int register_ftrace_direct(struct ftrace_ops *ops, u= nsigned long addr) if (ftrace_hash_empty(hash)) return -EINVAL; =20 + /* This is a "raw" address, and this should never happen. */ + if (WARN_ON_ONCE(ftrace_is_jmp(addr))) + return -EINVAL; + mutex_lock(&direct_mutex); =20 + if (ops->flags & FTRACE_OPS_FL_JMP) + addr =3D ftrace_jmp_set(addr); + /* Make sure requested entries are not already registered.. */ size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { @@ -6138,6 +6146,13 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsig= ned long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 + /* This is a "raw" address, and this should never happen. */ + if (WARN_ON_ONCE(ftrace_is_jmp(addr))) + return -EINVAL; + + if (ops->flags & FTRACE_OPS_FL_JMP) + addr =3D ftrace_jmp_set(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; --=20 2.51.2 From nobody Tue Dec 2 02:49:47 2025 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E72220C490 for ; Tue, 18 Nov 2025 12:37:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469423; cv=none; b=DrBQcStJJrwCiWDr6SjSQjvQ475l12b9haLEtzobNwMZ/7gtshu32rLcdGei8VieHAY+OlAgiil8X4jnkWUTcYExilpHjFx4dXA108tyhA4ieBZ2K0hfacD3IoqxEER2gv51MI9empcoD9UB4YeG0f6f+HSmFulWeH0IMVsVyno= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469423; c=relaxed/simple; bh=yw0+14vG3MTkylBTsNGEjI1rTbbl51z2DrBpLhZnx4s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iLAOURJCgMTMx1A0/aaikDEtbWHSWFQePYpWEZXEpwm2vjV08S7HsoZFl1JhUeqLgx7ifGddAw2M6ryLNYTBxrDQr9SICWClpCTS+RNqak8NP6rOefd11GKW1v7fXOQi7L0EIo0VDu+wOPq8t5/Yl1lI95wG6wDpNUN4gekFjn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XJ6PDHPU; arc=none smtp.client-ip=209.85.210.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XJ6PDHPU" Received: by mail-pf1-f196.google.com with SMTP id d2e1a72fcca58-7baf61be569so3873797b3a.3 for ; Tue, 18 Nov 2025 04:37:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469421; x=1764074221; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KRkWsGzjZIg7j77yjUEwdtDGoZTowRE1mWBAeLLFj3U=; b=XJ6PDHPU/43Hl58ZLURBIl0aD1bLDiWd/ZXkuC7Ivd8A5Xl6SCaLUOPPzeOFplUUmY 8bdgrxGb8WfYWTclxDAQadta7nXpRXIwXDQTojnC1jqxEMU68wqJQ263XQShenChUM2l 6cY5SqlwkYbQ2xeOB9OsVEK7X8hle3Dngo6TttlwgMW/uQp8iFcRoUXDOVfJMRP1iNgZ GgAATu4QyyiSv9E8Tkzh/Ppl+O17bRcDTfESNfST2L7t+gjZNYRcAnH5imdeQLjno5AH RDbezKUFs2Kw5vxcwS6UHhnxV6MZfGHt/YwH9RkE4CZqlBNBiYKkpESBmKuFcFEIq/Na AQmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469421; x=1764074221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=KRkWsGzjZIg7j77yjUEwdtDGoZTowRE1mWBAeLLFj3U=; b=hXk6K2EkPio4bY7k2LZaNh0NDFBFwkwRMTzR+YwlnTIZB7CGu6koTzPTPm4p/1DhOj GkXLZs813kiFxa8O+m4zfQZfqkFQnx7sVqNV6qBweKZRpi77APJg+K7bnLGJPckAfZu9 YYFq+UlNkqCNc4H/65ztbP2szLSMzMPhiQkQgFbWTKgLyGT9Mk9XakCdBrhjydY2R7Jm XZeJwWhd1zmWJotApSzaBkozIfLRNPirdtcRD7lOhTtS3weC6J0EMFAPGTzTXNxwriqS /GV8xprEWl/ugDJgFllOw5NtHHPM77Jp7L6TIMK16ZjQV4gXSoeMk/PJnhLUqCugajwu CN8w== X-Forwarded-Encrypted: i=1; AJvYcCUAZctqRmZtUqHxDYJtFZxtQV7psuHZ54XE31Io1nZqkYJXJHyCtpgnkeN7dA0WkaeTeRVAiPPZ5VSevYk=@vger.kernel.org X-Gm-Message-State: AOJu0YxrMCt1KMIb47M8c7GqJCl8YAtrKSef0SdmWkls0y5IVmw2CPcY LIPeCCdlzeNCLMe/5NdzMbTLgPxmVh4IIfF/WQM1oEZiZiW26X2zAXYp X-Gm-Gg: ASbGncsepPxYbEPAyVhOX1wwf7MeffI5Sr8dSligcz6I4THsazI14oNUuXvFinR+x6X E1Q4+YlysK9surtRID+O64///nUOVlPFFtk8lKi1ew3/0MFxFg64KqD7Gt8fld7/0Kdc2Xi9vnw NBtiGtWTClby9WTPItt6oj5Jt9Y0zsPsDFsZAOdErJGyKog/hcs077QCVjRxz3fraOIgsYBT6+8 CoQceNUm278qVIrB5ySGYZ86RBA+WpZ0owBlh8hvIYujJttzdqU3k1APvfHVf1yTnZZD5gvH0Kc ZcgE6dNMLNYcctSwlg9elMCKhdQdGeZcBlT0GtcrWm817eCGMvK0IoNO0ri5HJjJFhZvHJXFbqq xa/cTGW6lUHR3wblKWQk8hI3/YcU4CaQqONAqrVL0RUnQ3XgnNBl0rn3/ryqTzqSa1jpPRzX2/o khgIJa3nCJjbE= X-Google-Smtp-Source: AGHT+IHLeC9ZWRL2VLtRiE54fQ6H+bxg9/eru6ZBoKg3v7IDmaLDmUU9coK6XPoveKCEmhT4BZJJbA== X-Received: by 2002:aa7:888c:0:b0:77f:2dc4:4c16 with SMTP id d2e1a72fcca58-7ba3be8c664mr17780783b3a.21.1763469421399; Tue, 18 Nov 2025 04:37:01 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.36.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:37:01 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 2/6] x86/ftrace: implement DYNAMIC_FTRACE_WITH_JMP Date: Tue, 18 Nov 2025 20:36:30 +0800 Message-ID: <20251118123639.688444-3-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the DYNAMIC_FTRACE_WITH_JMP for x86_64. In ftrace_call_replace, we will use JMP32_INSN_OPCODE instead of CALL_INSN_OPCODE if the address should use "jmp". Meanwhile, adjust the direct call in the ftrace_regs_caller. The RSB is balanced in the "jmp" mode. Take the function "foo" for example: original_caller: call foo -> foo: call fentry -> fentry: [do ftrace callbacks ] move tramp_addr to stack RET -> tramp_addr tramp_addr: [..] call foo_body -> foo_body: [..] RET -> back to tramp_addr [..] RET -> back to original_caller Signed-off-by: Menglong Dong --- arch/x86/Kconfig | 1 + arch/x86/kernel/ftrace.c | 7 ++++++- arch/x86/kernel/ftrace_64.S | 12 +++++++++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fa3b616af03a..462250a20311 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -230,6 +230,7 @@ config X86 select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 select HAVE_FTRACE_REGS_HAVING_PT_REGS if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + select HAVE_DYNAMIC_FTRACE_WITH_JMP if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64 select HAVE_EBPF_JIT diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 4450acec9390..0543b57f54ee 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -74,7 +74,12 @@ static const char *ftrace_call_replace(unsigned long ip,= unsigned long addr) * No need to translate into a callthunk. The trampoline does * the depth accounting itself. */ - return text_gen_insn(CALL_INSN_OPCODE, (void *)ip, (void *)addr); + if (ftrace_is_jmp(addr)) { + addr =3D ftrace_jmp_get(addr); + return text_gen_insn(JMP32_INSN_OPCODE, (void *)ip, (void *)addr); + } else { + return text_gen_insn(CALL_INSN_OPCODE, (void *)ip, (void *)addr); + } } =20 static int ftrace_verify_code(unsigned long ip, const char *old_code) diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 823dbdd0eb41..a132608265f6 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -285,8 +285,18 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR RET =20 +1: + testb $1, %al + jz 2f + andq $0xfffffffffffffffe, %rax + movq %rax, MCOUNT_REG_SIZE+8(%rsp) + restore_mcount_regs + /* Restore flags */ + popfq + RET + /* Swap the flags with orig_rax */ -1: movq MCOUNT_REG_SIZE(%rsp), %rdi +2: movq MCOUNT_REG_SIZE(%rsp), %rdi movq %rdi, MCOUNT_REG_SIZE-8(%rsp) movq %rax, MCOUNT_REG_SIZE(%rsp) =20 --=20 2.51.2 From nobody Tue Dec 2 02:49:47 2025 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B63AF351FA9 for ; Tue, 18 Nov 2025 12:37:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469430; cv=none; b=NTaEEHDWfM0W7zec2/Lh6hkPLaL+jv0E0s39H7Lf7bPlYpfIX7pNFo4alGObslmRxi07Rf5QIZNxW/Ji4G4lQAmql59VFyXyP3snQ/7bKucS/umpl8oul2QWPxuHMs8T3wSBWE578H806DR3iiMkV/Vf47gVTyOXJrOqRZpYatk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469430; c=relaxed/simple; bh=CuZzfhxpaY2RwGDUn1Vx9bQnsJNO8QHP2g34k19NaVo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gwaZzlNZLJcQZmFM+58fuefVnE+tHwBB9JGpO/Tmv96MmnnNXsJy1pSAmC1LWBHUA7E033UHmF7iHaF9rnCHEXqwEmL7PpsflqD0OfBw6bnU6xuOYVmVGrvBi8H0oHteWtfg8f9ItXIFUfZ80mjbxgzTudz2WKRunAHrzr+2gM0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=F/FOs5sp; arc=none smtp.client-ip=209.85.210.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="F/FOs5sp" Received: by mail-pf1-f196.google.com with SMTP id d2e1a72fcca58-7bb710d1d1dso1629550b3a.1 for ; Tue, 18 Nov 2025 04:37:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469428; x=1764074228; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Gr5zFCvf3ujgW8K/MBn3EvZvT1zNiYJuY8EfOcbbjUE=; b=F/FOs5spanaAWg1eMuStZ0swyfWqGCjFh3BNVwH8MObwzotT48DICKhvWuX+Yd1mhx jBpKQxxFcWEeFYDz/F8w/zLigNcE8//nXMsAH4q8mYl7AbZ7d7eUgLAEGG91QlVt7Kyl rkZTuk8KYu639iPWBf1hxj6vl42p70b90ds9++2VgJ6ZUfR8ZQmTbPnbUL3yz6gMZm3n FTtvTKe/ReebLrdyDvvS4Dw3/hGMX3sN4P56pYi9tFK0sXIZkiAxnKrzVoYHQUdnD4Lv d516NKodA0yCA+53y/w1XXIS8MHZylmkqq1ZpVYTGHjEENLNnB4lDrqRRfbE5s37Mwso UKKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469428; x=1764074228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Gr5zFCvf3ujgW8K/MBn3EvZvT1zNiYJuY8EfOcbbjUE=; b=FP+YrAYYwzXGN7LTfDhvMgW/yksD8Qp810tL6KU4L7f0sd11N+DNBRfC6ARFLrwnsP o9MZpoEmNe97HDgLFc/6bLdLIap83at3rH4jwpsfrlHuNFWB9F+QkqOQmsuxMkYNGqLS rCgG8CpAnXnGbHVq/Li1AwSev7KCP7ZDDnFEF73Z7d1zzimU/cD8se+FBor1wQnYjj9S Pe7qb6EyzVifX101BQ/FGR4KxjNlbYqcHJqFO6oxCBds5gpXAhpVmBe+PpBJ3NzrX7uo KLntWQxRvPzeSGYHLDz2WcYUIgoTFflvAb5zVuc/oPq39YYjlrqEyLKQr1ji7K36XGxg 2qdQ== X-Forwarded-Encrypted: i=1; AJvYcCVfkjGhezrnqGCDdgZq/LoDmix0xaE1J8QYlJrHa/XAFAIwqG5fi+l3ZW0l87CBxifI8tkZCP0VYlSXgH0=@vger.kernel.org X-Gm-Message-State: AOJu0YxTZJT3Pr/IkH57/e05oFAmtz78vTl9IaqnCbhV/uvTqP2MXLDK ZdORv4PbvYv7IBuxyLNc3fO78H8DMQ03h1QtVm4hTSgKWVqjdkCqBjXA X-Gm-Gg: ASbGncvGYdc3uz3FOl0RilIJHybQYSkAZ7SGqY/rkGn+8+2ELNeI8yV6yQ35VJ7pn6F 35k4wYUWBCx5e3R0LY45us1rn37uYUWF78ZuhxrJPOsfEwkfKkKHmFXvOmOV3Ieojs3hYkQ8899 dP8EAuuZoGu7L29CuwBbr4o10EW3yowCvYe473jJwG8WFh6E6AMyjvbxVNco7Lp490q9NZNsPKq rkNS9SS1lPoeto68xSl6cSW8mjCbU0nP98no4GpziWp+5vuF0StXx+VU65UPY0l2qZ2kf+VbJan eyvvQHrVZpkLCniuvTWFoyjZ/Wzd/tNxahlh78655apjjmWHqz+/YSeqzm8HgFtJWGRv8DBD+XZ J2LVCbUF3kDBf+h3uQTebIefLmaClIJBJt6sU1xWyZFn6GnxdcXGWenvZzZowHef2MdwR48sTiJ seMFaNA2D12Gk= X-Google-Smtp-Source: AGHT+IH3lGATztfAevxX/+Crp6j/wkxIW4bZ1e3m6KQ8glDX3xAvYvs6wWCH2053VFn2llBG12hFJg== X-Received: by 2002:a05:6a00:2408:b0:7a9:c21a:559a with SMTP id d2e1a72fcca58-7ba39bc0c02mr18696620b3a.8.1763469428198; Tue, 18 Nov 2025 04:37:08 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.37.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:37:07 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 3/6] bpf: fix the usage of BPF_TRAMP_F_SKIP_FRAME Date: Tue, 18 Nov 2025 20:36:31 +0800 Message-ID: <20251118123639.688444-4-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some places calculate the origin_call by checking if BPF_TRAMP_F_SKIP_FRAME is set. However, it should use BPF_TRAMP_F_ORIG_STACK for this propose. Just fix them. Signed-off-by: Menglong Dong Acked-by: Alexei Starovoitov --- arch/riscv/net/bpf_jit_comp64.c | 2 +- arch/x86/net/bpf_jit_comp.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp6= 4.c index 45cbc7c6fe49..21c70ae3296b 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1131,7 +1131,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, store_args(nr_arg_slots, args_off, ctx); =20 /* skip to actual body of traced function */ - if (flags & BPF_TRAMP_F_SKIP_FRAME) + if (flags & BPF_TRAMP_F_ORIG_STACK) orig_call +=3D RV_FENTRY_NINSNS * 4; =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 36a0d4db9f68..808d4343f6cf 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3289,7 +3289,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im =20 arg_stack_off =3D stack_size; =20 - if (flags & BPF_TRAMP_F_SKIP_FRAME) { + if (flags & BPF_TRAMP_F_CALL_ORIG) { /* skip patched call instruction and point orig_call to actual * body of the kernel function. */ --=20 2.51.2 From nobody Tue Dec 2 02:49:47 2025 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CAB331076D for ; Tue, 18 Nov 2025 12:37:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469436; cv=none; b=C4HS+dPIAv89se7aYXIXuonkgiu2Lzrnt1uHd3CCDZNExGMKU6nPISCbu3WgA1cOmRwriXL4lnZWc5YtpUytX48nD8qUWWnh7NisEE6GcqjR/eAIcYpqwHFM027mZ6ugROFLk3ZA0Wb7zFLo2KugWOYURVdksARU9wbeGnsnS+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469436; c=relaxed/simple; bh=6lH5TThLt0iLUPf10EryfpP2LJ24tamdXbqQbNSY8i0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qjeZI8wGnjDLNy5KYcBBoGXa6Rv6+VktqvMghO7RjWWrxCUTYrBnts0Xss/PXj4FlkgdE83xkg7GOV+j5MPd9Yf5beGHjSICcQaG0IdkY6ndqGsodcHPnyjg9Obc/yKYmiuJ+726ll+PxqdlJvdL7zSB9EfuOnkWj5o/GauT/LE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ux+GpTaA; arc=none smtp.client-ip=209.85.210.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ux+GpTaA" Received: by mail-pf1-f193.google.com with SMTP id d2e1a72fcca58-7ba49f92362so4261801b3a.1 for ; Tue, 18 Nov 2025 04:37:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469434; x=1764074234; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Og8VZW3HRIC9jFqJjaOXGmdza1e4WZ/zf5HErz8QF0g=; b=Ux+GpTaAqZ7Lf5x0guIF+CpU2qC8RAa3PJ0KGmpnxl00r5UKn8FSsFajBXrewiDZQC M9rUx0qCdKLpsDsEbfWaNKdkMJvye7Fus/clloP44buHM69z2FbROzhshCvfbSzgSRYn L7N11+qPk1Co8xTmrA061UrB7BaAh4MA9q7oLDCZYsdH90aG5F24mACT9m//wO9QkPrW /ENnjwT1CswCw8G1sqxdUgWJ8YFWkbugYiTr6uqK7OnFuF4+WX8cHNgizVQe7YAjAZLG b6DHV1FDR0XN8djl1Oem09XvvJeOuZxjGtMlLrrEoPKvpVYnClBtwdRokOnyM8OdgiBP EYqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469434; x=1764074234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Og8VZW3HRIC9jFqJjaOXGmdza1e4WZ/zf5HErz8QF0g=; b=XIGPE2c1Wcj6zLiAHvICSuJuFpmMHsKfIuKv8fA4MPv/4fAzRIp557I+j8w3mGfNnf RTWV0PV87PyWwIvcB1ynSQfqK137eFYDUFxaGpY/7ceGsUxkLlzeFkB75K7yoAIHA6en uSaFqZ0ZLqrF5nnAwFvF+WQrgOa//hHZnRi5vbX9T7UYp4U/eBBqNY21/cMTc9yLBkgT KwnrN3x4lhxYzn3Z3D9Gx0Cw5hsPqmAYTCRJtxDLyWq7kOdgXP4lV7rUeTWqkwgJwsVJ e3uzl95LTL6aLHw0FwxoB4gP17KxrK2vYKAg21lpM37b1AkYU5c+1RGVKeGEYDGRb1gN yPqQ== X-Forwarded-Encrypted: i=1; AJvYcCVxpvGOt2fl1FFY68zqmAJSPmSj633L53mC2rfIxrQmagY1aC8adlh9PqPnJ9ZFwSgasIu83GYddIObh3E=@vger.kernel.org X-Gm-Message-State: AOJu0YxWmO9lcKGYlkpB62M8oQTk8IQ8UZOpKgKUkOYWoHkbIRE3mfzn MjVLYR3Dv1NRorIBX/OqtdQoL2vNd49b05gLYeolYK1wMsyx2LMPwNPN X-Gm-Gg: ASbGncumh9+v44Bc+POcF7AORXfmIBkweYZTsJuLbAuk7Xf34wHKQ0mQGGObEhqJ6Jf RIe69vtYYNIiCx7eo/Foh6nvbkqiY/X4yo0cC1JIb+uU9mzE293EbJhBtrs3e47fdaDcQXGNXVn AwOu10fiEhtmhcBmBKEftj2BN5ib9dY0GtmCWEd1ABmTSQJOdI6bptAELanwwPdDuHXT8TS0tMk azJ+zXy0CrCR3dfvpWG1bWvyBeJMHIJ50kNATd1j+jvrQGhhhHDwq2UMDqcEvE1aUECy/rClmE1 G06sM75BGzmLFdRW+BsholX79o2K6jKCdPVL+Ne7DHeJUXuaMw63i+HVbQ4XbOdCe0hJUBDFieN 6HetUPFLKrmNdSsCWwWMBa3fqItkJsuO/VKsA4T4M/5yAL+QBZ5knMt6xO5MU+/7cJtuQ6/xQOI Ic5RDlOxgMPOw= X-Google-Smtp-Source: AGHT+IFzVQjXBpjmRtPxHUA4jCnIvxaMRfFlmVFQe/HhGKJNurpODbRokRgaX1qNfPIoEbksNhY5aQ== X-Received: by 2002:a05:6a21:9989:b0:35d:3533:3dd2 with SMTP id adf61e73a8af0-361152010d6mr4202690637.0.1763469433693; Tue, 18 Nov 2025 04:37:13 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.37.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:37:13 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 4/6] bpf,x86: adjust the "jmp" mode for bpf trampoline Date: Tue, 18 Nov 2025 20:36:32 +0800 Message-ID: <20251118123639.688444-5-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the origin call case, if BPF_TRAMP_F_SKIP_FRAME is not set, it means that the trampoline is not called, but "jmp". Introduce the function bpf_trampoline_use_jmp() to check if the trampoline is in "jmp" mode. Do some adjustment on the "jmp" mode for the x86_64. The main adjustment that we make is for the stack parameter passing case, as the stack alignment logic changes in the "jmp" mode without the "rip". What's more, the location of the parameters on the stack also changes. Signed-off-by: Menglong Dong --- v2: - rename bpf_trampoline_need_jmp() to bpf_trampoline_use_jmp() --- arch/x86/net/bpf_jit_comp.c | 16 +++++++++++----- include/linux/bpf.h | 12 ++++++++++++ 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 808d4343f6cf..632a83381c2d 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2847,9 +2847,10 @@ static int get_nr_used_regs(const struct btf_func_mo= del *m) } =20 static void save_args(const struct btf_func_model *m, u8 **prog, - int stack_size, bool for_call_origin) + int stack_size, bool for_call_origin, u32 flags) { int arg_regs, first_off =3D 0, nr_regs =3D 0, nr_stack_slots =3D 0; + bool use_jmp =3D bpf_trampoline_use_jmp(flags); int i, j; =20 /* Store function arguments to stack. @@ -2890,7 +2891,7 @@ static void save_args(const struct btf_func_model *m,= u8 **prog, */ for (j =3D 0; j < arg_regs; j++) { emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, - nr_stack_slots * 8 + 0x18); + nr_stack_slots * 8 + 16 + (!use_jmp) * 8); emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); =20 @@ -3284,7 +3285,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im * should be 16-byte aligned. Following code depend on * that stack_size is already 8-byte aligned. */ - stack_size +=3D (stack_size % 16) ? 0 : 8; + if (bpf_trampoline_use_jmp(flags)) { + /* no rip in the "jmp" case */ + stack_size +=3D (stack_size % 16) ? 8 : 0; + } else { + stack_size +=3D (stack_size % 16) ? 0 : 8; + } } =20 arg_stack_off =3D stack_size; @@ -3344,7 +3350,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -ip_off); } =20 - save_args(m, &prog, regs_off, false); + save_args(m, &prog, regs_off, false, flags); =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { /* arg1: mov rdi, im */ @@ -3377,7 +3383,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { restore_regs(m, &prog, regs_off); - save_args(m, &prog, arg_stack_off, true); + save_args(m, &prog, arg_stack_off, true, flags); =20 if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { /* Before calling the original function, load the diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 09d5dc541d1c..4187b7578580 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1264,6 +1264,18 @@ typedef void (*bpf_trampoline_exit_t)(struct bpf_pro= g *prog, u64 start, bpf_trampoline_enter_t bpf_trampoline_enter(const struct bpf_prog *prog); bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog); =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP +static inline bool bpf_trampoline_use_jmp(u64 flags) +{ + return flags & BPF_TRAMP_F_CALL_ORIG && !(flags & BPF_TRAMP_F_SKIP_FRAME); +} +#else +static inline bool bpf_trampoline_use_jmp(u64 flags) +{ + return false; +} +#endif + struct bpf_ksym { unsigned long start; unsigned long end; --=20 2.51.2 From nobody Tue Dec 2 02:49:47 2025 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D74233563CE for ; Tue, 18 Nov 2025 12:37:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469445; cv=none; b=bLTM+BUJRO74Rl2vZ+lVWv4dXN41EB+WQcrcKcY4GhlVUp1Vd50O4MNbB1qKYAhNR2/9pm385jbqCukgrn3WxU+TD30uKY0n/FdBYN+owGGyk+9sOuRE0U+JEJfAHGpRaI69oD1u+yc5Oo4FsrDwse80MtUtmEnRZ/RvVMOrjQs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469445; c=relaxed/simple; bh=A189KruAXLdLXnVG6BxgtUuscbMj8iOLzy3Zi4GQuZk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ise2IsWjWF0Bp28W9mOcDd4Mb5I9uFOi9KjjDD8VRGj07y73wHBqFMSPno0YHxDUSrVJWcZMSzmd1NNOgM5/SX2Gv973Ya5vUNqxWtfCATUR9OkQiMvhQLUnZjW3eIoynahDOVPGcgLn2bhwQBA5HSTOFMl4L25yNEsbEQ8kbhQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OVOei3An; arc=none smtp.client-ip=209.85.210.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OVOei3An" Received: by mail-pf1-f196.google.com with SMTP id d2e1a72fcca58-7ba55660769so3639845b3a.1 for ; Tue, 18 Nov 2025 04:37:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469441; x=1764074241; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b2DQCf6tSRMTRKkx+h6Y6ZXBkfTNVyQ5trU9zCAP1Bg=; b=OVOei3Anc9xdJdzzpxFS1gH3c8EAGMKUtNyxW0BMV15H8Z9VFy3Wrct1xjz7d7V99S +XrNWFHVfQG+tfyJgXiDgtb1c24hJZBTc2lUCaEJoZylZ9TfxUhqPqpkN9gps7LmAJNJ lOEw37l0/fteqz8WIi47Sbo2DRHqHEEhPdSTGwV+AxVrw36cglZqzLlwlh1HoNwx469T 4EePjGeB9BL8r3QERphyyYLqprfX9QOvYOy/IINOET9eedOIg25pM43PKH3Y3LudY2gZ 8XxVX5bud9SJzxf+8FNEWhrJApCaS8SrfyB6Thg+PlC61wXIEHS+M42K9lN2WxLhkS0U RfmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469441; x=1764074241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=b2DQCf6tSRMTRKkx+h6Y6ZXBkfTNVyQ5trU9zCAP1Bg=; b=i7fREOJIJytSZt0lSA9O+Jn8cjOqkF3Jg0eICNmbYUK8JdlrQMgsPJMPajpiFjGwax Pw7Tp/20L+5aZeEi0cHSa5jtuPa8i34egPzlD9i9Iccc9zgkeSrsqRJl2hn9ijxJEcOi C6Z0ixRu4CEV7Iroh2yMfV0+cmYfbytYrjzpvLEEvLnTK4dKLdXEQarwVVgult+e6xIn 7/b2/ZwRrjRSY89V+x0sVbv9Px3zFseLwDsh/kvTg54L8extIY4tPkFb2/pqUrTPCaRh t5gKHNYnLQbLwNIy3saELS4nf7OJojukpB5m0w2LKB2JTt5efJIoHfzYPwExE/pfmuSv 63Dg== X-Forwarded-Encrypted: i=1; AJvYcCVQ3swKBB1XJ85bTeTCC7PQH4WPH8AZy2J0g8lGjqN65QfCL9+IDwJiJ+LkMFeizX0LNIL1xTX9Hzeln2Q=@vger.kernel.org X-Gm-Message-State: AOJu0YwOCdY/fNgHWu+JG+3XkWZzUwPch81MNc3bA8q/2jvVg/7Uzbwc 5+GP8p7rSgietBNQstq9+z4kZHUgpLRFBkG4YZ9O/5yL7RDn04bl5wyR X-Gm-Gg: ASbGncse0DsW4XbPVtpFnJ6IZMArQpPkccZZY08ECVw8v0Lb9d4UkiEiJU5c1aLebg4 PG9CM2lWEJnemRrp8di/cPTLvLgcyCVFQMm2GpSsqAOb/4Z1MBCHV4/w6DW8qWFeoADv98BmTP6 nG9wfhVPhmpBAbAYwz4NCFL7fyWKMd206IfZFYdHwnSg4WrAyoyql8EPoFyI6JXxQQLPKfuUv2J Xqf0mzz4zXNoszMYYS5YN0mBmrCk0xtKPb1I/QEurP5eaoW7nLmyB7jLXK7ghdSYj4mvekTa3ge BnfGHVFwnd2oBjT3dt8+kzJexBtz2LQLR0e3r+Hg5Cot86On3uQDgvO8KhqQHhvz2yX1pluhevx 0uIvg1KKhk3L29kPVbx9L06vZ5IF6yCyRiR08vQXyH5MZVXTmF33EEWDKjK5JBindtNrtEFCE5p qKKNyozv7I3Zw= X-Google-Smtp-Source: AGHT+IHYF1kg7lEYD3hrKWG9nsBvEr03AjbqXaDOXUHhDgx1fssUvuChaG0Qu/zvI8hSD1oVLBViJg== X-Received: by 2002:a05:6a20:a125:b0:342:9cb7:64a3 with SMTP id adf61e73a8af0-35ba22a4fcdmr21086580637.34.1763469441102; Tue, 18 Nov 2025 04:37:21 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.37.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:37:20 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 5/6] bpf: specify the old and new poke_type for bpf_arch_text_poke Date: Tue, 18 Nov 2025 20:36:33 +0800 Message-ID: <20251118123639.688444-6-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the origin logic, the bpf_arch_text_poke() assume that the old and new instructions have the same opcode. However, they can have different opcode if we want to replace a "call" insn with a "jmp" insn. Therefore, add the new function parameter "old_t" along with the "new_t", which are used to indicate the old and new poke type. Meanwhile, adjust the implement of bpf_arch_text_poke() for all the archs. "BPF_MOD_NOP" is added to make the code more readable. In bpf_arch_text_poke(), we still check if the new and old address is NULL to determine if nop insn should be used, which I think is more safe. Signed-off-by: Menglong Dong --- v3: - fix compile error in powerpc v2: - add new function parameter to bpf_arch_text_poke instead of introduce bpf_arch_text_poke_type() --- arch/arm64/net/bpf_jit_comp.c | 14 ++++++------- arch/loongarch/net/bpf_jit.c | 9 +++++--- arch/powerpc/net/bpf_jit_comp.c | 10 +++++---- arch/riscv/net/bpf_jit_comp64.c | 9 +++++--- arch/s390/net/bpf_jit_comp.c | 7 ++++--- arch/x86/net/bpf_jit_comp.c | 37 +++++++++++++++++++-------------- include/linux/bpf.h | 6 ++++-- kernel/bpf/core.c | 5 +++-- kernel/bpf/trampoline.c | 20 ++++++++++++------ 9 files changed, 71 insertions(+), 46 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 0c9a50a1e73e..c64df579b7e0 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2923,8 +2923,9 @@ static int gen_branch_or_nop(enum aarch64_insn_branch= _type type, void *ip, * The dummy_tramp is used to prevent another CPU from jumping to unknown * locations during the patching process, making the patching process easi= er. */ -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, - void *old_addr, void *new_addr) +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { int ret; u32 old_insn; @@ -2968,14 +2969,13 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke= _type poke_type, !poking_bpf_entry)) return -EINVAL; =20 - if (poke_type =3D=3D BPF_MOD_CALL) - branch_type =3D AARCH64_INSN_BRANCH_LINK; - else - branch_type =3D AARCH64_INSN_BRANCH_NOLINK; - + branch_type =3D old_t =3D=3D BPF_MOD_CALL ? AARCH64_INSN_BRANCH_LINK : + AARCH64_INSN_BRANCH_NOLINK; if (gen_branch_or_nop(branch_type, ip, old_addr, plt, &old_insn) < 0) return -EFAULT; =20 + branch_type =3D new_t =3D=3D BPF_MOD_CALL ? AARCH64_INSN_BRANCH_LINK : + AARCH64_INSN_BRANCH_NOLINK; if (gen_branch_or_nop(branch_type, ip, new_addr, plt, &new_insn) < 0) return -EFAULT; =20 diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c index cbe53d0b7fb0..2e7dacbbef5c 100644 --- a/arch/loongarch/net/bpf_jit.c +++ b/arch/loongarch/net/bpf_jit.c @@ -1284,11 +1284,12 @@ void *bpf_arch_text_copy(void *dst, void *src, size= _t len) return ret ? ERR_PTR(-EINVAL) : dst; } =20 -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, - void *old_addr, void *new_addr) +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { int ret; - bool is_call =3D (poke_type =3D=3D BPF_MOD_CALL); + bool is_call; u32 old_insns[LOONGARCH_LONG_JUMP_NINSNS] =3D {[0 ... 4] =3D INSN_NOP}; u32 new_insns[LOONGARCH_LONG_JUMP_NINSNS] =3D {[0 ... 4] =3D INSN_NOP}; =20 @@ -1298,6 +1299,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype poke_type, if (!is_bpf_text_address((unsigned long)ip)) return -ENOTSUPP; =20 + is_call =3D old_t =3D=3D BPF_MOD_CALL; ret =3D emit_jump_or_nops(old_addr, ip, old_insns, is_call); if (ret) return ret; @@ -1305,6 +1307,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype poke_type, if (memcmp(ip, old_insns, LOONGARCH_LONG_JUMP_NBYTES)) return -EFAULT; =20 + is_call =3D new_t =3D=3D BPF_MOD_CALL; ret =3D emit_jump_or_nops(new_addr, ip, new_insns, is_call); if (ret) return ret; diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_com= p.c index 88ad5ba7b87f..5e976730b2f5 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -1107,8 +1107,9 @@ static void do_isync(void *info __maybe_unused) * execute isync (or some CSI) so that they don't go back into the * trampoline again. */ -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, - void *old_addr, void *new_addr) +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { unsigned long bpf_func, bpf_func_end, size, offset; ppc_inst_t old_inst, new_inst; @@ -1119,7 +1120,6 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype poke_type, return -EOPNOTSUPP; =20 bpf_func =3D (unsigned long)ip; - branch_flags =3D poke_type =3D=3D BPF_MOD_CALL ? BRANCH_SET_LINK : 0; =20 /* We currently only support poking bpf programs */ if (!__bpf_address_lookup(bpf_func, &size, &offset, name)) { @@ -1132,7 +1132,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype poke_type, * an unconditional branch instruction at im->ip_after_call */ if (offset) { - if (poke_type !=3D BPF_MOD_JUMP) { + if (old_t =3D=3D BPF_MOD_CALL || new_t =3D=3D BPF_MOD_CALL) { pr_err("%s (0x%lx): calls are not supported in bpf prog body\n", __func= __, bpf_func); return -EOPNOTSUPP; @@ -1166,6 +1166,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype poke_type, } =20 old_inst =3D ppc_inst(PPC_RAW_NOP()); + branch_flags =3D old_t =3D=3D BPF_MOD_CALL ? BRANCH_SET_LINK : 0; if (old_addr) { if (is_offset_in_branch_range(ip - old_addr)) create_branch(&old_inst, ip, (unsigned long)old_addr, branch_flags); @@ -1174,6 +1175,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype poke_type, branch_flags); } new_inst =3D ppc_inst(PPC_RAW_NOP()); + branch_flags =3D new_t =3D=3D BPF_MOD_CALL ? BRANCH_SET_LINK : 0; if (new_addr) { if (is_offset_in_branch_range(ip - new_addr)) create_branch(&new_inst, ip, (unsigned long)new_addr, branch_flags); diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp6= 4.c index 21c70ae3296b..5f9457e910e8 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -852,17 +852,19 @@ static int gen_jump_or_nops(void *target, void *ip, u= 32 *insns, bool is_call) return emit_jump_and_link(is_call ? RV_REG_T0 : RV_REG_ZERO, rvoff, false= , &ctx); } =20 -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, - void *old_addr, void *new_addr) +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { u32 old_insns[RV_FENTRY_NINSNS], new_insns[RV_FENTRY_NINSNS]; - bool is_call =3D poke_type =3D=3D BPF_MOD_CALL; + bool is_call; int ret; =20 if (!is_kernel_text((unsigned long)ip) && !is_bpf_text_address((unsigned long)ip)) return -ENOTSUPP; =20 + is_call =3D old_t =3D=3D BPF_MOD_CALL; ret =3D gen_jump_or_nops(old_addr, ip, old_insns, is_call); if (ret) return ret; @@ -870,6 +872,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_typ= e poke_type, if (memcmp(ip, old_insns, RV_FENTRY_NBYTES)) return -EFAULT; =20 + is_call =3D new_t =3D=3D BPF_MOD_CALL; ret =3D gen_jump_or_nops(new_addr, ip, new_insns, is_call); if (ret) return ret; diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index cf461d76e9da..a2072cabba76 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2413,8 +2413,9 @@ bool bpf_jit_supports_far_kfunc_call(void) return true; } =20 -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, - void *old_addr, void *new_addr) +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { struct bpf_plt expected_plt, current_plt, new_plt, *plt; struct { @@ -2431,7 +2432,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_t= ype t, if (insn.opc !=3D (0xc004 | (old_addr ? 0xf0 : 0))) return -EINVAL; =20 - if (t =3D=3D BPF_MOD_JUMP && + if ((new_t =3D=3D BPF_MOD_JUMP || old_t =3D=3D BPF_MOD_JUMP) && insn.disp =3D=3D ((char *)new_addr - (char *)ip) >> 1) { /* * The branch already points to the destination, diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 632a83381c2d..b69dc7194e2c 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -597,7 +597,8 @@ static int emit_jump(u8 **pprog, void *func, void *ip) return emit_patch(pprog, func, ip, 0xE9); } =20 -static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, +static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, void *new_addr) { const u8 *nop_insn =3D x86_nops[5]; @@ -607,9 +608,9 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text= _poke_type t, int ret; =20 memcpy(old_insn, nop_insn, X86_PATCH_SIZE); - if (old_addr) { + if (old_t !=3D BPF_MOD_NOP && old_addr) { prog =3D old_insn; - ret =3D t =3D=3D BPF_MOD_CALL ? + ret =3D old_t =3D=3D BPF_MOD_CALL ? emit_call(&prog, old_addr, ip) : emit_jump(&prog, old_addr, ip); if (ret) @@ -617,9 +618,9 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text= _poke_type t, } =20 memcpy(new_insn, nop_insn, X86_PATCH_SIZE); - if (new_addr) { + if (new_t !=3D BPF_MOD_NOP && new_addr) { prog =3D new_insn; - ret =3D t =3D=3D BPF_MOD_CALL ? + ret =3D new_t =3D=3D BPF_MOD_CALL ? emit_call(&prog, new_addr, ip) : emit_jump(&prog, new_addr, ip); if (ret) @@ -640,8 +641,9 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text= _poke_type t, return ret; } =20 -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, - void *old_addr, void *new_addr) +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { if (!is_kernel_text((long)ip) && !is_bpf_text_address((long)ip)) @@ -655,7 +657,7 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_typ= e t, if (is_endbr(ip)) ip +=3D ENDBR_INSN_SIZE; =20 - return __bpf_arch_text_poke(ip, t, old_addr, new_addr); + return __bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr); } =20 #define EMIT_LFENCE() EMIT3(0x0F, 0xAE, 0xE8) @@ -897,12 +899,13 @@ static void bpf_tail_call_direct_fixup(struct bpf_pro= g *prog) target =3D array->ptrs[poke->tail_call.key]; if (target) { ret =3D __bpf_arch_text_poke(poke->tailcall_target, - BPF_MOD_JUMP, NULL, + BPF_MOD_NOP, BPF_MOD_JUMP, + NULL, (u8 *)target->bpf_func + poke->adj_off); BUG_ON(ret < 0); ret =3D __bpf_arch_text_poke(poke->tailcall_bypass, - BPF_MOD_JUMP, + BPF_MOD_JUMP, BPF_MOD_NOP, (u8 *)poke->tailcall_target + X86_PATCH_SIZE, NULL); BUG_ON(ret < 0); @@ -3985,6 +3988,7 @@ void bpf_arch_poke_desc_update(struct bpf_jit_poke_de= scriptor *poke, struct bpf_prog *new, struct bpf_prog *old) { u8 *old_addr, *new_addr, *old_bypass_addr; + enum bpf_text_poke_type t; int ret; =20 old_bypass_addr =3D old ? NULL : poke->bypass_addr; @@ -3997,21 +4001,22 @@ void bpf_arch_poke_desc_update(struct bpf_jit_poke_= descriptor *poke, * the kallsyms check. */ if (new) { + t =3D old_addr ? BPF_MOD_JUMP : BPF_MOD_NOP; ret =3D __bpf_arch_text_poke(poke->tailcall_target, - BPF_MOD_JUMP, + t, BPF_MOD_JUMP, old_addr, new_addr); BUG_ON(ret < 0); if (!old) { ret =3D __bpf_arch_text_poke(poke->tailcall_bypass, - BPF_MOD_JUMP, + BPF_MOD_JUMP, BPF_MOD_NOP, poke->bypass_addr, NULL); BUG_ON(ret < 0); } } else { + t =3D old_bypass_addr ? BPF_MOD_JUMP : BPF_MOD_NOP; ret =3D __bpf_arch_text_poke(poke->tailcall_bypass, - BPF_MOD_JUMP, - old_bypass_addr, + t, BPF_MOD_JUMP, old_bypass_addr, poke->bypass_addr); BUG_ON(ret < 0); /* let other CPUs finish the execution of program @@ -4020,9 +4025,9 @@ void bpf_arch_poke_desc_update(struct bpf_jit_poke_de= scriptor *poke, */ if (!ret) synchronize_rcu(); + t =3D old_addr ? BPF_MOD_JUMP : BPF_MOD_NOP; ret =3D __bpf_arch_text_poke(poke->tailcall_target, - BPF_MOD_JUMP, - old_addr, NULL); + t, BPF_MOD_NOP, old_addr, NULL); BUG_ON(ret < 0); } } diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 4187b7578580..d5e2af29c7c8 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3708,12 +3708,14 @@ static inline u32 bpf_xdp_sock_convert_ctx_access(e= num bpf_access_type type, #endif /* CONFIG_INET */ =20 enum bpf_text_poke_type { + BPF_MOD_NOP, BPF_MOD_CALL, BPF_MOD_JUMP, }; =20 -int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, - void *addr1, void *addr2); +int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr); =20 void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke, struct bpf_prog *new, struct bpf_prog *old); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ef4448f18aad..c8ae6ab31651 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3150,8 +3150,9 @@ int __weak skb_copy_bits(const struct sk_buff *skb, i= nt offset, void *to, return -EFAULT; } =20 -int __weak bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, - void *addr1, void *addr2) +int __weak bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, + enum bpf_text_poke_type new_t, void *old_addr, + void *new_addr) { return -ENOTSUPP; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 04104397c432..0230ad19533e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -183,7 +183,8 @@ static int unregister_fentry(struct bpf_trampoline *tr,= void *old_addr) if (tr->func.ftrace_managed) ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); else - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); + ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, BPF_MOD_NOP, + old_addr, NULL); =20 return ret; } @@ -200,7 +201,10 @@ static int modify_fentry(struct bpf_trampoline *tr, vo= id *old_addr, void *new_ad else ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); + ret =3D bpf_arch_text_poke(ip, + old_addr ? BPF_MOD_CALL : BPF_MOD_NOP, + new_addr ? BPF_MOD_CALL : BPF_MOD_NOP, + old_addr, new_addr); } return ret; } @@ -225,7 +229,8 @@ static int register_fentry(struct bpf_trampoline *tr, v= oid *new_addr) return ret; ret =3D register_ftrace_direct(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); + ret =3D bpf_arch_text_poke(ip, BPF_MOD_NOP, BPF_MOD_CALL, + NULL, new_addr); } =20 return ret; @@ -336,8 +341,9 @@ static void bpf_tramp_image_put(struct bpf_tramp_image = *im) * call_rcu_tasks() is not necessary. */ if (im->ip_after_call) { - int err =3D bpf_arch_text_poke(im->ip_after_call, BPF_MOD_JUMP, - NULL, im->ip_epilogue); + int err =3D bpf_arch_text_poke(im->ip_after_call, BPF_MOD_NOP, + BPF_MOD_JUMP, NULL, + im->ip_epilogue); WARN_ON(err); if (IS_ENABLED(CONFIG_TASKS_RCU)) call_rcu_tasks(&im->rcu, __bpf_tramp_image_put_rcu_tasks); @@ -570,7 +576,8 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_= link *link, if (err) return err; tr->extension_prog =3D link->link.prog; - return bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL, + return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP, + BPF_MOD_JUMP, NULL, link->link.prog->bpf_func); } if (cnt >=3D BPF_MAX_TRAMP_LINKS) @@ -618,6 +625,7 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tram= p_link *link, if (kind =3D=3D BPF_TRAMP_REPLACE) { WARN_ON_ONCE(!tr->extension_prog); err =3D bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, + BPF_MOD_NOP, tr->extension_prog->bpf_func, NULL); tr->extension_prog =3D NULL; guard(mutex)(&tgt_prog->aux->ext_mutex); --=20 2.51.2 From nobody Tue Dec 2 02:49:47 2025 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B9C0354AD5 for ; Tue, 18 Nov 2025 12:37:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.65 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469449; cv=none; b=pLXnGao9RJLyIMzTMJdeKtOYgjV4E2bNIKNw6szKZdgjeMVnwmZ+VWG5oaUEFRq8L23mgzjAiwbug17hfk05yij2NxWPq+d/gLxz6JXUm6hwEfBTUqwhba0brILlFgptW5s18P/Q3U95o/nZPm57vZrqS4GxlCUV7Xc7uqfFPaA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763469449; c=relaxed/simple; bh=+LW1LDwlw5aMWSZC+RME7xkgV6VVEeYQmuAgzKDmvtU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ejuNUxBuUoPIlg4muexzgtgg1/JAozGPQvhL/mG5D1kJSiGT5AKbMQXAKpLj2nCEW8I9LfNOzJeFqmeuVLdj05O9o08i4auvSxUpfibR+Zwx52aKcNFHPhRDDduUYzS7y2/ga2wMvpyLwPA0hYPh6XNqwzL5kVJd7z+ilNrPEAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZdobFG89; arc=none smtp.client-ip=209.85.216.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZdobFG89" Received: by mail-pj1-f65.google.com with SMTP id 98e67ed59e1d1-3436cbb723fso4151673a91.2 for ; Tue, 18 Nov 2025 04:37:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763469446; x=1764074246; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GSVMHccR7qFQowMQrwveg0nRJjHYor4cBFPpOOVsWCI=; b=ZdobFG89XihdwouF7VTmof8NXxPPpnWRSF90gOqUX6zhmDlGfmbHn2i/29Wxo1sU2g ccGhFTB42xzYqgyitVQF/talHTSAEkDnbtmLPfu0tz+Dt5jOhujSUUT6j21NutwqBOmM 96+jO+OOtuMRVbqKtzc1Dofeiny0hjYohyA1lPERR0yBqmgDhr80rFn9sKlpjNJLVU6G LZufg3zl7ltmHNaAmVW5az43B5u/IYQ9IQx96g7lFtdKhtuqeKE3UtPUPeg4Y/8K/sO/ q5BWawOIq5k8Y7FeklGiIP0NsbL3NAuY3sxRkaUUoitmCJW/v4+1SaBwg4mAc2npL4yC 4zDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763469446; x=1764074246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=GSVMHccR7qFQowMQrwveg0nRJjHYor4cBFPpOOVsWCI=; b=wMaVBeNvncZAIWAX/YDlXRkC1tdCS+YrB2MPVrO4Ol21M+de9tR/XKgYLQLaa1Dztp R4GlB7d/n5oHb+TGLfIj5J6zpf/d6TrLhtCpRX0gu3aNLtxHYarTx4YzbXH14kOghipc 5IjzwQT070jSeCDdugPNlKKuo+2M2tnEqeDNAtgSFsL594rjpm2stdC+sxoFRGzIhBEH rXWGYQ9XcHW85M4OvFOw+JK/NLNXRfRme7F3uSaOyWdncPYxJ4I0tgaBZfoYGii/d+AC QggojyeLO1DcWjJrPWaLMocFq/SXGs79aGNMbV2zNNPhQYA7F0PWkN1xgTEQA8KFKz72 gXcw== X-Forwarded-Encrypted: i=1; AJvYcCXGxKLRWZXspICJ7ixSSqEAYWcaRWVGxy+W6b88YhKNkmava6WwO4uhhwE0ZgaktNP1PUGAgJPNrlNHKiI=@vger.kernel.org X-Gm-Message-State: AOJu0YxQFt/U4xYozXC20v2gbaG99YhgGc+jwgEh0hAMVbyzPvRGnoK+ 081DXnc8T3suqJzxTrd+poC7++hMvQdz7jbNgtAAkShF7YKlmWzKuDZf X-Gm-Gg: ASbGncsQY77AL0Ij8CvBe2lBZBC7qx8qkOrp+lSSmVbF3dbiZq7pY9S0SKZNP4evDol 0BnBdb47XutZjgKjQqz+WhCRgD3xPd96aO17lK5oTtoHPhtnUU6O8lZirGPMeEALzStM0eTYm2Q 5eHfgGMHTtMfaMTErYbkxSW7jSoFXRVDxc1NqcXQqQ/aTSzOvwm7BRMdZOMf905wZRqTGykftjz bVEOyjQTXG3vEYpgzKCPkdRiTiPLDdeXwKJGGesJrlfKO0LecrjHJQRBmE9298dYtP9+o2vSOGo XsN8005UaQs9Spjjs/UOtHuPPb/wT0695diTCDWQR3KYyGw6UcQvVkZWvmX3CJuo/aevcJrrj8s IXk/gZGXS6jQcJmGvnRii1Xg5fBv76Xw31UxIldIiDytchh1lQ2ODPIoIwthzEtpmWA+Z0V1IRX mkHLPWet9Cy/ys/UZzGlHnyA== X-Google-Smtp-Source: AGHT+IHlB7JZdM0iuSFbQkeMsYxejQo0JNDWZPI7r1VSRlyryqeoQgKWNGwlsPQuVu65rP/M3ItuMw== X-Received: by 2002:a17:90b:58cc:b0:343:684c:f8ad with SMTP id 98e67ed59e1d1-343f9e92249mr18880629a91.4.1763469446351; Tue, 18 Nov 2025 04:37:26 -0800 (PST) Received: from 7950hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7b92772e7f2sm16331496b3a.57.2025.11.18.04.37.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Nov 2025 04:37:26 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: ast@kernel.org, rostedt@goodmis.org Cc: daniel@iogearbox.net, john.fastabend@gmail.com, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH bpf-next v3 6/6] bpf: implement "jmp" mode for trampoline Date: Tue, 18 Nov 2025 20:36:34 +0800 Message-ID: <20251118123639.688444-7-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251118123639.688444-1-dongml2@chinatelecom.cn> References: <20251118123639.688444-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the "jmp" mode for the bpf trampoline. For the ftrace_managed case, we need only to set the FTRACE_OPS_FL_JMP on the tr->fops if "jmp" is needed. For the bpf poke case, we will check the origin poke type with the "origin_flags", and current poke type with "tr->flags". The function bpf_trampoline_update_fentry() is introduced to do the job. The "jmp" mode will only be enabled with CONFIG_DYNAMIC_FTRACE_WITH_JMP enabled and BPF_TRAMP_F_SHARE_IPMODIFY is not set. With BPF_TRAMP_F_SHARE_IPMODIFY, we need to get the origin call ip from the stack, so we can't use the "jmp" mode. Signed-off-by: Menglong Dong Acked-by: Steven Rostedt (Google) --- v3: - wrap the write to tr->fops->flags with CONFIG_DYNAMIC_FTRACE_WITH_JMP - reset BPF_TRAMP_F_SKIP_FRAME when the second try of modify_fentry in bpf_trampoline_update() v2: - rename bpf_text_poke to bpf_trampoline_update_fentry - remove the BPF_TRAMP_F_JMPED and check the current mode with the origin flags instead. --- kernel/bpf/trampoline.c | 75 +++++++++++++++++++++++++++++++---------- 1 file changed, 58 insertions(+), 17 deletions(-) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 0230ad19533e..976d89011b15 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -175,24 +175,42 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u= 64 key) return tr; } =20 -static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) +static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 ori= g_flags, + void *old_addr, void *new_addr) { + enum bpf_text_poke_type new_t =3D BPF_MOD_CALL, old_t =3D BPF_MOD_CALL; void *ip =3D tr->func.addr; + + if (!new_addr) + new_t =3D BPF_MOD_NOP; + else if (bpf_trampoline_use_jmp(tr->flags)) + new_t =3D BPF_MOD_JUMP; + + if (!old_addr) + old_t =3D BPF_MOD_NOP; + else if (bpf_trampoline_use_jmp(orig_flags)) + old_t =3D BPF_MOD_JUMP; + + return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr); +} + +static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, + void *old_addr) +{ int ret; =20 if (tr->func.ftrace_managed) ret =3D unregister_ftrace_direct(tr->fops, (long)old_addr, false); else - ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, BPF_MOD_NOP, - old_addr, NULL); + ret =3D bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL); =20 return ret; } =20 -static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *= new_addr, +static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, + void *old_addr, void *new_addr, bool lock_direct_mutex) { - void *ip =3D tr->func.addr; int ret; =20 if (tr->func.ftrace_managed) { @@ -201,10 +219,8 @@ static int modify_fentry(struct bpf_trampoline *tr, vo= id *old_addr, void *new_ad else ret =3D modify_ftrace_direct_nolock(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, - old_addr ? BPF_MOD_CALL : BPF_MOD_NOP, - new_addr ? BPF_MOD_CALL : BPF_MOD_NOP, - old_addr, new_addr); + ret =3D bpf_trampoline_update_fentry(tr, orig_flags, old_addr, + new_addr); } return ret; } @@ -229,8 +245,7 @@ static int register_fentry(struct bpf_trampoline *tr, v= oid *new_addr) return ret; ret =3D register_ftrace_direct(tr->fops, (long)new_addr); } else { - ret =3D bpf_arch_text_poke(ip, BPF_MOD_NOP, BPF_MOD_CALL, - NULL, new_addr); + ret =3D bpf_trampoline_update_fentry(tr, 0, NULL, new_addr); } =20 return ret; @@ -416,7 +431,7 @@ static int bpf_trampoline_update(struct bpf_trampoline = *tr, bool lock_direct_mut return PTR_ERR(tlinks); =20 if (total =3D=3D 0) { - err =3D unregister_fentry(tr, tr->cur_image->image); + err =3D unregister_fentry(tr, orig_flags, tr->cur_image->image); bpf_tramp_image_put(tr->cur_image); tr->cur_image =3D NULL; goto out; @@ -440,9 +455,20 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS again: - if ((tr->flags & BPF_TRAMP_F_SHARE_IPMODIFY) && - (tr->flags & BPF_TRAMP_F_CALL_ORIG)) - tr->flags |=3D BPF_TRAMP_F_ORIG_STACK; + if (tr->flags & BPF_TRAMP_F_CALL_ORIG) { + if (tr->flags & BPF_TRAMP_F_SHARE_IPMODIFY) { + /* The BPF_TRAMP_F_SKIP_FRAME can be cleared in the + * first try, reset it in the second try. + */ + tr->flags |=3D BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SKIP_FRAME; + } else if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_JMP)) { + /* Use "jmp" instead of "call" for the trampoline + * in the origin call case, and we don't need to + * skip the frame. + */ + tr->flags &=3D ~BPF_TRAMP_F_SKIP_FRAME; + } + } #endif =20 size =3D arch_bpf_trampoline_size(&tr->func.model, tr->flags, @@ -473,10 +499,18 @@ static int bpf_trampoline_update(struct bpf_trampolin= e *tr, bool lock_direct_mut if (err) goto out_free; =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP + if (bpf_trampoline_use_jmp(tr->flags)) + tr->fops->flags |=3D FTRACE_OPS_FL_JMP; + else + tr->fops->flags &=3D ~FTRACE_OPS_FL_JMP; +#endif + WARN_ON(tr->cur_image && total =3D=3D 0); if (tr->cur_image) /* progs already running at this address */ - err =3D modify_fentry(tr, tr->cur_image->image, im->image, lock_direct_m= utex); + err =3D modify_fentry(tr, orig_flags, tr->cur_image->image, + im->image, lock_direct_mutex); else /* first time registering */ err =3D register_fentry(tr, im->image); @@ -499,8 +533,15 @@ static int bpf_trampoline_update(struct bpf_trampoline= *tr, bool lock_direct_mut tr->cur_image =3D im; out: /* If any error happens, restore previous flags */ - if (err) + if (err) { tr->flags =3D orig_flags; +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP + if (bpf_trampoline_use_jmp(tr->flags)) + tr->fops->flags |=3D FTRACE_OPS_FL_JMP; + else + tr->fops->flags &=3D ~FTRACE_OPS_FL_JMP; +#endif + } kfree(tlinks); return err; =20 --=20 2.51.2