From nobody Sun Feb 8 13:27:23 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1AA7C334C26 for ; Thu, 18 Dec 2025 13:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766063245; cv=none; b=JHTsS+au0XPEet2eZor2I22h3i2ivP+uHuZ9S79O8etloQIGO4j5X45gpP9fJ51C9vyxX9ALmM7SWsYmqseWO33GYGuFdrTVnfBtCQblM2PnlOEnGW6muwc2TkzBO1QMkLCoSC99b+hQ6v1R96fvE3vY2Z3uj4k0rVgUICnaVNA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766063245; c=relaxed/simple; bh=zbKUAm5oIbrYUr4Pq0Qw8fCPvshez5LuSrD/UJz2biw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZuzONdBUcWIx5DNV7JK4ycEmOdsqIX8xKGBwghIK2kmGN68kdbkc7IRRrgBrE9EzekfCy+brWIme941q15hGcJFJeSyDpvSV6BJ9Tl/Ie0xK57BlLX60J2Bd6YznEK46iSkloCW6JLW3w+VTABisIYKPVzyxbFKzCJ1+Pis2bZU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=joh7WqP5; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="joh7WqP5" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2a102494058so8848505ad.0 for ; Thu, 18 Dec 2025 05:07:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766063241; x=1766668041; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xfWGrzFwQxVXKLSf8bK35Njw60Z0UXLSaG1EWbBTvoM=; b=joh7WqP5+e0bTINEg1VVmEsk5qcgnr/knqedVyFaR1w8zUYQkGUOcSWEiToU0S+F9Q Cn0UXcvFiZYfBR4q0PX5kqURxbWjLTxZtRX+eupKzm+BnP5no0+o1lFhhF/lQ8at2BQ3 Ap98lWv0QSYAKSvLi8VXIKN3obLOlQrIOUg629vBgt54Sq8B8uusGtfGeuUrmYqkfMcM 2BM7Jt4mQcx8S0A7UQVIFhn/vbnLcTR16ff5/EaoTdOViNjNW1zC/vdmPktiwaMJXbFw nARbCmklIBasTWFaxXVTK5o/0Qckp8NGkbsRyZGn/mhcbrqEynh81AabOMq0vU47QEn9 L0Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766063241; x=1766668041; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=xfWGrzFwQxVXKLSf8bK35Njw60Z0UXLSaG1EWbBTvoM=; b=HY3e2G3kUMtqA/Hj9Y8QNv1qoCSgyueOQEqwKlYK4LeOSIv1ha/EdHlOQyTisUvDRK 7yJEKjYOs1eNQ3szRWbuq7fE0DYKtNTF/9sIJLMufSpNTY8EHPhv/O+Vty0uo67RCfbg FX44LFehwg6eKI6m8o4r4T4vwkvkqf6TxZN5I+xmEtvD3GlCi6Nsmmmcb1FEG4H5lxuX D5eWaFCkD1ZXInCCN0YupmNuOBE+/jnktIv+31W/GgJJj1fijpURuJmEi+kmZUGGhjw0 iC6eusKAMttjXnseosVkC0exuJBQs5Gbi4f/I2xcSAE/gv9lEk+QoJM5rbU7a11J9uMY z/nA== X-Forwarded-Encrypted: i=1; AJvYcCUDwAW1CLHjn5u9bGr3uagSg+/8YLjRQzai36RMTGZwrJGa0ECjsk93IoewUrN5oT9AKopas9LRD4pWa1k=@vger.kernel.org X-Gm-Message-State: AOJu0YxGbPJ7nOI74gY1bkEQcaoNj7mtahF726dHoqQdAF6u9Pg7kMYm i5SjSYM99M2v7vefrrC2YkxBkUROLziIjYRgaGrOX3uLI+VZFAqfBFd4 X-Gm-Gg: AY/fxX70Oid5OmqTTEISlQ+sWkaPmAVtdHl8vAqpDo6LvMPXhpog5v37C0MxOunEuF+ DPwr5ixLFpFs/ZvCoEtmxhZbUk1KhhIHkzOJky2HSQzgBz9XhyLKnDvyQTh4UMZ/hkn3SXobDHj k27TguI/IGfmOvmuV6hhsxVTa1cTrraW8U1KKxoKpOyPuSHW5YrTBrDGcOzKBl0je3wyomCPfXG wXqBoUIIfjHhBQXSCnQDV3IxDrxrIRqkMfygSITCfedMsoIeQwQArf79HqUxBrjIVA8MXa44ShS /gN971QMxkwR8xcLdPXp4gRwWkVb4emmbIZ1oll2kFE+bZiVbUXYioZgW3dSomstyN2lcBGVv7G cBW5oenV3ReLmet5c/F9AWl3Vr0ltpbn0c2Jhcp2goeZiCeUJwSvefTTO1AdPEmIMvmMZTRlO5R 4isZADY5XR7Ge/HR9ugUiX2rD8qkecRhj2PqA= X-Google-Smtp-Source: AGHT+IE2dPr6IKueNkbs60vHXI/kEmriix+hjdpYj9PL6qm8QcZvc/lPn2FJdvkRkQDLRAi3HNJglg== X-Received: by 2002:a17:903:2452:b0:2a0:acb8:9e80 with SMTP id d9443c01a7336-2a2cac7e29cmr28263405ad.29.1766063241197; Thu, 18 Dec 2025 05:07:21 -0800 (PST) Received: from mi-ThinkStation-K.mioffice.cn ([43.224.245.232]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2d1926cd2sm25539905ad.77.2025.12.18.05.07.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Dec 2025 05:07:20 -0800 (PST) From: liujing40 X-Google-Original-From: liujing40 To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, mhiramat@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, liujing40@xiaomi.com Subject: [PATCH 1/2] bpf: Prepare for kprobe multi link fallback patch Date: Thu, 18 Dec 2025 21:06:28 +0800 Message-Id: <20251218130629.365398-2-liujing40@xiaomi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20251218130629.365398-1-liujing40@xiaomi.com> References: <20251218130629.365398-1-liujing40@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This moves some functions and structs around to make the following patch easier to read. Signed-off-by: Jing Liu --- kernel/trace/bpf_trace.c | 304 +++++++++++++++++++-------------------- 1 file changed, 152 insertions(+), 152 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index fe28d86f7c35..1fd07c10378f 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2291,67 +2291,6 @@ struct bpf_kprobe_multi_run_ctx { unsigned long entry_ip; }; =20 -struct user_syms { - const char **syms; - char *buf; -}; - -#ifndef CONFIG_HAVE_FTRACE_REGS_HAVING_PT_REGS -static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); -#define bpf_kprobe_multi_pt_regs_ptr() this_cpu_ptr(&bpf_kprobe_multi_pt_r= egs) -#else -#define bpf_kprobe_multi_pt_regs_ptr() (NULL) -#endif - -static unsigned long ftrace_get_entry_ip(unsigned long fentry_ip) -{ - unsigned long ip =3D ftrace_get_symaddr(fentry_ip); - - return ip ? : fentry_ip; -} - -static int copy_user_syms(struct user_syms *us, unsigned long __user *usym= s, u32 cnt) -{ - unsigned long __user usymbol; - const char **syms =3D NULL; - char *buf =3D NULL, *p; - int err =3D -ENOMEM; - unsigned int i; - - syms =3D kvmalloc_array(cnt, sizeof(*syms), GFP_KERNEL); - if (!syms) - goto error; - - buf =3D kvmalloc_array(cnt, KSYM_NAME_LEN, GFP_KERNEL); - if (!buf) - goto error; - - for (p =3D buf, i =3D 0; i < cnt; i++) { - if (__get_user(usymbol, usyms + i)) { - err =3D -EFAULT; - goto error; - } - err =3D strncpy_from_user(p, (const char __user *) usymbol, KSYM_NAME_LE= N); - if (err =3D=3D KSYM_NAME_LEN) - err =3D -E2BIG; - if (err < 0) - goto error; - syms[i] =3D p; - p +=3D err + 1; - } - - us->syms =3D syms; - us->buf =3D buf; - return 0; - -error: - if (err) { - kvfree(syms); - kvfree(buf); - } - return err; -} - static void kprobe_multi_put_modules(struct module **mods, u32 cnt) { u32 i; @@ -2360,12 +2299,6 @@ static void kprobe_multi_put_modules(struct module *= *mods, u32 cnt) module_put(mods[i]); } =20 -static void free_user_syms(struct user_syms *us) -{ - kvfree(us->syms); - kvfree(us->buf); -} - static void bpf_kprobe_multi_link_release(struct bpf_link *link) { struct bpf_kprobe_multi_link *kmulti_link; @@ -2469,6 +2402,152 @@ static const struct bpf_link_ops bpf_kprobe_multi_l= ink_lops =3D { #endif }; =20 +static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) +{ + struct bpf_kprobe_multi_run_ctx *run_ctx; + + run_ctx =3D container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ct= x, + session_ctx.run_ctx); + return run_ctx->entry_ip; +} + +struct modules_array { + struct module **mods; + int mods_cnt; + int mods_cap; +}; + +static int add_module(struct modules_array *arr, struct module *mod) +{ + struct module **mods; + + if (arr->mods_cnt =3D=3D arr->mods_cap) { + arr->mods_cap =3D max(16, arr->mods_cap * 3 / 2); + mods =3D krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KER= NEL); + if (!mods) + return -ENOMEM; + arr->mods =3D mods; + } + + arr->mods[arr->mods_cnt] =3D mod; + arr->mods_cnt++; + return 0; +} + +static bool has_module(struct modules_array *arr, struct module *mod) +{ + int i; + + for (i =3D arr->mods_cnt - 1; i >=3D 0; i--) { + if (arr->mods[i] =3D=3D mod) + return true; + } + return false; +} + +static int get_modules_for_addrs(struct module ***mods, unsigned long *add= rs, u32 addrs_cnt) +{ + struct modules_array arr =3D {}; + u32 i, err =3D 0; + + for (i =3D 0; i < addrs_cnt; i++) { + bool skip_add =3D false; + struct module *mod; + + scoped_guard(rcu) { + mod =3D __module_address(addrs[i]); + /* Either no module or it's already stored */ + if (!mod || has_module(&arr, mod)) { + skip_add =3D true; + break; /* scoped_guard */ + } + if (!try_module_get(mod)) + err =3D -EINVAL; + } + if (skip_add) + continue; + if (err) + break; + err =3D add_module(&arr, mod); + if (err) { + module_put(mod); + break; + } + } + + /* We return either err < 0 in case of error, ... */ + if (err) { + kprobe_multi_put_modules(arr.mods, arr.mods_cnt); + kfree(arr.mods); + return err; + } + + /* or number of modules found if everything is ok. */ + *mods =3D arr.mods; + return arr.mods_cnt; +} + +struct user_syms { + const char **syms; + char *buf; +}; + +#ifndef CONFIG_HAVE_FTRACE_REGS_HAVING_PT_REGS +static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); +#define bpf_kprobe_multi_pt_regs_ptr() this_cpu_ptr(&bpf_kprobe_multi_pt_r= egs) +#else +#define bpf_kprobe_multi_pt_regs_ptr() (NULL) +#endif + +static unsigned long ftrace_get_entry_ip(unsigned long fentry_ip) +{ + unsigned long ip =3D ftrace_get_symaddr(fentry_ip); + + return ip ? : fentry_ip; +} + +static int copy_user_syms(struct user_syms *us, unsigned long __user *usym= s, u32 cnt) +{ + unsigned long __user usymbol; + const char **syms =3D NULL; + char *buf =3D NULL, *p; + int err =3D -ENOMEM; + unsigned int i; + + syms =3D kvmalloc_array(cnt, sizeof(*syms), GFP_KERNEL); + if (!syms) + goto error; + + buf =3D kvmalloc_array(cnt, KSYM_NAME_LEN, GFP_KERNEL); + if (!buf) + goto error; + + for (p =3D buf, i =3D 0; i < cnt; i++) { + if (__get_user(usymbol, usyms + i)) { + err =3D -EFAULT; + goto error; + } + err =3D strncpy_from_user(p, (const char __user *) usymbol, KSYM_NAME_LE= N); + if (err =3D=3D KSYM_NAME_LEN) + err =3D -E2BIG; + if (err < 0) + goto error; + syms[i] =3D p; + p +=3D err + 1; + } + + us->syms =3D syms; + us->buf =3D buf; + return 0; + +error: + if (err) { + kvfree(syms); + kvfree(buf); + } + return err; +} + static void bpf_kprobe_multi_cookie_swap(void *a, void *b, int size, const= void *priv) { const struct bpf_kprobe_multi_link *link =3D priv; @@ -2520,15 +2599,6 @@ static u64 bpf_kprobe_multi_cookie(struct bpf_run_ct= x *ctx) return *cookie; } =20 -static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) -{ - struct bpf_kprobe_multi_run_ctx *run_ctx; - - run_ctx =3D container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ct= x, - session_ctx.run_ctx); - return run_ctx->entry_ip; -} - static __always_inline int kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, unsigned long entry_ip, struct ftrace_regs *fregs, @@ -2597,6 +2667,12 @@ kprobe_multi_link_exit_handler(struct fprobe *fp, un= signed long fentry_ip, fregs, true, data); } =20 +static void free_user_syms(struct user_syms *us) +{ + kvfree(us->syms); + kvfree(us->buf); +} + static int symbols_cmp_r(const void *a, const void *b, const void *priv) { const char **str_a =3D (const char **) a; @@ -2627,82 +2703,6 @@ static void symbols_swap_r(void *a, void *b, int siz= e, const void *priv) } } =20 -struct modules_array { - struct module **mods; - int mods_cnt; - int mods_cap; -}; - -static int add_module(struct modules_array *arr, struct module *mod) -{ - struct module **mods; - - if (arr->mods_cnt =3D=3D arr->mods_cap) { - arr->mods_cap =3D max(16, arr->mods_cap * 3 / 2); - mods =3D krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KER= NEL); - if (!mods) - return -ENOMEM; - arr->mods =3D mods; - } - - arr->mods[arr->mods_cnt] =3D mod; - arr->mods_cnt++; - return 0; -} - -static bool has_module(struct modules_array *arr, struct module *mod) -{ - int i; - - for (i =3D arr->mods_cnt - 1; i >=3D 0; i--) { - if (arr->mods[i] =3D=3D mod) - return true; - } - return false; -} - -static int get_modules_for_addrs(struct module ***mods, unsigned long *add= rs, u32 addrs_cnt) -{ - struct modules_array arr =3D {}; - u32 i, err =3D 0; - - for (i =3D 0; i < addrs_cnt; i++) { - bool skip_add =3D false; - struct module *mod; - - scoped_guard(rcu) { - mod =3D __module_address(addrs[i]); - /* Either no module or it's already stored */ - if (!mod || has_module(&arr, mod)) { - skip_add =3D true; - break; /* scoped_guard */ - } - if (!try_module_get(mod)) - err =3D -EINVAL; - } - if (skip_add) - continue; - if (err) - break; - err =3D add_module(&arr, mod); - if (err) { - module_put(mod); - break; - } - } - - /* We return either err < 0 in case of error, ... */ - if (err) { - kprobe_multi_put_modules(arr.mods, arr.mods_cnt); - kfree(arr.mods); - return err; - } - - /* or number of modules found if everything is ok. */ - *mods =3D arr.mods; - return arr.mods_cnt; -} - static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt) { u32 i; --=20 2.25.1 From nobody Sun Feb 8 13:27:23 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CD2D3358A2 for ; Thu, 18 Dec 2025 13:07:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766063249; cv=none; b=V5hokcygRGiOv4Eyr3XBBPevYEraKoVyYdxNNs+jkWJqhW4uthAhjXzQKxcsRkVwxE0cT/vF826MUW5F6ghl5ZZwx8jMo2vRcVRZLEJ6ROBLt/cLktmwVaeFn2OxpamEWC1KaxRmYUjiO/fu31awM7NrZ4TwOlV2PjipCMUWphs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766063249; c=relaxed/simple; bh=VlgCi+jSecHlyNZTuI51p/UzqZ3eqid22YMKSEzHJVs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ft+Ae8+dOeXXgG13PB661lVaTKZUHxUzmV1G4RSwT1BKO1N24f4Exh02xhC3zcQUo96Y9Eb3zVvtQbrPxHZgQ6bBbwDgebSQ0CnKWZO1grr5YpE48tt2x5VQbN7fHFEY/LxQj3LYBGAuMhZP+/E4t3ocvRTyrUjxgoj29uQ5J7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bn3a84az; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bn3a84az" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2a07f8dd9cdso7034275ad.1 for ; Thu, 18 Dec 2025 05:07:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766063246; x=1766668046; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Kx6inJTls3xrAZIgzRv+a16gJkeMsKLZ2KcFkZHak+4=; b=bn3a84azrTXogIQ+44Ng1SbnUh3zi+oM4DwmYC0HcQsfqgBKrxEEROxA7hyKTFzXnU Zp43krwXkg1mQ/4Z/erUodgJIxbumr5IR0Hcxcii3JCVpHY2LomfWPS7/Q/eDSPQDuCf MLby7FNKDRESz0oBLV2RDsjWNez2WAkwZlVPZ2t3ZwH+i+Ldf5spYlEiX/m5kc9L1jVt yZX4tcitA81jjf1EVd8/gHABxPu/hAlMD+NADS17C3eMKgqsIoBMW192QZwgCoA4Ih5L XzZKTQaAskeUhPv/q6wWtUquvJNbE0W2wNJEWJwu84WdJLDcAgsf5i2V9nqjNOSpa2XW jwnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766063246; x=1766668046; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Kx6inJTls3xrAZIgzRv+a16gJkeMsKLZ2KcFkZHak+4=; b=CNvV2ZBNs9Eh6nmJOELVtxgROD45bFvfUHQQZHsjOcftxSDGFWIm3mA9zSz2mCg1Fq Tostl4HnXHd3cvjdqGNeBf/H9ASGD8em/Od9m64dyn9H6O5cwnPsMYL80YkQ61G0rD5Z pPQ3seUGDS0e3DxnWSc3pNtPQDFBTwvQFhuejNbaFmA3BAVHNajt/stc2m7JDzyweuGD NowfL1+I92cszQRVeV6bTL4Cvf6Z694hea667ZqczmqMy9t1haAjOTw+qJiBNoHIy7yu aAotogSUQjDO5rm2uvP6t3j9ldxC+c/6ErlpKQTPBhv25kfIHpmiEqvctKfcS13/wAAB EgmA== X-Forwarded-Encrypted: i=1; AJvYcCX3ZaGK7zXJKBSJTJnai/r/XzN+CAdKGKDR/Ny3OaTt+sS0DPQTD2L6qZ4U8CR5nH0QA0Lx4XMPsnQZbSQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwwBUv4+Spm66SwZKg/aCzJOOp6RXX/RZV3lPUUYs9MofTF40JH u+AbWU2RXhADZPGjHTXi1RbxdNsLIpWaGA4c03cRFVKQrYsA6++b52wd X-Gm-Gg: AY/fxX618zGilASKXUvzjyGvJz9vO7nNiBne04ZnhspX4B4UXlZdWLlSz7ia1IO0U4T rWf2Nk4Vyhm3vQf5WPHNHez7gUzkIictr+iruN8ETydX+l6Ahisa4KpHp/HqxEkmo7XvMaBwXtl Ii3FzlMCFPvJQQTPIqQGlixS/p1phPxCyr4K+rPHnzMeol6winf80BwH401Ke49zgPagkEavWaz 3Qqg/6BoSgF9zm5JDo8ZilAZ5hduQfzxZ9iHAxzaYl3/KV3HBE89Qu80DEUPZbfsqJb6LuJNKQJ cITzN1QHYB3v0y0EQESHc8Ycety/7S29CixK5QCm4Z2dKgZ3vQkuBEm6zwLeNblA6XkolsUiq+f zqym8NZuhSKFVtRummjGnHgkn/mXb9gnk8tGXhIbGIMG+kRoiE5VZAFeNnRBmCaBMo1KgzSzDKh ATmxmoDNT8cy/tbp7niHgwH7kEYJvb02YYeR0= X-Google-Smtp-Source: AGHT+IG4+LOle8Wc5Em5iycYNT/mJTH6jErIVEzgVHREcuYF/aW9Gar4t8YSlLxlaIZZyWiB+NY99Q== X-Received: by 2002:a17:903:3843:b0:2a0:d629:903f with SMTP id d9443c01a7336-2a0d6299386mr122532325ad.9.1766063246272; Thu, 18 Dec 2025 05:07:26 -0800 (PST) Received: from mi-ThinkStation-K.mioffice.cn ([43.224.245.232]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2d1926cd2sm25539905ad.77.2025.12.18.05.07.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Dec 2025 05:07:25 -0800 (PST) From: liujing40 X-Google-Original-From: liujing40 To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, mhiramat@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, liujing40@xiaomi.com Subject: [PATCH 2/2] bpf: Implement kretprobe fallback for kprobe multi link Date: Thu, 18 Dec 2025 21:06:29 +0800 Message-Id: <20251218130629.365398-3-liujing40@xiaomi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20251218130629.365398-1-liujing40@xiaomi.com> References: <20251218130629.365398-1-liujing40@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When fprobe is not available, provide a fallback implementation of kprobe_multi using the traditional kretprobe API. Uses kretprobe's entry_handler and handler callbacks to simulate fprobe's entry/exit functionality. Signed-off-by: Jing Liu --- kernel/trace/bpf_trace.c | 307 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 295 insertions(+), 12 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 1fd07c10378f..426a1c627508 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2274,12 +2274,44 @@ struct bpf_session_run_ctx { void *data; }; =20 -#ifdef CONFIG_FPROBE +#if defined(CONFIG_FPROBE) || defined(CONFIG_KRETPROBES) +#ifndef CONFIG_FPROBE +struct bpf_kprobe { + struct bpf_kprobe_multi_link *link; + u64 cookie; + struct kretprobe rp; +}; + +static void bpf_kprobe_unregister(struct bpf_kprobe *kps, u32 cnt) +{ + for (int i =3D 0; i < cnt; i++) + unregister_kretprobe(&kps[i].rp); +} + +static int bpf_kprobe_register(struct bpf_kprobe *kps, u32 cnt) +{ + int ret =3D 0, i; + + for (i =3D 0; i < cnt; i++) { + ret =3D register_kretprobe(&kps[i].rp); + if (ret < 0) { + bpf_kprobe_unregister(kps, i); + break; + } + } + return ret; +} +#endif + struct bpf_kprobe_multi_link { struct bpf_link link; +#ifdef CONFIG_FPROBE struct fprobe fp; unsigned long *addrs; u64 *cookies; +#else + struct bpf_kprobe *kprobes; +#endif u32 cnt; u32 mods_cnt; struct module **mods; @@ -2287,7 +2319,11 @@ struct bpf_kprobe_multi_link { =20 struct bpf_kprobe_multi_run_ctx { struct bpf_session_run_ctx session_ctx; +#ifdef CONFIG_FPROBE struct bpf_kprobe_multi_link *link; +#else + struct bpf_kprobe *kprobe; +#endif unsigned long entry_ip; }; =20 @@ -2304,7 +2340,11 @@ static void bpf_kprobe_multi_link_release(struct bpf= _link *link) struct bpf_kprobe_multi_link *kmulti_link; =20 kmulti_link =3D container_of(link, struct bpf_kprobe_multi_link, link); +#ifdef CONFIG_FPROBE unregister_fprobe(&kmulti_link->fp); +#else + bpf_kprobe_unregister(kmulti_link->kprobes, kmulti_link->cnt); +#endif kprobe_multi_put_modules(kmulti_link->mods, kmulti_link->mods_cnt); } =20 @@ -2313,8 +2353,12 @@ static void bpf_kprobe_multi_link_dealloc(struct bpf= _link *link) struct bpf_kprobe_multi_link *kmulti_link; =20 kmulti_link =3D container_of(link, struct bpf_kprobe_multi_link, link); +#ifdef CONFIG_FPROBE kvfree(kmulti_link->addrs); kvfree(kmulti_link->cookies); +#else + kvfree(kmulti_link->kprobes); +#endif kfree(kmulti_link->mods); kfree(kmulti_link); } @@ -2326,6 +2370,7 @@ static int bpf_kprobe_multi_link_fill_link_info(const= struct bpf_link *link, u64 __user *uaddrs =3D u64_to_user_ptr(info->kprobe_multi.addrs); struct bpf_kprobe_multi_link *kmulti_link; u32 ucount =3D info->kprobe_multi.count; + bool kallsyms_show =3D kallsyms_show_value(current_cred()); int err =3D 0, i; =20 if (!uaddrs ^ !ucount) @@ -2336,7 +2381,12 @@ static int bpf_kprobe_multi_link_fill_link_info(cons= t struct bpf_link *link, kmulti_link =3D container_of(link, struct bpf_kprobe_multi_link, link); info->kprobe_multi.count =3D kmulti_link->cnt; info->kprobe_multi.flags =3D kmulti_link->link.flags; +#ifdef CONFIG_FPROBE info->kprobe_multi.missed =3D kmulti_link->fp.nmissed; +#else + for (i =3D 0; i < kmulti_link->cnt; i++) + info->kprobe_multi.missed +=3D kmulti_link->kprobes[i].rp.nmissed; +#endif =20 if (!uaddrs) return 0; @@ -2345,6 +2395,7 @@ static int bpf_kprobe_multi_link_fill_link_info(const= struct bpf_link *link, else ucount =3D kmulti_link->cnt; =20 +#ifdef CONFIG_FPROBE if (ucookies) { if (kmulti_link->cookies) { if (copy_to_user(ucookies, kmulti_link->cookies, ucount * sizeof(u64))) @@ -2357,7 +2408,7 @@ static int bpf_kprobe_multi_link_fill_link_info(const= struct bpf_link *link, } } =20 - if (kallsyms_show_value(current_cred())) { + if (kallsyms_show) { if (copy_to_user(uaddrs, kmulti_link->addrs, ucount * sizeof(u64))) return -EFAULT; } else { @@ -2366,6 +2417,16 @@ static int bpf_kprobe_multi_link_fill_link_info(cons= t struct bpf_link *link, return -EFAULT; } } +#else + for (i =3D 0; i < ucount; i++) { + if (ucookies && put_user(kmulti_link->kprobes[i].cookie, ucookies + i)) + return -EFAULT; + + if (put_user(kallsyms_show ? (uintptr_t)kmulti_link->kprobes[i].rp.kp.ad= dr : 0, + uaddrs + i)) + return -EFAULT; + } +#endif return err; } =20 @@ -2374,21 +2435,32 @@ static void bpf_kprobe_multi_show_fdinfo(const stru= ct bpf_link *link, struct seq_file *seq) { struct bpf_kprobe_multi_link *kmulti_link; + unsigned long kprobe_multi_missed =3D 0; =20 kmulti_link =3D container_of(link, struct bpf_kprobe_multi_link, link); +#ifdef CONFIG_FPROBE + kprobe_multi_missed =3D kmulti_link->fp.nmissed; +#else + for (int i =3D 0; i < kmulti_link->cnt; i++) + kprobe_multi_missed +=3D kmulti_link->kprobes[i].rp.nmissed; +#endif =20 seq_printf(seq, "kprobe_cnt:\t%u\n" "missed:\t%lu\n", kmulti_link->cnt, - kmulti_link->fp.nmissed); + kprobe_multi_missed); =20 seq_printf(seq, "%s\t %s\n", "cookie", "func"); for (int i =3D 0; i < kmulti_link->cnt; i++) { - seq_printf(seq, - "%llu\t %pS\n", - kmulti_link->cookies[i], - (void *)kmulti_link->addrs[i]); +#ifdef CONFIG_FPROBE + u64 cookie =3D kmulti_link->cookies[i]; + void *addr =3D (void *)kmulti_link->addrs[i]; +#else + u64 cookie =3D kmulti_link->kprobes[i].cookie; + void *addr =3D (void *)kmulti_link->kprobes[i].rp.kp.addr; +#endif + seq_printf(seq, "%llu\t %pS\n", cookie, addr); } } #endif @@ -2445,17 +2517,22 @@ static bool has_module(struct modules_array *arr, s= truct module *mod) return false; } =20 -static int get_modules_for_addrs(struct module ***mods, unsigned long *add= rs, u32 addrs_cnt) +static int get_modules_for_addrs(struct bpf_kprobe_multi_link *link) { struct modules_array arr =3D {}; u32 i, err =3D 0; =20 - for (i =3D 0; i < addrs_cnt; i++) { + for (i =3D 0; i < link->cnt; i++) { bool skip_add =3D false; struct module *mod; +#ifdef CONFIG_FPROBE + unsigned long addr =3D link->addrs[i]; +#else + unsigned long addr =3D (unsigned long)link->kprobes[i].rp.kp.addr; +#endif =20 scoped_guard(rcu) { - mod =3D __module_address(addrs[i]); + mod =3D __module_address(addr); /* Either no module or it's already stored */ if (!mod || has_module(&arr, mod)) { skip_add =3D true; @@ -2483,10 +2560,11 @@ static int get_modules_for_addrs(struct module ***m= ods, unsigned long *addrs, u3 } =20 /* or number of modules found if everything is ok. */ - *mods =3D arr.mods; + link->mods =3D arr.mods; return arr.mods_cnt; } =20 +#ifdef CONFIG_FPROBE struct user_syms { const char **syms; char *buf; @@ -2843,7 +2921,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr= *attr, struct bpf_prog *pr link); } =20 - err =3D get_modules_for_addrs(&link->mods, addrs, cnt); + err =3D get_modules_for_addrs(link); if (err < 0) { bpf_link_cleanup(&link_primer); return err; @@ -2866,6 +2944,211 @@ int bpf_kprobe_multi_link_attach(const union bpf_at= tr *attr, struct bpf_prog *pr return err; } #else /* !CONFIG_FPROBE */ +static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx) +{ + struct bpf_kprobe_multi_run_ctx *run_ctx; + + run_ctx =3D container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ct= x, + session_ctx.run_ctx); + return run_ctx->kprobe->cookie; +} + +static __always_inline int +kprobe_multi_link_prog_run(struct bpf_kprobe *kprobe, unsigned long entry_= ip, + struct pt_regs *regs, bool is_return, void *data) +{ + struct bpf_kprobe_multi_link *link =3D kprobe->link; + struct bpf_kprobe_multi_run_ctx run_ctx =3D { + .session_ctx =3D { + .is_return =3D is_return, + .data =3D data, + }, + .kprobe =3D kprobe, + .entry_ip =3D entry_ip, + }; + struct bpf_run_ctx *old_run_ctx; + int err; + + cant_sleep(); + + if (unlikely(__this_cpu_inc_return(bpf_prog_active) !=3D 1)) { + bpf_prog_inc_misses_counter(link->link.prog); + err =3D 1; + goto out; + } + + rcu_read_lock(); + migrate_disable(); + old_run_ctx =3D bpf_set_run_ctx(&run_ctx.session_ctx.run_ctx); + err =3D bpf_prog_run(link->link.prog, regs); + bpf_reset_run_ctx(old_run_ctx); + migrate_enable(); + rcu_read_unlock(); + + out: + __this_cpu_dec(bpf_prog_active); + return err; +} + +static int +kprobe_multi_link_handler(struct kretprobe_instance *ri, struct pt_regs *r= egs) +{ + struct kretprobe *rp =3D get_kretprobe(ri); + struct bpf_kprobe *kprobe; + int err; + + if (unlikely(!rp)) + return 1; + + kprobe =3D container_of(rp, struct bpf_kprobe, rp); + err =3D kprobe_multi_link_prog_run(kprobe, get_entry_ip((uintptr_t)rp->kp= .addr), + regs, false, ri->data); + return is_kprobe_session(kprobe->link->link.prog) ? err : 0; +} + +static int +kprobe_multi_link_exit_handler(struct kretprobe_instance *ri, struct pt_re= gs *regs) +{ + struct kretprobe *rp =3D get_kretprobe(ri); + struct bpf_kprobe *kprobe; + + if (unlikely(!rp)) + return 0; + + kprobe =3D container_of(rp, struct bpf_kprobe, rp); + kprobe_multi_link_prog_run(kprobe, get_entry_ip((uintptr_t)rp->kp.addr), + regs, true, ri->data); + return 0; +} + +int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_pr= og *prog) +{ + struct bpf_kprobe_multi_link *link =3D NULL; + struct bpf_link_primer link_primer; + struct bpf_kprobe *kprobes =3D NULL; + u32 flags, cnt; + u64 __user *ucookies; + unsigned long __user *uaddrs; + unsigned long __user *usyms; + int err, i; + + /* no support for 32bit archs yet */ + if (sizeof(u64) !=3D sizeof(void *)) + return -EOPNOTSUPP; + + if (attr->link_create.flags) + return -EINVAL; + + if (!is_kprobe_multi(prog)) + return -EINVAL; + + /* Writing to context is not allowed for kprobes. */ + if (prog->aux->kprobe_write_ctx) + return -EINVAL; + + flags =3D attr->link_create.kprobe_multi.flags; + if (flags & ~BPF_F_KPROBE_MULTI_RETURN) + return -EINVAL; + + uaddrs =3D u64_to_user_ptr(attr->link_create.kprobe_multi.addrs); + usyms =3D u64_to_user_ptr(attr->link_create.kprobe_multi.syms); + if (!!uaddrs =3D=3D !!usyms) + return -EINVAL; + + cnt =3D attr->link_create.kprobe_multi.cnt; + if (!cnt) + return -EINVAL; + if (cnt > MAX_KPROBE_MULTI_CNT) + return -E2BIG; + + ucookies =3D u64_to_user_ptr(attr->link_create.kprobe_multi.cookies); + kprobes =3D kvcalloc(cnt, sizeof(*kprobes), GFP_KERNEL); + link =3D kzalloc(sizeof(*link), GFP_KERNEL); + if (!link || !kprobes) { + err =3D -ENOMEM; + goto error; + } + + for (i =3D 0; i < cnt; i++) { + unsigned long addr; + + if (uaddrs) { + if (__get_user(addr, uaddrs + i)) { + err =3D -EFAULT; + goto error; + } + } else { + unsigned long __user usymbol; + char buf[KSYM_NAME_LEN]; + + if (__get_user(usymbol, usyms + i)) { + err =3D -EFAULT; + goto error; + } + err =3D strncpy_from_user(buf, (const char __user *) usymbol, KSYM_NAME= _LEN); + if (err =3D=3D KSYM_NAME_LEN) + err =3D -E2BIG; + if (err < 0) + goto error; + + addr =3D kallsyms_lookup_name(buf); + if (!addr) + goto error; + } + if (prog->kprobe_override && !within_error_injection_list(addr)) { + err =3D -EINVAL; + goto error; + } + if (ucookies && __get_user(kprobes[i].cookie, ucookies + i)) { + err =3D -EFAULT; + goto error; + } + + kprobes[i].link =3D link; + kprobes[i].rp.kp.addr =3D (kprobe_opcode_t *)addr; + + if (!(flags & BPF_F_KPROBE_MULTI_RETURN)) + kprobes[i].rp.entry_handler =3D kprobe_multi_link_handler; + if ((flags & BPF_F_KPROBE_MULTI_RETURN) || is_kprobe_session(prog)) + kprobes[i].rp.handler =3D kprobe_multi_link_exit_handler; + if (is_kprobe_session(prog)) + kprobes[i].rp.data_size =3D sizeof(u64); + } + + bpf_link_init(&link->link, BPF_LINK_TYPE_KPROBE_MULTI, + &bpf_kprobe_multi_link_lops, prog, attr->link_create.attach_type); + + err =3D bpf_link_prime(&link->link, &link_primer); + if (err) + goto error; + + link->kprobes =3D kprobes; + link->cnt =3D cnt; + link->link.flags =3D flags; + + err =3D get_modules_for_addrs(link); + if (err < 0) { + bpf_link_cleanup(&link_primer); + return err; + } + link->mods_cnt =3D err; + + err =3D bpf_kprobe_register(kprobes, cnt); + if (err) { + kprobe_multi_put_modules(link->mods, link->mods_cnt); + bpf_link_cleanup(&link_primer); + return err; + } + + return bpf_link_settle(&link_primer); + +error: + kvfree(kprobes); + kfree(link); + return err; +} +#endif +#else /* !CONFIG_FPROBE && !CONFIG_KRETPROBES*/ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_pr= og *prog) { return -EOPNOTSUPP; --=20 2.25.1