From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8D31195FE8; Wed, 28 May 2025 03:49:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404182; cv=none; b=c2tze6T9z3d+wRYohi7V3ulY88YCO2IoaAqNlcD93iLiqNTK9CUg4Y4S/BfUCPefxy/4CBH/pN3/07vPN/9g7nj4uE4PaEv3u539k1jk5gYh5X/iwRPkVIuETR6a0yNrCDUIN1JSJ18Liu+Dr10tZJiQ6zTw8jUHp+qyu7pf6w0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404182; c=relaxed/simple; bh=T7NssPaqVeOHvj7JvgtrQS8i+xBkkBIwEC99uOHnS98=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gM6VrjgARgZqiOD8qjow8Yf2Zx2kOgS+e33DgAWf1WG9PkkdqvWR8WI8OJdDzxKFsOJl8Z/gn3WRmfohg8JPQZSfOXLxGEsO+XynRL9SZwPWf9/yJsC4v5eF1BQEhwhhUTxZj1rAkWfbW/ZtY1E2/0rlgyTXRIGNWL6eG5Sb/jA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YKSqJ2qw; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YKSqJ2qw" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-2347012f81fso30641275ad.2; Tue, 27 May 2025 20:49:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404179; x=1749008979; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DTmlP++5ctATzqFYw3YvfkZEdLmcOEH5s5X3SK2U8so=; b=YKSqJ2qw5JXi4oD2RhLnIgA2IQIcjEGC3jOzVttmrhWEjZM8MUWTi/blVtLewzDBNx pyg5HN+kD7il0JVtNVcQQmj0W99x0FvkqH1Hy+Oue6K/Jkixo19Rl3JSNaPmIKBJt+wv gMNlfSsWMCdjps1pCN8TWQpzKug2D9KQnHE7id8DlunHY0OiukgbfpRfrKe93Xoe7Nd1 VVobEYZ3srwrsnpYizMstc4YGy1SLmsHXtqwe0/TkTlK1MHbcVb021DTDTGyqHCnVFMb CAwMDckv9y9rs3kerYa9Ww1bxi73hNMcPMYS8G95n493kKFw6Ei42bnWe7LbCYN0uptz 7bNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404179; x=1749008979; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DTmlP++5ctATzqFYw3YvfkZEdLmcOEH5s5X3SK2U8so=; b=rzasdIqwYga+vdqFFuN3kNXV/43j6qABfVdKRorYBOncFF8dSC2dqDhVfYJj8scKMu f2n0aaZNi0zRkaGjuxajcpsZPy5msVDcr0UhHrDVboImbhGRrS24FEjQuMjz1wALqRgV /P4t8TiJ55hkomh6uiG94wQ6xp6rH0UL9Jc+HHsmW/bpFpr/4nqhVFrL6vWCjq6bgQUv Ni2E5g+wFgQ9Ur0jmPEyMl6OdxB7lV2+hzqZudlwuqKi3lVl0FpJnBaJ7gKnYANWG8ah 9Aegg372xKci4fXL/9uzB/C5NTLdnqGIp9O6YSR5p78CEjkbt0ArxZYI672Llb/UMK7w CBYA== X-Forwarded-Encrypted: i=1; AJvYcCWhyAVyGfouwoKnoloyyQ45Ju4YpvxtU0gY8ZjT0ZYdARYaSFmNUxSZ+q6Bb1iLPcJ0soKX05Bd/MrU6Qw=@vger.kernel.org X-Gm-Message-State: AOJu0YyJFPj97DMXrHZCcdm0LUaVIA5sKrUyZOUm7I+1/OHYUcFC+VJg XTMV8d0krd5Q3zR2LxUdowpf6hMxlrrgCTdG8w8yoSdWnf24xU0fgRWL X-Gm-Gg: ASbGnct1YkKOswQF4s95Wf7w1l26NFIuc5ITcuVm5P0SQWefJ2xQPDCXSxqWO5yxaVj oQQVSjlsJdctRMU00lLKOyGbVmkqd/z6EV5a0Kf3TQVfMdrk7ZtTCVGX5AhRUr/0Ne3gfQQ5o+f lnrH1YXYqPbeN7bctDAkizqvg1pWB3TPFe61I8TPjf/rJtK8ECTX8XRehhjvF62FDvZB5SD+N94 lWwvIyNii3Qc7s/yy3T4Ljic0ZAw/1zzGxIGHJjdqYFx3YEt1vUBeTUa7NlPtTy6H+skoGxzAer 3De3sdN66dPVT4RXPxb3ANL+nD5dVuVcZLhtgGW9tM1/vR3JPxaekN7Jocd6KPuEHbNn X-Google-Smtp-Source: AGHT+IF8gNtZfMhjGeixcimdN3aY3HBVzUm76AiS8MOrlteDo7Y0hyQxCu0Qwd0OXbZ2LWMAbnqR/Q== X-Received: by 2002:a17:903:2283:b0:234:9092:9dda with SMTP id d9443c01a7336-23490929e16mr112651645ad.24.1748404178596; Tue, 27 May 2025 20:49:38 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:38 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 01/25] add per-function metadata storage support Date: Wed, 28 May 2025 11:46:48 +0800 Message-Id: <20250528034712.138701-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, there isn't a way to set and get per-function metadata with a low overhead, which is not convenient for some situations. Take BPF trampoline for example, we need to create a trampoline for each kernel function, as we have to store some information of the function to the trampoline, such as BPF progs, function arg count, etc. The performance overhead and memory consumption can be higher to create these trampolines. With the supporting of per-function metadata storage, we can store these information to the metadata, and create a global BPF trampoline for all the kernel functions. In the global trampoline, we get the information that we need from the function metadata through the ip (function address) with almost no overhead. Another beneficiary can be fprobe. For now, fprobe will add all the functions that it hooks into a hash table. And in fprobe_entry(), it will lookup all the handlers of the function in the hash table. The performance can suffer from the hash table lookup. We can optimize it by adding the handler to the function metadata instead. Support per-function metadata storage in the function padding, and previous discussion can be found in [1]. Generally speaking, we have two way to implement this feature: 1. Create a function metadata array, and prepend a insn which can hold the index of the function metadata in the array. And store the insn to the function padding. 2. Allocate the function metadata with kmalloc(), and prepend a insn which hold the pointer of the metadata. And store the insn to the function padding. Compared with way 2, way 1 consume less space, but we need to do more work on the global function metadata array. And we implement this function in the way 1. Link: https://lore.kernel.org/bpf/CADxym3anLzM6cAkn_z71GDd_VeKiqqk1ts=3Dxui= P7pr4PO6USPA@mail.gmail.com/ [1] Signed-off-by: Menglong Dong --- include/linux/kfunc_md.h | 44 +++ kernel/trace/Makefile | 1 + kernel/trace/kfunc_md.c | 566 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 611 insertions(+) create mode 100644 include/linux/kfunc_md.h create mode 100644 kernel/trace/kfunc_md.c diff --git a/include/linux/kfunc_md.h b/include/linux/kfunc_md.h new file mode 100644 index 000000000000..21c0b879cc03 --- /dev/null +++ b/include/linux/kfunc_md.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KFUNC_MD_H +#define _LINUX_KFUNC_MD_H + +#define KFUNC_MD_FL_DEAD (1 << 0) /* the md shouldn't be reused */ + +#ifndef __ASSEMBLER__ + +#include +#include + +struct kfunc_md_array; + +struct kfunc_md { +#ifndef CONFIG_FUNCTION_METADATA_PADDING + /* this is used for the hash table mode */ + struct hlist_node hash; + /* this is used for table mode */ + struct rcu_head rcu; +#endif + unsigned long func; +#ifdef CONFIG_FUNCTION_METADATA + /* the array is used for the fast mode */ + struct kfunc_md_array *array; +#endif + struct percpu_ref pcref; + u32 flags; + u16 users; + u8 nr_args; +}; + +struct kfunc_md *kfunc_md_get(unsigned long ip); +struct kfunc_md *kfunc_md_get_noref(unsigned long ip); +struct kfunc_md *kfunc_md_create(unsigned long ip, int nr_args); +void kfunc_md_put_entry(struct kfunc_md *meta); +void kfunc_md_put(unsigned long ip); +void kfunc_md_lock(void); +void kfunc_md_unlock(void); +void kfunc_md_exit(struct kfunc_md *md); +void kfunc_md_enter(struct kfunc_md *md); +bool kfunc_md_arch_support(int *insn, int *data); + +#endif +#endif diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile index 057cd975d014..d8c19ff1e55e 100644 --- a/kernel/trace/Makefile +++ b/kernel/trace/Makefile @@ -52,6 +52,7 @@ obj-$(CONFIG_TRACING) +=3D trace_seq.o obj-$(CONFIG_TRACING) +=3D trace_stat.o obj-$(CONFIG_TRACING) +=3D trace_printk.o obj-$(CONFIG_TRACING) +=3D pid_list.o +obj-$(CONFIG_TRACING) +=3D kfunc_md.o obj-$(CONFIG_TRACING_MAP) +=3D tracing_map.o obj-$(CONFIG_PREEMPTIRQ_DELAY_TEST) +=3D preemptirq_delay_test.o obj-$(CONFIG_SYNTH_EVENT_GEN_TEST) +=3D synth_event_gen_test.o diff --git a/kernel/trace/kfunc_md.c b/kernel/trace/kfunc_md.c new file mode 100644 index 000000000000..9571081f6560 --- /dev/null +++ b/kernel/trace/kfunc_md.c @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ + +#include +#include +#include +#include +#include + +#include + +#ifndef CONFIG_FUNCTION_METADATA_PADDING + +DEFINE_STATIC_KEY_TRUE(kfunc_md_use_padding); +static int __insn_offset, __data_offset; +#define insn_offset __insn_offset +#define data_offset __data_offset + +#define KFUNC_MD_HASH_BITS 10 +static struct hlist_head kfunc_md_table[1 << KFUNC_MD_HASH_BITS]; + +#else +#define insn_offset KFUNC_MD_INSN_OFFSET +#define data_offset KFUNC_MD_DATA_OFFSET +#endif + +#define insn_size KFUNC_MD_INSN_SIZE + +#define ENTRIES_PER_PAGE (PAGE_SIZE / sizeof(struct kfunc_md)) + +#define KFUNC_MD_ARRAY_FL_DEAD 0 + +struct kfunc_md_array { + struct kfunc_md *mds; + u32 kfunc_md_count; + unsigned long flags; + atomic_t kfunc_md_used; + union { + struct work_struct work; + struct rcu_head rcu; + }; +}; + +static struct kfunc_md_array empty_array =3D { + .mds =3D NULL, + .kfunc_md_count =3D 0, +}; +/* used for the padding-based function metadata */ +static struct kfunc_md_array __rcu *kfunc_mds =3D &empty_array; + +/* any function metadata write should hold this lock */ +static DEFINE_MUTEX(kfunc_md_mutex); + + +#ifndef CONFIG_FUNCTION_METADATA_PADDING + +static struct hlist_head *kfunc_md_hash_head(unsigned long ip) +{ + return &kfunc_md_table[hash_ptr((void *)ip, KFUNC_MD_HASH_BITS)]; +} + +static struct kfunc_md *kfunc_md_hash_get(unsigned long ip) +{ + struct hlist_head *head; + struct kfunc_md *md; + + head =3D kfunc_md_hash_head(ip); + hlist_for_each_entry_rcu_notrace(md, head, hash) { + if (md->func =3D=3D ip) + return md; + } + + return NULL; +} + +static void kfunc_md_hash_release(struct percpu_ref *pcref) +{ + struct kfunc_md *md; + + md =3D container_of(pcref, struct kfunc_md, pcref); + kfree_rcu(md, rcu); +} + +static struct kfunc_md *kfunc_md_hash_create(unsigned long ip, int nr_args) +{ + struct kfunc_md *md =3D kfunc_md_hash_get(ip); + struct hlist_head *head; + int err; + + if (md) { + md->users++; + return md; + } + + md =3D kzalloc(sizeof(*md), GFP_KERNEL); + if (!md) + return NULL; + + md->users =3D 1; + md->func =3D ip; + md->nr_args =3D nr_args; + + err =3D percpu_ref_init(&md->pcref, kfunc_md_hash_release, 0, GFP_KERNEL); + if (err) { + kfree(md); + return NULL; + } + + head =3D kfunc_md_hash_head(ip); + hlist_add_tail_rcu(&md->hash, head); + atomic_inc(&kfunc_mds->kfunc_md_used); + + return md; +} + +static void kfunc_md_hash_put(struct kfunc_md *md) +{ + if (WARN_ON_ONCE(md->users <=3D 0)) + return; + + md->users--; + if (md->users > 0) + return; + + hlist_del_rcu(&md->hash); + percpu_ref_kill(&md->pcref); + atomic_dec(&kfunc_mds->kfunc_md_used); +} + +static bool kfunc_md_fast(void) +{ + return static_branch_likely(&kfunc_md_use_padding); +} +#else + +static void kfunc_md_hash_put(struct kfunc_md *md) +{ +} + +static struct kfunc_md *kfunc_md_hash_get(unsigned long ip) +{ + return NULL; +} + +static struct kfunc_md *kfunc_md_hash_create(unsigned long ip, int nr_args) +{ + return NULL; +} + +#define kfunc_md_fast() 1 +#endif /* CONFIG_FUNCTION_METADATA_PADDING */ + +#ifdef CONFIG_FUNCTION_METADATA +static void kfunc_md_release(struct percpu_ref *pcref); + +static __always_inline u32 kfunc_md_get_index(unsigned long ip) +{ + return *(u32 *)(ip - data_offset); +} + +static struct kfunc_md_array *kfunc_md_array_alloc(struct kfunc_md_array *= old) +{ + struct kfunc_md_array *new_mds; + int len =3D old->kfunc_md_count; + struct kfunc_md *md; + int err, i; + + new_mds =3D kmalloc(sizeof(*new_mds), __GFP_ZERO | GFP_KERNEL); + if (!new_mds) + return NULL; + + /* if the length of old kfunc md array is zero, we make ENTRIES_PER_PAGE + * as the default size of the new kfunc md array. + */ + new_mds->kfunc_md_count =3D (len * 2) ?: ENTRIES_PER_PAGE; + new_mds->mds =3D kvmalloc_array(new_mds->kfunc_md_count, sizeof(*new_mds-= >mds), + __GFP_ZERO | GFP_KERNEL); + if (!new_mds->mds) { + kfree(new_mds); + return NULL; + } + + if (len) { + memcpy(new_mds->mds, old->mds, sizeof(*new_mds->mds) * len); + new_mds->kfunc_md_used =3D old->kfunc_md_used; + } + + for (i =3D 0; i < new_mds->kfunc_md_count; i++) { + md =3D &new_mds->mds[i]; + + if (md->users) { + err =3D percpu_ref_init(&md->pcref, kfunc_md_release, + 0, GFP_KERNEL); + if (err) + goto pcref_fail; + md->array =3D new_mds; + } + } + + return new_mds; + +pcref_fail: + for (int j =3D 0; j < i; j++) { + md =3D &new_mds->mds[j]; + if (md->users) + percpu_ref_exit(&md->pcref); + } + kvfree(new_mds->mds); + kfree(new_mds); + return NULL; +} + +static void kfunc_md_array_release_deferred(struct work_struct *work) +{ + struct kfunc_md_array *mds; + + mds =3D container_of(work, struct kfunc_md_array, work); + /* the kfunc metadata array is not used anywhere, we can free it + * directly. + */ + if (atomic_read(&mds->kfunc_md_used) =3D=3D 0) { + for (int i =3D 0; i < mds->kfunc_md_count; i++) { + if (mds->mds[i].users) + percpu_ref_exit(&mds->mds[i].pcref); + } + + kvfree(mds->mds); + kfree_rcu(mds, rcu); + return; + } + + for (int i =3D 0; i < mds->kfunc_md_count; i++) { + if (mds->mds[i].users) + percpu_ref_kill(&mds->mds[i].pcref); + } +} + +static void kfunc_md_array_release(struct rcu_head *rcu) +{ + struct kfunc_md_array *mds; + + mds =3D container_of(rcu, struct kfunc_md_array, rcu); + if (mds =3D=3D &empty_array) + return; + + INIT_WORK(&mds->work, kfunc_md_array_release_deferred); + schedule_work(&mds->work); +} + +static void kfunc_md_release(struct percpu_ref *pcref) +{ + struct kfunc_md *md; + + md =3D container_of(pcref, struct kfunc_md, pcref); + if (test_bit(KFUNC_MD_ARRAY_FL_DEAD, &md->array->flags)) { + if (atomic_dec_and_test(&md->array->kfunc_md_used)) { + call_rcu_tasks(&md->array->rcu, kfunc_md_array_release); + return; + } + } + percpu_ref_exit(&md->pcref); + /* clear the flags, so it can be reused */ + md->flags =3D 0; +} + +static int kfunc_md_text_poke(unsigned long ip, void *insn, void *nop) +{ + void *target; + int ret =3D 0; + u8 *prog; + + target =3D (void *)(ip - insn_offset); + mutex_lock(&text_mutex); + if (insn) { + if (!memcmp(target, insn, insn_size)) + goto out; + + if (memcmp(target, nop, insn_size)) { + ret =3D -EBUSY; + goto out; + } + prog =3D insn; + } else { + if (!memcmp(target, nop, insn_size)) + goto out; + prog =3D nop; + } + + ret =3D kfunc_md_arch_poke(target, prog, insn_offset); +out: + mutex_unlock(&text_mutex); + return ret; +} + +/* Get next usable function metadata. On success, return the usable + * kfunc_md and store the index of it to *index. If no usable kfunc_md is + * found in kfunc_mds, a larger array will be allocated. + */ +static struct kfunc_md *kfunc_md_fast_next(u32 *index) +{ + struct kfunc_md_array *mds, *new_mds; + struct kfunc_md *md; + u32 i; + + mds =3D kfunc_mds; +do_retry: + if (likely(atomic_read(&mds->kfunc_md_used) < mds->kfunc_md_count)) { + /* maybe we can manage the used function metadata entry + * with a bit map ? + */ + for (i =3D 0; i < mds->kfunc_md_count; i++) { + md =3D &mds->mds[i]; + if (!md->users && !(md->flags & KFUNC_MD_FL_DEAD)) { + atomic_inc(&mds->kfunc_md_used); + *index =3D i; + return md; + } + } + } + + /* no available function metadata, so allocate a bigger function + * metadata array. + * + * TODO: we increase the array length here, and we also need to + * shrink it somewhere. + */ + new_mds =3D kfunc_md_array_alloc(mds); + if (!new_mds) + return NULL; + + rcu_assign_pointer(kfunc_mds, new_mds); + /* release of the old kfunc metadata array. + * + * First step, set KFUNC_MD_ARRAY_FL_DEAD on it. The old mds will + * not be accessed by anyone anymore from now on. + * + * Second step, call rcu to wakeup the work queue to call + * kfunc_md_array_release_deferred() in kfunc_md_array_release. + * + * Third step, kill all the percpu ref of the mds in + * kfunc_md_array_release_deferred(). + * + * Fourth step, decrease the mds->kfunc_md_used in the callback of + * the percpu ref. And the callback is kfunc_md_release(). + * + * Fifth step, wakeup the work queue to call + * kfunc_md_array_release_deferred() if old->kfunc_md_used is decreased + * to 0, and the old mds will be freed. + */ + set_bit(KFUNC_MD_ARRAY_FL_DEAD, &mds->flags); + call_rcu_tasks(&mds->rcu, kfunc_md_array_release); + mds =3D new_mds; + + goto do_retry; +} + +static void kfunc_md_fast_put(struct kfunc_md *md) +{ + u8 nop_insn[insn_size]; + + if (WARN_ON_ONCE(md->users <=3D 0)) + return; + + md->users--; + if (md->users > 0) + return; + + if (WARN_ON_ONCE(!kfunc_md_arch_exist(md->func, insn_offset))) + return; + + atomic_dec(&md->array->kfunc_md_used); + kfunc_md_arch_nops(nop_insn); + /* release the metadata by recovering the function padding to NOPS */ + kfunc_md_text_poke(md->func, NULL, nop_insn); + /* mark it as dead, so it will not be reused before we release it + * fully in kfunc_md_release(). + */ + md->flags |=3D KFUNC_MD_FL_DEAD; + percpu_ref_kill(&md->pcref); +} + +/* Get a exist metadata by the function address, and NULL will be returned + * if not exist. + * + * NOTE: rcu lock or kfunc_md_lock should be held during reading the metad= ata, + * and kfunc_md_lock should be held if writing happens. + */ +static struct kfunc_md *kfunc_md_fast_get(unsigned long ip) +{ + struct kfunc_md *md; + u32 index; + + if (kfunc_md_arch_exist(ip, insn_offset)) { + index =3D kfunc_md_get_index(ip); + md =3D READ_ONCE(kfunc_mds->mds) + index; + return md; + } + return NULL; +} + +/* Get a exist metadata by the function address, and create one if not + * exist. Reference of the metadata will increase 1. + * + * NOTE: always call this function with kfunc_md_lock held, and all + * updating to metadata should also hold the kfunc_md_lock. + */ +static struct kfunc_md *kfunc_md_fast_create(unsigned long ip, int nr_args) +{ + u8 nop_insn[insn_size], insn[insn_size]; + struct kfunc_md *md; + u32 index; + int err; + + md =3D kfunc_md_fast_get(ip); + if (md) { + md->users++; + return md; + } + + md =3D kfunc_md_fast_next(&index); + if (!md) + return NULL; + + memset(md, 0, sizeof(*md)); + err =3D percpu_ref_init(&md->pcref, kfunc_md_release, 0, GFP_KERNEL); + if (err) + return NULL; + + kfunc_md_arch_pretend(insn, index); + kfunc_md_arch_nops(nop_insn); + + if (kfunc_md_text_poke(ip, insn, nop_insn)) { + atomic_dec(&kfunc_mds->kfunc_md_used); + percpu_ref_exit(&md->pcref); + return NULL; + } + + md->users =3D 1; + md->func =3D ip; + md->array =3D kfunc_mds; + md->nr_args =3D nr_args; + + return md; +} +#else + +static void kfunc_md_fast_put(struct kfunc_md *md) +{ +} + +static struct kfunc_md *kfunc_md_fast_get(unsigned long ip) +{ + return NULL; +} + +static struct kfunc_md *kfunc_md_fast_create(unsigned long ip, int nr_args) +{ + return NULL; +} + +#endif /* !CONFIG_FUNCTION_METADATA */ + +void kfunc_md_enter(struct kfunc_md *md) +{ + percpu_ref_get(&md->pcref); +} +EXPORT_SYMBOL_GPL(kfunc_md_enter); + +void kfunc_md_exit(struct kfunc_md *md) +{ + percpu_ref_put(&md->pcref); +} +EXPORT_SYMBOL_GPL(kfunc_md_exit); + +void kfunc_md_unlock(void) +{ + mutex_unlock(&kfunc_md_mutex); +} +EXPORT_SYMBOL_GPL(kfunc_md_unlock); + +void kfunc_md_lock(void) +{ + mutex_lock(&kfunc_md_mutex); +} +EXPORT_SYMBOL_GPL(kfunc_md_lock); + +#undef CALL +#define CALL(fast, slow, type, ...) ({ \ + type ___ret; \ + if (kfunc_md_fast()) \ + ___ret =3D fast(__VA_ARGS__); \ + else \ + ___ret =3D slow(__VA_ARGS__); \ + ___ret; \ +}) + +#undef CALL_VOID +#define CALL_VOID(fast, slow, ...) do { \ + if (kfunc_md_fast()) \ + fast(__VA_ARGS__); \ + else \ + slow(__VA_ARGS__); \ +} while (0) + +struct kfunc_md *kfunc_md_get_noref(unsigned long ip) +{ + return CALL(kfunc_md_fast_get, kfunc_md_hash_get, struct kfunc_md *, + ip); +} +EXPORT_SYMBOL_GPL(kfunc_md_get_noref); + +struct kfunc_md *kfunc_md_get(unsigned long ip) +{ + struct kfunc_md *md; + + md =3D CALL(kfunc_md_fast_get, kfunc_md_hash_get, struct kfunc_md *, + ip); + if (md) + md->users++; + return md; +} +EXPORT_SYMBOL_GPL(kfunc_md_get); + +void kfunc_md_put(unsigned long ip) +{ + struct kfunc_md *md =3D kfunc_md_get_noref(ip); + + if (md) + CALL_VOID(kfunc_md_fast_put, kfunc_md_hash_put, md); +} +EXPORT_SYMBOL_GPL(kfunc_md_put); + +/* Decrease the reference of the md, release it if "md->users <=3D 0" */ +void kfunc_md_put_entry(struct kfunc_md *md) +{ + if (!md) + return; + + CALL_VOID(kfunc_md_fast_put, kfunc_md_hash_put, md); +} +EXPORT_SYMBOL_GPL(kfunc_md_put_entry); + +struct kfunc_md *kfunc_md_create(unsigned long ip, int nr_args) +{ + return CALL(kfunc_md_fast_create, kfunc_md_hash_create, + struct kfunc_md *, ip, nr_args); +} +EXPORT_SYMBOL_GPL(kfunc_md_create); + +bool __weak kfunc_md_arch_support(int *insn, int *data) +{ + return false; +} + +static int __init kfunc_md_init_test(void) +{ +#ifndef CONFIG_FUNCTION_METADATA_PADDING + /* When the CONFIG_FUNCTION_METADATA_PADDING is not available, try + * to probe the usable function padding dynamically. + */ + if (!kfunc_md_arch_support(&__insn_offset, &__data_offset)) + static_branch_disable(&kfunc_md_use_padding); +#endif + return 0; +} +late_initcall(kfunc_md_init_test); --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF4AD1E98F3; Wed, 28 May 2025 03:49:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404184; cv=none; b=mH08ZSeSR2S8vqkEqs8EIyHPaY05wY61YRYR8HKRSov0yTsMc4dErb1yNeMXDfVFFosHh6gKNcjcgR5Met3N33P2QuQuezzCDLl+6FF9LnwWmT4Fvck0iA825RNcH6oLNrZ+sBVcWWq+Yw2y+sR5tznJT44dgt7X7flz1LCwSn4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404184; c=relaxed/simple; bh=brjzoRfQr/GEtdVUjTUnIgAaWrgzva+2OjgAcyy2xVE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oFj36sbPVijsE+Cn1bRErbre7UQUy5aitsCU3uEUVaN614pWkaa4vDKsVehp5Q7d6F8ogne4gOa3mMvLr7XdXn812EcA7ThCjoX4XnC2M09GJm6Aptm1/zayiR3RemfhadOn5zQEESSZXe/jk9n7fQqUfdDVSG6kBYAJomrebuM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=T3J8oCZU; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="T3J8oCZU" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-234bfe37cccso6984225ad.0; Tue, 27 May 2025 20:49:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404181; x=1749008981; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bsDc4QLUgbCBBYtWAxHqZcjIGHY4TkYNVyWhaF+ewHY=; b=T3J8oCZUaWNxJVHkoyQSWJHhSI5jUbxSrMqfGP52DAKYTjqunr7SChIfkBGwZZ0SeL A8y2dg5WmwZQLzndzD31FYj17wDJnT/b0XfZrFW4uoD1HoIiZNu3zSx8AjVtpEtzYMtr y21xTlU/yepOVDGEGTTBwyid/wm+gDWtOyBVBSrqjQU1EeIgCXMCrGu5DBew3n20AZLl moYphd0YT0RjR+V5J+b4iCvH7iyIv0XXYjinxioAwmW7yL0NYO88Rj318LqplGp21ifO uUnsa6EgYklgE27S2hb9XsbWrbPDDcOnPDAWUWaAXZN13+Lg1vyl8Eh6Y9XfGeBZngf6 Gncw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404181; x=1749008981; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bsDc4QLUgbCBBYtWAxHqZcjIGHY4TkYNVyWhaF+ewHY=; b=dxwj2MnpsSUzwTNBD3z3ioHDymSNdUSWaZSCd/g1W0a1JqYlFbHrINk59RV/MuBG5d XaVkZlzeyo5w04cKSf45/uv3Drj471QgvNaHakfbRIxZogIubBRQWC92kW+RtvhZ4f4/ kQeua4eCWGGx+pSTtTuRfepNSe86bpGctYkRrZEtgNXKHtqm8cKjo8WcMi6AHU2XqmEs eYCQ25iz6w0gGZUonWAK8w8hkp7gPdVBkC6sT7nPOJhxm9u1Op4f8N4LX/RgRLHRnTCh NGG+FhIbwOvbw5PWWMvJWMesrREaN/NbGMzOwGZ3RamnEog2Tt0brRkFVH5E7dJoWERM 4ohg== X-Forwarded-Encrypted: i=1; AJvYcCUKnYzWswMbmGPdgUQy0S63v240pW2XSOqXUzxPZQeOUrr3lxQLSQlQWYcNDVicfD3Nm5YNkvt0C4yhZvE=@vger.kernel.org X-Gm-Message-State: AOJu0YzD8nCkxul0ulsCDSuUM5uKZ1V6q1o2ovUuNd69geF9a1RqbVNz TExeTP2VO57NgCrXParo8jGrdxHpGwFyYrsVOQ/vNXLgQgRMHT8VsLcE X-Gm-Gg: ASbGncugMAyoKBaVPYiEibhR5HC5TWskpX/HeSGh2LctYdNXXojuH/gP8QFaJY47pev ypc67dzWCdbcw55B7km9C2n4INe4H8CVMKaFHR+JjSyJXv4Qv27YPFuwC0mJ6E5wc/LRTZQexAb 3+m0UP/qZBXboPrCxtNZGLq1qy+jJ6/6WGgEfP4fOC7jI6MxBKDWpCoAuiWjiMkGL9NYApagZdF HjKjcb5CkaAR9nBSEsdt9qOnrnTGWE0/Z1UQZHvbvkW7CGmg4spb0U1efWC6mGs8/153qCVXbZb fj5KuGg854ktur/p6kLMHw6Kvi3SIvvVr2eRr2BICiO0hPW5GS0tclP1l7iMBYtKxAOYOfxLBcj sRbA= X-Google-Smtp-Source: AGHT+IFFmuM2KuoJrIGiY+X6Xa/3xOqPHrS80F9eVMW7zRPeXztvJ1YZLoiJAKud8BF+8v168sJ6oA== X-Received: by 2002:a17:903:2988:b0:224:e0f:4b5 with SMTP id d9443c01a7336-23414f3aa7bmr195721765ad.7.1748404180895; Tue, 27 May 2025 20:49:40 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:40 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 02/25] x86: implement per-function metadata storage for x86 Date: Wed, 28 May 2025 11:46:49 +0800 Message-Id: <20250528034712.138701-3-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With CONFIG_CALL_PADDING enabled, there will be 16-bytes padding space before all the kernel functions. And some kernel features can use it, such as MITIGATION_CALL_DEPTH_TRACKING, CFI_CLANG, FINEIBT, etc. In my research, MITIGATION_CALL_DEPTH_TRACKING will consume the tail 9-bytes in the function padding, CFI_CLANG will consume the head 5-bytes, and FINEIBT will consume all the 16 bytes if it is enabled. So there will be no space for us if MITIGATION_CALL_DEPTH_TRACKING and CFI_CLANG are both enabled, or FINEIBT is enabled. In order to implement the padding-based function metadata, we need 5-bytes to prepend a "mov %eax xxx" insn in x86_64, which can hold a 4-bytes index. So we have following logic: 1. use the head 5-bytes if CFI_CLANG is not enabled 2. use the tail 5-bytes if MITIGATION_CALL_DEPTH_TRACKING and FINEIBT are not enabled 3. try to probe if fineibt or the call thunks is enabled after the kernel boot dynamically On the third case, we implement the function metadata by hash table if "cfi_mode=3D=3DCFI_FINEIBT || thunks_initialized". Therefore, we need to ma= ke thunks_initialized global in arch/x86/kernel/callthunks.c Signed-off-by: Menglong Dong --- arch/x86/Kconfig | 26 +++++++++++++++++ arch/x86/include/asm/alternative.h | 2 ++ arch/x86/include/asm/ftrace.h | 47 ++++++++++++++++++++++++++++++ arch/x86/kernel/callthunks.c | 2 +- arch/x86/kernel/ftrace.c | 26 +++++++++++++++++ 5 files changed, 102 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 4b9f378e05f6..0405288c42c6 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2459,6 +2459,32 @@ config PREFIX_SYMBOLS def_bool y depends on CALL_PADDING && !CFI_CLANG =20 +config FUNCTION_METADATA + bool "Per-function metadata storage support" + default y + depends on CC_HAS_ENTRY_PADDING && OBJTOOL + help + Support function padding based per-function metadata storage for + kernel functions, and get the metadata of the function by its + address with almost no overhead. + + The index of the metadata will be stored in the function padding + and consumes 5-bytes. + + Hash table based function metadata will be used if this option + is not enabled. + +config FUNCTION_METADATA_PADDING + bool "function padding is available for metadata" + default y + depends on FUNCTION_METADATA && !FINEIBT && !(CFI_CLANG && CALL_THUNKS) + select CALL_PADDING + help + Function padding is available for the function metadata. If this + option is disabled, function metadata will try to probe if there + are usable function padding during the system boot. If not, the + hash table based function metadata will be used instead. + menuconfig CPU_MITIGATIONS bool "Mitigations for CPU vulnerabilities" default y diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alte= rnative.h index 4a37a8bd87fd..951edf1857c3 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -103,6 +103,8 @@ struct callthunk_sites { }; =20 #ifdef CONFIG_CALL_THUNKS +extern bool thunks_initialized; + extern void callthunks_patch_builtin_calls(void); extern void callthunks_patch_module_calls(struct callthunk_sites *sites, struct module *mod); diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 93156ac4ffe0..ed1fdfce824e 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -4,6 +4,21 @@ =20 #include =20 +#ifdef CONFIG_FUNCTION_METADATA_PADDING + +#ifdef CONFIG_CFI_CLANG +/* use the space that CALL_THUNKS suppose to use */ +#define KFUNC_MD_INSN_OFFSET (5) +#define KFUNC_MD_DATA_OFFSET (4) +#else +/* use the space that CFI_CLANG suppose to use */ +#define KFUNC_MD_INSN_OFFSET (CONFIG_FUNCTION_PADDING_BYTES) +#define KFUNC_MD_DATA_OFFSET (CONFIG_FUNCTION_PADDING_BYTES - 1) +#endif +#endif + +#define KFUNC_MD_INSN_SIZE (5) + #ifdef CONFIG_FUNCTION_TRACER #ifndef CC_USING_FENTRY # error Compiler does not support fentry? @@ -154,6 +169,38 @@ static inline bool arch_trace_is_compat_syscall(struct= pt_regs *regs) } #endif /* CONFIG_FTRACE_SYSCALLS && CONFIG_IA32_EMULATION */ #endif /* !COMPILE_OFFSETS */ + +#ifdef CONFIG_FUNCTION_METADATA +#include + +static inline bool kfunc_md_arch_exist(unsigned long ip, int insn_offset) +{ + return *(u8 *)(ip - insn_offset) =3D=3D 0xB8; +} + +static inline void kfunc_md_arch_pretend(u8 *insn, u32 index) +{ + *insn =3D 0xB8; + *(u32 *)(insn + 1) =3D index; +} + +static inline void kfunc_md_arch_nops(u8 *insn) +{ + *(insn++) =3D BYTES_NOP1; + *(insn++) =3D BYTES_NOP1; + *(insn++) =3D BYTES_NOP1; + *(insn++) =3D BYTES_NOP1; + *(insn++) =3D BYTES_NOP1; +} + +static inline int kfunc_md_arch_poke(void *ip, u8 *insn, int insn_offset) +{ + text_poke(ip, insn, insn_offset); + text_poke_sync(); + return 0; +} +#endif /* CONFIG_FUNCTION_METADATA */ + #endif /* !__ASSEMBLER__ */ =20 #endif /* _ASM_X86_FTRACE_H */ diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index d86d7d6e750c..6ed49904cd61 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -56,7 +56,7 @@ struct core_text { const char *name; }; =20 -static bool thunks_initialized __ro_after_init; +bool thunks_initialized __ro_after_init; =20 static const struct core_text builtin_coretext =3D { .base =3D (unsigned long)_text, diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index cace6e8d7cc7..2504c2556508 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -26,6 +26,7 @@ #include #include #include +#include =20 #include =20 @@ -569,6 +570,31 @@ void arch_ftrace_trampoline_free(struct ftrace_ops *op= s) ops->trampoline =3D 0; } =20 +#if defined(CONFIG_FUNCTION_METADATA) && !defined(CONFIG_FUNCTION_METADATA= _PADDING) +bool kfunc_md_arch_support(int *insn, int *data) +{ + /* when fineibt is enabled, the 16-bytes padding are all used */ + if (IS_ENABLED(CONFIG_FINEIBT) && cfi_mode =3D=3D CFI_FINEIBT) + return false; + + if (IS_ENABLED(CONFIG_CALL_THUNKS) && IS_ENABLED(CONFIG_CFI_CLANG)) { + /* when call thunks and cfi are both enabled, no enough space + * for us. + */ + if (thunks_initialized) + return false; + /* use the tail 5-bytes for function meta data */ + *insn =3D 5; + *data =3D 4; + + return true; + } + + WARN_ON_ONCE(1); + return true; +} +#endif + #endif /* CONFIG_X86_64 */ #endif /* CONFIG_DYNAMIC_FTRACE */ =20 --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC07B1EF36B; Wed, 28 May 2025 03:49:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404186; cv=none; b=NxK8PYuHsWNMI/m7wYEZODOqzzIc0+Bpjpvrzh/zh5JKzRARoMi7KGh5OLErUv6KMLuSGSQUFJyOgaWtTCPtph5qbVGhEaZLg7/lHh4AVE05VNzFbI28GWv9bxsbB0kUDmUcPmyEAiTisJbic8Q5HW0l5PUYBssCOONMT1jF/Ww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404186; c=relaxed/simple; bh=PzcQU/jU8BoBn8QTc/rCoOpUq8QKomNni3rt844DF8U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PAKugwmchozDIeHOgoprMEDkOAw0gekr0snv+WyE850NAmu6rqAqBZmvv8z1iTOVfgAB714KDOMMYiZoj+KIpnSQr+UMONGZ0jG/eQVeye+tYtneBjqITpKqzAQ7WqkL2Av21gBpPfZUPBFUhpH4EHOQcMfgwa/NjyVuWA4gu7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mDTDdCqe; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mDTDdCqe" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-234ade5a819so11268865ad.1; Tue, 27 May 2025 20:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404183; x=1749008983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zYPTFZRK7Anqseg021BaLIF06orSQsH0yc5V0T15hTo=; b=mDTDdCqefiPSvRnwve7/gZS7seJXwqsqOIfAc9gnk8itF2ttwdAHaty32ezClRMe/P xt3WUrd8gNzNEkXaTfyqrs0mPixGHaIj5V7JZb7IRVnqAHPxv8NvovKsbSYy8KZBoSDX cgzFud8qgWV21LWJbERsdCr0obQBGSbEUftmVQIOTDV1ZoFPFvRF+JJA2NPum25YOfzy EuNb1cvxdhh3LTdeG2CNZ51+76hKZ2WXXOOfmVLgARXaKO3HZxA48Ny5LlSyiMlx1os2 Ta2x1ouEpmdPRsWpZY7bHIOv+1RfrOdVSgUEfxAOphNo2DxUFl8e5hYuY6qHjZdF0xH4 IQkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404183; x=1749008983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zYPTFZRK7Anqseg021BaLIF06orSQsH0yc5V0T15hTo=; b=pTHZMzTdkc3fIWXjHcTsvPy2JuxanVq10tEr4WB8aabvdFV+3SZshyiE2IWKVaJI5B IJ7c+/SF9uYy85PuAMzl0y8mQIsqLD16OT3yAwWrvlOFcVlGmSkOXVMWNh8SexUqmLSQ rr7JxkISkKfbxnFjCMdTGY5V3EpwJdGMOAO6pHEmJIxGjzFVvQvbtp5uWh9tReluZPzC bjrKa5rr7C0/SCDBDP5ftfOGV3CS3HApY6Q0H+lqrne0rzbXvmLAfv/BX63e1x4yN1rX lrOsV2d2Bp3DQ+8FgswKSf/OXEmAyErRq+xw14n65CZweXsmS3ImKolCxYIEVnRiUbK4 OhoA== X-Forwarded-Encrypted: i=1; AJvYcCWP3gkxaeuWzLpCj2mtDwHoAhrXT6TsFH8Q6ue2nzYbSNnu17Pcg21/093XMUTmCaDQf7S+7UTv3sFdfHA=@vger.kernel.org X-Gm-Message-State: AOJu0Yw8drU/oTb5a64qbafCAlEgKgsTWhvupM10TfRVj0iW3QD79q5P AunZNMLBWv9/iNq9iEaekm5s2FA81+ynFX+DZ74SDUt2CANvy3uwZsR7 X-Gm-Gg: ASbGnctcJPcwg+KUk4gicdK9jC5KAwjvYrTaZYVhdMkrHeytbib9CSV1lcD060kdWD3 DhEszU5kxIVUTWl4VWKLqRskfuX6SGHbg84w9UrBCeF+kLkbsb47gl/2+BgmL1yNgJ36qQZT72N wrcTY/wXN85Bd7MR9EpWCoLyr7UVpiTAa9CBQUzl9zfHLJNs6bPvht+yWxU6NoAdN5REd8BxdAx y3uBhHsGyVavi4cdb+Lg2mmUeBXmQR1ujfREUxA9dSw2HObjPKIElkuE4TYF/TRcS6CRGirE1p3 PBuTdEhYNAvapZrl5XyPoQcyFDKsLwG0kGgDE0j2VkLTJFYziMcg6EenxD0Hn4On/XqL X-Google-Smtp-Source: AGHT+IHZdGdpsTzeRfx9xKiCMkdWV+5vKYFIU4x8VuwMZsmqeTu/rcGsgPCx9pnheEwC+vB2FDl/Fw== X-Received: by 2002:a17:902:e842:b0:22e:4a2e:8ae7 with SMTP id d9443c01a7336-23414f7d0ebmr288136065ad.22.1748404183038; Tue, 27 May 2025 20:49:43 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:42 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 03/25] arm64: implement per-function metadata storage for arm64 Date: Wed, 28 May 2025 11:46:50 +0800 Message-Id: <20250528034712.138701-4-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The per-function metadata storage is already used by ftrace if CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS is enabled, and it store the pointer of the callback directly to the function padding, which consume 8-bytes, in the commit baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS"). So we can directly store the index to the function padding too, without a prepending. With CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS enabled, the function is 8-bytes aligned, and we will compile the kernel with extra 8-bytes (2 NOPS) padding space. Otherwise, the function is 4-bytes aligned, and only extra 4-bytes (1 NOPS) is needed for us. However, we have the same problem with Mark in the commit above: we can't use the function padding together with CFI_CLANG, which can make the clang compiles a wrong offset to the pre-function type hash. So we fallback to the hash table mode for function metadata if CFI_CLANG is enabled. Signed-off-by: Menglong Dong --- arch/arm64/Kconfig | 21 ++++++++++++++++++++ arch/arm64/Makefile | 23 ++++++++++++++++++++-- arch/arm64/include/asm/ftrace.h | 34 +++++++++++++++++++++++++++++++++ arch/arm64/kernel/ftrace.c | 13 +++++++++++-- 4 files changed, 87 insertions(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a182295e6f08..db504df07072 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1549,6 +1549,27 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. =20 +config FUNCTION_METADATA + bool "Per-function metadata storage support" + default y + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE if !FTRACE_MCOUNT_USE_PATCHABLE_F= UNCTION_ENTRY + depends on !CFI_CLANG + help + Support function padding based per-function metadata storage for + kernel functions, and get the metadata of the function by its + address with almost no overhead. + + The index of the metadata will be stored in the function padding, + which will consume 4-bytes. If FUNCTION_ALIGNMENT_8B is enabled, + extra 8-bytes function padding will be reserved during compiling. + Otherwise, only extra 4-bytes function padding is needed. + + Hash table based function metadata will be used if this option + is not enabled. + +config FUNCTION_METADATA_PADDING + def_bool FUNCTION_METADATA + source "kernel/Kconfig.hz" =20 config ARCH_SPARSEMEM_ENABLE diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 1d5dfcd1c13e..576d6ab94dc5 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -144,12 +144,31 @@ endif =20 CHECKFLAGS +=3D -D__aarch64__ =20 +ifeq ($(CONFIG_FUNCTION_METADATA_PADDING),y) + ifeq ($(CONFIG_FUNCTION_ALIGNMENT_8B),y) + __padding_nops :=3D 2 + else + __padding_nops :=3D 1 + endif +else + __padding_nops :=3D 0 +endif + ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS),y) + __padding_nops :=3D $(shell echo $(__padding_nops) + 2 | bc) KBUILD_CPPFLAGS +=3D -DCC_USING_PATCHABLE_FUNCTION_ENTRY - CC_FLAGS_FTRACE :=3D -fpatchable-function-entry=3D4,2 + CC_FLAGS_FTRACE :=3D -fpatchable-function-entry=3D$(shell echo $(__paddi= ng_nops) + 2 | bc),$(__padding_nops) else ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_ARGS),y) + CC_FLAGS_FTRACE :=3D -fpatchable-function-entry=3D$(shell echo $(__paddi= ng_nops) + 2 | bc),$(__padding_nops) KBUILD_CPPFLAGS +=3D -DCC_USING_PATCHABLE_FUNCTION_ENTRY - CC_FLAGS_FTRACE :=3D -fpatchable-function-entry=3D2 +else ifeq ($(CONFIG_FUNCTION_METADATA_PADDING),y) + CC_FLAGS_FTRACE +=3D -fpatchable-function-entry=3D$(__padding_nops),$(__= padding_nops) + ifneq ($(CONFIG_FUNCTION_TRACER),y) + KBUILD_CFLAGS +=3D $(CC_FLAGS_FTRACE) + # some file need to remove this cflag when CONFIG_FUNCTION_TRACER + # is not enabled, so we need to export it here + export CC_FLAGS_FTRACE + endif endif =20 ifeq ($(CONFIG_KASAN_SW_TAGS), y) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrac= e.h index bfe3ce9df197..9aafb3103829 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -24,6 +24,16 @@ #define FTRACE_PLT_IDX 0 #define NR_FTRACE_PLTS 1 =20 +#ifdef CONFIG_FUNCTION_METADATA_PADDING +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS +#define KFUNC_MD_DATA_OFFSET (AARCH64_INSN_SIZE * 3) +#else +#define KFUNC_MD_DATA_OFFSET AARCH64_INSN_SIZE +#endif +#define KFUNC_MD_INSN_SIZE AARCH64_INSN_SIZE +#define KFUNC_MD_INSN_OFFSET KFUNC_MD_DATA_OFFSET +#endif + /* * Currently, gcc tends to save the link register after the local variables * on the stack. This causes the max stack tracer to report the function @@ -216,6 +226,30 @@ static inline bool arch_syscall_match_sym_name(const c= har *sym, */ return !strcmp(sym + 8, name); } + +#ifdef CONFIG_FUNCTION_METADATA_PADDING +#include + +static inline bool kfunc_md_arch_exist(unsigned long ip, int insn_offset) +{ + return !aarch64_insn_is_nop(*(u32 *)(ip - insn_offset)); +} + +static inline void kfunc_md_arch_pretend(u8 *insn, u32 index) +{ + *(u32 *)insn =3D index; +} + +static inline void kfunc_md_arch_nops(u8 *insn) +{ + *(u32 *)insn =3D aarch64_insn_gen_nop(); +} + +static inline int kfunc_md_arch_poke(void *ip, u8 *insn, int insn_offset) +{ + return aarch64_insn_patch_text_nosync(ip, *(u32 *)insn); +} +#endif #endif /* ifndef __ASSEMBLY__ */ =20 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 5a890714ee2e..869946dabdd0 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -88,8 +88,10 @@ unsigned long ftrace_call_adjust(unsigned long addr) * to `BL `, which is at `addr + 4` bytes in either case. * */ - if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) - return addr + AARCH64_INSN_SIZE; + if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) { + addr +=3D AARCH64_INSN_SIZE; + goto out; + } =20 /* * When using patchable-function-entry with pre-function NOPs, addr is @@ -139,6 +141,13 @@ unsigned long ftrace_call_adjust(unsigned long addr) =20 /* Skip the first NOP after function entry */ addr +=3D AARCH64_INSN_SIZE; +out: + if (IS_ENABLED(CONFIG_FUNCTION_METADATA_PADDING)) { + if (IS_ENABLED(CONFIG_FUNCTION_ALIGNMENT_8B)) + addr +=3D 2 * AARCH64_INSN_SIZE; + else + addr +=3D AARCH64_INSN_SIZE; + } =20 return addr; } --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 270811F3BA9; Wed, 28 May 2025 03:49:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404187; cv=none; b=Vnhtz+4k6Gc0oy9P3xFf6/c29l8cIXvUNKQlohoZJt7FdsS1AcSVA1ns9zoCslQ+gUnH6Hw3ubU1qpYr2zbC5z7OP3JDFtEw8kPybsJqedKnzCYByXmdNIUe3XznqeQJ3suVJuPU6KaynG8p4jsUyCFUt2xE4Q89T5CnWMDvBRU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404187; c=relaxed/simple; bh=id6lILaKdgBMWZnpVnlNqUcvsPute9DuoXqlXPJA44M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MRhuOg7R2DaJ9TmBierJoEF9Iu1baqHYZkyHXi+w9YEKmh91FsUBnGYC0ggoZxIOuI3pKG0rrr0eGmuDFYOP0ru9cFvHPzEnZv7s+ATdUw/kzCkDL6cd+Rd5woRvOs3rE6CFJFv+fnjteqful98M8SVtckm5pLfYXKflWsKJ1ms= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=O4UobeVk; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="O4UobeVk" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-234c5b57557so4023365ad.3; Tue, 27 May 2025 20:49:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404185; x=1749008985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4y9BozBA9c7G/gYDE1RLJDCqW9V05vfoBze499QJjhM=; b=O4UobeVkNlZhQoGr1n4S9Yqb9XmBbR9BYo9cmADT7dHko4B93HkST0yrSTdTridsze gh8j3ZVC0XVc6CbRFZ4T9aN9XHP3McMJnwbCRSaSwqLznKUNMuFHNoRro90r0SfzGjCJ 2TdYduJXW/CI6Hoy0gTieOevIF8Ux6APGra8EoHQZML6et3bNpTeRm1rCHkFVOlXnLmU M6mI5dYX99HzuK+rkzyM67gHUtgDw8Vq5OCKHIZzxzpD4fFH5nv5nddZJn9XJq/nR2Jf drlcj2yUkeFqh36aJeJw2vZ10//bUW+XeYYNb7sDLsWsfWpnHN0g6noBgq9S/ochS6rS kFmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404185; x=1749008985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4y9BozBA9c7G/gYDE1RLJDCqW9V05vfoBze499QJjhM=; b=boeFxjf/zM1/eZ8BIJ1MuLBXPLjsZpYf2KW4wIBG2E4QFtO31xr5RBcXRPN0Wcs6kU k3RDJvty93CJ7aIFOIuS3UTsizkiElcAFbfI6LS+r9fNHQHJXI5v8nQ9pt9gm030ilf1 KCfvfeESOiTJDutfyMAWbOrgtQfi0VhZ3N4IdTlR71zeSu1fQ28viwQCPZl+a4s3q79c eKkRqXgon6MHOTN9HhGffI4qya8SqIx8UyeknnLT4AWOkCyu2nzeNwMA9aOlkZ40kJK/ bMLWchOfQyaFP6LSXcZaRCnQvlmX5tqTMqBTBfANptpP7eTfMCYHPSKZSbdg6EviiDOz TCyA== X-Forwarded-Encrypted: i=1; AJvYcCXtZN0O6FS/bOuKE4dxcuZwYr0sXuGRtG/21lYgyXL+GBZqrzyQDJow9FX2CZqjTFDHrm72ECU/yv9sDLs=@vger.kernel.org X-Gm-Message-State: AOJu0Yzv8RSpRMh7QvouV8rbFkNjL72VBq86mtf5XD/hvpIQrX1zyNYC dO7tfy2bvkywvEtxycHbWayVzySHQnpGy3bPWBPLwLybzuTPUBBO4f4t X-Gm-Gg: ASbGncv+KMSzWe/6uuuXCtmCRJOWPEFiNBCXILkWBmct6+Jc5EvfFO3+FoPVDWoCO6T VKbVxV2czqn5XGvKCGqe9YorXmy9NopEJJDHLzSxh7QKGw1hTalyJBSE3fqfbwYqpnUgjSHF5W4 em9hmEJ0xc7J6tMJkaRGxaQHGkgxxvyxuV6dcTKz0MkgDNuHCIY4q36x47bErPtm06NrW3XwRb+ CMUclYrkrZj1mr6ot4gVsR7wZTgpWAuaminsmQv6mSbb3ISguNEcAkVK/EysQuCgWc7dGGDASnM q4U4z9ZhO4Nv45m6iGg0YDhM75ZzuID+JH04XPuXjfXQCzenwNv1Rvfu5JzVYIqtCNYL7hJ9dH6 ENtY= X-Google-Smtp-Source: AGHT+IHgMrvnRk4C4QBdIIOR03RXWg25Wd0c+6j2FeK13WU7gMN5nz5BZ9oiNEdlJQNbPpve9T5v4g== X-Received: by 2002:a17:903:40c5:b0:234:9fed:ccb2 with SMTP id d9443c01a7336-2349fedce43mr78841355ad.29.1748404185208; Tue, 27 May 2025 20:49:45 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:44 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 04/25] bpf: make kfunc_md support global trampoline link Date: Wed, 28 May 2025 11:46:51 +0800 Message-Id: <20250528034712.138701-5-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce the struct kfunc_md_tramp_prog for BPF_PROG_TYPE_TRACING, and add the field "bpf_progs" to struct kfunc_md. These filed will be used in the next patch of bpf global trampoline. And the KFUNC_MD_FL_TRACING_ORIGIN is introduced to indicate that origin call is needed on this function. Add the function kfunc_md_bpf_link and kfunc_md_bpf_unlink to add or remove bpf prog to kfunc_md. Meanwhile, introduce kunfc_md_bpf_ips() to get all the kernel functions in kfunc_mds that contains bpf progs. The KFUNC_MD_FL_BPF_REMOVING indicate that a removing operation is in progress, and we shouldn't return it if "bpf_prog_cnt<=3D1" in kunfc_md_bpf_ips(). Signed-off-by: Menglong Dong --- include/linux/kfunc_md.h | 17 ++++++ kernel/trace/kfunc_md.c | 118 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) diff --git a/include/linux/kfunc_md.h b/include/linux/kfunc_md.h index 21c0b879cc03..f1b1012eeab2 100644 --- a/include/linux/kfunc_md.h +++ b/include/linux/kfunc_md.h @@ -3,12 +3,21 @@ #define _LINUX_KFUNC_MD_H =20 #define KFUNC_MD_FL_DEAD (1 << 0) /* the md shouldn't be reused */ +#define KFUNC_MD_FL_TRACING_ORIGIN (1 << 1) +#define KFUNC_MD_FL_BPF_REMOVING (1 << 2) =20 #ifndef __ASSEMBLER__ =20 #include #include =20 +struct kfunc_md_tramp_prog { + struct kfunc_md_tramp_prog *next; + struct bpf_prog *prog; + u64 cookie; + struct rcu_head rcu; +}; + struct kfunc_md_array; =20 struct kfunc_md { @@ -19,6 +28,7 @@ struct kfunc_md { struct rcu_head rcu; #endif unsigned long func; + struct kfunc_md_tramp_prog *bpf_progs[BPF_TRAMP_MAX]; #ifdef CONFIG_FUNCTION_METADATA /* the array is used for the fast mode */ struct kfunc_md_array *array; @@ -26,6 +36,7 @@ struct kfunc_md { struct percpu_ref pcref; u32 flags; u16 users; + u8 bpf_prog_cnt; u8 nr_args; }; =20 @@ -40,5 +51,11 @@ void kfunc_md_exit(struct kfunc_md *md); void kfunc_md_enter(struct kfunc_md *md); bool kfunc_md_arch_support(int *insn, int *data); =20 +int kfunc_md_bpf_ips(void ***ips); + +int kfunc_md_bpf_unlink(struct kfunc_md *md, struct bpf_prog *prog, int ty= pe); +int kfunc_md_bpf_link(struct kfunc_md *md, struct bpf_prog *prog, int type, + u64 cookie); + #endif #endif diff --git a/kernel/trace/kfunc_md.c b/kernel/trace/kfunc_md.c index 9571081f6560..ebb4e46d482d 100644 --- a/kernel/trace/kfunc_md.c +++ b/kernel/trace/kfunc_md.c @@ -131,6 +131,23 @@ static bool kfunc_md_fast(void) { return static_branch_likely(&kfunc_md_use_padding); } + +static int kfunc_md_hash_bpf_ips(void **ips) +{ + struct hlist_head *head; + struct kfunc_md *md; + int c =3D 0, i; + + for (i =3D 0; i < (1 << KFUNC_MD_HASH_BITS); i++) { + head =3D &kfunc_md_table[i]; + hlist_for_each_entry(md, head, hash) { + if (md->bpf_prog_cnt > !!(md->flags & KFUNC_MD_FL_BPF_REMOVING)) + ips[c++] =3D (void *)md->func; + } + } + + return c; +} #else =20 static void kfunc_md_hash_put(struct kfunc_md *md) @@ -148,6 +165,11 @@ static struct kfunc_md *kfunc_md_hash_create(unsigned = long ip, int nr_args) } =20 #define kfunc_md_fast() 1 + +static int kfunc_md_hash_bpf_ips(void **ips) +{ + return 0; +} #endif /* CONFIG_FUNCTION_METADATA_PADDING */ =20 #ifdef CONFIG_FUNCTION_METADATA @@ -442,6 +464,19 @@ static struct kfunc_md *kfunc_md_fast_create(unsigned = long ip, int nr_args) =20 return md; } + +static int kfunc_md_fast_bpf_ips(void **ips) +{ + struct kfunc_md *md; + int i, c =3D 0; + + for (i =3D 0; i < kfunc_mds->kfunc_md_count; i++) { + md =3D &kfunc_mds->mds[i]; + if (md->users && md->bpf_prog_cnt > !!(md->flags & KFUNC_MD_FL_BPF_REMOV= ING)) + ips[c++] =3D (void *)md->func; + } + return c; +} #else =20 static void kfunc_md_fast_put(struct kfunc_md *md) @@ -458,6 +493,10 @@ static struct kfunc_md *kfunc_md_fast_create(unsigned = long ip, int nr_args) return NULL; } =20 +static int kfunc_md_fast_bpf_ips(void **ips) +{ + return 0; +} #endif /* !CONFIG_FUNCTION_METADATA */ =20 void kfunc_md_enter(struct kfunc_md *md) @@ -547,6 +586,85 @@ struct kfunc_md *kfunc_md_create(unsigned long ip, int= nr_args) } EXPORT_SYMBOL_GPL(kfunc_md_create); =20 +int kfunc_md_bpf_ips(void ***ips) +{ + void **tmp; + int c; + + c =3D atomic_read(&kfunc_mds->kfunc_md_used); + if (!c) + return 0; + + tmp =3D kmalloc_array(c, sizeof(*tmp), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + rcu_read_lock(); + c =3D CALL(kfunc_md_fast_bpf_ips, kfunc_md_hash_bpf_ips, int, tmp); + rcu_read_unlock(); + + *ips =3D tmp; + + return c; +} + +int kfunc_md_bpf_link(struct kfunc_md *md, struct bpf_prog *prog, int type, + u64 cookie) +{ + struct kfunc_md_tramp_prog *tramp_prog, **last; + + tramp_prog =3D md->bpf_progs[type]; + /* check if the prog is already linked */ + while (tramp_prog) { + if (tramp_prog->prog =3D=3D prog) + return -EEXIST; + tramp_prog =3D tramp_prog->next; + } + + tramp_prog =3D kmalloc(sizeof(*tramp_prog), GFP_KERNEL); + if (!tramp_prog) + return -ENOMEM; + + tramp_prog->prog =3D prog; + tramp_prog->cookie =3D cookie; + tramp_prog->next =3D NULL; + + /* add the new prog to the list tail */ + last =3D &md->bpf_progs[type]; + while (*last) + last =3D &(*last)->next; + *last =3D tramp_prog; + + md->bpf_prog_cnt++; + if (type =3D=3D BPF_TRAMP_FEXIT || type =3D=3D BPF_TRAMP_MODIFY_RETURN) + md->flags |=3D KFUNC_MD_FL_TRACING_ORIGIN; + + return 0; +} + +int kfunc_md_bpf_unlink(struct kfunc_md *md, struct bpf_prog *prog, int ty= pe) +{ + struct kfunc_md_tramp_prog *cur, **prev; + + prev =3D &md->bpf_progs[type]; + while (*prev && (*prev)->prog !=3D prog) + prev =3D &(*prev)->next; + + cur =3D *prev; + if (!cur) + return -EINVAL; + + *prev =3D cur->next; + kfree_rcu(cur, rcu); + md->bpf_prog_cnt--; + + if (!md->bpf_progs[BPF_TRAMP_FEXIT] && + !md->bpf_progs[BPF_TRAMP_MODIFY_RETURN]) + md->flags &=3D ~KFUNC_MD_FL_TRACING_ORIGIN; + + return 0; +} + bool __weak kfunc_md_arch_support(int *insn, int *data) { return false; --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53DC61FAC4B; Wed, 28 May 2025 03:49:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404190; cv=none; b=il+M48ksbThwQazPzf6IWIpUl6Jas+RvNmAglIWBXdwKBt771aYtvCnhEl1zeyLathxHr7/AM5QBato+Tmna/i/CpDEQAUPmlUJQ8rpk2lACvk2u6/yVCThaUmsHyGq49UW7WLsM5LX+fBB52eMt8XPp8c8hqcYytSDK+7LqEOw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404190; c=relaxed/simple; bh=yDOTdN1geg/TbHZc+fDhUVZt5n0IxaqnyAa52LAD0ag=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aZAC+BbAVsE65yeAiV96MbdhPgFh1f4/2qZlvTpmInQIUKkIKWG6bd/6LqFjAmJaTBZbMUI5ModWTLhiGCmMOi6LQZe8sD/z0BglxuuCjVtCvdf0bIHDttzxfhOdhJ71FklkIINk5iwWx2iFIoA818fk/XHjVk4i6VIzIhTIs7c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hM0fgQvf; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hM0fgQvf" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-23278ce78efso28586395ad.2; Tue, 27 May 2025 20:49:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404187; x=1749008987; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QqTzPgLr+m5glEqqQjNvOSOGxBWQn19GNEWF6ihzcEE=; b=hM0fgQvf4jYg/3V90PzRKWNvs/NYPyF2fYku+am1OuxEeBYLvr7ycWcpRCPAoy9u9R k/Iw/45Yj7tkS55NimEYIryxfV+ZTLl74uMXLYK6VWWxhwIhVHZJhF+w785Jbrpl034A HCXGyxbmXrFmn3IKXg3CxMytpFDpDE6VDN24ZZoS46jAzWz8+vjMnZzWNO/uFaoCjJLZ WFoo/6E/bz4BVUK/etQmpsQXHMJ8+VG++ZD4UtTYO1ruGJmYF56jSpB1uiIM7cJ5abMG FR7yN4V1q91JKjUQNzZZbrDTmOTHw/5H1LyEKwbd5PAgKoTw0axbVpItLZORfoWqAaSa 8z6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404187; x=1749008987; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QqTzPgLr+m5glEqqQjNvOSOGxBWQn19GNEWF6ihzcEE=; b=hXikZXKz748zT6HwvCZrHPHxWkt1R1oi5Tj7a7yMs28tWwn5eSuSxa6r/Si2VKVlKM BgUnt/N5onn4z2EcTwfxB3ISErcszBy74xhK768tvitEXMdCmIg296iHPvq3X8ibdr/q kgwVT1cmxO+5fAZyouVBEruBS5smryOpcRq2XynI1BczFvkSBH6T8+q+WoeqLOOUNK3r /CK1mgGVM9yvGF/HFVtkdtrYvtGYGfpCY6xcgX0GLXUGb/Z09/Ey3D9KZ/1+AgfUNViG dngjrL1FUrEzgKByz4kePB51nifts6HjTjMROXNS9I0iKsrLhGNFbjm4nXuzaWk/5x7F ANqg== X-Forwarded-Encrypted: i=1; AJvYcCUBkI8mFwg+oCUlSjidUewL7/Vr4/EmOPseJ0rifIGEJEBktZTxq3w8r85u94QHR+7OZROGL1QvIClVEAs=@vger.kernel.org X-Gm-Message-State: AOJu0YyGTHlXhAKTazuiHkNBNT5Dn+x1WI+hE8uRR+jNJx37+GNJFz55 bzggUuMbqg4WunmI8xywTHtA0NZVjj1nHfBjoOQ/4E6WruuGd8dqC8Xe X-Gm-Gg: ASbGncuOLorDBwKh8GOsU4rxvqHPxPYUnor4LScDRN63OWVqxnY4z6N7E04YlO9zG3a fVsaiu6yr7OgQZdQn5Nt+oIC3ex0evcUQSEd/eRmHw91RSYgMw5Cymo5rr6IavZv2owK3iAgRp2 jLJgWU8xueqU9rhCWNKWaVVxJHTCK+wjeRXL7xIGMeQ8Yso5EMECUt4Nnr4SbY0Z7M6wX4COgqP boxjf9YvxqW/4ILE8m3k6ncXtVnCEsPFVasbEg936Cj3FmH5X2niijKNp6juwFTtz+jBXd0SDD2 qI9fKfYjCewv7jVLGzy++WvFZpaPZUMk8CrWsebZZCQrMp6miaKcxM46RE8inJETJriM X-Google-Smtp-Source: AGHT+IGdQntgd4Zc/Jhr6r4tuWsvlozIWuOfn03xxWPEcd5HkuvUovYpYC60uVO9F3+QdLAtAPYREg== X-Received: by 2002:a17:903:32c8:b0:223:f9a4:3f99 with SMTP id d9443c01a7336-23414f9f230mr244494005ad.29.1748404187451; Tue, 27 May 2025 20:49:47 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:46 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 05/25] x86,bpf: add bpf_global_caller for global trampoline Date: Wed, 28 May 2025 11:46:52 +0800 Message-Id: <20250528034712.138701-6-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the bpf global trampoline "bpf_global_caller" for x86_64. The logic of it is similar to the bpf trampoline: 1. save the regs for function args. For now, only the function with args count no more than 6 is supported 2. save rbx and r12, which will be used to store the prog list and return value of __bpf_prog_enter_recur 3. get the origin function address on the stack. To get the real function address, we make it "&=3D $0xfffffffffffffff0", as it is always 16-bytes aligned 4. get the function metadata by calling kfunc_md_get_noref() 5. get the function args count from the kfunc_md and store it on the stack. 6. get the kfunc_md flags and store it on the stack. Call kfunc_md_enter() if origin call is needed 7. get the prog list for FENTRY, and run all the progs in the list with bpf_caller_prog_run 8. goto the end if origin call is not necessary 9. get the prog list for MODIFY_RETURN, and run all the progs in the list with bpf_caller_prog_run 10.restore the regs and do the origin call. We get the ip of the origin function by the rip in the stack 11.save the return value of the origin call to the stack. 12.get the prog list for FEXIT, and run all the progs in the list with bpf_caller_prog_run 13.restore rbx, r12, r13. In order to rebalance the RSB, we call bpf_global_caller_rsb here. Indirect call is used in bpf_caller_prog_run, as we load and call the function address from the stack in the origin call case. What's more, we get the bpf progs from the kfunc_md and call it indirectly. We make the indirect call with CALL_NOSPEC, and I'm not sure if it can prevent the Spectre. I just saw others do it in the same way :/ We use the r13 to keep the address where we put the return value of the origin call on the stack. The offset of it is "FUNC_ARGS_OFFSET + 8 * nr_args". The calling of kfunc_md_get_noref() should be within rcu_read_lock, which I don't, as this will increase the overhead of a function call. And I'm considering to make the calling of the bpf prog list within the rcu lock: rcu_read_lock() kfunc_md_get_noref() call fentry progs call modify_return progs rcu_read_unlock() call origin rcu_read_lock() call fexit progs rcu_read_unlock() I'm not sure why the general bpf trampoline don't do it this way. Because this will make the trampoline hold the rcu lock too long? Signed-off-by: Menglong Dong --- arch/x86/Kconfig | 4 + arch/x86/kernel/asm-offsets.c | 15 +++ arch/x86/kernel/ftrace_64.S | 231 ++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 4 + kernel/bpf/trampoline.c | 6 +- 5 files changed, 257 insertions(+), 3 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 0405288c42c6..6d37f814701a 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -153,6 +153,7 @@ config X86 select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH + select ARCH_HAS_BPF_GLOBAL_CALLER if X86_64 select BUILDTIME_TABLE_SORT select CLKEVT_I8253 select CLOCKSOURCE_WATCHDOG @@ -431,6 +432,9 @@ config PGTABLE_LEVELS default 3 if X86_PAE default 2 =20 +config ARCH_HAS_BPF_GLOBAL_CALLER + bool + menu "Processor type and features" =20 config SMP diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index ad4ea6fb3b6c..a35831be3054 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -12,7 +12,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -115,4 +117,17 @@ static void __used common(void) OFFSET(ARIA_CTX_rounds, aria_ctx, rounds); #endif =20 + DEFINE(RUN_CTX_SIZE, sizeof(struct bpf_tramp_run_ctx)); + OFFSET(RUN_CTX_cookie, bpf_tramp_run_ctx, bpf_cookie); + + OFFSET(BPF_PROG_func, bpf_prog, bpf_func); + + DEFINE(KFUNC_MD_SIZE, sizeof(struct kfunc_md)); + OFFSET(KFUNC_MD_progs, kfunc_md, bpf_progs); + OFFSET(KFUNC_MD_addr, kfunc_md, func); + OFFSET(KFUNC_MD_flags, kfunc_md, flags); + OFFSET(KFUNC_MD_nr_args, kfunc_md, nr_args); + + OFFSET(KFUNC_MD_PROG_prog, kfunc_md_tramp_prog, prog); + OFFSET(KFUNC_MD_PROG_cookie, kfunc_md_tramp_prog, cookie); } diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 367da3638167..62269a67bf3a 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -384,3 +385,233 @@ SYM_CODE_START(return_to_handler) X86_FEATURE_CALL_DEPTH SYM_CODE_END(return_to_handler) #endif + +/* + * the stack layout for bpf_global_caller: + * + * callee rip + * rbp + * ---------------------- rbp + * return value - for origin call + * arg6 + * ...... + * arg1 + * arg count + * origin ip - for bpf_get_func_ip() + * rbx - keep pointer to kfunc_md_tramp_prog + * bpf_tramp_run_ctx + * kfunc_md_ptr + * kfunc_md_flags + * r12 - keep the start time + * r13 - keep the return value address, for origin call + * + * Note: the return value can be in the position of arg6, arg5, etc, + * depending on the number of args. That's why we need the %r13 + */ + +#define FUNC_RETURN_OFFSET (-8) + +#define FUNC_ARGS_SIZE (6 * 8) +#define FUNC_ARGS_OFFSET (FUNC_RETURN_OFFSET - FUNC_ARGS_SIZE) +#define FUNC_ARGS_1 (FUNC_ARGS_OFFSET + 0 * 8) +#define FUNC_ARGS_2 (FUNC_ARGS_OFFSET + 1 * 8) +#define FUNC_ARGS_3 (FUNC_ARGS_OFFSET + 2 * 8) +#define FUNC_ARGS_4 (FUNC_ARGS_OFFSET + 3 * 8) +#define FUNC_ARGS_5 (FUNC_ARGS_OFFSET + 4 * 8) +#define FUNC_ARGS_6 (FUNC_ARGS_OFFSET + 5 * 8) + +/* the args count, rbp - 8 * 8 */ +#define FUNC_ARGS_COUNT_OFFSET (FUNC_ARGS_OFFSET - 1 * 8) +#define FUNC_ORIGIN_IP (FUNC_ARGS_OFFSET - 2 * 8) /* -9 * 8 */ +#define RBX_OFFSET (FUNC_ARGS_OFFSET - 3 * 8) + +/* bpf_tramp_run_ctx, rbp - RUN_CTX_OFFSET */ +#define RUN_CTX_OFFSET (RBX_OFFSET - RUN_CTX_SIZE) +#define KFUNC_MD_OFFSET (RUN_CTX_OFFSET - 1 * 8) +#define KFUNC_MD_FLAGS_OFFSET (KFUNC_MD_OFFSET - 8) +#define R12_OFFSET (KFUNC_MD_OFFSET - 16) +#define R13_OFFSET (KFUNC_MD_OFFSET - 24) +#define STACK_SIZE (-1 * R13_OFFSET) + +/* restore the regs before we return or before we do the origin call */ +.macro tramp_restore_regs + movq FUNC_ARGS_1(%rbp), %rdi + movq FUNC_ARGS_2(%rbp), %rsi + movq FUNC_ARGS_3(%rbp), %rdx + movq FUNC_ARGS_4(%rbp), %rcx + movq FUNC_ARGS_5(%rbp), %r8 + movq FUNC_ARGS_6(%rbp), %r9 + .endm + +/* save the args to stack, only regs is supported for now */ +.macro tramp_save_regs + movq %rdi, FUNC_ARGS_1(%rbp) + movq %rsi, FUNC_ARGS_2(%rbp) + movq %rdx, FUNC_ARGS_3(%rbp) + movq %rcx, FUNC_ARGS_4(%rbp) + movq %r8, FUNC_ARGS_5(%rbp) + movq %r9, FUNC_ARGS_6(%rbp) + .endm + +#define BPF_TRAMP_FENTRY 0 +#define BPF_TRAMP_FEXIT 1 +#define BPF_TRAMP_MODIFY_RETURN 2 + +.macro bpf_caller_prog_run type + /* check if the prog list is NULL */ +1: testq %rbx, %rbx + jz 3f + + /* load bpf prog to the 1st arg */ + movq KFUNC_MD_PROG_prog(%rbx), %rdi + + /* load the pointer of tramp_run_ctx to the 2nd arg */ + leaq RUN_CTX_OFFSET(%rbp), %rsi + /* save the bpf cookie to the tramp_run_ctx */ + movq KFUNC_MD_PROG_cookie(%rbx), %rax + movq %rax, RUN_CTX_cookie(%rsi) + call __bpf_prog_enter_recur + /* save the start time to r12 */ + movq %rax, %r12 + testq %rax, %rax + jz 2f + + movq KFUNC_MD_PROG_prog(%rbx), %rdi + /* load the JITed prog to rax */ + movq BPF_PROG_func(%rdi), %rax + /* load func args array to the 1st arg */ + leaq FUNC_ARGS_OFFSET(%rbp), %rdi + + /* load and call the JITed bpf func */ + CALL_NOSPEC rax +.if \type=3D=3DBPF_TRAMP_MODIFY_RETURN + /* modify_return case, save the return value */ + movq %rax, (%r13) +.endif + + /* load bpf prog to the 1st arg */ +2: movq KFUNC_MD_PROG_prog(%rbx), %rdi + /* load the rbx(start time) to the 2nd arg */ + movq %r12, %rsi + /* load the pointer of tramp_run_ctx to the 3rd arg */ + leaq RUN_CTX_OFFSET(%rbp), %rdx + call __bpf_prog_exit_recur + +.if \type=3D=3DBPF_TRAMP_MODIFY_RETURN + /* modify_return case, break the loop and skip the origin function call */ + cmpq $0, (%r13) + jne do_bpf_fexit +.endif + /* load the next tramp prog to rbx */ + movq 0(%rbx), %rbx + jmp 1b + +3: + .endm + +SYM_FUNC_START(bpf_global_caller) + ANNOTATE_NOENDBR + + /* prepare the stack space and store the args to the stack */ + pushq %rbp + movq %rsp, %rbp + subq $STACK_SIZE, %rsp + tramp_save_regs + + CALL_DEPTH_ACCOUNT + + /* save rbx and r12, which will be used later */ + movq %rbx, RBX_OFFSET(%rbp) + movq %r12, R12_OFFSET(%rbp) + + /* get the function address */ + movq 8(%rbp), %rdi + /* for x86_64, the function is 16-bytes aligned */ + andq $0xfffffffffffffff0, %rdi + /* save the origin function ip */ + movq %rdi, FUNC_ORIGIN_IP(%rbp) + + /* get the function meta data */ + call kfunc_md_get_noref + testq %rax, %rax + jz do_bpf_out + movq %rax, %rbx + movq %rbx, KFUNC_MD_OFFSET(%rbp) + + /* save the function args count */ + movzbq KFUNC_MD_nr_args(%rbx), %rax + movq %rax, FUNC_ARGS_COUNT_OFFSET(%rbp) + + /* call kfunc_md_enter only if we need origin call */ + movl KFUNC_MD_flags(%rbx), %edi + movl %edi, KFUNC_MD_FLAGS_OFFSET(%rbp) + andl $KFUNC_MD_FL_TRACING_ORIGIN, %edi + jz 1f + + /* save the address of the return value to r13 */ + movq %r13, R13_OFFSET(%rbp) + leaq FUNC_ARGS_OFFSET(%rbp, %rax, 8), %r13 + + movq %rbx, %rdi + call kfunc_md_enter + + /* try run fentry progs */ +1: movq (KFUNC_MD_progs + BPF_TRAMP_FENTRY * 8)(%rbx), %rbx + bpf_caller_prog_run BPF_TRAMP_FENTRY + + /* check if we need to do the origin call */ + movl KFUNC_MD_FLAGS_OFFSET(%rbp), %eax + andl $KFUNC_MD_FL_TRACING_ORIGIN, %eax + jz do_bpf_out + + /* try run modify_return progs */ + movq KFUNC_MD_OFFSET(%rbp), %r12 + movq (KFUNC_MD_progs + BPF_TRAMP_MODIFY_RETURN * 8)(%r12), %rbx + movq $0, (%r13) + bpf_caller_prog_run BPF_TRAMP_MODIFY_RETURN + + /* do the origin call */ + tramp_restore_regs + /* call the origin function from the stack, just like BPF_TRAMP_F_ORIG_ST= ACK */ + movq 8(%rbp), %rax + CALL_NOSPEC rax + movq %rax, (%r13) + +do_bpf_fexit: + /* for origin case, run fexit and return */ + movq KFUNC_MD_OFFSET(%rbp), %r12 + movq (KFUNC_MD_progs + BPF_TRAMP_FEXIT * 8)(%r12), %rbx + bpf_caller_prog_run BPF_TRAMP_FEXIT + movq KFUNC_MD_OFFSET(%rbp), %rdi + call kfunc_md_exit + + movq (%r13), %rax + movq RBX_OFFSET(%rbp), %rbx + movq R12_OFFSET(%rbp), %r12 + movq R13_OFFSET(%rbp), %r13 + leave + + /* rebalance the RSB. We can simply use: + * + * leaq 8(%rsp), %rsp + * RET + * + * instead here if we don't want do the rebalance. + */ + movq $bpf_global_caller_rsb, (%rsp) + RET +SYM_INNER_LABEL(bpf_global_caller_rsb, SYM_L_LOCAL) + ANNOTATE_NOENDBR + RET + +do_bpf_out: + /* for no origin call case, restore regs and return */ + tramp_restore_regs + + movq RBX_OFFSET(%rbp), %rbx + movq R12_OFFSET(%rbp), %r12 + leave + RET + +SYM_FUNC_END(bpf_global_caller) +STACK_FRAME_NON_STANDARD(bpf_global_caller) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5b25d278409b..8979e397ea06 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3554,6 +3554,10 @@ int bpf_bprintf_prepare(char *fmt, u32 fmt_size, con= st u64 *raw_args, u32 num_args, struct bpf_bprintf_data *data); void bpf_bprintf_cleanup(struct bpf_bprintf_data *data); =20 +u64 __bpf_prog_enter_recur(struct bpf_prog *prog, struct bpf_tramp_run_ctx= *run_ctx); +void __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start, + struct bpf_tramp_run_ctx *run_ctx); + #ifdef CONFIG_BPF_LSM void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype); void bpf_cgroup_atype_put(int cgroup_atype); diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index c4b1a98ff726..da4be23f03c3 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -894,7 +894,7 @@ static __always_inline u64 notrace bpf_prog_start_time(= void) * [2..MAX_U64] - execute bpf prog and record execution time. * This is start time. */ -static u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, struct bp= f_tramp_run_ctx *run_ctx) +u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, struct bpf_tramp= _run_ctx *run_ctx) __acquires(RCU) { rcu_read_lock(); @@ -934,8 +934,8 @@ static void notrace update_prog_stats(struct bpf_prog *= prog, } } =20 -static void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start, - struct bpf_tramp_run_ctx *run_ctx) +void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start, + struct bpf_tramp_run_ctx *run_ctx) __releases(RCU) { bpf_reset_run_ctx(run_ctx->saved_run_ctx); --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70BBF205AA8; Wed, 28 May 2025 03:49:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404192; cv=none; b=LmekRS/ekLurX7H+3IfH3ydo2OZu6VfnseBacsAoJgSIt/EsxLy2uQxu5ROB65pvsfVyz1d5xUTSRzrpQGiLeU+fUNKCQ40WphgYcIWw4J2fvccWlwhGBuud3DODAc/3j6mxihj7vQk+PP2F/4815AOGmQcPo+sbbPmTTt1+Ppo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404192; c=relaxed/simple; bh=qClQDl93AFZQUdkbU/XbVWIXBVJtxF9wpQ2WNufjqEc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=k9pnnFggGC9Y6q/Uksf3vXi/UmcoCUtWlD/BjYUvy3Sp+bnRhfTh00qrifoJZa+hfke83IvJLmOFb2ECrztM/haXm648y2/cYZbMSttk5fMASa9QumAJVNDP7DvYib3pSZyJn4WdcSjJIF1XEcgL+YsBBuJoD5Rt32Bq4F3VQ0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YelcOWo7; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YelcOWo7" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-2347b7d6aeeso25625285ad.2; Tue, 27 May 2025 20:49:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404190; x=1749008990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OOx80xOj6UGzmt1Y5Dc93j8W1eKLEy1T+1F8mwGiOv4=; b=YelcOWo7gtmh2YD1Ne/cPCCWha23KBR5lXJ73Fbp8e9OdBZnBVcVZNWv7PAms0XGux dn2b/4Wh1WbMQ0GJIJd1vQ5pDanvtd8sjGOctKeZ8hKuL2lQIwumUS9DkLHiIfSUf422 9HBoqjEo17E56RAzyKnokmgsuAtin4MPu+mVQRLNruzkg39zYwtH1Od3ff133kMVMtBJ sc7qwJSAG6eKPgpvzCEw5OyIgoZJadXWoahI2K9Ccj9/30vqh0ViaBo/F0sppdTpRjqI OUG2XvibTdLW0OtJtOV+ddrtSXIbHbMyqbQ/Z3GBmbhmXxAr1yU0ZjqR5qugdx1pOLhG 3T8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404190; x=1749008990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OOx80xOj6UGzmt1Y5Dc93j8W1eKLEy1T+1F8mwGiOv4=; b=UNcF4tjCEsN77tPP9/+tbckjyIMB3O4lwlPEunIPfMlL+WGvlZM8l2QMFjVIyO3Ajv egpWETc7DpY3w19Yy0l2ufQ3Dghj29WtfWCPdHVzXDDpESglFHdvaGNSm+6W2g1F30jg yL+CU33EuVzsyRLoxsNNM1/3MX5zNilPJl4HhhsZ+X5r7Opj77hxOChxm3uvd4gfQkIr sPOVmJvFvKkeApCMSmIqqbGy8V9hfFdrHKWCTvpc94HpUCqhBhlik1n9cmQ87hYuMfLW 9Q8Rmnc09aq0L5JwLxq4N9uMPv3en/m7njuKCoZ2xtdKzwHYVqZtaNjKFdg334oyoOW+ z5Kw== X-Forwarded-Encrypted: i=1; AJvYcCWT2KqqqWM5QKARCu5Sn+VsopCl9w/iYhlW3kPjqL7b2v6R1/FJ3krNmtJ9lzIwYi+GbryF91Uynk5wyUM=@vger.kernel.org X-Gm-Message-State: AOJu0YyUFvSYzDNu7HXZjV+p5ZmzYsJ+7sGCBQoI6GVQ8Dych2uoPWaP v2wq/2ov3ZhfUAN1ONeRjEdwlchulxoF4Iw0tNGalz9cS06nOSj7bKxo X-Gm-Gg: ASbGnctQNm03mcWuBEpI5LJ381M/JtnoL7jvk9lyGqRs9VNJ5Kuqq8i6VlbvZW0NEmr Or2/FFgaHwA/eGrD45SgVSZzUyusYR0ZDf31JN7UL89DiO1QXKT9Km6vTHFNL3K0yATBg2gK3EZ 4tiiNFH8SXWDYAUsTPvHI1dwQUbMobaPXSRQOeskQazgf2dK19OZUMqmk/9tYvIF5rGcGWMcDGc 9/JFolDZArUGFM6C7mDRZIWwdZ7AkABNMb53kj1/kMByF2o0sVgFw+tRb5pDmVyGKn2cWmfI96S 8hf1aKUQqIICfjltDt8t8PEiOpfG9K4MdBSmhCCC5CrCitnqqWguL+887CR8hQcMD4vw X-Google-Smtp-Source: AGHT+IE6ZFsizyOf792pJ/ak7j1cTxvY1Gh9OREX0qMHn+XSFclTKgeM3tAhcUYnIIYWMAUoZ6pujg== X-Received: by 2002:a17:903:1311:b0:234:bca7:2921 with SMTP id d9443c01a7336-234bca736d0mr30224255ad.33.1748404189612; Tue, 27 May 2025 20:49:49 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:49 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 06/25] ftrace: factor out ftrace_direct_update from register_ftrace_direct Date: Wed, 28 May 2025 11:46:53 +0800 Message-Id: <20250528034712.138701-7-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Factor out ftrace_direct_update() from register_ftrace_direct(), which is used to add new entries to the direct_functions. This function will be used in the later patch. Signed-off-by: Menglong Dong --- kernel/trace/ftrace.c | 108 +++++++++++++++++++++++------------------- 1 file changed, 60 insertions(+), 48 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 61130bb34d6c..a1028942e743 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5910,53 +5910,18 @@ static void register_ftrace_direct_cb(struct rcu_he= ad *rhp) free_ftrace_hash(fhp); } =20 -/** - * register_ftrace_direct - Call a custom trampoline directly - * for multiple functions registered in @ops - * @ops: The address of the struct ftrace_ops object - * @addr: The address of the trampoline to call at @ops functions - * - * This is used to connect a direct calls to @addr from the nop locations - * of the functions registered in @ops (with by ftrace_set_filter_ip - * function). - * - * The location that it calls (@addr) must be able to handle a direct call, - * and save the parameters of the function being traced, and restore them - * (or inject new ones if needed), before returning. - * - * Returns: - * 0 on success - * -EINVAL - The @ops object was already registered with this call or - * when there are no functions in @ops object. - * -EBUSY - Another direct function is already attached (there can be o= nly one) - * -ENODEV - @ip does not point to a ftrace nop location (or not support= ed) - * -ENOMEM - There was an allocation failure. - */ -int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) +static int ftrace_direct_update(struct ftrace_hash *hash, unsigned long ad= dr) { - struct ftrace_hash *hash, *new_hash =3D NULL, *free_hash =3D NULL; struct ftrace_func_entry *entry, *new; + struct ftrace_hash *new_hash =3D NULL; int err =3D -EBUSY, size, i; =20 - if (ops->func || ops->trampoline) - return -EINVAL; - if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) - return -EINVAL; - if (ops->flags & FTRACE_OPS_FL_ENABLED) - return -EINVAL; - - hash =3D ops->func_hash->filter_hash; - if (ftrace_hash_empty(hash)) - return -EINVAL; - - mutex_lock(&direct_mutex); - /* Make sure requested entries are not already registered.. */ size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (ftrace_find_rec_direct(entry->ip)) - goto out_unlock; + goto out; } } =20 @@ -5969,7 +5934,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) size =3D FTRACE_HASH_MAX_BITS; new_hash =3D alloc_ftrace_hash(size); if (!new_hash) - goto out_unlock; + goto out; =20 /* Now copy over the existing direct entries */ size =3D 1 << direct_functions->size_bits; @@ -5977,7 +5942,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) hlist_for_each_entry(entry, &direct_functions->buckets[i], hlist) { new =3D add_hash_entry(new_hash, entry->ip); if (!new) - goto out_unlock; + goto out; new->direct =3D entry->direct; } } @@ -5988,16 +5953,67 @@ int register_ftrace_direct(struct ftrace_ops *ops, = unsigned long addr) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { new =3D add_hash_entry(new_hash, entry->ip); if (!new) - goto out_unlock; + goto out; /* Update both the copy and the hash entry */ new->direct =3D addr; entry->direct =3D addr; } } =20 - free_hash =3D direct_functions; rcu_assign_pointer(direct_functions, new_hash); new_hash =3D NULL; + err =3D 0; +out: + if (new_hash) + free_ftrace_hash(new_hash); + + return err; +} + +/** + * register_ftrace_direct - Call a custom trampoline directly + * for multiple functions registered in @ops + * @ops: The address of the struct ftrace_ops object + * @addr: The address of the trampoline to call at @ops functions + * + * This is used to connect a direct calls to @addr from the nop locations + * of the functions registered in @ops (with by ftrace_set_filter_ip + * function). + * + * The location that it calls (@addr) must be able to handle a direct call, + * and save the parameters of the function being traced, and restore them + * (or inject new ones if needed), before returning. + * + * Returns: + * 0 on success + * -EINVAL - The @ops object was already registered with this call or + * when there are no functions in @ops object. + * -EBUSY - Another direct function is already attached (there can be o= nly one) + * -ENODEV - @ip does not point to a ftrace nop location (or not support= ed) + * -ENOMEM - There was an allocation failure. + */ +int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) +{ + struct ftrace_hash *hash, *free_hash =3D NULL; + int err =3D -EBUSY; + + if (ops->func || ops->trampoline) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) + return -EINVAL; + if (ops->flags & FTRACE_OPS_FL_ENABLED) + return -EINVAL; + + hash =3D ops->func_hash->filter_hash; + if (ftrace_hash_empty(hash)) + return -EINVAL; + + mutex_lock(&direct_mutex); + + free_hash =3D direct_functions; + err =3D ftrace_direct_update(hash, addr); + if (err) + goto out_unlock; =20 ops->func =3D call_direct_funcs; ops->flags =3D MULTI_FLAGS; @@ -6005,15 +6021,11 @@ int register_ftrace_direct(struct ftrace_ops *ops, = unsigned long addr) ops->direct_call =3D addr; =20 err =3D register_ftrace_function_nolock(ops); - - out_unlock: - mutex_unlock(&direct_mutex); - if (free_hash && free_hash !=3D EMPTY_HASH) call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb); =20 - if (new_hash) - free_ftrace_hash(new_hash); + out_unlock: + mutex_unlock(&direct_mutex); =20 return err; } --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D3DF20C001; Wed, 28 May 2025 03:49:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404194; cv=none; b=EvMdxrAwbU4nvd+hW3l8aGz26U22w17TPEWNjW+2pd4Ybg2yTj2qLAZeIJfrBIDnQc6RNYqqh+P4h28DdYBu+FDoJv7blfDGjLGaCdGPcAunumG/80Nr5FyefvvM0EXLZuWtszd8V5e0zo0LFgVbXmFIzS8Vj6lP6wEfqOhoEeQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404194; c=relaxed/simple; bh=hoagNbJPn239NTj8+NoBSGh3wAvcPZsWqDyYt8a0HGY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PJbcYWMj0NzzRL/PasI5k5dNGiULgIcQdl9MC1jSeTUhi+cCImZA3SUD3NJ8jcLJzTCzlV+BtMfy17/HZxmU0oEA02LdB9ZNoGy5+414BXr3fS2X63BAX8nEqwLTtrUyJELcWZT3Ah4rMh8s/WS7sX5uSBurw8iX7d6uaEvwzSs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SM9Wq0Jr; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SM9Wq0Jr" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-22c33677183so28641495ad.2; Tue, 27 May 2025 20:49:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404192; x=1749008992; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u6Gzs/Wo0vrVWTxQToES+CRwnjtt6Uid9yBONVnTXyo=; b=SM9Wq0Jr20Ar7aD5/jr/N67J7TApKQ9Vaos32wuGHkaN4lUs8hwPfIb9aS5XAGNGOt U+sY+BAZiCcNgzVNzO1diQh/UU2ClaNESlpvvpLYi08JmJiGZtrFcVpCpq6qy6i8nWBT YnrL17nbLW7MroMxrVkiPyRXmk/ZObhmLLnUq2gVbaKQzVIYUw9sl52j0LzETTqFkQk5 SpZJ3fCsSJAgIQgH316Y1KE8TQLXAFuRt9M7eVed9ZEJ7dZq7RRVxY6hoHqVq4uTCFSr mojd6nTF0COVt5U2dJD4LXm5cJD4EZxsqnWvZomY8HQ62iRkGc12WR1hevOCTuUvMPiI QmwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404192; x=1749008992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u6Gzs/Wo0vrVWTxQToES+CRwnjtt6Uid9yBONVnTXyo=; b=J+wvmuVfwj07NV7G7r0eZZLeUnP7pMBlJr2oeT6hI8eVNE4zt1DHXOk7TUr0AmmfdF 13Up+AugGQPuk/DfB7Gif56Qp7S4fF635wqzwumu5ntQL+cwQjoFVYTvHvxZZPEs4uLs 355gS5J4Nm0ErRL7vk9idU7FJNT5BRpaXtprzzvaen/FK7sLPDDq51ZrZichwzwMsrk0 tzVksljooa8oz5aEH/PEw/TO7DYG+MeQtQMVKj18w0PVoF1eejkBtXN3KZdpYGYgw+0+ jk0svyFNqLTOMw8QG+pp8PD9kpwyXMmkncB20zRmwt7iulNTHnnktN59jBz4uJAIg6Wh 3H1Q== X-Forwarded-Encrypted: i=1; AJvYcCVvvjJz13ZjAoxP/yvIs+MErWjPwgJVXBv7zYZQe/KbqFPFVgQuGlvNqmhHc8u6uygSAQICNouWgpiWnr0=@vger.kernel.org X-Gm-Message-State: AOJu0Yy2ezjYZr0Nvl21aKqhSbEWsbs8u+oODVFD8kxPsaNqm2WHgT6R NHgZjU8Q5ATaDDuLD17T1RThSqZJPKJp2sMwntsaNeMsERDf3XbF5k3R X-Gm-Gg: ASbGncuf9cQ0Tq8Vl/Dvr763d9ZDBQWMC89OkqrRTVoR5Ys8I6y3m1pzmuCnAZN1tiI 5hU5VCynsA8s2jgovVHy8xftGj6TccA3w0ks4m3Eyey7DIxWQKo/95BLWmIYNgSTnlx/C8mKC5x 74ZMNxZmM6Vb5/WjLidQ9WPwIfSffDOrmFy7JVETR90o7BDWnOHEStEfmUUHjQ5Z8LhQco5YAVx vQ0yoSlVOC2mFbeHgJVjWrxksNN6iMMgusOJyyUZlqfnKgMdn1k7bpimu/4xdIzH+Ee3DgOPpyB Kk96ovT99M9kzySWhWZpeVkgeJWpAYNaOX8pHD2eBRrGUYTuLp9SslEv9wZUTyZttA+p X-Google-Smtp-Source: AGHT+IGM0AOHtSF2ccrQGi6tNXesN8k9w9fUc1m2GNea3HWDHe6VMVannBjtamAWqHBOJhD+/kS6ow== X-Received: by 2002:a17:902:fc84:b0:234:c65f:6c0c with SMTP id d9443c01a7336-234c65f6d0fmr25229395ad.15.1748404191731; Tue, 27 May 2025 20:49:51 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:51 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 07/25] ftrace: add reset_ftrace_direct_ips Date: Wed, 28 May 2025 11:46:54 +0800 Message-Id: <20250528034712.138701-8-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, we can change the address of a direct ftrace_ops with modify_ftrace_direct(). However, we can't change the functions to filter for a direct ftrace_ops. Therefore, we introduce the function reset_ftrace_direct_ips() to do such things, and this function will reset the functions to filter for a direct ftrace_ops. This function do such thing in following steps: 1. filter out the new functions from ips that don't exist in the ops->func_hash->filter_hash and add them to the new hash. 2. add all the functions in the new ftrace_hash to direct_functions by ftrace_direct_update(). 3. reset the functions to filter of the ftrace_ops to the ips with ftrace_set_filter_ips(). 4. remove the functions that in the old ftrace_hash, but not in the new ftrace_hash from direct_functions. Signed-off-by: Menglong Dong --- include/linux/ftrace.h | 7 ++++ kernel/trace/ftrace.c | 75 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index fbabc3d848b3..40727d3f125d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -526,6 +526,8 @@ int modify_ftrace_direct_nolock(struct ftrace_ops *ops,= unsigned long addr); =20 void ftrace_stub_direct_tramp(void); =20 +int reset_ftrace_direct_ips(struct ftrace_ops *ops, unsigned long *ips, + unsigned int cnt); #else struct ftrace_ops; static inline unsigned long ftrace_find_rec_direct(unsigned long ip) @@ -549,6 +551,11 @@ static inline int modify_ftrace_direct_nolock(struct f= trace_ops *ops, unsigned l { return -ENODEV; } +static inline int reset_ftrace_direct_ips(struct ftrace_ops *ops, unsigned= long *ips, + unsigned int cnt) +{ + return -ENODEV; +} =20 /* * This must be implemented by the architecture. diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index a1028942e743..0befb4c93e89 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6181,6 +6181,81 @@ int modify_ftrace_direct(struct ftrace_ops *ops, uns= igned long addr) return err; } EXPORT_SYMBOL_GPL(modify_ftrace_direct); + +/* reset the ips for a direct ftrace (add or remove) */ +int reset_ftrace_direct_ips(struct ftrace_ops *ops, unsigned long *ips, + unsigned int cnt) +{ + struct ftrace_hash *hash, *free_hash; + struct ftrace_func_entry *entry, *del; + unsigned long ip; + int err, size; + + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + + mutex_lock(&direct_mutex); + hash =3D alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); + if (!hash) { + err =3D -ENOMEM; + goto out_unlock; + } + + /* find out the new functions from ips and add to hash */ + for (int i =3D 0; i < cnt; i++) { + ip =3D ftrace_location(ips[i]); + if (!ip) { + err =3D -ENOENT; + goto out_unlock; + } + if (__ftrace_lookup_ip(ops->func_hash->filter_hash, ip)) + continue; + err =3D __ftrace_match_addr(hash, ip, 0); + if (err) + goto out_unlock; + } + + free_hash =3D direct_functions; + /* add the new ips to direct hash. */ + err =3D ftrace_direct_update(hash, ops->direct_call); + if (err) + goto out_unlock; + + if (free_hash && free_hash !=3D EMPTY_HASH) + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb); + + free_ftrace_hash(hash); + hash =3D alloc_and_copy_ftrace_hash(FTRACE_HASH_DEFAULT_BITS, + ops->func_hash->filter_hash); + if (!hash) { + err =3D -ENOMEM; + goto out_unlock; + } + err =3D ftrace_set_filter_ips(ops, ips, cnt, 0, 1); + + /* remove the entries that don't exist in our filter_hash anymore + * from the direct_functions. + */ + size =3D 1 << hash->size_bits; + for (int i =3D 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + if (__ftrace_lookup_ip(ops->func_hash->filter_hash, entry->ip)) + continue; + del =3D __ftrace_lookup_ip(direct_functions, entry->ip); + if (del && del->direct =3D=3D ops->direct_call) { + remove_hash_entry(direct_functions, del); + kfree(del); + } + } + } +out_unlock: + mutex_unlock(&direct_mutex); + if (hash) + free_ftrace_hash(hash); + return err; +} #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 /** --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C824E21322B; Wed, 28 May 2025 03:49:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404197; cv=none; b=IX94QfyipkS/E81uLQmjtfQV+sPFhPF3oaRkPw/Ob9ldtg0+yk/F69nVywtPkvcOEAZnFGr4qRpU3APSqY2Lo0vs10f4gbSIYtCuhSUq88f1xR6Bl+eIyx6+/PUh5Fhm11RVDi3y31hXTtixCDgcjrVmfeax/uQsoclyz72gEww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404197; c=relaxed/simple; bh=Q2ZhADwFji9fpJ66IuQW1xzWb9NKosAUzd+v8YlATa8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O7eDyEA7kJfi2QM/5zjXxH1ZpdOAJ3zWCB16k40q5/1cDp4b6Up5k7wISl/vbnlHNnA+j58iT6Ed0ZEY9tK4+Cp1vTTVdiuywMmZMhYAI9ZPAVZrGNpRmbyp9jiCaPphe6EsUWk3ckL0zX1jmj085juvgJoSB62F/7oo8B4oanM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=d7VNdeoh; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d7VNdeoh" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-231c86bffc1so40748525ad.0; Tue, 27 May 2025 20:49:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404194; x=1749008994; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BDS2dmniJXm2+zWW98g5diGeXn4aawjb/QQTXBKdBMQ=; b=d7VNdeohNBoGGKBzS03Urp93Vz2kHVMKyM2ESwRrRFRDHJouvDv3bo1i08PVQW2qJz OOGF8bPkK0j9lWqKeHwIiX6w1WJQDJ5QL4d8SUzQ05lN0M78nnBzLEoWCuqjFRg7++MC 49ixp1a7W1q52ZtOi7ky1RHO2UEOCnnO7o/Rq+CJYMgOOPIzVGmtTD9ScQ+Vrb1/u+tG kocIeW7N7sb7AZz5M/cOdpKxrnhdUquP0yky9/bCIdyPPx80Ww2CPOfdkKP6X2Exvnbk qp6HRVE98MqNE0M86qbFgN0qfpeowZYECzSgprbAsqKcuHoK0eNiFZgYnfqQYIm3YCnG iASw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404194; x=1749008994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BDS2dmniJXm2+zWW98g5diGeXn4aawjb/QQTXBKdBMQ=; b=EZHYjEKErArK4lmdu7NxDSd5eCD2cfmXOveYICsMgU48LT0/RN6dEJeMuEcEl+q5I3 iwzSrRwvx+Wb4/aMlCiKQBoXdRcI5miMMf6jPrRXTr9z+IRzcy2Nxxu7gu/JX1fzO3/h q4a3K1FgfZthzmrCX4Acb2XzJUobmz/Kr7sMt5bt2tvgnxeiG/Q2/jUFEIUxS2X3UPQV 5BZf3d67wg+xMMAwLR8uFaoF+k0BBuaQsiWu2MJ45hy4NcsfRWjFovP7ZkysDeb20QJc WmNYj6D1ILFoh9uSccSwnAfm9h1YB+sMCDTzosbmiJvkYNtBxyVm/XFwRQ0tOubwdUTf 0+jQ== X-Forwarded-Encrypted: i=1; AJvYcCWOqwoWOUB2jCbGYkJGuYA/F0KcjP9Fc3FlcHyDnO+WQDks105T9iwxjUOQYj+ysoJirUzFIwCQmGyToQ0=@vger.kernel.org X-Gm-Message-State: AOJu0YxcxknlvA2yk8DBvVrmKyV2oH/V0zsXf5nCg4qxTWIgIp3rapXQ NTzOW5nWl0HSHa6whRSXfg6GImEsQvn+TWcaW2MwaigHp6NJVwUaeRjF X-Gm-Gg: ASbGncvLtNme3Okw0XYM2M5NTnBlF48zGoTmWTLOO9MAVEAe+1hYwP/TrpmjMycReZ/ KZWRunNs13d3F7ymyXhFPlfKDftkNxX0FtSTfFce3UkzNq78tRG6nRG/eT+s1k1/+7Wp8Yjyqb2 oFekWDOIy0U1PsfbFTfp3YiNOa+4ePJ+quSov9M80sOBCZn0IROJREeXD/zlA0ULVYpxgVdTlWQ +aC56t1jWGb216Lt/R9fN9Mc6+O2GxhcmLA41UOVMHG1wrfJg14i+h7PAlQ4q7yA3UYJmVDIZK3 qDxzWx0ghy80xTocMd/4UfHh01QShfIkZJ4wz8y1gpRE+0e0NgsoP1cIzLvftTT6sUpZSYqQIjd w/q0= X-Google-Smtp-Source: AGHT+IHMnQL99sUjCpafDVBKNEwv0T97vx90aeArCF4R3+H/Zy3yyGJtPef038xTQz0YoeD5R0PH/Q== X-Received: by 2002:a17:903:44c:b0:22e:3b65:9286 with SMTP id d9443c01a7336-23414fc7372mr173251535ad.49.1748404193924; Tue, 27 May 2025 20:49:53 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:53 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 08/25] bpf: introduce bpf_gtramp_link Date: Wed, 28 May 2025 11:46:55 +0800 Message-Id: <20250528034712.138701-9-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce the struct bpf_gtramp_link, which is used to attach a bpf prog to multi functions. Meanwhile, introduce corresponding function bpf_gtrampoline_{link,unlink}_prog. The lock global_tr_lock is held during global trampoline link and unlink. Why we define the global_tr_lock as rw_semaphore? Well, it should be mutex here, but we will use the rw_semaphore in the later patch for the trampoline override case :/ When unlink the global trampoline link, we mark all the function in the bpf_gtramp_link with KFUNC_MD_FL_BPF_REMOVING and update the global trampoline with bpf_gtrampoline_update(). If this is the last bpf prog in the kfunc_md, the function will be remove from the filter_hash of the ftrace_ops of bpf_global_trampoline. Then, we remove the bpf prog from the kfunc_md, and free the kfunc_md if necessary. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 31 +++++++ kernel/bpf/trampoline.c | 183 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 214 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 8979e397ea06..7527399bab5b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -58,12 +58,15 @@ struct bpf_token; struct user_namespace; struct super_block; struct inode; +struct bpf_tramp_link; +struct bpf_gtramp_link; =20 extern struct idr btf_idr; extern spinlock_t btf_idr_lock; extern struct kobject *btf_kobj; extern struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma; extern bool bpf_global_ma_set; +extern struct bpf_global_trampoline global_tr; =20 typedef u64 (*bpf_callback_t)(u64, u64, u64, u64, u64); typedef int (*bpf_iter_init_seq_priv_t)(void *private_data, @@ -1279,6 +1282,12 @@ struct bpf_trampoline { struct bpf_tramp_image *cur_image; }; =20 +struct bpf_global_trampoline { + struct list_head list; + struct ftrace_ops *fops; + void *image; +}; + struct bpf_attach_target_info { struct btf_func_model fmodel; long tgt_addr; @@ -1382,6 +1391,12 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, void bpf_trampoline_put(struct bpf_trampoline *tr); int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int nu= m_funcs); =20 +#ifdef CONFIG_ARCH_HAS_BPF_GLOBAL_CALLER +void bpf_global_caller(void); +#endif +int bpf_gtrampoline_link_prog(struct bpf_gtramp_link *link); +int bpf_gtrampoline_unlink_prog(struct bpf_gtramp_link *link); + /* * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn * indirection with a direct call to the bpf program. If the architecture = does @@ -1746,6 +1761,22 @@ struct bpf_shim_tramp_link { struct bpf_trampoline *trampoline; }; =20 +struct bpf_gtramp_link_entry { + struct bpf_prog *tgt_prog; + struct bpf_trampoline *trampoline; + void *addr; + struct btf *attach_btf; + u64 cookie; + u32 btf_id; + u32 nr_args; +}; + +struct bpf_gtramp_link { + struct bpf_link link; + struct bpf_gtramp_link_entry *entries; + u32 entry_cnt; +}; + struct bpf_tracing_link { struct bpf_tramp_link link; enum bpf_attach_type attach_type; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index da4be23f03c3..be06dd76505a 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -13,6 +13,7 @@ #include #include #include +#include =20 /* dummy _ops. The verifier will operate on target program's ops. */ const struct bpf_verifier_ops bpf_extension_verifier_ops =3D { @@ -29,6 +30,10 @@ static struct hlist_head trampoline_table[TRAMPOLINE_TAB= LE_SIZE]; /* serializes access to trampoline_table */ static DEFINE_MUTEX(trampoline_mutex); =20 +struct bpf_global_trampoline global_tr; +static DECLARE_RWSEM(global_tr_lock); +static const struct bpf_link_ops bpf_shim_tramp_link_lops; + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_dire= ct_mutex); =20 @@ -645,6 +650,172 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_link = *link, return err; } =20 +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && defined(CONFIG_ARC= H_HAS_BPF_GLOBAL_CALLER) +static int bpf_gtrampoline_update(struct bpf_global_trampoline *tr) +{ + struct ftrace_ops *fops; + int ips_count, err =3D 0; + void **ips =3D NULL; + + ips_count =3D kfunc_md_bpf_ips(&ips); + if (ips_count < 0) { + err =3D ips_count; + goto out; + } + + fops =3D tr->fops; + if (ips_count =3D=3D 0) { + if (!(fops->flags & FTRACE_OPS_FL_ENABLED)) + goto out; + err =3D unregister_ftrace_direct(fops, (unsigned long)tr->image, + true); + goto out; + } + + if (fops->flags & FTRACE_OPS_FL_ENABLED) { + err =3D reset_ftrace_direct_ips(fops, (unsigned long *)ips, + ips_count); + goto out; + } + + err =3D ftrace_set_filter_ips(tr->fops, (unsigned long *)ips, + ips_count, 0, 1); + if (err) + goto out; + + err =3D register_ftrace_direct(fops, (unsigned long)tr->image); +out: + kfree(ips); + + return err; +} +#else +static int bpf_gtrampoline_update(struct bpf_global_trampoline *tr) +{ + return -ENODEV; +} +#endif + +static int __bpf_gtrampoline_unlink_prog(struct bpf_gtramp_link *link, + u32 cnt) +{ + enum bpf_tramp_prog_type kind; + struct kfunc_md *md; + int err =3D 0; + + kind =3D bpf_attach_type_to_tramp(link->link.prog); + kfunc_md_lock(); + for (int i =3D 0; i < cnt; i++) { + md =3D kfunc_md_get_noref((long)link->entries[i].addr); + if (WARN_ON_ONCE(!md)) { + err =3D -EINVAL; + break; + } + + if (md->tramp) + bpf_gtrampoline_remove(md->tramp, link->link.prog, false); + + md->flags &=3D ~KFUNC_MD_FL_BPF_REMOVING; + err =3D kfunc_md_bpf_unlink(md, link->link.prog, kind); + kfunc_md_put_entry(md); + if (err) + break; + } + kfunc_md_unlock(); + + return err; +} + +int bpf_gtrampoline_unlink_prog(struct bpf_gtramp_link *link) +{ + struct kfunc_md *md; + int err; + + + /* hold the global trampoline lock, to make the target functions + * consist during we unlink the prog. + */ + down_read(&global_tr_lock); + /* update the kfunc_md status, meanwhile update corresponding fops */ + kfunc_md_lock(); + for (int i =3D 0; i < link->entry_cnt; i++) { + md =3D kfunc_md_get_noref((long)link->entries[i].addr); + if (WARN_ON_ONCE(!md)) + continue; + + md->flags |=3D KFUNC_MD_FL_BPF_REMOVING; + } + kfunc_md_unlock(); + + bpf_gtrampoline_update(&global_tr); + + /* update the ftrace filter first, then the corresponding kfunc_md */ + err =3D __bpf_gtrampoline_unlink_prog(link, link->entry_cnt); + up_read(&global_tr_lock); + + return err; +} + +int bpf_gtrampoline_link_prog(struct bpf_gtramp_link *link) +{ + struct bpf_gtramp_link_entry *entry; + enum bpf_tramp_prog_type kind; + struct bpf_prog *prog; + struct kfunc_md *md; + bool update =3D false; + int err =3D 0, i; + + prog =3D link->link.prog; + kind =3D bpf_attach_type_to_tramp(prog); + + /* hold the global trampoline lock, to make the target functions + * consist during we link the prog. + */ + down_read(&global_tr_lock); + + /* update the bpf prog to all the corresponding function metadata */ + for (i =3D 0; i < link->entry_cnt; i++) { + entry =3D &link->entries[i]; + /* it seems that we hold this lock too long, we can use rcu + * lock instead. + */ + kfunc_md_lock(); + md =3D kfunc_md_create((long)entry->addr, entry->nr_args); + if (md) { + /* the function is not in the filter hash of gtr, + * we need update the global trampoline. + */ + if (!md->bpf_prog_cnt) + update =3D true; + err =3D kfunc_md_bpf_link(md, prog, kind, entry->cookie); + } else { + err =3D -ENOMEM; + } + + if (err) { + kfunc_md_put_entry(md); + kfunc_md_unlock(); + goto on_fallback; + } + kfunc_md_unlock(); + } + + if (update) { + err =3D bpf_gtrampoline_update(&global_tr); + if (err) + goto on_fallback; + } + up_read(&global_tr_lock); + + return 0; + +on_fallback: + __bpf_gtrampoline_unlink_prog(link, i); + up_read(&global_tr_lock); + + return err; +} + #if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM) static void bpf_shim_tramp_link_release(struct bpf_link *link) { @@ -1131,6 +1302,18 @@ static int __init init_trampolines(void) { int i; =20 +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + global_tr.fops =3D kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL); + if (!global_tr.fops) + return -ENOMEM; + + global_tr.fops->private =3D &global_tr; + global_tr.fops->ops_func =3D bpf_tramp_ftrace_ops_func; +#endif +#ifdef CONFIG_ARCH_HAS_BPF_GLOBAL_CALLER + global_tr.image =3D bpf_global_caller; +#endif + for (i =3D 0; i < TRAMPOLINE_TABLE_SIZE; i++) INIT_HLIST_HEAD(&trampoline_table[i]); return 0; --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91B292135BB; Wed, 28 May 2025 03:49:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404198; cv=none; b=XRneGGEIuVis5ekKiGBNPfMatDyL1Eb4unFQEdGGIyitqzy8kznfRXQmD9TjwCBKjKvgUvd9NLR3bCJ/SRypO3LMtN7RxrEvUlpFi/IiP+vVhGRum9oWQDlHo2rzmHStq0yUVLBLS/g+WCs+3ZlDKzK1eJsOYOcsh2FpibLLRu4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404198; c=relaxed/simple; bh=2qsWsLKzCQWmPBn92SOSDDTtGP108ttyYy+xr/NrD5s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e2SCPID3GGEmyKUXOKeHxRPLDm1Y5oaAYR/7zXN37cG3kyqavvLwn0bnfivM3uIph5sIIQSO6B62SIc0LRtxTUfypEos0eEgdmIytYdq+PMHZVFnmqmCbOsFkkH1JVzxZohw/NL7UIUyVVhy87Hcj6GCfg3H5uBO4+8Eiby/EQU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fj7cRkqR; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fj7cRkqR" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-2347d505445so16471895ad.2; Tue, 27 May 2025 20:49:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404196; x=1749008996; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N2KhlOaE735i3tZSmxmCD+tiCv4kDZN8CVb3bdoSz/Q=; b=fj7cRkqRsqazW840Mnfx6iX5pFpZI4sFAoUl+KzjvQ1WEPe9nXrRtoo6KQD0aMcl6x fbVuLZOj6YYQMEni7PDB6fTdZvKpIbZ4bnMkwdBgX2pZfaCHCKq3hxRW9OXY3Mii9N3G xPY4DMwTlJuZDyY4Bg9KyZXp3Hrgi941vtXrJfOSqrIbqwG6j3DzbwoFeP5GUjvD10Gg /N8Vo536m7hF7PdGHOKlensrL+5UxdGdIYA+in3+eecQplhd1H6tSPPhH4IqoN+6Sj/K N4EJ+BRMfbG1FB2vCSxMYw7xEfdnXc34+li9s2Uve4S8aZxoV/fRjOPAgFmhsFUvJiq3 +wPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404196; x=1749008996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N2KhlOaE735i3tZSmxmCD+tiCv4kDZN8CVb3bdoSz/Q=; b=M/FrEC5mVNnll/Uhj4oEjC/+on6OwktjwFuM8Dgc7xO3oU3H5YvI2pDs8q6FVeOJCh 7cE6DU9hm7+aD0qt/kmxnp7O2APhoUuYcWvKQKf/fY5ltu9Bv3ZCIPPbL850ufjA7S4O 1EDJMr3gF6UcvGLKfnVtyste1nnyw8644/7L04N0bHGtsgnGZ36tPlIv7WfflQIlaDcS G3xJAVaKDmIsYSsFDSdNqJd3C5QFAvRE3Yo++Ve2OPUq801WqdhNGeTzR5Ouh5VAVdzq 4K2o+yh8x4byCHgUEr9xOwsoEtz3oq15xZUY7tYg3xZnqju0Ix9FZcnMhf5Sd8HoH7V/ dDLw== X-Forwarded-Encrypted: i=1; AJvYcCUrcTWByDGsjSbCDHLXvS8iul8VLydCoPc6rWqd+imdmVj9wCeBqKIX4IW1RrQzhec1+l7kETUoEdGegRU=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2u55m0vIWLcmqNx5YinyfFi/SI7UlMFx7kQuAqmcLuPg8tRY1 Iuk8XYLscbg9pkq9xLmEY3BWhPRX16jCRUt607uRpd4fKbPfMyVW13Ng X-Gm-Gg: ASbGncv5UtwY0svP6967PdWKll9uicTdh021rcvYcn9V6TRvGNhu0tkS5IZ3yEFUb+Q R1ht8y8FDGn8GFqCQCXwoi3X0oz3JW1RRW2qtln+4f3PzA5Bdl5+8Jp1uHcoE+mzsOua8W8OpxF MCPR/vf9NpSHQoU0O3eKRBnv8FCPEcQgwxoNj4XKyjQdJG2a24YUMZySLYAzNUJCKPDd8TT/3bv j951pTgl2bz/hxz5wB4yBh90rk+M+p97DR+usRmO9KzmUzk8CZDhUQ6X60NoesoJY0NggWbq892 07ojhzPRrV3D2QnpOYNT5IdFqVY3+QdP2Oh1tQC0Y7yOl9sX1i+39zuZ0RixfCcO+kFf X-Google-Smtp-Source: AGHT+IF2BYBHNpymSQL9fQ8dcUhjPfH7A+R7PWRcP7iUBtt9L1t/RZGT/4upzoc1pvqrNcdXfIaUYQ== X-Received: by 2002:a17:902:dac4:b0:231:c90e:292d with SMTP id d9443c01a7336-23414fe34admr222979455ad.44.1748404196070; Tue, 27 May 2025 20:49:56 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:55 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 09/25] bpf: tracing: add support to record and check the accessed args Date: Wed, 28 May 2025 11:46:56 +0800 Message-Id: <20250528034712.138701-10-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In this commit, we add the 'accessed_args' field to struct bpf_prog_aux, which is used to record the accessed index of the function args in btf_ctx_access(). Meanwhile, we add the function btf_check_func_part_match() to compare the accessed function args of two function prototype. This function will be used in the following commit. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 4 ++ kernel/bpf/btf.c | 108 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 110 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7527399bab5b..abf504e95ff2 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1601,6 +1601,7 @@ struct bpf_prog_aux { const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ const char *attach_func_name; + u64 accessed_args; struct bpf_prog **func; void *jit_data; /* JIT specific data. arch dependent */ struct bpf_jit_poke_descriptor *poke_tab; @@ -2779,6 +2780,9 @@ struct bpf_reg_state; int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog); int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_pr= og *prog, struct btf *btf, const struct btf_type *t); +int btf_check_func_part_match(struct btf *btf1, const struct btf_type *t1, + struct btf *btf2, const struct btf_type *t2, + u64 func_args); const char *btf_find_decl_tag_value(const struct btf *btf, const struct bt= f_type *pt, int comp_idx, const char *tag_key); int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *p= t, diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 0f7828380895..64538625ee91 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6392,19 +6392,24 @@ static bool is_void_or_int_ptr(struct btf *btf, con= st struct btf_type *t) } =20 static u32 get_ctx_arg_idx(struct btf *btf, const struct btf_type *func_pr= oto, - int off) + int off, int *aligned_idx) { const struct btf_param *args; const struct btf_type *t; u32 offset =3D 0, nr_args; int i; =20 + if (aligned_idx) + *aligned_idx =3D -ENOENT; + if (!func_proto) return off / 8; =20 nr_args =3D btf_type_vlen(func_proto); args =3D (const struct btf_param *)(func_proto + 1); for (i =3D 0; i < nr_args; i++) { + if (aligned_idx && offset =3D=3D off) + *aligned_idx =3D i; t =3D btf_type_skip_modifiers(btf, args[i].type, NULL); offset +=3D btf_type_is_ptr(t) ? 8 : roundup(t->size, 8); if (off < offset) @@ -6671,7 +6676,7 @@ bool btf_ctx_access(int off, int size, enum bpf_acces= s_type type, tname, off); return false; } - arg =3D get_ctx_arg_idx(btf, t, off); + arg =3D get_ctx_arg_idx(btf, t, off, NULL); args =3D (const struct btf_param *)(t + 1); /* if (t =3D=3D NULL) Fall back to default BPF prog with * MAX_BPF_FUNC_REG_ARGS u64 arguments. @@ -6681,6 +6686,9 @@ bool btf_ctx_access(int off, int size, enum bpf_acces= s_type type, /* skip first 'void *__data' argument in btf_trace_##name typedef */ args++; nr_args--; + prog->aux->accessed_args |=3D (1 << (arg + 1)); + } else { + prog->aux->accessed_args |=3D (1 << arg); } =20 if (arg > nr_args) { @@ -7540,6 +7548,102 @@ int btf_check_type_match(struct bpf_verifier_log *l= og, const struct bpf_prog *pr return btf_check_func_type_match(log, btf1, t1, btf2, t2); } =20 +static u32 get_ctx_arg_total_size(struct btf *btf, const struct btf_type *= t) +{ + const struct btf_param *args; + u32 size =3D 0, nr_args; + int i; + + nr_args =3D btf_type_vlen(t); + args =3D (const struct btf_param *)(t + 1); + for (i =3D 0; i < nr_args; i++) { + t =3D btf_type_skip_modifiers(btf, args[i].type, NULL); + size +=3D btf_type_is_ptr(t) ? 8 : roundup(t->size, 8); + } + + return size; +} + +/* This function is similar to btf_check_func_type_match(), except that it + * only compare some function args of the function prototype t1 and t2. + */ +int btf_check_func_part_match(struct btf *btf1, const struct btf_type *fun= c1, + struct btf *btf2, const struct btf_type *func2, + u64 func_args) +{ + const struct btf_param *args1, *args2; + u32 nargs1, i, offset =3D 0; + const char *s1, *s2; + + if (!btf_type_is_func_proto(func1) || !btf_type_is_func_proto(func2)) + return -EINVAL; + + args1 =3D (const struct btf_param *)(func1 + 1); + args2 =3D (const struct btf_param *)(func2 + 1); + nargs1 =3D btf_type_vlen(func1); + + for (i =3D 0; i <=3D nargs1; i++) { + const struct btf_type *t1, *t2; + + if (!(func_args & (1 << i))) + goto next; + + if (i < nargs1) { + int t2_index; + + /* get the index of the arg corresponding to args1[i] + * by the offset. + */ + get_ctx_arg_idx(btf2, func2, offset, &t2_index); + if (t2_index < 0) + return -EINVAL; + + t1 =3D btf_type_skip_modifiers(btf1, args1[i].type, NULL); + t2 =3D btf_type_skip_modifiers(btf2, args2[t2_index].type, + NULL); + } else { + /* i =3D=3D nargs1, this is the index of return value of t1 */ + if (get_ctx_arg_total_size(btf1, func1) !=3D + get_ctx_arg_total_size(btf2, func2)) + return -EINVAL; + + /* check the return type of t1 and t2 */ + t1 =3D btf_type_skip_modifiers(btf1, func1->type, NULL); + t2 =3D btf_type_skip_modifiers(btf2, func2->type, NULL); + } + + if (t1->info !=3D t2->info || + (btf_type_has_size(t1) && t1->size !=3D t2->size)) + return -EINVAL; + if (btf_type_is_int(t1) || btf_is_any_enum(t1)) + goto next; + + if (btf_type_is_struct(t1)) + goto on_struct; + + if (!btf_type_is_ptr(t1)) + return -EINVAL; + + t1 =3D btf_type_skip_modifiers(btf1, t1->type, NULL); + t2 =3D btf_type_skip_modifiers(btf2, t2->type, NULL); + if (!btf_type_is_struct(t1) || !btf_type_is_struct(t2)) + return -EINVAL; + +on_struct: + s1 =3D btf_name_by_offset(btf1, t1->name_off); + s2 =3D btf_name_by_offset(btf2, t2->name_off); + if (strcmp(s1, s2)) + return -EINVAL; +next: + if (i < nargs1) { + t1 =3D btf_type_skip_modifiers(btf1, args1[i].type, NULL); + offset +=3D btf_type_is_ptr(t1) ? 8 : roundup(t1->size, 8); + } + } + + return 0; +} + static bool btf_is_dynptr_ptr(const struct btf *btf, const struct btf_type= *t) { const char *name; --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE6321EF0A6; Wed, 28 May 2025 03:49:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404200; cv=none; b=GJQ4/yuVT/bu9EQOxao/DFo3dNKBFdHWCni1bJ9Tb0yLnP3LIK2nG4Km9JazQddfxW2AlojUIcbxneH14yIsN++C/KMcq2Lxod0FVcRW7J3CH6xW3Wzx6ktKZfA7KMZAYV/ZPOxZV5ExNuRXkx+SgyT8QnBMaAZiFMpUKpUGlHY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404200; c=relaxed/simple; bh=6PlYWOy3amN0lotfPfL/VKwZ5CAcJfl9znOLLpFy40k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=m0oHVzE6k8NuC/dDDKjrT99DdknkTXz6ofP7ydhQ/aD/2u79X4IARXMYoWYj6a7PLidv9LYYsGlNsc1ob0Yda0OoQZODQhGiw7oOHxQEAewzuVGhSTOMUBhysqxb77s700NCQkQ7k0Sb5RvXtuALLjpxdm2tK0xQk4BK+xxSetw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cPjcONJ4; arc=none smtp.client-ip=209.85.210.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cPjcONJ4" Received: by mail-pf1-f196.google.com with SMTP id d2e1a72fcca58-742c9563fafso2876534b3a.0; Tue, 27 May 2025 20:49:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404198; x=1749008998; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gBEP94FNiEfzTaba1ZYVkx6Mjhh8FN4Fehm643kC5no=; b=cPjcONJ4uRUbhQtl9IK0c+kCa25QEPCAl3ErQCbXoRsZMxvu1t9vIwZyHPUyjB88w/ CmP0BAjABk29sF3YuUc8hkclRElNO748Z7+KiCRbE8BNNS5SCYyMjnZB/4a11llaegK+ zVENj7IzPdha/PPNzTWjkNSircJZW1va4j5VFx6FMTT8yzeZM87fpTkHXiLJmDSnDt0R dn85k1KoNl/YHy1y3qlcIwUp+q/U4pXDJKJg+TzcXRLMQiBAdfRap0nXTm5xKjr4UcXw E0+XXQK9ohnm93n3nqzEbqfj2q9RKs8XnRODikFPzrnxY5EIj711lf2o7A5tFtATGT0D C/5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404198; x=1749008998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gBEP94FNiEfzTaba1ZYVkx6Mjhh8FN4Fehm643kC5no=; b=ctAR1HdjhqWMOhoWnRrCTqeawSKl5QwD78Bq3Tzs/aBZhmSwZDIiFvfPXwYf1d94Pe S0IIPFzQ48nA/YbVdNQagltWj3awUeiXiblrTpueknB/5Pdj8vGN9bWSayMxei+2spWF kxdmeiReZVF+9BsZMNm83q/yLy6Objznoa0rcvLETQC1Lx6WpEfMHOM2S/2KlinXJwiv +RsVDgfLdnrxzAzl8yoGN+hYejqTPSJrBnEds4tUMCoAbILgSAbw5hnwfyaAV9cuH4ts G+a04dmRkT9TMZKMb1g154czL8PPRnMmnvIkjcO/GwLkJ2S6w92KRKfwY/aM/NbZVud3 67pQ== X-Forwarded-Encrypted: i=1; AJvYcCWJdPwRvcqA3qj9Def4vfQxHMufxvWn3Mwwx5GvX/9R8u0RJkd2xzz3DiSSaHy5NoGq9oo+C83hhpQRey4=@vger.kernel.org X-Gm-Message-State: AOJu0YwlusfdYpGSZdaEKi4qNYRX+fQTyB1UVKxbnGYWlECjNl+sSydm Yd5kSW5h32MKJU94De4I01DQxOvg+hYvtADTfVVgkNA388kG1cY9dWMC X-Gm-Gg: ASbGncvmIhdOrXuXgpQJYE8dHSZS0mJRSZEQQN1ZK5ivE0ezMl6xOTvHPod2dDeO1f2 Nupxad0vzh92lNhkmZ0U3s14fT2hsZQJmISLHga0u8Ez3rXKIY8FUUgiMmUYbOz9Tj+rRj4uMUc NLqFnCSQhksNWsMzoM2fc7ZjS2/mcQm2Z7iSh6IAEyZtr26ad0f7bIIaNw5ty9OTu17qIThK3AL DkhFUxJQpDm6A/hxjOwk15d0UvChXDbp8DVpjnVPd1UVprzTdTDV56rnvmhMczU9fd8w9YQZV44 AF4k6jkQIdYcCk0+/Vy1rW++cDtophI8i8mJzADEwTG9GIpdllnuNE5vnjFczR2VFzJY X-Google-Smtp-Source: AGHT+IEeiWx5NKO1Aq4jkogMNl0g4N1L39Hs4Nq2xJgLb9lD2vlJsx2wdh2oOQ8VcI8whFVdzgpnYA== X-Received: by 2002:a17:903:2f8a:b0:234:a139:1206 with SMTP id d9443c01a7336-234d2c2e18amr10836605ad.40.1748404198179; Tue, 27 May 2025 20:49:58 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:57 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 10/25] bpf: refactor the modules_array to ptr_array Date: Wed, 28 May 2025 11:46:57 +0800 Message-Id: <20250528034712.138701-11-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor the struct modules_array to more general struct ptr_array, which is used to store the pointers. Meanwhile, introduce the bpf_try_add_ptr(), which checks the existing of the ptr before adding it to the array. Seems it should be moved to another files in "lib", and I'm not sure where to add it now, and let's move it to kernel/bpf/syscall.c for now. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 10 +++++++++ kernel/bpf/syscall.c | 36 ++++++++++++++++++++++++++++++ kernel/trace/bpf_trace.c | 48 ++++++---------------------------------- 3 files changed, 53 insertions(+), 41 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index abf504e95ff2..c35da9d91125 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -315,6 +315,16 @@ struct bpf_map { s64 __percpu *elem_count; }; =20 +struct ptr_array { + void **ptrs; + int cnt; + int cap; +}; + +int bpf_add_ptr(struct ptr_array *arr, void *ptr); +bool bpf_has_ptr(struct ptr_array *arr, struct module *mod); +int bpf_try_add_ptr(struct ptr_array *arr, void *ptr); + static inline const char *btf_field_type_name(enum btf_field_type type) { switch (type) { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4b5f29168618..e22a23aa03d1 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -619,6 +619,42 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int= nid, return ret; } =20 +int bpf_add_ptr(struct ptr_array *arr, void *ptr) +{ + void **ptrs; + + if (arr->cnt =3D=3D arr->cap) { + arr->cap =3D max(16, arr->cap * 3 / 2); + ptrs =3D krealloc_array(arr->ptrs, arr->cap, sizeof(*ptrs), GFP_KERNEL); + if (!ptrs) + return -ENOMEM; + arr->ptrs =3D ptrs; + } + + arr->ptrs[arr->cnt] =3D ptr; + arr->cnt++; + return 0; +} + +bool bpf_has_ptr(struct ptr_array *arr, struct module *mod) +{ + int i; + + for (i =3D arr->cnt - 1; i >=3D 0; i--) { + if (arr->ptrs[i] =3D=3D mod) + return true; + } + return false; +} + +int bpf_try_add_ptr(struct ptr_array *arr, void *ptr) +{ + if (bpf_has_ptr(arr, ptr)) + return -EEXIST; + if (bpf_add_ptr(arr, ptr)) + return -ENOMEM; + return 0; +} =20 static int btf_field_cmp(const void *a, const void *b) { diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 132c8be6f635..8f134f291b81 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2779,43 +2779,9 @@ static void symbols_swap_r(void *a, void *b, int siz= e, const void *priv) } } =20 -struct modules_array { - struct module **mods; - int mods_cnt; - int mods_cap; -}; - -static int add_module(struct modules_array *arr, struct module *mod) -{ - struct module **mods; - - if (arr->mods_cnt =3D=3D arr->mods_cap) { - arr->mods_cap =3D max(16, arr->mods_cap * 3 / 2); - mods =3D krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KER= NEL); - if (!mods) - return -ENOMEM; - arr->mods =3D mods; - } - - arr->mods[arr->mods_cnt] =3D mod; - arr->mods_cnt++; - return 0; -} - -static bool has_module(struct modules_array *arr, struct module *mod) -{ - int i; - - for (i =3D arr->mods_cnt - 1; i >=3D 0; i--) { - if (arr->mods[i] =3D=3D mod) - return true; - } - return false; -} - static int get_modules_for_addrs(struct module ***mods, unsigned long *add= rs, u32 addrs_cnt) { - struct modules_array arr =3D {}; + struct ptr_array arr =3D {}; u32 i, err =3D 0; =20 for (i =3D 0; i < addrs_cnt; i++) { @@ -2825,7 +2791,7 @@ static int get_modules_for_addrs(struct module ***mod= s, unsigned long *addrs, u3 scoped_guard(rcu) { mod =3D __module_address(addrs[i]); /* Either no module or it's already stored */ - if (!mod || has_module(&arr, mod)) { + if (!mod || bpf_has_ptr(&arr, mod)) { skip_add =3D true; break; /* scoped_guard */ } @@ -2836,7 +2802,7 @@ static int get_modules_for_addrs(struct module ***mod= s, unsigned long *addrs, u3 continue; if (err) break; - err =3D add_module(&arr, mod); + err =3D bpf_add_ptr(&arr, mod); if (err) { module_put(mod); break; @@ -2845,14 +2811,14 @@ static int get_modules_for_addrs(struct module ***m= ods, unsigned long *addrs, u3 =20 /* We return either err < 0 in case of error, ... */ if (err) { - kprobe_multi_put_modules(arr.mods, arr.mods_cnt); - kfree(arr.mods); + kprobe_multi_put_modules((struct module **)arr.ptrs, arr.cnt); + kfree(arr.ptrs); return err; } =20 /* or number of modules found if everything is ok. */ - *mods =3D arr.mods; - return arr.mods_cnt; + *mods =3D (struct module **)arr.ptrs; + return arr.cnt; } =20 static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt) --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F9DB218ADE; Wed, 28 May 2025 03:50:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404202; cv=none; b=K1JZFt3dokFLmMykTXb/O1+lnaIQJoCMK+HaR/ItZXznQyQ6mVr93Eu0E3olH/UKqhUF1sZm+8nBuSK5jHB19oexhVAWRmt3WW1meUTD214XA6q6uJ0yQICuH9PzipS3p2P0QizvSJDiyRAdLgZw3aFWH7REPwAb/Jv4swMmSG0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404202; c=relaxed/simple; bh=XfHTqGgIuYKDaki6V1V1qv8At05h/K/OYdumPMde5mk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=m8hcvXczdKFzgZYT/WINiNLxJiLwgXjJQ4X0+gBTDpydKkW+6QR/Tj49KSLqS20SaEcq9g/2mqA0SIgW2ztAL+zDPffe1ns2Slg36BwglnD22bf/FQAftByD3J393a3iDzJvpxrgawJr/7Kp0pwhjmmx7XrYNlzMly6/IaZi9h4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mbv1heEL; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mbv1heEL" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-2347b7d6aeeso25626295ad.2; Tue, 27 May 2025 20:50:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404200; x=1749009000; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v7XLlY7hAS+lvBDHPoWxg90r1GSSdzm3rmlEr8zDuWw=; b=mbv1heELfk+dc+hyx+YuDRJ6RzNzVgkmCFf16r5kUaeJHS6AgtX0KAtS7b9ZeiVKWv lau4YzIU4/xXLKpcF1W7G7vM0Fi6mjtcVwPV8syKHNP2b6gL8NVRt2Ei/6UQlM5B8rWx tZ1NCkH7fkgiLGb9LYO0QuzLWgVfTwIAZShsxukTe2qyhIJHPui+rFtXOHGD1XsVbSp1 pMlz5dJvQMhxCkQuESu0JRP4ZSPeOxWWEFZ/XwJQ13IvVQwrhXZEN8lucHwKejdfaR7u BoQk64JaRC2bBf7fFyxSx6KsdBTvcGa8sfOCNib6JtoAbnpriA/C7en9WhsbafiqkThe qwEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404200; x=1749009000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v7XLlY7hAS+lvBDHPoWxg90r1GSSdzm3rmlEr8zDuWw=; b=W1ykDD5BmExx2X8VlF7Uy94/FJa8hHfAbxGspdRbddxXWYtSuXYIzP//A9IzNOCj3l Mb2k63lrQcIgRrLm3IIDuHGmgLHXDuvXbHiMmKwpO3UMryd62Au3homWqVk7POsXwuLx HxuR7v0GJmwmcB55hD9XZAtzSjwUxxcXFdReqEUxysTpKi+fzTxp5+Sp2FB9VFgbO0Jn wlRPpRXLQetlkKwYMWFGx3PQReFIZnd0Nh2RCl+SImO6qLu6OZK1cC4AXBlEK+Eiifvk KTcP4Ec2ZljFu6nde2Q39s+Lp7HArn7fWCn56BvCZeKkRwlp2qUfUWUv9r+1ySpVvaFC TpVQ== X-Forwarded-Encrypted: i=1; AJvYcCViEFtoOATvda8jCqf8abdfs8iJKl06JXLPmHxFEWKKmxnxpPMLniNAFiZnlNYyQU49SUBedo8gIX+geXo=@vger.kernel.org X-Gm-Message-State: AOJu0YzL/wgFQnnd4cqdmVz38ug2bemAleT4b4u0lL117P52rjX+Et/u wz4AbRzn46YMEXg78vtoX4khbyh8VPOSerafdgS97Uf6i/To8D3i7Xhc+hb03yrg X-Gm-Gg: ASbGncseaoQ+/qFOiOe5N/JmA0kk+JqvuvEWVm6fNF902yw+raK14kuvchP0OGji901 t8VgrKqO8HgMs6VoRZLTL6cHjEm5p2nxDjYJ7XBTwS+NVdU5I/8Tcrslfpbi/Izw75Z0cDONfNJ ps8kFvLxuutTr9KuufDSgN7rSV+790KM1KqmXTH4LNX7pfh997viNbh4pwYguMUAwh/fcn6SgEn Hg8Hy2A7QcKi/+S/S+DtLm5DMQ4WlQBvI6b+jXCkIkW14Vxiq+Y9kT+HSCAkZF7z2A59SyRmwT8 WGgmxQEasJ9QZI/NASHCjefZlVeP0HiUGJsGc+X6ziEmlC9qKufE1/I1NGq8fAQ7JANSqu6ZZ2U WTh4= X-Google-Smtp-Source: AGHT+IFgGdV9N1DbyDMHm8JQfixq2yy5gE1OVQFBwQtcwt5vNfeEYP2KUBZc9OT9Zl5GgvxilgA5CQ== X-Received: by 2002:a17:902:c412:b0:234:a44c:ff8c with SMTP id d9443c01a7336-234a44d0091mr58217715ad.18.1748404200323; Tue, 27 May 2025 20:50:00 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.49.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:49:59 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 11/25] bpf: verifier: add btf to the function args of bpf_check_attach_target Date: Wed, 28 May 2025 11:46:58 +0800 Message-Id: <20250528034712.138701-12-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add target btf to the function args of bpf_check_attach_target(), then the caller can specify the btf to check. Signed-off-by: Menglong Dong --- include/linux/bpf_verifier.h | 1 + kernel/bpf/syscall.c | 6 ++++-- kernel/bpf/trampoline.c | 1 + kernel/bpf/verifier.c | 8 +++++--- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 78c97e12ea4e..85ed52a4e50b 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -894,6 +894,7 @@ static inline void bpf_trampoline_unpack_key(u64 key, u= 32 *obj_id, u32 *btf_id) int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id, struct bpf_attach_target_info *tgt_info); void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index e22a23aa03d1..60865a27d7d3 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3593,9 +3593,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *= prog, * need a new trampoline and a check for compatibility */ struct bpf_attach_target_info tgt_info =3D {}; + struct btf *btf; =20 - err =3D bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, - &tgt_info); + btf =3D tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; + err =3D bpf_check_attach_target(NULL, prog, tgt_prog, btf, + btf_id, &tgt_info); if (err) goto out_unlock; =20 diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index be06dd76505a..3d7fd59107ed 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -907,6 +907,7 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *pr= og, int err; =20 err =3D bpf_check_attach_target(NULL, prog, NULL, + prog->aux->attach_btf, prog->aux->attach_btf_id, &tgt_info); if (err) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d5807d2efc92..b3927db15254 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -23078,6 +23078,7 @@ static int check_non_sleepable_error_inject(u32 btf= _id) int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id, struct bpf_attach_target_info *tgt_info) { @@ -23090,7 +23091,6 @@ int bpf_check_attach_target(struct bpf_verifier_log= *log, const struct btf_type *t; bool conservative =3D true; const char *tname, *fname; - struct btf *btf; long addr =3D 0; struct module *mod =3D NULL; =20 @@ -23098,7 +23098,6 @@ int bpf_check_attach_target(struct bpf_verifier_log= *log, bpf_log(log, "Tracing programs must provide btf_id\n"); return -EINVAL; } - btf =3D tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; if (!btf) { bpf_log(log, "FENTRY/FEXIT program can only be attached to another program annotated= with BTF\n"); @@ -23477,6 +23476,7 @@ static int check_attach_btf_id(struct bpf_verifier_= env *env) struct bpf_attach_target_info tgt_info =3D {}; u32 btf_id =3D prog->aux->attach_btf_id; struct bpf_trampoline *tr; + struct btf *btf; int ret; u64 key; =20 @@ -23501,7 +23501,9 @@ static int check_attach_btf_id(struct bpf_verifier_= env *env) prog->type !=3D BPF_PROG_TYPE_EXT) return 0; =20 - ret =3D bpf_check_attach_target(&env->log, prog, tgt_prog, btf_id, &tgt_i= nfo); + btf =3D tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; + ret =3D bpf_check_attach_target(&env->log, prog, tgt_prog, btf, + btf_id, &tgt_info); if (ret) return ret; =20 --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 640D61EF38F; Wed, 28 May 2025 03:50:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404205; cv=none; b=iqjKu5PTrvWr7NNZu85UVcAL0StfhZNiAJdz9lLz0L3UeUJisoGIjRiu2x7sy5hJS8ZSzGXRJNOvJLwLvhZGifGG2SpHX3fFUfliT9Lnxw7R4Cevo/yFaknZZ6BeoZ/JHlongQNLzvB5WN3i/zWpvI7LLzMX4rp9zmDM6aZB9mk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404205; c=relaxed/simple; bh=Zs3cP7UFli99gzff2jGHioUvnLhMgTpoIoM6EBMVvUM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EWPTlc0wO0XGjXAeje5Xt83wLnf8GqdhLIOkz7HIWlwe8s0PKREK5iMf6VeREzhB+aSmvIQvDphPmSaKR2LqokB2t0G+l3R9MwnUgC/b5B2Xa17g/d5jHiGclIcYMKE6uC1cO+KnvmEQzPj9okBPXbmo9r9U0ecaov0lyF19Cf0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BJhFYxel; arc=none smtp.client-ip=209.85.215.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BJhFYxel" Received: by mail-pg1-f194.google.com with SMTP id 41be03b00d2f7-b26c5fd40a9so359780a12.1; Tue, 27 May 2025 20:50:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404202; x=1749009002; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QTd0uBOUtnBte3BNrocZYUMCmEJy6MJODQUX7kabL/c=; b=BJhFYxelILflZwKebl6L1twkLIRAJ6Pw+hj6sqxIEn3sra/Y/vO3NKR/8wiDrPBZRb gwdpeOpLhyDKBLb5dm1G+yVla7+NM6ZuO/JJ6q1nSO1xi6wVgNvQCK81C/YsZX+7TsLu DLKduDLMmBbODpHdiEMxg765vkzJnAvMo1fO5tz7ItF8zAk3JoPXZKAD7ubF68+OdW/w H0G1wviIGxrN17O676sSMQNgeQzSvbjZ3/LHNB1S2qggxG8WNv335p3qSGhd5EhYLs4Y YeyvTm2t/YzU6hgya4t+vDvxVYvhZ94dWqXjJ+432aOncBXKnc3R3fT8/7ehy6LX0Beh TXDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404202; x=1749009002; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QTd0uBOUtnBte3BNrocZYUMCmEJy6MJODQUX7kabL/c=; b=I/natiuMdcNSFge+UC35lAFsF18GNjHopTsmE6VTanfByj4rijFQFStQkSXDy85MCh wt/WC2jlwUOdD4IL7k3qsR1u6TihstvJnksYkBW9LjHQ9wMKRa8Svs6NpnSwSrsQKJQB kKCIWPkPjiQmeZSn+fbpW8bSAVkmqm9Kz4rPGGGzhZm2ENFZ9egKPWKshnqX8rKPbHTU HYwnzpc9WsTMi7HMtARFSF23WoOBFYQ0bV+vgd8ddn7aoyRmb2vF6eqvUhZJtwNlywlo 2i1ridl+LURTPUX/YNxtd7fpN1Nh8TPT2uikV28ABZxnOc/eKt5gNpyA229nLDA1Sa0W CWXA== X-Forwarded-Encrypted: i=1; AJvYcCUCqNgnkzgdF5j4OyBu6gV0qNipk4Xo99TRxadPdA7khtyMSFfT4AtuVoHwD1z0REnrOPdQkkzx8zEJOyE=@vger.kernel.org X-Gm-Message-State: AOJu0Ywn6NMqTUeV5zro+9eilSv7OkbPfeIDGccRAJVbtnGN/TxSfmqA XRNUaTPpme36Hwwl/fyAmSc4zeTyR0RN544nN7cR+aZZyLqLsbsU7oIj X-Gm-Gg: ASbGncs1eJ0OW8n1Hbub1Wsy4w/+abjA3+4P4A/6SOt61Ey9KtZLh9eOC1FgoZjDkFh WTeQnW0HsPVB5MLFMJNlWgD6Zq1mpSt3Q9zPxTC9nmf8fxZXUZGZZ7WRJkP/rbivfC0sskevfVi pc7W+2iy7Rd+SvPMi3sSxAzUdmqKjBnmHEXmpG1TUnmqEFK6VZjMZSiEw/leKNdzHkH12Errbo0 /gbaPCzI+6NRfTRrmObI2cV7Oehk1mOWbJ+nVAyFp4G6ve1abIJ2qDmPGQEcJoli5xUdeXXtJOC 77fvAHRdt2zvAQ0Y1d5tRHXdy6PSHO402+xiIHS36eMxGL6Gdp8MywHNLLZxCeH/WqhTku+wSfY 4d+Y= X-Google-Smtp-Source: AGHT+IGQDQDeKxLuC6waHd36AvNbYhvq1YmynrRV69vPb94pJiDcTkMsvXDljCzNwZn40vkbp6Bx9A== X-Received: by 2002:a17:902:ea03:b0:234:adce:3eb8 with SMTP id d9443c01a7336-234b749ddf5mr48676305ad.12.1748404202391; Tue, 27 May 2025 20:50:02 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:02 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 12/25] bpf: verifier: move btf_id_deny to bpf_check_attach_target Date: Wed, 28 May 2025 11:46:59 +0800 Message-Id: <20250528034712.138701-13-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the checking of btf_id_deny and noreturn_deny from check_attach_btf_id() to bpf_check_attach_target(). Therefore, we can do such checking during attaching for tracing multi-link in the later patches. Signed-off-by: Menglong Dong --- kernel/bpf/verifier.c | 125 ++++++++++++++++++++++-------------------- 1 file changed, 65 insertions(+), 60 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b3927db15254..5d2e70425c1d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -23075,6 +23075,52 @@ static int check_non_sleepable_error_inject(u32 bt= f_id) return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id); } =20 +BTF_SET_START(btf_id_deny) +BTF_ID_UNUSED +#ifdef CONFIG_SMP +BTF_ID(func, migrate_disable) +BTF_ID(func, migrate_enable) +#endif +#if !defined CONFIG_PREEMPT_RCU && !defined CONFIG_TINY_RCU +BTF_ID(func, rcu_read_unlock_strict) +#endif +#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE) +BTF_ID(func, preempt_count_add) +BTF_ID(func, preempt_count_sub) +#endif +#ifdef CONFIG_PREEMPT_RCU +BTF_ID(func, __rcu_read_lock) +BTF_ID(func, __rcu_read_unlock) +#endif +BTF_SET_END(btf_id_deny) + +/* fexit and fmod_ret can't be used to attach to __noreturn functions. + * Currently, we must manually list all __noreturn functions here. Once a = more + * robust solution is implemented, this workaround can be removed. + */ +BTF_SET_START(noreturn_deny) +#ifdef CONFIG_IA32_EMULATION +BTF_ID(func, __ia32_sys_exit) +BTF_ID(func, __ia32_sys_exit_group) +#endif +#ifdef CONFIG_KUNIT +BTF_ID(func, __kunit_abort) +BTF_ID(func, kunit_try_catch_throw) +#endif +#ifdef CONFIG_MODULES +BTF_ID(func, __module_put_and_kthread_exit) +#endif +#ifdef CONFIG_X86_64 +BTF_ID(func, __x64_sys_exit) +BTF_ID(func, __x64_sys_exit_group) +#endif +BTF_ID(func, do_exit) +BTF_ID(func, do_group_exit) +BTF_ID(func, kthread_complete_and_exit) +BTF_ID(func, kthread_exit) +BTF_ID(func, make_task_dead) +BTF_SET_END(noreturn_deny) + int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, @@ -23398,6 +23444,25 @@ int bpf_check_attach_target(struct bpf_verifier_lo= g *log, =20 break; } + + if (prog->type =3D=3D BPF_PROG_TYPE_LSM) { + ret =3D bpf_lsm_verify_prog(log, prog); + if (ret < 0) { + module_put(mod); + return ret; + } + } else if (prog->type =3D=3D BPF_PROG_TYPE_TRACING && + btf_id_set_contains(&btf_id_deny, btf_id)) { + module_put(mod); + return -EINVAL; + } else if ((prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT || + prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN) && + btf_id_set_contains(&noreturn_deny, btf_id)) { + module_put(mod); + bpf_log(log, "Attaching fexit/fmod_ret to __noreturn functions is reject= ed.\n"); + return -EINVAL; + } + tgt_info->tgt_addr =3D addr; tgt_info->tgt_name =3D tname; tgt_info->tgt_type =3D t; @@ -23405,52 +23470,6 @@ int bpf_check_attach_target(struct bpf_verifier_lo= g *log, return 0; } =20 -BTF_SET_START(btf_id_deny) -BTF_ID_UNUSED -#ifdef CONFIG_SMP -BTF_ID(func, migrate_disable) -BTF_ID(func, migrate_enable) -#endif -#if !defined CONFIG_PREEMPT_RCU && !defined CONFIG_TINY_RCU -BTF_ID(func, rcu_read_unlock_strict) -#endif -#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE) -BTF_ID(func, preempt_count_add) -BTF_ID(func, preempt_count_sub) -#endif -#ifdef CONFIG_PREEMPT_RCU -BTF_ID(func, __rcu_read_lock) -BTF_ID(func, __rcu_read_unlock) -#endif -BTF_SET_END(btf_id_deny) - -/* fexit and fmod_ret can't be used to attach to __noreturn functions. - * Currently, we must manually list all __noreturn functions here. Once a = more - * robust solution is implemented, this workaround can be removed. - */ -BTF_SET_START(noreturn_deny) -#ifdef CONFIG_IA32_EMULATION -BTF_ID(func, __ia32_sys_exit) -BTF_ID(func, __ia32_sys_exit_group) -#endif -#ifdef CONFIG_KUNIT -BTF_ID(func, __kunit_abort) -BTF_ID(func, kunit_try_catch_throw) -#endif -#ifdef CONFIG_MODULES -BTF_ID(func, __module_put_and_kthread_exit) -#endif -#ifdef CONFIG_X86_64 -BTF_ID(func, __x64_sys_exit) -BTF_ID(func, __x64_sys_exit_group) -#endif -BTF_ID(func, do_exit) -BTF_ID(func, do_group_exit) -BTF_ID(func, kthread_complete_and_exit) -BTF_ID(func, kthread_exit) -BTF_ID(func, make_task_dead) -BTF_SET_END(noreturn_deny) - static bool can_be_sleepable(struct bpf_prog *prog) { if (prog->type =3D=3D BPF_PROG_TYPE_TRACING) { @@ -23533,20 +23552,6 @@ static int check_attach_btf_id(struct bpf_verifier= _env *env) return bpf_iter_prog_supported(prog); } =20 - if (prog->type =3D=3D BPF_PROG_TYPE_LSM) { - ret =3D bpf_lsm_verify_prog(&env->log, prog); - if (ret < 0) - return ret; - } else if (prog->type =3D=3D BPF_PROG_TYPE_TRACING && - btf_id_set_contains(&btf_id_deny, btf_id)) { - return -EINVAL; - } else if ((prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT || - prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN) && - btf_id_set_contains(&noreturn_deny, btf_id)) { - verbose(env, "Attaching fexit/fmod_ret to __noreturn functions is reject= ed.\n"); - return -EINVAL; - } - key =3D bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_i= d); tr =3D bpf_trampoline_get(key, &tgt_info); if (!tr) --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42B5021A95D; Wed, 28 May 2025 03:50:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404206; cv=none; b=BQ9Rz1L4MLc7UP8plny7HAiZriQTnCJQKt2vLRBW6M15CKC41WR3uhLv8zqdVHzilTBggGt4teAk9TiE/WgmSrzHQvkHxW+VpW/x9FsZzH8AIaBHT1qO2plt6sSsF8ss2P2rrJBc91JJCezOapjcQsPiVM2TH5pFC0Mp9CpOzX4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404206; c=relaxed/simple; bh=3+KAGe0mMKvHWbDVS0yeauVnAfqDNvt543iPPCKcClE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mtMvdUbmxfiGx58QK5Pc0HNSCIDYktq7pTMdQqzN/Unbt4yI2ScXHAHUMFxwS7XFPRlHFVaRBmaLcgelS7ekRYXnF3qoFu06Sfxr4wBDH/bAGoon3wePwphlOVEqge59OdNavbuy4XPnwBts5LZeQ/4gxZ1V7RSDu2OwNvOP4Ps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XXZ2g+VE; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XXZ2g+VE" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-234ade5a819so11270205ad.1; Tue, 27 May 2025 20:50:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404204; x=1749009004; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4AN7saxVdEQOZ24jtiS71Z11sBzLh/We020pOkc7UII=; b=XXZ2g+VEFzqp0pnYOxZjzwe6ct9wMDfdr9Ik8fn07GC9NPiAjZrnqQdMRMLZa5T7dp PcBkMqLbJxNxkoZ/VDKqbDMeh/SXGYfrJ40wKD+5Zx5rwBrv8cyd1+dyTTJgLHks71OC RME/R+dYepZW2WOFi2bhWFZQ3FETDLeZxYI62BKFmw22fpRpk+g4wrTZz/eR5XmHnbIp 7pkhCMixv+1/GodCoe8tKd2yCvYjIlrvyKzorYHCUVDK3/JxYIu0WWLxpj1GUN5vgWwZ X0XvnIuQFvh24kLkyymheDX38EcV/dtFQYgIY/PTxfCYirMVbwb7mtBrnyXKKHp+sHwh piEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404204; x=1749009004; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4AN7saxVdEQOZ24jtiS71Z11sBzLh/We020pOkc7UII=; b=nHX/FBZpJuCvzG+Hrrw4F+HY+97AM1OnDdvBpWCfS0K3fMQnRHCIvX+1jiDD5IhtWV 5qo2i5zz/mfHMuogx0f5oGzHJ9hHiuoSS5qoSZPiGOA59nvvqMz4j8rnoBcGI6lwG8Bt kux3KGqcPg5nr4I5A4lH3gwQJzugiBYNxE5Xut2HFiBO7/2ct3wi5YpjbRJkV6CnPJD5 byZInVpuD52gagr7K0V3Sxiq0TIE13cYtIClN9A7hjMJH8d2daim8ZRnggXoSjq/IjIJ FYZB+sWjTmE5WjvhlsZw2ZwfB4GURYxLBdVGMTSgO+TuTrGZpXWZL7jkechaRIORSsXN kZNw== X-Forwarded-Encrypted: i=1; AJvYcCW+ewUNqneKWGkNG0znThPT8IXca7MuyiH758X+WptgT621kPo3W0c5zQZbdhyBqiVqEbP+nDMZFtllFn4=@vger.kernel.org X-Gm-Message-State: AOJu0YzPoFVDDc20ODeHEnOBmBTFfmnB13LNRK3AX9cV4UOfj6S0kYDC 37oNyOwELMqRS+nWMiBMe7xbFI6iuyNMxOGaHmW3e4K7rGkcJ++qekTj X-Gm-Gg: ASbGnctU6UrOHzKiXZfon89E9Ui5H4p2Y7kQ82AaYTkCVWrNt3npnpKHDCWv3kNcj0d V7QW/CjsQmrDGdjikUSs5a7DebeCb9d8guMVnXOlqGXtt8VQmUZUsoDDgagkwXY7ab+afvaNbE7 PjhBLxI0LxD8WehUAryLLDA8XEQjEvEcizDfUqLjNvVaxc2oiCJEMblK/oPfBADLQJm0QMNfs9A qSMvAiPL31xr9IuWo3dUB9NJRtki/cmP2KRnaQrHxgnZi4PxdLcjGSnO3BMGol1xCINP7hlrWMZ 3A87b2mvx+bLyoUDMPzfnQYQJW2sLl/C3M6GSj6UyhJePgozlKBSYsWok2umM/kadzsr X-Google-Smtp-Source: AGHT+IG+E65Yzbk/55Dmvh6JNx45fYvnIXnjlLRmlCxJdxBJ3L66qt+G/U2TIaq1ic9zNbrT29lP4w== X-Received: by 2002:a17:903:2290:b0:234:d7b2:2ab9 with SMTP id d9443c01a7336-234d7b22ca6mr1525905ad.12.1748404204531; Tue, 27 May 2025 20:50:04 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:04 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 13/25] x86,bpf: factor out __arch_get_bpf_regs_nr Date: Wed, 28 May 2025 11:47:00 +0800 Message-Id: <20250528034712.138701-14-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Factor the function __arch_get_bpf_regs_nr() to get the regs count that used by the function args. The arch_get_bpf_regs_nr() will return -ENOTSUPP if the regs is not enough to hold the function args. Signed-off-by: Menglong Dong --- arch/x86/net/bpf_jit_comp.c | 36 +++++++++++++++++++++++++++++------- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 5 +++++ 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 9e5fe2ba858f..84bb668f3bee 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2945,6 +2945,33 @@ static int invoke_bpf_mod_ret(const struct btf_func_= model *m, u8 **pprog, return 0; } =20 +static int __arch_bpf_get_regs_nr(const struct btf_func_model *m) +{ + int nr_regs =3D m->nr_args; + + /* extra registers for struct arguments */ + for (int i =3D 0; i < m->nr_args; i++) { + if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) + nr_regs +=3D (m->arg_size[i] + 7) / 8 - 1; + } + + return nr_regs; +} + +int arch_bpf_get_regs_nr(const struct btf_func_model *m) +{ + int nr_regs =3D __arch_bpf_get_regs_nr(m); + + /* The maximum number of registers that can be used to pass + * arguments is 6. If the number of registers exceeds this, + * return -ENOTSUPP. + */ + if (nr_regs > 6) + return -EOPNOTSUPP; + + return nr_regs; +} + /* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */ #define LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack) \ __LOAD_TCC_PTR(-round_up(stack, 8) - 8) @@ -3015,7 +3042,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im struct bpf_tramp_links *tlinks, void *func_addr) { - int i, ret, nr_regs =3D m->nr_args, stack_size =3D 0; + int i, ret, nr_regs, stack_size =3D 0; int regs_off, nregs_off, ip_off, run_ctx_off, arg_stack_off, rbx_off; struct bpf_tramp_links *fentry =3D &tlinks[BPF_TRAMP_FENTRY]; struct bpf_tramp_links *fexit =3D &tlinks[BPF_TRAMP_FEXIT]; @@ -3033,15 +3060,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf= _tramp_image *im, void *rw_im WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) && (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET))); =20 - /* extra registers for struct arguments */ - for (i =3D 0; i < m->nr_args; i++) { - if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) - nr_regs +=3D (m->arg_size[i] + 7) / 8 - 1; - } - /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 * are passed through regs, the remains are through stack. */ + nr_regs =3D __arch_bpf_get_regs_nr(m); if (nr_regs > MAX_BPF_FUNC_ARGS) return -ENOTSUPP; =20 diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c35da9d91125..080bb966d026 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1221,6 +1221,7 @@ void arch_free_bpf_trampoline(void *image, unsigned i= nt size); int __must_check arch_protect_bpf_trampoline(void *image, unsigned int siz= e); int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, struct bpf_tramp_links *tlinks, void *func_addr); +int arch_bpf_get_regs_nr(const struct btf_func_model *m); =20 u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5d2e70425c1d..9c4e29bc98c0 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -22901,6 +22901,11 @@ static int do_check_main(struct bpf_verifier_env *= env) } =20 =20 +int __weak arch_bpf_get_regs_nr(const struct btf_func_model *m) +{ + return -ENODEV; +} + static void print_verification_stats(struct bpf_verifier_env *env) { int i; --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB4C321B185; Wed, 28 May 2025 03:50:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404209; cv=none; b=WF1f/BlmZI9W87QT2Hmj/OBLQWyI7aVbTqw+bj5VY8GVZp40VBGw2weV+ozNftSo96YNbbtWPy+GGrJ1t3oMeHXWEL4hfJG+jLuJUduD27YDkiMwlTJeQkB8Hyp1MRqRB42PrWKGvkITPgBFQGT6pnDPuzCXHV57XBBeTBVW/7c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404209; c=relaxed/simple; bh=+GQpoZXDPO7rGQs0NfF2G75ig1nN7rowJn5wSET/d3o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WRLd1e4GMSxjLNcGMUpFdt9EzNt3kcTcupdWYUv93bduuC1iWKqXDQVSrzkByUkrw//8rC9ftgLlyIyRPKRTx7y7jpmmowv9j7PCd8XxbBeniJGdtWSqMrCqZqTHapo9BWxmQYGO6kdnmaKcTtJNl49rwxKTf613uplTflpA5pc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=O1k0LxWX; arc=none smtp.client-ip=209.85.216.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="O1k0LxWX" Received: by mail-pj1-f68.google.com with SMTP id 98e67ed59e1d1-30e5430ed0bso3878637a91.3; Tue, 27 May 2025 20:50:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404207; x=1749009007; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pd1iJVXmvy8UHPpKSuMU0wcMCTE7OFRL0Fq7hy/IwPI=; b=O1k0LxWXMxcl0CI+buidq8BRWXJCkaQlehoLGkXe52eUnfoEwKzCHbcZw4eJl1P+bM M3rz1hJpdU2mkdw/q1P5t03XgkyKhlyQU00G9iYW9A4ZQBE1381l3I0uGgV58C9ykGJS 3uo5kKvjKSZLCEO+fhQsQWR0vGS6/gDa751q0W1PZcxsz6xt0naj5Jhd09u5Lt/MUhlI Fzox184HK++1Ftk8fDgfYxXo3Swu7BsQ11y864sVeNJ7sH/6NKozXMwYkbGudZ0cOjso h1dclMy5vTDxme/K8vmMD1IhHBkjpArm/1GlLtNvJTqJ7C+aDr0S+15HeJ7OuAMqv9Vd MbKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404207; x=1749009007; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pd1iJVXmvy8UHPpKSuMU0wcMCTE7OFRL0Fq7hy/IwPI=; b=woYFxOBJJnBMd8oKTxunOxXBjVQhHhJgR/6Gakma/dtpz5RULOLfFSJUj1mP1ej7o2 QxTwv5fYBSXxqk/YdzfPTg1G9tHca7ebESRPf3zhfuu3Yejr7tuKT/2jTP0tN+82k6xO oPHQIEsERNhDwM8YUNfL75loDR35kER2boCnLgkQ3ZD9ucHWArTwsqEq8WzMLKb3iBiK H95pPhRzI823FKys1i13Kc6FhC38ncMFYgsdf83w8P5hSlfgg7P+BBoPn/cq5YchZcMk MVI97LrAVnSLumzFYPa7kiTLSZtlmOdSyKi8xF4pxWK+uCQs9cIBzgiDWKd94FhtgT3n oKGg== X-Forwarded-Encrypted: i=1; AJvYcCXTBHneBZSDjVX0mVxGuPg5uRXbIRdAQTJ7SYtzxGttUH4DgiYcwcy46XWjk2j+xV3O6Y3J1NoKDf/EEDQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwUFNNvKOTjPuXZt2/yI/9qAWdXYEmpZYBNs6EB2qze+PqPu5VG 2d37LVtfgOF8/w/vKB5bmKaTgTEL0FJTei6f+N3MoYPdRTta0UJyNq5W1aGFRmOt X-Gm-Gg: ASbGncvbmvg+yeinegF/6D28pOrfJn/3NlhzH7BHLz61aZDSFi9PamNPQv7ZHrZPPFV BLXB1JAztAGT/jS7KDz51wrXFLCGRD9XZ9YZriKo+qQlG5ViSLMP2Mg1NMEjaJMElLhe81h468/ /vqTcxA3A8uRpV8uEn6y54lQZeCvBKIJQ5nipiapxgQymYIpQvY/I3myngjM3EVaHSpti5HykoX PzO9SSGGo+XQK0MeIkKtI6WCe7iTuxerp/O+72QkqIV45VGdDPtpWc4QNCQ/4eYLc+iTMyAgoaL zUqUivSu+/CyzcV0icB76tVEjVx9VVawN3GKGC7t/lw9n4DTm3oyp42v5Kz8lZyG46zC X-Google-Smtp-Source: AGHT+IE9KH+zIs6JCpp9NMS9hc8oxqAOFu3qZ83LuK5e1fLc5u6ULkMEO9E+4nONeHSydvcsq8VxYA== X-Received: by 2002:a17:90b:55cb:b0:2ee:ad18:b309 with SMTP id 98e67ed59e1d1-3110f113859mr23293747a91.3.1748404206746; Tue, 27 May 2025 20:50:06 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:06 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 14/25] bpf: tracing: add multi-link support Date: Wed, 28 May 2025 11:47:01 +0800 Message-Id: <20250528034712.138701-15-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In this commit, we add the support to allow attaching a tracing BPF program to multi hooks, which is similar to BPF_TRACE_KPROBE_MULTI. The use case is obvious. For now, we have to create a BPF program for each kernel function, for which we want to trace, even through all the program have the same (or similar logic). This can consume extra memory, and make the program loading slow if we have plenty of kernel function to trace. The KPROBE_MULTI maybe a alternative, but it can't do what TRACING do. For example, the kretprobe can't obtain the function args, but the FEXIT can. For now, we support to create multi-link for fentry/fexit/modify_return with the following new attach types that we introduce: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI We introduce the struct bpf_tracing_multi_link for this purpose, which can hold all the kernel modules, target bpf program (for attaching to bpf program) or target btf (for attaching to kernel function) that we referenced. During loading, the first target is used for verification by the verifer. And during attaching, we check the consistency of all the targets with the first target. Signed-off-by: Menglong Dong --- include/linux/bpf.h | 11 + include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 10 + kernel/bpf/btf.c | 5 + kernel/bpf/syscall.c | 365 +++++++++++++++++++++++++++++++++ kernel/bpf/trampoline.c | 7 +- kernel/bpf/verifier.c | 25 ++- net/bpf/test_run.c | 3 + net/core/bpf_sk_storage.c | 2 + tools/include/uapi/linux/bpf.h | 10 + 10 files changed, 435 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 080bb966d026..7191ad25d519 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1802,6 +1802,17 @@ struct bpf_raw_tp_link { u64 cookie; }; =20 +struct bpf_tracing_multi_link { + struct bpf_gtramp_link link; + enum bpf_attach_type attach_type; + struct bpf_prog **tgt_progs; + struct btf **tgt_btfs; + struct module **mods; + u32 prog_cnt; + u32 btf_cnt; + u32 mods_cnt; +}; + struct bpf_link_primer { struct bpf_link *link; struct file *file; diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index fa78f49d4a9a..139d5436ce4c 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -154,3 +154,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi) +BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 16e95398c91c..45dfaf40230e 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1120,6 +1120,9 @@ enum bpf_attach_type { BPF_NETKIT_PEER, BPF_TRACE_KPROBE_SESSION, BPF_TRACE_UPROBE_SESSION, + BPF_TRACE_FENTRY_MULTI, + BPF_TRACE_FEXIT_MULTI, + BPF_MODIFY_RETURN_MULTI, __MAX_BPF_ATTACH_TYPE }; =20 @@ -1144,6 +1147,7 @@ enum bpf_link_type { BPF_LINK_TYPE_UPROBE_MULTI =3D 12, BPF_LINK_TYPE_NETKIT =3D 13, BPF_LINK_TYPE_SOCKMAP =3D 14, + BPF_LINK_TYPE_TRACING_MULTI =3D 15, __MAX_BPF_LINK_TYPE, }; =20 @@ -1765,6 +1769,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 cnt; + __aligned_u64 tgt_fds; + __aligned_u64 btf_ids; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 64538625ee91..03045f7d428f 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6100,6 +6100,9 @@ static int btf_validate_prog_ctx_type(struct bpf_veri= fier_log *log, const struct case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: /* allow u64* as ctx */ if (btf_is_int(t) && t->size =3D=3D 8) return 0; @@ -6705,6 +6708,7 @@ bool btf_ctx_access(int off, int size, enum bpf_acces= s_type type, fallthrough; case BPF_LSM_CGROUP: case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: /* When LSM programs are attached to void LSM hooks * they use FEXIT trampolines and when attached to * int LSM hooks, they use MODIFY_RETURN trampolines. @@ -6723,6 +6727,7 @@ bool btf_ctx_access(int off, int size, enum bpf_acces= s_type type, t =3D btf_type_by_id(btf, t->type); break; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: /* For now the BPF_MODIFY_RETURN can only be attached to * functions that return an int. */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 60865a27d7d3..0cd989381128 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -37,6 +37,7 @@ #include #include #include +#include =20 #include #include @@ -3466,6 +3467,34 @@ static const struct bpf_link_ops bpf_tracing_link_lo= ps =3D { .fill_link_info =3D bpf_tracing_link_fill_link_info, }; =20 +static int bpf_tracing_check_multi(struct bpf_prog *prog, + struct bpf_prog *tgt_prog, + struct btf *btf2, + const struct btf_type *t2) +{ + const struct btf_type *t1; + struct btf *btf1; + + /* this case is already valided in bpf_check_attach_target() */ + if (prog->type =3D=3D BPF_PROG_TYPE_EXT) + return 0; + + btf1 =3D prog->aux->dst_prog ? prog->aux->dst_prog->aux->btf : + prog->aux->attach_btf; + if (!btf1) + return -EOPNOTSUPP; + + btf2 =3D btf2 ?: tgt_prog->aux->btf; + t1 =3D prog->aux->attach_func_proto; + + /* the target is the same as the origin one, this is a re-attach */ + if (t1 =3D=3D t2) + return 0; + + return btf_check_func_part_match(btf1, t1, btf2, t2, + prog->aux->accessed_args); +} + static int bpf_tracing_prog_attach(struct bpf_prog *prog, int tgt_prog_fd, u32 btf_id, @@ -3665,6 +3694,335 @@ static int bpf_tracing_prog_attach(struct bpf_prog = *prog, return err; } =20 +static void __bpf_tracing_multi_link_release(struct bpf_tracing_multi_link= *link) +{ + int i; + + if (link->mods_cnt) { + for (i =3D 0; i < link->mods_cnt; i++) + module_put(link->mods[i]); + kfree(link->mods); + } + + if (link->prog_cnt) { + for (i =3D 0; i < link->prog_cnt; i++) + bpf_prog_put(link->tgt_progs[i]); + kfree(link->tgt_progs); + } + + if (link->btf_cnt) { + for (i =3D 0; i < link->btf_cnt; i++) + btf_put(link->tgt_btfs[i]); + kfree(link->tgt_btfs); + } + + kfree(link->link.entries); +} + +static void bpf_tracing_multi_link_release(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *multi_link =3D + container_of(link, struct bpf_tracing_multi_link, link.link); + + bpf_gtrampoline_unlink_prog(&multi_link->link); + __bpf_tracing_multi_link_release(multi_link); +} + +static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *tr_link =3D + container_of(link, struct bpf_tracing_multi_link, link.link); + + kfree(tr_link); +} + +static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_tracing_multi_link *tr_link =3D + container_of(link, struct bpf_tracing_multi_link, link.link); + int i; + + for (i =3D 0; i < tr_link->link.entry_cnt; i++) { + seq_printf(seq, + "attach_type:\t%d\n" + "target_addr:\t%p\n", + tr_link->attach_type, + tr_link->link.entries[i].addr); + } +} + +static const struct bpf_link_ops bpf_tracing_multi_link_lops =3D { + .release =3D bpf_tracing_multi_link_release, + .dealloc =3D bpf_tracing_multi_link_dealloc, + .show_fdinfo =3D bpf_tracing_multi_link_show_fdinfo, +}; + +#define MAX_TRACING_MULTI_CNT 102400 + +static int bpf_tracing_get_target(u32 fd, struct bpf_prog **tgt_prog, + struct btf **tgt_btf) +{ + struct bpf_prog *prog =3D NULL; + struct btf *btf =3D NULL; + int err =3D 0; + + if (fd) { + prog =3D bpf_prog_get(fd); + if (!IS_ERR(prog)) + goto found; + + prog =3D NULL; + /* "fd" is the fd of the kernel module BTF */ + btf =3D btf_get_by_fd(fd); + if (IS_ERR(btf)) { + err =3D PTR_ERR(btf); + goto err; + } + if (!btf_is_kernel(btf)) { + btf_put(btf); + err =3D -EOPNOTSUPP; + goto err; + } + } else { + btf =3D bpf_get_btf_vmlinux(); + if (IS_ERR(btf)) { + err =3D PTR_ERR(btf); + goto err; + } + if (!btf) { + err =3D -EINVAL; + goto err; + } + btf_get(btf); + } +found: + *tgt_prog =3D prog; + *tgt_btf =3D btf; + return 0; +err: + *tgt_prog =3D NULL; + *tgt_btf =3D NULL; + return err; +} + +static int bpf_tracing_multi_link_check(const union bpf_attr *attr, u32 **= btf_ids, + u32 **tgt_fds, u64 **cookies, + u32 cnt) +{ + void __user *ubtf_ids; + void __user *utgt_fds; + void __user *ucookies; + void *tmp; + int i; + + if (!cnt) + return -EINVAL; + + if (cnt > MAX_TRACING_MULTI_CNT) + return -E2BIG; + + ucookies =3D u64_to_user_ptr(attr->link_create.tracing_multi.cookies); + if (ucookies) { + tmp =3D kvmalloc_array(cnt, sizeof(**cookies), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *cookies =3D tmp; + if (copy_from_user(tmp, ucookies, cnt * sizeof(**cookies))) + return -EFAULT; + } + + utgt_fds =3D u64_to_user_ptr(attr->link_create.tracing_multi.tgt_fds); + if (utgt_fds) { + tmp =3D kvmalloc_array(cnt, sizeof(**tgt_fds), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *tgt_fds =3D tmp; + if (copy_from_user(tmp, utgt_fds, cnt * sizeof(**tgt_fds))) + return -EFAULT; + } + + ubtf_ids =3D u64_to_user_ptr(attr->link_create.tracing_multi.btf_ids); + if (!ubtf_ids) + return -EINVAL; + + tmp =3D kvmalloc_array(cnt, sizeof(**btf_ids), GFP_KERNEL); + if (!tmp) + return -ENOMEM; + + *btf_ids =3D tmp; + if (copy_from_user(tmp, ubtf_ids, cnt * sizeof(**btf_ids))) + return -EFAULT; + + for (i =3D 0; i < cnt; i++) { + if (!(*btf_ids)[i]) + return -EINVAL; + } + + return 0; +} + +static void bpf_tracing_multi_link_ptr_fill(struct bpf_tracing_multi_link = *link, + struct ptr_array *progs, + struct ptr_array *mods, + struct ptr_array *btfs) +{ + link->mods =3D (struct module **) mods->ptrs; + link->mods_cnt =3D mods->cnt; + link->tgt_btfs =3D (struct btf **) btfs->ptrs; + link->btf_cnt =3D btfs->cnt; + link->tgt_progs =3D (struct bpf_prog **) progs->ptrs; + link->prog_cnt =3D progs->cnt; +} + +static int bpf_tracing_prog_attach_multi(const union bpf_attr *attr, + struct bpf_prog *prog) +{ + struct bpf_tracing_multi_link *link =3D NULL; + u32 cnt, *btf_ids =3D NULL, *tgt_fds =3D NULL; + struct bpf_link_primer link_primer; + struct ptr_array prog_array =3D { }; + struct ptr_array btf_array =3D { }; + struct ptr_array mod_array =3D { }; + u64 *cookies =3D NULL; + int err =3D 0, i; + + if ((prog->expected_attach_type !=3D BPF_TRACE_FENTRY_MULTI && + prog->expected_attach_type !=3D BPF_TRACE_FEXIT_MULTI && + prog->expected_attach_type !=3D BPF_MODIFY_RETURN_MULTI) || + prog->type !=3D BPF_PROG_TYPE_TRACING) + return -EINVAL; + + cnt =3D attr->link_create.tracing_multi.cnt; + err =3D bpf_tracing_multi_link_check(attr, &btf_ids, &tgt_fds, &cookies, + cnt); + if (err) + goto err_out; + + link =3D kzalloc(sizeof(*link), GFP_USER); + if (!link) { + err =3D -ENOMEM; + goto err_out; + } + link->link.entries =3D kzalloc(sizeof(*link->link.entries) * cnt, + GFP_USER); + if (!link->link.entries) { + err =3D -ENOMEM; + goto err_out; + } + + bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING_MULTI, + &bpf_tracing_multi_link_lops, prog); + link->attach_type =3D prog->expected_attach_type; + + mutex_lock(&prog->aux->dst_mutex); + + for (i =3D 0; i < cnt; i++) { + struct bpf_attach_target_info tgt_info =3D {}; + struct bpf_gtramp_link_entry *entry; + struct bpf_prog *tgt_prog =3D NULL; + u32 tgt_fd, btf_id =3D btf_ids[i]; + struct btf *tgt_btf =3D NULL; + struct module *mod =3D NULL; + int nr_regs; + + entry =3D &link->link.entries[i]; + tgt_fd =3D tgt_fds ? tgt_fds[i] : 0; + err =3D bpf_tracing_get_target(tgt_fd, &tgt_prog, &tgt_btf); + if (err) + goto err_out_unlock; + + if (tgt_prog) { + err =3D bpf_try_add_ptr(&prog_array, tgt_prog); + if (err) { + bpf_prog_put(tgt_prog); + if (err !=3D -EEXIST) + goto err_out_unlock; + } + } + + if (tgt_btf) { + err =3D bpf_try_add_ptr(&btf_array, tgt_btf); + if (err) { + btf_put(tgt_btf); + if (err !=3D -EEXIST) + goto err_out_unlock; + } + } + + prog->aux->attach_tracing_prog =3D tgt_prog && + tgt_prog->type =3D=3D BPF_PROG_TYPE_TRACING && + prog->type =3D=3D BPF_PROG_TYPE_TRACING; + + err =3D bpf_check_attach_target(NULL, prog, tgt_prog, tgt_btf, + btf_id, &tgt_info); + if (err) + goto err_out_unlock; + + nr_regs =3D arch_bpf_get_regs_nr(&tgt_info.fmodel); + if (nr_regs < 0) { + err =3D nr_regs; + goto err_out_unlock; + } + + mod =3D tgt_info.tgt_mod; + if (mod) { + err =3D bpf_try_add_ptr(&mod_array, mod); + if (err) { + module_put(mod); + if (err !=3D -EEXIST) + goto err_out_unlock; + } + } + + err =3D bpf_tracing_check_multi(prog, tgt_prog, tgt_btf, + tgt_info.tgt_type); + if (err) + goto err_out_unlock; + + entry->cookie =3D cookies ? cookies[i] : 0; + entry->addr =3D (void *)tgt_info.tgt_addr; + entry->tgt_prog =3D tgt_prog; + entry->attach_btf =3D tgt_btf; + entry->btf_id =3D btf_id; + entry->nr_args =3D nr_regs; + + link->link.entry_cnt++; + } + + err =3D bpf_gtrampoline_link_prog(&link->link); + if (err) + goto err_out_unlock; + + err =3D bpf_link_prime(&link->link.link, &link_primer); + if (err) { + bpf_gtrampoline_unlink_prog(&link->link); + goto err_out_unlock; + } + + bpf_tracing_multi_link_ptr_fill(link, &prog_array, &mod_array, + &btf_array); + mutex_unlock(&prog->aux->dst_mutex); + + kfree(btf_ids); + kfree(tgt_fds); + kfree(cookies); + return bpf_link_settle(&link_primer); +err_out_unlock: + bpf_tracing_multi_link_ptr_fill(link, &prog_array, &mod_array, + &btf_array); + __bpf_tracing_multi_link_release(link); + mutex_unlock(&prog->aux->dst_mutex); +err_out: + kfree(btf_ids); + kfree(tgt_fds); + kfree(cookies); + kfree(link); + return err; +} + static void bpf_raw_tp_link_release(struct bpf_link *link) { struct bpf_raw_tp_link *raw_tp =3D @@ -4133,6 +4491,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_= type) case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: return BPF_PROG_TYPE_TRACING; case BPF_LSM_MAC: return BPF_PROG_TYPE_LSM; @@ -5439,6 +5800,10 @@ static int link_create(union bpf_attr *attr, bpfptr_= t uattr) ret =3D bpf_iter_link_attach(attr, uattr, prog); else if (prog->expected_attach_type =3D=3D BPF_LSM_CGROUP) ret =3D cgroup_bpf_link_attach(attr, prog); + else if (prog->expected_attach_type =3D=3D BPF_TRACE_FENTRY_MULTI || + prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT_MULTI || + prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN_MULTI) + ret =3D bpf_tracing_prog_attach_multi(attr, prog); else ret =3D bpf_tracing_prog_attach(prog, attr->link_create.target_fd, diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 3d7fd59107ed..b92d1d4f1033 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -116,7 +116,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *pro= g) =20 return (ptype =3D=3D BPF_PROG_TYPE_TRACING && (eatype =3D=3D BPF_TRACE_FENTRY || eatype =3D=3D BPF_TRACE_FEXIT || - eatype =3D=3D BPF_MODIFY_RETURN)) || + eatype =3D=3D BPF_MODIFY_RETURN || + eatype =3D=3D BPF_TRACE_FENTRY_MULTI || eatype =3D=3D BPF_TRACE_FEXIT_M= ULTI || + eatype =3D=3D BPF_MODIFY_RETURN_MULTI)) || (ptype =3D=3D BPF_PROG_TYPE_LSM && eatype =3D=3D BPF_LSM_MAC); } =20 @@ -515,10 +517,13 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tr= amp(struct bpf_prog *prog) { switch (prog->expected_attach_type) { case BPF_TRACE_FENTRY: + case BPF_TRACE_FENTRY_MULTI: return BPF_TRAMP_FENTRY; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: return BPF_TRAMP_MODIFY_RETURN; case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: return BPF_TRAMP_FEXIT; case BPF_LSM_MAC: if (!prog->aux->attach_func_proto->type) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9c4e29bc98c0..bbfe8ae39f3c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16945,10 +16945,13 @@ static int check_return_code(struct bpf_verifier_= env *env, int regno, const char switch (env->prog->expected_attach_type) { case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: range =3D retval_range(0, 0); break; case BPF_TRACE_RAW_TP: case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: return 0; case BPF_TRACE_ITER: break; @@ -22264,7 +22267,9 @@ static int do_misc_fixups(struct bpf_verifier_env *= env) if (prog_type =3D=3D BPF_PROG_TYPE_TRACING && insn->imm =3D=3D BPF_FUNC_get_func_ret) { if (eatype =3D=3D BPF_TRACE_FEXIT || - eatype =3D=3D BPF_MODIFY_RETURN) { + eatype =3D=3D BPF_MODIFY_RETURN || + eatype =3D=3D BPF_TRACE_FEXIT_MULTI || + eatype =3D=3D BPF_MODIFY_RETURN_MULTI) { /* Load nr_args from ctx - 8 */ insn_buf[0] =3D BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8); insn_buf[1] =3D BPF_ALU64_IMM(BPF_LSH, BPF_REG_0, 3); @@ -23246,7 +23251,9 @@ int bpf_check_attach_target(struct bpf_verifier_log= *log, if (tgt_prog->type =3D=3D BPF_PROG_TYPE_TRACING && prog_extension && (tgt_prog->expected_attach_type =3D=3D BPF_TRACE_FENTRY || - tgt_prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT)) { + tgt_prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT || + tgt_prog->expected_attach_type =3D=3D BPF_TRACE_FENTRY_MULTI || + tgt_prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT_MULTI)) { /* Program extensions can extend all program types * except fentry/fexit. The reason is the following. * The fentry/fexit programs are used for performance @@ -23345,6 +23352,9 @@ int bpf_check_attach_target(struct bpf_verifier_log= *log, case BPF_LSM_CGROUP: case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_MODIFY_RETURN_MULTI: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", btf_id); @@ -23430,7 +23440,8 @@ int bpf_check_attach_target(struct bpf_verifier_log= *log, bpf_log(log, "%s is not sleepable\n", tname); return ret; } - } else if (prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN) { + } else if (prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN || + prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN_MULTI) { if (tgt_prog) { module_put(mod); bpf_log(log, "can't modify return codes of BPF programs\n"); @@ -23483,6 +23494,9 @@ static bool can_be_sleepable(struct bpf_prog *prog) case BPF_TRACE_FEXIT: case BPF_MODIFY_RETURN: case BPF_TRACE_ITER: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: return true; default: return false; @@ -23557,6 +23571,11 @@ static int check_attach_btf_id(struct bpf_verifier= _env *env) return bpf_iter_prog_supported(prog); } =20 + if (prog->expected_attach_type =3D=3D BPF_TRACE_FENTRY_MULTI || + prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT_MULTI || + prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN_MULTI) + return 0; + key =3D bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_i= d); tr =3D bpf_trampoline_get(key, &tgt_info); if (!tr) diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index aaf13a7d58ed..bcb609425fda 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -696,6 +696,8 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, switch (prog->expected_attach_type) { case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: if (bpf_fentry_test1(1) !=3D 2 || bpf_fentry_test2(2, 3) !=3D 5 || bpf_fentry_test3(4, 5, 6) !=3D 15 || @@ -709,6 +711,7 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, goto out; break; case BPF_MODIFY_RETURN: + case BPF_MODIFY_RETURN_MULTI: ret =3D bpf_modify_return_test(1, &b); if (b !=3D 2) side_effect++; diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c index 2e538399757f..c5b1fd714b58 100644 --- a/net/core/bpf_sk_storage.c +++ b/net/core/bpf_sk_storage.c @@ -369,6 +369,8 @@ static bool bpf_sk_storage_tracing_allowed(const struct= bpf_prog *prog) return true; case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: return !!strncmp(prog->aux->attach_func_name, "bpf_sk_storage", strlen("bpf_sk_storage")); default: diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 16e95398c91c..45dfaf40230e 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1120,6 +1120,9 @@ enum bpf_attach_type { BPF_NETKIT_PEER, BPF_TRACE_KPROBE_SESSION, BPF_TRACE_UPROBE_SESSION, + BPF_TRACE_FENTRY_MULTI, + BPF_TRACE_FEXIT_MULTI, + BPF_MODIFY_RETURN_MULTI, __MAX_BPF_ATTACH_TYPE }; =20 @@ -1144,6 +1147,7 @@ enum bpf_link_type { BPF_LINK_TYPE_UPROBE_MULTI =3D 12, BPF_LINK_TYPE_NETKIT =3D 13, BPF_LINK_TYPE_SOCKMAP =3D 14, + BPF_LINK_TYPE_TRACING_MULTI =3D 15, __MAX_BPF_LINK_TYPE, }; =20 @@ -1765,6 +1769,12 @@ union bpf_attr { */ __u64 cookie; } tracing; + struct { + __u32 cnt; + __aligned_u64 tgt_fds; + __aligned_u64 btf_ids; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAF0F21B9C0; Wed, 28 May 2025 03:50:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404211; cv=none; b=pUkNFUScg5M0CipdY9ffGA5+imvhXdQ8f7kqMfqNvLaNxovzWHbAtHy3yPsaN2ha5o+R7ucuvH6SS6ltI1XfDAYpUalqs0jUICgBJc9xNY4tmIhfiChfstYOghPRi9FtFMJBkpUPAxX3/FHbf8p2En5EKOZjjFppwOcsJuHVfdo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404211; c=relaxed/simple; bh=tT3QgbeZaVPngHmlrmhR/+LqwS5dNFFCHOpwwRuz1fQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=A+/CkIc6k9qbxoJIJD+8Am5WnJNki8LsuBq3YxY1nbQVjMy6IoYPbafastaHbO/zvP4hEnjFrJAMqklvBY1iBw16x8gOgmu5aFaqkt3S5y3KlpB0UNDDDcuiQ6XWXNtlYcomGI2knQLSiBSUP/lz8Tyx89+cEIMaZwOTPbwqNWE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=A9KEJM8I; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A9KEJM8I" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-22e331215dbso3210885ad.1; Tue, 27 May 2025 20:50:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404209; x=1749009009; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YAYS5lKsbl835bzscg3XIO5IGldYqod4Ozc4rgXcA6c=; b=A9KEJM8IoE8GMGlHQQEdsKDuWz9vNnO+EU1prXPMRFM61HGv+WLpMEmo7aSYPr63hh 8ivNsf7uhKcfWP6NghqhqbmSBS5qTDkWTUTWvZuUveJuZZaAq7ue0XyLvY0W03cBE9Al XrmtHP1wgBXxYOehBvjXowGUUE7o0wP0qSCwTKMaLHKvJ7+FJmnqaWTzp7U4854CLnew +RmGzwNJuHlkQlWxGXdmJ2TY9ZtbYADjZEhWHuT5ZGbf6fLwYY8mlSLfSQatOvrAsiek mHq4K0oBGXmvSc3+IOgUQH+Ghnvd8UlEg9Yp3Kv8gNTLTWnAw6MsQv0I+gAX1b4z81mY UoNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404209; x=1749009009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YAYS5lKsbl835bzscg3XIO5IGldYqod4Ozc4rgXcA6c=; b=cRblM+pghtVgTGY5/GRGP+o2+8j7wOG+zO8FdYn+aIMoy8stlQJ+vsm9+eKWPv3QRt XDMfHT1xvlA1OUcBL+mnKDmTW503SgWvCSIZvaA2VWIOQHaN/Yr1ZccJmg+4+BczLJL3 xwxHV9gPZ6P0EwcmqrnT8BAlwZfsNubEt6Y5qupPv9IOzuaUjKqsLrvSq8ftuYj1hJdl rAV0U09oU5tWD3TOX5nuJ4cfEI0DOZ3TskJ0ohtg2yRwvtoy4LJprG7D0H8rVSzescLr kVVuy0EgY+mTLh/WIpYcgVmEB++BW9ft8dII7mZlw4K4W1o79zoz3ibcMKMn5YR7isWS UC+g== X-Forwarded-Encrypted: i=1; AJvYcCV+IhhRXLZHHiOM9vSm9e2Tbg22CKovbdoMrK1TzlJKWLi8zCV5wISHMqNgckS8fpFA20T7jn+RZpHRy2M=@vger.kernel.org X-Gm-Message-State: AOJu0YxrJlKKYEuiyk3rpWyZSKHHQF51qtalTPY5s84Qe85G4l1WIPzt vdcTT4is0C4enpHpL+zb6akPqIYQHANwjU1Kky9JCUS21BpPmDlIhTYi X-Gm-Gg: ASbGncs3psRRGSU1V7Z93EHbvOHEHNZjzh22BM/6DWcwlF9kbFte+FmjV3u+DOMuinj PSDXhrQpg45dj9R2boc2g45Zh3E3xbqgqu7Crirtze7PHrqXizmtew/pZTd0URdCdnUDyljQRsC QhemzKker6UypV3qfXfLcQ1hfUJb3aK6qomDJh9CGeckgkIroQ5jAoXDrYjaWtzUCpPBA3suw9W xd5VKUFr4qbvyVWF4qyARXxQEmOGWVn1sjpvu/F0NDYItGOxgf9F7dMLX+zP8AOk5Ll3L0OwvRX B67GlM3HrU7UkeaxgaqjS7u4KuYlP6cQawxnnIxND3M8LHQ6KShud3GS2Z2tOURYxq0w X-Google-Smtp-Source: AGHT+IEBQQaFiaEybooAwzIyji1mQLsF6d+czJF3HzEWnNxRCVv+hTVem/J4PfY5SyVXBw/Pyk0YBQ== X-Received: by 2002:a17:903:32c5:b0:231:ad5a:fe9c with SMTP id d9443c01a7336-234b74b7d7emr41679175ad.15.1748404208894; Tue, 27 May 2025 20:50:08 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:08 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 15/25] ftrace: factor out __unregister_ftrace_direct Date: Wed, 28 May 2025 11:47:02 +0800 Message-Id: <20250528034712.138701-16-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Factor out __unregister_ftrace_direct, which doesn't hold the direct_mutex lock. Signed-off-by: Menglong Dong --- kernel/trace/ftrace.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 0befb4c93e89..5b6b74ea4c20 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -113,6 +113,8 @@ bool ftrace_pids_enabled(struct ftrace_ops *ops) } =20 static void ftrace_update_trampoline(struct ftrace_ops *ops); +static int __unregister_ftrace_direct(struct ftrace_ops *ops, unsigned lon= g addr, + bool free_filters); =20 /* * ftrace_disabled is set when an anomaly is discovered. @@ -6046,8 +6048,8 @@ EXPORT_SYMBOL_GPL(register_ftrace_direct); * 0 on success * -EINVAL - The @ops object was not properly registered. */ -int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr, - bool free_filters) +static int __unregister_ftrace_direct(struct ftrace_ops *ops, unsigned lon= g addr, + bool free_filters) { struct ftrace_hash *hash =3D ops->func_hash->filter_hash; int err; @@ -6057,10 +6059,8 @@ int unregister_ftrace_direct(struct ftrace_ops *ops,= unsigned long addr, if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) return -EINVAL; =20 - mutex_lock(&direct_mutex); err =3D unregister_ftrace_function(ops); remove_direct_functions_hash(hash, addr); - mutex_unlock(&direct_mutex); =20 /* cleanup for possible another register call */ ops->func =3D NULL; @@ -6070,6 +6070,18 @@ int unregister_ftrace_direct(struct ftrace_ops *ops,= unsigned long addr, ftrace_free_filter(ops); return err; } + +int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr, + bool free_filters) +{ + int err; + + mutex_lock(&direct_mutex); + err =3D __unregister_ftrace_direct(ops, addr, free_filters); + mutex_unlock(&direct_mutex); + + return err; +} EXPORT_SYMBOL_GPL(unregister_ftrace_direct); =20 static int --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C63901F17E8; Wed, 28 May 2025 03:50:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404213; cv=none; b=FlRZfuvEdax3cceJ5+zrrSO1OkDoIn1BZmFwHqyI+HE+SN5vhkIKmM0cnmqwP2H4NbTQ7nylF9riW6N9H8EOg40hsCQc6lUNUUcdejGwJ0ET2iLpcf7Vb33LVk5E0y3yDGfElsrerpCcoA//ZkBhnshO0vBhpai0++K7RZilju0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404213; c=relaxed/simple; bh=1fwUg9kN1ZusHPzIWHFgs+cuOvftS0FFPnEIprFNV2A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WiBKGHgv8MuupV3PLnt9QEx3I4fzqvh0zG5txasJ7P2eSPGWRQKyYIQSjuwxdzXs/97igmn811Of7D7GLXn0s4LRVarHEMZ3hmO/SugHUJT3L0OqO0MapVt7AbyJJg04+QyrHYwO9g/cozeS4DJC/PO5w5TJ+yjY1rrtqqe9wgM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GR5Fi5x+; arc=none smtp.client-ip=209.85.215.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GR5Fi5x+" Received: by mail-pg1-f193.google.com with SMTP id 41be03b00d2f7-b26d7ddbfd7so3498800a12.0; Tue, 27 May 2025 20:50:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404211; x=1749009011; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tKvUSOWXMccipDrf7DsG0TgpjJedGkWJTSZCOMyRAZY=; b=GR5Fi5x+lfDb+iWkgfkuomPSHH5pmM5KpvqPb2nj5GJ9mltJM2ptPY/2oP+IGgdUYh Jy3CrevsC3dLGOogbRJKA8Ua6X4uv9YhnPl3IRwVgj1TbCNo+/uxvPiJ5oFty/i0nG3J /IrM2SteTp2wrj99dTn1osswIQzWoA8wi6wHTQIjEuOUDMujkklDz1sw+IP57N6t305U 74CtarBnS56MOg1s4ZquAKH1fpAOIqJWkndOhNBr0rF4mqcL2EyNpRPJ5MBKaINHrKjs d615WpzeqSOTIJUHM7uQ21cGz+Wns222V4yrV6tk+rVMjOlFeEfZNB1vZPodr8dUoBwg ftjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404211; x=1749009011; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tKvUSOWXMccipDrf7DsG0TgpjJedGkWJTSZCOMyRAZY=; b=Y/KlCUu8kNGdhwf6dJR2gCztMJFbvD6/6SqDF7xMbUFykFt/wbBKNry/XDatU5iSjU y4vJLw06UxtDBtjfuUK+nqvjWXbErQBe8xFWcyVebfpI6Vx24evAYgnsyRIcOcV7ZS9I iWtvxW8iSit5fF6u68C5mZXd7WfaJ4ShZ9TYiynsIiWAIkr56lXq5haUTg3R5fQc1YRd 7tZZfUDcMCNFYTxPQjmsSFUsBRLjOEhXLDCNqwusahgOM1lS9nFL2MxN9MWyd6XPtSbs ZwML1rTs0G/JRV/joxZsHZSaWJW8HH51JfKKNTkFK5t7kcgXSSAT6kwFvlBF9fITiyoE UUyA== X-Forwarded-Encrypted: i=1; AJvYcCUYtJlAjNWw/vLGvwLZW9OLMLudkChweTj71bsqXq9xGhfDFdT3YuFNzs/ytK1A35RbQEw1wjukJ6eOSPU=@vger.kernel.org X-Gm-Message-State: AOJu0YwOm96wtApmimaTzI/U9Z6qFQ/rTxhCdD+JNPJCUiD2g8AYkVg6 AInTLVroV4EC0G5p/dwjDYslktKJx0O4BsaIpuoyjorpeADoertfZAPk X-Gm-Gg: ASbGncsLEzO5ACv3XkUoAAM/wKKG/Ol7uZE1cHWS+LO1YpuyLzO+jveqLndwtRfkNbO 4yQ++yjWSXsjnKfTHimasXiSwQhM9pJwQcPsxspLI8SngMTFsZxIkmXtTSm+eaDcphissWrr/Ac /PJ7FGvGpiTbYxyc9CH9YG/lJOo/M9/LQPx6jHg3fD2NqQvqYesAhBkJSUDU6xIU9IR0c6qoQU1 e2u8t78rZjR0MSaqWAh50iTn6EAaXOC6MG2j5/kyEAc110NHVuig0DDO5QvuNHkF9KGKJCqy0sg WmCYG/WoI7ENP2Y45eszLfifbeT/FwlpCCHlhmXMwpPDswDlljpiLQDtIq9oxEhXy2C4 X-Google-Smtp-Source: AGHT+IGG2Sc/hwcLt/vu9eqQkgCFuls5UzM2jnZ1L14nDNHAJZrPgOboZLInAAAPUefuR7XUCLG7lA== X-Received: by 2002:a17:902:cccb:b0:231:ecd5:e719 with SMTP id d9443c01a7336-23414fb5561mr294495325ad.24.1748404211034; Tue, 27 May 2025 20:50:11 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:10 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 16/25] ftrace: supporting replace direct ftrace_ops Date: Wed, 28 May 2025 11:47:03 +0800 Message-Id: <20250528034712.138701-17-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce the function replace_ftrace_direct(). This is used to replace the direct ftrace_ops for a function, and will be used in the next patch. Let's call the origin ftrace_ops A, and the new ftrace_ops B. First, we register B directly, and the callback of the functions in A and B will fallback to the ftrace_ops_list case. Then, we modify the address of the entry in the direct_functions to B->direct_call, and remove it from A. This will update the dyn_rec and make the functions call b->direct_call directly. If no function in A->filter_hash, just unregister it. So a record can have more than one direct ftrace_ops, and we need check if there is any direct ops for the record before remove the FTRACE_OPS_FL_DIRECT in __ftrace_hash_rec_update(). Signed-off-by: Menglong Dong --- include/linux/ftrace.h | 8 ++++ kernel/trace/ftrace.c | 87 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 93 insertions(+), 2 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 40727d3f125d..1d162e331e99 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -528,6 +528,9 @@ void ftrace_stub_direct_tramp(void); =20 int reset_ftrace_direct_ips(struct ftrace_ops *ops, unsigned long *ips, unsigned int cnt); +int replace_ftrace_direct(struct ftrace_ops *ops, struct ftrace_ops *src_o= ps, + unsigned long addr); + #else struct ftrace_ops; static inline unsigned long ftrace_find_rec_direct(unsigned long ip) @@ -556,6 +559,11 @@ static inline int reset_ftrace_direct_ips(struct ftrac= e_ops *ops, unsigned long { return -ENODEV; } +static inline int replace_ftrace_direct(struct ftrace_ops *ops, struct ftr= ace_ops *src_ops, + unsigned long addr) +{ + return -ENODEV; +} =20 /* * This must be implemented by the architecture. diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 5b6b74ea4c20..7f2313e4c3d9 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -1727,6 +1727,24 @@ static bool skip_record(struct dyn_ftrace *rec) !(rec->flags & FTRACE_FL_ENABLED); } =20 +static struct ftrace_ops * +ftrace_find_direct_ops_any_other(struct dyn_ftrace *rec, struct ftrace_ops= *op_exclude) +{ + struct ftrace_ops *op; + unsigned long ip =3D rec->ip; + + do_for_each_ftrace_op(op, ftrace_ops_list) { + + if (op =3D=3D op_exclude || !(op->flags & FTRACE_OPS_FL_DIRECT)) + continue; + + if (hash_contains_ip(ip, op->func_hash)) + return op; + } while_for_each_ftrace_op(op); + + return NULL; +} + /* * This is the main engine to the ftrace updates to the dyn_ftrace records. * @@ -1831,8 +1849,10 @@ static bool __ftrace_hash_rec_update(struct ftrace_o= ps *ops, * function, then that function should no longer * be direct. */ - if (ops->flags & FTRACE_OPS_FL_DIRECT) - rec->flags &=3D ~FTRACE_FL_DIRECT; + if (ops->flags & FTRACE_OPS_FL_DIRECT) { + if (!ftrace_find_direct_ops_any_other(rec, ops)) + rec->flags &=3D ~FTRACE_FL_DIRECT; + } =20 /* * If the rec had REGS enabled and the ops that is @@ -6033,6 +6053,69 @@ int register_ftrace_direct(struct ftrace_ops *ops, u= nsigned long addr) } EXPORT_SYMBOL_GPL(register_ftrace_direct); =20 +int replace_ftrace_direct(struct ftrace_ops *ops, struct ftrace_ops *src_o= ps, + unsigned long addr) +{ + struct ftrace_hash *hash; + struct ftrace_func_entry *entry, *iter; + int err =3D -EBUSY, size, count; + + if (ops->func || ops->trampoline) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) + return -EINVAL; + if (ops->flags & FTRACE_OPS_FL_ENABLED) + return -EINVAL; + + hash =3D ops->func_hash->filter_hash; + if (ftrace_hash_empty(hash)) + return -EINVAL; + + mutex_lock(&direct_mutex); + + ops->func =3D call_direct_funcs; + ops->flags =3D MULTI_FLAGS; + ops->trampoline =3D FTRACE_REGS_ADDR; + ops->direct_call =3D addr; + + err =3D register_ftrace_function_nolock(ops); + if (err) + goto out_unlock; + + hash =3D ops->func_hash->filter_hash; + size =3D 1 << hash->size_bits; + for (int i =3D 0; i < size; i++) { + hlist_for_each_entry(iter, &hash->buckets[i], hlist) { + entry =3D __ftrace_lookup_ip(direct_functions, iter->ip); + if (!entry) { + err =3D -ENOENT; + goto out_unlock; + } + WRITE_ONCE(entry->direct, addr); + /* remove the ip from the hash, and this will make the trampoline + * be called directly. + */ + count =3D src_ops->func_hash->filter_hash->count; + if (count <=3D 1) { + if (WARN_ON_ONCE(!count)) + continue; + err =3D __unregister_ftrace_direct(src_ops, src_ops->direct_call, + true); + } else { + err =3D ftrace_set_filter_ip(src_ops, iter->ip, 1, 0); + } + if (err) + goto out_unlock; + } + } + +out_unlock: + mutex_unlock(&direct_mutex); + + return err; +} +EXPORT_SYMBOL_GPL(replace_ftrace_direct); + /** * unregister_ftrace_direct - Remove calls to custom trampoline * previously registered by register_ftrace_direct for @ops object. --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E10C21C9F6; Wed, 28 May 2025 03:50:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404216; cv=none; b=h4JDecmsd4szLk1Zgy8hg9b+OXoWZ7PNIaKHS/mp+G5FBp/K0lbAvZUl9l7iyUHzE/WhGZzkMCESUyaALuh8iWRA49DJdA1cZ/ywwrUP7rjRAgYZgLz3I2Sr8ADtcZBG+p/PttqiWHJg2p5hJfhrp9ARaYGO7dR88DTk4Wo2bO8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404216; c=relaxed/simple; bh=aApBRpTSVGFJFyeFt0SJiX2fLyAA1yK3SMrzMhlMm/A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OdYdYn2WrOg82XXVkTdOjE/8/5pPWox50TlRZa4ay/bBtKalxXUVpiCxuFZBWkLMTTAY2Jrp3eVEhPvJDmPEHcc9PmUMkUDlzJ6WfiY9X9UX5fIWvFzDEU7iybvXvFUe/qvQc6Mm4VgURcpUcqlhkdH4/Q1S7PobSyCUDDx3TfM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aRVPTMrS; arc=none smtp.client-ip=209.85.214.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aRVPTMrS" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-234b440afa7so8923315ad.0; Tue, 27 May 2025 20:50:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404213; x=1749009013; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PMw9XBLwB3ktFFUUL8YNkGDIrwciXTJUnDt5CzTt340=; b=aRVPTMrSZkwiXEi/tfYZ8W0LHXmHsnN/3TGh8Z7H7Ra3tAhYN1zgPnSqAvErcTpOX9 erGi4wwiYJyJQgOQLbnB1YIPL5Cl5LsoZkmfmkHqENeKlGKYF+c1YABXc0guqB5uzQrt A+68mWUQQae2TkFfa8kCUNZs4Oc7mJhaJJ1iOEtoUXQER9C57MpcQl+3Hvq5R1XDF1zN 2TdcpTF25tgv0Vukdmvpa1MaJ6YT+GYXr56dDP8rMsjD8Il7lpe22Y0R9Y/M9kXZEJwJ HHUr3sk1rEkZQWCXHUPxGxLyBuwbi2qEMLirf81vlU8ly2NkncR0nHqSvcVnzxm722k4 GAkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404213; x=1749009013; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PMw9XBLwB3ktFFUUL8YNkGDIrwciXTJUnDt5CzTt340=; b=ulzJw9sxLXHMGDywAo2hkTYtC0fzL5D8VlMW3zve6oi0fwwaYpLc/nQ7dPC+4j/3dL hBptNuC04Ln9W+O4t1tkSvNELt57Frjp3aJOQsDtTD+zmSq04t4pRv/udlqd7i0bJ7eo XCmFgiSNWoWePCPw54NWLtaY9PvKSYPlk+z07MkD/13jjHGVOdmZHKCKxGwIfhjO2eat BD+Shi5SI4a4XIW3/r8fHpioRO5VndZPy6xF0CBS3A3/OceEhnslcZusrqDyRVlKc7Ej aopEvB8iXjyiL6MlBNPA0cUvMUHCJHSuhJuQDGs5nB9/8lrwOuoj8qPuuy4Hk4eVWQ/h qe7g== X-Forwarded-Encrypted: i=1; AJvYcCV2QLR/VVbRJuwnrJNWVtTB9SPQEv6+T5neqx4tja43zxepeDHv5CbupRN9g50k6wRtJ4RDbvFD/cSnTP8=@vger.kernel.org X-Gm-Message-State: AOJu0YyanT+roGAYA7aIdPzqyIYDZVbGBHrB/QF7v8uFtOiCF/yzuM2F lvoy4DkZdFD3479dk9CRcQDz3A9Zc5uwFVse2EQBE5OfrTrJVJPhokUyCwQLLfk2 X-Gm-Gg: ASbGncvo7wLb+8PtKRyGgqHkDsAcaZr4voA/5F5LAETCJ743f5UON527ESUZHmnpFq3 5NM0LW/qQWsgEnp/MEp1tFV8P0WRw1RWbe3V/SORSJA/pOJGIZpriu+IBRjm8oYcP6Rb/7KwC54 +S7PweCSOJDOvdccTdufQoYxdtcxb3hgo8VB8BPipKs5dv+FEaieEo5ox1lyFjIXe6KKjRHlYmI 1JOShiiAH1k8uGPI/JXLTVoCX9r/Z3W5x3kVwERu5hzac36Sb9y7fRWC5fhNb4Oe8NTIBxiGxZw bD4rGn1Nrvv4xjfAoiDFwEmA363n/Wtbt+2pD05sv52vYewoKJLPUZKVsijzH3uWDgZ3 X-Google-Smtp-Source: AGHT+IEhvXf0piOAs0CO8nTX0yvduSmqoZK1c/X1qcWFu8WrWXKrIACVSBz3ph9fBjE8NSK62i9qGA== X-Received: by 2002:a17:902:cccb:b0:231:ecd5:e719 with SMTP id d9443c01a7336-23414fb5561mr294497265ad.24.1748404213253; Tue, 27 May 2025 20:50:13 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:12 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 17/25] bpf: make trampoline compatible with global trampoline Date: Wed, 28 May 2025 11:47:04 +0800 Message-Id: <20250528034712.138701-18-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, the bpf global trampoline can't work together with trampoline. For example, we will fail on attaching the FENTRY_MULTI to the functions that FENTRY exists, and FENTRY will also fail if FENTRY_MULTI exists. We make the global trampoline work together with trampoline in this commit. It is not easy. The most difficult part is synchronization between bpf_gtrampoline_link_prog and bpf_trampoline_link_prog, and we use a rw_semaphore here, which is quite ugly. We hold the write lock in bpf_gtrampoline_link_prog and read lock in bpf_trampoline_link_prog. We introduce the function bpf_gtrampoline_link_tramp() to make bpf_gtramp_link fit bpf_trampoline, which will be called in bpf_gtrampoline_link_prog(). If the bpf_trampoline of the function exist in the kfunc_md or we find it with bpf_trampoline_lookup_exist(), it means that we need do the fitting. The fitting is simple, we create a bpf_shim_tramp_link for our prog and link it to the bpf_trampoline with __bpf_trampoline_link_prog(). It's a little complex for the bpf_trampoline_link_prog() case. We create bpf_shim_tramp_link for all the bpf progs in kfunc_md and add it to the bpf_trampoline before we call __bpf_trampoline_link_prog() in bpf_gtrampoline_replace(). And we will fallback in bpf_gtrampoline_replace_finish() if error is returned by __bpf_trampoline_link_prog(). In __bpf_gtrampoline_unlink_prog(), we will call bpf_gtrampoline_remove() to release the bpf_shim_tramp_link, and the bpf prog will be unlinked if it is ever linked successfully in bpf_link_free(). Another solution is to fit into the existing trampoline. For example, we can add the bpf prog to the kfunc_md if tracing_multi bpf prog is attached on the target function when we attach a tracing bpf prog. And we can also update the tracing_multi prog to the trampoline if tracing prog exists on the target function. I think this will make the compatibility much easier. The code in this part is very ugly and messy, and I think it will be a liberation to split it out to another series :/ Signed-off-by: Menglong Dong --- include/linux/bpf.h | 6 + include/linux/kfunc_md.h | 2 + kernel/bpf/syscall.c | 2 +- kernel/bpf/trampoline.c | 291 +++++++++++++++++++++++++++++++++++++-- kernel/trace/kfunc_md.c | 9 +- 5 files changed, 293 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7191ad25d519..0f4605be87fc 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1173,6 +1173,11 @@ struct btf_func_model { */ #define BPF_TRAMP_F_INDIRECT BIT(8) =20 +/* Indicate that bpf global trampoline is also used on this function and + * the trampoline is replacing it. + */ +#define BPF_TRAMP_F_REPLACE BIT(9) + /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. */ @@ -2554,6 +2559,7 @@ void bpf_link_put(struct bpf_link *link); int bpf_link_new_fd(struct bpf_link *link); struct bpf_link *bpf_link_get_from_fd(u32 ufd); struct bpf_link *bpf_link_get_curr_or_next(u32 *id); +void bpf_link_free(struct bpf_link *link); =20 void bpf_token_inc(struct bpf_token *token); void bpf_token_put(struct bpf_token *token); diff --git a/include/linux/kfunc_md.h b/include/linux/kfunc_md.h index f1b1012eeab2..956e16f96d82 100644 --- a/include/linux/kfunc_md.h +++ b/include/linux/kfunc_md.h @@ -29,6 +29,8 @@ struct kfunc_md { #endif unsigned long func; struct kfunc_md_tramp_prog *bpf_progs[BPF_TRAMP_MAX]; + /* fallback case, there is already a trampoline on this function */ + struct bpf_trampoline *tramp; #ifdef CONFIG_FUNCTION_METADATA /* the array is used for the fast mode */ struct kfunc_md_array *array; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0cd989381128..c1c92c2b2cfc 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3184,7 +3184,7 @@ static void bpf_link_defer_dealloc_mult_rcu_gp(struct= rcu_head *rcu) } =20 /* bpf_link_free is guaranteed to be called from process context */ -static void bpf_link_free(struct bpf_link *link) +void bpf_link_free(struct bpf_link *link) { const struct bpf_link_ops *ops =3D link->ops; =20 diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index b92d1d4f1033..81b62aae9faf 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -14,6 +14,7 @@ #include #include #include +#include =20 /* dummy _ops. The verifier will operate on target program's ops. */ const struct bpf_verifier_ops bpf_extension_verifier_ops =3D { @@ -142,20 +143,44 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } =20 -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +static struct bpf_trampoline *__bpf_trampoline_lookup_exist(u64 key) { struct bpf_trampoline *tr; struct hlist_head *head; - int i; =20 - mutex_lock(&trampoline_mutex); head =3D &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; hlist_for_each_entry(tr, head, hlist) { if (tr->key =3D=3D key) { refcount_inc(&tr->refcnt); - goto out; + return tr; } } + + return NULL; +} + +static struct bpf_trampoline *bpf_trampoline_lookup_exist(u64 key) +{ + struct bpf_trampoline *tr; + + mutex_lock(&trampoline_mutex); + tr =3D __bpf_trampoline_lookup_exist(key); + mutex_unlock(&trampoline_mutex); + + return tr; +} + +static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +{ + struct bpf_trampoline *tr; + struct hlist_head *head; + int i; + + mutex_lock(&trampoline_mutex); + tr =3D __bpf_trampoline_lookup_exist(key); + if (tr) + goto out; + tr =3D kzalloc(sizeof(*tr), GFP_KERNEL); if (!tr) goto out; @@ -172,6 +197,7 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64= key) =20 tr->key =3D key; INIT_HLIST_NODE(&tr->hlist); + head =3D &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; hlist_add_head(&tr->hlist, head); refcount_set(&tr->refcnt, 1); mutex_init(&tr->mutex); @@ -228,7 +254,11 @@ static int register_fentry(struct bpf_trampoline *tr, = void *new_addr) =20 if (tr->func.ftrace_managed) { ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); - ret =3D register_ftrace_direct(tr->fops, (long)new_addr); + if (tr->flags & BPF_TRAMP_F_REPLACE) + ret =3D replace_ftrace_direct(tr->fops, global_tr.fops, + (long)new_addr); + else + ret =3D register_ftrace_direct(tr->fops, (long)new_addr); } else { ret =3D bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); } @@ -236,6 +266,17 @@ static int register_fentry(struct bpf_trampoline *tr, = void *new_addr) return ret; } =20 +static int +bpf_trampoline_get_count(const struct bpf_trampoline *tr) +{ + int count =3D 0; + + for (int kind =3D 0; kind < BPF_TRAMP_MAX; kind++) + count +=3D tr->progs_cnt[kind]; + + return count; +} + static struct bpf_tramp_links * bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool= *ip_arg) { @@ -608,15 +649,173 @@ static int __bpf_trampoline_link_prog(struct bpf_tra= mp_link *link, return err; } =20 +static int bpf_gtrampoline_get_link(struct bpf_trampoline *tr, struct bpf_= prog *prog, + u64 cookie, int kind, + struct bpf_shim_tramp_link **link) +{ + struct bpf_shim_tramp_link *__link; + + __link =3D kzalloc(sizeof(*__link), GFP_KERNEL); + if (!__link) + return -ENOMEM; + + __link->link.cookie =3D cookie; + + bpf_link_init(&__link->link.link, BPF_LINK_TYPE_UNSPEC, + &bpf_shim_tramp_link_lops, prog); + + /* the bpf_shim_tramp_link will hold a reference on the prog and tr */ + refcount_inc(&tr->refcnt); + bpf_prog_inc(prog); + *link =3D __link; + + return 0; +} + +static struct bpf_tramp_link * +bpf_gtrampoline_find_link(struct bpf_trampoline *tr, struct bpf_prog *prog) +{ + struct bpf_tramp_link *link; + + for (int kind =3D 0; kind < BPF_TRAMP_MAX; kind++) { + hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { + if (link->link.prog =3D=3D prog) + return link; + } + } + + return NULL; +} + +static int bpf_gtrampoline_remove(struct bpf_trampoline *tr, struct bpf_pr= og *prog, + bool remove_list) +{ + struct bpf_shim_tramp_link *slink; + int kind; + + slink =3D (struct bpf_shim_tramp_link *)bpf_gtrampoline_find_link(tr, pro= g); + if (WARN_ON_ONCE(!slink)) + return -EINVAL; + + if (!slink->trampoline && remove_list) { + kind =3D bpf_attach_type_to_tramp(prog); + hlist_del_init(&slink->link.tramp_hlist); + tr->progs_cnt[kind]--; + } + bpf_link_free(&slink->link.link); + + return 0; +} + +static int bpf_gtrampoline_replace(struct bpf_trampoline *tr) +{ + struct kfunc_md_tramp_prog *progs; + struct bpf_shim_tramp_link *link; + struct kfunc_md *md; + int err =3D 0, count; + + kfunc_md_lock(); + md =3D kfunc_md_get((unsigned long)tr->func.addr); + if (!md || md->tramp) { + kfunc_md_put_entry(md); + kfunc_md_unlock(); + return 0; + } + kfunc_md_unlock(); + + rcu_read_lock(); + md =3D kfunc_md_get_noref((unsigned long)tr->func.addr); + if (!md || md->tramp) + goto on_fail; + + count =3D bpf_trampoline_get_count(tr); + /* we are attaching a new link, so +1 here */ + count +=3D md->bpf_prog_cnt + 1; + if (count > BPF_MAX_TRAMP_LINKS) { + err =3D -E2BIG; + goto on_fail; + } + + for (int kind =3D 0; kind < BPF_TRAMP_MAX; kind++) { + progs =3D md->bpf_progs[kind]; + while (progs) { + err =3D bpf_gtrampoline_get_link(tr, progs->prog, progs->cookie, + kind, &link); + if (err) + goto on_fail; + + hlist_add_head(&link->link.tramp_hlist, &tr->progs_hlist[kind]); + tr->progs_cnt[kind]++; + progs =3D progs->next; + link->trampoline =3D tr; + } + } + + tr->flags |=3D BPF_TRAMP_F_REPLACE; + rcu_read_unlock(); + + return 0; + +on_fail: + kfunc_md_put_entry(md); + rcu_read_unlock(); + + return err; +} + +static void bpf_gtrampoline_replace_finish(struct bpf_trampoline *tr, int = err) +{ + struct kfunc_md_tramp_prog *progs; + struct kfunc_md *md; + + if (!(tr->flags & BPF_TRAMP_F_REPLACE)) + return; + + kfunc_md_lock(); + md =3D kfunc_md_get_noref((unsigned long)tr->func.addr); + /* this shouldn't happen, as the md->tramp can only be set with + * global_tr_lock. + */ + if (WARN_ON_ONCE(!md || md->tramp)) + return; + + if (err) { + for (int kind =3D 0; kind < BPF_TRAMP_MAX; kind++) { + progs =3D md->bpf_progs[kind]; + while (progs) { + /* the progs is already added to trampoline + * and we need clean it on this case. + */ + bpf_gtrampoline_remove(tr, progs->prog, true); + progs =3D progs->next; + } + } + } else { + md->tramp =3D tr; + } + + kfunc_md_put_entry(md); + kfunc_md_unlock(); +} + int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr, struct bpf_prog *tgt_prog) { int err; =20 - mutex_lock(&tr->mutex); - err =3D __bpf_trampoline_link_prog(link, tr, tgt_prog); - mutex_unlock(&tr->mutex); + down_read(&global_tr_lock); + + err =3D bpf_gtrampoline_replace(tr); + if (!err) { + mutex_lock(&tr->mutex); + err =3D __bpf_trampoline_link_prog(link, tr, tgt_prog); + mutex_unlock(&tr->mutex); + } + + bpf_gtrampoline_replace_finish(tr, err); + up_read(&global_tr_lock); + return err; } =20 @@ -745,7 +944,7 @@ int bpf_gtrampoline_unlink_prog(struct bpf_gtramp_link = *link) kfunc_md_lock(); for (int i =3D 0; i < link->entry_cnt; i++) { md =3D kfunc_md_get_noref((long)link->entries[i].addr); - if (WARN_ON_ONCE(!md)) + if (WARN_ON_ONCE(!md) || md->tramp) continue; =20 md->flags |=3D KFUNC_MD_FL_BPF_REMOVING; @@ -761,13 +960,65 @@ int bpf_gtrampoline_unlink_prog(struct bpf_gtramp_lin= k *link) return err; } =20 +static int bpf_gtrampoline_link_tramp(struct bpf_gtramp_link_entry *entry, + struct bpf_prog *prog) +{ + struct bpf_trampoline *tr, *new_tr =3D NULL; + struct bpf_shim_tramp_link *slink =3D NULL; + struct kfunc_md *md; + int err, kind; + u64 key; + + kfunc_md_lock(); + md =3D kfunc_md_get_noref((long)entry->addr); + kind =3D bpf_attach_type_to_tramp(prog); + if (!md->tramp) { + key =3D bpf_trampoline_compute_key(NULL, entry->attach_btf, + entry->btf_id); + new_tr =3D bpf_trampoline_lookup_exist(key); + md->tramp =3D new_tr; + } + + /* check if we need to be replaced by trampoline */ + tr =3D md->tramp; + kfunc_md_unlock(); + if (!tr) + return 0; + + mutex_lock(&tr->mutex); + err =3D bpf_gtrampoline_get_link(tr, prog, entry->cookie, kind, &slink); + if (err) + goto err_out; + + err =3D __bpf_trampoline_link_prog(&slink->link, tr, NULL); + if (err) + goto err_out; + mutex_unlock(&tr->mutex); + + bpf_trampoline_put(new_tr); + /* this can only be set on the link success */ + slink->trampoline =3D tr; + tr->flags |=3D BPF_TRAMP_F_REPLACE; + + return 0; +err_out: + mutex_unlock(&tr->mutex); + + bpf_trampoline_put(new_tr); + if (slink) { + bpf_trampoline_put(tr); + bpf_link_free(&slink->link.link); + } + return err; +} + int bpf_gtrampoline_link_prog(struct bpf_gtramp_link *link) { struct bpf_gtramp_link_entry *entry; enum bpf_tramp_prog_type kind; struct bpf_prog *prog; struct kfunc_md *md; - bool update =3D false; + bool update =3D false, linked; int err =3D 0, i; =20 prog =3D link->link.prog; @@ -785,6 +1036,7 @@ int bpf_gtrampoline_link_prog(struct bpf_gtramp_link *= link) * lock instead. */ kfunc_md_lock(); + linked =3D false; md =3D kfunc_md_create((long)entry->addr, entry->nr_args); if (md) { /* the function is not in the filter hash of gtr, @@ -793,16 +1045,27 @@ int bpf_gtrampoline_link_prog(struct bpf_gtramp_link= *link) if (!md->bpf_prog_cnt) update =3D true; err =3D kfunc_md_bpf_link(md, prog, kind, entry->cookie); + if (!err) + linked =3D true; } else { err =3D -ENOMEM; } + kfunc_md_unlock(); =20 - if (err) { - kfunc_md_put_entry(md); - kfunc_md_unlock(); - goto on_fallback; + if (!err) { + err =3D bpf_gtrampoline_link_tramp(entry, prog); + if (!err) + continue; } + + /* on error case, fallback the md and previous */ + kfunc_md_lock(); + md =3D kfunc_md_get_noref((long)entry->addr); + if (linked) + kfunc_md_bpf_unlink(md, prog, kind); + kfunc_md_put_entry(md); kfunc_md_unlock(); + goto on_fallback; } =20 if (update) { diff --git a/kernel/trace/kfunc_md.c b/kernel/trace/kfunc_md.c index ebb4e46d482d..5d61a8be3768 100644 --- a/kernel/trace/kfunc_md.c +++ b/kernel/trace/kfunc_md.c @@ -141,7 +141,8 @@ static int kfunc_md_hash_bpf_ips(void **ips) for (i =3D 0; i < (1 << KFUNC_MD_HASH_BITS); i++) { head =3D &kfunc_md_table[i]; hlist_for_each_entry(md, head, hash) { - if (md->bpf_prog_cnt > !!(md->flags & KFUNC_MD_FL_BPF_REMOVING)) + if (md->bpf_prog_cnt > !!(md->flags & KFUNC_MD_FL_BPF_REMOVING) && + !md->tramp) ips[c++] =3D (void *)md->func; } } @@ -472,7 +473,8 @@ static int kfunc_md_fast_bpf_ips(void **ips) =20 for (i =3D 0; i < kfunc_mds->kfunc_md_count; i++) { md =3D &kfunc_mds->mds[i]; - if (md->users && md->bpf_prog_cnt > !!(md->flags & KFUNC_MD_FL_BPF_REMOV= ING)) + if (md->users && md->bpf_prog_cnt > !!(md->flags & KFUNC_MD_FL_BPF_REMOV= ING) && + !md->tramp) ips[c++] =3D (void *)md->func; } return c; @@ -662,6 +664,9 @@ int kfunc_md_bpf_unlink(struct kfunc_md *md, struct bpf= _prog *prog, int type) !md->bpf_progs[BPF_TRAMP_MODIFY_RETURN]) md->flags &=3D ~KFUNC_MD_FL_TRACING_ORIGIN; =20 + if (!md->bpf_prog_cnt) + md->tramp =3D NULL; + return 0; } =20 --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CF9D21C9F9; Wed, 28 May 2025 03:50:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404217; cv=none; b=BDuA29gYf89VaMiNqFN85XauTRWKCK4YTL1yiNop6Dika8y1wd4htBq5KAtY8Zd6ZgkLhLMdGeBTqgbgMcLKlb+Z9F7ZBwdZf/GrxGkCOuyzzFnraj0SbO/lTixXekVvGGDl/RBQQnIDfF5NmndWPRfNdmtB0yfGwige+lP2d3Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404217; c=relaxed/simple; bh=OA+FxIGlKhg819WfsELB4yUlJn708Xx4wh4sMB6f6gU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LA6ESrQD4iiMNRW11HG6wrIDbrAYVkcpuXpA7Ojgb7OgUT2AlKqYLxnd0h/QwYPJDcksamqolyDHGNhdx3/US7pl1EuhNE24i1vusX1eMTPD20w+7QeWdkdFWyT79BrLoUaPX6DqXHK3PHtx8W6/aOA0K5WTv/k5LOuLQgLWD8Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TAo2jUyj; arc=none smtp.client-ip=209.85.215.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TAo2jUyj" Received: by mail-pg1-f196.google.com with SMTP id 41be03b00d2f7-b0b2d0b2843so2412935a12.2; Tue, 27 May 2025 20:50:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404215; x=1749009015; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WffP/cSbyBFMF+/uXvI9bocvXjk7B9YFHd0hp/gMMUk=; b=TAo2jUyj0CofIR7R/EO1PIYcH5b9CdUg5UWROkNV3b+lxFtiRdgjodP9gSDVU2NrBo C4F9dRMWg7c6vLFiRLjttCoSZzmC/hKG9V4yz+f0zqFaMCniYfX5i83M7QhWH9iSdW+u g1T2tt8qM3FZNrsj+L2bWVZUQYgqhBCW9ED4vq07pY1SqQEqbpRB263Jcfs2Zz4JAtJT kGYc4f1CnSyxlhJP5eho27BMr+xvUbPvoag+G9YBwo6H5ISjpzfz5at+aiBohxtIhFOX hA69ANp3EMt5qcVh8JW8riYTfhVu9XZLGvF3k4szMDzkHb1iSz6oii1HhB4avzr220p+ rHXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404215; x=1749009015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WffP/cSbyBFMF+/uXvI9bocvXjk7B9YFHd0hp/gMMUk=; b=IcccYVw8Fi6RntT+YBM7Gf/ANoIjgoXgF3x6VJXxCgGdVqQlRKZ5mo4mDlL7jRjtZR 5zu+hxq4sGnQ50DIdVywtzSdzz311XxddvA9qzQtqkkmm3koBcqG4OVwKfFE80QNPicE Bg030dUC32kCvMdgkh6Snk6zax6Op4ZqTOeHWXnhtoJjU+JF7m4of/eoorDottUrgflf NplFh4IQWFbplewxSbMfKr66P8g8yjB9OQXdy3W3ocz8eH+aAI+7NUURcF5UjiLCOBB7 XUF6D4ziDHUGjeu+WOQ312YvdWxNLPwVWyP6u33F3gSjQ1fx7RgMGzrzei2psG70oPp2 Tokg== X-Forwarded-Encrypted: i=1; AJvYcCUKQblZlCmnICR3oANA63NCvgKaKuAK9vzRrcjHuSaqkSymlSxxS3AqzUKEyQIx4TWkNtz1vpLM1In926s=@vger.kernel.org X-Gm-Message-State: AOJu0YwfKd/09694KEsXC62+MKQ8tfIUkNVmlU5WevdnLZElDzbOSdZA TsWgVP3qlEbg5wr+bye6+0VHNhVcQEpRQJrHjjYh+EJlcfiCAT2u0K04 X-Gm-Gg: ASbGncu86K2l6nK5S+9PCBMZEvuj1KxAiFUmklFw90mH9TxPAml1t1svqLedLIIsow2 plIQVEDQzQeu/OT+Y5va5jAl8uJZxYtd+aCuZ57crN4jt/60nc8GWAGYKtX4qwfoKNc5jqSE8z3 JuqZxefRZ8fqwAg8+XYa/uWS1hqjZz/t1E6bANCX5Ro85XNmF8v1aeTux4oKSkFCOWcvGDYoJ9+ DXKkPFePRIa9Vcd2qa0WuMTTxhhIqP7oRBIXKrX0mj6S8GV1MVKMJ2OHA929BdGKYOuYgqbi+WV HGno7N8TuUhG5NRAvtyY+sBak39zd8rhRUmClAop/dKaF2xu5CW7AX39vfAGZA2b8B1d X-Google-Smtp-Source: AGHT+IEO4NrcYqigTspQOA9dpUSJowPXjoJZkbYF2Lxs7j/SRyuV59rn8R0FzF4o1MHWYhKhpnLV/w== X-Received: by 2002:a17:902:ecd2:b0:223:7006:4db2 with SMTP id d9443c01a7336-234d2adc24amr10290065ad.31.1748404215441; Tue, 27 May 2025 20:50:15 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:15 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 18/25] libbpf: don't free btf if tracing_multi progs existing Date: Wed, 28 May 2025 11:47:05 +0800 Message-Id: <20250528034712.138701-19-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" By default, the kernel btf that we load during loading program will be freed after the programs are loaded in bpf_object_load(). However, we still need to use these btf for tracing of multi-link during attaching. Therefore, we don't free the btfs until the bpf object is closed if any bpf programs of the type multi-link tracing exist. Meanwhile, introduce the new api bpf_object__free_btf() to manually free the btfs after attaching. Signed-off-by: Menglong Dong --- tools/lib/bpf/libbpf.c | 16 +++++++++++++++- tools/lib/bpf/libbpf.h | 2 ++ tools/lib/bpf/libbpf.map | 1 + 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index e9c641a2fb20..cfe81e1640d8 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -8581,6 +8581,20 @@ static void bpf_object_post_load_cleanup(struct bpf_= object *obj) obj->btf_vmlinux =3D NULL; } =20 +static void bpf_object_early_free_btf(struct bpf_object *obj) +{ + struct bpf_program *prog; + + bpf_object__for_each_program(prog, obj) { + if (prog->expected_attach_type =3D=3D BPF_TRACE_FENTRY_MULTI || + prog->expected_attach_type =3D=3D BPF_TRACE_FEXIT_MULTI || + prog->expected_attach_type =3D=3D BPF_MODIFY_RETURN_MULTI) + return; + } + + bpf_object_post_load_cleanup(obj); +} + static int bpf_object_prepare(struct bpf_object *obj, const char *target_b= tf_path) { int err; @@ -8652,7 +8666,7 @@ static int bpf_object_load(struct bpf_object *obj, in= t extra_log_level, const ch err =3D bpf_gen__finish(obj->gen_loader, obj->nr_programs, obj->nr_maps= ); } =20 - bpf_object_post_load_cleanup(obj); + bpf_object_early_free_btf(obj); obj->state =3D OBJ_LOADED; /* doesn't matter if successfully or not */ =20 if (err) { diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index d39f19c8396d..ded98e8cf327 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -323,6 +323,8 @@ LIBBPF_API struct bpf_program * bpf_object__find_program_by_name(const struct bpf_object *obj, const char *name); =20 +LIBBPF_API void bpf_object__free_btfs(struct bpf_object *obj); + LIBBPF_API int libbpf_prog_type_by_name(const char *name, enum bpf_prog_type *prog_type, enum bpf_attach_type *expected_attach_type); diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 1205f9a4fe04..23df00ae0b73 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -415,6 +415,7 @@ LIBBPF_1.4.0 { bpf_token_create; btf__new_split; btf_ext__raw_data; + bpf_object__free_btfs; } LIBBPF_1.3.0; =20 LIBBPF_1.5.0 { --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D58721CC7F; Wed, 28 May 2025 03:50:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.193 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404220; cv=none; b=t1ZKuQt7nbKRgUnPnyw4BwmxYScl+tA2Nt9YmfTvmuE9174mRy1ON7Oi/MD0pPEQSxkPCj7n9miGYIZoNBLihGuhWdxHQ3Fwa2pj7HgorAHAq0kkj0SM1LzilXsWLraISaifMOrGzu0iwQFOl+izjjX2fwzp94radL+KjwbL5xM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404220; c=relaxed/simple; bh=5V+8heoQF3gSTeQDws/G0dOXiPgoppADqa7baGsuzPE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NihKajXar4IHL2/Gvmj1ILFp07H4q5vo9DzyOhK7E+UFbd529mQ9m6pYQlW3tabOOmLzL4vuu/HynEmCfVCKVN9C94EUHtJkkQbl3Fsd8Bo/CjwrKYWD5CGwZErTODjip2zfI6cWUl/cIbozTHJyqe0VVv4mca6aLP6sdX9eOoQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EyxBi+0r; arc=none smtp.client-ip=209.85.214.193 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EyxBi+0r" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-234488c2ea6so28143715ad.3; Tue, 27 May 2025 20:50:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404218; x=1749009018; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OOAklLICFXXiJE09qqPApCXuoD1jwY/692BPEgyWx/I=; b=EyxBi+0rJv/PI4VAbkIVz3xeOa3V2+UEcV8dILYmAGWeyFD2+3bOO4DsOrfC0d/SYB Nzxq6OK8x71sdEmDeKntd791VCW+sViBu95q3eb3IkFL0Y1BB0oNwDAlG7jgWRSmH4lA hNLZHQkL443p9MVNwBB+gqTKd9z1ZGp/9ldt3dpKIKM0tV23T7SpQD02wuC+eGB6+2en ZgDSJf8qcwVjCSqOfvlfu2GKSvpwfvkSXjh06yS6/Le1pmdrtxJN9WEOqKUbvQxfSXyR IYZbdseWa6B6w6o7OtjNuMhaRs/kTQrmjnJsHEulkrPRDi+58zMt7Vt69rk3qNjg18Jt uHGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404218; x=1749009018; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OOAklLICFXXiJE09qqPApCXuoD1jwY/692BPEgyWx/I=; b=LHTwwSAnJZktsrtI4j1Vch/ezRaKbkUom6tjDJfZKzoA9aBnRe1Ms3oWG60OUJk0zs 2BidjIE+U8fb/g3YEATGIhC85nZHQbip9VwJ0LlDbeGsysUQALiRvGhIFzZwQYSStWEs UiLZnaapE9iAf3Hmpbw2eXvYhaqKSJ2jGAFUoIRzEY34OlyxVEUoQBl+1GhaAdP1SmyU ab9Y2h4qb8Sak1fEJt5WmXKdPB+maFC3KnUEXxp4PGG7u29nlCCKan0aDx/pPxAL9l08 aythrYyCil12xtTD1zXvqdDA20ucTAxnm/+GvlN8SAqXtzALKyDDWWsiIssutkLXThtS V86w== X-Forwarded-Encrypted: i=1; AJvYcCUvsqDa8XI3Edqxj+IiZzxT9DFE/ED+X5AMHdNizjBOQjk1bK6Ho0/0NnavqHOF8LPZT9NrqwN7uga9pCs=@vger.kernel.org X-Gm-Message-State: AOJu0YxrM1EM7aRf6/zcn990nlp05k5j3d3Ar0jH+2c86pWJpwrcJzM3 OrmQFcWQmUpp/tESIOI94TH/TYVznw6f7lFKRNMhOZALBsMUUtSh6dX9 X-Gm-Gg: ASbGncuTI3OpJ2SD6Lq9JKp2FGDr7OgsBVyCvnQF6aKjkyGkogCuKDJDil7p0fu12gF 8VvTQ+Ar8n2fr1cUcDm4MQwN+23z83ROnZjONqlHxRoKv+3xsY/j2HNLBoH0foxIfL/wlPP8maP h3bQaPNYsEqLteflb5EU9EWWPmJM96Uv9eLNYrGnndtTghwX1/jKeS6C5QvZI4hwE0K+QuWVobq ouWcSyIc6RqaUMEpE/ARtUlVrVYxQa5J1Fk9fY/YCXGy7XVp8SzGDqlv92fGXd8wmENPkWgO9bi VClr72SKG3/5b+nOK0fB9/v2AhQLo7vWEdHMde2nvYlNrdIoKWu186zJtGjv/vBPn3dP X-Google-Smtp-Source: AGHT+IFjRDfKk14PTcCXJFZU5cazoUTE4t7ucC5c5KcvLGRVl1mcIVWu47Z++9kntaSV5DajC+EV3Q== X-Received: by 2002:a17:902:e807:b0:22e:50fa:50d6 with SMTP id d9443c01a7336-23414fb1934mr238367355ad.37.1748404217643; Tue, 27 May 2025 20:50:17 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:17 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 19/25] libbpf: support tracing_multi Date: Wed, 28 May 2025 11:47:06 +0800 Message-Id: <20250528034712.138701-20-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add supporting for the attach types of: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI Signed-off-by: Menglong Dong --- tools/bpf/bpftool/common.c | 3 + tools/lib/bpf/bpf.c | 10 +++ tools/lib/bpf/bpf.h | 6 ++ tools/lib/bpf/libbpf.c | 168 ++++++++++++++++++++++++++++++++++++- tools/lib/bpf/libbpf.h | 19 +++++ tools/lib/bpf/libbpf.map | 1 + 6 files changed, 204 insertions(+), 3 deletions(-) diff --git a/tools/bpf/bpftool/common.c b/tools/bpf/bpftool/common.c index ecfa790adc13..8e681fe3dd6b 100644 --- a/tools/bpf/bpftool/common.c +++ b/tools/bpf/bpftool/common.c @@ -1162,6 +1162,9 @@ const char *bpf_attach_type_input_str(enum bpf_attach= _type t) case BPF_TRACE_FENTRY: return "fentry"; case BPF_TRACE_FEXIT: return "fexit"; case BPF_MODIFY_RETURN: return "mod_ret"; + case BPF_TRACE_FENTRY_MULTI: return "fentry_multi"; + case BPF_TRACE_FEXIT_MULTI: return "fexit_multi"; + case BPF_MODIFY_RETURN_MULTI: return "mod_ret_multi"; case BPF_SK_REUSEPORT_SELECT: return "sk_skb_reuseport_select"; case BPF_SK_REUSEPORT_SELECT_OR_MIGRATE: return "sk_skb_reuseport_select_= or_migrate"; default: return libbpf_bpf_attach_type_str(t); diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index a9c3e33d0f8a..75a917de1a3c 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -797,6 +797,16 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, tracing)) return libbpf_err(-EINVAL); break; + case BPF_TRACE_FENTRY_MULTI: + case BPF_TRACE_FEXIT_MULTI: + case BPF_MODIFY_RETURN_MULTI: + attr.link_create.tracing_multi.btf_ids =3D ptr_to_u64(OPTS_GET(opts, tra= cing_multi.btf_ids, 0)); + attr.link_create.tracing_multi.tgt_fds =3D ptr_to_u64(OPTS_GET(opts, tra= cing_multi.tgt_fds, 0)); + attr.link_create.tracing_multi.cookies =3D ptr_to_u64(OPTS_GET(opts, tra= cing_multi.cookies, 0)); + attr.link_create.tracing_multi.cnt =3D OPTS_GET(opts, tracing_multi.cnt,= 0); + if (!OPTS_ZEROED(opts, tracing_multi)) + return libbpf_err(-EINVAL); + break; case BPF_NETFILTER: attr.link_create.netfilter.pf =3D OPTS_GET(opts, netfilter.pf, 0); attr.link_create.netfilter.hooknum =3D OPTS_GET(opts, netfilter.hooknum,= 0); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 777627d33d25..c279b3bc80be 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -422,6 +422,12 @@ struct bpf_link_create_opts { struct { __u64 cookie; } tracing; + struct { + __u32 cnt; + const __u32 *btf_ids; + const __u32 *tgt_fds; + const __u64 *cookies; + } tracing_multi; struct { __u32 pf; __u32 hooknum; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index cfe81e1640d8..0c4ed5d237e5 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -136,6 +136,9 @@ static const char * const attach_type_name[] =3D { [BPF_NETKIT_PEER] =3D "netkit_peer", [BPF_TRACE_KPROBE_SESSION] =3D "trace_kprobe_session", [BPF_TRACE_UPROBE_SESSION] =3D "trace_uprobe_session", + [BPF_TRACE_FENTRY_MULTI] =3D "trace_fentry_multi", + [BPF_TRACE_FEXIT_MULTI] =3D "trace_fexit_multi", + [BPF_MODIFY_RETURN_MULTI] =3D "modify_return_multi", }; =20 static const char * const link_type_name[] =3D { @@ -410,6 +413,8 @@ enum sec_def_flags { SEC_XDP_FRAGS =3D 16, /* Setup proper attach type for usdt probes. */ SEC_USDT =3D 32, + /* attachment target is multi-link */ + SEC_ATTACH_BTF_MULTI =3D 64, }; =20 struct bpf_sec_def { @@ -7417,9 +7422,9 @@ static int libbpf_prepare_prog_load(struct bpf_progra= m *prog, opts->expected_attach_type =3D BPF_TRACE_UPROBE_MULTI; } =20 - if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) { + if ((def & (SEC_ATTACH_BTF | SEC_ATTACH_BTF_MULTI)) && !prog->attach_btf_= id) { int btf_obj_fd =3D 0, btf_type_id =3D 0, err; - const char *attach_name; + const char *attach_name, *name_end; =20 attach_name =3D strchr(prog->sec_name, '/'); if (!attach_name) { @@ -7438,7 +7443,27 @@ static int libbpf_prepare_prog_load(struct bpf_progr= am *prog, } attach_name++; /* skip over / */ =20 - err =3D libbpf_find_attach_btf_id(prog, attach_name, &btf_obj_fd, &btf_t= ype_id); + name_end =3D strchr(attach_name, ','); + /* for multi-link tracing, use the first target symbol during + * loading. + */ + if ((def & SEC_ATTACH_BTF_MULTI) && name_end) { + int len =3D name_end - attach_name + 1; + char *first_tgt; + + first_tgt =3D malloc(len); + if (!first_tgt) + return -ENOMEM; + libbpf_strlcpy(first_tgt, attach_name, len); + first_tgt[len - 1] =3D '\0'; + err =3D libbpf_find_attach_btf_id(prog, first_tgt, &btf_obj_fd, + &btf_type_id); + free(first_tgt); + } else { + err =3D libbpf_find_attach_btf_id(prog, attach_name, &btf_obj_fd, + &btf_type_id); + } + if (err) return err; =20 @@ -9507,6 +9532,7 @@ static int attach_kprobe_session(const struct bpf_pro= gram *prog, long cookie, st static int attach_uprobe_multi(const struct bpf_program *prog, long cookie= , struct bpf_link **link); static int attach_lsm(const struct bpf_program *prog, long cookie, struct = bpf_link **link); static int attach_iter(const struct bpf_program *prog, long cookie, struct= bpf_link **link); +static int attach_trace_multi(const struct bpf_program *prog, long cookie,= struct bpf_link **link); =20 static const struct bpf_sec_def section_defs[] =3D { SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE), @@ -9553,6 +9579,13 @@ static const struct bpf_sec_def section_defs[] =3D { SEC_DEF("fentry.s+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF | SEC_SLE= EPABLE, attach_trace), SEC_DEF("fmod_ret.s+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF | SEC_= SLEEPABLE, attach_trace), SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEP= ABLE, attach_trace), + SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_tra= ce), + SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_ATTACH_BTF_= MULTI, attach_trace_multi), + SEC_DEF("fmod_ret.multi+", TRACING, BPF_MODIFY_RETURN_MULTI, SEC_ATTACH_B= TF_MULTI, attach_trace_multi), + SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_ATTACH_BTF_M= ULTI, attach_trace_multi), + SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_ATTACH_BT= F_MULTI | SEC_SLEEPABLE, attach_trace_multi), + SEC_DEF("fmod_ret.multi.s+", TRACING, BPF_MODIFY_RETURN_MULTI, SEC_ATTACH= _BTF_MULTI | SEC_SLEEPABLE, attach_trace_multi), + SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_ATTACH_BTF_= MULTI | SEC_SLEEPABLE, attach_trace_multi), SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace), SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, atta= ch_lsm), @@ -12787,6 +12820,135 @@ static int attach_trace(const struct bpf_program = *prog, long cookie, struct bpf_ return libbpf_get_error(*link); } =20 +struct bpf_link *bpf_program__attach_trace_multi_opts(const struct bpf_pro= gram *prog, + const struct bpf_trace_multi_opts *opts) +{ + LIBBPF_OPTS(bpf_link_create_opts, link_opts); + __u32 *btf_ids =3D NULL, *tgt_fds =3D NULL; + struct bpf_link *link =3D NULL; + char errmsg[STRERR_BUFSIZE]; + int prog_fd, pfd, cnt, err; + + if (!OPTS_VALID(opts, bpf_trace_multi_opts)) + return libbpf_err_ptr(-EINVAL); + + prog_fd =3D bpf_program__fd(prog); + if (prog_fd < 0) { + pr_warn("prog '%s': can't attach before loaded\n", prog->name); + return libbpf_err_ptr(-EINVAL); + } + + cnt =3D OPTS_GET(opts, cnt, 0); + if (opts->syms) { + int btf_obj_fd, btf_type_id, i; + + if (opts->btf_ids || opts->tgt_fds) { + pr_warn("can set both opts->syms and opts->btf_ids\n"); + return libbpf_err_ptr(-EINVAL); + } + + btf_ids =3D malloc(sizeof(*btf_ids) * cnt); + tgt_fds =3D malloc(sizeof(*tgt_fds) * cnt); + if (!btf_ids || !tgt_fds) { + err =3D -ENOMEM; + goto err_free; + } + for (i =3D 0; i < cnt; i++) { + btf_obj_fd =3D btf_type_id =3D 0; + + err =3D find_kernel_btf_id(prog->obj, opts->syms[i], + prog->expected_attach_type, &btf_obj_fd, + &btf_type_id); + if (err) + goto err_free; + btf_ids[i] =3D btf_type_id; + tgt_fds[i] =3D btf_obj_fd; + } + link_opts.tracing_multi.btf_ids =3D btf_ids; + link_opts.tracing_multi.tgt_fds =3D tgt_fds; + } else { + link_opts.tracing_multi.btf_ids =3D OPTS_GET(opts, btf_ids, 0); + link_opts.tracing_multi.tgt_fds =3D OPTS_GET(opts, tgt_fds, 0); + } + + link =3D calloc(1, sizeof(*link)); + if (!link) { + err =3D -ENOMEM; + goto err_free; + } + link->detach =3D &bpf_link__detach_fd; + + link_opts.tracing_multi.cookies =3D OPTS_GET(opts, cookies, 0); + link_opts.tracing_multi.cnt =3D cnt; + + pfd =3D bpf_link_create(prog_fd, 0, bpf_program__expected_attach_type(pro= g), &link_opts); + if (pfd < 0) { + err =3D -errno; + pr_warn("prog '%s': failed to attach: %s\n", + prog->name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg))); + goto err_free; + } + link->fd =3D pfd; + + free(btf_ids); + free(tgt_fds); + return link; +err_free: + free(btf_ids); + free(tgt_fds); + free(link); + return libbpf_err_ptr(err); +} + +static int attach_trace_multi(const struct bpf_program *prog, long cookie,= struct bpf_link **link) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + int i, err, len, cnt =3D 1; + char **syms, *buf, *name; + const char *spec; + + spec =3D strchr(prog->sec_name, '/'); + if (!spec || !*(++spec)) + return -EINVAL; + + len =3D strlen(spec) + 1; + buf =3D malloc(len); + if (!buf) + return -ENOMEM; + + libbpf_strlcpy(buf, spec, len); + for (i =3D 0; i < len; i++) { + if (buf[i] =3D=3D ',') + cnt++; + } + + syms =3D malloc(sizeof(*syms) * cnt); + if (!syms) { + err =3D -ENOMEM; + goto out_free; + } + + opts.syms =3D (const char **)syms; + opts.cnt =3D cnt; + name =3D buf; + err =3D -EINVAL; + while (name) { + if (*name =3D=3D '\0') + goto out_free; + *(syms++) =3D name; + name =3D strchr(name, ','); + if (name) + *(name++) =3D '\0'; + } + + *link =3D bpf_program__attach_trace_multi_opts(prog, &opts); + err =3D libbpf_get_error(*link); +out_free: + free(buf); + free(opts.syms); + return err; +} + static int attach_lsm(const struct bpf_program *prog, long cookie, struct = bpf_link **link) { *link =3D bpf_program__attach_lsm(prog); diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index ded98e8cf327..d7f0db7ab586 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -833,6 +833,25 @@ bpf_program__attach_xdp(const struct bpf_program *prog= , int ifindex); LIBBPF_API struct bpf_link * bpf_program__attach_freplace(const struct bpf_program *prog, int target_fd, const char *attach_func_name); +struct bpf_trace_multi_opts { + /* size of this struct, for forward/backward compatibility */ + size_t sz; + /* array of function symbols to attach */ + const char **syms; + /* array of the btf type id to attach */ + __u32 *btf_ids; + /* array of the target fds */ + __u32 *tgt_fds; + /* array of the cookies */ + __u64 *cookies; + /* number of elements in syms/btf_ids/cookies arrays */ + size_t cnt; +}; +#define bpf_trace_multi_opts__last_field cnt + +LIBBPF_API struct bpf_link * +bpf_program__attach_trace_multi_opts(const struct bpf_program *prog, + const struct bpf_trace_multi_opts *opts); =20 struct bpf_netfilter_opts { /* size of this struct, for forward/backward compatibility */ diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 23df00ae0b73..fab014528b86 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -416,6 +416,7 @@ LIBBPF_1.4.0 { btf__new_split; btf_ext__raw_data; bpf_object__free_btfs; + bpf_program__attach_trace_multi_opts; } LIBBPF_1.3.0; =20 LIBBPF_1.5.0 { --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D58521CC71; Wed, 28 May 2025 03:50:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404222; cv=none; b=CggqQuerExIoauh+l2AzaowKhzu/yhSQuuBvmfnsTSq6AUVSs3iynZDcM3HDpAn+TYXVA4XhpXwS10DPYy2CWt96UCZBZDz/M1xcm2/qMRCag1fdKErlZjh2juqWer4d8SKuAbBt0itVGTK60e7aInyOTClBpHN0u/SglmS+qy4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404222; c=relaxed/simple; bh=Cg0Y83b329LN/NLe5Rd1wETYp97M1UfTmvlXP4dlxQA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZXjFUVrJY7B/nH2/JvGPzlRjnJ8mN4xxqgKpavwogUu+3cgXDC6gyMjnGKButozb3rUI3su4snQJfbGI4mQGka1IaF3Yb5g/8+t611IVZ8DLTQ3cYhX3GZiGpCF1YLrHYaS5KJeVm4UKZ5lMXo6b4g2Aonp5gNKgMn2PMmlpfSA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cRAHr2Y/; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cRAHr2Y/" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-23035b3edf1so30877705ad.3; Tue, 27 May 2025 20:50:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404220; x=1749009020; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ppMvLXTeAVvwICb0b3sYSiEahNB4W8tGcpU/rQDfwuU=; b=cRAHr2Y/y+obMbKpBPI6GwhsvMpEWZXpmeRgyA1BFtnw+6LIaWSFJlmZeD4zMsmQ7g ZvFULvNVLVBtMKE7S/EatPiszGJBdYo4+DxO5e/QBnK+60VK3fS6+FlEtZAgdIRri6Pm E68jRhQtnsTinWvlmKNEuAP9vVIefbx7ZJaxv0Y+UBMouo2hBdNuo7iRozE8ppWGLrwo /Du20/uzLSq2hUEZ/6emFMhcjtsUtZ51UAoK4telD65GlF/VNl8NFOXlAUqwmcHktm6C yEJW0ETMakWHub5WTMSc0C1mK02Kq6OTcpICNhHxwRoCWM0lDFL2CzjAyhK0xOm5TD+r N6Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404220; x=1749009020; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ppMvLXTeAVvwICb0b3sYSiEahNB4W8tGcpU/rQDfwuU=; b=BnctoszIztFf0yKx4qE1uHrLCbRfkp3/zpdwX5XB+LsxJoPzyt0aMPdPWGNwfZ+rSG yd8W5d+tIUun3+R2yFBOOeimMaAVTOmwXB1skpgtrN9OL464MO8UZ8Vj7Dr/05tKxCVt f/hZ8M5eUxyxr42yAZiP9TXdPbkrl8e1rJAf/f+2Kx1K57hgl6yiejOPjghwek7qr9Y0 FVeZihV1izQuHXZ273qtQt2x/dwAeGPQ2Z1e4w7Id06oMi09h+MNtdiWI3jvp4XQ3hmR q1XGBWrk0h3Tr5vV8KW+uOhLyMGAAdpb5W5ia1A+efbP66YR0dkA4x+Kx2+Q/yZHYM0W d00A== X-Forwarded-Encrypted: i=1; AJvYcCUiJmLabYD98P5QOgf3vCXAa5/1FXzPcGwHM/8xrNYL8eH0mzmBDGg5XaXeq+J8UE4pR2kOGi3wCAsZoPI=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2lZadbUzaLFDkrZKf4Q6kNYl6JG4OjLSza9++U9krkjTFZ2r2 SZSzUfzr1GS6NMCtuQnw31O/xq44JM79N+INZlo7ZGi6XmPXD3nks7oO X-Gm-Gg: ASbGncuAJoBV7sxJWxgjRYqlaOnMtQ/ekjh/+dq7DagYXzO0eR/ZIEMNnTL1//d4oCv xmcDf5PEat8VZIMBNuJIYio0kahgfisGrTG+ZLfMCUGz4M7nVi8JGYNLBbSOE5FyHcFMDqMG8OJ AREzub26Z79r6g61sHdJLcflzxuROx7XxBd8E2MOaFz0V7E59czYhPQyV/MQiMHa0b2Z/TwtPY6 oM3cqhAk29Zc2XtAhHkIGTmJ2yjTPpo4xd7PZBqr97/GbHJoZKw+MfKf3b7QJw3dB9oDWkjoSBo zt+iMSULzEi2h/+nattoP4khDHDcXepBAs/If7AYQxypue9lYXZd3pj2F22Q+PMeIDY+Pd5E6TB M/wg= X-Google-Smtp-Source: AGHT+IGY4I9vvI1XK8m9Bv5+nxTB1QcaQm+Nu44QbsEkVPydM4PCaf7ATyYKIvKCS4IIHIi+e1yqAA== X-Received: by 2002:a17:903:986:b0:234:c8f6:1b03 with SMTP id d9443c01a7336-234d2c4aa08mr13167895ad.47.1748404219819; Tue, 27 May 2025 20:50:19 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:19 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 20/25] libbpf: add btf type hash lookup support Date: Wed, 28 May 2025 11:47:07 +0800 Message-Id: <20250528034712.138701-21-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For now, the libbpf find the btf type id by loop all the btf types and compare its name, which is inefficient if we have many functions to lookup. We add the "use_hash" to the function args of find_kernel_btf_id() to indicate if we should lookup the btf type id by hash. The hash table will be initialized if it has not yet. Signed-off-by: Menglong Dong --- tools/lib/bpf/btf.c | 102 +++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/btf.h | 6 +++ tools/lib/bpf/libbpf.c | 37 +++++++++++--- tools/lib/bpf/libbpf.map | 3 ++ 4 files changed, 140 insertions(+), 8 deletions(-) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index f1d495dc66bb..a0df16296a94 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -35,6 +35,7 @@ struct btf { void *raw_data; /* raw BTF data in non-native endianness */ void *raw_data_swapped; + struct hashmap *func_hash; __u32 raw_size; /* whether target endianness differs from the native one */ bool swapped_endian; @@ -131,6 +132,12 @@ struct btf { int ptr_sz; }; =20 +struct btf_type_key { + __u32 dummy; + const char *name; + int kind; +}; + static inline __u64 ptr_to_u64(const void *ptr) { return (__u64) (unsigned long) ptr; @@ -938,6 +945,100 @@ static __s32 btf_find_by_name_kind(const struct btf *= btf, int start_id, return libbpf_err(-ENOENT); } =20 +static size_t btf_hash_name(long key, void *btf) +{ + const struct btf_type *t =3D (const struct btf_type *)key; + const char *name; + + if (t->name_off > BTF_MAX_NAME_OFFSET) + name =3D ((struct btf_type_key *)key)->name; + else + name =3D btf__name_by_offset(btf, t->name_off); + + return str_hash(name); +} + +static bool btf_name_equal(long key1, long key2, void *btf) +{ + const struct btf_type *t1 =3D (const struct btf_type *)key1, + *t2 =3D (const struct btf_type *)key2; + const char *name1, *name2; + int k1, k2; + + name1 =3D btf__name_by_offset(btf, t1->name_off); + k1 =3D btf_kind(t1); + + if (t2->name_off > BTF_MAX_NAME_OFFSET) { + struct btf_type_key *t2_key =3D (struct btf_type_key *)key2; + + name2 =3D t2_key->name; + k2 =3D t2_key->kind; + } else { + name2 =3D btf__name_by_offset(btf, t2->name_off); + k2 =3D btf_kind(t2); + } + + return k1 =3D=3D k2 && strcmp(name1, name2) =3D=3D 0; +} + +__s32 btf__make_hash(struct btf *btf) +{ + __u32 i, nr_types =3D btf__type_cnt(btf); + struct hashmap *map; + + if (btf->func_hash) + return 0; + + map =3D hashmap__new(btf_hash_name, btf_name_equal, (void *)btf); + if (!map) + return libbpf_err(-ENOMEM); + + for (i =3D btf->start_id; i < nr_types; i++) { + const struct btf_type *t =3D btf__type_by_id(btf, i); + int err; + + /* only function need this */ + if (btf_kind(t) !=3D BTF_KIND_FUNC) + continue; + + err =3D hashmap__add(map, t, i); + if (err =3D=3D -EEXIST) { + pr_warn("btf type exist: name=3D%s\n", + btf__name_by_offset(btf, t->name_off)); + continue; + } + + if (err) + return libbpf_err(err); + } + + btf->func_hash =3D map; + return 0; +} + +bool btf__hash_hash(struct btf *btf) +{ + return !!btf->func_hash; +} + +int btf__find_by_func_hash(struct btf *btf, const char *type_name, __u32 k= ind) +{ + struct btf_type_key key =3D { + .dummy =3D 0xffffffff, + .name =3D type_name, + .kind =3D kind, + }; + long t; + + if (!btf->func_hash) + return -ENOENT; + + if (hashmap__find(btf->func_hash, &key, &t)) + return t; + + return -ENOENT; +} + __s32 btf__find_by_name_kind_own(const struct btf *btf, const char *type_n= ame, __u32 kind) { @@ -974,6 +1075,7 @@ void btf__free(struct btf *btf) if (btf->fd >=3D 0) close(btf->fd); =20 + hashmap__free(btf->func_hash); if (btf_is_modifiable(btf)) { /* if BTF was modified after loading, it will have a split * in-memory representation for header, types, and strings diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h index 4392451d634b..8639377a1f3b 100644 --- a/tools/lib/bpf/btf.h +++ b/tools/lib/bpf/btf.h @@ -335,6 +335,12 @@ btf_dump__dump_type_data(struct btf_dump *d, __u32 id, const void *data, size_t data_sz, const struct btf_dump_type_data_opts *opts); =20 + +LIBBPF_API __s32 btf__make_hash(struct btf *btf); +LIBBPF_API bool btf__hash_hash(struct btf *btf); +LIBBPF_API int +btf__find_by_func_hash(struct btf *btf, const char *type_name, __u32 kind); + /* * A set of helpers for easier BTF types handling. * diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 0c4ed5d237e5..4a903102e0c7 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -634,6 +634,7 @@ struct extern_desc { =20 struct module_btf { struct btf *btf; + struct hashmap *btf_name_hash; char *name; __u32 id; int fd; @@ -717,6 +718,7 @@ struct bpf_object { * it at load time. */ struct btf *btf_vmlinux; + struct hashmap *btf_name_hash; /* Path to the custom BTF to be used for BPF CO-RE relocations as an * override for vmlinux BTF. */ @@ -1004,7 +1006,7 @@ static int find_ksym_btf_id(struct bpf_object *obj, c= onst char *ksym_name, struct module_btf **res_mod_btf); =20 #define STRUCT_OPS_VALUE_PREFIX "bpf_struct_ops_" -static int find_btf_by_prefix_kind(const struct btf *btf, const char *pref= ix, +static int find_btf_by_prefix_kind(struct btf *btf, const char *prefix, const char *name, __u32 kind); =20 static int @@ -10040,7 +10042,7 @@ void btf_get_kernel_prefix_kind(enum bpf_attach_typ= e attach_type, } } =20 -static int find_btf_by_prefix_kind(const struct btf *btf, const char *pref= ix, +static int find_btf_by_prefix_kind(struct btf *btf, const char *prefix, const char *name, __u32 kind) { char btf_type_name[BTF_MAX_NAME_SIZE]; @@ -10054,6 +10056,10 @@ static int find_btf_by_prefix_kind(const struct bt= f *btf, const char *prefix, */ if (ret < 0 || ret >=3D sizeof(btf_type_name)) return -ENAMETOOLONG; + + if (btf__hash_hash(btf)) + return btf__find_by_func_hash(btf, btf_type_name, kind); + return btf__find_by_name_kind(btf, btf_type_name, kind); } =20 @@ -10126,9 +10132,9 @@ static int libbpf_find_prog_btf_id(const char *name= , __u32 attach_prog_fd, int t =20 static int find_kernel_btf_id(struct bpf_object *obj, const char *attach_n= ame, enum bpf_attach_type attach_type, - int *btf_obj_fd, int *btf_type_id) + int *btf_obj_fd, int *btf_type_id, bool use_hash) { - int ret, i, mod_len; + int ret, i, mod_len, err; const char *fn_name, *mod_name =3D NULL; =20 fn_name =3D strchr(attach_name, ':'); @@ -10139,6 +10145,11 @@ static int find_kernel_btf_id(struct bpf_object *o= bj, const char *attach_name, } =20 if (!mod_name || strncmp(mod_name, "vmlinux", mod_len) =3D=3D 0) { + if (use_hash) { + err =3D btf__make_hash(obj->btf_vmlinux); + if (err) + return err; + } ret =3D find_attach_btf_id(obj->btf_vmlinux, mod_name ? fn_name : attach_name, attach_type); @@ -10161,6 +10172,11 @@ static int find_kernel_btf_id(struct bpf_object *o= bj, const char *attach_name, if (mod_name && strncmp(mod->name, mod_name, mod_len) !=3D 0) continue; =20 + if (use_hash) { + err =3D btf__make_hash(mod->btf); + if (err) + return err; + } ret =3D find_attach_btf_id(mod->btf, mod_name ? fn_name : attach_name, attach_type); @@ -10210,7 +10226,7 @@ static int libbpf_find_attach_btf_id(struct bpf_pro= gram *prog, const char *attac } else { err =3D find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, - btf_type_id); + btf_type_id, false); } if (err) { pr_warn("prog '%s': failed to find kernel BTF type ID of '%s': %s\n", @@ -12854,11 +12870,16 @@ struct bpf_link *bpf_program__attach_trace_multi_= opts(const struct bpf_program * goto err_free; } for (i =3D 0; i < cnt; i++) { - btf_obj_fd =3D btf_type_id =3D 0; + /* only use btf type function hashmap when the count + * is big enough. + */ + bool func_hash =3D cnt > 1024; + =20 + btf_obj_fd =3D btf_type_id =3D 0; err =3D find_kernel_btf_id(prog->obj, opts->syms[i], prog->expected_attach_type, &btf_obj_fd, - &btf_type_id); + &btf_type_id, func_hash); if (err) goto err_free; btf_ids[i] =3D btf_type_id; @@ -13936,7 +13957,7 @@ int bpf_program__set_attach_target(struct bpf_progr= am *prog, return libbpf_err(err); err =3D find_kernel_btf_id(prog->obj, attach_func_name, prog->expected_attach_type, - &btf_obj_fd, &btf_id); + &btf_obj_fd, &btf_id, false); if (err) return libbpf_err(err); } diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index fab014528b86..100b14de9b22 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -445,4 +445,7 @@ LIBBPF_1.6.0 { bpf_program__line_info_cnt; btf__add_decl_attr; btf__add_type_attr; + btf__hash_hash; + btf__find_by_func_hash; + btf__make_hash; } LIBBPF_1.5.0; --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D41C821E091; Wed, 28 May 2025 03:50:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404224; cv=none; b=h8uV1m0+mzvEC9t+VHBJPDdx86whRsM12l+y/cbFaAR0qi5Dp7E2oK2YYg22/cpkNO27riaVhWBtLiAFKr8JFaoOdjZ4LNuXe8/lYlwv1a5mWVRwn/HRQhsKNUcE0plFbGWl7ziomS4XTedpw4BcKT/RWThhVmq/Rbmw/xVZqfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404224; c=relaxed/simple; bh=sJmwkTgPZfMSTWNcMK4YlEf4SczAz7/vvyliNjgTZgk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CNrLYQLxkHC1jvUsXDsTcqdmaUpn5JA0JuJWKP+6L/SFqXLKFODqHotAqzoaydeKLgVT/Wz6PUP5bJJz1j4el9VadvFhGNyeqbS/ZRne/O2C+txB3heckho2GBvlcs4tqLGiP7CX8tn1ah02PVWDaBvrcmQFABV1HuYGRJOllB4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FFFvXvF7; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FFFvXvF7" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-23035b3edf1so30877885ad.3; Tue, 27 May 2025 20:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404222; x=1749009022; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v4ULvkeWo6R+wt2v8eLeuZgd3DEWdnv8cuyh2RcE8bQ=; b=FFFvXvF7l+TwmHzDBLpn4q1IpTT3b+BeLoKzNBVVoGvqVD+6v/ZZM/4M9bhDwToplN 4ErdwlmNBpo6bHghquk8EOJTvVptpT+SoPZ0PkcqMXJ+zbOWdWPMVJjWSy0bGE/yyMvB rDdBXWHk0CrXPQOa5fs3q9CfZl6giSW3uY8XFyMWQnIjHCR1AzoSNWegMJAExxbSPiFL LJWKBpgYLcbtbIchTblm4wHU7FcB/AgSPVjBJVmx3sPrrI1nJnQFA28AIMO+scRKCcBU oRElkvcA7s+czj5MhbZbki6O8aLQ1Nub5w71jkWmvxU7aEQ36VKGTLiot/CVLm5uoumd ftGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404222; x=1749009022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v4ULvkeWo6R+wt2v8eLeuZgd3DEWdnv8cuyh2RcE8bQ=; b=QD1udeNPlp2OOYG9MijunMPw6Yvs4e2yvyzbl9EaMdvO0IqU01zpc5o99Xgh3lnB2H YHy/FyDFqnSrJ4yg6+PwSKTHsVquR/Kia3ywsZGxc4LoAVpaC7x3hFUXR97DX+JE32j0 yvhCyuoZME+C7cbHiW8YOAMVbp01tedXfsR5UM2k801YNme369rZt9HUxt1jcu7VRrTY B3E9GS6H7ReTrwkQ9v49tTYYx4o5uBsQlDPhsMk2bDOBKSKqS0vBVJvFY6EDV+CVigc+ pyRPp8PgRWjS7vQkwef2amO7mhm/YS/TWjCWSXN+csoCGDjN9mj6OXYt593P23UuRzmY sNdA== X-Forwarded-Encrypted: i=1; AJvYcCVrx0GTZftIaWiT8XpAm4uGdlHyBn4ldcEqRzvP1qCBtsZ4WUs2BuodsH5KwPDbrogc2rBUgHcAp5fpuRs=@vger.kernel.org X-Gm-Message-State: AOJu0YyBrz3T0EJD1c+eP3BpOU9gErMj9qKzYauUeZcX/M1NRLyUzNdh yNNd6leF3hLC+8ZcJ+3Js1YpytcI7qa6AScktpFImb842ra4qeeF3E1G X-Gm-Gg: ASbGncuj/bH1eLk7LsvC9i2Kpwde/ifVTzyOeiZ3ZMrYmQLPvVFF2aoNQTdCEbKTOA+ mdheU1q2pgg4qyYOf3sRc46yOhrfQ1Se1aVlzBZsbcoZXDKcLBepwFvuUMhskB48/OWENybEy/O H/B0Z/aWynNuwYi5gPWVfN4LwLJfev1hzSK2IvDCXUzg8vJhbjnTKgFAliq+3b/JvU1wp9ShMjv SpMl2QQWBmffsKKcx/Bgn43FJIMRtVTuPRGJ2QZg0JyOq9Y3G4DtSb2ZzdpIMmpG0HQ0cjvlYDr QPpkfsx796XWktzwTgD4hWnLYHHplrwSz9py0QzqWm5D2OsaLABs9ixfpjPD6Gw3FfJi X-Google-Smtp-Source: AGHT+IGrTl+R2r5VL+P6flkdaWz94R5YsISZycgZQK7Co5/DR4zquz85jMGX1WLjHOVrY2O7lEJ0/g== X-Received: by 2002:a17:903:986:b0:234:c8f6:1b03 with SMTP id d9443c01a7336-234d2c4aa08mr13169505ad.47.1748404221993; Tue, 27 May 2025 20:50:21 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:21 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 21/25] libbpf: add skip_invalid and attach_tracing for tracing_multi Date: Wed, 28 May 2025 11:47:08 +0800 Message-Id: <20250528034712.138701-22-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We add skip_invalid and attach_tracing for tracing_multi for the selftests. When we try to attach all the functions in available_filter_functions with tracing_multi, we can't tell if the target symbol can be attached successfully, and the attaching will fail. When skip_invalid is set to true, we will check if it can be attached in libbpf, and skip the invalid entries. We will skip the symbols in the following cases: 1. the btf type not exist 2. the btf type is not a function proto 3. the function args count more that 6 4. the return type is struct or union 5. any function args is struct or union The 5th rule can be a manslaughter, but it's ok for the testings. "attach_tracing" is used to convert a TRACING prog to TRACING_MULTI. For example, we can set the attach type to FENTRY_MULTI before we load the skel. And we can attach the prog with bpf_program__attach_trace_multi_opts() with "attach_tracing=3D1". The libbpf will attach the target btf type of the prog automatically. This is also used to reuse the selftests of tracing. (Oh my goodness! What am I doing?) Signed-off-by: Menglong Dong --- tools/lib/bpf/libbpf.c | 97 ++++++++++++++++++++++++++++++++++++------ tools/lib/bpf/libbpf.h | 6 ++- 2 files changed, 89 insertions(+), 14 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 4a903102e0c7..911fda3f678c 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -10132,7 +10132,8 @@ static int libbpf_find_prog_btf_id(const char *name= , __u32 attach_prog_fd, int t =20 static int find_kernel_btf_id(struct bpf_object *obj, const char *attach_n= ame, enum bpf_attach_type attach_type, - int *btf_obj_fd, int *btf_type_id, bool use_hash) + int *btf_obj_fd, int *btf_type_id, bool use_hash, + const struct btf **btf) { int ret, i, mod_len, err; const char *fn_name, *mod_name =3D NULL; @@ -10156,6 +10157,8 @@ static int find_kernel_btf_id(struct bpf_object *ob= j, const char *attach_name, if (ret > 0) { *btf_obj_fd =3D 0; /* vmlinux BTF */ *btf_type_id =3D ret; + if (btf) + *btf =3D obj->btf_vmlinux; return 0; } if (ret !=3D -ENOENT) @@ -10183,6 +10186,8 @@ static int find_kernel_btf_id(struct bpf_object *ob= j, const char *attach_name, if (ret > 0) { *btf_obj_fd =3D mod->fd; *btf_type_id =3D ret; + if (btf) + *btf =3D mod->btf; return 0; } if (ret =3D=3D -ENOENT) @@ -10226,7 +10231,7 @@ static int libbpf_find_attach_btf_id(struct bpf_pro= gram *prog, const char *attac } else { err =3D find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, - btf_type_id, false); + btf_type_id, false, NULL); } if (err) { pr_warn("prog '%s': failed to find kernel BTF type ID of '%s': %s\n", @@ -12836,6 +12841,53 @@ static int attach_trace(const struct bpf_program *= prog, long cookie, struct bpf_ return libbpf_get_error(*link); } =20 +static bool is_trace_valid(const struct btf *btf, int btf_type_id, const c= har *name) +{ + const struct btf_type *t; + + t =3D skip_mods_and_typedefs(btf, btf_type_id, NULL); + if (btf_is_func(t)) { + const struct btf_param *args; + __u32 nargs, m; + + t =3D skip_mods_and_typedefs(btf, t->type, NULL); + if (!btf_is_func_proto(t)) { + pr_debug("skipping no function btf type for %s\n", + name); + return false; + } + + args =3D (const struct btf_param *)(t + 1); + nargs =3D btf_vlen(t); + if (nargs > 6) { + pr_debug("skipping args count more than 6 for %s\n", + name); + return false; + } + + t =3D skip_mods_and_typedefs(btf, t->type, NULL); + if (btf_is_struct(t) || btf_is_union(t) || + (nargs && args[nargs - 1].type =3D=3D 0)) { + pr_debug("skipping invalid return type for %s\n", + name); + return false; + } + + for (m =3D 0; m < nargs; m++) { + t =3D skip_mods_and_typedefs(btf, args[m].type, NULL); + if (btf_is_struct(t) || btf_is_union(t)) { + pr_debug("skipping not supported arg type %s\n", + name); + break; + } + } + if (m < nargs) + return false; + } + + return true; +} + struct bpf_link *bpf_program__attach_trace_multi_opts(const struct bpf_pro= gram *prog, const struct bpf_trace_multi_opts *opts) { @@ -12856,7 +12908,7 @@ struct bpf_link *bpf_program__attach_trace_multi_op= ts(const struct bpf_program * =20 cnt =3D OPTS_GET(opts, cnt, 0); if (opts->syms) { - int btf_obj_fd, btf_type_id, i; + int btf_obj_fd, btf_type_id, i, j =3D 0; =20 if (opts->btf_ids || opts->tgt_fds) { pr_warn("can set both opts->syms and opts->btf_ids\n"); @@ -12870,23 +12922,41 @@ struct bpf_link *bpf_program__attach_trace_multi_= opts(const struct bpf_program * goto err_free; } for (i =3D 0; i < cnt; i++) { + const struct btf *btf =3D NULL; + bool func_hash; + /* only use btf type function hashmap when the count * is big enough. */ - bool func_hash =3D cnt > 1024; - - + func_hash =3D cnt > 1024; btf_obj_fd =3D btf_type_id =3D 0; err =3D find_kernel_btf_id(prog->obj, opts->syms[i], - prog->expected_attach_type, &btf_obj_fd, - &btf_type_id, func_hash); - if (err) - goto err_free; - btf_ids[i] =3D btf_type_id; - tgt_fds[i] =3D btf_obj_fd; + prog->expected_attach_type, &btf_obj_fd, + &btf_type_id, func_hash, &btf); + if (err) { + if (!opts->skip_invalid) + goto err_free; + + pr_debug("can't find btf type for %s, skip\n", + opts->syms[i]); + continue; + } + + if (opts->skip_invalid && + !is_trace_valid(btf, btf_type_id, opts->syms[i])) + continue; + + btf_ids[j] =3D btf_type_id; + tgt_fds[j] =3D btf_obj_fd; + j++; } + cnt =3D j; link_opts.tracing_multi.btf_ids =3D btf_ids; link_opts.tracing_multi.tgt_fds =3D tgt_fds; + } else if (opts->attach_tracing) { + link_opts.tracing_multi.btf_ids =3D &prog->attach_btf_id; + link_opts.tracing_multi.tgt_fds =3D &prog->attach_btf_obj_fd; + cnt =3D 1; } else { link_opts.tracing_multi.btf_ids =3D OPTS_GET(opts, btf_ids, 0); link_opts.tracing_multi.tgt_fds =3D OPTS_GET(opts, tgt_fds, 0); @@ -13957,7 +14027,8 @@ int bpf_program__set_attach_target(struct bpf_progr= am *prog, return libbpf_err(err); err =3D find_kernel_btf_id(prog->obj, attach_func_name, prog->expected_attach_type, - &btf_obj_fd, &btf_id, false); + &btf_obj_fd, &btf_id, false, + NULL); if (err) return libbpf_err(err); } diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index d7f0db7ab586..c087525ad25a 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -846,8 +846,12 @@ struct bpf_trace_multi_opts { __u64 *cookies; /* number of elements in syms/btf_ids/cookies arrays */ size_t cnt; + /* skip the invalid btf type before attaching */ + bool skip_invalid; + /* attach a TRACING prog as TRACING_MULTI */ + bool attach_tracing; }; -#define bpf_trace_multi_opts__last_field cnt +#define bpf_trace_multi_opts__last_field attach_tracing =20 LIBBPF_API struct bpf_link * bpf_program__attach_trace_multi_opts(const struct bpf_program *prog, --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDCC01F5413; Wed, 28 May 2025 03:50:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404226; cv=none; b=jTKbQXEwTtAsx1DTD0L5AW+JHWzhBgWhDnvXd4VWrZcAYo2dlloHj8FSYnAfHOOVOCTk8sXEWTNNs1V4pBahObrilJk0sVxCYUht/R62vFkO76rVY9qlfSzeQtGggvtI+SNZR02LyHMnocjIDjA1molPzxlrWPXCV68NBs7mkAY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404226; c=relaxed/simple; bh=OvK/Q/DfXMU6KN25/kQRxR9LncmbU2X3WDmeadWtX7Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Y1YaLm2HBhtyI8RcDxvXMIVJ1CvyIp4ME6Q5CMDrp9k4HIwNJiyiX6fcxhyJCbl+QxKZvfek2qFc14XFHes2kbxH/xDCBrbhbHhoDweDmKbpRtN0HTo+OzA/3J4Re2LzFjXfCaUPYB2GW9AFx4mHK/gCQT2aHZZ+mmnelv8CgTw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f7XJ3fwC; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f7XJ3fwC" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-234ade5a819so11271675ad.1; Tue, 27 May 2025 20:50:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404224; x=1749009024; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KS7M6SZuFz0KIT+q3oUQfoKiK682acaipuzojcNU9pw=; b=f7XJ3fwC0imi8+XxddHxa6CiSMneohc2d7alfId8DXMVY0DpCAM2th9oEYTXHSwy4B MAU1eCVGK+2sFdMBozAl4TNO9Lwe/dUG4egr+hj+C1WSlNH6NCMS2HjD4b8quig9sUpB tsh1u7uji48lVfec2JfT2fq3RAeG164WW/JSjzWJ+DkuRBkbDsSgbIzypusx0s6DPRGa fnHmRtbhyGaUv2PozBePWA3m2wfN55T8fpYX4kOcL9MEQ9c28H7cSdUB5FnY6MpV3G0q u22NQ3Y8GpcusXxP88leLCxm9Vi3yuhXK42sE6HDpuTrvk9+yF1//1Gj611AqOMwJqLe U+jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404224; x=1749009024; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KS7M6SZuFz0KIT+q3oUQfoKiK682acaipuzojcNU9pw=; b=CyM59Fx0ij5mknyZiCRt6yS7XRC3yfcakAneKfl1Wo9CjeU6k7dWNZtDpzBWkEeiUp iDYsbeazN+tYSVt5M1dxjobExnH4mUthXIRIacmOlPMM9Hc+0pc+3Xd2/6KWE4iJp6m8 toka9oTZo20JzSkZYnjqmAAyPV8r0OhjQyDzsqat7ILjBsYiEOfmhCvDQ7k36Pd2JGX7 OqWYrgi/Q863+kSPC3GgXR0XHau3kBpIlSeTDWHQhtoT9zme6qK6gxPL+1GCyJufbk8n hw6heo6FpzqLDVN560sSBa3BHDKzU84JT74NGa295XV7jP8WFa4nXu1SJCX2sYM+GfBQ lQjQ== X-Forwarded-Encrypted: i=1; AJvYcCUKtdVMTS/i/d5OypAaxufHxUKakusO8xuu6Zug1U6M9mYMSjvUo+2GJUOZeN9kopuowoiR9HQuFN7ldUI=@vger.kernel.org X-Gm-Message-State: AOJu0YwkExOWjlm5uAFYvsr44Pa+OO6vbKimZ535d3TEnFXuyrDIRqfa SzFez1sEkYff+ZmeSjffHbqWjSoVwa4AJ6JqCO+PVGYwF5uueJxsPBLF X-Gm-Gg: ASbGncsDPZbyKEnffy4nhNCMIuEdzVaTS3uWv3OeABWKNITzL2k281YoB5QADnMxF20 WgxRH0W9gxXNIitfp4r2LYlBJY0XceRBnBT18zFHmDqp0gsSj0th4lHlSpPCgzClizLf58nv8CQ iJQwZABm4i691PpxsWtsTtBv9uPL6YiT1FKKJsrfvUdoa/KyQKWX4N609Khc9eJew7H8V0SH1YB Og9qdPQpNj4cPnVMf6MHfFUqASowwfhoPQVBOuSSVJjHq5AVJCgYELllU2N1G1tC3R6UILog9xp 2oyj5qk24GS4k8hrzhW4m7eU0po14MaNUmjzORJ5M5+jUwiBqT2no3lC15sMJ3wJOb3t2wtVSDH olzY= X-Google-Smtp-Source: AGHT+IH0eSpGHXA08+6NiTdGlaPyUo6UXH/UGs2FC/ewWlNyd1asZxmq/TnGKqHq+2JIpPmABK5pvg== X-Received: by 2002:a17:902:ec8a:b0:231:d461:5326 with SMTP id d9443c01a7336-23414fe6ffbmr208440535ad.50.1748404224137; Tue, 27 May 2025 20:50:24 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:23 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 22/25] selftests/bpf: use the glob_match() from libbpf in test_progs.c Date: Wed, 28 May 2025 11:47:09 +0800 Message-Id: <20250528034712.138701-23-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The glob_match() in test_progs.c has almost the same logic with the glob_match() in libbpf.c, so we replace it to make the code simple. Signed-off-by: Menglong Dong --- tools/testing/selftests/bpf/test_progs.c | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selft= ests/bpf/test_progs.c index 309d9d4a8ace..e246fe4b7b70 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -17,6 +17,7 @@ #include #include #include +#include "bpf/libbpf_internal.h" #include "json_writer.h" =20 #include "network_helpers.h" @@ -129,28 +130,6 @@ static int traffic_monitor_print_fn(const char *format= , va_list args) return 0; } =20 -/* Adapted from perf/util/string.c */ -static bool glob_match(const char *str, const char *pat) -{ - while (*str && *pat && *pat !=3D '*') { - if (*str !=3D *pat) - return false; - str++; - pat++; - } - /* Check wild card */ - if (*pat =3D=3D '*') { - while (*pat =3D=3D '*') - pat++; - if (!*pat) /* Tail wild card matches all */ - return true; - while (*str) - if (glob_match(str++, pat)) - return true; - } - return !*str && !*pat; -} - #define EXIT_NO_TEST 2 #define EXIT_ERR_SETUP_INFRA 3 =20 --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4955A1F790F; Wed, 28 May 2025 03:50:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404229; cv=none; b=hvfqMEO+X7QIH/JAH4JKdtcpEGsAmE1luXVoP2kEGg/r4WoDnQ671oocFeo3/KCQ3OhtB2MGrpD3h8meg+8wviSq/e5aJqiEI8eabaNpzydtnwGWpCuWen6PI8r9nvoP4GUiOm8yLKVe/g1n0c9umqZeam/r5+PnpBt0M2KWd7o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404229; c=relaxed/simple; bh=AcaO69fi2Cm16me+V0/KdP3xPRhkcRRYM/GGL20HM90=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GvQPwJmzYQSxElEmnWJX2X/KqtO3f6uE7v7toCxzCiCmVJCoOxzYyAZLjjqDTk/XGNMbTufGleVCwcUC8JMoneptjk2DrdwIDB8xvBYLmQhdEadl5/dmRPF/Z5GKScLTgdFMFb+FPXUIM5OvWEgo6UoluZxdWaU8q3GPMvBXQbs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OQgSsfYu; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OQgSsfYu" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-22e16234307so4362895ad.0; Tue, 27 May 2025 20:50:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404226; x=1749009026; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cv8Gaih+91J3pEdij4+7GRCFDqzELtyA4Kb4EEM5piM=; b=OQgSsfYu1I7GYiFcgXDqOPhGv8XsQ9LzYhcYafWUFJOBKpN8140YCtSFgzZ3k+8Lw7 3jHZIBcIKYOKyGKvB3EG6qgSe2QqdZn2J1rORcwXY7AzUpVF+yW519e3p3Tow+D7iFz0 7y8svWEaSApd6pv9mHNACZJQaRHslCnH6dLkKBuscq0XZ9pKFe+en/zY4G0TH6pn/YFO qlpjKPUYd099+U58+ZnfEJG1NA9pVyBcuCiJesBEC3hvG7BbTBpr0o66uC38Wztn+XW8 dbDdC2LLXX/bMiOfqQEoRhMWgb2sY/EUKU1V7nDyojpCOxBI5Tccx1nMGAL+zVYZMijI fC1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404226; x=1749009026; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cv8Gaih+91J3pEdij4+7GRCFDqzELtyA4Kb4EEM5piM=; b=rUE6W+HB/PTiiDxRKerMmb8VD/oRkwN8D6GlFXPb2d4+cejSzmHA2Vu+1erjDCyL52 GG1JRa7VCByDXC/HI7/VLBKmaZS/PpbXg6z3W9TGiStwqGQUA2oKtLCET/N8GMnst1Kf O7xItq+sLTyNSyafHLFNJg5SkvsqhlNjjvcwtwIWiRj2BbPLSH211pJoZ56EE04jT3Ta CYNrYsDdjUy+JCDYcB7P6fGMzDl5McVbZwad0sOGItihBx++4TZRa+U5htlLEOk767Ga fCgofBlXdgc8YA3nAmZ4CU0PTsj77NJ3uRCRsMZLzsSG8j3prFq0CpBm6AdoG0JJtxMG Nq3w== X-Forwarded-Encrypted: i=1; AJvYcCXw9QcMobtJx7rliQM4MCpnQJbB2n1FTk3lpBUC/ykCD/eYCR+mFAlzX72eaPPacjZTcgwWPdyaP1lGavE=@vger.kernel.org X-Gm-Message-State: AOJu0YwqTqJ5dhLxmzLVoOAwvT86O2sMPM81/7Ek8MYY800dEL2LbNyE bnh7QdzXpsC2V9vtOuuJmq5jVjyvLnd1SPJUULfmxlUl/JFyQlLHJ4OE X-Gm-Gg: ASbGncvywImuCENZ6nq/pTuR9n843UnSn3knL3IViGWj7r+qTIxA6jUTiKKQFlTzY2D jbG/FRzKT/3qHmeZqj8zFOb2iALThBojViIpojjSPImd/m2Y/+TmICLeBLSfknFYh/oZ3ihinEV T6tRwDrzFVT/A1lBKdqRoICcnE4MZ/j9JS21wOiZwZL4DK8n6w2pBwob1HtlOdoN5+VvtwgaGYK LqHnklTEuI+3XQoAvIjmwRWPyC5eLuYswMhWCns1xMjCNAdtYCWPw4PH9uVq2ousGIidqzD5gNX l/74vusfNHeEHQBsZ6UpUwj900Npw2cbKnt6bSwxwfpJU3lheA80wyy+XSS2cUBBNwly X-Google-Smtp-Source: AGHT+IEt91bPXG7soyKF2aKcPaT2KY1bHpuur1X67g7WjyjNsrjONQCoB0VAJ9kcDLP+2eGzwYxV1g== X-Received: by 2002:a17:903:2f89:b0:215:6c5f:d142 with SMTP id d9443c01a7336-234b74dba8amr44353475ad.20.1748404226348; Tue, 27 May 2025 20:50:26 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:25 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 23/25] selftests/bpf: add get_ksyms and get_addrs to test_progs.c Date: Wed, 28 May 2025 11:47:10 +0800 Message-Id: <20250528034712.138701-24-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We need to get all the kernel function that can be traced sometimes, so we move the get_syms() and get_addrs() in kprobe_multi_test.c to test_progs.c and rename it to bpf_get_ksyms() and bpf_get_addrs(). Signed-off-by: Menglong Dong --- .../bpf/prog_tests/kprobe_multi_test.c | 220 +----------------- tools/testing/selftests/bpf/test_progs.c | 214 +++++++++++++++++ tools/testing/selftests/bpf/test_progs.h | 2 + 3 files changed, 219 insertions(+), 217 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c b/t= ools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c index e19ef509ebf8..171706e78da8 100644 --- a/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +++ b/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c @@ -422,220 +422,6 @@ static void test_unique_match(void) kprobe_multi__destroy(skel); } =20 -static size_t symbol_hash(long key, void *ctx __maybe_unused) -{ - return str_hash((const char *) key); -} - -static bool symbol_equal(long key1, long key2, void *ctx __maybe_unused) -{ - return strcmp((const char *) key1, (const char *) key2) =3D=3D 0; -} - -static bool is_invalid_entry(char *buf, bool kernel) -{ - if (kernel && strchr(buf, '[')) - return true; - if (!kernel && !strchr(buf, '[')) - return true; - return false; -} - -static bool skip_entry(char *name) -{ - /* - * We attach to almost all kernel functions and some of them - * will cause 'suspicious RCU usage' when fprobe is attached - * to them. Filter out the current culprits - arch_cpu_idle - * default_idle and rcu_* functions. - */ - if (!strcmp(name, "arch_cpu_idle")) - return true; - if (!strcmp(name, "default_idle")) - return true; - if (!strncmp(name, "rcu_", 4)) - return true; - if (!strcmp(name, "bpf_dispatcher_xdp_func")) - return true; - if (!strncmp(name, "__ftrace_invalid_address__", - sizeof("__ftrace_invalid_address__") - 1)) - return true; - return false; -} - -/* Do comparision by ignoring '.llvm.' suffixes. */ -static int compare_name(const char *name1, const char *name2) -{ - const char *res1, *res2; - int len1, len2; - - res1 =3D strstr(name1, ".llvm."); - res2 =3D strstr(name2, ".llvm."); - len1 =3D res1 ? res1 - name1 : strlen(name1); - len2 =3D res2 ? res2 - name2 : strlen(name2); - - if (len1 =3D=3D len2) - return strncmp(name1, name2, len1); - if (len1 < len2) - return strncmp(name1, name2, len1) <=3D 0 ? -1 : 1; - return strncmp(name1, name2, len2) >=3D 0 ? 1 : -1; -} - -static int load_kallsyms_compare(const void *p1, const void *p2) -{ - return compare_name(((const struct ksym *)p1)->name, ((const struct ksym = *)p2)->name); -} - -static int search_kallsyms_compare(const void *p1, const struct ksym *p2) -{ - return compare_name(p1, p2->name); -} - -static int get_syms(char ***symsp, size_t *cntp, bool kernel) -{ - size_t cap =3D 0, cnt =3D 0; - char *name =3D NULL, *ksym_name, **syms =3D NULL; - struct hashmap *map; - struct ksyms *ksyms; - struct ksym *ks; - char buf[256]; - FILE *f; - int err =3D 0; - - ksyms =3D load_kallsyms_custom_local(load_kallsyms_compare); - if (!ASSERT_OK_PTR(ksyms, "load_kallsyms_custom_local")) - return -EINVAL; - - /* - * The available_filter_functions contains many duplicates, - * but other than that all symbols are usable in kprobe multi - * interface. - * Filtering out duplicates by using hashmap__add, which won't - * add existing entry. - */ - - if (access("/sys/kernel/tracing/trace", F_OK) =3D=3D 0) - f =3D fopen("/sys/kernel/tracing/available_filter_functions", "r"); - else - f =3D fopen("/sys/kernel/debug/tracing/available_filter_functions", "r"); - - if (!f) - return -EINVAL; - - map =3D hashmap__new(symbol_hash, symbol_equal, NULL); - if (IS_ERR(map)) { - err =3D libbpf_get_error(map); - goto error; - } - - while (fgets(buf, sizeof(buf), f)) { - if (is_invalid_entry(buf, kernel)) - continue; - - free(name); - if (sscanf(buf, "%ms$*[^\n]\n", &name) !=3D 1) - continue; - if (skip_entry(name)) - continue; - - ks =3D search_kallsyms_custom_local(ksyms, name, search_kallsyms_compare= ); - if (!ks) { - err =3D -EINVAL; - goto error; - } - - ksym_name =3D ks->name; - err =3D hashmap__add(map, ksym_name, 0); - if (err =3D=3D -EEXIST) { - err =3D 0; - continue; - } - if (err) - goto error; - - err =3D libbpf_ensure_mem((void **) &syms, &cap, - sizeof(*syms), cnt + 1); - if (err) - goto error; - - syms[cnt++] =3D ksym_name; - } - - *symsp =3D syms; - *cntp =3D cnt; - -error: - free(name); - fclose(f); - hashmap__free(map); - if (err) - free(syms); - return err; -} - -static int get_addrs(unsigned long **addrsp, size_t *cntp, bool kernel) -{ - unsigned long *addr, *addrs, *tmp_addrs; - int err =3D 0, max_cnt, inc_cnt; - char *name =3D NULL; - size_t cnt =3D 0; - char buf[256]; - FILE *f; - - if (access("/sys/kernel/tracing/trace", F_OK) =3D=3D 0) - f =3D fopen("/sys/kernel/tracing/available_filter_functions_addrs", "r"); - else - f =3D fopen("/sys/kernel/debug/tracing/available_filter_functions_addrs"= , "r"); - - if (!f) - return -ENOENT; - - /* In my local setup, the number of entries is 50k+ so Let us initially - * allocate space to hold 64k entries. If 64k is not enough, incrementally - * increase 1k each time. - */ - max_cnt =3D 65536; - inc_cnt =3D 1024; - addrs =3D malloc(max_cnt * sizeof(long)); - if (addrs =3D=3D NULL) { - err =3D -ENOMEM; - goto error; - } - - while (fgets(buf, sizeof(buf), f)) { - if (is_invalid_entry(buf, kernel)) - continue; - - free(name); - if (sscanf(buf, "%p %ms$*[^\n]\n", &addr, &name) !=3D 2) - continue; - if (skip_entry(name)) - continue; - - if (cnt =3D=3D max_cnt) { - max_cnt +=3D inc_cnt; - tmp_addrs =3D realloc(addrs, max_cnt); - if (!tmp_addrs) { - err =3D -ENOMEM; - goto error; - } - addrs =3D tmp_addrs; - } - - addrs[cnt++] =3D (unsigned long)addr; - } - - *addrsp =3D addrs; - *cntp =3D cnt; - -error: - free(name); - fclose(f); - if (err) - free(addrs); - return err; -} - static void do_bench_test(struct kprobe_multi_empty *skel, struct bpf_kpro= be_multi_opts *opts) { long attach_start_ns, attach_end_ns; @@ -670,7 +456,7 @@ static void test_kprobe_multi_bench_attach(bool kernel) char **syms =3D NULL; size_t cnt =3D 0; =20 - if (!ASSERT_OK(get_syms(&syms, &cnt, kernel), "get_syms")) + if (!ASSERT_OK(bpf_get_ksyms(&syms, &cnt, kernel), "bpf_get_ksyms")) return; =20 skel =3D kprobe_multi_empty__open_and_load(); @@ -696,13 +482,13 @@ static void test_kprobe_multi_bench_attach_addr(bool = kernel) size_t cnt =3D 0; int err; =20 - err =3D get_addrs(&addrs, &cnt, kernel); + err =3D bpf_get_addrs(&addrs, &cnt, kernel); if (err =3D=3D -ENOENT) { test__skip(); return; } =20 - if (!ASSERT_OK(err, "get_addrs")) + if (!ASSERT_OK(err, "bpf_get_addrs")) return; =20 skel =3D kprobe_multi_empty__open_and_load(); diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selft= ests/bpf/test_progs.c index e246fe4b7b70..26cc50bbed8b 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -18,6 +18,7 @@ #include #include #include "bpf/libbpf_internal.h" +#include "bpf/hashmap.h" #include "json_writer.h" =20 #include "network_helpers.h" @@ -646,6 +647,219 @@ int bpf_find_map(const char *test, struct bpf_object = *obj, const char *name) return bpf_map__fd(map); } =20 +static size_t symbol_hash(long key, void *ctx __maybe_unused) +{ + return str_hash((const char *) key); +} + +static bool symbol_equal(long key1, long key2, void *ctx __maybe_unused) +{ + return strcmp((const char *) key1, (const char *) key2) =3D=3D 0; +} + +static bool is_invalid_entry(char *buf, bool kernel) +{ + if (kernel && strchr(buf, '[')) + return true; + if (!kernel && !strchr(buf, '[')) + return true; + return false; +} + +static bool skip_entry(char *name) +{ + /* + * We attach to almost all kernel functions and some of them + * will cause 'suspicious RCU usage' when fprobe is attached + * to them. Filter out the current culprits - arch_cpu_idle + * default_idle and rcu_* functions. + */ + if (!strcmp(name, "arch_cpu_idle")) + return true; + if (!strcmp(name, "default_idle")) + return true; + if (!strncmp(name, "rcu_", 4)) + return true; + if (!strcmp(name, "bpf_dispatcher_xdp_func")) + return true; + if (!strncmp(name, "__ftrace_invalid_address__", + sizeof("__ftrace_invalid_address__") - 1)) + return true; + return false; +} + +/* Do comparison by ignoring '.llvm.' suffixes. */ +static int compare_name(const char *name1, const char *name2) +{ + const char *res1, *res2; + int len1, len2; + + res1 =3D strstr(name1, ".llvm."); + res2 =3D strstr(name2, ".llvm."); + len1 =3D res1 ? res1 - name1 : strlen(name1); + len2 =3D res2 ? res2 - name2 : strlen(name2); + + if (len1 =3D=3D len2) + return strncmp(name1, name2, len1); + if (len1 < len2) + return strncmp(name1, name2, len1) <=3D 0 ? -1 : 1; + return strncmp(name1, name2, len2) >=3D 0 ? 1 : -1; +} + +static int load_kallsyms_compare(const void *p1, const void *p2) +{ + return compare_name(((const struct ksym *)p1)->name, ((const struct ksym = *)p2)->name); +} + +static int search_kallsyms_compare(const void *p1, const struct ksym *p2) +{ + return compare_name(p1, p2->name); +} + +int bpf_get_ksyms(char ***symsp, size_t *cntp, bool kernel) +{ + size_t cap =3D 0, cnt =3D 0; + char *name =3D NULL, *ksym_name, **syms =3D NULL; + struct hashmap *map; + struct ksyms *ksyms; + struct ksym *ks; + char buf[256]; + FILE *f; + int err =3D 0; + + ksyms =3D load_kallsyms_custom_local(load_kallsyms_compare); + if (!ASSERT_OK_PTR(ksyms, "load_kallsyms_custom_local")) + return -EINVAL; + + /* + * The available_filter_functions contains many duplicates, + * but other than that all symbols are usable to trace. + * Filtering out duplicates by using hashmap__add, which won't + * add existing entry. + */ + + if (access("/sys/kernel/tracing/trace", F_OK) =3D=3D 0) + f =3D fopen("/sys/kernel/tracing/available_filter_functions", "r"); + else + f =3D fopen("/sys/kernel/debug/tracing/available_filter_functions", "r"); + + if (!f) + return -EINVAL; + + map =3D hashmap__new(symbol_hash, symbol_equal, NULL); + if (IS_ERR(map)) { + err =3D libbpf_get_error(map); + goto error; + } + + while (fgets(buf, sizeof(buf), f)) { + if (is_invalid_entry(buf, kernel)) + continue; + + free(name); + if (sscanf(buf, "%ms$*[^\n]\n", &name) !=3D 1) + continue; + if (skip_entry(name)) + continue; + + ks =3D search_kallsyms_custom_local(ksyms, name, search_kallsyms_compare= ); + if (!ks) { + err =3D -EINVAL; + goto error; + } + + ksym_name =3D ks->name; + err =3D hashmap__add(map, ksym_name, 0); + if (err =3D=3D -EEXIST) { + err =3D 0; + continue; + } + if (err) + goto error; + + err =3D libbpf_ensure_mem((void **) &syms, &cap, + sizeof(*syms), cnt + 1); + if (err) + goto error; + + syms[cnt++] =3D ksym_name; + } + + *symsp =3D syms; + *cntp =3D cnt; + +error: + free(name); + fclose(f); + hashmap__free(map); + if (err) + free(syms); + return err; +} + +int bpf_get_addrs(unsigned long **addrsp, size_t *cntp, bool kernel) +{ + unsigned long *addr, *addrs, *tmp_addrs; + int err =3D 0, max_cnt, inc_cnt; + char *name =3D NULL; + size_t cnt =3D 0; + char buf[256]; + FILE *f; + + if (access("/sys/kernel/tracing/trace", F_OK) =3D=3D 0) + f =3D fopen("/sys/kernel/tracing/available_filter_functions_addrs", "r"); + else + f =3D fopen("/sys/kernel/debug/tracing/available_filter_functions_addrs"= , "r"); + + if (!f) + return -ENOENT; + + /* In my local setup, the number of entries is 50k+ so Let us initially + * allocate space to hold 64k entries. If 64k is not enough, incrementally + * increase 1k each time. + */ + max_cnt =3D 65536; + inc_cnt =3D 1024; + addrs =3D malloc(max_cnt * sizeof(long)); + if (addrs =3D=3D NULL) { + err =3D -ENOMEM; + goto error; + } + + while (fgets(buf, sizeof(buf), f)) { + if (is_invalid_entry(buf, kernel)) + continue; + + free(name); + if (sscanf(buf, "%p %ms$*[^\n]\n", &addr, &name) !=3D 2) + continue; + if (skip_entry(name)) + continue; + + if (cnt =3D=3D max_cnt) { + max_cnt +=3D inc_cnt; + tmp_addrs =3D realloc(addrs, max_cnt); + if (!tmp_addrs) { + err =3D -ENOMEM; + goto error; + } + addrs =3D tmp_addrs; + } + + addrs[cnt++] =3D (unsigned long)addr; + } + + *addrsp =3D addrs; + *cntp =3D cnt; + +error: + free(name); + fclose(f); + if (err) + free(addrs); + return err; +} + int compare_map_keys(int map1_fd, int map2_fd) { __u32 key, next_key; diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selft= ests/bpf/test_progs.h index 870694f2a359..477a785c8b5f 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -468,6 +468,8 @@ int trigger_module_test_write(int write_sz); int write_sysctl(const char *sysctl, const char *value); int get_bpf_max_tramp_links_from(struct btf *btf); int get_bpf_max_tramp_links(void); +int bpf_get_ksyms(char ***symsp, size_t *cntp, bool kernel); +int bpf_get_addrs(unsigned long **addrsp, size_t *cntp, bool kernel); =20 struct netns_obj; struct netns_obj *netns_new(const char *name, bool open); --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79048221263; Wed, 28 May 2025 03:50:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404232; cv=none; b=UVs67ecSiaKzRuX96QhFgr0OKLUhveQAu6hTx2Ba9EAG0plkkrhPmT8jNOmkhFI7YwD28u6WLu7Q86fCnouFIKkFqnl3EkwZXfKbr/R0/tvseBLSEnnwSOn+RKxwFn5VnNHLJuxV/1P9Odm+DnNxRuSiXvj5PP+TIgNokm10vDk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404232; c=relaxed/simple; bh=aPQ3vmm3j5g5SSBlHLCGoS0Iq/r9v/hjD1ssl3W2V6w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IXUXm0YQdcwHQgytywHkqd38pnW88ENlz/v3vvctS7Yo7w8kVbb1u5HPn5oQsl29DA3m/OSi3OD86svpB6yKjXZE/oB+IHkGYhIr2ofYEz/UBDNO03krduAPtA9Qg3R/O51mwqJQF/6/736sfHh7jE3EhQMTM4juD/59i6/OiKI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QokRMHgd; arc=none smtp.client-ip=209.85.215.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QokRMHgd" Received: by mail-pg1-f196.google.com with SMTP id 41be03b00d2f7-b0db0b6a677so2849905a12.2; Tue, 27 May 2025 20:50:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404229; x=1749009029; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=quFKE0IyjJDzhZdIU0S08uOjSryGsB+MNJLbAiFEAVo=; b=QokRMHgdWVCZAxS8N21BWyIfmIH0ncRJT6wTfZFCAShsVKeEp98Ha9Oor1sxD/xVMu ZpY4sfddDnPGGahbiLyt0w1ZkDfa3ozAolD4Wm4FsFXdi++4FQYYx49sN4xEMOIBeTd8 JmZzVlBJaSKjPiFuRKzp2Bv98wQN/hiDXmDJdNJXj0tEN/LsMJ0R7YJ5Q2vB/DUi3SNu hB6YYrwHmIO7p6Os3UdOaJSpEpc2pqCyWjk+KIlzD6nvOWtl2jMxoeAOuoY9KWrkyQfC 0L5w4XlaaJ7y+6as4bOUcwk5qvIIiS4SVD2z3fWWMwJMH7pM2FTh2s0AY1zuHVuNbqtX 2pPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404229; x=1749009029; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=quFKE0IyjJDzhZdIU0S08uOjSryGsB+MNJLbAiFEAVo=; b=u6YRK502TLxX6UB+2eLozPyAYarYjknZPcxcjQGAiS6YelyjWZcOJUsWvALSgth+WR Adm+GhCmySqH6ewL4DdIp92tYtH7PxfBL2V8Y7id1TUkKrOHmXR7GMON+Qa1WqEzxH2V PXbU439Nk+HLpANWcUBxa+Q3c/eQgF2mum2V8D3dH5QLQ8fFVc4+Y5D2tOQ+y8etgSr1 WSjVn6lzoFifQCXNBXySOX0vYYs8NRS/CavcT/dwECceJuJJhWj9Xuw2XehVxPxPaQ76 8pfNA9Q2XEZPrUHq7yAtml446YVK7eilmAwrL6uY/bCrbLF5zdS9bfD+wa3u1OAAokYY qfDg== X-Forwarded-Encrypted: i=1; AJvYcCVsUz761DJ0nBHc/HU77s1DmATYm5LZENScLCdWXEuC3eM7noYHNt9vqf5mjswfEBzyUU9DktfEdyEHRBo=@vger.kernel.org X-Gm-Message-State: AOJu0Yx3INbDHbl3K8QkI9ps+BxqWYonoXBof/3qeLGhXK9bGLXaW87Z AYlYjFyd8J6vGueP6Qzq8LLF2rylBaNwhCwNMDwKpIc9e9LhWI3UX0Du X-Gm-Gg: ASbGnct7RMl1ntnPmBFgxl5ZxPJ0VYJr5+SK87x2NJlJMkN4oNi2Fha2rH5Q3bifAiG F+10qgAZqrlL3YUkWm+2iJ1K2qSAGzTBIau+TzkVC3K93E42HFKpv+KQIeWIuydcYbdhrgPhn2o uoPB1s+RI1C6ehmxwh9erEUhrq8AR41oqVJ4vjx4XmRY6IPOHEgUFHlsj9qNi5ToWCVzp+rm6g2 RzmpmWrXzoArcEq4NC80Z6wMEY63uzOpKG2wR4VWGYDVjXlZ+T/Kkgp8kGvtBhzBK0bz6GI7qLP ouik4qTrBZwaSdXMeN8siqFd7YhmcO/ABnx8RfApc+4tZwC8wIH0dX00XI1YefUnwqHi X-Google-Smtp-Source: AGHT+IFeCfedJJo38/XErRxppNs0m79/cOF9OhwnV++Lj8oL3ibJmss4a+Zgrfgqrq5+k2mxrLwefA== X-Received: by 2002:a17:902:f68a:b0:234:d679:72dc with SMTP id d9443c01a7336-234d67975a8mr3481135ad.6.1748404228639; Tue, 27 May 2025 20:50:28 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:28 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 24/25] selftests/bpf: add testcases for multi-link of tracing Date: Wed, 28 May 2025 11:47:11 +0800 Message-Id: <20250528034712.138701-25-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In this commit, we add some testcases for the following attach types: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI We reuse the testings in fentry_test.c, fexit_test.c and modify_return.c by attach the tracing bpf prog as tracing_multi. We add some functions to skip for tracing progs to bpf_get_ksyms() in this commit. Signed-off-by: Menglong Dong --- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/prog_tests/fentry_fexit.c | 22 +- .../selftests/bpf/prog_tests/fentry_test.c | 79 +++-- .../selftests/bpf/prog_tests/fexit_test.c | 79 +++-- .../selftests/bpf/prog_tests/modify_return.c | 60 ++++ .../bpf/prog_tests/tracing_multi_link.c | 276 ++++++++++++++++++ .../selftests/bpf/progs/fentry_multi_empty.c | 13 + .../bpf/progs/tracing_multi_override.c | 28 ++ .../selftests/bpf/progs/tracing_multi_test.c | 181 ++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 24 ++ tools/testing/selftests/bpf/test_progs.c | 112 +++++++ tools/testing/selftests/bpf/test_progs.h | 3 + 12 files changed, 831 insertions(+), 48 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/tracing_multi_li= nk.c create mode 100644 tools/testing/selftests/bpf/progs/fentry_multi_empty.c create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_overrid= e.c create mode 100644 tools/testing/selftests/bpf/progs/tracing_multi_test.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests= /bpf/Makefile index cf5ed3bee573..93cacb56591e 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -496,7 +496,7 @@ LINKED_SKELS :=3D test_static_linked.skel.h linked_func= s.skel.h \ test_subskeleton.skel.h test_subskeleton_lib.skel.h \ test_usdt.skel.h =20 -LSKELS :=3D fentry_test.c fexit_test.c fexit_sleep.c atomics.c \ +LSKELS :=3D fexit_sleep.c atomics.c \ trace_printk.c trace_vprintk.c map_ptr_kern.c \ core_kern.c core_kern_overflow.c test_ringbuf.c \ test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c b/tools/= testing/selftests/bpf/prog_tests/fentry_fexit.c index 130f5b82d2e6..84cc8b669684 100644 --- a/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c +++ b/tools/testing/selftests/bpf/prog_tests/fentry_fexit.c @@ -1,32 +1,32 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2019 Facebook */ #include -#include "fentry_test.lskel.h" -#include "fexit_test.lskel.h" +#include "fentry_test.skel.h" +#include "fexit_test.skel.h" =20 void test_fentry_fexit(void) { - struct fentry_test_lskel *fentry_skel =3D NULL; - struct fexit_test_lskel *fexit_skel =3D NULL; + struct fentry_test *fentry_skel =3D NULL; + struct fexit_test *fexit_skel =3D NULL; __u64 *fentry_res, *fexit_res; int err, prog_fd, i; LIBBPF_OPTS(bpf_test_run_opts, topts); =20 - fentry_skel =3D fentry_test_lskel__open_and_load(); + fentry_skel =3D fentry_test__open_and_load(); if (!ASSERT_OK_PTR(fentry_skel, "fentry_skel_load")) goto close_prog; - fexit_skel =3D fexit_test_lskel__open_and_load(); + fexit_skel =3D fexit_test__open_and_load(); if (!ASSERT_OK_PTR(fexit_skel, "fexit_skel_load")) goto close_prog; =20 - err =3D fentry_test_lskel__attach(fentry_skel); + err =3D fentry_test__attach(fentry_skel); if (!ASSERT_OK(err, "fentry_attach")) goto close_prog; - err =3D fexit_test_lskel__attach(fexit_skel); + err =3D fexit_test__attach(fexit_skel); if (!ASSERT_OK(err, "fexit_attach")) goto close_prog; =20 - prog_fd =3D fexit_skel->progs.test1.prog_fd; + prog_fd =3D bpf_program__fd(fexit_skel->progs.test1); err =3D bpf_prog_test_run_opts(prog_fd, &topts); ASSERT_OK(err, "ipv6 test_run"); ASSERT_OK(topts.retval, "ipv6 test retval"); @@ -40,6 +40,6 @@ void test_fentry_fexit(void) } =20 close_prog: - fentry_test_lskel__destroy(fentry_skel); - fexit_test_lskel__destroy(fexit_skel); + fentry_test__destroy(fentry_skel); + fexit_test__destroy(fexit_skel); } diff --git a/tools/testing/selftests/bpf/prog_tests/fentry_test.c b/tools/t= esting/selftests/bpf/prog_tests/fentry_test.c index aee1bc77a17f..9edd383feabd 100644 --- a/tools/testing/selftests/bpf/prog_tests/fentry_test.c +++ b/tools/testing/selftests/bpf/prog_tests/fentry_test.c @@ -1,26 +1,16 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2019 Facebook */ #include -#include "fentry_test.lskel.h" +#include "fentry_test.skel.h" #include "fentry_many_args.skel.h" =20 -static int fentry_test_common(struct fentry_test_lskel *fentry_skel) +static int fentry_test_check(struct fentry_test *fentry_skel) { + LIBBPF_OPTS(bpf_test_run_opts, topts); int err, prog_fd, i; - int link_fd; __u64 *result; - LIBBPF_OPTS(bpf_test_run_opts, topts); - - err =3D fentry_test_lskel__attach(fentry_skel); - if (!ASSERT_OK(err, "fentry_attach")) - return err; =20 - /* Check that already linked program can't be attached again. */ - link_fd =3D fentry_test_lskel__test1__attach(fentry_skel); - if (!ASSERT_LT(link_fd, 0, "fentry_attach_link")) - return -1; - - prog_fd =3D fentry_skel->progs.test1.prog_fd; + prog_fd =3D bpf_program__fd(fentry_skel->progs.test1); err =3D bpf_prog_test_run_opts(prog_fd, &topts); ASSERT_OK(err, "test_run"); ASSERT_EQ(topts.retval, 0, "test_run"); @@ -31,7 +21,28 @@ static int fentry_test_common(struct fentry_test_lskel *= fentry_skel) return -1; } =20 - fentry_test_lskel__detach(fentry_skel); + return 0; +} + +static int fentry_test_common(struct fentry_test *fentry_skel) +{ + struct bpf_link *link; + int err; + + err =3D fentry_test__attach(fentry_skel); + if (!ASSERT_OK(err, "fentry_attach")) + return err; + + /* Check that already linked program can't be attached again. */ + link =3D bpf_program__attach(fentry_skel->progs.test1); + if (!ASSERT_ERR_PTR(link, "fentry_attach_link")) + return -1; + + err =3D fentry_test_check(fentry_skel); + if (!ASSERT_OK(err, "fentry_test_check")) + return err; + + fentry_test__detach(fentry_skel); =20 /* zero results for re-attach test */ memset(fentry_skel->bss, 0, sizeof(*fentry_skel->bss)); @@ -40,10 +51,10 @@ static int fentry_test_common(struct fentry_test_lskel = *fentry_skel) =20 static void fentry_test(void) { - struct fentry_test_lskel *fentry_skel =3D NULL; + struct fentry_test *fentry_skel =3D NULL; int err; =20 - fentry_skel =3D fentry_test_lskel__open_and_load(); + fentry_skel =3D fentry_test__open_and_load(); if (!ASSERT_OK_PTR(fentry_skel, "fentry_skel_load")) goto cleanup; =20 @@ -55,7 +66,7 @@ static void fentry_test(void) ASSERT_OK(err, "fentry_second_attach"); =20 cleanup: - fentry_test_lskel__destroy(fentry_skel); + fentry_test__destroy(fentry_skel); } =20 static void fentry_many_args(void) @@ -84,10 +95,42 @@ static void fentry_many_args(void) fentry_many_args__destroy(fentry_skel); } =20 +static void fentry_multi_test(void) +{ + struct fentry_test *fentry_skel =3D NULL; + int err, prog_cnt; + + fentry_skel =3D fentry_test__open(); + if (!ASSERT_OK_PTR(fentry_skel, "fentry_skel_open")) + goto cleanup; + + prog_cnt =3D sizeof(fentry_skel->progs) / sizeof(long); + err =3D bpf_to_tracing_multi((void *)&fentry_skel->progs, prog_cnt); + if (!ASSERT_OK(err, "fentry_to_multi")) + goto cleanup; + + err =3D fentry_test__load(fentry_skel); + if (!ASSERT_OK(err, "fentry_skel_load")) + goto cleanup; + + err =3D bpf_attach_as_tracing_multi((void *)&fentry_skel->progs, + prog_cnt, + (void *)&fentry_skel->links); + if (!ASSERT_OK(err, "fentry_attach_multi")) + goto cleanup; + + err =3D fentry_test_check(fentry_skel); + ASSERT_OK(err, "fentry_first_attach"); +cleanup: + fentry_test__destroy(fentry_skel); +} + void test_fentry_test(void) { if (test__start_subtest("fentry")) fentry_test(); + if (test__start_subtest("fentry_multi")) + fentry_multi_test(); if (test__start_subtest("fentry_many_args")) fentry_many_args(); } diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_test.c b/tools/te= sting/selftests/bpf/prog_tests/fexit_test.c index 1c13007e37dd..5652d02b3ad9 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_test.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_test.c @@ -1,26 +1,16 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2019 Facebook */ #include -#include "fexit_test.lskel.h" +#include "fexit_test.skel.h" #include "fexit_many_args.skel.h" =20 -static int fexit_test_common(struct fexit_test_lskel *fexit_skel) +static int fexit_test_check(struct fexit_test *fexit_skel) { + LIBBPF_OPTS(bpf_test_run_opts, topts); int err, prog_fd, i; - int link_fd; __u64 *result; - LIBBPF_OPTS(bpf_test_run_opts, topts); - - err =3D fexit_test_lskel__attach(fexit_skel); - if (!ASSERT_OK(err, "fexit_attach")) - return err; =20 - /* Check that already linked program can't be attached again. */ - link_fd =3D fexit_test_lskel__test1__attach(fexit_skel); - if (!ASSERT_LT(link_fd, 0, "fexit_attach_link")) - return -1; - - prog_fd =3D fexit_skel->progs.test1.prog_fd; + prog_fd =3D bpf_program__fd(fexit_skel->progs.test1); err =3D bpf_prog_test_run_opts(prog_fd, &topts); ASSERT_OK(err, "test_run"); ASSERT_EQ(topts.retval, 0, "test_run"); @@ -31,7 +21,28 @@ static int fexit_test_common(struct fexit_test_lskel *fe= xit_skel) return -1; } =20 - fexit_test_lskel__detach(fexit_skel); + return 0; +} + +static int fexit_test_common(struct fexit_test *fexit_skel) +{ + struct bpf_link *link; + int err; + + err =3D fexit_test__attach(fexit_skel); + if (!ASSERT_OK(err, "fexit_attach")) + return err; + + /* Check that already linked program can't be attached again. */ + link =3D bpf_program__attach(fexit_skel->progs.test1); + if (!ASSERT_ERR_PTR(link, "fexit_attach_link")) + return -1; + + err =3D fexit_test_check(fexit_skel); + if (!ASSERT_OK(err, "fexit_test_check")) + return err; + + fexit_test__detach(fexit_skel); =20 /* zero results for re-attach test */ memset(fexit_skel->bss, 0, sizeof(*fexit_skel->bss)); @@ -40,10 +51,10 @@ static int fexit_test_common(struct fexit_test_lskel *f= exit_skel) =20 static void fexit_test(void) { - struct fexit_test_lskel *fexit_skel =3D NULL; + struct fexit_test *fexit_skel =3D NULL; int err; =20 - fexit_skel =3D fexit_test_lskel__open_and_load(); + fexit_skel =3D fexit_test__open_and_load(); if (!ASSERT_OK_PTR(fexit_skel, "fexit_skel_load")) goto cleanup; =20 @@ -55,7 +66,7 @@ static void fexit_test(void) ASSERT_OK(err, "fexit_second_attach"); =20 cleanup: - fexit_test_lskel__destroy(fexit_skel); + fexit_test__destroy(fexit_skel); } =20 static void fexit_many_args(void) @@ -84,10 +95,42 @@ static void fexit_many_args(void) fexit_many_args__destroy(fexit_skel); } =20 +static void fexit_test_multi(void) +{ + struct fexit_test *fexit_skel =3D NULL; + int err, prog_cnt; + + fexit_skel =3D fexit_test__open(); + if (!ASSERT_OK_PTR(fexit_skel, "fexit_skel_open")) + goto cleanup; + + prog_cnt =3D sizeof(fexit_skel->progs) / sizeof(long); + err =3D bpf_to_tracing_multi((void *)&fexit_skel->progs, prog_cnt); + if (!ASSERT_OK(err, "fexit_to_multi")) + goto cleanup; + + err =3D fexit_test__load(fexit_skel); + if (!ASSERT_OK(err, "fexit_skel_load")) + goto cleanup; + + err =3D bpf_attach_as_tracing_multi((void *)&fexit_skel->progs, + prog_cnt, + (void *)&fexit_skel->links); + if (!ASSERT_OK(err, "fexit_attach_multi")) + goto cleanup; + + err =3D fexit_test_check(fexit_skel); + ASSERT_OK(err, "fexit_first_attach"); +cleanup: + fexit_test__destroy(fexit_skel); +} + void test_fexit_test(void) { if (test__start_subtest("fexit")) fexit_test(); + if (test__start_subtest("fexit_multi")) + fexit_test_multi(); if (test__start_subtest("fexit_many_args")) fexit_many_args(); } diff --git a/tools/testing/selftests/bpf/prog_tests/modify_return.c b/tools= /testing/selftests/bpf/prog_tests/modify_return.c index a70c99c2f8c8..3ca454379e90 100644 --- a/tools/testing/selftests/bpf/prog_tests/modify_return.c +++ b/tools/testing/selftests/bpf/prog_tests/modify_return.c @@ -49,6 +49,56 @@ static void run_test(__u32 input_retval, __u16 want_side= _effect, __s16 want_ret) modify_return__destroy(skel); } =20 +static void run_multi_test(__u32 input_retval, __u16 want_side_effect, __s= 16 want_ret) +{ + struct modify_return *skel =3D NULL; + int err, prog_fd, prog_cnt; + __u16 side_effect; + __s16 ret; + LIBBPF_OPTS(bpf_test_run_opts, topts); + + skel =3D modify_return__open(); + if (!ASSERT_OK_PTR(skel, "skel_open")) + goto cleanup; + + /* stack function args is not supported by tracing multi-link yet, + * so we only enable the bpf progs without stack function args. + */ + bpf_program__set_expected_attach_type(skel->progs.fentry_test, + BPF_TRACE_FENTRY_MULTI); + bpf_program__set_expected_attach_type(skel->progs.fexit_test, + BPF_TRACE_FEXIT_MULTI); + bpf_program__set_expected_attach_type(skel->progs.fmod_ret_test, + BPF_MODIFY_RETURN_MULTI); + + err =3D modify_return__load(skel); + if (!ASSERT_OK(err, "skel_load")) + goto cleanup; + + prog_cnt =3D sizeof(skel->progs) / sizeof(long); + err =3D bpf_attach_as_tracing_multi((void *)&skel->progs, + prog_cnt, + (void *)&skel->links); + if (!ASSERT_OK(err, "modify_return__attach failed")) + goto cleanup; + + skel->bss->input_retval =3D input_retval; + prog_fd =3D bpf_program__fd(skel->progs.fmod_ret_test); + err =3D bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + + side_effect =3D UPPER(topts.retval); + ret =3D LOWER(topts.retval); + + ASSERT_EQ(ret, want_ret, "test_run ret"); + ASSERT_EQ(side_effect, want_side_effect, "modify_return side_effect"); + ASSERT_EQ(skel->bss->fentry_result, 1, "modify_return fentry_result"); + ASSERT_EQ(skel->bss->fexit_result, 1, "modify_return fexit_result"); + ASSERT_EQ(skel->bss->fmod_ret_result, 1, "modify_return fmod_ret_result"); +cleanup: + modify_return__destroy(skel); +} + /* TODO: conflict with get_func_ip_test */ void serial_test_modify_return(void) { @@ -59,3 +109,13 @@ void serial_test_modify_return(void) 0 /* want_side_effect */, -EINVAL * 2 /* want_ret */); } + +void serial_test_modify_return_multi(void) +{ + run_multi_test(0 /* input_retval */, + 2 /* want_side_effect */, + 33 /* want_ret */); + run_multi_test(-EINVAL /* input_retval */, + 1 /* want_side_effect */, + -EINVAL + 29 /* want_ret */); +} diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c b/= tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c new file mode 100644 index 000000000000..f730e26be911 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/tracing_multi_link.c @@ -0,0 +1,276 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ + +#include +#include "bpf/libbpf_internal.h" + +#include "tracing_multi_test.skel.h" +#include "tracing_multi_override.skel.h" +#include "fentry_multi_empty.skel.h" + +static void test_run(struct tracing_multi_test *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + skel->bss->pid =3D getpid(); + prog_fd =3D bpf_program__fd(skel->progs.fentry_cookie_test1); + err =3D bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->fentry_test1_result, 1, "fentry_test1_result"); + ASSERT_EQ(skel->bss->fentry_test2_result, 1, "fentry_test2_result"); + ASSERT_EQ(skel->bss->fentry_test3_result, 1, "fentry_test3_result"); + ASSERT_EQ(skel->bss->fentry_test4_result, 1, "fentry_test4_result"); + ASSERT_EQ(skel->bss->fentry_test5_result, 1, "fentry_test5_result"); + ASSERT_EQ(skel->bss->fentry_test6_result, 1, "fentry_test6_result"); + ASSERT_EQ(skel->bss->fentry_test7_result, 1, "fentry_test7_result"); + ASSERT_EQ(skel->bss->fentry_test8_result, 1, "fentry_test8_result"); +} + +static void test_skel_auto_api(void) +{ + struct tracing_multi_test *skel; + int err; + + skel =3D tracing_multi_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + return; + + /* disable all programs that should fail */ + bpf_program__set_autoattach(skel->progs.fentry_fail_test1, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test2, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test3, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test4, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test5, false); + bpf_program__set_autoattach(skel->progs.fentry_fail_test6, false); + + bpf_program__set_autoattach(skel->progs.fexit_fail_test1, false); + bpf_program__set_autoattach(skel->progs.fexit_fail_test2, false); + bpf_program__set_autoattach(skel->progs.fexit_fail_test3, false); + + err =3D tracing_multi_test__attach(skel); + if (!ASSERT_OK(err, "tracing_multi_test__attach")) + goto cleanup; + + test_run(skel); + +cleanup: + tracing_multi_test__destroy(skel); +} + +static void test_skel_manual_api(void) +{ + struct tracing_multi_test *skel; + struct bpf_link *link; + int err; + + skel =3D tracing_multi_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + return; + +#define ATTACH_PROG(name, success) \ +do { \ + link =3D bpf_program__attach(skel->progs.name); \ + err =3D libbpf_get_error(link); \ + if (!ASSERT_OK(success ? err : !err, \ + "bpf_program__attach: " #name)) \ + goto cleanup; \ + skel->links.name =3D err ? NULL : link; \ +} while (0) + + ATTACH_PROG(fentry_success_test1, true); + ATTACH_PROG(fentry_success_test2, true); + ATTACH_PROG(fentry_success_test3, true); + ATTACH_PROG(fentry_success_test4, true); + + ATTACH_PROG(fexit_success_test1, true); + ATTACH_PROG(fexit_success_test2, true); + + ATTACH_PROG(fentry_fail_test1, false); + ATTACH_PROG(fentry_fail_test2, false); + ATTACH_PROG(fentry_fail_test3, false); + ATTACH_PROG(fentry_fail_test4, false); + ATTACH_PROG(fentry_fail_test5, false); + ATTACH_PROG(fentry_fail_test6, false); + + ATTACH_PROG(fexit_fail_test1, false); + ATTACH_PROG(fexit_fail_test2, false); + ATTACH_PROG(fexit_fail_test3, false); + + ATTACH_PROG(fentry_cookie_test1, true); + + test_run(skel); + +cleanup: + tracing_multi_test__destroy(skel); +} + +static void test_attach_api(void) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + struct tracing_multi_test *skel; + struct bpf_link *link; + const char *syms[8] =3D { + "bpf_fentry_test1", + "bpf_fentry_test2", + "bpf_fentry_test3", + "bpf_fentry_test4", + "bpf_fentry_test5", + "bpf_fentry_test6", + "bpf_fentry_test7", + "bpf_fentry_test8", + }; + __u64 cookies[] =3D {1, 7, 2, 3, 4, 5, 6, 8}; + + skel =3D tracing_multi_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + return; + + opts.syms =3D syms; + opts.cookies =3D cookies; + opts.cnt =3D ARRAY_SIZE(syms); + link =3D bpf_program__attach_trace_multi_opts(skel->progs.fentry_cookie_t= est1, + &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_trace_multi_opts")) + goto cleanup; + skel->links.fentry_cookie_test1 =3D link; + + skel->bss->test_cookie =3D true; + test_run(skel); +cleanup: + tracing_multi_test__destroy(skel); +} + +static void test_attach_bench(bool kernel) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + struct fentry_multi_empty *skel; + long attach_start_ns, attach_end_ns; + long detach_start_ns, detach_end_ns; + double attach_delta, detach_delta; + struct bpf_link *link =3D NULL; + char **syms =3D NULL; + size_t cnt =3D 0; + + if (!ASSERT_OK(bpf_get_ksyms(&syms, &cnt, kernel), "get_syms")) + return; + + skel =3D fentry_multi_empty__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_empty__open_and_load")) + goto cleanup; + + opts.syms =3D (const char **) syms; + opts.cnt =3D cnt; + opts.skip_invalid =3D true; + + attach_start_ns =3D get_time_ns(); + link =3D bpf_program__attach_trace_multi_opts(skel->progs.fentry_multi_em= pty, + &opts); + attach_end_ns =3D get_time_ns(); + + if (!ASSERT_OK_PTR(link, "bpf_program__attach_trace_multi_opts")) + return; + + detach_start_ns =3D get_time_ns(); + bpf_link__destroy(link); + detach_end_ns =3D get_time_ns(); + + attach_delta =3D (attach_end_ns - attach_start_ns) / 1000000000.0; + detach_delta =3D (detach_end_ns - detach_start_ns) / 1000000000.0; + + printf("%s: found %lu functions\n", __func__, opts.cnt); + printf("%s: attached in %7.3lfs\n", __func__, attach_delta); + printf("%s: detached in %7.3lfs\n", __func__, detach_delta); + +cleanup: + fentry_multi_empty__destroy(skel); + if (syms) + free(syms); +} + +static void test_attach_override(bool fentry_over_multi) +{ + struct tracing_multi_override *skel; + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct bpf_link *link; + int err, prog_fd; + + skel =3D tracing_multi_override__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open_and_load")) + goto cleanup; + + if (fentry_over_multi) { + ATTACH_PROG(fentry_multi_override_test1, true); + ATTACH_PROG(fentry_override_test1, true); + } else { + ATTACH_PROG(fentry_override_test1, true); + ATTACH_PROG(fentry_multi_override_test1, true); + } + + prog_fd =3D bpf_program__fd(skel->progs.fentry_multi_override_test1); + err =3D bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "test_run"); + + ASSERT_EQ(skel->data->fentry_override_test1_result, 3, + "fentry_override_test1_result"); +cleanup: + tracing_multi_override__destroy(skel); +} + +static void test_attach_multi(void) +{ + struct tracing_multi_override *skel; + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct bpf_link *link; + int err, prog_fd; + + skel =3D tracing_multi_override__open(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_test__open")) + goto cleanup; + + /* don't load fentry_override_test1, it will create a trampoline */ + bpf_program__set_autoload(skel->progs.fentry_override_test1, false); + err =3D tracing_multi_override__load(skel); + if (!ASSERT_OK(err, "tracing_multi_test__load")) + goto cleanup; + + ATTACH_PROG(fentry_multi_override_test1, true); + ATTACH_PROG(fentry_multi_override_test2, true); + + prog_fd =3D bpf_program__fd(skel->progs.fentry_multi_override_test1); + err =3D bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "test_run"); + + ASSERT_EQ(skel->data->fentry_override_test1_result, 4, + "fentry_override_test1_result"); +cleanup: + tracing_multi_override__destroy(skel); +} + +void serial_test_tracing_multi_attach_bench(void) +{ + if (test__start_subtest("kernel")) + test_attach_bench(true); + if (test__start_subtest("modules")) + test_attach_bench(false); +} + +void test_tracing_multi_attach_test(void) +{ + if (test__start_subtest("skel_auto_api")) + test_skel_auto_api(); + if (test__start_subtest("skel_manual_api")) + test_skel_manual_api(); + if (test__start_subtest("attach_api")) + test_attach_api(); + if (test__start_subtest("attach_over_multi")) + test_attach_override(true); + if (test__start_subtest("attach_over_fentry")) + test_attach_override(false); + if (test__start_subtest("attach_multi")) + test_attach_multi(); +} diff --git a/tools/testing/selftests/bpf/progs/fentry_multi_empty.c b/tools= /testing/selftests/bpf/progs/fentry_multi_empty.c new file mode 100644 index 000000000000..a09ba216dff8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/fentry_multi_empty.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ +#include +#include +#include + +char _license[] SEC("license") =3D "GPL"; + +SEC("fentry.multi/bpf_fentry_test1") +int BPF_PROG(fentry_multi_empty) +{ + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/tracing_multi_override.c b/t= ools/testing/selftests/bpf/progs/tracing_multi_override.c new file mode 100644 index 000000000000..8001be433914 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tracing_multi_override.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ +#include +#include +#include + +__u64 fentry_override_test1_result =3D 1; + +SEC("fentry.multi/bpf_fentry_test1") +int BPF_PROG(fentry_multi_override_test1) +{ + fentry_override_test1_result++; + return 0; +} + +SEC("fentry.multi/bpf_fentry_test1") +int BPF_PROG(fentry_multi_override_test2) +{ + fentry_override_test1_result <<=3D 1; + return 0; +} + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(fentry_override_test1) +{ + fentry_override_test1_result++; + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/tracing_multi_test.c b/tools= /testing/selftests/bpf/progs/tracing_multi_test.c new file mode 100644 index 000000000000..fa27851896b9 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tracing_multi_test.c @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ +#include +#include +#include + +char _license[] SEC("license") =3D "GPL"; + +struct bpf_testmod_struct_arg_1 { + int a; +}; +struct bpf_testmod_struct_arg_2 { + long a; + long b; +}; + +__u64 test_result =3D 0; + +int pid =3D 0; +int test_cookie =3D 0; + +__u64 fentry_test1_result =3D 0; +__u64 fentry_test2_result =3D 0; +__u64 fentry_test3_result =3D 0; +__u64 fentry_test4_result =3D 0; +__u64 fentry_test5_result =3D 0; +__u64 fentry_test6_result =3D 0; +__u64 fentry_test7_result =3D 0; +__u64 fentry_test8_result =3D 0; + +extern const void bpf_fentry_test1 __ksym; +extern const void bpf_fentry_test2 __ksym; +extern const void bpf_fentry_test3 __ksym; +extern const void bpf_fentry_test4 __ksym; +extern const void bpf_fentry_test5 __ksym; +extern const void bpf_fentry_test6 __ksym; +extern const void bpf_fentry_test7 __ksym; +extern const void bpf_fentry_test8 __ksym; + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_13") +int BPF_PROG2(fentry_success_test1, struct bpf_testmod_struct_arg_2, a) +{ + test_result =3D a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_ar= g_10") +int BPF_PROG2(fentry_success_test2, int, a, struct bpf_testmod_struct_arg_= 2, b) +{ + test_result =3D a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_4") +int BPF_PROG2(fentry_success_test3, struct bpf_testmod_struct_arg_2, a, in= t, b, + int, c) +{ + test_result =3D c; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_2") +int BPF_PROG2(fentry_success_test4, struct bpf_testmod_struct_arg_2, a, in= t, b, + int, c) +{ + test_result =3D c; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_1") +int BPF_PROG2(fentry_fail_test1, struct bpf_testmod_struct_arg_2, a) +{ + test_result =3D a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_2") +int BPF_PROG2(fentry_fail_test2, struct bpf_testmod_struct_arg_2, a) +{ + test_result =3D a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_4") +int BPF_PROG2(fentry_fail_test3, struct bpf_testmod_struct_arg_2, a) +{ + test_result =3D a.a + a.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_ar= g_2") +int BPF_PROG2(fentry_fail_test4, int, a, struct bpf_testmod_struct_arg_2, = b) +{ + test_result =3D a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_ar= g_13") +int BPF_PROG2(fentry_fail_test5, int, a, struct bpf_testmod_struct_arg_2, = b) +{ + test_result =3D a + b.a + b.b; + return 0; +} + +SEC("fentry.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_ar= g_12") +int BPF_PROG2(fentry_fail_test6, struct bpf_testmod_struct_arg_2, a, int, = b, + int, c) +{ + test_result =3D c; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg= _2,bpf_testmod_test_struct_arg_3") +int BPF_PROG2(fexit_success_test1, struct bpf_testmod_struct_arg_2, a, int= , b, + int, c, int, retval) +{ + test_result =3D retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg= _12") +int BPF_PROG2(fexit_success_test2, int, a, struct bpf_testmod_struct_arg_2= , b, + int, c, int, retval) +{ + test_result =3D a + b.a + b.b + retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_1,bpf_testmod_test_struct_arg= _4") +int BPF_PROG2(fexit_fail_test1, struct bpf_testmod_struct_arg_2, a, int, b, + int, c, int, retval) +{ + test_result =3D retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg= _10") +int BPF_PROG2(fexit_fail_test2, int, a, struct bpf_testmod_struct_arg_2, b, + int, c, int, retval) +{ + test_result =3D a + b.a + b.b + retval; + return 0; +} + +SEC("fexit.multi/bpf_testmod_test_struct_arg_2,bpf_testmod_test_struct_arg= _11") +int BPF_PROG2(fexit_fail_test3, int, a, struct bpf_testmod_struct_arg_2, b, + int, c, int, retval) +{ + test_result =3D a + b.a + b.b + retval; + return 0; +} + +static void tracing_multi_check_cookie(unsigned long long *ctx) +{ + if (bpf_get_current_pid_tgid() >> 32 !=3D pid) + return; + + __u64 cookie =3D test_cookie ? bpf_get_attach_cookie(ctx) : 0; + __u64 addr =3D bpf_get_func_ip(ctx); + +#define SET(__var, __addr, __cookie) ({ \ + if (((const void *) addr =3D=3D __addr) && \ + (!test_cookie || (cookie =3D=3D __cookie))) \ + __var =3D 1; \ +}) + SET(fentry_test1_result, &bpf_fentry_test1, 1); + SET(fentry_test2_result, &bpf_fentry_test2, 7); + SET(fentry_test3_result, &bpf_fentry_test3, 2); + SET(fentry_test4_result, &bpf_fentry_test4, 3); + SET(fentry_test5_result, &bpf_fentry_test5, 4); + SET(fentry_test6_result, &bpf_fentry_test6, 5); + SET(fentry_test7_result, &bpf_fentry_test7, 6); + SET(fentry_test8_result, &bpf_fentry_test8, 8); +} + +SEC("fentry.multi/bpf_fentry_test1,bpf_fentry_test2,bpf_fentry_test3,bpf_f= entry_test4,bpf_fentry_test5,bpf_fentry_test6,bpf_fentry_test7,bpf_fentry_t= est8") +int BPF_PROG(fentry_cookie_test1) +{ + tracing_multi_check_cookie(ctx); + return 0; +} diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/t= esting/selftests/bpf/test_kmods/bpf_testmod.c index 2e54b95ad898..ebc4d5204136 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -128,6 +128,30 @@ bpf_testmod_test_struct_arg_9(u64 a, void *b, short c,= int d, void *e, char f, return bpf_testmod_test_struct_arg_result; } =20 +noinline int +bpf_testmod_test_struct_arg_10(int a, struct bpf_testmod_struct_arg_2 b) { + bpf_testmod_test_struct_arg_result =3D a + b.a + b.b; + return bpf_testmod_test_struct_arg_result; +} + +noinline struct bpf_testmod_struct_arg_2 * +bpf_testmod_test_struct_arg_11(int a, struct bpf_testmod_struct_arg_2 b, i= nt c) { + bpf_testmod_test_struct_arg_result =3D a + b.a + b.b + c; + return (void *)bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_struct_arg_12(int a, struct bpf_testmod_struct_arg_2 b, i= nt *c) { + bpf_testmod_test_struct_arg_result =3D a + b.a + b.b + *c; + return bpf_testmod_test_struct_arg_result; +} + +noinline int +bpf_testmod_test_struct_arg_13(struct bpf_testmod_struct_arg_2 b) { + bpf_testmod_test_struct_arg_result =3D b.a + b.b; + return bpf_testmod_test_struct_arg_result; +} + noinline int bpf_testmod_test_arg_ptr_to_struct(struct bpf_testmod_struct_arg_1 *a) { bpf_testmod_test_struct_arg_result =3D a->a; diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selft= ests/bpf/test_progs.c index 26cc50bbed8b..286a30c1c7ae 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -685,6 +685,68 @@ static bool skip_entry(char *name) if (!strncmp(name, "__ftrace_invalid_address__", sizeof("__ftrace_invalid_address__") - 1)) return true; + + /* skip functions in "btf_id_deny" */ + if (!strcmp(name, "migrate_disable")) + return true; + if (!strcmp(name, "migrate_enable")) + return true; + if (!strcmp(name, "rcu_read_unlock_strict")) + return true; + if (!strcmp(name, "preempt_count_add")) + return true; + if (!strcmp(name, "preempt_count_sub")) + return true; + if (!strcmp(name, "__rcu_read_lock")) + return true; + if (!strcmp(name, "__rcu_read_unlock")) + return true; + + /* Following symbols have multi definition in kallsyms, take + * "t_next" for example: + * + * ffffffff813c10d0 t t_next + * ffffffff813d31b0 t t_next + * ffffffff813e06b0 t t_next + * ffffffff813eb360 t t_next + * ffffffff81613360 t t_next + * + * but only one of them have corresponding mrecord: + * ffffffff81613364 t_next + * + * The kernel search the target function address by the symbol + * name "t_next" with kallsyms_lookup_name() during attaching + * and the function "0xffffffff813c10d0" can be matched, which + * doesn't have a corresponding mrecord. And this will make + * the attach failing. Skip the functions like this. + * + * The list maybe not whole, so we still can fail...... + */ + if (!strcmp(name, "kill_pid_usb_asyncio")) + return true; + if (!strcmp(name, "t_next")) + return true; + if (!strcmp(name, "t_stop")) + return true; + if (!strcmp(name, "t_start")) + return true; + if (!strcmp(name, "p_next")) + return true; + if (!strcmp(name, "p_stop")) + return true; + if (!strcmp(name, "p_start")) + return true; + if (!strcmp(name, "mem32_serial_out")) + return true; + if (!strcmp(name, "mem32_serial_in")) + return true; + if (!strcmp(name, "io_serial_in")) + return true; + if (!strcmp(name, "io_serial_out")) + return true; + if (!strcmp(name, "event_callback")) + return true; + return false; } =20 @@ -860,6 +922,56 @@ int bpf_get_addrs(unsigned long **addrsp, size_t *cntp= , bool kernel) return err; } =20 +int bpf_to_tracing_multi(struct bpf_program **progs, int prog_cnt) +{ + enum bpf_attach_type type; + int i, err; + + for (i =3D 0; i < prog_cnt; i++) { + type =3D bpf_program__get_expected_attach_type(progs[i]); + if (type =3D=3D BPF_TRACE_FENTRY) + type =3D BPF_TRACE_FENTRY_MULTI; + else if (type =3D=3D BPF_TRACE_FEXIT) + type =3D BPF_TRACE_FEXIT_MULTI; + else if (type =3D=3D BPF_MODIFY_RETURN) + type =3D BPF_MODIFY_RETURN_MULTI; + else + continue; + err =3D bpf_program__set_expected_attach_type(progs[i], type); + if (err) + return err; + } + + return 0; +} + +int bpf_attach_as_tracing_multi(struct bpf_program **progs, int prog_cnt, + struct bpf_link **link) +{ + struct bpf_link *__link; + int err, type; + + for (int i =3D 0; i < prog_cnt; i++) { + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + + type =3D bpf_program__get_expected_attach_type(progs[i]); + if (type !=3D BPF_TRACE_FENTRY_MULTI && + type !=3D BPF_TRACE_FEXIT_MULTI && + type !=3D BPF_MODIFY_RETURN_MULTI) + continue; + + opts.attach_tracing =3D true; + __link =3D bpf_program__attach_trace_multi_opts(progs[i], &opts); + err =3D libbpf_get_error(link); + if (err) + return err; + + link[i] =3D __link; + } + + return 0; +} + int compare_map_keys(int map1_fd, int map2_fd) { __u32 key, next_key; diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selft= ests/bpf/test_progs.h index 477a785c8b5f..d0878bf605af 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -470,6 +470,9 @@ int get_bpf_max_tramp_links_from(struct btf *btf); int get_bpf_max_tramp_links(void); int bpf_get_ksyms(char ***symsp, size_t *cntp, bool kernel); int bpf_get_addrs(unsigned long **addrsp, size_t *cntp, bool kernel); +int bpf_to_tracing_multi(struct bpf_program **progs, int prog_cnt); +int bpf_attach_as_tracing_multi(struct bpf_program **progs, int prog_cnt, + struct bpf_link **link); =20 struct netns_obj; struct netns_obj *netns_new(const char *name, bool open); --=20 2.39.5 From nobody Sun Feb 8 13:42:12 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC234221277; Wed, 28 May 2025 03:50:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404233; cv=none; b=Z90eO4r8gQqIt9CzVPun90A8PPtfT/2d/Kf1/5JDNG7zGgTriCR93kezybo2gueRmQZX77BeM7QYMYtKeYL2tlJJxSUsnX9Y+CjHY6zHmxdSl32tn1j79D7rUPn0aYwzkv7+rpevPifKDr+tCyTo78B3VMwUe3VQ4TlTzAsZD1c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748404233; c=relaxed/simple; bh=rciDrG8G6L4Dh6dkq2H8+0cYFsdIDFcdIk/SXzFy2Vw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hKQ1aLovyIqpPXRNtRR6T0KXUWGa3UYyP2E3JhUNsJDGIazCO3vnTqDKWt/O5vAQsoDkK0nP6CE1PWO1Lb2IQPErZjwZzcylGoXEWb4JOnTTEehBTZa2FbwN7enVUYJpI6vVWAtZZQ19ml07FisP1qoBKLomQQRT1SLf0+nyb7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mPQCK9uV; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mPQCK9uV" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-23446bb8785so37436685ad.0; Tue, 27 May 2025 20:50:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748404231; x=1749009031; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hu9/X1JOi8Os36x5RU4FpTPbcc8XO6FTrIp+jwMDpec=; b=mPQCK9uVQrI/1QxlD5BoZkw4kJB8zOGFt+dWbUJUmo2FwhuQDYYx8jwlDa/0Hzw735 ihks9MVokUBv6TEDIgvEaWVZsg2hovST3kFymNEHnBziPplKwCmy+9Ucc9HzbGprD3/b IgzHEDWHMsgzpRmsrj+1CKzCxrnfLHGP0+6xkI+KJZCq3AsG2KUjCbcoHDxWquwZT+yY 6IIe/NAuNKwawgsPlk0YHMm2MnLe9ywLEC3WLPbkWDkBMmgesiasw8Q3v3jeTT6SIprj +NdqQYm0PRkTiTf8NZjLXn6fMxY59ME9NhroTZtq9T1GMBCsiNOLh8O3SaR3Q69/u2Qu xR9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748404231; x=1749009031; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hu9/X1JOi8Os36x5RU4FpTPbcc8XO6FTrIp+jwMDpec=; b=dXFifQfGWYQFC1PIZc/FsAkB0+y08WC3YwjgRHNJpsAQyV7xk4CYUZxV9lP3qC1RGc J7XDnTR7Hko2Dq0JqMGMpFxNmvSr6upJMYDSn3RVovm7hmm0c2c6WGro3iCKHme+b9LT R5hjUvrnhpP5WilnmQOkJeKmf4Oq0EbaZy0StK/aBZX4cwvfExXFw3IeBP2++3eNskNA 28uXmHiDtHol/mgB9pg+1dFaLqCF805D0iwJ5Yj3E0BPyOPev3vq6fSZ04PUhSglCORI IPQzuJo+iRx38b757m++gsQqRe9kiPFRpBoPhKHYrOFDKHz3188sNv+pxlfRb9GCH0ya juyA== X-Forwarded-Encrypted: i=1; AJvYcCUKIym+JVckzYCFeSuJ9s8zdiSrzlDPCVX3czHo+tCwREten7DGulVxcHJ1lDwwZDwnWluOpTtWcT3nPWw=@vger.kernel.org X-Gm-Message-State: AOJu0Yx24sShMuGxS2GDTRrv3HlaxKzK0iBd6nOyqy3fPnaaSl8YDkBF zOFXTCkPkOs3CLqAi04MGYFkXx832ikRGCHVXofoySpR+Aa7nTfnFxLw X-Gm-Gg: ASbGnctPEiGRS0BFEhhqUFUq+Td92xpkmmpFr5aL6mMLVT9rs1xXeyChPYA4XKq8Uil 5JwUGF+UpfuLiBRkTUjCd//FiHYqzdrEgwe8RE9u00ZGY3Jsko3hZ6fGfc5VXSwwW1SYKY6PwqU /HfAPURwi6URTnRcvGcoqA4VqQDpnACSUB4VkGXGZ9guMutgK1XI86s7UpFAJi7Nqx8GpfEfvLa HWdw3FO0LduAYWIkgmDC/Rvwytvrr+p9qDBD82n7a8Rl3Qyac9x/8AqAvxmKUnQC/0CQVSKZcju FljQUOqm346S8dmFxZ7JU8AWRzPDstAvCkGJICE4F93IzbF84mtMDKbfhLzMlRNbZFWn X-Google-Smtp-Source: AGHT+IHb6XuBb0eSurxKAfp2Gz9/QNo3XaqUB3jxZEb20L9m4arA2I30pbXGEjnNIAQjFRKJrZLISQ== X-Received: by 2002:a17:902:e5cb:b0:234:c549:da10 with SMTP id d9443c01a7336-234c549dbedmr29711375ad.47.1748404230906; Tue, 27 May 2025 20:50:30 -0700 (PDT) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-234d35ac417sm2074505ad.169.2025.05.27.20.50.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 May 2025 20:50:30 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: alexei.starovoitov@gmail.com, rostedt@goodmis.org, jolsa@kernel.org Cc: bpf@vger.kernel.org, Menglong Dong , linux-kernel@vger.kernel.org Subject: [PATCH bpf-next 25/25] selftests/bpf: add performance bench test for trace prog Date: Wed, 28 May 2025 11:47:12 +0800 Message-Id: <20250528034712.138701-26-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250528034712.138701-1-dongml2@chinatelecom.cn> References: <20250528034712.138701-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add testcase for the performance of the trace bpf progs. In this testcase, bpf_fentry_test1() will be called 10000000 times in bpf_testmod_bench_run, and the time consumed will be returned. Following cases is considered: - nop: nothing is attached to bpf_fentry_test1() - fentry: a empty FENTRY bpf program is attached to bpf_fentry_test1() - fentry_multi_single: a empty FENTRY_MULTI bpf program is attached to bpf_fentry_test1() - fentry_multi_all: a empty FENTRY_MULTI bpf program is attached to all the kernel functions - kprobe_multi_single: a empty KPROBE_MULTI bpf program is attached to bpf_fentry_test1() - kprobe_multi_all: a empty KPROBE_MULTI bpf program is attached to all the kernel functions And we can get the result by running: ./test_progs -t tracing_multi_bench -v | grep time Signed-off-by: Menglong Dong --- .../selftests/bpf/prog_tests/trace_bench.c | 149 ++++++++++++++++++ .../selftests/bpf/progs/fentry_empty.c | 13 ++ .../testing/selftests/bpf/progs/trace_bench.c | 21 +++ .../selftests/bpf/test_kmods/bpf_testmod.c | 16 ++ 4 files changed, 199 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/trace_bench.c create mode 100644 tools/testing/selftests/bpf/progs/fentry_empty.c create mode 100644 tools/testing/selftests/bpf/progs/trace_bench.c diff --git a/tools/testing/selftests/bpf/prog_tests/trace_bench.c b/tools/t= esting/selftests/bpf/prog_tests/trace_bench.c new file mode 100644 index 000000000000..673c9acf358c --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/trace_bench.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ + +#include +#include "bpf/libbpf_internal.h" + +#include "fentry_multi_empty.skel.h" +#include "fentry_empty.skel.h" +#include "kprobe_multi_empty.skel.h" +#include "trace_bench.skel.h" + +static void test_bench_run(const char *name) +{ + struct trace_bench *skel; + __u64 bench_result; + int err; + + skel =3D trace_bench__open_and_load(); + if (!ASSERT_OK_PTR(skel, "trace_bench__open_and_load")) + return; + + err =3D trace_bench__attach(skel); + if (!ASSERT_OK(err, "trace_bench__attach")) + goto cleanup; + + ASSERT_OK(trigger_module_test_read(1), "trigger_read"); + + bench_result =3D skel->bss->bench_result / 1000; + printf("bench time for %s: %lld.%03lldms\n", name, bench_result / 1000, + bench_result % 1000); +cleanup: + trace_bench__destroy(skel); +} + +static void test_fentry_multi(bool load_all, char *name) +{ + LIBBPF_OPTS(bpf_trace_multi_opts, opts); + struct fentry_multi_empty *skel; + char **syms =3D NULL; + struct bpf_link *link; + size_t cnt =3D 0; + int err; + + skel =3D fentry_multi_empty__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_empty__open_and_load")) + goto cleanup; + + if (!load_all) { + err =3D fentry_multi_empty__attach(skel); + if (!ASSERT_OK(err, "fentry_multi_empty__attach")) + goto cleanup; + goto do_test; + } + + if (!ASSERT_OK(bpf_get_ksyms(&syms, &cnt, true), "get_syms")) + return; + opts.syms =3D (const char **) syms; + opts.cnt =3D cnt; + opts.skip_invalid =3D true; + link =3D bpf_program__attach_trace_multi_opts(skel->progs.fentry_multi_em= pty, + &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_trace_multi_opts")) + goto cleanup; + skel->links.fentry_multi_empty =3D link; + printf("attach %d functions before testings\n", (int)opts.cnt); + +do_test: + test_bench_run(name); +cleanup: + fentry_multi_empty__destroy(skel); + if (syms) + free(syms); +} + +static void test_fentry_single(void) +{ + struct fentry_empty *skel; + int err; + + skel =3D fentry_empty__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_empty__open_and_load")) + return; + + err =3D fentry_empty__attach(skel); + if (!ASSERT_OK(err, "fentry_empty__attach")) + goto cleanup; + + test_bench_run("fentry_single"); +cleanup: + fentry_empty__destroy(skel); +} + +static void test_kprobe_multi(bool load_all, char *name) +{ + LIBBPF_OPTS(bpf_kprobe_multi_opts, opts); + char *test_func =3D "bpf_fentry_test1"; + struct kprobe_multi_empty *skel; + struct bpf_link *link; + char **syms =3D NULL; + size_t cnt =3D 0; + + if (!ASSERT_OK(bpf_get_ksyms(&syms, &cnt, true), "get_syms")) + return; + + skel =3D kprobe_multi_empty__open_and_load(); + if (!ASSERT_OK_PTR(skel, "kprobe_multi_empty__open_and_load")) + goto cleanup; + + if (load_all) { + opts.syms =3D (const char **) syms; + opts.cnt =3D cnt; + } else { + opts.syms =3D (const char **) &test_func; + opts.cnt =3D 1; + } + link =3D bpf_program__attach_kprobe_multi_opts(skel->progs.test_kprobe_em= pty, + NULL, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_kprobe_multi_opts")) + goto cleanup; + skel->links.test_kprobe_empty =3D link; + + if (load_all) + printf("attach %d functions before testings\n", (int)opts.cnt); + test_bench_run(name); + +cleanup: + kprobe_multi_empty__destroy(skel); + if (syms) + free(syms); +} + +void test_trace_bench(void) +{ + if (test__start_subtest("nop")) + test_bench_run("nop"); + + if (test__start_subtest("fentry_single")) + test_fentry_single(); + + if (test__start_subtest("fentry_multi_single")) + test_fentry_multi(false, "fentry_multi_single"); + if (test__start_subtest("fentry_multi_all")) + test_fentry_multi(true, "fentry_multi_all"); + + if (test__start_subtest("kprobe_multi_single")) + test_kprobe_multi(false, "kprobe_multi_single"); + if (test__start_subtest("kprobe_multi_all")) + test_kprobe_multi(true, "kprobe_multi_all"); +} diff --git a/tools/testing/selftests/bpf/progs/fentry_empty.c b/tools/testi= ng/selftests/bpf/progs/fentry_empty.c new file mode 100644 index 000000000000..f2bfaf04d56a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/fentry_empty.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ +#include +#include +#include + +char _license[] SEC("license") =3D "GPL"; + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(fentry_empty) +{ + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/trace_bench.c b/tools/testin= g/selftests/bpf/progs/trace_bench.c new file mode 100644 index 000000000000..98373871414a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/trace_bench.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 ChinaTelecom */ + +#include +#include +#include + +char _license[] SEC("license") =3D "GPL"; + +__u64 bench_result; + +SEC("fexit.multi/bpf_testmod_bench_run") +int BPF_PROG(fexit_bench_done) +{ + __u64 ret =3D 0; + + bpf_get_func_ret(ctx, &ret); + bench_result =3D ret; + + return 0; +} diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/t= esting/selftests/bpf/test_kmods/bpf_testmod.c index ebc4d5204136..d21775eba211 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -405,6 +405,20 @@ noinline int bpf_testmod_fentry_test11(u64 a, void *b,= short c, int d, return a + (long)b + c + d + (long)e + f + g + h + i + j + k; } =20 +extern int bpf_fentry_test1(int a); +noinline u64 bpf_testmod_bench_run(void) +{ + u64 start =3D ktime_get_boottime_ns(); + u64 time; + + for (int i =3D 0; i < 10000000; i++) + bpf_fentry_test1(i); + + time =3D ktime_get_boottime_ns() - start; + + return time; +} + int bpf_testmod_fentry_ok; =20 noinline ssize_t @@ -443,6 +457,8 @@ bpf_testmod_test_read(struct file *file, struct kobject= *kobj, =20 (void)trace_bpf_testmod_test_raw_tp_null(NULL); =20 + (void)bpf_testmod_bench_run(); + bpf_testmod_test_struct_ops3(); =20 struct_arg3 =3D kmalloc((sizeof(struct bpf_testmod_struct_arg_3) + --=20 2.39.5