From nobody Fri Dec 19 12:06:44 2025 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28BC527144E for ; Fri, 10 Oct 2025 03:39:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760067543; cv=none; b=qBdeOnSZLMjrb8dKHgQUhs0TnYXPFap+ljCGIz8E4kCZYzQ4ZcvZiO3091n3/+wiZSmtVRi7OddAjlASNvCVru5b6XKMYzeJR84ZIN6YQgzgZexsyeYU/hQb0zkroB5I/AccXKGOtik4+6oG1J7Xl1NglGmua/4jEpzLfJdKkCQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760067543; c=relaxed/simple; bh=BpJi/vaZmaMrSd2P2X1czVc4BiMq7wfJtbSVngqW5m0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mLzkAbwLwRxTkfTywCesNrQOLNqp4SILqMU3qkaW/nWHq70EX7G3hu8W6s4lW7kgntl+rH08QnoSWt56m0azUagz9Jv/ox+qjtpshy5r3cFyH+yi9LP6st3KFFroiqjxxBakMdSLT6097MUFZndjfErgg4kwqjQPPgIteipoY3o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hsvWAjRA; arc=none smtp.client-ip=209.85.210.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hsvWAjRA" Received: by mail-pf1-f196.google.com with SMTP id d2e1a72fcca58-78125ed4052so1930585b3a.0 for ; Thu, 09 Oct 2025 20:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1760067539; x=1760672339; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RtrdkRoYKffOfWTuEywjN8EL4Ba6ONqO4F8W9HPcWYc=; b=hsvWAjRAGwQSOiRVOj7znFPqa11b/PNFnvWStn0BVQWrO3mYqGPn+9CiHhqbv7jhOQ qv0YToW3sKC+jUkjsAtvJzEfMK81kPHhIBzdXrcXaCWoQdmV/iZ3skeeb/q6n7RPWBCy aetRG2PXaAnGljzDPqYxI6otWwQAb2/Ba0NMK+7P/rl09FQfnKJHxmBcnKpN8Bg1EWbi JT6/jHKj9QQ2i8NyYh0Vhl/yjkHEsv+2HLn4k901/l1SRSvjS9EA+16/IQkkYqg+nAcc NA3FBmFdXpqXdNvVQMpvBkfkr9R//Nj3YcWHuNUBjaxmvc8+yMzahKLxWNugKKO5js00 4aEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760067539; x=1760672339; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RtrdkRoYKffOfWTuEywjN8EL4Ba6ONqO4F8W9HPcWYc=; b=pNQ8g7ZS9OmDDjKIR85M9nEcdTGOEHYTPY2tC2c8MXwB8BmllqyRhoRhbjCb1/KPbM nhtb4YKMPS5zJ3Nx+IgMx4LtjR6VhJinuTO8YKgLE0ZFmlelibGDurZjxYFCWnnhJ3WL dfj7MqKqgGFoH/rs2zxs56x6ibnkh4bpbY4G/4EZXeC1upbAV4auVG6fzKaaecGRffXO NqvGtu5Ug+VlWiMSkf4Hpf3yGGEa24cHehUsoOVx4IasBD0vhnXhHhZAJ8ayy7uftsOx CQzQKpNS3sUFVkvMNWf3RmItotaW0MmOWo3grTxnh6KpEVJocgS5Hfbl2dWTuVEmNEKK dzsw== X-Forwarded-Encrypted: i=1; AJvYcCXfjK4JHTOd9hOtTmpgbunCil1kNp97HGc2QK+fSvuISI4nwW0ffqjURvHsEzmsXLCPV+WZhY+lGnpz658=@vger.kernel.org X-Gm-Message-State: AOJu0YxOQCzy6eIxqrvAPFXhkGjEEVi7OACI2GHVIkDwkrddVhCKb724 JdRYrh/YhMQpFndTBvm5wI36Zal2b9TytkfVGWUXfmsP3nLNE0vOFS61 X-Gm-Gg: ASbGncuSUQj2pz0R22yiK48lqADm6JxD6In8PvAysK03HJjWybBStCvMtYexdmWGFG4 6Zm+FqxWRyA8oiBj2cDrv1p+417njI/g/xjqL1Sno1b9CucdshDBy4eSeL6eHQe/Vd2jyy1IGZP dFQN2sd8rIW8Wma2HntA8qfI8E1c2v/T03JmArPKkoc2eabJF7O1ghfBqZqwiifnf4pgPiJMp6s Hgao1WZgW3lncsoRw6cr9odPIEFvmNYQjKH6YHsyA5HcDjVzjqZ8VjNSdQfWf7tO0XSEuCivFXt AdzKkKXA0jO9snX/I59vyzRO/Ulj2bQc8ytdhhp/F7h5Lhbp9bozdPigmb2oxp7x0SRQYtGIm8O b837ZVm9hP9kZqyHW/xCTYK7cle94Bba2QdOmfoYIJhh6VeKZ X-Google-Smtp-Source: AGHT+IFtBEHP618h3t53kD4WHUk6lpqNdymniKN8BAkFUVnrFnhXgl8zSMIDyAukKdjyqa7r+hV2CQ== X-Received: by 2002:a05:6a20:3d06:b0:251:1b8c:5643 with SMTP id adf61e73a8af0-32da845fdf3mr13573976637.50.1760067539326; Thu, 09 Oct 2025 20:38:59 -0700 (PDT) Received: from 7940hx ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7992d993853sm1260148b3a.74.2025.10.09.20.38.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Oct 2025 20:38:58 -0700 (PDT) From: Menglong Dong X-Google-Original-From: Menglong Dong To: mhiramat@kernel.org Cc: rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiang.biao@linux.dev, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH v2 1/2] tracing: fprobe: optimization for entry only case Date: Fri, 10 Oct 2025 11:38:46 +0800 Message-ID: <20251010033847.31008-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20251010033847.31008-1-dongml2@chinatelecom.cn> References: <20251010033847.31008-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable For now, fgraph is used for the fprobe, even if we need trace the entry only. However, the performance of ftrace is better than fgraph, and we can use ftrace_ops for this case. Then performance of kprobe-multi increases from 54M to 69M. Before this commit: $ ./benchs/run_bench_trigger.sh kprobe-multi kprobe-multi : 54.663 =C2=B1 0.493M/s After this commit: $ ./benchs/run_bench_trigger.sh kprobe-multi kprobe-multi : 69.447 =C2=B1 0.143M/s Mitigation is disable during the bench testing above. Signed-off-by: Menglong Dong --- v2: - add some document for fprobe_fgraph_entry as Masami suggested - merge the rename of fprobe_entry into current patch - use ftrace_test_recursion_trylock() in fprobe_ftrace_entry() --- kernel/trace/fprobe.c | 104 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 93 insertions(+), 11 deletions(-) diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 99d83c08b9e2..bb02d6d09d6a 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -254,8 +254,9 @@ static inline int __fprobe_kprobe_handler(unsigned long= ip, unsigned long parent return ret; } =20 -static int fprobe_entry(struct ftrace_graph_ent *trace, struct fgraph_ops = *gops, - struct ftrace_regs *fregs) +/* fgraph_ops callback, this processes fprobes which have exit_handler. */ +static int fprobe_fgraph_entry(struct ftrace_graph_ent *trace, struct fgra= ph_ops *gops, + struct ftrace_regs *fregs) { unsigned long *fgraph_data =3D NULL; unsigned long func =3D trace->func; @@ -292,7 +293,7 @@ static int fprobe_entry(struct ftrace_graph_ent *trace,= struct fgraph_ops *gops, if (node->addr !=3D func) continue; fp =3D READ_ONCE(node->fp); - if (fp && !fprobe_disabled(fp)) + if (fp && !fprobe_disabled(fp) && fp->exit_handler) fp->nmissed++; } return 0; @@ -312,11 +313,11 @@ static int fprobe_entry(struct ftrace_graph_ent *trac= e, struct fgraph_ops *gops, if (node->addr !=3D func) continue; fp =3D READ_ONCE(node->fp); - if (!fp || fprobe_disabled(fp)) + if (unlikely(!fp || fprobe_disabled(fp) || !fp->exit_handler)) continue; =20 data_size =3D fp->entry_data_size; - if (data_size && fp->exit_handler) + if (data_size) data =3D fgraph_data + used + FPROBE_HEADER_SIZE_IN_LONG; else data =3D NULL; @@ -327,7 +328,7 @@ static int fprobe_entry(struct ftrace_graph_ent *trace,= struct fgraph_ops *gops, ret =3D __fprobe_handler(func, ret_ip, fp, fregs, data); =20 /* If entry_handler returns !0, nmissed is not counted but skips exit_ha= ndler. */ - if (!ret && fp->exit_handler) { + if (!ret) { int size_words =3D SIZE_IN_LONG(data_size); =20 if (write_fprobe_header(&fgraph_data[used], fp, size_words)) @@ -340,7 +341,7 @@ static int fprobe_entry(struct ftrace_graph_ent *trace,= struct fgraph_ops *gops, /* If any exit_handler is set, data must be used. */ return used !=3D 0; } -NOKPROBE_SYMBOL(fprobe_entry); +NOKPROBE_SYMBOL(fprobe_fgraph_entry); =20 static void fprobe_return(struct ftrace_graph_ret *trace, struct fgraph_ops *gops, @@ -379,11 +380,82 @@ static void fprobe_return(struct ftrace_graph_ret *tr= ace, NOKPROBE_SYMBOL(fprobe_return); =20 static struct fgraph_ops fprobe_graph_ops =3D { - .entryfunc =3D fprobe_entry, + .entryfunc =3D fprobe_fgraph_entry, .retfunc =3D fprobe_return, }; static int fprobe_graph_active; =20 +/* ftrace_ops callback, this processes fprobes which have only entry_handl= er. */ +static void fprobe_ftrace_entry(unsigned long ip, unsigned long parent_ip, + struct ftrace_ops *ops, struct ftrace_regs *fregs) +{ + struct fprobe_hlist_node *node; + struct rhlist_head *head, *pos; + struct fprobe *fp; + int bit; + + bit =3D ftrace_test_recursion_trylock(ip, parent_ip); + if (bit < 0) + return; + + rcu_read_lock(); + head =3D rhltable_lookup(&fprobe_ip_table, &ip, fprobe_rht_params); + + rhl_for_each_entry_rcu(node, pos, head, hlist) { + if (node->addr !=3D ip) + break; + fp =3D READ_ONCE(node->fp); + if (unlikely(!fp || fprobe_disabled(fp) || fp->exit_handler)) + continue; + + if (fprobe_shared_with_kprobes(fp)) + __fprobe_kprobe_handler(ip, parent_ip, fp, fregs, NULL); + else + __fprobe_handler(ip, parent_ip, fp, fregs, NULL); + } + rcu_read_unlock(); + ftrace_test_recursion_unlock(bit); +} +NOKPROBE_SYMBOL(fprobe_ftrace_entry); + +static struct ftrace_ops fprobe_ftrace_ops =3D { + .func =3D fprobe_ftrace_entry, + .flags =3D FTRACE_OPS_FL_SAVE_REGS, +}; +static int fprobe_ftrace_active; + +static int fprobe_ftrace_add_ips(unsigned long *addrs, int num) +{ + int ret; + + lockdep_assert_held(&fprobe_mutex); + + ret =3D ftrace_set_filter_ips(&fprobe_ftrace_ops, addrs, num, 0, 0); + if (ret) + return ret; + + if (!fprobe_ftrace_active) { + ret =3D register_ftrace_function(&fprobe_ftrace_ops); + if (ret) { + ftrace_free_filter(&fprobe_ftrace_ops); + return ret; + } + } + fprobe_ftrace_active++; + return 0; +} + +static void fprobe_ftrace_remove_ips(unsigned long *addrs, int num) +{ + lockdep_assert_held(&fprobe_mutex); + + fprobe_ftrace_active--; + if (!fprobe_ftrace_active) + unregister_ftrace_function(&fprobe_ftrace_ops); + if (num) + ftrace_set_filter_ips(&fprobe_ftrace_ops, addrs, num, 1, 0); +} + /* Add @addrs to the ftrace filter and register fgraph if needed. */ static int fprobe_graph_add_ips(unsigned long *addrs, int num) { @@ -498,9 +570,12 @@ static int fprobe_module_callback(struct notifier_bloc= k *nb, } while (node =3D=3D ERR_PTR(-EAGAIN)); rhashtable_walk_exit(&iter); =20 - if (alist.index > 0) + if (alist.index > 0) { ftrace_set_filter_ips(&fprobe_graph_ops.ops, alist.addrs, alist.index, 1, 0); + ftrace_set_filter_ips(&fprobe_ftrace_ops, + alist.addrs, alist.index, 1, 0); + } mutex_unlock(&fprobe_mutex); =20 kfree(alist.addrs); @@ -733,7 +808,11 @@ int register_fprobe_ips(struct fprobe *fp, unsigned lo= ng *addrs, int num) mutex_lock(&fprobe_mutex); =20 hlist_array =3D fp->hlist_array; - ret =3D fprobe_graph_add_ips(addrs, num); + if (fp->exit_handler) + ret =3D fprobe_graph_add_ips(addrs, num); + else + ret =3D fprobe_ftrace_add_ips(addrs, num); + if (!ret) { add_fprobe_hash(fp); for (i =3D 0; i < hlist_array->size; i++) { @@ -829,7 +908,10 @@ int unregister_fprobe(struct fprobe *fp) } del_fprobe_hash(fp); =20 - fprobe_graph_remove_ips(addrs, count); + if (fp->exit_handler) + fprobe_graph_remove_ips(addrs, count); + else + fprobe_ftrace_remove_ips(addrs, count); =20 kfree_rcu(hlist_array, rcu); fp->hlist_array =3D NULL; --=20 2.51.0