From nobody Fri Oct 2 09:19:25 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6A252FF657; Mon, 3 Aug 2026 08:15:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785744925; cv=none; b=prCus8D0aVxY0vfW8CGWM4NVpHLO2CqBsojetZwFPbzWyTkMrHbnky07C5ETCJZGEWll2vN1zwSS4A1znaBmeHgTQvOh0ApwPM02s4WJxRkr+AVsGNeZduylrh8PxCsIaMHeB9ThgSgdwp1mR5Ua8vBxmyEVtZIrG9mmPnJDYRs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785744925; c=relaxed/simple; bh=Am7APvg2KeYpIUxsui5r1UK6HfCdZ77h7rA1mlk+acc=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=YRScweGb68khXKMvOxzMKIwfWjTmQPsgvCnrIMmSzNjPzxFGpeLf3LuH/55xa4l1ZTaaY1KMJitt0sc+QjHda5zIwbKWlAkJV3kPtNnYQSiZ3uR8RdHkeG9imfp2PnlNpG7CmEljKkRY/xDc90oRqF7ApcCt9EQmO00dmjrDf4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=sFJOtzSc; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=h28MP1Xh; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="sFJOtzSc"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="h28MP1Xh" Date: Mon, 03 Aug 2026 08:15:20 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1785744922; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SP/xyPCWHc8nUuqkwojCjxsFRZ1mbkyvRLGPPPiFKMk=; b=sFJOtzScNGPadKwevlML5Q6RjUWNHHQC5mbkg3SPCx6/9wP3shMtHpNXDwnVEQt1g4QtdJ kvbl7rUNGT6a4x6Yh+4ZnjON1vN+aQPnywvqHf7nGhFzOVDvcVcA2yz12ngq2A1mxbpA3A AfEtCVXAmrsz1bEFSBcKh/m5SOjuOaRLocSy3Ce9Uq47ilvzYaSyASLufoj3AOW+pLXB+e 0ZAQWpQ4qOf0YD1nb+4AuUvOaLKBwNRMApwl+Wqexg5hzNtjYINpbbLl3Thd8KJylrdjvS l+F5f1LEd6j0m8jAAKfnIPYdeyIM+x1IeSHIrG+1rE3xFKx0aIaHaFcJ1WUphA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1785744922; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SP/xyPCWHc8nUuqkwojCjxsFRZ1mbkyvRLGPPPiFKMk=; b=h28MP1Xh9Ial+WmZJGBwObGp2QDRzOZVcNb3uhVTTg1dkq9wbUEqdN5KxK7SdxM0zLwZ0a WbEcfGfpJ/u5M7AQ== From: "tip-bot2 for Puranjay Mohan" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: perf/core] uprobes: Switch uretprobes_srcu to SRCU-fast-updown Cc: Puranjay Mohan , "Peter Zijlstra (Intel)" , Oleg Nesterov , Andrii Nakryiko , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260706172744.3920417-3-puranjay@kernel.org> References: <20260706172744.3920417-3-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <178574492040.1210945.14642221562432818278.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the perf/core branch of tip: Commit-ID: 36c8b02c3fe9ba56d6f11ec055cd879bc49871f7 Gitweb: https://git.kernel.org/tip/36c8b02c3fe9ba56d6f11ec055cd879bc= 49871f7 Author: Puranjay Mohan AuthorDate: Mon, 06 Jul 2026 10:27:42 -07:00 Committer: Peter Zijlstra CommitterDate: Mon, 03 Aug 2026 10:09:08 +02:00 uprobes: Switch uretprobes_srcu to SRCU-fast-updown uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit. Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits. The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it. The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage. Signed-off-by: Puranjay Mohan Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Oleg Nesterov Acked-by: Andrii Nakryiko Link: https://patch.msgid.link/20260706172744.3920417-3-puranjay@kernel.org --- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 18be159..d34dbc0 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -25,6 +25,7 @@ struct mm_struct; struct inode; struct notifier_block; struct page; +struct srcu_ctr; =20 /* * Allowed return values from uprobe consumer's handler callback @@ -106,7 +107,7 @@ enum hprobe_state { * underlying uprobe is not guaranteed anymore. __UPROBE_DEAD is just = an * internal marker and is handled transparently by hprobe_fetch() help= er. * - * When uprobe is SRCU-protected, we also record srcu_idx value, necessary= for + * When uprobe is SRCU-protected, we also record srcu_scp value, necessary= for * SRCU unlocking. * * See hprobe_expire() and hprobe_fetch() for details of race-free uprobe @@ -115,7 +116,7 @@ enum hprobe_state { */ struct hprobe { enum hprobe_state state; - int srcu_idx; + struct srcu_ctr __percpu *srcu_scp; struct uprobe *uprobe; }; =20 diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index e4f526c..a18529a 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -54,7 +54,7 @@ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ]; DEFINE_STATIC_PERCPU_RWSEM(dup_mmap_sem); =20 /* Covers return_instance's uprobe lifetime. */ -DEFINE_STATIC_SRCU(uretprobes_srcu); +DEFINE_STATIC_SRCU_FAST_UPDOWN(uretprobes_srcu); =20 /* Have a copy of original instruction */ #define UPROBE_COPY_INSN 0 @@ -707,12 +707,13 @@ static void put_uprobe(struct uprobe *uprobe) } =20 /* Initialize hprobe as SRCU-protected "leased" uprobe */ -static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprob= e, int srcu_idx) +static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprob= e, + struct srcu_ctr __percpu *srcu_scp) { WARN_ON(!uprobe); hprobe->state =3D HPROBE_LEASED; hprobe->uprobe =3D uprobe; - hprobe->srcu_idx =3D srcu_idx; + hprobe->srcu_scp =3D srcu_scp; } =20 /* Initialize hprobe as refcounted ("stable") uprobe (uprobe can be NULL).= */ @@ -720,7 +721,7 @@ static void hprobe_init_stable(struct hprobe *hprobe, s= truct uprobe *uprobe) { hprobe->state =3D uprobe ? HPROBE_STABLE : HPROBE_GONE; hprobe->uprobe =3D uprobe; - hprobe->srcu_idx =3D -1; + hprobe->srcu_scp =3D NULL; } =20 /* @@ -757,7 +758,7 @@ static void hprobe_finalize(struct hprobe *hprobe, enum= hprobe_state hstate) { switch (hstate) { case HPROBE_LEASED: - __srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx); + srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp); break; case HPROBE_STABLE: put_uprobe(hprobe->uprobe); @@ -829,7 +830,7 @@ static struct uprobe *hprobe_expire(struct hprobe *hpro= be, bool get) */ if (try_cmpxchg(&hprobe->state, &hstate, uprobe ? HPROBE_STABLE : HPROBE= _GONE)) { /* We won the race, we are the ones to unlock SRCU */ - __srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx); + srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp); return get && uprobe ? get_uprobe(uprobe) : uprobe; } =20 @@ -2035,7 +2036,7 @@ static void ri_timer(struct timer_list *timer) struct return_instance *ri; =20 /* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire= (). */ - guard(srcu)(&uretprobes_srcu); + guard(srcu_fast_updown)(&uretprobes_srcu); /* RCU protects return_instance from freeing. */ guard(rcu)(); =20 @@ -2132,7 +2133,7 @@ static int dup_utask(struct task_struct *t, struct up= robe_task *o_utask) t->utask =3D n_utask; =20 /* protect uprobes from freeing, we'll need try_get_uprobe() them */ - guard(srcu)(&uretprobes_srcu); + guard(srcu_fast_updown)(&uretprobes_srcu); =20 p =3D &n_utask->return_instances; for (o =3D o_utask->return_instances; o; o =3D o->next) { @@ -2244,8 +2245,8 @@ static void prepare_uretprobe(struct uprobe *uprobe, = struct pt_regs *regs, { struct uprobe_task *utask =3D current->utask; unsigned long orig_ret_vaddr, trampoline_vaddr; + struct srcu_ctr __percpu *srcu_scp; bool chained; - int srcu_idx; =20 if (!get_xol_area()) goto free; @@ -2283,8 +2284,12 @@ static void prepare_uretprobe(struct uprobe *uprobe,= struct pt_regs *regs, orig_ret_vaddr =3D utask->return_instances->orig_ret_vaddr; } =20 - /* __srcu_read_lock() because SRCU lock survives switch to user space */ - srcu_idx =3D __srcu_read_lock(&uretprobes_srcu); + /* + * Use srcu_down_read_fast() because the SRCU lock survives a switch to + * user space and can be unlocked from a different context by ri_timer() + * or dup_utask(). + */ + srcu_scp =3D srcu_down_read_fast(&uretprobes_srcu); =20 ri->func =3D instruction_pointer(regs); ri->stack =3D user_stack_pointer(regs); @@ -2293,7 +2298,7 @@ static void prepare_uretprobe(struct uprobe *uprobe, = struct pt_regs *regs, =20 utask->depth++; =20 - hprobe_init_leased(&ri->hprobe, uprobe, srcu_idx); + hprobe_init_leased(&ri->hprobe, uprobe, srcu_scp); ri->next =3D utask->return_instances; rcu_assign_pointer(utask->return_instances, ri); =20