From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 135A04A2E0F; Tue, 5 May 2026 17:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001005; cv=none; b=uSq8yktAsv5LHEPwewPKbt7kEossPChkFlAq+lcGjd2jtU79lq6GJs0hIDFXRdpkxnsjUz8LFqIFLLS0MELNkUSuCyqlOngUs8H0H1RarF9s2RPVqQG/iBYsUOruXQeZEuRadgphM8bMWsrFy9BIkX4XiPFSqdshnt6KdBkpedg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001005; c=relaxed/simple; bh=4EDmDHbkx8sWstAxmN5dypJ1uHXNHsE+7kGZ6BrByBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SZQRZPFX07lTcvNOFmKuF9GUaKvsGhguAt93yTt7ZtZ4nh8RleNjYk2dE584PgDTVcpJfkoiCxObaiQIV8MOe12njLzHGZSlNlF8CTha6QaXXHn0uz/dUnFhlZKG5IR5oXyB9L7OMHaLh7spKweGE5L3ZS2xSi129BBbNY1ZZXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=ssEo3Kfr; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="ssEo3Kfr" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 1BC05CFFEF; Tue, 05 May 2026 17:09:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000993; bh=SxdOxmJ+WRA4zY3tyFP5+GgWLQ1rokSaYX9143EcXHM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ssEo3KfrvTzaYppTeR/9SbdtJSPaIVu1WuRcuxgZD6c7mHL+wbdWTgbH50oMX9Rx4 jbiTDSsFgKtyohDLtiVfkwSixAPFGoCqLsroBDbbi8FZyVseJMQXFy3LQGSrAai9Nw 1x1vAXnXREgWmxSZd9lkIwqxSPbEE82eVuY97/ms= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin , Usama Arif Subject: [PATCH v6 1/7] tracing/lock: Remove unnecessary linux/sched.h include Date: Tue, 5 May 2026 17:09:30 +0000 Message-ID: <16c1eccc88e1a1e67028086151c7fbffddf980e5.1777999826.git.d@ilvokhin.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" None of the trace events in lock.h reference anything from linux/sched.h. Remove the unnecessary include. Signed-off-by: Dmitry Ilvokhin Acked-by: Usama Arif --- include/trace/events/lock.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/trace/events/lock.h b/include/trace/events/lock.h index 8e89baa3775f..da978f2afb45 100644 --- a/include/trace/events/lock.h +++ b/include/trace/events/lock.h @@ -5,7 +5,6 @@ #if !defined(_TRACE_LOCK_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_LOCK_H =20 -#include #include =20 /* flags for lock:contention_begin */ --=20 2.52.0 From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1BB964A2E16; Tue, 5 May 2026 17:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001002; cv=none; b=sNhgnYmF62jvSUXez4JR4MFW/4Txp0ICHwfa1wyrGDiTjf9K2sdnD1cY8LpcS2G0ZBNFSFt1oUhgAZAwID8wpbr10gF0afyWjMA4Uc44vX8wuBgHG9BCTy4q0Z7pL7h8B9+wSjoPeEGQJWSMweB+ZXjupSXhlKBdrBW4DLzketk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001002; c=relaxed/simple; bh=nkS2FNHvuD93wdW/3Ky4EndgW9Xd2ntRYc4kP5U6umY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E9h72AtWklIqcKh3qcHIEpGlDfjYdTo+b+ozRPd/eEZnLe4/sVPL4fmgQLzOgXbGbrWs2floNeTSPB6CLx1AcBZ+Wbul+N/EFoUIh4JiFEEpwJv04m7hdcEIKSCu/xk2YroIv2kVpO3lrSdxeT4c6Xb2NFgHz7++9y46kkLvSnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=sWU2rOJr; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="sWU2rOJr" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 8367ACFFF3; Tue, 05 May 2026 17:09:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000993; bh=Jiq9HHreSKv9gJ1OxNDx05z/LXXiM1SwacfPI7vtSKY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=sWU2rOJrbDjkjTDizt2f9U5ELWqEaXAc5QmT9jVUAdoxkMtiiOnFCeUpOz6Z+KEGX 0SGGytVFpYd8E/QFVq933R0++oX41r40I8M6oQeN7FEE8k4o5N+4nxr6ZIgiZbqxxR nQ05tef5x5Wsdba66GspFDa0eZI8GtHF/1JeTlVk= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin , Usama Arif Subject: [PATCH v6 2/7] locking/percpu-rwsem: Extract __percpu_up_read() Date: Tue, 5 May 2026 17:09:31 +0000 Message-ID: <0a30ed3ffebe5753aabc083bbe93cc123227da76.1777999826.git.d@ilvokhin.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the percpu_up_read() slowpath out of the inline function into a new __percpu_up_read() to avoid binary size increase from adding a tracepoint to an inlined function. Signed-off-by: Dmitry Ilvokhin Acked-by: Usama Arif --- include/linux/percpu-rwsem.h | 15 +++------------ kernel/locking/percpu-rwsem.c | 18 ++++++++++++++++++ 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/include/linux/percpu-rwsem.h b/include/linux/percpu-rwsem.h index c8cb010d655e..39d5bf8e6562 100644 --- a/include/linux/percpu-rwsem.h +++ b/include/linux/percpu-rwsem.h @@ -107,6 +107,8 @@ static inline bool percpu_down_read_trylock(struct perc= pu_rw_semaphore *sem) return ret; } =20 +extern void __percpu_up_read(struct percpu_rw_semaphore *sem); + static inline void percpu_up_read(struct percpu_rw_semaphore *sem) { rwsem_release(&sem->dep_map, _RET_IP_); @@ -118,18 +120,7 @@ static inline void percpu_up_read(struct percpu_rw_sem= aphore *sem) if (likely(rcu_sync_is_idle(&sem->rss))) { this_cpu_dec(*sem->read_count); } else { - /* - * slowpath; reader will only ever wake a single blocked - * writer. - */ - smp_mb(); /* B matches C */ - /* - * In other words, if they see our decrement (presumably to - * aggregate zero, as that is the only time it matters) they - * will also see our critical section. - */ - this_cpu_dec(*sem->read_count); - rcuwait_wake_up(&sem->writer); + __percpu_up_read(sem); } preempt_enable(); } diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index ef234469baac..f3ee7a0d6047 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -288,3 +288,21 @@ void percpu_up_write(struct percpu_rw_semaphore *sem) rcu_sync_exit(&sem->rss); } EXPORT_SYMBOL_GPL(percpu_up_write); + +void __percpu_up_read(struct percpu_rw_semaphore *sem) +{ + lockdep_assert_preemption_disabled(); + /* + * slowpath; reader will only ever wake a single blocked + * writer. + */ + smp_mb(); /* B matches C */ + /* + * In other words, if they see our decrement (presumably to + * aggregate zero, as that is the only time it matters) they + * will also see our critical section. + */ + this_cpu_dec(*sem->read_count); + rcuwait_wake_up(&sem->writer); +} +EXPORT_SYMBOL_GPL(__percpu_up_read); --=20 2.52.0 From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E47374A2E23; Tue, 5 May 2026 17:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001003; cv=none; b=V0qWs+VKzckKaQ7HrSc7GvzMcpm+pvFvv7QOTo9LaBTnLZh5ua3OLnR2inqsa/ZAjIfsdNzPzWp/fdGX7k53obp2pZGgQwG5a0zJugfROuJUXHci5NGmleW5EgXxNak6oGAuHv7gASI8zg25v2P8BPjCaMd+h8cS2GsX55DVCM0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001003; c=relaxed/simple; bh=UKN3dov81u2ruVLah9uGMEJqVPN2l4vK0rTvVmk/6uU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=piPgnQ3T/kIhAyxB8CAoP6IdTD+5+LddCIq09C8XuMelkKheyi0aBkf0gNaQTWgckcqBewfoR8n1wcRW5Ti/lDnwSFOVbkzzyaCfR+SANmGHjZWSKa1tLY6gUVgkOEEjK9CQlYIUxnXNocWW52cXogeZ/M/MdztQ1SEiacOBKJ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=zcCtOI/d; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="zcCtOI/d" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id ECED4CFFF7; Tue, 05 May 2026 17:09:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000994; bh=TU6fnwDkVJ/mUD/VZ0OBa0Es886o/JD7esEKomLEV8o=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=zcCtOI/d+ZuvnwEzkIXl5D/JI/z6PvCNTaZw9QKwmjnme0R6jWKnWQL1dqEr7LjZ/ zS8X0JF9txAqSbwGjuc0jKcz3Xyfm5r1QknHBbg9XC67GAnznYDi/a5WDrY1ZtkK9V 0Uh/RsxUM9enkB10s7cXSbaVULny+m8bSeLRl+is= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin , "Paul E. McKenney" Subject: [PATCH v6 3/7] locking: Add contended_release tracepoint to sleepable locks Date: Tue, 5 May 2026 17:09:32 +0000 Message-ID: <81cac7754f10897977094baa1b93779c66d894f8.1777999826.git.d@ilvokhin.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the contended_release trace event. This tracepoint fires on the holder side when a contended lock is released, complementing the existing contention_begin/contention_end tracepoints which fire on the waiter side. This enables correlating lock hold time under contention with waiter events by lock address. Add trace_contended_release()/trace_call__contended_release() calls to the slowpath unlock paths of sleepable locks: mutex, rtmutex, semaphore, rwsem, percpu-rwsem, and RT-specific rwbase locks. Where possible, trace_contended_release() fires before the lock is released and before the waiter is woken. For some lock types, the tracepoint fires after the release but before the wake. Making the placement consistent across all lock types is not worth the added complexity. For reader/writer locks, the tracepoint fires for every reader releasing while a writer is waiting, not only for the last reader. Signed-off-by: Dmitry Ilvokhin Acked-by: Paul E. McKenney --- include/trace/events/lock.h | 17 +++++++++++++++++ kernel/locking/mutex.c | 4 ++++ kernel/locking/percpu-rwsem.c | 11 +++++++++++ kernel/locking/rtmutex.c | 1 + kernel/locking/rwbase_rt.c | 6 ++++++ kernel/locking/rwsem.c | 10 ++++++++-- kernel/locking/semaphore.c | 4 ++++ 7 files changed, 51 insertions(+), 2 deletions(-) diff --git a/include/trace/events/lock.h b/include/trace/events/lock.h index da978f2afb45..1ded869cd619 100644 --- a/include/trace/events/lock.h +++ b/include/trace/events/lock.h @@ -137,6 +137,23 @@ TRACE_EVENT(contention_end, TP_printk("%p (ret=3D%d)", __entry->lock_addr, __entry->ret) ); =20 +TRACE_EVENT(contended_release, + + TP_PROTO(void *lock), + + TP_ARGS(lock), + + TP_STRUCT__entry( + __field(void *, lock_addr) + ), + + TP_fast_assign( + __entry->lock_addr =3D lock; + ), + + TP_printk("%p", __entry->lock_addr) +); + #endif /* _TRACE_LOCK_H */ =20 /* This part must be outside protection */ diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 09534628dc01..43b7f7e281a0 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -1023,6 +1023,9 @@ static noinline void __sched __mutex_unlock_slowpath(= struct mutex *lock, unsigne wake_q_add(&wake_q, next); } =20 + if (trace_contended_release_enabled() && waiter) + trace_call__contended_release(lock); + if (owner & MUTEX_FLAG_HANDOFF) __mutex_handoff(lock, next); =20 @@ -1220,6 +1223,7 @@ EXPORT_SYMBOL(ww_mutex_lock_interruptible); =20 EXPORT_TRACEPOINT_SYMBOL_GPL(contention_begin); EXPORT_TRACEPOINT_SYMBOL_GPL(contention_end); +EXPORT_TRACEPOINT_SYMBOL_GPL(contended_release); =20 /** * atomic_dec_and_mutex_lock - return holding mutex if we dec to 0 diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index f3ee7a0d6047..f7e152c40d6d 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -263,6 +263,9 @@ void percpu_up_write(struct percpu_rw_semaphore *sem) { rwsem_release(&sem->dep_map, _RET_IP_); =20 + if (trace_contended_release_enabled() && wq_has_sleeper(&sem->waiters)) + trace_call__contended_release(sem); + /* * Signal the writer is done, no fast path yet. * @@ -292,6 +295,14 @@ EXPORT_SYMBOL_GPL(percpu_up_write); void __percpu_up_read(struct percpu_rw_semaphore *sem) { lockdep_assert_preemption_disabled(); + /* + * After percpu_up_write() completes, rcu_sync_is_idle() can still + * return false during the grace period, forcing readers into this + * slowpath. Only trace when a writer is actually waiting for + * readers to drain. + */ + if (trace_contended_release_enabled() && rcuwait_active(&sem->writer)) + trace_call__contended_release(sem); /* * slowpath; reader will only ever wake a single blocked * writer. diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 69759fde7d10..9a4c3bee50ac 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -1470,6 +1470,7 @@ static void __sched rt_mutex_slowunlock(struct rt_mut= ex_base *lock) raw_spin_lock_irqsave(&lock->wait_lock, flags); } =20 + trace_contended_release(lock); /* * The wakeup next waiter path does not suffer from the above * race. See the comments there. diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c index 82e078c0665a..2835c9ef9b3f 100644 --- a/kernel/locking/rwbase_rt.c +++ b/kernel/locking/rwbase_rt.c @@ -174,6 +174,8 @@ static void __sched __rwbase_read_unlock(struct rwbase_= rt *rwb, static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb, unsigned int state) { + if (trace_contended_release_enabled() && rt_mutex_owner(&rwb->rtmutex)) + trace_call__contended_release(rwb); /* * rwb->readers can only hit 0 when a writer is waiting for the * active readers to leave the critical section. @@ -205,6 +207,8 @@ static inline void rwbase_write_unlock(struct rwbase_rt= *rwb) unsigned long flags; =20 raw_spin_lock_irqsave(&rtm->wait_lock, flags); + if (trace_contended_release_enabled() && rt_mutex_has_waiters(rtm)) + trace_call__contended_release(rwb); __rwbase_write_unlock(rwb, WRITER_BIAS, flags); } =20 @@ -214,6 +218,8 @@ static inline void rwbase_write_downgrade(struct rwbase= _rt *rwb) unsigned long flags; =20 raw_spin_lock_irqsave(&rtm->wait_lock, flags); + if (trace_contended_release_enabled() && rt_mutex_has_waiters(rtm)) + trace_call__contended_release(rwb); /* Release it and account current as reader */ __rwbase_write_unlock(rwb, WRITER_BIAS - 1, flags); } diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index bf647097369c..b9c180ac1eee 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -1387,6 +1387,8 @@ static inline void __up_read(struct rw_semaphore *sem) rwsem_clear_reader_owned(sem); tmp =3D atomic_long_add_return_release(-RWSEM_READER_BIAS, &sem->count); DEBUG_RWSEMS_WARN_ON(tmp < 0, sem); + if (trace_contended_release_enabled() && (tmp & RWSEM_FLAG_WAITERS)) + trace_call__contended_release(sem); if (unlikely((tmp & (RWSEM_LOCK_MASK|RWSEM_FLAG_WAITERS)) =3D=3D RWSEM_FLAG_WAITERS)) { clear_nonspinnable(sem); @@ -1413,8 +1415,10 @@ static inline void __up_write(struct rw_semaphore *s= em) preempt_disable(); rwsem_clear_owner(sem); tmp =3D atomic_long_fetch_add_release(-RWSEM_WRITER_LOCKED, &sem->count); - if (unlikely(tmp & RWSEM_FLAG_WAITERS)) + if (unlikely(tmp & RWSEM_FLAG_WAITERS)) { + trace_contended_release(sem); rwsem_wake(sem); + } preempt_enable(); } =20 @@ -1437,8 +1441,10 @@ static inline void __downgrade_write(struct rw_semap= hore *sem) tmp =3D atomic_long_fetch_add_release( -RWSEM_WRITER_LOCKED+RWSEM_READER_BIAS, &sem->count); rwsem_set_reader_owned(sem); - if (tmp & RWSEM_FLAG_WAITERS) + if (tmp & RWSEM_FLAG_WAITERS) { + trace_contended_release(sem); rwsem_downgrade_wake(sem); + } preempt_enable(); } =20 diff --git a/kernel/locking/semaphore.c b/kernel/locking/semaphore.c index 74d41433ba13..233730c25933 100644 --- a/kernel/locking/semaphore.c +++ b/kernel/locking/semaphore.c @@ -230,6 +230,10 @@ void __sched up(struct semaphore *sem) sem->count++; else __up(sem, &wake_q); + + if (trace_contended_release_enabled() && !wake_q_empty(&wake_q)) + trace_call__contended_release(sem); + raw_spin_unlock_irqrestore(&sem->lock, flags); if (!wake_q_empty(&wake_q)) wake_up_q(&wake_q); --=20 2.52.0 From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBF8D4A2E2C; Tue, 5 May 2026 17:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001003; cv=none; b=ACiq/ExeTGG6kuYiWcLbvmJufD4Zg/ZEYWRUECqlBfE9PvI5gWItWNVUCuhVjyBPYg5ldbNyZjjz1+SmQKiRnRi3zKCf3cTUdj+Jwt4M1n6XKhS+wfh/AOSCyjfzOfq9UBwewgmEUyjUUAJAzjL2Hahbgq1nTEepOw3CymwDbDA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001003; c=relaxed/simple; bh=Hxgm4nyRXG7am7evRAcxkckPyVlEd1UOmeLxYJCJOHI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WcQrsH8PsJ1JTXsWulOS77jBzEJmQ0Cz2ea5yYFch5j0zO7EpXnf+W3Id2qoa/oKKzc9ib9XdDwDv1AyKpEHZI280BNd7TbggYfs48fK1oFPFvmEsNBcikqST/uP2v+9cHYlAmyLMMr+3eAzQjoBmqoVoayttGlNoSs+1+N56Ng= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=NddL6V50; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="NddL6V50" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 5D462CFFFB; Tue, 05 May 2026 17:09:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000994; bh=0iNiXafam+/e0hkdJzF6nIr0O4d4HoUoH1x5V8VkIAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=NddL6V50gn9QHy1qO45QfNnsCGY22x/FGJ9pE4goYDRaBNmo5J1FEz/pCZ86C3hfQ bXOIi3qyfbj1qjkBuNbWqzN7Yja6FSueSZPaV5S5oU1HXvzY9z7SUhaiQWQKfOUvLe QG3eIABqUNEWkkCnlSPfPahP+NN2+iFMyyJjTW5U= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin , "Paul E. McKenney" Subject: [PATCH v6 4/7] locking: Factor out queued_spin_release() Date: Tue, 5 May 2026 17:09:33 +0000 Message-ID: <64c202b8a76a7d98515cf10cc1f99ecb0a9a7ccf.1777999826.git.d@ilvokhin.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce queued_spin_release() as an arch-overridable unlock primitive, and make queued_spin_unlock() a generic wrapper around it. This is a preparatory refactoring for the next commit, which adds contended_release tracepoint instrumentation to queued_spin_unlock(). Rename the existing arch-specific queued_spin_unlock() overrides on x86 (paravirt) and MIPS to queued_spin_release(). No functional change. Signed-off-by: Dmitry Ilvokhin Reviewed-by: Paul E. McKenney --- arch/mips/include/asm/spinlock.h | 6 +++--- arch/x86/include/asm/paravirt-spinlock.h | 6 +++--- include/asm-generic/qspinlock.h | 15 ++++++++++++--- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/arch/mips/include/asm/spinlock.h b/arch/mips/include/asm/spinl= ock.h index 6ce2117e49f6..c349162f15eb 100644 --- a/arch/mips/include/asm/spinlock.h +++ b/arch/mips/include/asm/spinlock.h @@ -13,12 +13,12 @@ =20 #include =20 -#define queued_spin_unlock queued_spin_unlock +#define queued_spin_release queued_spin_release /** - * queued_spin_unlock - release a queued spinlock + * queued_spin_release - release a queued spinlock * @lock : Pointer to queued spinlock structure */ -static inline void queued_spin_unlock(struct qspinlock *lock) +static inline void queued_spin_release(struct qspinlock *lock) { /* This could be optimised with ARCH_HAS_MMIOWB */ mmiowb(); diff --git a/arch/x86/include/asm/paravirt-spinlock.h b/arch/x86/include/as= m/paravirt-spinlock.h index 7beffcb08ed6..ac75e0736198 100644 --- a/arch/x86/include/asm/paravirt-spinlock.h +++ b/arch/x86/include/asm/paravirt-spinlock.h @@ -49,9 +49,9 @@ static __always_inline bool pv_vcpu_is_preempted(long cpu) ALT_NOT(X86_FEATURE_VCPUPREEMPT)); } =20 -#define queued_spin_unlock queued_spin_unlock +#define queued_spin_release queued_spin_release /** - * queued_spin_unlock - release a queued spinlock + * queued_spin_release - release a queued spinlock * @lock : Pointer to queued spinlock structure * * A smp_store_release() on the least-significant byte. @@ -66,7 +66,7 @@ static inline void queued_spin_lock_slowpath(struct qspin= lock *lock, u32 val) pv_queued_spin_lock_slowpath(lock, val); } =20 -static inline void queued_spin_unlock(struct qspinlock *lock) +static inline void queued_spin_release(struct qspinlock *lock) { kcsan_release(); pv_queued_spin_unlock(lock); diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index bf47cca2c375..df76f34645a0 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -115,12 +115,12 @@ static __always_inline void queued_spin_lock(struct q= spinlock *lock) } #endif =20 -#ifndef queued_spin_unlock +#ifndef queued_spin_release /** - * queued_spin_unlock - release a queued spinlock + * queued_spin_release - release a queued spinlock * @lock : Pointer to queued spinlock structure */ -static __always_inline void queued_spin_unlock(struct qspinlock *lock) +static __always_inline void queued_spin_release(struct qspinlock *lock) { /* * unlock() needs release semantics: @@ -129,6 +129,15 @@ static __always_inline void queued_spin_unlock(struct = qspinlock *lock) } #endif =20 +/** + * queued_spin_unlock - unlock a queued spinlock + * @lock : Pointer to queued spinlock structure + */ +static __always_inline void queued_spin_unlock(struct qspinlock *lock) +{ + queued_spin_release(lock); +} + #ifndef virt_spin_lock static __always_inline bool virt_spin_lock(struct qspinlock *lock) { --=20 2.52.0 From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B3304A33F3; Tue, 5 May 2026 17:10:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001005; cv=none; b=NQyKfdWbHTehJLBmfZbk9rGdRlL0dgUIaqsJ5YrycNMCNECYXjKYReZddbgDSCQB/f9k1xdZwk1pXjZ1vuiIFKk4q7FtpDklu7Z41GefB8TxOU64dlImMRMr5alIJw+d3xp1M+zaXo1fFWCBWKWINp2NOmbKVvMJ4zP5llgh26c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001005; c=relaxed/simple; bh=5FtJ1S2CDU3Odzvxg8LA/McfTvcWX7yofYFuutItafU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kIwZMOgXKsfatibD2NkhNshI7HVFqs6SkzTNTqfdC3RYmMxk7gg6gNoZx5RJC0dE7t+iVvSAnKYKaevf/hY4T0PtW5sEfZELGaLtPePvacszZV+kRpCm533hv0CseQZtlGvg5ksqdr+bejWGzYegG0ESSSx9jneAMIExZN+UCUM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=OSHLOw1z; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="OSHLOw1z" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id C1410D0000; Tue, 05 May 2026 17:09:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000995; bh=INT5iVx0BJhEgv/8g1hlou4GKubElqnqSOStPxLCWSY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=OSHLOw1zDhJR+x0r5tP3lBldueD1O2a5vME5r5oHVcjUs5p8dH/1Yr2pO1Qw54Nr+ 0YBFA6hi2CbPKnKzw+a/zlZteQUiGT6Leq4Udz8CuKYMktDxrfqYeYBrNnmfOv1Hrp u4pXd3jpKmhYH2W/lXI6NxVpRBNQJybvo3y89KTY= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin , "Paul E. McKenney" Subject: [PATCH v6 5/7] locking: Add contended_release tracepoint to qspinlock Date: Tue, 5 May 2026 17:09:34 +0000 Message-ID: <5d7ea75ffe74a785e6b234ada9f23c6373d4b4c1.1777999826.git.d@ilvokhin.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the arch-overridable queued_spin_release(), introduced in the previous commit, to ensure the tracepoint works correctly across all architectures, including those with custom unlock implementations (e.g. x86 paravirt). When the tracepoint is disabled, the only addition to the hot path is a single NOP instruction (the static branch). When enabled, the contention check, trace call, and unlock are combined in an out-of-line function to minimize hot path impact, avoiding the compiler needing to preserve the lock pointer in a callee-saved register across the trace call. Binary size impact (x86_64, defconfig): uninlined unlock (common case): +680 bytes (+0.00%) inlined unlock (worst case): +83659 bytes (+0.21%) The inlined unlock case could not be achieved through Kconfig options on x86_64 as PREEMPT_BUILD unconditionally selects UNINLINE_SPIN_UNLOCK on x86_64. The UNINLINE_SPIN_UNLOCK guards were manually inverted to force inline the unlock path and estimate the worst case binary size increase. In practice, configurations with UNINLINE_SPIN_UNLOCK=3Dn have already opted against binary size optimization, so the inlined worst case is unlikely to be a concern. Architectures with fully custom qspinlock implementations (e.g. PowerPC) are not covered by this change. Signed-off-by: Dmitry Ilvokhin Acked-by: Paul E. McKenney --- include/asm-generic/qspinlock.h | 18 ++++++++++++++++++ kernel/locking/qspinlock.c | 8 ++++++++ 2 files changed, 26 insertions(+) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index df76f34645a0..915a4c2777f6 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -41,6 +41,7 @@ =20 #include #include +#include =20 #ifndef queued_spin_is_locked /** @@ -129,12 +130,29 @@ static __always_inline void queued_spin_release(struc= t qspinlock *lock) } #endif =20 +DECLARE_TRACEPOINT(contended_release); + +extern void queued_spin_release_traced(struct qspinlock *lock); + /** * queued_spin_unlock - unlock a queued spinlock * @lock : Pointer to queued spinlock structure + * + * Generic tracing wrapper around the arch-overridable + * queued_spin_release(). */ static __always_inline void queued_spin_unlock(struct qspinlock *lock) { + /* + * Trace and release are combined in queued_spin_release_traced() so + * the compiler does not need to preserve the lock pointer across the + * function call, avoiding callee-saved register save/restore on the + * hot path. + */ + if (tracepoint_enabled(contended_release)) { + queued_spin_release_traced(lock); + return; + } queued_spin_release(lock); } =20 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index af8d122bb649..649fdca69288 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -104,6 +104,14 @@ static __always_inline u32 __pv_wait_head_or_lock(str= uct qspinlock *lock, #define queued_spin_lock_slowpath native_queued_spin_lock_slowpath #endif =20 +void __lockfunc queued_spin_release_traced(struct qspinlock *lock) +{ + if (queued_spin_is_contended(lock)) + trace_call__contended_release(lock); + queued_spin_release(lock); +} +EXPORT_SYMBOL(queued_spin_release_traced); + #endif /* _GEN_PV_LOCK_SLOWPATH */ =20 /** --=20 2.52.0 From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B43A4A33F4; Tue, 5 May 2026 17:10:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001004; cv=none; b=mBBy/8hV66YFRb8+bl2q8mbyrhGEzIY00uuVcjR6VsGYFjdOes2vccXw9LScRxUYvkHDZT8qFFgJqK9pdbHJwVO3/HOJK/s15abFkj9O3FoxL5hcdW4spB5Q19uYjDLLfMHOSI+FrYvMTDmC41K1odMKWA3wCwUbShoBvo9rZ/4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001004; c=relaxed/simple; bh=mraJN1NbsGD/dzKMa2w2GJIg2A7ETfaDkmweFrXmhjE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OERVQ9QsR8Yw2lxAXsCV1RLrfYxBOz/npKfxV91/7xY5CqIpm0FJvIeQXDoFakj0Y1EL3VZiKARRN1kB7jWx0W7OSkqOYAiSVUqOowIqcTlsBYzDY/FHk9PskFYgKRezlv0gjiy5noezW/9evif8rWZkHFKjZYV0vnNIOashFgw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=lt719uei; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="lt719uei" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 388DFD0005; Tue, 05 May 2026 17:09:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000995; bh=HNQRtWDHTSdHuulO3xaZF7TbUmQJOtiaHzb+LTQXOgk=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=lt719ueizrhf62Y1Z7/DZbnmjEx0GJ0GjDpQhigoaaIbE/RpBOAJkHGdfOMuYyBjk 4l56I3tj+XiqaPZPebIxNQMWRr+l90s8NGj8CwP7j64JI9O3RyLiI7Cd4dIRMRt2Pg WnkSFzXSeAx6cbAJJHStQMGcmpN8eYCWX+ayIshY= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin , "Paul E. McKenney" Subject: [PATCH v6 6/7] locking: Factor out __queued_read_unlock()/__queued_write_unlock() Date: Tue, 5 May 2026 17:09:35 +0000 Message-ID: <8e88613c73f0603c4440ba3a62eb604a5dddc57b.1777999826.git.d@ilvokhin.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This is a preparatory refactoring for the next commit, which adds contended_release tracepoint instrumentation and needs to call the unlock from both traced and non-traced paths. No functional change. Signed-off-by: Dmitry Ilvokhin Acked-by: Paul E. McKenney --- include/asm-generic/qrwlock.h | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 75b8f4601b28..4b627bafba8b 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -101,16 +101,26 @@ static inline void queued_write_lock(struct qrwlock *= lock) queued_write_lock_slowpath(lock); } =20 +static __always_inline void __queued_read_unlock(struct qrwlock *lock) +{ + /* + * Atomically decrement the reader count + */ + (void)atomic_sub_return_release(_QR_BIAS, &lock->cnts); +} + /** * queued_read_unlock - release read lock of a queued rwlock * @lock : Pointer to queued rwlock structure */ static inline void queued_read_unlock(struct qrwlock *lock) { - /* - * Atomically decrement the reader count - */ - (void)atomic_sub_return_release(_QR_BIAS, &lock->cnts); + __queued_read_unlock(lock); +} + +static __always_inline void __queued_write_unlock(struct qrwlock *lock) +{ + smp_store_release(&lock->wlocked, 0); } =20 /** @@ -119,7 +129,7 @@ static inline void queued_read_unlock(struct qrwlock *l= ock) */ static inline void queued_write_unlock(struct qrwlock *lock) { - smp_store_release(&lock->wlocked, 0); + __queued_write_unlock(lock); } =20 /** --=20 2.52.0 From nobody Sat Jun 13 23:56:14 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F107C4A340F; Tue, 5 May 2026 17:10:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001005; cv=none; b=gqsKncyK/jFF279xiUH4i7BQnFM9xp8ST8Hlmd7VFy7NvuSASm3bAsAfXHQ0+pYd9dI6w3v9IWO50YN7kgpljHInvlB/Ue9Yej5Wa3scRCEn3K3PHpPvYecYp1BpHJSnGAnJFApCmweM2uyaBtgOj0te1NWGOLD50lFdPYClubI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778001005; c=relaxed/simple; bh=N8txh3pEvw+ELeHqMi98HdQAesu/7T09Grff+J+0IVc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KqaNk+aQmNfvEI6KGDXOaBzqMv62z6nyiTuaF3MADeD9CYa7DbeczZbmlKXrZn3qkqfSC6RthpzrhiZDafSfzIZfnKy8ygfuFxlLMCRRFpBGNKVOobBvE41HumYKvm8T9uVXc28DJlp57qv0Q4yJ5ReAwxx5MDuiXshpNz/sM+w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=1OW0Qc3m; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="1OW0Qc3m" Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id A1FB9D000F; Tue, 05 May 2026 17:09:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1778000995; bh=5x6FaQ8RAPWiFTMehIyIvGpXzwgZYwE2l4ogWqmfm5I=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=1OW0Qc3mLC4qVMBQ1Uc315DJ5RE3HdXkq8W9njLbvKRJQ08noQiB1vYfPi+KCYhwr 31pqiATb64htMJIHCUdBhkjmLNs49yVSXJkrXzmdhsqj62zVihyRvPPrrMxT3pARkb 4Mpf91HRD+GSXIZ31isxB5rn4U4bOLuKGOKx+XtE= From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, virtualization@lists.linux.dev, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin Subject: [PATCH v6 7/7] locking: Add contended_release tracepoint to qrwlock Date: Tue, 5 May 2026 17:09:36 +0000 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend the contended_release tracepoint to queued rwlocks, using the same out-of-line traced unlock approach as queued spinlocks. Signed-off-by: Dmitry Ilvokhin --- include/asm-generic/qrwlock.h | 22 ++++++++++++++++++++++ kernel/locking/qrwlock.c | 16 ++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 4b627bafba8b..274c19006125 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -14,6 +14,7 @@ #define __ASM_GENERIC_QRWLOCK_H =20 #include +#include #include #include =20 @@ -35,6 +36,10 @@ */ extern void queued_read_lock_slowpath(struct qrwlock *lock); extern void queued_write_lock_slowpath(struct qrwlock *lock); +extern void queued_read_unlock_traced(struct qrwlock *lock); +extern void queued_write_unlock_traced(struct qrwlock *lock); + +DECLARE_TRACEPOINT(contended_release); =20 /** * queued_read_trylock - try to acquire read lock of a queued rwlock @@ -115,6 +120,17 @@ static __always_inline void __queued_read_unlock(struc= t qrwlock *lock) */ static inline void queued_read_unlock(struct qrwlock *lock) { + /* + * Trace and unlock are combined in the traced unlock variant so + * the compiler does not need to preserve the lock pointer across + * the function call, avoiding callee-saved register save/restore + * on the hot path. + */ + if (tracepoint_enabled(contended_release)) { + queued_read_unlock_traced(lock); + return; + } + __queued_read_unlock(lock); } =20 @@ -129,6 +145,12 @@ static __always_inline void __queued_write_unlock(stru= ct qrwlock *lock) */ static inline void queued_write_unlock(struct qrwlock *lock) { + /* See comment in queued_read_unlock(). */ + if (tracepoint_enabled(contended_release)) { + queued_write_unlock_traced(lock); + return; + } + __queued_write_unlock(lock); } =20 diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index d2ef312a8611..5ae4b0372719 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -90,3 +90,19 @@ void __lockfunc queued_write_lock_slowpath(struct qrwloc= k *lock) trace_contention_end(lock, 0); } EXPORT_SYMBOL(queued_write_lock_slowpath); + +void __lockfunc queued_read_unlock_traced(struct qrwlock *lock) +{ + if (queued_rwlock_is_contended(lock)) + trace_call__contended_release(lock); + __queued_read_unlock(lock); +} +EXPORT_SYMBOL(queued_read_unlock_traced); + +void __lockfunc queued_write_unlock_traced(struct qrwlock *lock) +{ + if (queued_rwlock_is_contended(lock)) + trace_call__contended_release(lock); + __queued_write_unlock(lock); +} +EXPORT_SYMBOL(queued_write_unlock_traced); --=20 2.52.0