From nobody Tue Apr 7 22:01:07 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 051173DD511 for ; Wed, 11 Mar 2026 11:53:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773229995; cv=none; b=Nb+Mgb24yr7ih8VNThrpNO5sj6gJRp8+kv4u6ZwHxlbWdVqeZBEqZJFEkW3a5fMzeKdzohNweBguk8nz8X5aimBx9TpU708UkkkCNy3XQO9Szq9ZA+38+DzBemHv0okHzfj7zdnqVap1RtLir6oOce7lQ2vPv7GOd/cNcLsccRM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773229995; c=relaxed/simple; bh=z68fgqc2iOMqfVA3tcxpjxIvdUnGOzEv71RskaThPjA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ooQYZJgnbsfYCEhmjWi8+j2w+XdxVLwBw6z4FbXWWtzv40fCjWdmy+EHEQIc7Lf9ScvD8ZZO69OVzvUEYfVwsdZ4f0IWSrJI9CNza5Ta8gsEwRbV/IEGCBP/e4jS5833q6GK/iZHk3u9tbcUIyUIxWBZnt7tCyY6po3ZgSNq9sc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=a+Mg7yZj; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="a+Mg7yZj" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-829928e512aso3545670b3a.2 for ; Wed, 11 Mar 2026 04:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773229992; x=1773834792; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BbAh84gjMQujNaKw3Rut1VBI8uWVKCNQdZxq2kKwp6Y=; b=a+Mg7yZjPaMPDdxMssgTZTNPO/Iy5+wLiY1KF6us9O/cDBKGlXUr2iTa0wIUfP3siX 3cmjZsDglvDeHtuPculW3jO6vazoQlEjvlWp0RBO+2IIaBjKmvv7M3Ss8kcMgYdYrueX jW4LHOzGGQZCMjQ70V7cm8bd4puUMbgS7wl5+hknlsvqJ/P60+08+hV86vikgpIdcxYP dc9rBLrIw4XazLb8Lyzgi6ZhyneGkoNktLWVXFkXaB2QhBM5bg4Xtewk5p4OhY/B8ucR LQ/RGsrh3D3qVYax1pifb7p4wBFSKHM/3WUL0TN00Kkx1SHJ9u+5BNbytFEDtUsad/hr /S/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773229992; x=1773834792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BbAh84gjMQujNaKw3Rut1VBI8uWVKCNQdZxq2kKwp6Y=; b=bfoETS8HOiJJRx0YCeFYQHzU0sfs2N6gqtYoHKD9UjKTQCths3XuYwzAJIePzLkSJh d9ssI6QUj4P38OGKqewl73v5gqjsVPa6xijANU+lMW7oJj/fY/k3g/z2K9QC9m382Ccd IVwqmJIX6gtSz41WE98hNwarRlR59r8qwgoPXUWmzYaOJqN9Ydu/CUXfVeu+tIA9+BrQ krWzeVgF/PQbRlH1yHXloccBg/WwPnd5TZx1/mbVR9+y1cC+/JATVDwiH4fB20Q2sWSq 7TZl9RJh1IwPQvrrUBL3h2EKTbpzCUDLdDIBvD3eabNR7LHYa06E4c2HS+RLM5yHbeuV /Jqg== X-Gm-Message-State: AOJu0YxJhXQUaXjM8M2+eqQnJjipR2vUrycGrfZpT6N7S8mke+4KyTWx BFo9z2TT/n7raFXpIGdRj12QbJuegyA/rcbLV4YA+eeq3zBwGd6vo3sn X-Gm-Gg: ATEYQzytkmHSUx4bawyJvoiPZZV1H3EVjy4DWlKZgufhManG9CxML6FsftDdU3e6LgD 9NEgDkdw1zfd0oIjXyyo1XU0IpivGwWa6L99d+aIgf8DdZIIusI6/SRYCHN0OZrZUhNBrmKQKWQ QFhG6rJgiVIfvs4vT+X6EVyWBE2LDmwYlTwST/B4LiBqNIHsLvQ8UGXD4/UZNG6XgP4pqdL9u0L hWIALb/lk2NS9HV2v1cvVcVi0Aa9t0KFTmRZ49CcQNXABalPS/mqL6zvaBjUNUfkITBga/7LYaR pGO90KoI4g0DgkoGCqVTiG2gCdbkoAU51g7tOLsyPhLLY4hGTiRW1RlL4fUBGDD+oDoE9/mZuAO 19YCoHUkfJzQ5zvAF5tYzJmvLoEIwUHJAdH2mWoLVpqtZzwjszCUo0cv9GBT4CnkRDfGvaY+20f HtWIOFIWs1aftOt0C4/S7u7tiUKDZQgUMVGCjA42Zqn3eBFaIPt8n5TAsvBeDEg8k= X-Received: by 2002:a05:6a00:1c8f:b0:81f:5127:5d48 with SMTP id d2e1a72fcca58-829f7036d89mr2176235b3a.30.1773229992190; Wed, 11 Mar 2026 04:53:12 -0700 (PDT) Received: from localhost.localdomain ([2409:891f:1b46:28af:1445:b4e2:b8bf:2d47]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-829f6df3aedsm2500588b3a.18.2026.03.11.04.53.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 11 Mar 2026 04:53:11 -0700 (PDT) From: Yafang Shao To: peterz@infradead.org, mingo@redhat.com, will@kernel.org, boqun@kernel.org, longman@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, david.laight.linux@gmail.com Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yafang Shao Subject: [RFC PATCH v2 1/3] locking/mutex: Add slow path variants for lock/unlock Date: Wed, 11 Mar 2026 19:52:48 +0800 Message-ID: <20260311115250.78488-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260311115250.78488-1-laoar.shao@gmail.com> References: <20260311115250.78488-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Background =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D One of our latency-sensitive services reported random CPU pressure spikes. After a thorough investigation, we finally identified the root cause of the CPU pressure spikes. The key kernel stacks are as follows: - Task A 2026-02-14-16:53:40.938243: [CPU198] 2156302(bpftrace) cgrp:4019437 pod:401= 9253 find_kallsyms_symbol+142 module_address_lookup+104 kallsyms_lookup_buildid+203 kallsyms_lookup+20 print_rec+64 t_show+67 seq_read_iter+709 seq_read+165 vfs_read+165 ksys_read+103 __x64_sys_read+25 do_syscall_64+56 entry_SYSCALL_64_after_hwframe+100 This task (2156302, bpftrace) is reading the /sys/kernel/tracing/available_filter_functions to check if a function is traceable: https://github.com/bpftrace/bpftrace/blob/master/src/tracefs/tracefs.h#L21 Reading the available_filter_functions file is time-consuming, as it contains tens of thousands of functions: $ cat /sys/kernel/tracing/available_filter_functions | wc -l 59221 $ time cat /sys/kernel/tracing/available_filter_functions > /dev/null real 0m0.458s user 0m0.001s sys 0m0.457s Consequently, the ftrace_lock is held by this task for an extended period. - Other Tasks 2026-02-14-16:53:41.437094: [CPU79] 2156308(bpftrace) cgrp:4019437 pod:4019= 253 mutex_spin_on_owner+108 __mutex_lock.constprop.0+1132 __mutex_lock_slowpath+19 mutex_lock+56 t_start+51 seq_read_iter+250 seq_read+165 vfs_read+165 ksys_read+103 __x64_sys_read+25 do_syscall_64+56 entry_SYSCALL_64_after_hwframe+100 Since ftrace_lock is held by Task-A and Task-A is actively running on a CPU, all other tasks waiting for the same lock will spin on their respective CPUs. This leads to increased CPU pressure. Reproduction =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This issue can be reproduced simply by running `cat available_filter_functions`. - Single process reading available_filter_functions: $ time cat /sys/kernel/tracing/available_filter_functions > /dev/null real 0m0.458s user 0m0.001s sys 0m0.457s - Six processes reading available_filter_functions simultaneously: for i in `seq 0 5`; do time cat /sys/kernel/tracing/available_filter_functions > /dev/null & done The results are as follows: real 0m2.666s user 0m0.001s sys 0m2.557s real 0m2.718s user 0m0.000s sys 0m2.655s real 0m2.718s user 0m0.001s sys 0m2.600s real 0m2.733s user 0m0.001s sys 0m2.554s real 0m2.735s user 0m0.000s sys 0m2.573s real 0m2.738s user 0m0.000s sys 0m2.664s As more processes are added, the system time increases correspondingly. Solution =3D=3D=3D=3D=3D=3D=3D=3D One approach is to optimize the reading of available_filter_functions to make it as fast as possible. However, the risk lies in the contention caused by optimistic spin locking. Therefore, we need to consider an alternative solution that avoids optimistic spinning for heavy mutexes that may be held for long durations. Note that we do not want to disable CONFIG_MUTEX_SPIN_ON_OWNER entirely, as that could lead to unexpected performance regressions. In this patch, two new APIs are introduced to allow heavy locks to selectively disable optimistic spinning. slow_mutex_lock() - lock a mutex without optimistic spinning slow_mutex_unlock() - unlock the slow mutex - The result of this optimization After applying this slow mutex to ftrace_lock and concurrently running six processes, the results are as follows: real 0m2.691s user 0m0.001s sys 0m0.458s real 0m2.785s user 0m0.001s sys 0m0.467s real 0m2.787s user 0m0.000s sys 0m0.469s real 0m2.787s user 0m0.000s sys 0m0.466s real 0m2.788s user 0m0.001s sys 0m0.468s real 0m2.789s user 0m0.000s sys 0m0.471s The system time remains similar to that of running a single process. Signed-off-by: Yafang Shao --- include/linux/mutex.h | 4 ++++ kernel/locking/mutex.c | 41 ++++++++++++++++++++++++++++++++++------- 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/include/linux/mutex.h b/include/linux/mutex.h index ecaa0440f6ec..eed0e87c084c 100644 --- a/include/linux/mutex.h +++ b/include/linux/mutex.h @@ -189,11 +189,13 @@ extern int __must_check mutex_lock_interruptible_nest= ed(struct mutex *lock, extern int __must_check _mutex_lock_killable(struct mutex *lock, unsigned int subclass, struct lockdep_map *nest_lock) __cond_acquires(0,= lock); extern void mutex_lock_io_nested(struct mutex *lock, unsigned int subclass= ) __acquires(lock); +extern void slow_mutex_lock_nested(struct mutex *lock, unsigned int subcla= ss); =20 #define mutex_lock(lock) mutex_lock_nested(lock, 0) #define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(loc= k, 0) #define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL) #define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0) +#define slow_mutex_lock(lock) slow_mutex_lock_nested(lock, 0) =20 #define mutex_lock_nest_lock(lock, nest_lock) \ do { \ @@ -215,6 +217,7 @@ extern void mutex_lock(struct mutex *lock) __acquires(l= ock); extern int __must_check mutex_lock_interruptible(struct mutex *lock) __con= d_acquires(0, lock); extern int __must_check mutex_lock_killable(struct mutex *lock) __cond_acq= uires(0, lock); extern void mutex_lock_io(struct mutex *lock) __acquires(lock); +extern void slow_mutex_lock(struct mutex *lock) __acquires(lock); =20 # define mutex_lock_nested(lock, subclass) mutex_lock(lock) # define mutex_lock_interruptible_nested(lock, subclass) mutex_lock_interr= uptible(lock) @@ -247,6 +250,7 @@ extern int mutex_trylock(struct mutex *lock) __cond_acq= uires(true, lock); #endif =20 extern void mutex_unlock(struct mutex *lock) __releases(lock); +#define slow_mutex_unlock(lock) mutex_unlock(lock) =20 extern int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock) __= cond_acquires(true, lock); =20 diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 2a1d165b3167..5766d824b3fe 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -443,8 +443,11 @@ static inline int mutex_can_spin_on_owner(struct mutex= *lock) */ static __always_inline bool mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx, - struct mutex_waiter *waiter) + struct mutex_waiter *waiter, const bool slow) { + if (slow) + return false; + if (!waiter) { /* * The purpose of the mutex_can_spin_on_owner() function is @@ -577,7 +580,8 @@ EXPORT_SYMBOL(ww_mutex_unlock); static __always_inline int __sched __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int s= ubclass, struct lockdep_map *nest_lock, unsigned long ip, - struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx) + struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx, + const bool slow) { DEFINE_WAKE_Q(wake_q); struct mutex_waiter waiter; @@ -615,7 +619,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas =20 trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN); if (__mutex_trylock(lock) || - mutex_optimistic_spin(lock, ww_ctx, NULL)) { + mutex_optimistic_spin(lock, ww_ctx, NULL, slow)) { /* got the lock, yay! */ lock_acquired(&lock->dep_map, ip); if (ww_ctx) @@ -716,7 +720,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas * to run. */ clear_task_blocked_on(current, lock); - if (mutex_optimistic_spin(lock, ww_ctx, &waiter)) + if (mutex_optimistic_spin(lock, ww_ctx, &waiter, slow)) break; set_task_blocked_on(current, lock); trace_contention_begin(lock, LCB_F_MUTEX); @@ -773,14 +777,21 @@ static int __sched __mutex_lock(struct mutex *lock, unsigned int state, unsigned int subclass, struct lockdep_map *nest_lock, unsigned long ip) { - return __mutex_lock_common(lock, state, subclass, nest_lock, ip, NULL, fa= lse); + return __mutex_lock_common(lock, state, subclass, nest_lock, ip, NULL, fa= lse, false); +} + +static int __sched +__slow_mutex_lock(struct mutex *lock, unsigned int state, unsigned int sub= class, + struct lockdep_map *nest_lock, unsigned long ip) +{ + return __mutex_lock_common(lock, state, subclass, nest_lock, ip, NULL, fa= lse, true); } =20 static int __sched __ww_mutex_lock(struct mutex *lock, unsigned int state, unsigned int subcl= ass, unsigned long ip, struct ww_acquire_ctx *ww_ctx) { - return __mutex_lock_common(lock, state, subclass, NULL, ip, ww_ctx, true); + return __mutex_lock_common(lock, state, subclass, NULL, ip, ww_ctx, true,= false); } =20 /** @@ -861,11 +872,17 @@ mutex_lock_io_nested(struct mutex *lock, unsigned int= subclass) =20 token =3D io_schedule_prepare(); __mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, - subclass, NULL, _RET_IP_, NULL, 0); + subclass, NULL, _RET_IP_, NULL, 0, false); io_schedule_finish(token); } EXPORT_SYMBOL_GPL(mutex_lock_io_nested); =20 +void __sched +slow_mutex_lock_nested(struct mutex *lock, unsigned int subclass) +{ + __slow_mutex_lock(lock, TASK_UNINTERRUPTIBLE, subclass, NULL, _RET_IP_); +} + static inline int ww_mutex_deadlock_injection(struct ww_mutex *lock, struct ww_acquire_ctx *= ctx) { @@ -923,6 +940,16 @@ ww_mutex_lock_interruptible(struct ww_mutex *lock, str= uct ww_acquire_ctx *ctx) } EXPORT_SYMBOL_GPL(ww_mutex_lock_interruptible); =20 +#else + +void __sched slow_mutex_lock(struct mutex *lock) +{ + might_sleep(); + + if (!__mutex_trylock_fast(lock)) + __slow_mutex_lock(lock, TASK_UNINTERRUPTIBLE, 0, NULL, _RET_IP_); +} + #endif =20 /* --=20 2.47.3 From nobody Tue Apr 7 22:01:07 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D2BD3DE424 for ; Wed, 11 Mar 2026 11:53:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773229998; cv=none; b=PKzvoL3QXx4oeASA5sTx3e6yIpq9YjZptg8rdYzrl1hu96+Rsx1YUAr4FXJALqsSRUd17K8GI5Be2lcIMppqY8V6ucIoh2V4O7ty4vkCaFRUSh8n+uoGxadM+OwMRwiRYy1QUOoaPkf5W15PIkDUtP9z2lCRJyc/jxm7EAd0O/4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773229998; c=relaxed/simple; bh=0Ifq6XmVDs1fpC0j/ZIbpmcmAOkRHB7zh4IlFJ+hrPA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bwICWk17La6iqBRAXMjV7QMv79smIHC6drAH3D/+fzyYLazwUWHfsQ8jDRZK/0L4DS1FV2ZELJ8SOum7mhnR4O/A6MJgXhcu3hLsFKAeMQXskVXrCvEIOmmgl6baQoF+a74q9pCsPC+WQzOllqQpPV54jwPw6+45MFpT727Ol/w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UYer6CG2; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UYer6CG2" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-829abaaa92bso2586956b3a.1 for ; Wed, 11 Mar 2026 04:53:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773229996; x=1773834796; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8J94ffxrGM62oAfDCHAJ0pM0R8c1cTRpTTHeUMpVEkI=; b=UYer6CG2Qk+1RQ0sFdcJogoULr9wpNO4LFRhlyocSUTeRH0zC/WN5npgdGZpWteCn8 qWCaVSElzNrOP4fX1WLf4O3SH412LNz7cpCb9/G1btS4jckBqrFBZwv1hleXw6CshboL MjMV1gVUfgVcsbCKKDvGldNt4WiVPag7YRHMAi9VRS/x3SMDHcfWSbk4v9smy2emHBYu VitmMMj/JkdP9cxhOBdE2RR1ZQnotXJqgJdWf1SkNxO1Qs7oLmOz0KBNoiEGQsoXEt4X gw/U720tNcH4HoypLTCI4VcjgTNE3eknMYHTU1uwy/k0Tc3d3qJWApVZhFz7DtxBYFVX BmFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773229996; x=1773834796; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8J94ffxrGM62oAfDCHAJ0pM0R8c1cTRpTTHeUMpVEkI=; b=IjoQXecQIxDzqcgSfyvqZBibwi1TEEkh61dlA/zfV1h5B4TxKYfP91PeJ9INr1qtCr MXEjNPug6HMVh0K93nzIowmHaqeXa6K/MeX9wOsBTAMn5H4he0cMNW8VEjHkApLafOmd n4ne2N/dUU2urEdPCK7l834QBI2+c4dZb2ye2yZDuhCfVpeIX3qoU1ztGy1mXE4c1gce +0YiWbWSxGP9CawLTaWVs2TMS2Om+CLG/DaEXtmeKS2vB0WnLGyh0GqXewEyavQYv173 TL8f3kkRpjWz/WFUXxH6a74r9PRAWP/DcHKWhAL9guZ4E9ZSSQvTSP/ouK5FrvnKic6o p1zA== X-Gm-Message-State: AOJu0Yyix9PqOwnZi2+qixEW3ocrOz17424Hpo9+7le62CykraekuGL+ mCvQOTi6/GExlvH/aEFku6Xma6k4ry6SYX5/dXyXzviRKFyWfsHO4xUZ X-Gm-Gg: ATEYQzxgaGvayUWLAx4C3uG1lu8qYpqzPjJofYBctzTAygxqohEIBnlN31756pl29vy EcoA5X+4E6p7YO/oxYH0h0AzS9GbVKgQUGhuPE3gxiCzepy5SY3YGW5C+32WRJzuMiYs2dw4RFn zT7RFiLLmk91ML608gBLl/jL26Ud4Te9dYbd6WbQSgWjzegNuw1u7MqCNu6Pd5qH6XzjcdlhM0A xHSTnLAkxoqlD+eH8WE2YRhjRfVZ5e6zusrl9sRbKyG0e6PN8booS5yU+q8dTJ4yF7CO/f0Kh3n 0LEmZtIp5uXCwcgCM5g9zbfrJNvS7yj3qIJDASbPaSQveQHhZEuTX6DYuUKf1WR3RPsOCN4Sy+B Duv+BhmXtwJ2MzNVf4ZwHruZ89NfYryRTx5Wy6dPCBO6x1p9lta1NQV0uTH9rkPbo+G8v+4JMH5 OTjyD2ISj6SECdR8eI2pLnVcj6bs2G0Bga35gZFVB6LAnM3ahxBx+82R+M+ByKYeQ= X-Received: by 2002:a05:6a00:44c5:b0:829:800b:9fe with SMTP id d2e1a72fcca58-829f706890cmr2213115b3a.39.1773229996428; Wed, 11 Mar 2026 04:53:16 -0700 (PDT) Received: from localhost.localdomain ([2409:891f:1b46:28af:1445:b4e2:b8bf:2d47]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-829f6df3aedsm2500588b3a.18.2026.03.11.04.53.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 11 Mar 2026 04:53:16 -0700 (PDT) From: Yafang Shao To: peterz@infradead.org, mingo@redhat.com, will@kernel.org, boqun@kernel.org, longman@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, david.laight.linux@gmail.com Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yafang Shao Subject: [RFC PATCH v2 2/3] locking/rtmutex: Add slow path variants for lock/unlock Date: Wed, 11 Mar 2026 19:52:49 +0800 Message-ID: <20260311115250.78488-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260311115250.78488-1-laoar.shao@gmail.com> References: <20260311115250.78488-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add slow mutex APIs for rtmutex: slow_rt_mutex_lock: lock a rtmutex without optimistic spinning slow_rt_mutex_unlock: unlock the slow rtmutex Signed-off-by: Yafang Shao --- include/linux/rtmutex.h | 3 +++ kernel/locking/rtmutex.c | 37 +++++++++++++++++----------- kernel/locking/rtmutex_api.c | 47 ++++++++++++++++++++++++++++++------ 3 files changed, 66 insertions(+), 21 deletions(-) diff --git a/include/linux/rtmutex.h b/include/linux/rtmutex.h index ede4c6bf6f22..22294a916ddc 100644 --- a/include/linux/rtmutex.h +++ b/include/linux/rtmutex.h @@ -109,6 +109,7 @@ extern void __rt_mutex_init(struct rt_mutex *lock, cons= t char *name, struct lock =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC extern void rt_mutex_lock_nested(struct rt_mutex *lock, unsigned int subcl= ass); +extern void slow_rt_mutex_lock_nested(struct rt_mutex *lock, unsigned int = subclass); extern void _rt_mutex_lock_nest_lock(struct rt_mutex *lock, struct lockdep= _map *nest_lock); #define rt_mutex_lock(lock) rt_mutex_lock_nested(lock, 0) #define rt_mutex_lock_nest_lock(lock, nest_lock) \ @@ -116,9 +117,11 @@ extern void _rt_mutex_lock_nest_lock(struct rt_mutex *= lock, struct lockdep_map * typecheck(struct lockdep_map *, &(nest_lock)->dep_map); \ _rt_mutex_lock_nest_lock(lock, &(nest_lock)->dep_map); \ } while (0) +#define slow_rt_mutex_lock(lock) slow_rt_mutex_lock_nested(lock, 0) =20 #else extern void rt_mutex_lock(struct rt_mutex *lock); +extern void slow_rt_mutex_lock(struct rt_mutex *lock); #define rt_mutex_lock_nested(lock, subclass) rt_mutex_lock(lock) #define rt_mutex_lock_nest_lock(lock, nest_lock) rt_mutex_lock(lock) #endif diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index c80902eacd79..663ff96cb1be 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -1480,10 +1480,13 @@ static __always_inline void __rt_mutex_unlock(struc= t rt_mutex_base *lock) #ifdef CONFIG_SMP static bool rtmutex_spin_on_owner(struct rt_mutex_base *lock, struct rt_mutex_waiter *waiter, - struct task_struct *owner) + struct task_struct *owner, + const bool slow) { bool res =3D true; =20 + if (slow) + return false; rcu_read_lock(); for (;;) { /* If owner changed, trylock again. */ @@ -1517,7 +1520,8 @@ static bool rtmutex_spin_on_owner(struct rt_mutex_bas= e *lock, #else static bool rtmutex_spin_on_owner(struct rt_mutex_base *lock, struct rt_mutex_waiter *waiter, - struct task_struct *owner) + struct task_struct *owner, + const bool slow) { return false; } @@ -1606,7 +1610,8 @@ static int __sched rt_mutex_slowlock_block(struct rt_= mutex_base *lock, unsigned int state, struct hrtimer_sleeper *timeout, struct rt_mutex_waiter *waiter, - struct wake_q_head *wake_q) + struct wake_q_head *wake_q, + const bool slow) __releases(&lock->wait_lock) __acquires(&lock->wait_lock) { struct rt_mutex *rtm =3D container_of(lock, struct rt_mutex, rtmutex); @@ -1642,7 +1647,7 @@ static int __sched rt_mutex_slowlock_block(struct rt_= mutex_base *lock, owner =3D NULL; raw_spin_unlock_irq_wake(&lock->wait_lock, wake_q); =20 - if (!owner || !rtmutex_spin_on_owner(lock, waiter, owner)) { + if (!owner || !rtmutex_spin_on_owner(lock, waiter, owner, slow)) { lockevent_inc(rtmutex_slow_sleep); rt_mutex_schedule(); } @@ -1693,7 +1698,8 @@ static int __sched __rt_mutex_slowlock(struct rt_mute= x_base *lock, unsigned int state, enum rtmutex_chainwalk chwalk, struct rt_mutex_waiter *waiter, - struct wake_q_head *wake_q) + struct wake_q_head *wake_q, + const bool slow) { struct rt_mutex *rtm =3D container_of(lock, struct rt_mutex, rtmutex); struct ww_mutex *ww =3D ww_container_of(rtm); @@ -1718,7 +1724,7 @@ static int __sched __rt_mutex_slowlock(struct rt_mute= x_base *lock, =20 ret =3D task_blocks_on_rt_mutex(lock, waiter, current, ww_ctx, chwalk, wa= ke_q); if (likely(!ret)) - ret =3D rt_mutex_slowlock_block(lock, ww_ctx, state, NULL, waiter, wake_= q); + ret =3D rt_mutex_slowlock_block(lock, ww_ctx, state, NULL, waiter, wake_= q, slow); =20 if (likely(!ret)) { /* acquired the lock */ @@ -1749,7 +1755,8 @@ static int __sched __rt_mutex_slowlock(struct rt_mute= x_base *lock, static inline int __rt_mutex_slowlock_locked(struct rt_mutex_base *lock, struct ww_acquire_ctx *ww_ctx, unsigned int state, - struct wake_q_head *wake_q) + struct wake_q_head *wake_q, + const bool slow) { struct rt_mutex_waiter waiter; int ret; @@ -1758,7 +1765,7 @@ static inline int __rt_mutex_slowlock_locked(struct r= t_mutex_base *lock, waiter.ww_ctx =3D ww_ctx; =20 ret =3D __rt_mutex_slowlock(lock, ww_ctx, state, RT_MUTEX_MIN_CHAINWALK, - &waiter, wake_q); + &waiter, wake_q, slow); =20 debug_rt_mutex_free_waiter(&waiter); lockevent_cond_inc(rtmutex_slow_wake, !wake_q_empty(wake_q)); @@ -1773,7 +1780,8 @@ static inline int __rt_mutex_slowlock_locked(struct r= t_mutex_base *lock, */ static int __sched rt_mutex_slowlock(struct rt_mutex_base *lock, struct ww_acquire_ctx *ww_ctx, - unsigned int state) + unsigned int state, + const bool slow) { DEFINE_WAKE_Q(wake_q); unsigned long flags; @@ -1797,7 +1805,7 @@ static int __sched rt_mutex_slowlock(struct rt_mutex_= base *lock, * irqsave/restore variants. */ raw_spin_lock_irqsave(&lock->wait_lock, flags); - ret =3D __rt_mutex_slowlock_locked(lock, ww_ctx, state, &wake_q); + ret =3D __rt_mutex_slowlock_locked(lock, ww_ctx, state, &wake_q, slow); raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q); rt_mutex_post_schedule(); =20 @@ -1805,14 +1813,14 @@ static int __sched rt_mutex_slowlock(struct rt_mute= x_base *lock, } =20 static __always_inline int __rt_mutex_lock(struct rt_mutex_base *lock, - unsigned int state) + unsigned int state, const bool slow) { lockdep_assert(!current->pi_blocked_on); =20 if (likely(rt_mutex_try_acquire(lock))) return 0; =20 - return rt_mutex_slowlock(lock, NULL, state); + return rt_mutex_slowlock(lock, NULL, state, slow); } #endif /* RT_MUTEX_BUILD_MUTEX */ =20 @@ -1827,7 +1835,8 @@ static __always_inline int __rt_mutex_lock(struct rt_= mutex_base *lock, * @wake_q: The wake_q to wake tasks after we release the wait_lock */ static void __sched rtlock_slowlock_locked(struct rt_mutex_base *lock, - struct wake_q_head *wake_q) + struct wake_q_head *wake_q, + const bool slow) __releases(&lock->wait_lock) __acquires(&lock->wait_lock) { struct rt_mutex_waiter waiter; @@ -1863,7 +1872,7 @@ static void __sched rtlock_slowlock_locked(struct rt_= mutex_base *lock, owner =3D NULL; raw_spin_unlock_irq_wake(&lock->wait_lock, wake_q); =20 - if (!owner || !rtmutex_spin_on_owner(lock, &waiter, owner)) { + if (!owner || !rtmutex_spin_on_owner(lock, &waiter, owner, slow)) { lockevent_inc(rtlock_slow_sleep); schedule_rtlock(); } diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c index 59dbd29cb219..b196cdd35ff1 100644 --- a/kernel/locking/rtmutex_api.c +++ b/kernel/locking/rtmutex_api.c @@ -37,21 +37,29 @@ subsys_initcall(init_rtmutex_sysctl); * The atomic acquire/release ops are compiled away, when either the * architecture does not support cmpxchg or when debugging is enabled. */ -static __always_inline int __rt_mutex_lock_common(struct rt_mutex *lock, +static __always_inline int ___rt_mutex_lock_common(struct rt_mutex *lock, unsigned int state, struct lockdep_map *nest_lock, - unsigned int subclass) + unsigned int subclass, + const bool slow) { int ret; =20 might_sleep(); mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, _RET_IP_); - ret =3D __rt_mutex_lock(&lock->rtmutex, state); + ret =3D __rt_mutex_lock(&lock->rtmutex, state, slow); if (ret) mutex_release(&lock->dep_map, _RET_IP_); return ret; } =20 +static __always_inline int __rt_mutex_lock_common(struct rt_mutex *lock, + unsigned int state, + struct lockdep_map *nest_lock, + unsigned int subclass) +{ + return ___rt_mutex_lock_common(lock, state, nest_lock, subclass, false); +} void rt_mutex_base_init(struct rt_mutex_base *rtb) { __rt_mutex_base_init(rtb); @@ -77,6 +85,11 @@ void __sched _rt_mutex_lock_nest_lock(struct rt_mutex *l= ock, struct lockdep_map } EXPORT_SYMBOL_GPL(_rt_mutex_lock_nest_lock); =20 +void __sched slow_rt_mutex_lock_nested(struct rt_mutex *lock, unsigned int= subclass) +{ + ___rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, subclass, true); +} + #else /* !CONFIG_DEBUG_LOCK_ALLOC */ =20 /** @@ -89,6 +102,11 @@ void __sched rt_mutex_lock(struct rt_mutex *lock) __rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, 0); } EXPORT_SYMBOL_GPL(rt_mutex_lock); + +void __sched slow_rt_mutex_lock(struct rt_mutex *lock) +{ + ___rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, 0, true); +} #endif =20 /** @@ -401,7 +419,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_ba= se *lock, raw_spin_lock_irq(&lock->wait_lock); /* sleep on the mutex */ set_current_state(TASK_INTERRUPTIBLE); - ret =3D rt_mutex_slowlock_block(lock, NULL, TASK_INTERRUPTIBLE, to, waite= r, NULL); + ret =3D rt_mutex_slowlock_block(lock, NULL, TASK_INTERRUPTIBLE, to, waite= r, NULL, false); /* * try_to_take_rt_mutex() sets the waiter bit unconditionally. We might * have to fix that up. @@ -521,17 +539,18 @@ static void __mutex_rt_init_generic(struct mutex *mut= ex) debug_check_no_locks_freed((void *)mutex, sizeof(*mutex)); } =20 -static __always_inline int __mutex_lock_common(struct mutex *lock, +static __always_inline int ___mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclass, struct lockdep_map *nest_lock, - unsigned long ip) + unsigned long ip, + const bool slow) { int ret; =20 might_sleep(); mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip); - ret =3D __rt_mutex_lock(&lock->rtmutex, state); + ret =3D __rt_mutex_lock(&lock->rtmutex, state, slow); if (ret) mutex_release(&lock->dep_map, ip); else @@ -539,6 +558,15 @@ static __always_inline int __mutex_lock_common(struct = mutex *lock, return ret; } =20 +static __always_inline int __mutex_lock_common(struct mutex *lock, + unsigned int state, + unsigned int subclass, + struct lockdep_map *nest_lock, + unsigned long ip) +{ + ___mutex_lock_common(lock, state, subclass, nest_lock, ip, false); +} + #ifdef CONFIG_DEBUG_LOCK_ALLOC void mutex_rt_init_lockdep(struct mutex *mutex, const char *name, struct l= ock_class_key *key) { @@ -644,6 +672,11 @@ int __sched mutex_trylock(struct mutex *lock) return __rt_mutex_trylock(&lock->rtmutex); } EXPORT_SYMBOL(mutex_trylock); + +void __sched slow_mutex_lock(struct mutex *lock) +{ + ___mutex_lock_common(lock, state, subclass, nest_lock, ip, true); +} #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ =20 void __sched mutex_unlock(struct mutex *lock) --=20 2.47.3 From nobody Tue Apr 7 22:01:07 2026 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1920A3DE42A for ; Wed, 11 Mar 2026 11:53:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773230003; cv=none; b=O/2RKaGTO0ohjEoKo2w2cGo/dZi0ZSp+OOB993/wfVj385d+fEaGTO+d8zJjMrYLt8CLsSvdqaFwQRvdM+fD8uv/xqBnnWuR/WhtgPQdtJJa6LFXjAWzWan3CyXc7JKxgQvH2TSnm7r3FUAJRklp4QDAup5gFKYkBdSVOJ2MBQk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773230003; c=relaxed/simple; bh=HhxX14UkA4Md0FHrn0YoGqjjtgcFeX0wk8i706bX378=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ejh6bAvcNVDelLiuyyH1p/+JfxMA8qjTr+vpLfl+Hj3soe4rN5ynFaSdKTTckiU4Bnw/UzpMISeNHBkZHiYxZFDU+MMV57x3IlcR9yqNzyIAn6YD3UReppfYLwrwleBiAi82WybwMvBo37aWJ0xIvmWISNyBSuXhvbw1I/VSYzE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CbSOKF5M; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CbSOKF5M" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-829b2019b39so2540283b3a.3 for ; Wed, 11 Mar 2026 04:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773230000; x=1773834800; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5XUaqMDdHb6xWSLB7LHHRKEGcmYQutagR+4ik5kvWaI=; b=CbSOKF5MrEPdoMCmt9kqnbnUc8KZEpXee2cAccJ/lcqSCwFkmY7+MFmyCFlxQo5QqT 2g/hBCKYUQogSpxrAQXLmUSJHMo9LitdSaQ3j7GfVKwc7ADNZAKpeVRqR9+uwVxhx5bk o0GuMW/v7XfV+8suAiDPO83SpqYHiEpsUz17u2aLCNN+Xg8eRNT9CuQeQ44Upv2ZlAIO uvknmxvqNeLsT8+S7UrUbGZW4xjiT1pKjYjca0xuBSovtFe27z8S+KFdby5/0RNcs5wW 3Z7KDMqB6fqwe1ViCtIDDLVi+NazBfY6GKccH180YxdSnswb1+3hNXgGVsANpD+UwlhZ NReQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773230000; x=1773834800; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5XUaqMDdHb6xWSLB7LHHRKEGcmYQutagR+4ik5kvWaI=; b=G69znRTuLtR1S8H0szMNs5r/NeaolaoMlfEM3FwJEicz9EKPDgnzfUW3p9/ScVoE8x uOxWuHNBlf7412eV/eFt3lVkOdTff2D9leR0+yd+LruvubIqph8iatYtrHfcsouqxCcr yNYNmZU7DBv63WTH/+uO23fFerAQWpM3tnmRMGs5XGuIHettIcpMbI3uFEPraFo1No5P Q0oCMI2wKcacDbvM1/oLM9+DyXb1BtvjAAtgGYWpHhW9oSZl3WzhRhU9WzWuV38gKt/g 7l0Df4OATT7DAyIyt6Ok5QHKfUsZxh5ihLA6CV2ieMBcBO3iGgcl6vnzqoZAbp70jJ5V IH0w== X-Gm-Message-State: AOJu0Yz7lKyczTLpIut3JYTcSbIFcl4H/eKJOGXKIXd2LAXitG3u3Tvr aaHPaNkZD/cz57Y/pNE6I+z3H0CUamKKJ26DO0NqOQ68RWT/vsN82/b3 X-Gm-Gg: ATEYQzxYHRTZUwYqcf3EZvmkmJtmQWpMWzbtH6+zgZPX+RUFspLyXR9CWyFsG1iNlUF 3Tz6uqYW8xC4zFGQTa8qRLkV6uIq5kGdB7gYprrhTH3QWiqeBE63jHnBoa0XrbboXk6HH7D5m0R 8Ro2JuohiB4CiynahuWayi89QESsMQvcheIcWG4JEVfKRbzRMfBWrVJ3OfOBStbiy3pMnlbctA7 Gymwgv1aYMFP+xdVsDFBkObRGJ5F6CZlPhpDXgiNGCLav8E5PD7Ge7f0H4dfp8b/9OwXEvfKNx0 uAR4q+nfOMh4BQJJ2TDwXCWhghJuosj8rPQCWKEUt0uczyeytI6Ek7t70fBv4P5ZCVihkYf0gFC 9PTkjWVVzIrAzxEbsxfT1jNipAB+L3LFAkdut9x8lxMtV0sSvm2l/kozO9WPaI2rUXpTv3nMbty tjlXzGJJnbWyL6Uya1liUBnECGnqtQZp2qXM0PmsbuzSrbotQ4DYdiTsIBglk7nzBxKNC+XAoEP w== X-Received: by 2002:a05:6a00:ab89:b0:81e:d18a:489d with SMTP id d2e1a72fcca58-829f71da9e4mr2364945b3a.42.1773230000395; Wed, 11 Mar 2026 04:53:20 -0700 (PDT) Received: from localhost.localdomain ([2409:891f:1b46:28af:1445:b4e2:b8bf:2d47]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-829f6df3aedsm2500588b3a.18.2026.03.11.04.53.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 11 Mar 2026 04:53:19 -0700 (PDT) From: Yafang Shao To: peterz@infradead.org, mingo@redhat.com, will@kernel.org, boqun@kernel.org, longman@redhat.com, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, mathieu.desnoyers@efficios.com, david.laight.linux@gmail.com Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yafang Shao Subject: [RFC PATCH v2 3/3] ftrace: Disable optimistic spinning for ftrace_lock Date: Wed, 11 Mar 2026 19:52:50 +0800 Message-ID: <20260311115250.78488-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260311115250.78488-1-laoar.shao@gmail.com> References: <20260311115250.78488-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The ftrace_lock may be held for a relatively long duration. For example, reading the available_filter_functions file takes considerable time: $ time cat /sys/kernel/tracing/available_filter_functions &> /dev/null real 0m0.457s user 0m0.001s sys 0m0.455s When the lock owner is continuously running, other tasks waiting for the lock will spin repeatedly until they hit a cond_resched() point, wasting CPU cycles. ftrace_lock is currently used in the following scenarios: - Debugging - Live patching Neither of these scenarios are in the application critical path. Therefore, it is reasonable to make tasks sleep when they cannot acquire the lock immediately, rather than spinning and consuming CPU resources. Performance Comparison =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - Before this change - Single task reading available_filter_functions: real 0m0.457s user 0m0.001s sys 0m0.455s - Six concurrent processes: real 0m2.666s user 0m0.001s sys 0m2.557s real 0m2.718s user 0m0.000s sys 0m2.655s real 0m2.718s user 0m0.001s sys 0m2.600s real 0m2.733s user 0m0.001s sys 0m2.554s real 0m2.735s user 0m0.000s sys 0m2.573s real 0m2.738s user 0m0.000s sys 0m2.664s - After this change - Single task: real 0m0.454s user 0m0.002s sys 0m0.453s - Six concurrent processes: real 0m2.691s user 0m0.001s sys 0m0.458s real 0m2.785s user 0m0.001s sys 0m0.467s real 0m2.787s user 0m0.000s sys 0m0.469s real 0m2.787s user 0m0.000s sys 0m0.466s real 0m2.788s user 0m0.001s sys 0m0.468s real 0m2.789s user 0m0.000s sys 0m0.471s The system time significantly decreases in the concurrent case, as tasks now sleep while waiting for the lock instead of busy-spinning. Signed-off-by: Yafang Shao --- kernel/trace/ftrace.c | 106 +++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 53 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 827fb9a0bf0d..00c195e280c5 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -1284,10 +1284,10 @@ static void clear_ftrace_mod_list(struct list_head = *head) if (!head) return; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); list_for_each_entry_safe(p, n, head, list) free_ftrace_mod(p); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 void free_ftrace_hash(struct ftrace_hash *hash) @@ -4254,7 +4254,7 @@ static void *t_start(struct seq_file *m, loff_t *pos) void *p =3D NULL; loff_t l; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 if (unlikely(ftrace_disabled)) return NULL; @@ -4305,7 +4305,7 @@ static void *t_start(struct seq_file *m, loff_t *pos) =20 static void t_stop(struct seq_file *m, void *p) { - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 void * __weak @@ -4362,11 +4362,11 @@ static __init void ftrace_check_work_func(struct wo= rk_struct *work) struct ftrace_page *pg; struct dyn_ftrace *rec; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); do_for_each_ftrace_rec(pg, rec) { test_for_valid_rec(rec); } while_for_each_ftrace_rec(); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 static int __init ftrace_check_for_weak_functions(void) @@ -5123,7 +5123,7 @@ static void process_mod_list(struct list_head *head, = struct ftrace_ops *ops, if (!new_hash) goto out; /* warn? */ =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 list_for_each_entry_safe(ftrace_mod, n, head, list) { =20 @@ -5145,7 +5145,7 @@ static void process_mod_list(struct list_head *head, = struct ftrace_ops *ops, ftrace_mod->func =3D func; } =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 list_for_each_entry_safe(ftrace_mod, n, &process_mods, list) { =20 @@ -5159,11 +5159,11 @@ static void process_mod_list(struct list_head *head= , struct ftrace_ops *ops, if (enable && list_empty(head)) new_hash->flags &=3D ~FTRACE_HASH_FL_MOD; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 ftrace_hash_move_and_update_ops(ops, orig_hash, new_hash, enable); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 out: mutex_unlock(&ops->func_hash->regex_lock); @@ -5465,7 +5465,7 @@ register_ftrace_function_probe(char *glob, struct tra= ce_array *tr, return -EINVAL; =20 =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); /* Check if the probe_ops is already registered */ list_for_each_entry(iter, &tr->func_probes, list) { if (iter->probe_ops =3D=3D probe_ops) { @@ -5476,7 +5476,7 @@ register_ftrace_function_probe(char *glob, struct tra= ce_array *tr, if (!probe) { probe =3D kzalloc_obj(*probe); if (!probe) { - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); return -ENOMEM; } probe->probe_ops =3D probe_ops; @@ -5488,7 +5488,7 @@ register_ftrace_function_probe(char *glob, struct tra= ce_array *tr, =20 acquire_probe_locked(probe); =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 /* * Note, there's a small window here that the func_hash->filter_hash @@ -5540,7 +5540,7 @@ register_ftrace_function_probe(char *glob, struct tra= ce_array *tr, } } =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 if (!count) { /* Nothing was added? */ @@ -5560,7 +5560,7 @@ register_ftrace_function_probe(char *glob, struct tra= ce_array *tr, ret =3D ftrace_startup(&probe->ops, 0); =20 out_unlock: - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 if (!ret) ret =3D count; @@ -5619,7 +5619,7 @@ unregister_ftrace_function_probe_func(char *glob, str= uct trace_array *tr, return -EINVAL; } =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); /* Check if the probe_ops is already registered */ list_for_each_entry(iter, &tr->func_probes, list) { if (iter->probe_ops =3D=3D probe_ops) { @@ -5636,7 +5636,7 @@ unregister_ftrace_function_probe_func(char *glob, str= uct trace_array *tr, =20 acquire_probe_locked(probe); =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 mutex_lock(&probe->ops.func_hash->regex_lock); =20 @@ -5679,7 +5679,7 @@ unregister_ftrace_function_probe_func(char *glob, str= uct trace_array *tr, goto out_unlock; } =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 WARN_ON(probe->ref < count); =20 @@ -5703,7 +5703,7 @@ unregister_ftrace_function_probe_func(char *glob, str= uct trace_array *tr, probe_ops->free(probe_ops, tr, entry->ip, probe->data); kfree(entry); } - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 out_unlock: mutex_unlock(&probe->ops.func_hash->regex_lock); @@ -5714,7 +5714,7 @@ unregister_ftrace_function_probe_func(char *glob, str= uct trace_array *tr, return ret; =20 err_unlock_ftrace: - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); return ret; } =20 @@ -5943,9 +5943,9 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char= *buf, int len, goto out_regex_unlock; } =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); ret =3D ftrace_hash_move_and_update_ops(ops, orig_hash, hash, enable); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 out_regex_unlock: mutex_unlock(&ops->func_hash->regex_lock); @@ -6205,7 +6205,7 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr) * Now the ftrace_ops_list_func() is called to do the direct callers. * We can safely change the direct functions attached to each entry. */ - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { @@ -6219,7 +6219,7 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr) /* Prevent store tearing if a trampoline concurrently accesses the value = */ WRITE_ONCE(ops->direct_call, addr); =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 out: /* Removing the tmp_ops will add the updated direct callers to the functi= ons */ @@ -6625,7 +6625,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, = struct ftrace_hash *hash, b * Now the ftrace_ops_list_func() is called to do the direct callers. * We can safely change the direct functions attached to each entry. */ - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 size =3D 1 << hash->size_bits; for (i =3D 0; i < size; i++) { @@ -6637,7 +6637,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, = struct ftrace_hash *hash, b } } =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 out: /* Removing the tmp_ops will add the updated direct callers to the functi= ons */ @@ -6980,10 +6980,10 @@ int ftrace_regex_release(struct inode *inode, struc= t file *file) } else orig_hash =3D &iter->ops->func_hash->notrace_hash; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); ftrace_hash_move_and_update_ops(iter->ops, orig_hash, iter->hash, filter_hash); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 mutex_unlock(&iter->ops->func_hash->regex_lock); @@ -7464,12 +7464,12 @@ void ftrace_create_filter_files(struct ftrace_ops *= ops, */ void ftrace_destroy_filter_files(struct ftrace_ops *ops) { - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); if (ops->flags & FTRACE_OPS_FL_ENABLED) ftrace_shutdown(ops, 0); ops->flags |=3D FTRACE_OPS_FL_DELETED; ftrace_free_filter(ops); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) @@ -7571,7 +7571,7 @@ static int ftrace_process_locs(struct module *mod, if (!start_pg) return -ENOMEM; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 /* * Core and each module needs their own pages, as @@ -7661,7 +7661,7 @@ static int ftrace_process_locs(struct module *mod, local_irq_restore(flags); ret =3D 0; out: - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 /* We should have used all pages unless we skipped some */ if (pg_unuse) { @@ -7868,7 +7868,7 @@ void ftrace_release_mod(struct module *mod) struct ftrace_page *tmp_page =3D NULL; struct ftrace_page *pg; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 /* * To avoid the UAF problem after the module is unloaded, the @@ -7913,7 +7913,7 @@ void ftrace_release_mod(struct module *mod) last_pg =3D &pg->next; } out_unlock: - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 /* Need to synchronize with ftrace_location_range() */ if (tmp_page) @@ -7938,7 +7938,7 @@ void ftrace_module_enable(struct module *mod) struct dyn_ftrace *rec; struct ftrace_page *pg; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 if (ftrace_disabled) goto out_unlock; @@ -8008,7 +8008,7 @@ void ftrace_module_enable(struct module *mod) ftrace_arch_code_modify_post_process(); =20 out_unlock: - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 process_cached_mods(mod->name); } @@ -8267,7 +8267,7 @@ void ftrace_free_mem(struct module *mod, void *start_= ptr, void *end_ptr) key.ip =3D start; key.flags =3D end; /* overload flags, as it is unsigned long */ =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 /* * If we are freeing module init memory, then check if @@ -8310,7 +8310,7 @@ void ftrace_free_mem(struct module *mod, void *start_= ptr, void *end_ptr) /* More than one function may be in this block */ goto again; } - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 list_for_each_entry_safe(func, func_next, &clear_hash, list) { clear_func_from_hashes(func); @@ -8686,22 +8686,22 @@ static void clear_ftrace_pids(struct trace_array *t= r, int type) =20 void ftrace_clear_pids(struct trace_array *tr) { - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 clear_ftrace_pids(tr, TRACE_PIDS | TRACE_NO_PIDS); =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 static void ftrace_pid_reset(struct trace_array *tr, int type) { - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); clear_ftrace_pids(tr, type); =20 ftrace_update_pid_func(); ftrace_startup_all(0); =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 /* Greater than any max PID */ @@ -8713,7 +8713,7 @@ static void *fpid_start(struct seq_file *m, loff_t *p= os) struct trace_pid_list *pid_list; struct trace_array *tr =3D m->private; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); rcu_read_lock_sched(); =20 pid_list =3D rcu_dereference_sched(tr->function_pids); @@ -8740,7 +8740,7 @@ static void fpid_stop(struct seq_file *m, void *p) __releases(RCU) { rcu_read_unlock_sched(); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); } =20 static int fpid_show(struct seq_file *m, void *v) @@ -8766,7 +8766,7 @@ static void *fnpid_start(struct seq_file *m, loff_t *= pos) struct trace_pid_list *pid_list; struct trace_array *tr =3D m->private; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); rcu_read_lock_sched(); =20 pid_list =3D rcu_dereference_sched(tr->function_no_pids); @@ -9057,7 +9057,7 @@ static int prepare_direct_functions_for_ipmodify(stru= ct ftrace_ops *ops) unsigned long ip =3D entry->ip; bool found_op =3D false; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); do_for_each_ftrace_op(op, ftrace_ops_list) { if (!(op->flags & FTRACE_OPS_FL_DIRECT)) continue; @@ -9066,7 +9066,7 @@ static int prepare_direct_functions_for_ipmodify(stru= ct ftrace_ops *ops) break; } } while_for_each_ftrace_op(op); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 if (found_op) { if (!op->ops_func) @@ -9106,7 +9106,7 @@ static void cleanup_direct_functions_after_ipmodify(s= truct ftrace_ops *ops) unsigned long ip =3D entry->ip; bool found_op =3D false; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); do_for_each_ftrace_op(op, ftrace_ops_list) { if (!(op->flags & FTRACE_OPS_FL_DIRECT)) continue; @@ -9115,7 +9115,7 @@ static void cleanup_direct_functions_after_ipmodify(s= truct ftrace_ops *ops) break; } } while_for_each_ftrace_op(op); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 /* The cleanup is optional, ignore any errors */ if (found_op && op->ops_func) @@ -9153,11 +9153,11 @@ static int register_ftrace_function_nolock(struct f= trace_ops *ops) =20 ftrace_ops_init(ops); =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); =20 ret =3D ftrace_startup(ops, 0); =20 - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 return ret; } @@ -9200,9 +9200,9 @@ int unregister_ftrace_function(struct ftrace_ops *ops) { int ret; =20 - mutex_lock(&ftrace_lock); + slow_mutex_lock(&ftrace_lock); ret =3D ftrace_shutdown(ops, 0); - mutex_unlock(&ftrace_lock); + slow_mutex_unlock(&ftrace_lock); =20 cleanup_direct_functions_after_ipmodify(ops); return ret; --=20 2.47.3