From nobody Thu Aug 13 09:15:08 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21657421254; Tue, 4 Aug 2026 07:16:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827779; cv=none; b=IYHmIs8IiJs8Jt6E62w0kS1l6ScQAJL3GEt8crDQn1Pyql6lG38Mo5I8lcFI1FUTAD1r8+bAfXS0O3WndnR2yP0GXBeAoEmY4RmJeNR815eYnO4UjOICaMllYLvbFhu3Dai6rp4t0PeJ0PVIHGbzRcE7wqRh4nae8nT04vG7ohs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827779; c=relaxed/simple; bh=JLH7QpKgZ10gQbmXPhOq6m2povwsMw0BnTXBtwhbETo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q71Ljcc6cFVTIMmxh6zsc/4IouW6HXvMv9TdLfCJjtNKkMePu0+vBqOjILl0hK3i7PvFpToU8glNB6XKXTXHMJSFG+WNSEK5el3FottzDQEVAoj5cxPB6+hIzx85icu3AlQ8GlA9FXNmYzBllhjVGt6bfqiTR9swwYbHJgepg30= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=FVNDO3r4; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="FVNDO3r4" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1785827769; bh=Hgqlc//hwpz+K1crcBkrYPgGS5kJOu9yEyqJUnHTuno=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=FVNDO3r4J4MOXT5qmRaXYUu2pNzl9ZurajzLzmFoGAVSzfbChQnc8xUZc5dM6cHjl dl4A/X2nBgfn5GZv2d97IF3LH92AggOHh7cpwlOahEJLSUKNdgOV9q4kQ+BBqIcgV2 hjQ84vsVQbCOHXjwIMxPrXZCCUeUGcpJzNJ/G9uA= Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id AF538E16D0; Tue, 04 Aug 2026 07:16:08 +0000 (UTC) From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , "K. Y. Srinivasan" , Haiyang Zhang , Wei Liu , Dexuan Cui , Long Li , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Paolo Bonzini , Vitaly Kuznetsov , Josh Poimboeuf , Jason Baron , Alice Ryhl , Steven Rostedt , Ard Biesheuvel , Boris Ostrovsky , Arnd Bergmann , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-hyperv@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-arch@vger.kernel.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin Subject: [PATCH 1/5] x86/paravirt: Use static_call() for the paravirt spinlock ops Date: Tue, 4 Aug 2026 07:15:41 +0000 Message-ID: <9a32ae399eb804a02a31af04dcabe7e7ee4f3fdf.1785778551.git.d@ilvokhin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra queued_spin_lock_slowpath() and queued_spin_unlock() are dispatched through pv_ops_lock via the paravirt-ops ALTERNATIVE machinery, which picks the target (native inline store / hypervisor call) once at boot and cannot change at runtime. Convert both to static_call(). The site becomes a direct call patched in place (one byte smaller), and on native the unlock still collapses to the inline "movb $0, (%rdi)" store, so the fast path is unchanged. Unlike the ALTERNATIVE mechanism, a static_call() target can also be updated at runtime via static_call_update(). This is a prerequisite for the contended_release tracepoint, which has to swap in a traced unlock while the system is running. [ ilvokhin: commit message; fix PARAVIRT_SPINLOCKS=3Dn build; teach __static_call_validate() about the inline unlock insn; make the slowpath site module-safe: static_call_mod() + EXPORT_STATIC_CALL_TRAMP(); pass @lock to the callee-save unlock, fixing a boot hang under CALL_DEPTH_TRACKING. Boot tested native + KVM PV guest. ] Link: https://lore.kernel.org/all/20260603120811.GW3493090@noisy.programmin= g.kicks-ass.net/ Co-developed-by: Dmitry Ilvokhin Signed-off-by: Dmitry Ilvokhin Acked-by: Juergen Gross --- arch/x86/hyperv/hv_spinlock.c | 4 ++-- arch/x86/include/asm/cpufeatures.h | 1 - arch/x86/include/asm/paravirt-spinlock.h | 19 +++++++++++------ arch/x86/kernel/kvm.c | 5 ++--- arch/x86/kernel/paravirt-spinlocks.c | 12 +++++------ arch/x86/kernel/static_call.c | 27 ++++++++++++++++++++++++ arch/x86/xen/spinlock.c | 5 ++--- tools/arch/x86/include/asm/cpufeatures.h | 1 - 8 files changed, 51 insertions(+), 23 deletions(-) diff --git a/arch/x86/hyperv/hv_spinlock.c b/arch/x86/hyperv/hv_spinlock.c index 210b494e4de0..6b4bdea18218 100644 --- a/arch/x86/hyperv/hv_spinlock.c +++ b/arch/x86/hyperv/hv_spinlock.c @@ -78,8 +78,8 @@ void __init hv_init_spinlocks(void) pr_info("PV spinlocks enabled\n"); =20 __pv_init_lock_hash(); - pv_ops_lock.queued_spin_lock_slowpath =3D __pv_queued_spin_lock_slowpath; - pv_ops_lock.queued_spin_unlock =3D PV_CALLEE_SAVE(__pv_queued_spin_unlock= ); + static_call_update(queued_spin_lock_slowpath, __pv_queued_spin_lock_slowp= ath); + static_call_update(queued_spin_unlock, __raw_callee_save___pv_queued_spin= _unlock); pv_ops_lock.wait =3D hv_qlock_wait; pv_ops_lock.kick =3D hv_qlock_kick; pv_ops_lock.vcpu_is_preempted =3D PV_CALLEE_SAVE(hv_vcpu_is_preempted); diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpuf= eatures.h index 1b4a48bff18f..e41fe5c24841 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -225,7 +225,6 @@ #define X86_FEATURE_EPT_AD ( 8*32+17) /* "ept_ad" Intel Extended Page Tab= le access-dirty bit */ #define X86_FEATURE_VMCALL ( 8*32+18) /* Hypervisor supports the VMCALL i= nstruction */ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* VMware prefers VMMCALL hype= rcall instruction */ -#define X86_FEATURE_PVUNLOCK ( 8*32+20) /* PV unlock function */ #define X86_FEATURE_VCPUPREEMPT ( 8*32+21) /* PV vcpu_is_preempted functi= on */ #define X86_FEATURE_TDX_GUEST ( 8*32+22) /* "tdx_guest" Intel Trust Domai= n Extensions Guest */ =20 diff --git a/arch/x86/include/asm/paravirt-spinlock.h b/arch/x86/include/as= m/paravirt-spinlock.h index 7beffcb08ed6..ff735830de4a 100644 --- a/arch/x86/include/asm/paravirt-spinlock.h +++ b/arch/x86/include/asm/paravirt-spinlock.h @@ -3,6 +3,7 @@ #define _ASM_X86_PARAVIRT_SPINLOCK_H =20 #include +#include =20 #ifdef CONFIG_SMP #include @@ -11,9 +12,6 @@ struct qspinlock; =20 struct pv_lock_ops { - void (*queued_spin_lock_slowpath)(struct qspinlock *lock, u32 val); - struct paravirt_callee_save queued_spin_unlock; - void (*wait)(u8 *ptr, u8 val); void (*kick)(int cpu); =20 @@ -26,20 +24,27 @@ extern struct pv_lock_ops pv_ops_lock; extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 v= al); extern void __pv_init_lock_hash(void); extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val= ); +extern void __raw_callee_save___native_queued_spin_unlock(struct qspinlock= *lock); extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lo= ck); extern bool nopvspin; =20 +DECLARE_STATIC_CALL(queued_spin_lock_slowpath, native_queued_spin_lock_slo= wpath); +DECLARE_STATIC_CALL(queued_spin_unlock, __raw_callee_save___native_queued_= spin_unlock); + static __always_inline void pv_queued_spin_lock_slowpath(struct qspinlock = *lock, u32 val) { - PVOP_VCALL2(pv_ops_lock, queued_spin_lock_slowpath, lock, val); + static_call_mod(queued_spin_lock_slowpath)(lock, val); } =20 static __always_inline void pv_queued_spin_unlock(struct qspinlock *lock) { - PVOP_ALT_VCALLEE1(pv_ops_lock, queued_spin_unlock, lock, - "movb $0, (%%" _ASM_ARG1 ")", - ALT_NOT(X86_FEATURE_PVUNLOCK)); + PVOP_CALL_ARGS; + __STATIC_CALL_MOD_ADDRESSABLE(queued_spin_unlock); + asm volatile ("call " STATIC_CALL_TRAMP_STR(queued_spin_unlock) + : PVOP_VCALLEE_CLOBBERS, ASM_CALL_CONSTRAINT + : PVOP_CALL_ARG1(lock) + : "memory", "cc"); } =20 static __always_inline bool pv_vcpu_is_preempted(long cpu) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index dcef84da304b..253c159c4abe 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -1136,9 +1136,8 @@ void __init kvm_spinlock_init(void) pr_info("PV spinlocks enabled\n"); =20 __pv_init_lock_hash(); - pv_ops_lock.queued_spin_lock_slowpath =3D __pv_queued_spin_lock_slowpath; - pv_ops_lock.queued_spin_unlock =3D - PV_CALLEE_SAVE(__pv_queued_spin_unlock); + static_call_update(queued_spin_lock_slowpath, __pv_queued_spin_lock_slowp= ath); + static_call_update(queued_spin_unlock, __raw_callee_save___pv_queued_spin= _unlock); pv_ops_lock.wait =3D kvm_wait; pv_ops_lock.kick =3D kvm_kick_cpu; =20 diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravir= t-spinlocks.c index 95452444868f..ddc19dc28ba1 100644 --- a/arch/x86/kernel/paravirt-spinlocks.c +++ b/arch/x86/kernel/paravirt-spinlocks.c @@ -25,9 +25,14 @@ __visible void __native_queued_spin_unlock(struct qspinl= ock *lock) } PV_CALLEE_SAVE_REGS_THUNK(__native_queued_spin_unlock); =20 +DEFINE_STATIC_CALL(queued_spin_lock_slowpath, native_queued_spin_lock_slow= path); +EXPORT_STATIC_CALL_TRAMP(queued_spin_lock_slowpath); +DEFINE_STATIC_CALL(queued_spin_unlock, __raw_callee_save___native_queued_s= pin_unlock); +EXPORT_STATIC_CALL_TRAMP(queued_spin_unlock); + bool pv_is_native_spin_unlock(void) { - return pv_ops_lock.queued_spin_unlock.func =3D=3D + return static_call_query(queued_spin_unlock) =3D=3D __raw_callee_save___native_queued_spin_unlock; } =20 @@ -45,16 +50,11 @@ bool pv_is_native_vcpu_is_preempted(void) =20 void __init paravirt_set_cap(void) { - if (!pv_is_native_spin_unlock()) - setup_force_cpu_cap(X86_FEATURE_PVUNLOCK); - if (!pv_is_native_vcpu_is_preempted()) setup_force_cpu_cap(X86_FEATURE_VCPUPREEMPT); } =20 struct pv_lock_ops pv_ops_lock =3D { - .queued_spin_lock_slowpath =3D native_queued_spin_lock_slowpath, - .queued_spin_unlock =3D PV_CALLEE_SAVE(__native_queued_spin_unlock), .wait =3D paravirt_nop, .kick =3D paravirt_nop, .vcpu_is_preempted =3D PV_CALLEE_SAVE(__native_vcpu_is_preempted), diff --git a/arch/x86/kernel/static_call.c b/arch/x86/kernel/static_call.c index 61592e41a6b1..bab9406e6d6a 100644 --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -4,6 +4,12 @@ #include #include =20 +/* Declared locally to avoid pulling asm/paravirt-spinlock.h header. */ +#ifdef CONFIG_PARAVIRT_SPINLOCKS +struct qspinlock; +void __raw_callee_save___native_queued_spin_unlock(struct qspinlock *lock); +#endif + enum insn_type { CALL =3D 0, /* site call */ NOP =3D 1, /* site cond-call */ @@ -31,6 +37,17 @@ static const u8 retinsn[] =3D { RET_INSN_OPCODE, 0xcc, 0= xcc, 0xcc, 0xcc }; */ static const u8 warninsn[] =3D { 0x67, 0x48, 0x0f, 0xb9, 0x3a }; =20 +#ifdef CONFIG_PARAVIRT_SPINLOCKS +/* + * ds ds movb $0, (_ASM_ARG1) + */ +#ifdef CONFIG_64BIT +static const u8 unlockinsn[] =3D { 0x3e, 0x3e, 0xc6, 0x07, 0x00 }; +#else +static const u8 unlockinsn[] =3D { 0x3e, 0x3e, 0xc6, 0x00, 0x00 }; +#endif +#endif + static u8 __is_Jcc(u8 *insn) /* Jcc.d32 */ { u8 ret =3D 0; @@ -78,6 +95,12 @@ static void __ref __static_call_transform(void *insn, en= um insn_type type, emulate =3D code; code =3D &warninsn; } +#ifdef CONFIG_PARAVIRT_SPINLOCKS + if (func =3D=3D &__raw_callee_save___native_queued_spin_unlock) { + emulate =3D code; + code =3D &unlockinsn; + } +#endif break; =20 case NOP: @@ -139,6 +162,10 @@ static void __static_call_validate(u8 *insn, bool tail= , bool tramp) !memcmp(insn, xor5rax, 5) || !memcmp(insn, warninsn, 5)) return; +#ifdef CONFIG_PARAVIRT_SPINLOCKS + if (!memcmp(insn, unlockinsn, 5)) + return; +#endif } =20 /* diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c index 83ac24ead289..f718e535ea7c 100644 --- a/arch/x86/xen/spinlock.c +++ b/arch/x86/xen/spinlock.c @@ -134,9 +134,8 @@ void __init xen_init_spinlocks(void) printk(KERN_DEBUG "xen: PV spinlocks enabled\n"); =20 __pv_init_lock_hash(); - pv_ops_lock.queued_spin_lock_slowpath =3D __pv_queued_spin_lock_slowpath; - pv_ops_lock.queued_spin_unlock =3D - PV_CALLEE_SAVE(__pv_queued_spin_unlock); + static_call_update(queued_spin_lock_slowpath, __pv_queued_spin_lock_slowp= ath); + static_call_update(queued_spin_unlock, __raw_callee_save___pv_queued_spin= _unlock); pv_ops_lock.wait =3D xen_qlock_wait; pv_ops_lock.kick =3D xen_qlock_kick; pv_ops_lock.vcpu_is_preempted =3D PV_CALLEE_SAVE(xen_vcpu_stolen); diff --git a/tools/arch/x86/include/asm/cpufeatures.h b/tools/arch/x86/incl= ude/asm/cpufeatures.h index 86d17b195e79..61541f042f74 100644 --- a/tools/arch/x86/include/asm/cpufeatures.h +++ b/tools/arch/x86/include/asm/cpufeatures.h @@ -225,7 +225,6 @@ #define X86_FEATURE_EPT_AD ( 8*32+17) /* "ept_ad" Intel Extended Page Tab= le access-dirty bit */ #define X86_FEATURE_VMCALL ( 8*32+18) /* Hypervisor supports the VMCALL i= nstruction */ #define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* VMware prefers VMMCALL hype= rcall instruction */ -#define X86_FEATURE_PVUNLOCK ( 8*32+20) /* PV unlock function */ #define X86_FEATURE_VCPUPREEMPT ( 8*32+21) /* PV vcpu_is_preempted functi= on */ #define X86_FEATURE_TDX_GUEST ( 8*32+22) /* "tdx_guest" Intel Trust Domai= n Extensions Guest */ =20 --=20 2.53.0-Meta From nobody Thu Aug 13 09:15:08 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23A9F4266AA; Tue, 4 Aug 2026 07:16:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827778; cv=none; b=JgintB5okmb+8txISDrRVcZQLSS200IG2TIEU8MJTvn+fWv4VbNYVWnjvys3d4JvoFsgzgTzdnUP0CDMe5Vs3Ll2xSQcJC28Dmvgk2pxFVyLhtTtJwiFPE66tNhu0Fj1PkcUuEC+rM2gjE4xOW3n9dpTp+S65tFAnZVphA/cgZ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827778; c=relaxed/simple; bh=DfoCtgOrQUc0eNb0GLPqnOjGGw/w2KYMm3OrmdISGyI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RC4GLJ2N1Qa+a0RJ3IvzZ+ULtAJl73x3FWzClfISHRRLfiZHp6+VU8Ya0/rtnJJWFyaubC+1GkcEcu/+iGyTTGBQGmSFvhEZv0I4Vi/Orhizmra1mXcoSpmhhNKwawjDojO9b7G7ZWZHnacG1VlnRk6tFKNBTYb+Ko+Cz2QT9rg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=wuKIY1D4; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="wuKIY1D4" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1785827769; bh=WuLqet5KqDLEqzcC11hIv1M/iVcVPFYVpjgSQGthfWs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=wuKIY1D4nBbR5uq8gfZNxdmHgxtiW+Zwfk36Z3rrQRTooGhRRML5EXyzobMw0uJxv 5P/K81r2u82h7iiaVnYm4EpGO+40bXq+eTassRL9b5566xMP0Fz5dav+H08KKCBg4i JYRM7O6cFcYUV5X3SXyn9SghO75pg22YrPEy33FY= Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 24FC1E16D4; Tue, 04 Aug 2026 07:16:09 +0000 (UTC) From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , "K. Y. Srinivasan" , Haiyang Zhang , Wei Liu , Dexuan Cui , Long Li , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Paolo Bonzini , Vitaly Kuznetsov , Josh Poimboeuf , Jason Baron , Alice Ryhl , Steven Rostedt , Ard Biesheuvel , Boris Ostrovsky , Arnd Bergmann , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-hyperv@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-arch@vger.kernel.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin Subject: [PATCH 2/5] locking: Factor out queued_spin_release() Date: Tue, 4 Aug 2026 07:15:42 +0000 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The contended_release tracepoint needs to hook queued_spin_unlock(), but architectures with a custom unlock define queued_spin_unlock() directly, leaving no single generic place to add the tracing. Introduce queued_spin_release() as the arch-overridable release primitive and make queued_spin_unlock() a generic wrapper around it. An architecture that only customizes the release can then override queued_spin_release() and inherit the generic wrapper. Rename the MIPS override to queued_spin_release() accordingly. x86 paravirt overrides queued_spin_unlock() directly and is left unchanged. No functional change intended. Signed-off-by: Dmitry Ilvokhin Acked-by: Juergen Gross --- arch/mips/include/asm/spinlock.h | 6 +++--- include/asm-generic/qspinlock.h | 17 ++++++++++++++--- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/arch/mips/include/asm/spinlock.h b/arch/mips/include/asm/spinl= ock.h index 6ce2117e49f6..c349162f15eb 100644 --- a/arch/mips/include/asm/spinlock.h +++ b/arch/mips/include/asm/spinlock.h @@ -13,12 +13,12 @@ =20 #include =20 -#define queued_spin_unlock queued_spin_unlock +#define queued_spin_release queued_spin_release /** - * queued_spin_unlock - release a queued spinlock + * queued_spin_release - release a queued spinlock * @lock : Pointer to queued spinlock structure */ -static inline void queued_spin_unlock(struct qspinlock *lock) +static inline void queued_spin_release(struct qspinlock *lock) { /* This could be optimised with ARCH_HAS_MMIOWB */ mmiowb(); diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index bf47cca2c375..ae45289e8ec7 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -115,12 +115,12 @@ static __always_inline void queued_spin_lock(struct q= spinlock *lock) } #endif =20 -#ifndef queued_spin_unlock +#ifndef queued_spin_release /** - * queued_spin_unlock - release a queued spinlock + * queued_spin_release - release a queued spinlock * @lock : Pointer to queued spinlock structure */ -static __always_inline void queued_spin_unlock(struct qspinlock *lock) +static __always_inline void queued_spin_release(struct qspinlock *lock) { /* * unlock() needs release semantics: @@ -129,6 +129,17 @@ static __always_inline void queued_spin_unlock(struct = qspinlock *lock) } #endif =20 +#ifndef queued_spin_unlock +/** + * queued_spin_unlock - unlock a queued spinlock + * @lock : Pointer to queued spinlock structure + */ +static __always_inline void queued_spin_unlock(struct qspinlock *lock) +{ + queued_spin_release(lock); +} +#endif + #ifndef virt_spin_lock static __always_inline bool virt_spin_lock(struct qspinlock *lock) { --=20 2.53.0-Meta From nobody Thu Aug 13 09:15:08 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B6A1426D19; Tue, 4 Aug 2026 07:16:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827780; cv=none; b=MYdZVVKH1EWWC/H/oetGLlNurVzQW1x45NP3wAtCcwNUPiQ5fntqOphLCFiqBnusGPJclJxzCbbYnblluC3aLky8bKPKmew3Mke0w+6w7Er8OOsKNmGzF7wbeDC8iNt7vHYlCNrNDgvacA+9b1N0URdHzyghcKFtxvwzH3z8big= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827780; c=relaxed/simple; bh=kPxo41qOkG/sFD78eLz3Pjf+SGUj+xLlnvMzVJD2ctM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PW4gp5P6LTd0AEei18FIwesyIL5b01MQKxQt0VGjzsQN5UOXjNZix+2x0IwO+lfxhi+QQycK3VJk/tJUzvEqxXKHbs9aX7uKU55pO8m99lIrG6hC+kOO4Nmc76L0xuyAx+Nq5sOjIQOVWRm6DwP5TZhR3iHd6AQZ0oJOHhh1GTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=0OMfUdlq; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="0OMfUdlq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1785827769; bh=fHFQBsOvzK2/mUZmCSCVwo8JOcS+aODOMFtgBaMfo18=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=0OMfUdlqT7tXp+xgFH8Lqhq8QKQ7xel10QLPov1Fj6BEEyus3KjXzGsZvJvOoCmFc HoubPtrG0hOfQ94rwZ6ss/UcnbQNjRe7uPtfYREtEbeVVgOyI7BCepcQi8X/rHThP1 7w23gEsazcnNYc12vfymcaG7+J/NylkdvJYVuzVs= Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 8F84BE16D9; Tue, 04 Aug 2026 07:16:09 +0000 (UTC) From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , "K. Y. Srinivasan" , Haiyang Zhang , Wei Liu , Dexuan Cui , Long Li , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Paolo Bonzini , Vitaly Kuznetsov , Josh Poimboeuf , Jason Baron , Alice Ryhl , Steven Rostedt , Ard Biesheuvel , Boris Ostrovsky , Arnd Bergmann , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-hyperv@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-arch@vger.kernel.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin Subject: [PATCH 3/5] locking/qspinlock: Add contended_release tracepoint Date: Tue, 4 Aug 2026 07:15:43 +0000 Message-ID: <0d998e22a0c595f670cfc6725bb683323aced5cb.1785778551.git.d@ilvokhin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf lock contention --lock-owner" cannot attribute a contended spinlock to its holder. The waiter-side contention_begin event records that a spinlock is contended, but not by whom. Firing contended_release in the holder's context at unlock is the only way to capture the holder of a contended spinlock. Combine the contention check, trace call and release in an out-of-line queued_spin_release_traced() so the compiler need not preserve the lock pointer in a callee-saved register across the call. The check in queued_spin_unlock() is paid on every unlock, even while the tracepoint is disabled: a static-branch NOP on x86_64, and a few more instructions to manage a stack frame elsewhere. Gate it behind CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (default n) so nobody pays for a tracepoint they do not use. Sleeping locks fire contended_release regardless. On x86 this generic path is used only with PARAVIRT_SPINLOCKS=3Dn (e.g. defconfig). PARAVIRT_SPINLOCKS=3Dy kernels keep the paravirt static_call unlock and are wired up separately. All below are with the QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE option enabled. _raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but disabled. The unlock is the single 'movb'. The only instruction added to the executed path is the 2-byte static-branch NOP. The CALL to the traced helper and the JMP back are emitted out of line and are reached only once the static branch is patched on: endbr64 ; 4 bytes xchg %ax,%ax ; 2 static-branch NOP ; (added) movb $0x0,(%rdi) ; 3 unlock (single store) A: decl %gs:__preempt_count ; 7 je B ; 2 jmp __x86_return_thunk ; 5 call queued_spin_release_traced ; 5 out of line, reached ; only when the ; tracepoint is on jmp A ; 2 (added) B: call __SCT__preempt_schedule ; 5 jmp __x86_return_thunk ; 5 Baseline is the same stream without the NOP and the out-of-line CALL/JMP: 31 bytes vs 40 (+9 bytes). Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all standard configs out-of-line unlock. Architectures with inlined unlock (s390 (always), csky and loongarch (both when !PREEMPTION)) will see a bigger increase in binary size. On the same path (x86_64, PARAVIRT_SPINLOCKS=3Dn) with the tracepoint disabled, a _raw_spin_unlock()-heavy nginx workload [1] shows no measurable difference between baseline and patched kernels in throughput, latency, cycles, instructions, IPC, or L1 instruction-cache misses (kernel and total): all deltas stay within run-to-run noise. Unlike x86, on arm64 the frame setup code (STP, MOV and LDP) lands on the executed path in addition to static-branch NOP. Binary size impact on arm64, defconfig: +932 bytes (+0.00%). The _raw_spin_unlock()-heavy nginx workload reflects the larger hot path: L1 instruction-cache misses rise ~1.4% (kernel and total) and instruction count ~0.4%, consistent with the per-unlock frame. cpu_cycles, throughput and latency show no measurable change and are within run-to-run noise. Architectures with fully custom qspinlock implementations (e.g. PowerPC) are not covered by this change. [1]: https://lore.kernel.org/all/aiphFXe_TPNPxZ_n@shell.ilvokhin.com/ Signed-off-by: Dmitry Ilvokhin Acked-by: Juergen Gross --- include/asm-generic/qspinlock.h | 21 +++++++++++++++++++++ kernel/Kconfig.locks | 20 ++++++++++++++++++++ kernel/locking/qspinlock.c | 22 ++++++++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index ae45289e8ec7..2ca94e41823b 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -41,6 +41,7 @@ =20 #include #include +#include =20 #ifndef queued_spin_is_locked /** @@ -130,12 +131,32 @@ static __always_inline void queued_spin_release(struc= t qspinlock *lock) #endif =20 #ifndef queued_spin_unlock + +DECLARE_TRACEPOINT(contended_release); + +extern void queued_spin_release_traced(struct qspinlock *lock); + /** * queued_spin_unlock - unlock a queued spinlock * @lock : Pointer to queued spinlock structure + * + * Generic tracing wrapper around the arch-overridable + * queued_spin_release(). */ static __always_inline void queued_spin_unlock(struct qspinlock *lock) { + /* + * Trace and release are combined in queued_spin_release_traced() so + * the compiler does not need to preserve the lock pointer across the + * function call, avoiding callee-saved register save/restore on the + * hot path. queued_spin_release() is therefore called both here and in + * queued_spin_release_traced(). Keep the two in sync. + */ + if (IS_ENABLED(CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE) && + tracepoint_enabled(contended_release)) { + queued_spin_release_traced(lock); + return; + } queued_spin_release(lock); } #endif diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index 4198f0273ecd..1c6423aafcd4 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -243,6 +243,26 @@ config QUEUED_SPINLOCKS def_bool y if ARCH_USE_QUEUED_SPINLOCKS depends on SMP =20 +config QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE + bool "Trace contended_release on queued spinlocks" + depends on QUEUED_SPINLOCKS && TRACEPOINTS + help + Fire the lock:contended_release tracepoint when a contended queued + spinlock is released, so it is possible to attribute a contended + spinlock to its holder. + + Architectures that can patch the unlock site do this at no cost and + do not need this option. + + Everywhere else the check is compiled into queued_spin_unlock() and + a small cost is paid on every unlock even when the tracepoint is + disabled: a static-branch NOP and possibly a few more instructions + to manage a stack frame. + + Sleeping locks fire lock:contended_release regardless of this option. + + If unsure, say N. + config BPF_ARCH_SPINLOCK bool =20 diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index af8d122bb649..33fe6d437c8f 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -104,6 +104,28 @@ static __always_inline u32 __pv_wait_head_or_lock(str= uct qspinlock *lock, #define queued_spin_lock_slowpath native_queued_spin_lock_slowpath #endif =20 +#if !defined(queued_spin_unlock) && \ + IS_ENABLED(CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE) +/* + * Out-of-line trace-and-release path for queued_spin_unlock(), used when + * the contended_release tracepoint is enabled. + * + * queued_spin_release() is duplicated here on purpose: doing the release + * in this function (rather than tracing here and releasing in the caller) + * lets queued_spin_unlock() return right after the call, so the + * tracepoint-disabled hot path never has to keep lock live across a call + * in a callee-saved register. Keep this release in sync with the one in + * queued_spin_unlock(). + */ +void __lockfunc queued_spin_release_traced(struct qspinlock *lock) +{ + if (queued_spin_is_contended(lock)) + trace_call__contended_release(lock); + queued_spin_release(lock); +} +EXPORT_SYMBOL(queued_spin_release_traced); +#endif + #endif /* _GEN_PV_LOCK_SLOWPATH */ =20 /** --=20 2.53.0-Meta From nobody Thu Aug 13 09:15:08 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B5D9426D16; Tue, 4 Aug 2026 07:16:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827779; cv=none; b=iHaPMvfkWSvDlR1G5CoSmaRAdRo7PK3mRao+pnd/nIZW4g3cHD93eyPJ2xEELFbh3mHEn78cAgVIlRUdmSOIXfZpXbILtjtP0wU+Z0Yb/ZtUxBGA9h8XzbMDlcSciXECDe+ee1Mb6b+c/zLqx7Fl7nr4jIJvwiA4Yf4c3NqfQK8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827779; c=relaxed/simple; bh=bfWfrFj1fW9kMK2LtEt9QZqHXwJoM32GS8ORP5hs3tY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C4kX6/QDUp3gNITVj2F4vSBR+G/chSUxm+JQwABVeR2JGxl+YbzXeLopH5APsyWAtWAQmb7Ek/tA5lmouC7QNTZUJPL38XfI8M6mRT2pj9YeEsqdAcL95/tH2yoWNlfrFwnw+xjaW8fodKzbojPLadVedjzSfEl3Bl4E8qO7HfE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=pap3mJ3/; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="pap3mJ3/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1785827770; bh=Bz2CpdeYvHJ4wdtpHF06PqmuJczwotX2rk+VZpOkhHc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=pap3mJ3/HxJEJFDk1E9V+MfQ4ojuK31y3RhNzciiECTAVYuygUg+AAUpw8Vg2iPD9 bo0bIm+yzmE6QIWLIch8VLiCc2NJUocvXXQ2cbg2o8M207KKA1P5DizNPLx/31xYgC RZ1Z6my1XJzOWuENeYHFx8DmI08HwyPl+3SWt/mM= Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 05AB4E16DC; Tue, 04 Aug 2026 07:16:10 +0000 (UTC) From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , "K. Y. Srinivasan" , Haiyang Zhang , Wei Liu , Dexuan Cui , Long Li , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Paolo Bonzini , Vitaly Kuznetsov , Josh Poimboeuf , Jason Baron , Alice Ryhl , Steven Rostedt , Ard Biesheuvel , Boris Ostrovsky , Arnd Bergmann , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-hyperv@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-arch@vger.kernel.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin Subject: [PATCH 4/5] tracing/lock: Use TRACE_EVENT_FN() for contended_release Date: Tue, 4 Aug 2026 07:15:44 +0000 Message-ID: <1c2fcccfb584c075c02890c484f22c76a1948bf1.1785778551.git.d@ilvokhin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" queued_spin_unlock() gates its contended_release trace call behind a static branch, so a NOP sits on the unlock path even while the tracepoint is disabled. Removing that requires replacing the unlock implementation only while contended_release is enabled, which needs a callback when the tracepoint is toggled. Convert contended_release to TRACE_EVENT_FN() and add weak no-op arch_contended_release_trace_reg()/arch_contended_release_trace_unreg() hooks. The default hooks are empty, so this is a no-op until an architecture overrides them. No functional change intended. Signed-off-by: Dmitry Ilvokhin Acked-by: Juergen Gross --- include/trace/events/lock.h | 10 ++++++++-- kernel/locking/mutex.c | 4 ++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/trace/events/lock.h b/include/trace/events/lock.h index 1ded869cd619..b1d5b18c4514 100644 --- a/include/trace/events/lock.h +++ b/include/trace/events/lock.h @@ -137,7 +137,11 @@ TRACE_EVENT(contention_end, TP_printk("%p (ret=3D%d)", __entry->lock_addr, __entry->ret) ); =20 -TRACE_EVENT(contended_release, +/* kernel/locking/mutex.c */ +int arch_contended_release_trace_reg(void); +void arch_contended_release_trace_unreg(void); + +TRACE_EVENT_FN(contended_release, =20 TP_PROTO(void *lock), =20 @@ -151,7 +155,9 @@ TRACE_EVENT(contended_release, __entry->lock_addr =3D lock; ), =20 - TP_printk("%p", __entry->lock_addr) + TP_printk("%p", __entry->lock_addr), + + arch_contended_release_trace_reg, arch_contended_release_trace_unreg ); =20 #endif /* _TRACE_LOCK_H */ diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 8a85912d7ee6..942a939cee95 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -1272,6 +1272,10 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(contention_begin); EXPORT_TRACEPOINT_SYMBOL_GPL(contention_end); EXPORT_TRACEPOINT_SYMBOL_GPL(contended_release); =20 +__weak int arch_contended_release_trace_reg(void) { return 0; } + +__weak void arch_contended_release_trace_unreg(void) { } + /** * atomic_dec_and_mutex_lock - return holding mutex if we dec to 0 * @cnt: the atomic which we are to dec --=20 2.53.0-Meta From nobody Thu Aug 13 09:15:08 2026 Received: from mail.ilvokhin.com (mail.ilvokhin.com [178.62.254.231]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10B1E42AFB6; Tue, 4 Aug 2026 07:16:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=178.62.254.231 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827781; cv=none; b=pHic0dWImEerwFFImvggAIFH/NfDvvW6fUwT9hNQ6nZ5ZiH4Kq6cIMf1NdBYRk5tVOd/7esuD1Q6ut392e1GKYSZn1mB6VzUdDOxNdBy1wlB7FOr/yWuc2C8CmJDMy/paSj7dfI0opTZcHIvn44VB5iNBN7NrPC8VGAqwjTqH1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785827781; c=relaxed/simple; bh=JESu/AclIL3+BmuW3VoymqC5DgpEXPwYsbzIHy7VIuI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n1Z0xmP17e4oShp+UkHpthf6QeTrRgbwe4rY5Om2hOZpSVDiGX/nmF7CcSmGeNLI9rZ/g/Z6t1KEdK9igmTJFS/W4N0dgGRV4+f6uYdLRJjMY8CiMBYq+2IczIDcQkpCx6VBj3rrL980QzddPA2VxDPe6+1aVJ5Kivbzt42NFXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com; spf=pass smtp.mailfrom=ilvokhin.com; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b=YCa2mhDC; arc=none smtp.client-ip=178.62.254.231 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ilvokhin.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ilvokhin.com header.i=@ilvokhin.com header.b="YCa2mhDC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ilvokhin.com; s=mail; t=1785827770; bh=MJXFdixHataYYt87xEtn0QdfMJOi3nH/EXHHrFBbQBU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=YCa2mhDCHjFFX0O6d3S1tYEYBR/Jb4ai4lQlH0qCvYfntcHb1/+MEaIyrBX3QT0u1 jViT9cjtE41ag6sP28YwsfmBHSjMYS3PwUlLcqd8dqfNVR1ReEl98E/Q9nIoYF3BTz G+fzkT2LOVauhNUbkR9kOT4y87KoehBEuITFVAbI= Received: from localhost.localdomain (shell.ilvokhin.com [138.68.190.75]) (Authenticated sender: d@ilvokhin.com) by mail.ilvokhin.com (Postfix) with ESMTPSA id 77194E16E0; Tue, 04 Aug 2026 07:16:10 +0000 (UTC) From: Dmitry Ilvokhin To: Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , Waiman Long , Thomas Bogendoerfer , "K. Y. Srinivasan" , Haiyang Zhang , Wei Liu , Dexuan Cui , Long Li , Thomas Gleixner , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Juergen Gross , Ajay Kaher , Alexey Makhalov , Broadcom internal kernel review list , Paolo Bonzini , Vitaly Kuznetsov , Josh Poimboeuf , Jason Baron , Alice Ryhl , Steven Rostedt , Ard Biesheuvel , Boris Ostrovsky , Arnd Bergmann , Masami Hiramatsu , Mathieu Desnoyers Cc: linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-hyperv@vger.kernel.org, virtualization@lists.linux.dev, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-arch@vger.kernel.org, linux-trace-kernel@vger.kernel.org, kernel-team@meta.com, Dmitry Ilvokhin Subject: [PATCH 5/5] x86/paravirt: Trace contended_release on unlock Date: Tue, 4 Aug 2026 07:15:45 +0000 Message-ID: <17fa67f9fa4cf93f1150725e89f5f916e41a9b6f.1785778551.git.d@ilvokhin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On PARAVIRT_SPINLOCKS=3Dy kernels queued_spin_unlock() is dispatched through a static_call(). Those PARAVIRT_SPINLOCKS=3Dy kernels are quite popular. Gating contended_release behind a static branch would leave a NOP on the unlock hot path even, when the tracepoint is disabled. Since the static_call() is already present, swap its target to a traced unlock, when the tracepoint is enabled instead. When contended_release tracepoint is disabled the target is the plain unlock (an inline store on native x86_64), so the unlock path is unchanged and the tracepoint is truly zero-cost. Provide two traced variants, native_queued_spin_unlock_traced() and pv_queued_spin_unlock_traced(), so each tail-calls its own base unlock directly rather than recursing through the now-traced static_call(). Teach pv_is_native_spin_unlock() that the traced native variant still counts as native. Only PARAVIRT_SPINLOCKS=3Dy is affected. PARAVIRT_SPINLOCKS=3Dn keeps the generic static-branch path. Suggested-by: Peter Zijlstra Signed-off-by: Dmitry Ilvokhin Acked-by: Juergen Gross --- arch/x86/include/asm/paravirt-spinlock.h | 2 + arch/x86/kernel/paravirt-spinlocks.c | 53 +++++++++++++++++++++++- 2 files changed, 53 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/paravirt-spinlock.h b/arch/x86/include/as= m/paravirt-spinlock.h index ff735830de4a..302bc2ba3a75 100644 --- a/arch/x86/include/asm/paravirt-spinlock.h +++ b/arch/x86/include/asm/paravirt-spinlock.h @@ -99,6 +99,8 @@ bool __raw_callee_save___native_vcpu_is_preempted(long cp= u); =20 void __init native_pv_lock_init(void); __visible void __native_queued_spin_unlock(struct qspinlock *lock); +__visible void native_queued_spin_unlock_traced(struct qspinlock *lock); +__visible void pv_queued_spin_unlock_traced(struct qspinlock *lock); bool pv_is_native_spin_unlock(void); __visible bool __native_vcpu_is_preempted(long cpu); bool pv_is_native_vcpu_is_preempted(void); diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravir= t-spinlocks.c index ddc19dc28ba1..ca12b3655307 100644 --- a/arch/x86/kernel/paravirt-spinlocks.c +++ b/arch/x86/kernel/paravirt-spinlocks.c @@ -7,6 +7,7 @@ #include #include #include +#include =20 DEFINE_STATIC_KEY_FALSE(virt_spin_lock_key); =20 @@ -30,10 +31,58 @@ EXPORT_STATIC_CALL_TRAMP(queued_spin_lock_slowpath); DEFINE_STATIC_CALL(queued_spin_unlock, __raw_callee_save___native_queued_s= pin_unlock); EXPORT_STATIC_CALL_TRAMP(queued_spin_unlock); =20 +/* + * Traced unlock variants, swapped in via static_call while the + * contended_release tracepoint is enabled. Two of them, so each tail call= s its + * own base directly. + */ +__visible void native_queued_spin_unlock_traced(struct qspinlock *lock) +{ + if (queued_spin_is_contended(lock)) + trace_call__contended_release(lock); + native_queued_spin_unlock(lock); +} +PV_CALLEE_SAVE_REGS_THUNK(native_queued_spin_unlock_traced); + +__visible void pv_queued_spin_unlock_traced(struct qspinlock *lock) +{ + if (queued_spin_is_contended(lock)) + trace_call__contended_release(lock); + __raw_callee_save___pv_queued_spin_unlock(lock); +} +PV_CALLEE_SAVE_REGS_THUNK(pv_queued_spin_unlock_traced); + bool pv_is_native_spin_unlock(void) { - return static_call_query(queued_spin_unlock) =3D=3D - __raw_callee_save___native_queued_spin_unlock; + void *unlock =3D static_call_query(queued_spin_unlock); + + return unlock =3D=3D __raw_callee_save___native_queued_spin_unlock || + unlock =3D=3D __raw_callee_save_native_queued_spin_unlock_traced; +} + +int arch_contended_release_trace_reg(void) +{ + void *cur =3D static_call_query(queued_spin_unlock); + + if (cur =3D=3D __raw_callee_save___native_queued_spin_unlock) + static_call_update(queued_spin_unlock, + __raw_callee_save_native_queued_spin_unlock_traced); + else if (cur =3D=3D __raw_callee_save___pv_queued_spin_unlock) + static_call_update(queued_spin_unlock, + __raw_callee_save_pv_queued_spin_unlock_traced); + return 0; +} + +void arch_contended_release_trace_unreg(void) +{ + void *cur =3D static_call_query(queued_spin_unlock); + + if (cur =3D=3D __raw_callee_save_native_queued_spin_unlock_traced) + static_call_update(queued_spin_unlock, + __raw_callee_save___native_queued_spin_unlock); + else if (cur =3D=3D __raw_callee_save_pv_queued_spin_unlock_traced) + static_call_update(queued_spin_unlock, + __raw_callee_save___pv_queued_spin_unlock); } =20 __visible bool __native_vcpu_is_preempted(long cpu) --=20 2.53.0-Meta