From nobody Tue Apr 7 23:40:11 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0228B40DFD5 for ; Wed, 11 Mar 2026 13:02:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773234151; cv=none; b=VAngj/Nut9Ta30VmPInIyvv7pZfGy7pt7i6vyfn/fcroZzmvnp0fhAUlHbEIB2UwJ3neXNX9978fFFB8C8LHWU1gOymY2ieR1EaiAEDoP422UMcYbxWP2jaKZDUnNHdOMTRje49dM1bQd8zLavoN5notk/p44qku7d5/hKkfE1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773234151; c=relaxed/simple; bh=alxhS5zNt5yhZDlpWUot0Dy3jmoiPvwWrfIvY63/tcE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SUd6oAsTXSRG3ooKevUBFb7x6EnE8zHwEftC79P02xZHqUEWqCAOikixBvvwcu6y22nqadJ2/yDGbWHcOQlyUZYDiGByIOvjA16avnRCI7QyKF+CHpujdfU8EiM/jP9fD95sK6otGU2Nj4P8434HVauRGkO4KlCDtLApOmD+muc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Tt2zLps+; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Tt2zLps+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1773234149; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wDwbdAOWQrkyyH+jPzV483o27gJ9IQVrrz9O2kAMCZY=; b=Tt2zLps+R9ZZhFEfQhnnq22hI7gsMqHD617VThf0HUtrg7/dT3zjA8x2CfPHkkr7rt2ii/ 9kLfVrllwOPmXsq0KLLoMcrBpUaSIT4LJEOkhtOTy1mdYVn6zAn/5nEVreATdba+FZH2Dc ZXn0sd88LpHz5cn+GqTWiiA8rLKfsWc= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-663-pe716fNZPJqrBdtiFVtqJQ-1; Wed, 11 Mar 2026 09:02:23 -0400 X-MC-Unique: pe716fNZPJqrBdtiFVtqJQ-1 X-Mimecast-MFC-AGG-ID: pe716fNZPJqrBdtiFVtqJQ_1773234140 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F26DA19560B5; Wed, 11 Mar 2026 13:02:18 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.80.206]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 63EFE3000223; Wed, 11 Mar 2026 13:02:12 +0000 (UTC) From: Wander Lairson Costa To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Masami Hiramatsu , Mathieu Desnoyers , Andrew Morton , linux-kernel@vger.kernel.org (open list:SCHEDULER), linux-trace-kernel@vger.kernel.org (open list:TRACING) Cc: acme@kernel.org, williams@redhat.com, gmonaco@redhat.com, Wander Lairson Costa Subject: [PATCH v3 1/4] tracing/preemptirq: Optimize preempt_disable/enable() tracepoint overhead Date: Wed, 11 Mar 2026 09:50:15 -0300 Message-ID: <20260311125021.197638-2-wander@redhat.com> In-Reply-To: <20260311125021.197638-1-wander@redhat.com> References: <20260311125021.197638-1-wander@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Content-Type: text/plain; charset="utf-8" When CONFIG_TRACE_PREEMPT_TOGGLE is enabled, preempt_count_add() and preempt_count_sub() become external function calls (defined in kernel/sched/core.c) rather than inlined operations. These functions also perform preempt_count() checks and call trace_preempt_on/off() unconditionally, even when no tracing consumer is active. Reduce this overhead by splitting the #if logic in preempt.h into three cases. When CONFIG_DEBUG_PREEMPT or CONFIG_PREEMPT_TRACER is set, keep external function calls because DEBUG_PREEMPT needs runtime validation checks, and PREEMPT_TRACER needs the preemptoff latency tracer hooks (tracer_preempt_on/off, called via trace_preempt_on/off). When CONFIG_TRACE_PREEMPT_TOGGLE alone is set, provide new inline versions of preempt_count_add/sub() that check the tracepoint static key via the __preempt_trace_enabled() macro before calling into the tracing path. The macro evaluates to true when the preempt_enable or preempt_disable tracepoint has subscribers AND the preempt count equals val (indicating the first preempt disable or last preempt enable), preserving the original preempt_latency_start/stop semantics. When none of the above are set, use pure inline macros with no tracing overhead. The preempt_count_dec_and_test() macro is refactored out of the three-way #if into a separate block shared by the first two cases, since both need it to call the (potentially inline) preempt_count_sub() before checking should_resched(). The inline path calls thin __trace_preempt_on/off() wrappers (added in trace_preemptirq.c) that invoke trace_preempt_on/off(), keeping the full tracepoint machinery out of the inline code. The #include is placed inside the CONFIG_TRACE_PREEMPT_TOGGLE block rather than at the top of the file to avoid a circular include dependency on architectures where asm/irqflags.h includes linux/preempt.h (e.g. m68k): preempt.h -> tracepoint-defs.h -> static_key.h -> jump_label.h -> atomic.h -> irqflags.h -> asm/irqflags.h -> preempt.h (guarded) If the include were at the top, this chain would be traversed before hardirq_count() is defined (at line 110), causing a build failure on m68k. Placing it inside the #elif block ensures it is only pulled in when CONFIG_TRACE_PREEMPT_TOGGLE is enabled and avoids the cycle for configurations that do not select it. In core.c, narrow the compilation guard for the external preempt_count_add/sub() from CONFIG_DEBUG_PREEMPT || CONFIG_TRACE_PREEMPT_TOGGLE to CONFIG_DEBUG_PREEMPT || CONFIG_PREEMPT_TRACER, since CONFIG_TRACE_PREEMPT_TOGGLE is now handled inline. Signed-off-by: Wander Lairson Costa Suggested-by: Steven Rostedt --- include/linux/preempt.h | 49 +++++++++++++++++++++++++++++++-- kernel/sched/core.c | 2 +- kernel/trace/trace_preemptirq.c | 19 +++++++++++++ 3 files changed, 66 insertions(+), 4 deletions(-) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index d964f965c8ffc..f59a92f930d81 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -189,17 +189,60 @@ static __always_inline unsigned char interrupt_contex= t_level(void) */ #define in_atomic_preempt_off() (preempt_count() !=3D PREEMPT_DISABLE_OFFS= ET) =20 -#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE) +#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER) extern void preempt_count_add(int val); extern void preempt_count_sub(int val); -#define preempt_count_dec_and_test() \ - ({ preempt_count_sub(1); should_resched(0); }) +#elif defined(CONFIG_TRACE_PREEMPT_TOGGLE) +/* + * Avoid the circular dependency on architectures where asm/irqflags.h + * includes linux/preempt.h (e.g. m68k): + * + * preempt.h <--------------------+ + * tracepoint-defs.h | + * static_key.h | + * jump_label.h | + * atomic.h | + * irqflags.h | + * asm/irqflags.h | + * preempt.h --------------+ + */ +#include + +extern void __trace_preempt_on(void); +extern void __trace_preempt_off(void); + +DECLARE_TRACEPOINT(preempt_enable); +DECLARE_TRACEPOINT(preempt_disable); + +#define __preempt_trace_enabled(type, val) \ + (tracepoint_enabled(preempt_##type) && preempt_count() =3D=3D (val)) + +static __always_inline void preempt_count_add(int val) +{ + __preempt_count_add(val); + + if (__preempt_trace_enabled(disable, val)) + __trace_preempt_off(); +} + +static __always_inline void preempt_count_sub(int val) +{ + if (__preempt_trace_enabled(enable, val)) + __trace_preempt_on(); + + __preempt_count_sub(val); +} #else #define preempt_count_add(val) __preempt_count_add(val) #define preempt_count_sub(val) __preempt_count_sub(val) #define preempt_count_dec_and_test() __preempt_count_dec_and_test() #endif =20 +#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE) +#define preempt_count_dec_and_test() \ + ({ preempt_count_sub(1); should_resched(0); }) +#endif + #define __preempt_count_inc() __preempt_count_add(1) #define __preempt_count_dec() __preempt_count_sub(1) =20 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b7f77c165a6e0..125e5d71d1bd3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5733,7 +5733,7 @@ static inline void sched_tick_stop(int cpu) { } #endif /* !CONFIG_NO_HZ_FULL */ =20 #if defined(CONFIG_PREEMPTION) && (defined(CONFIG_DEBUG_PREEMPT) || \ - defined(CONFIG_TRACE_PREEMPT_TOGGLE)) + defined(CONFIG_PREEMPT_TRACER)) /* * If the value passed in is equal to the current preempt count * then we just disabled preemption. Start timing the latency. diff --git a/kernel/trace/trace_preemptirq.c b/kernel/trace/trace_preemptir= q.c index 0c42b15c38004..9f098fcb28012 100644 --- a/kernel/trace/trace_preemptirq.c +++ b/kernel/trace/trace_preemptirq.c @@ -115,6 +115,25 @@ NOKPROBE_SYMBOL(trace_hardirqs_off); =20 #ifdef CONFIG_TRACE_PREEMPT_TOGGLE =20 +#if !defined(CONFIG_DEBUG_PREEMPT) && !defined(CONFIG_PREEMPT_TRACER) +EXPORT_TRACEPOINT_SYMBOL(preempt_disable); +EXPORT_TRACEPOINT_SYMBOL(preempt_enable); + +void __trace_preempt_on(void) +{ + trace_preempt_on(CALLER_ADDR0, get_lock_parent_ip()); +} +EXPORT_SYMBOL(__trace_preempt_on); +NOKPROBE_SYMBOL(__trace_preempt_on); + +void __trace_preempt_off(void) +{ + trace_preempt_off(CALLER_ADDR0, get_lock_parent_ip()); +} +EXPORT_SYMBOL(__trace_preempt_off); +NOKPROBE_SYMBOL(__trace_preempt_off); +#endif /* !CONFIG_DEBUG_PREEMPT */ + void trace_preempt_on(unsigned long a0, unsigned long a1) { trace(preempt_enable, TP_ARGS(a0, a1)); --=20 2.53.0