From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF027C433EF for ; Thu, 19 May 2022 21:41:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245181AbiESVlZ (ORCPT ); Thu, 19 May 2022 17:41:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237913AbiESVlR (ORCPT ); Thu, 19 May 2022 17:41:17 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1305C40A26 for ; Thu, 19 May 2022 14:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=BHW2swRVhzZshQYFgj6YJOVsdD0o08beiukGyK4KHoU=; b=cWDbHddtLDV1hGcqPaCt0WKFp7 7EiSTXucfdzKVVScMNhw1DNzUXwp5Z/qD9AME6ALhtdtueEAdXTBCn8A+n0dvwAcUaElgey3zbjGR pS5z5rw+Kc3LBxWfPw3KZdYGme89vvJD+kyJfoNl5dRA4h6XHKAaZ8omH2/Vg0CezutPzv0cNS7xJ osXyCS73lIQ07ups2dLdA/IeilE8H9t5MN+JooJ8Sw/wRYLXXFxKinCrId46fLQEdOlCmcb2HFPNC srAxjFaNfsL5WUQqd4655g/j4fgZ0dcgP6mPA6SneieANzjW3VsXCA5IgR66mZ0Yvy1C5zEBZFnb/ GigN05tg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntN-00D7Pj-Ad; Thu, 19 May 2022 21:41:10 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 234A9300BE6; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 030882021B5C3; Thu, 19 May 2022 23:41:06 +0200 (CEST) Message-ID: <20220519213421.627364681@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:51 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org, eranian@google.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 1/9] x86/perf/amd: Remove tracing from perf_lopwr_cb() References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The perf_lopwr_cb() is called from the idle routines; there is no RCU there, we must not enter tracing. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/events/amd/brs.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) --- a/arch/x86/events/amd/brs.c +++ b/arch/x86/events/amd/brs.c @@ -41,18 +41,15 @@ static inline unsigned int brs_to(int id return MSR_AMD_SAMP_BR_FROM + 2 * idx + 1; } =20 -static inline void set_debug_extn_cfg(u64 val) +static __always_inline void set_debug_extn_cfg(u64 val) { /* bits[4:3] must always be set to 11b */ - wrmsrl(MSR_AMD_DBG_EXTN_CFG, val | 3ULL << 3); + __wrmsr(MSR_AMD_DBG_EXTN_CFG, val | 3ULL << 3, val >> 32); } =20 -static inline u64 get_debug_extn_cfg(void) +static __always_inline u64 get_debug_extn_cfg(void) { - u64 val; - - rdmsrl(MSR_AMD_DBG_EXTN_CFG, val); - return val; + return __rdmsr(MSR_AMD_DBG_EXTN_CFG); } =20 static bool __init amd_brs_detect(void) @@ -338,7 +335,7 @@ void amd_pmu_brs_sched_task(struct perf_ * called from ACPI processor_idle.c or acpi_pad.c * with interrupts disabled */ -void perf_amd_brs_lopwr_cb(bool lopwr_in) +void noinstr perf_amd_brs_lopwr_cb(bool lopwr_in) { struct cpu_hw_events *cpuc =3D this_cpu_ptr(&cpu_hw_events); union amd_debug_extn_cfg cfg; From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66784C433EF for ; Thu, 19 May 2022 21:41:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245182AbiESVln (ORCPT ); Thu, 19 May 2022 17:41:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244465AbiESVlW (ORCPT ); Thu, 19 May 2022 17:41:22 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0344E40A26 for ; Thu, 19 May 2022 14:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=66jVoWk+OA85s9ib6uVCg5Wh95Rp51ur8O6i8KwQNnE=; b=DJcr+aw1GwQzMhU+yRZ2IEAMuu 8XHLx8RqIkp+uFv5OgMru0jrpLgXEnzyXspCZSJYiYbv6jI3TDzn5Sivpr6pQKAGJOu89WV37HdM0 AtXVnemEUpp8ZyWaxgH4gaczPYahAdzW4Q2Bq6YNMHcTq0PP5hc0BzCQA5xcqjdzN41eVZqpy032Y IDlTRYi+E6UryFOD2dtxgue/dc03aN0TSDex/Br6vBhWYiI3yD/lKaPPKqMJPgiLRvm2Pqqp+jCEQ zFaHL+bv0Oc3URS/lSQ6fDy+JYXYRWQgJ3dThj0yJVrO2j7ODdITR+8pr7aQygnMgpVNOHKV79Uf5 52ikL+NA==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntN-0028dl-Cr; Thu, 19 May 2022 21:41:10 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 1CB8A3002BE; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 04F812021B5DF; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213421.687917982@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:52 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 2/9] x86/idle: Replace x86_idle with a static_call References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Typical boot time setup; no need to suffer an indirect call for that. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Frederic Weisbecker --- arch/x86/kernel/process.c | 50 +++++++++++++++++++++++++----------------= ----- 1 file changed, 28 insertions(+), 22 deletions(-) --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -692,7 +693,23 @@ void __switch_to_xtra(struct task_struct unsigned long boot_option_idle_override =3D IDLE_NO_OVERRIDE; EXPORT_SYMBOL(boot_option_idle_override); =20 -static void (*x86_idle)(void); +/* + * We use this if we don't have any better idle routine.. + */ +void __cpuidle default_idle(void) +{ + raw_safe_halt(); +} +#if defined(CONFIG_APM_MODULE) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE) +EXPORT_SYMBOL(default_idle); +#endif + +DEFINE_STATIC_CALL_NULL(x86_idle, default_idle); + +static bool x86_idle_set(void) +{ + return !!static_call_query(x86_idle); +} =20 #ifndef CONFIG_SMP static inline void play_dead(void) @@ -715,28 +732,17 @@ void arch_cpu_idle_dead(void) /* * Called from the generic idle code. */ -void arch_cpu_idle(void) -{ - x86_idle(); -} - -/* - * We use this if we don't have any better idle routine.. - */ -void __cpuidle default_idle(void) +void __cpuidle arch_cpu_idle(void) { - raw_safe_halt(); + static_call(x86_idle)(); } -#if defined(CONFIG_APM_MODULE) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE) -EXPORT_SYMBOL(default_idle); -#endif =20 #ifdef CONFIG_XEN bool xen_set_default_idle(void) { - bool ret =3D !!x86_idle; + bool ret =3D x86_idle_set(); =20 - x86_idle =3D default_idle; + static_call_update(x86_idle, default_idle); =20 return ret; } @@ -859,20 +865,20 @@ void select_idle_routine(const struct cp if (boot_option_idle_override =3D=3D IDLE_POLL && smp_num_siblings > 1) pr_warn_once("WARNING: polling idle and HT enabled, performance may degr= ade\n"); #endif - if (x86_idle || boot_option_idle_override =3D=3D IDLE_POLL) + if (x86_idle_set() || boot_option_idle_override =3D=3D IDLE_POLL) return; =20 if (boot_cpu_has_bug(X86_BUG_AMD_E400)) { pr_info("using AMD E400 aware idle routine\n"); - x86_idle =3D amd_e400_idle; + static_call_update(x86_idle, amd_e400_idle); } else if (prefer_mwait_c1_over_halt(c)) { pr_info("using mwait in idle threads\n"); - x86_idle =3D mwait_idle; + static_call_update(x86_idle, mwait_idle); } else if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { pr_info("using TDX aware idle routine\n"); - x86_idle =3D tdx_safe_halt; + static_call_update(x86_idle, tdx_safe_halt); } else - x86_idle =3D default_idle; + static_call_update(x86_idle, default_idle); } =20 void amd_e400_c1e_apic_setup(void) @@ -925,7 +931,7 @@ static int __init idle_setup(char *str) * To continue to load the CPU idle driver, don't touch * the boot_option_idle_override. */ - x86_idle =3D default_idle; + static_call_update(x86_idle, default_idle); boot_option_idle_override =3D IDLE_HALT; } else if (!strcmp(str, "nomwait")) { /* From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8FB0C433FE for ; Thu, 19 May 2022 21:42:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245278AbiESVmL (ORCPT ); Thu, 19 May 2022 17:42:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245179AbiESVlY (ORCPT ); Thu, 19 May 2022 17:41:24 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6A584EA0D for ; Thu, 19 May 2022 14:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=4Ryv8V/o6iO/NxByO0hQtge0QtXFF+VpHW1PBt63HGE=; b=WBvhQDsCtvUv3fASEBtX4EZBXk u2tcPobZYmRlY6xxZ2I+UCaV/sJheSv+kEXv1F/PR+tvmiMCfUwxOEKu/3BDRz8AQzhSK5iV0b8Wj lyp9minffd1H2Hp100McpbZSi8TdLhffjk3RjtBD2WTOYDUoP2Z/hzNpyZQscToNb1uSIq6FwhlA+ ukY0BqtCf8rKa36rePo8M8166lFg8g2YGaYScOpGVO8gZhbKxbONBC/DwieGnDxqFEPogiUZxf3Ok gRcNLjc7SUdihWYWzIF+EkBXd3NHE9fW6i7KxKcAqmU3Vw7PIJ1nT4lVmkdZUUzTVrgqDZZP6bJuM b6MHNE+g==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntN-0028dk-Ck; Thu, 19 May 2022 21:41:10 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 211F5300859; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 08D862021B1A0; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213421.748352112@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:53 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 3/9] cpuidle: Move IRQ state validation References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make cpuidle_enter_state() consistent with the s2idle variant and verify ->enter() always returns with interrupts disabled. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/cpuidle.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -234,7 +234,11 @@ int cpuidle_enter_state(struct cpuidle_d stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) rcu_idle_enter(); + entered_state =3D target_state->enter(dev, drv, index); + if (WARN_ON_ONCE(!irqs_disabled())) + raw_local_irq_disable(); + if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) rcu_idle_exit(); start_critical_timings(); @@ -246,12 +250,8 @@ int cpuidle_enter_state(struct cpuidle_d /* The cpu is no longer idle or about to enter idle. */ sched_idle_set_state(NULL); =20 - if (broadcast) { - if (WARN_ON_ONCE(!irqs_disabled())) - local_irq_disable(); - + if (broadcast) tick_broadcast_exit(); - } =20 if (!cpuidle_state_is_coupled(drv, index)) local_irq_enable(); From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EF8AC433EF for ; Thu, 19 May 2022 21:41:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245188AbiESVlp (ORCPT ); Thu, 19 May 2022 17:41:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245163AbiESVlX (ORCPT ); Thu, 19 May 2022 17:41:23 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0361447ADC for ; Thu, 19 May 2022 14:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=uJmPFNr5Fp4mKF9rK6zqxxWonCmQ8KmmZmAyYt1PvWA=; b=K3LmTo2dbOJmxFmxEz54YIWdhu RnFLBHAejqDMjZW2q/K+jFOFBYU1B+zQ7jZPVgXUYLcnHbHjXPBj3CWsBCrNPT+/Qmm4LXhc1SoC6 JAwXzu4bNANTMPVpryJ+Bbd49wfrQ1VzAvQdzvxTFuJfzZ0fvctntuf9EV0S5vQksZsqpsPuxlwGc nHWjCgosdBM3sb7vUXyf5mDyh3RVGTGTHpmZzapNrkAgf8tcakdcFy4ueTLcAQJk77BGg7TW72XwB vjf94jJAaa0u4rQrXLhmtbjYoUEUa2FZIMcZlqEBhlBVqx0OxLbXoeIEG9oeBZhjsXBTJFdt7PmQA t2ab9HPg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntN-0028dj-BW; Thu, 19 May 2022 21:41:10 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 28B3D300F06; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 0EB2E2021AF6B; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213421.808983977@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:54 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 4/9] idle: Fix rcu_idle_*() usage References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The whole disable-RCU, enable-IRQS dance is very intricate since changing IRQ state is traced, which depends on RCU. Add two helpers for the cpuidle case that mirror the entry code. Signed-off-by: Peter Zijlstra (Intel) --- arch/arm/mach-imx/cpuidle-imx6q.c | 5 ++-- drivers/acpi/processor_idle.c | 18 +++++++++------ drivers/cpuidle/cpuidle.c | 11 +++++---- include/linux/sched/idle.h | 29 ++++++++++++++++++++++++ kernel/sched/idle.c | 45 ++++++++++-----------------------= ----- kernel/time/tick-broadcast.c | 2 - 6 files changed, 62 insertions(+), 48 deletions(-) --- a/arch/arm/mach-imx/cpuidle-imx6q.c +++ b/arch/arm/mach-imx/cpuidle-imx6q.c @@ -5,6 +5,7 @@ =20 #include #include +#include #include =20 #include @@ -24,9 +25,9 @@ static int imx6q_enter_wait(struct cpuid imx6_set_lpm(WAIT_UNCLOCKED); raw_spin_unlock(&cpuidle_lock); =20 - rcu_idle_enter(); + cpuidle_rcu_enter(); cpu_do_idle(); - rcu_idle_exit(); + cpuidle_rcu_exit(); =20 raw_spin_lock(&cpuidle_lock); if (num_idle_cpus-- =3D=3D num_online_cpus()) --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -607,7 +607,7 @@ static DEFINE_RAW_SPINLOCK(c3_lock); * @cx: Target state context * @index: index of target state */ -static int acpi_idle_enter_bm(struct cpuidle_driver *drv, +static noinstr int acpi_idle_enter_bm(struct cpuidle_driver *drv, struct acpi_processor *pr, struct acpi_processor_cx *cx, int index) @@ -626,6 +626,8 @@ static int acpi_idle_enter_bm(struct cpu */ bool dis_bm =3D pr->flags.bm_control; =20 + instrumentation_begin(); + /* If we can skip BM, demote to a safe state. */ if (!cx->bm_sts_skip && acpi_idle_bm_check()) { dis_bm =3D false; @@ -647,11 +649,11 @@ static int acpi_idle_enter_bm(struct cpu raw_spin_unlock(&c3_lock); } =20 - rcu_idle_enter(); + cpuidle_rcu_enter(); =20 acpi_idle_do_entry(cx); =20 - rcu_idle_exit(); + cpuidle_rcu_exit(); =20 /* Re-enable bus master arbitration */ if (dis_bm) { @@ -661,11 +663,13 @@ static int acpi_idle_enter_bm(struct cpu raw_spin_unlock(&c3_lock); } =20 + instrumentation_end(); + return index; } =20 -static int acpi_idle_enter(struct cpuidle_device *dev, - struct cpuidle_driver *drv, int index) +static noinstr int acpi_idle_enter(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) { struct acpi_processor_cx *cx =3D per_cpu(acpi_cstate[index], dev->cpu); struct acpi_processor *pr; @@ -693,8 +697,8 @@ static int acpi_idle_enter(struct cpuidl return index; } =20 -static int acpi_idle_enter_s2idle(struct cpuidle_device *dev, - struct cpuidle_driver *drv, int index) +static noinstr int acpi_idle_enter_s2idle(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) { struct acpi_processor_cx *cx =3D per_cpu(acpi_cstate[index], dev->cpu); =20 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -150,12 +151,12 @@ static void enter_s2idle_proper(struct c */ stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_enter(); + cpuidle_rcu_enter(); target_state->enter_s2idle(dev, drv, index); if (WARN_ON_ONCE(!irqs_disabled())) - local_irq_disable(); + raw_local_irq_disable(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_exit(); + cpuidle_rcu_enter(); tick_unfreeze(); start_critical_timings(); =20 @@ -233,14 +234,14 @@ int cpuidle_enter_state(struct cpuidle_d =20 stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_enter(); + cpuidle_rcu_enter(); =20 entered_state =3D target_state->enter(dev, drv, index); if (WARN_ON_ONCE(!irqs_disabled())) raw_local_irq_disable(); =20 if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_exit(); + cpuidle_rcu_exit(); start_critical_timings(); =20 sched_clock_idle_wakeup_event(); --- a/include/linux/sched/idle.h +++ b/include/linux/sched/idle.h @@ -88,4 +88,33 @@ static inline void current_clr_polling(v preempt_fold_need_resched(); } =20 +static __always_inline void cpuidle_rcu_enter(void) +{ + lockdep_assert_irqs_disabled(); + /* + * Idle is allowed to (temporary) enable IRQs. It + * will return with IRQs disabled. + * + * Trace IRQs enable here, then switch off RCU, and have + * arch_cpu_idle() use raw_local_irq_enable(). Note that + * rcu_idle_enter() relies on lockdep IRQ state, so switch that + * last -- this is very similar to the entry code. + */ + trace_hardirqs_on_prepare(); + lockdep_hardirqs_on_prepare(); + instrumentation_end(); + rcu_idle_enter(); + lockdep_hardirqs_on(_THIS_IP_); +} + +static __always_inline void cpuidle_rcu_exit(void) +{ + /* + * Carefully undo the above. + */ + lockdep_hardirqs_off(_THIS_IP_); + rcu_idle_exit(); + instrumentation_begin(); +} + #endif /* _LINUX_SCHED_IDLE_H */ --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -51,18 +51,22 @@ __setup("hlt", cpu_idle_nopoll_setup); =20 static noinline int __cpuidle cpu_idle_poll(void) { + instrumentation_begin(); trace_cpu_idle(0, smp_processor_id()); stop_critical_timings(); - rcu_idle_enter(); - local_irq_enable(); + cpuidle_rcu_enter(); =20 + raw_local_irq_enable(); while (!tif_need_resched() && (cpu_idle_force_poll || tick_check_broadcast_expired())) cpu_relax(); + raw_local_irq_disable(); =20 - rcu_idle_exit(); + cpuidle_rcu_exit(); start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); + local_irq_enable(); + instrumentation_end(); =20 return 1; } @@ -85,44 +87,21 @@ void __weak arch_cpu_idle(void) */ void __cpuidle default_idle_call(void) { - if (current_clr_polling_and_test()) { - local_irq_enable(); - } else { - + instrumentation_begin(); + if (!current_clr_polling_and_test()) { trace_cpu_idle(1, smp_processor_id()); stop_critical_timings(); =20 - /* - * arch_cpu_idle() is supposed to enable IRQs, however - * we can't do that because of RCU and tracing. - * - * Trace IRQs enable here, then switch off RCU, and have - * arch_cpu_idle() use raw_local_irq_enable(). Note that - * rcu_idle_enter() relies on lockdep IRQ state, so switch that - * last -- this is very similar to the entry code. - */ - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(); - rcu_idle_enter(); - lockdep_hardirqs_on(_THIS_IP_); - + cpuidle_rcu_enter(); arch_cpu_idle(); - - /* - * OK, so IRQs are enabled here, but RCU needs them disabled to - * turn itself back on.. funny thing is that disabling IRQs - * will cause tracing, which needs RCU. Jump through hoops to - * make it 'work'. - */ raw_local_irq_disable(); - lockdep_hardirqs_off(_THIS_IP_); - rcu_idle_exit(); - lockdep_hardirqs_on(_THIS_IP_); - raw_local_irq_enable(); + cpuidle_rcu_exit(); =20 start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); } + local_irq_enable(); + instrumentation_end(); } =20 static int call_cpuidle_s2idle(struct cpuidle_driver *drv, --- a/kernel/time/tick-broadcast.c +++ b/kernel/time/tick-broadcast.c @@ -622,7 +622,7 @@ struct cpumask *tick_get_broadcast_onesh * to avoid a deep idle transition as we are about to get the * broadcast IPI right away. */ -int tick_check_broadcast_expired(void) +int noinstr tick_check_broadcast_expired(void) { return cpumask_test_cpu(smp_processor_id(), tick_broadcast_force_mask); } From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87478C433F5 for ; Thu, 19 May 2022 21:41:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245253AbiESVlu (ORCPT ); Thu, 19 May 2022 17:41:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245170AbiESVlX (ORCPT ); Thu, 19 May 2022 17:41:23 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 037B448336 for ; Thu, 19 May 2022 14:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=NtxpRvMipUusyaSMRuF1GQDqmhBvdAgxyA3nzuh6DWA=; b=c0ExzCD7vdgcK8CuVpLeNS58qh z1gw/+kXW09MsN4iKGZNlIKznt/1t5bWNKdVzQff3gBQZ8swGJh3MLaVB1r1zjspUe3SEIIPUNjv4 CtFYlGnl1GeK4KsSmsV0hpt+GJWZRCKMSfaq2i+VQcTBTqo/joN8IGJJb01EalZUhkZsqWi52XzFd ZWHxIyAyP0BQxc8QJOVI1FSBFoXo1NNwfgNGePFsx0nB8LgKFXiYj0yaL+VA6UP9wQusKw3aTZIUy 09GMAsFY9LK/66uO0H3TjCtnIyDTQh0lz5BUqSnHPW3ehSdcdFlOAQhQqDTRoOmAZtyC1YLYljoII 71pnN0kg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntP-0028du-Bd; Thu, 19 May 2022 21:41:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4A49E301152; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 10ADC2021AF7C; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213421.869214636@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:55 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 5/9] rcu: Fix rcu_idle_exit() References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current rcu_idle_exit() is terminally broken because it uses local_irq_{save,restore}(), which are traced which uses RCU. However, now that all the callers are sure to have IRQs disabled, we can remove these calls. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Paul E. McKenney --- kernel/rcu/tree.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -659,7 +659,7 @@ static noinstr void rcu_eqs_enter(bool u * If you add or remove a call to rcu_idle_enter(), be sure to test with * CONFIG_RCU_EQS_DEBUG=3Dy. */ -void rcu_idle_enter(void) +void noinstr rcu_idle_enter(void) { lockdep_assert_irqs_disabled(); rcu_eqs_enter(false); @@ -896,13 +896,10 @@ static void noinstr rcu_eqs_exit(bool us * If you add or remove a call to rcu_idle_exit(), be sure to test with * CONFIG_RCU_EQS_DEBUG=3Dy. */ -void rcu_idle_exit(void) +void noinstr rcu_idle_exit(void) { - unsigned long flags; - - local_irq_save(flags); + lockdep_assert_irqs_disabled(); rcu_eqs_exit(false); - local_irq_restore(flags); } EXPORT_SYMBOL_GPL(rcu_idle_exit); From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B91AC433EF for ; Thu, 19 May 2022 21:41:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245260AbiESVl4 (ORCPT ); Thu, 19 May 2022 17:41:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245176AbiESVlY (ORCPT ); Thu, 19 May 2022 17:41:24 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 038884CD56 for ; Thu, 19 May 2022 14:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=zScd4cADrYcj3rMhEyHUI7u7RAFkUNduwGg8qRRyi/k=; b=h1ABYcts9iJeZ40gBprKn+tFhi 12z3MW69jE7oLeBeuXGt9SKS33sPCX+AyZ7+568QTuFy0X/phqWpyd6H5CMXKPr8kQ7TCItRhMdCt 7RdsKkY6BmETHmJcGOrZqwiuOwsRhefpbJWbjtA1CZMl/CXnNgx8eJrPjZyK1VQo7qsPA6YcCiI7d E/48ILz1k7o6LEgxsEfOEBZo51562ScBm66VIpjuVZ6gXIB69XlR4AJgBHdV5jYxUJNDgOrdM23Zr 7Y+W8EG/gMuXgbC7wTLUHguGRGmksdPp183dMz3bJweT8ZFfy5kVAm5nKqAVMHzPvviLELwUwjyMZ N68J6kQw==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntP-0028ds-A6; Thu, 19 May 2022 21:41:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4A4CA301A50; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 167F12021AF7E; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213421.938914850@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:56 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 6/9] acpi_idle: Remove tracing References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All the idle routines are called with RCU disabled, as such there must not be any tracing inside. Signed-off-by: Peter Zijlstra (Intel) --- drivers/acpi/processor_idle.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -108,8 +108,8 @@ static const struct dmi_system_id proces static void __cpuidle acpi_safe_halt(void) { if (!tif_need_resched()) { - safe_halt(); - local_irq_disable(); + raw_safe_halt(); + raw_local_irq_disable(); } } =20 @@ -524,16 +524,21 @@ static int acpi_idle_bm_check(void) return bm_status; } =20 -static void wait_for_freeze(void) +static __cpuidle void io_idle(unsigned long addr) { + /* IO port based C-state */ + inb(addr); + #ifdef CONFIG_X86 /* No delay is needed if we are in guest */ if (boot_cpu_has(X86_FEATURE_HYPERVISOR)) return; #endif - /* Dummy wait op - must do something useless after P_LVL2 read - because chipsets cannot guarantee that STPCLK# signal - gets asserted in time to freeze execution properly. */ + /* + * Dummy wait op - must do something useless after P_LVL2 read + * because chipsets cannot guarantee that STPCLK# signal + * gets asserted in time to freeze execution properly. + */ inl(acpi_gbl_FADT.xpm_timer_block.address); } =20 @@ -553,9 +558,7 @@ static void __cpuidle acpi_idle_do_entry } else if (cx->entry_method =3D=3D ACPI_CSTATE_HALT) { acpi_safe_halt(); } else { - /* IO port based C-state */ - inb(cx->address); - wait_for_freeze(); + io_idle(cx->address); } =20 perf_lopwr_cb(false); @@ -577,8 +580,7 @@ static int acpi_idle_play_dead(struct cp if (cx->entry_method =3D=3D ACPI_CSTATE_HALT) safe_halt(); else if (cx->entry_method =3D=3D ACPI_CSTATE_SYSTEMIO) { - inb(cx->address); - wait_for_freeze(); + io_idle(cx->address); } else return -ENODEV; From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809FEC433EF for ; Thu, 19 May 2022 21:41:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245206AbiESVlh (ORCPT ); Thu, 19 May 2022 17:41:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243201AbiESVlV (ORCPT ); Thu, 19 May 2022 17:41:21 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0477E4D9E6 for ; Thu, 19 May 2022 14:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=4pFy5yUDyOn3t6yQ5LMrAaM08lEHZzH/aMff1iGk6z8=; b=qxWfOID7UF8TYDA8cO+Bwcr5Yz 90IqcFX8KvkE84q8PtXbSeF9qf942jx61MkfSJCa+uWeSyTi3t80pBTU8f/EMSSBIsNhXy8LIo3gt WTMq9ouMDYlOZuspBASafJZIJ/hGnWzkAku35uYqG1oxuw/YPUvLzypNqq1r6NRJpGGwuyt9akRAq I/x16EGv7Dkm5vrPKBNQPUSvisK95D0lwu9iMpLG8xZoyi3U9+YyNejFqaz4EjUrQJLwszXT7Twk5 YOEeBS3t8c/WsOT2BMTapP4R/rbZh++wu2+OOLFnDd1AwtLl20brjufm4LHtu/fG9ReMbAMGnhj0S zyRZ0ZVg==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntP-00D7Pn-A6; Thu, 19 May 2022 21:41:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4A39C3010C4; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1AACA2021AF80; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213421.999009886@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:57 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 7/9] cpuidle: Annotate poll_idle() References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The __cpuidle functions will become a noinstr class, as such they need explicit annotations. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/poll_state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -13,7 +13,10 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - u64 time_start =3D local_clock(); + u64 time_start; + + instrumentation_begin(); + time_start =3D local_clock(); =20 dev->poll_time_limit =3D false; =20 @@ -37,6 +40,7 @@ static int __cpuidle poll_idle(struct cp } } current_clr_polling(); + instrumentation_end(); =20 return index; } From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC3A2C433F5 for ; Thu, 19 May 2022 21:42:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245292AbiESVmD (ORCPT ); Thu, 19 May 2022 17:42:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245178AbiESVlY (ORCPT ); Thu, 19 May 2022 17:41:24 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3AAD443E9 for ; Thu, 19 May 2022 14:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=oun646QTF9CG3dxCBz/TlExnrNW2cE/edsQDkLu+uEc=; b=KZxpeCkNdnxI52U/JUB5io+pIB oyUs5oh/mpDrn+rCOeiHyykwwxL80IY+PO6dJ/9w18T8x6wazfGZ/QdUWPMNPVngZEbb1+WCKSYZe cnwks2b4QzY1cLtud8xah/jKvA9l0o2Gj+ST2GyWBllKTVTMKFPTUwND6HMIfHZMcXUA3/OyVL5IP TsOg2BMPdI8vpZp5eorUdt8LOXTZIOKMtPt9zPm3MgE9MQTnfVpyql230ZoldjJJMEjNxZypZ6VZv 0pv68VBvqC4MgAym5zQsgmEw7qnAhJmZw7fKEWoIAADUX7Vy9jHqGNFv3ULWrVRMmIxPO3pA/L60z Sd4BIqxQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntP-0028dv-CO; Thu, 19 May 2022 21:41:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4A4DF301BCC; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1E0412021AF82; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213422.059293174@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:58 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 8/9] objtool/idle: Validate __cpuidle code as noinstr References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Idle code is very like entry code in that RCU isn't available. As such, add a little validation. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/irqflags.h | 11 ++++------- arch/x86/include/asm/mwait.h | 2 +- include/linux/compiler_types.h | 8 ++++++-- include/linux/cpu.h | 3 --- tools/objtool/check.c | 15 ++++++++++++++- 5 files changed, 25 insertions(+), 14 deletions(-) --- a/arch/x86/include/asm/irqflags.h +++ b/arch/x86/include/asm/irqflags.h @@ -8,9 +8,6 @@ =20 #include =20 -/* Provide __cpuidle; we can't safely include */ -#define __cpuidle __section(".cpuidle.text") - /* * Interrupt control: */ @@ -45,13 +42,13 @@ static __always_inline void native_irq_e asm volatile("sti": : :"memory"); } =20 -static inline __cpuidle void native_safe_halt(void) +static __always_inline void native_safe_halt(void) { mds_idle_clear_cpu_buffers(); asm volatile("sti; hlt": : :"memory"); } =20 -static inline __cpuidle void native_halt(void) +static __always_inline void native_halt(void) { mds_idle_clear_cpu_buffers(); asm volatile("hlt": : :"memory"); @@ -84,7 +81,7 @@ static __always_inline void arch_local_i * Used in the idle loop; sti takes one instruction cycle * to complete: */ -static inline __cpuidle void arch_safe_halt(void) +static __always_inline void arch_safe_halt(void) { native_safe_halt(); } @@ -93,7 +90,7 @@ static inline __cpuidle void arch_safe_h * Used when interrupts are already enabled or to * shutdown the processor: */ -static inline __cpuidle void halt(void) +static __always_inline void halt(void) { native_halt(); } --- a/arch/x86/include/asm/mwait.h +++ b/arch/x86/include/asm/mwait.h @@ -104,7 +104,7 @@ static inline void __sti_mwait(unsigned * New with Core Duo processors, MWAIT can take some hints based on CPU * capability. */ -static inline void mwait_idle_with_hints(unsigned long eax, unsigned long = ecx) +static __always_inline void mwait_idle_with_hints(unsigned long eax, unsig= ned long ecx) { if (static_cpu_has_bug(X86_BUG_MONITOR) || !current_set_polling_and_test(= )) { if (static_cpu_has_bug(X86_BUG_CLFLUSH_MONITOR)) { --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -225,10 +225,14 @@ struct ftrace_likely_data { #endif =20 /* Section for code which can't be instrumented at all */ -#define noinstr \ - noinline notrace __attribute((__section__(".noinstr.text"))) \ +#define __noinstr_section(section) \ + noinline notrace __attribute((__section__(section))) \ __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage =20 +#define noinstr __noinstr_section(".noinstr.text") + +#define __cpuidle __noinstr_section(".cpuidle.text") + #endif /* __KERNEL__ */ =20 #endif /* __ASSEMBLY__ */ --- a/include/linux/cpu.h +++ b/include/linux/cpu.h @@ -171,9 +171,6 @@ void __noreturn cpu_startup_entry(enum c =20 void cpu_idle_poll_ctrl(bool enable); =20 -/* Attach to any functions which should be considered cpuidle. */ -#define __cpuidle __section(".cpuidle.text") - bool cpu_in_idle(unsigned long pc); =20 void arch_cpu_idle(void); --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -373,7 +373,8 @@ static int decode_instructions(struct ob sec->text =3D true; =20 if (!strcmp(sec->name, ".noinstr.text") || - !strcmp(sec->name, ".entry.text")) + !strcmp(sec->name, ".entry.text") || + !strcmp(sec->name, ".cpuidle.text")) sec->noinstr =3D true; =20 for (offset =3D 0; offset < sec->sh.sh_size; offset +=3D insn->len) { @@ -3077,6 +3078,12 @@ static inline bool noinstr_call_dest(str return true; =20 /* + * If the symbol is a static_call trampoline, we can't tell. + */ + if (func->static_call_tramp) + return true; + + /* * The __ubsan_handle_*() calls are like WARN(), they only happen when * something 'BAD' happened. At the risk of taking the machine down, * let them proceed to get the message out. @@ -3645,6 +3652,12 @@ static int validate_noinstr_sections(str if (sec) { warnings +=3D validate_section(file, sec); warnings +=3D validate_unwind_hints(file, sec); + } + + sec =3D find_section_by_name(file->elf, ".cpuidle.text"); + if (sec) { + warnings +=3D validate_section(file, sec); + warnings +=3D validate_unwind_hints(file, sec); } =20 return warnings; From nobody Thu May 7 19:48:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF47EC433F5 for ; Thu, 19 May 2022 21:41:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245197AbiESVle (ORCPT ); Thu, 19 May 2022 17:41:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242961AbiESVlV (ORCPT ); Thu, 19 May 2022 17:41:21 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0357446669 for ; Thu, 19 May 2022 14:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LG88lYYU67rv6HELiYc+nfr45EdkFjfKOgei8WMKd6w=; b=wXuHlkHKsedTeAWSJ9zwysjUxE nYylHK7rLhWkSxw/I5hrqbHSr4HrndrEZ2uiggeS4HsTx+AekLopGHm4mfZV9uOLhAPSNeutJQorF dR6tTDthfseafN9heHoCVIktgsJaZmeBZ1s75r/IKe7NTpR9yC82HGA7+DzsfAyQs3TBQ6y62J/Dc U6zvwInmfTAQ+s1sAMn00R9cW1kkrAhYPmHukS53kTNJig8+5Ys3azTnVUiTNi96eXmOeXlsh1Lxd FAMvFup7TAWZNcRak8K8xkAMUCAUCLrQsyYndQB8V2gKjd9WoF9oz5HK+raBnyx7KeTECYxndaYuo zEtCdgEQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrntP-00D7Pm-A6; Thu, 19 May 2022 21:41:11 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4A4FA301C50; Thu, 19 May 2022 23:41:07 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 225222021AF84; Thu, 19 May 2022 23:41:07 +0200 (CEST) Message-ID: <20220519213422.119695559@infradead.org> User-Agent: quilt/0.66 Date: Thu, 19 May 2022 23:27:59 +0200 From: Peter Zijlstra To: frederic@kernel.org, paulmck@kernel.org, rjw@rjwysocki.net, x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, jpoimboe@kernel.org Subject: [RFC][PATCH 9/9] arch/idle: Change arch_cpu_idle() IRQ behaviour References: <20220519212750.656413111@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current arch_cpu_idle() is called with IRQs disabled, but will return with IRQs enabled. However, the very first thing the generic code does after calling arch_cpu_idle() is raw_local_irq_disable(). This means that architectures that can idle with IRQs disabled end up doing a pointless 'enable-disable' dance. Therefore, push this IRQ disabling into the idle function, meaning that those architectures can avoid the pointless IRQ state flipping. Signed-off-by: Peter Zijlstra (Intel) --- arch/alpha/kernel/process.c | 1 - arch/arc/kernel/process.c | 3 +++ arch/arm/kernel/process.c | 1 - arch/arm/mach-gemini/board-dt.c | 3 ++- arch/arm64/kernel/idle.c | 1 - arch/csky/kernel/process.c | 1 - arch/csky/kernel/smp.c | 2 +- arch/h8300/kernel/process.c | 1 + arch/hexagon/kernel/process.c | 1 - arch/ia64/kernel/process.c | 1 + arch/microblaze/kernel/process.c | 1 - arch/mips/kernel/idle.c | 8 +++----- arch/nios2/kernel/process.c | 1 - arch/openrisc/kernel/process.c | 1 + arch/parisc/kernel/process.c | 2 -- arch/powerpc/kernel/idle.c | 5 ++--- arch/riscv/kernel/process.c | 1 - arch/s390/kernel/idle.c | 1 - arch/sh/kernel/idle.c | 1 + arch/sparc/kernel/leon_pmc.c | 4 ++++ arch/sparc/kernel/process_32.c | 1 - arch/sparc/kernel/process_64.c | 3 ++- arch/um/kernel/process.c | 1 - arch/x86/coco/tdx/tdx.c | 3 +++ arch/x86/kernel/process.c | 14 ++++---------- arch/xtensa/kernel/process.c | 1 + kernel/sched/idle.c | 2 -- 27 files changed, 29 insertions(+), 36 deletions(-) --- a/arch/alpha/kernel/process.c +++ b/arch/alpha/kernel/process.c @@ -57,7 +57,6 @@ EXPORT_SYMBOL(pm_power_off); void arch_cpu_idle(void) { wtint(0); - raw_local_irq_enable(); } =20 void arch_cpu_idle_dead(void) --- a/arch/arc/kernel/process.c +++ b/arch/arc/kernel/process.c @@ -114,6 +114,8 @@ void arch_cpu_idle(void) "sleep %0 \n" : :"I"(arg)); /* can't be "r" has to be embedded const */ + + raw_local_irq_disable(); } =20 #else /* ARC700 */ @@ -122,6 +124,7 @@ void arch_cpu_idle(void) { /* sleep, but enable both set E1/E2 (levels of interrupts) before committ= ing */ __asm__ __volatile__("sleep 0x3 \n"); + raw_local_irq_disable(); } =20 #endif --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -78,7 +78,6 @@ void arch_cpu_idle(void) arm_pm_idle(); else cpu_do_idle(); - raw_local_irq_enable(); } =20 void arch_cpu_idle_prepare(void) --- a/arch/arm/mach-gemini/board-dt.c +++ b/arch/arm/mach-gemini/board-dt.c @@ -42,8 +42,9 @@ static void gemini_idle(void) */ =20 /* FIXME: Enabling interrupts here is racy! */ - local_irq_enable(); + raw_local_irq_enable(); cpu_do_idle(); + raw_local_irq_disable(); } =20 static void __init gemini_init_machine(void) --- a/arch/arm64/kernel/idle.c +++ b/arch/arm64/kernel/idle.c @@ -42,5 +42,4 @@ void noinstr arch_cpu_idle(void) * tricks */ cpu_do_idle(); - raw_local_irq_enable(); } --- a/arch/csky/kernel/process.c +++ b/arch/csky/kernel/process.c @@ -102,6 +102,5 @@ void arch_cpu_idle(void) #ifdef CONFIG_CPU_PM_STOP asm volatile("stop\n"); #endif - raw_local_irq_enable(); } #endif --- a/arch/csky/kernel/smp.c +++ b/arch/csky/kernel/smp.c @@ -314,7 +314,7 @@ void arch_cpu_idle_dead(void) while (!secondary_stack) arch_cpu_idle(); =20 - local_irq_disable(); + raw_local_irq_disable(); =20 asm volatile( "mov sp, %0\n" --- a/arch/h8300/kernel/process.c +++ b/arch/h8300/kernel/process.c @@ -59,6 +59,7 @@ void arch_cpu_idle(void) { raw_local_irq_enable(); __asm__("sleep"); + raw_local_irq_disable(); } =20 void machine_restart(char *__unused) --- a/arch/hexagon/kernel/process.c +++ b/arch/hexagon/kernel/process.c @@ -44,7 +44,6 @@ void arch_cpu_idle(void) { __vmwait(); /* interrupts wake us up, but irqs are still disabled */ - raw_local_irq_enable(); } =20 /* --- a/arch/ia64/kernel/process.c +++ b/arch/ia64/kernel/process.c @@ -241,6 +241,7 @@ void arch_cpu_idle(void) (*mark_idle)(1); =20 raw_safe_halt(); + raw_local_irq_disable(); =20 if (mark_idle) (*mark_idle)(0); --- a/arch/microblaze/kernel/process.c +++ b/arch/microblaze/kernel/process.c @@ -138,5 +138,4 @@ int dump_fpu(struct pt_regs *regs, elf_f =20 void arch_cpu_idle(void) { - raw_local_irq_enable(); } --- a/arch/mips/kernel/idle.c +++ b/arch/mips/kernel/idle.c @@ -33,13 +33,13 @@ static void __cpuidle r3081_wait(void) { unsigned long cfg =3D read_c0_conf(); write_c0_conf(cfg | R30XX_CONF_HALT); - raw_local_irq_enable(); } =20 void __cpuidle r4k_wait(void) { raw_local_irq_enable(); __r4k_wait(); + raw_local_irq_disable(); } =20 /* @@ -57,7 +57,6 @@ void __cpuidle r4k_wait_irqoff(void) " .set arch=3Dr4000 \n" " wait \n" " .set pop \n"); - raw_local_irq_enable(); } =20 /* @@ -77,7 +76,6 @@ static void __cpuidle rm7k_wait_irqoff(v " wait \n" " mtc0 $1, $12 # stalls until W stage \n" " .set pop \n"); - raw_local_irq_enable(); } =20 /* @@ -103,6 +101,8 @@ static void __cpuidle au1k_wait(void) " nop \n" " .set pop \n" : : "r" (au1k_wait), "r" (c0status)); + + raw_local_irq_disable(); } =20 static int __initdata nowait; @@ -245,8 +245,6 @@ void arch_cpu_idle(void) { if (cpu_wait) cpu_wait(); - else - raw_local_irq_enable(); } =20 #ifdef CONFIG_CPU_IDLE --- a/arch/nios2/kernel/process.c +++ b/arch/nios2/kernel/process.c @@ -33,7 +33,6 @@ EXPORT_SYMBOL(pm_power_off); =20 void arch_cpu_idle(void) { - raw_local_irq_enable(); } =20 /* --- a/arch/openrisc/kernel/process.c +++ b/arch/openrisc/kernel/process.c @@ -87,6 +87,7 @@ void arch_cpu_idle(void) raw_local_irq_enable(); if (mfspr(SPR_UPR) & SPR_UPR_PMP) mtspr(SPR_PMR, mfspr(SPR_PMR) | SPR_PMR_DME); + raw_local_irq_disable(); } =20 void (*pm_power_off) (void) =3D machine_power_off; --- a/arch/parisc/kernel/process.c +++ b/arch/parisc/kernel/process.c @@ -187,8 +187,6 @@ void arch_cpu_idle_dead(void) =20 void __cpuidle arch_cpu_idle(void) { - raw_local_irq_enable(); - /* nop on real hardware, qemu will idle sleep. */ asm volatile("or %%r10,%%r10,%%r10\n":::); } --- a/arch/powerpc/kernel/idle.c +++ b/arch/powerpc/kernel/idle.c @@ -51,10 +51,9 @@ void arch_cpu_idle(void) * Some power_save functions return with * interrupts enabled, some don't. */ - if (irqs_disabled()) - raw_local_irq_enable(); + if (!irqs_disabled()) + raw_local_irq_disable(); } else { - raw_local_irq_enable(); /* * Go into low thread priority and possibly * low power mode. --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -39,7 +39,6 @@ extern asmlinkage void ret_from_kernel_t void arch_cpu_idle(void) { cpu_do_idle(); - raw_local_irq_enable(); } =20 void __show_regs(struct pt_regs *regs) --- a/arch/s390/kernel/idle.c +++ b/arch/s390/kernel/idle.c @@ -66,7 +66,6 @@ void arch_cpu_idle(void) idle->idle_count++; account_idle_time(cputime_to_nsecs(idle_time)); raw_write_seqcount_end(&idle->seqcount); - raw_local_irq_enable(); } =20 static ssize_t show_idle_count(struct device *dev, --- a/arch/sh/kernel/idle.c +++ b/arch/sh/kernel/idle.c @@ -25,6 +25,7 @@ void default_idle(void) raw_local_irq_enable(); /* Isn't this racy ? */ cpu_sleep(); + raw_local_irq_disable(); clear_bl_bit(); } =20 --- a/arch/sparc/kernel/leon_pmc.c +++ b/arch/sparc/kernel/leon_pmc.c @@ -57,6 +57,8 @@ static void pmc_leon_idle_fixup(void) "lda [%0] %1, %%g0\n" : : "r"(address), "i"(ASI_LEON_BYPASS)); + + raw_local_irq_disable(); } =20 /* @@ -70,6 +72,8 @@ static void pmc_leon_idle(void) =20 /* For systems without power-down, this will be no-op */ __asm__ __volatile__ ("wr %g0, %asr19\n\t"); + + raw_local_irq_disable(); } =20 /* Install LEON Power Down function */ --- a/arch/sparc/kernel/process_32.c +++ b/arch/sparc/kernel/process_32.c @@ -71,7 +71,6 @@ void arch_cpu_idle(void) { if (sparc_idle) (*sparc_idle)(); - raw_local_irq_enable(); } =20 /* XXX cli/sti -> local_irq_xxx here, check this works once SMP is fixed. = */ --- a/arch/sparc/kernel/process_64.c +++ b/arch/sparc/kernel/process_64.c @@ -59,7 +59,6 @@ void arch_cpu_idle(void) { if (tlb_type !=3D hypervisor) { touch_nmi_watchdog(); - raw_local_irq_enable(); } else { unsigned long pstate; =20 @@ -90,6 +89,8 @@ void arch_cpu_idle(void) "wrpr %0, %%g0, %%pstate" : "=3D&r" (pstate) : "i" (PSTATE_IE)); + + raw_local_irq_disable(); } } =20 --- a/arch/um/kernel/process.c +++ b/arch/um/kernel/process.c @@ -216,7 +216,6 @@ void arch_cpu_idle(void) { cpu_tasks[current_thread_info()->cpu].pid =3D os_getpid(); um_idle_sleep(); - raw_local_irq_enable(); } =20 int __cant_sleep(void) { --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -178,6 +178,9 @@ void __cpuidle tdx_safe_halt(void) */ if (__halt(irq_disabled, do_sti)) WARN_ONCE(1, "HLT instruction emulation failed\n"); + + /* XXX I can't make sense of what @do_sti actually does */ + raw_local_irq_disable(); } =20 static bool read_msr(struct pt_regs *regs) --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -699,6 +699,7 @@ EXPORT_SYMBOL(boot_option_idle_override) void __cpuidle default_idle(void) { raw_safe_halt(); + raw_local_irq_disable(); } #if defined(CONFIG_APM_MODULE) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE) EXPORT_SYMBOL(default_idle); @@ -804,13 +805,7 @@ static void amd_e400_idle(void) =20 default_idle(); =20 - /* - * The switch back from broadcast mode needs to be called with - * interrupts disabled. - */ - raw_local_irq_disable(); tick_broadcast_exit(); - raw_local_irq_enable(); } =20 /* @@ -849,12 +844,11 @@ static __cpuidle void mwait_idle(void) } =20 __monitor((void *)¤t_thread_info()->flags, 0, 0); - if (!need_resched()) + if (!need_resched()) { __sti_mwait(0, 0); - else - raw_local_irq_enable(); + raw_local_irq_disable(); + } } else { - raw_local_irq_enable(); } __current_clr_polling(); } --- a/arch/xtensa/kernel/process.c +++ b/arch/xtensa/kernel/process.c @@ -120,6 +120,7 @@ void coprocessor_flush_all(struct thread void arch_cpu_idle(void) { platform_idle(); + raw_local_irq_disable(); } =20 /* --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -79,7 +79,6 @@ void __weak arch_cpu_idle_dead(void) { } void __weak arch_cpu_idle(void) { cpu_idle_force_poll =3D 1; - raw_local_irq_enable(); } =20 /** @@ -96,7 +95,6 @@ void __cpuidle default_idle_call(void) =20 cpuidle_rcu_enter(); arch_cpu_idle(); - raw_local_irq_disable(); cpuidle_rcu_exit(); =20 start_critical_timings();