From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68CC8EB64D9 for ; Thu, 15 Jun 2023 20:34:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232248AbjFOUd7 (ORCPT ); Thu, 15 Jun 2023 16:33:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231837AbjFOUdy (ORCPT ); Thu, 15 Jun 2023 16:33:54 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3F21271E for ; Thu, 15 Jun 2023 13:33:52 -0700 (PDT) Message-ID: <20230615193330.263684884@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861231; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=acEqf4q7Qtm/dIVEXjGN26S1eTXxFBpCuNJbm8wwWUk=; b=wtYkDT5Wp6cmzquQsLIDVXHRu82TcABoicmjPmIBCdv4bMY3rUr6L5NL4TdVubX56rzsp8 CQm0ipkWL7tBKZhiNUZ9kF8rGD70UM6+NB8N1JxmlNWUyWtxs4n9/wAeb1y2dcaJYPXcJ6 4W5TcaKUDXYBuI6EPmCtStHLc7GA0Mu1qy04W49HlwotirGkrJkPsW/ndRf9GADvx48Vqd wV+j/su6/vnHZ1aExpqUOvdGMM9D73RmxpBN0Mr/rpHWluywWIjiJZr+svPAScO77gnsqn Cowk7vti+/cWmE5gipPNLYhsxTOzd0mn+tYK6bPSuCd8fyjc1+VJWuCoWm4EZg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861231; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=acEqf4q7Qtm/dIVEXjGN26S1eTXxFBpCuNJbm8wwWUk=; b=zkPF7gIjgtraA0L54z3AInrIrOMZ6k6R6IooaRpMMHKcIrF9KVYs2f10pRQy0BvTa6wtJK W0qnqkggPK+sXKCA== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman Subject: [patch v3 1/7] x86/smp: Make stop_other_cpus() more robust References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:33:50 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Tony reported intermittent lockups on poweroff. His analysis identified the wbinvd() in stop_this_cpu() as the culprit. This was added to ensure that on SME enabled machines a kexec() does not leave any stale data in the caches when switching from encrypted to non-encrypted mode or vice versa. That wbindv() is conditional on the SME feature bit which is read directly from CPUID. But that readout does not check whether the CPUID leaf is available or not. If it's not available the CPU will return the value of the highest supported leaf instead. Depending on the content the "SME" bit might be set or not. That's incorrect but harmless. Making the CPUID readout conditional makes the observed hangs go away, but it does not fix the underlying problem: CPU0 CPU1 stop_other_cpus() send_IPIs(REBOOT); stop_this_cpu() while (num_online_cpus() > 1); set_online(false); proceed... -> hang wbinvd() WBINVD is an expensive operation and if multiple CPUs issue it at the same time the resulting delays are even larger. But CPU0 already observed num_online_cpus() going down to 1 and proceeds which causes the system to hang. This issue exists independent of WBINVD, but the delays caused by WBINVD make it more prominent. Make this more robust by adding a cpumask which is initialized to the online CPU mask before sending the IPIs and CPUs clear their bit in stop_this_cpu() after the WBINVD completed. Check for that cpumask to become empty in stop_other_cpus() instead of watching num_online_cpus(). The cpumask cannot plug all holes either, but it's better than a raw counter and allows to restrict the NMI fallback IPI to be sent only to the CPUs which have not reported within the timeout window. Fixes: 08f253ec3767 ("x86/cpu: Clear SME feature flag when not in use") Reported-by: Tony Battersby Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/all/3817d810-e0f1-8ef8-0bbd-663b919ca49b@cybe= rnetics.com Reviewed-by: Ashok Raj Reviewed-by: Borislav Petkov (AMD) Tested-by: Tony Battersby --- V3: Use a cpumask to make the NMI case slightly safer - Ashok --- arch/x86/include/asm/cpu.h | 2 + arch/x86/kernel/process.c | 23 +++++++++++++- arch/x86/kernel/smp.c | 71 +++++++++++++++++++++++++++++++---------= ----- 3 files changed, 73 insertions(+), 23 deletions(-) --- a/arch/x86/include/asm/cpu.h +++ b/arch/x86/include/asm/cpu.h @@ -98,4 +98,6 @@ extern u64 x86_read_arch_cap_msr(void); int intel_find_matching_signature(void *mc, unsigned int csig, int cpf); int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type); =20 +extern struct cpumask cpus_stop_mask; + #endif /* _ASM_X86_CPU_H */ --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -759,13 +759,23 @@ bool xen_set_default_idle(void) } #endif =20 +struct cpumask cpus_stop_mask; + void __noreturn stop_this_cpu(void *dummy) { + unsigned int cpu =3D smp_processor_id(); + local_irq_disable(); + /* - * Remove this CPU: + * Remove this CPU from the online mask and disable it + * unconditionally. This might be redundant in case that the reboot + * vector was handled late and stop_other_cpus() sent an NMI. + * + * According to SDM and APM NMIs can be accepted even after soft + * disabling the local APIC. */ - set_cpu_online(smp_processor_id(), false); + set_cpu_online(cpu, false); disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); =20 @@ -783,6 +793,15 @@ void __noreturn stop_this_cpu(void *dumm */ if (cpuid_eax(0x8000001f) & BIT(0)) native_wbinvd(); + + /* + * This brings a cache line back and dirties it, but + * native_stop_other_cpus() will overwrite cpus_stop_mask after it + * observed that all CPUs reported stop. This write will invalidate + * the related cache line on this CPU. + */ + cpumask_clear_cpu(cpu, &cpus_stop_mask); + for (;;) { /* * Use native_halt() so that memory contents don't change --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -146,31 +147,43 @@ static int register_stop_handler(void) =20 static void native_stop_other_cpus(int wait) { - unsigned long flags; - unsigned long timeout; + unsigned int cpu =3D smp_processor_id(); + unsigned long flags, timeout; =20 if (reboot_force) return; =20 - /* - * Use an own vector here because smp_call_function - * does lots of things not suitable in a panic situation. - */ + /* Only proceed if this is the first CPU to reach this code */ + if (atomic_cmpxchg(&stopping_cpu, -1, cpu) !=3D -1) + return; =20 /* - * We start by using the REBOOT_VECTOR irq. - * The irq is treated as a sync point to allow critical - * regions of code on other cpus to release their spin locks - * and re-enable irqs. Jumping straight to an NMI might - * accidentally cause deadlocks with further shutdown/panic - * code. By syncing, we give the cpus up to one second to - * finish their work before we force them off with the NMI. + * 1) Send an IPI on the reboot vector to all other CPUs. + * + * The other CPUs should react on it after leaving critical + * sections and re-enabling interrupts. They might still hold + * locks, but there is nothing which can be done about that. + * + * 2) Wait for all other CPUs to report that they reached the + * HLT loop in stop_this_cpu() + * + * 3) If #2 timed out send an NMI to the CPUs which did not + * yet report + * + * 4) Wait for all other CPUs to report that they reached the + * HLT loop in stop_this_cpu() + * + * #3 can obviously race against a CPU reaching the HLT loop late. + * That CPU will have reported already and the "have all CPUs + * reached HLT" condition will be true despite the fact that the + * other CPU is still handling the NMI. Again, there is no + * protection against that as "disabled" APICs still respond to + * NMIs. */ - if (num_online_cpus() > 1) { - /* did someone beat us here? */ - if (atomic_cmpxchg(&stopping_cpu, -1, safe_smp_processor_id()) !=3D -1) - return; + cpumask_copy(&cpus_stop_mask, cpu_online_mask); + cpumask_clear_cpu(cpu, &cpus_stop_mask); =20 + if (!cpumask_empty(&cpus_stop_mask)) { /* sync above data before sending IRQ */ wmb(); =20 @@ -183,24 +196,34 @@ static void native_stop_other_cpus(int w * CPUs reach shutdown state. */ timeout =3D USEC_PER_SEC; - while (num_online_cpus() > 1 && timeout--) + while (!cpumask_empty(&cpus_stop_mask) && timeout--) udelay(1); } =20 /* if the REBOOT_VECTOR didn't work, try with the NMI */ - if (num_online_cpus() > 1) { + if (!cpumask_empty(&cpus_stop_mask)) { /* * If NMI IPI is enabled, try to register the stop handler * and send the IPI. In any case try to wait for the other * CPUs to stop. */ if (!smp_no_nmi_ipi && !register_stop_handler()) { + u32 dm; + /* Sync above data before sending IRQ */ wmb(); =20 pr_emerg("Shutting down cpus with NMI\n"); =20 - apic_send_IPI_allbutself(NMI_VECTOR); + dm =3D apic->dest_mode_logical ? APIC_DEST_LOGICAL : APIC_DEST_PHYSICAL; + dm |=3D APIC_DM_NMI; + + for_each_cpu(cpu, &cpus_stop_mask) { + u32 apicid =3D apic->cpu_present_to_apicid(cpu); + + apic_icr_write(dm, apicid); + apic_wait_icr_idle(); + } } /* * Don't wait longer than 10 ms if the caller didn't @@ -208,7 +231,7 @@ static void native_stop_other_cpus(int w * one or more CPUs do not reach shutdown state. */ timeout =3D USEC_PER_MSEC * 10; - while (num_online_cpus() > 1 && (wait || timeout--)) + while (!cpumask_empty(&cpus_stop_mask) && (wait || timeout--)) udelay(1); } =20 @@ -216,6 +239,12 @@ static void native_stop_other_cpus(int w disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); local_irq_restore(flags); + + /* + * Ensure that the cpus_stop_mask cache lines are invalidated on + * the other CPUs. See comment vs. SME in stop_this_cpu(). + */ + cpumask_clear(&cpus_stop_mask); } =20 /* From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 781B8EB64DB for ; Thu, 15 Jun 2023 20:34:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232525AbjFOUeC (ORCPT ); Thu, 15 Jun 2023 16:34:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232027AbjFOUdz (ORCPT ); Thu, 15 Jun 2023 16:33:55 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C9EC2711 for ; Thu, 15 Jun 2023 13:33:54 -0700 (PDT) Message-ID: <20230615193330.322186388@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=F5oJwB0GE84n/VmQ+gAa3Veuk6SxbBp4c68eFnUy6Y0=; b=UMuAs2SM92SW0z9tELVFy5u/AMkL0NDmXH+pafezqQnMYC6OcnR88nEjpLcIpn5Z31tJ5C W91OTZbh7MzxP0ZllmRwkGkeyQN5UNiiJzrNYM+LXF15SvSD08/SZwXgKw1zfeg4uHevjR F8k9caqbsp+XR4MbKy6RyPOuTt7yVGEiI0lizgpovAvqGYur+DGqGXWOTcthd6Hov1hqZV 5CoSfIv9E6/BcqfiQGAUo7BhhyODWKAF8yfLhx+0zXqJdpAA5V2+7RIwrwCKjQE3SvO/Dp EP6QVqvQBCZ4A5Cc9S2SKdFgu0mV2lwucvNBc4xvYhl/2NHl6QVSZLgS7OD6QA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=F5oJwB0GE84n/VmQ+gAa3Veuk6SxbBp4c68eFnUy6Y0=; b=c6anQgBHPb6ZHy+bj8yaIUYehPRUCxgGiX9lejOeWUl50NRgTNB9O6jeTDFaMRCzcZoeNN luLww6OHL36NQdAA== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman Subject: [patch v3 2/7] x86/smp: Dont access non-existing CPUID leaf References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:33:52 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tony Battersby stop_this_cpu() tests CPUID leaf 0x8000001f::EAX unconditionally. CPUs return the content of the highest supported leaf when a non-existing leaf is read. So the result of the test is lottery except on AMD CPUs which support that leaf. While harmless it's incorrect and causes the conditional wbinvd() to be issued where not required. Check whether the leaf is supported before reading it. [ tglx: Adjusted changelog ] Fixes: 08f253ec3767 ("x86/cpu: Clear SME feature flag when not in use") Signed-off-by: Tony Battersby Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/r/3817d810-e0f1-8ef8-0bbd-663b919ca49b@cybern= etics.com Reviewed-by: Borislav Petkov (AMD) Reviewed-by: Mario Limonciello --- arch/x86/kernel/process.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -763,6 +763,7 @@ struct cpumask cpus_stop_mask; =20 void __noreturn stop_this_cpu(void *dummy) { + struct cpuinfo_x86 *c =3D this_cpu_ptr(&cpu_info); unsigned int cpu =3D smp_processor_id(); =20 local_irq_disable(); @@ -777,7 +778,7 @@ void __noreturn stop_this_cpu(void *dumm */ set_cpu_online(cpu, false); disable_local_APIC(); - mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); + mcheck_cpu_clear(c); =20 /* * Use wbinvd on processors that support SME. This provides support @@ -791,7 +792,7 @@ void __noreturn stop_this_cpu(void *dumm * Test the CPUID bit directly because the machine might've cleared * X86_FEATURE_SME due to cmdline options. */ - if (cpuid_eax(0x8000001f) & BIT(0)) + if (c->extended_cpuid_level >=3D 0x8000001f && (cpuid_eax(0x8000001f) & B= IT(0))) native_wbinvd(); =20 /* From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F9D1EB64DA for ; Thu, 15 Jun 2023 20:34:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237562AbjFOUeG (ORCPT ); Thu, 15 Jun 2023 16:34:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232067AbjFOUd4 (ORCPT ); Thu, 15 Jun 2023 16:33:56 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C8E12711 for ; Thu, 15 Jun 2023 13:33:56 -0700 (PDT) Message-ID: <20230615193330.378358382@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=1+WMbgfctMXWtOKcGKW/wJHvdbPeyuqNojFzRQeoCU8=; b=nUb0Dcd9Wxw6PxWAFilnXXJDm3n0Hf3nSKfqXxWsDlM7mbktaUcmJctrcP9SYfyabnw2hC 5qyHXKmjYRXGw+qgkO1C4Jde5AlWm4C+rOpMSBFJXGXKZL3zU2Jmp+TbGHUoRIBY92hvdY C9fJFhu+lqjrx3t0EQzpEbBZXwcbMEGQxTgFKXB2WwO/C4a9OzrjqH1l2lBzElxvquE1r+ WIg38DEhP61uumeEFZsfRX4iKLRh9ITsbMPB2iCwWEVMAMSrBrftJWdcinwGmes0wKsBF4 xD9Iw4pDCKXMhaVLpeWAQ3nC8BGmRXj4b5EGW+5NWXq+JdMWKyNXNIUxLO1SgA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=1+WMbgfctMXWtOKcGKW/wJHvdbPeyuqNojFzRQeoCU8=; b=vfWJWIssuqpvGT6HKL9HUhoEwkD5Lk4wGLv8q2r4naRUuPPfJ2L5kKHHKCsTr4Sc8Fo3p2 W1YK1YIMjsjup5Ag== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman Subject: [patch v3 3/7] x86/smp: Remove pointless wmb()s from native_stop_other_cpus() References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:33:54 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The wmb()s before sending the IPIs are not synchronizing anything. If at all then the apic IPI functions have to provide or act as appropriate barriers. Remove these cargo cult barriers which have no explanation of what they are synchronizing. Signed-off-by: Thomas Gleixner Reviewed-by: Borislav Petkov (AMD) --- V3: Remove second instance and reword changelog - PeterZ --- arch/x86/kernel/smp.c | 6 ------ 1 file changed, 6 deletions(-) --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -184,9 +184,6 @@ static void native_stop_other_cpus(int w cpumask_clear_cpu(cpu, &cpus_stop_mask); =20 if (!cpumask_empty(&cpus_stop_mask)) { - /* sync above data before sending IRQ */ - wmb(); - apic_send_IPI_allbutself(REBOOT_VECTOR); =20 /* @@ -210,9 +207,6 @@ static void native_stop_other_cpus(int w if (!smp_no_nmi_ipi && !register_stop_handler()) { u32 dm; =20 - /* Sync above data before sending IRQ */ - wmb(); - pr_emerg("Shutting down cpus with NMI\n"); =20 dm =3D apic->dest_mode_logical ? APIC_DEST_LOGICAL : APIC_DEST_PHYSICAL; From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 427BDEB64D9 for ; Thu, 15 Jun 2023 20:34:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237760AbjFOUeL (ORCPT ); Thu, 15 Jun 2023 16:34:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232118AbjFOUd6 (ORCPT ); Thu, 15 Jun 2023 16:33:58 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8AB57271E for ; Thu, 15 Jun 2023 13:33:57 -0700 (PDT) Message-ID: <20230615193330.434553750@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=tVfoJ5rLMlFkeEBpRbPrXavA+I9P+UPy7NG09ZxG1es=; b=S4BfXxBXNNJ5B3Kp+5KHAQzKf+LHuMTTDNRQ2Am/JAHpHjLeI40iWPmcHky8GX7D86imf9 WN5xDCrx8DQwywDGBBBpJZwb8xc33QMaTcTH/UNNUHjDh9kjbru/3f4eNm2AI5uYYhPJat l7BQM8YiYFtHnO5HPNh/5/7bMwzHPW8CxVdAdmR+k0lfNJymnq6f6rgubuJRByrdPs79Ag 2bBxV4ztnQiWNH6Kq1Vanpttnl2f5lZFk0d2JgqjaMO6A1rGBAhpovcKIR3Aqz0Y+1dxsd 6MeaxuZxcTaSfog5eYKbCXegS159q/o5SQNqzCHm4qDfD+Rb7LPThHYXYdQznA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=tVfoJ5rLMlFkeEBpRbPrXavA+I9P+UPy7NG09ZxG1es=; b=fAYM2QaSuEVf7/qNOLCtOFT0tvHkbuo21+sU5Ov4SwNWaS62e10Dqg6NP1nrDVaz+BUZZF GCNaWS7/oTKuSvCw== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman , Ashok Raj Subject: [patch v3 4/7] x86/smp: Use dedicated cache-line for mwait_play_dead() References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:33:55 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Monitoring idletask::thread_info::flags in mwait_play_dead() has been an obvious choice as all what is needed is a cache line which is not written by other CPUs. But there is a use case where a "dead" CPU needs to be brought out of that mwait(): kexec(). The CPU needs to be brought out of mwait before kexec() as kexec() can overwrite text, pagetables, stacks and the monitored cacheline of the original kernel. The latter causes mwait to resume execution which obviously causes havoc on the kexec kernel which results usually in triple faults. Use a dedicated per CPU storage to prepare for that. Signed-off-by: Thomas Gleixner Reviewed-by: Ashok Raj Reviewed-by: Borislav Petkov (AMD) --- arch/x86/kernel/smpboot.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -101,6 +101,17 @@ EXPORT_PER_CPU_SYMBOL(cpu_die_map); DEFINE_PER_CPU_READ_MOSTLY(struct cpuinfo_x86, cpu_info); EXPORT_PER_CPU_SYMBOL(cpu_info); =20 +struct mwait_cpu_dead { + unsigned int control; + unsigned int status; +}; + +/* + * Cache line aligned data for mwait_play_dead(). Separate on purpose so + * that it's unlikely to be touched by other CPUs. + */ +static DEFINE_PER_CPU_ALIGNED(struct mwait_cpu_dead, mwait_cpu_dead); + /* Logical package management. We might want to allocate that dynamically = */ unsigned int __max_logical_packages __read_mostly; EXPORT_SYMBOL(__max_logical_packages); @@ -1758,10 +1769,10 @@ EXPORT_SYMBOL_GPL(cond_wakeup_cpu0); */ static inline void mwait_play_dead(void) { + struct mwait_cpu_dead *md =3D this_cpu_ptr(&mwait_cpu_dead); unsigned int eax, ebx, ecx, edx; unsigned int highest_cstate =3D 0; unsigned int highest_subcstate =3D 0; - void *mwait_ptr; int i; =20 if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_AMD || @@ -1796,13 +1807,6 @@ static inline void mwait_play_dead(void) (highest_subcstate - 1); } =20 - /* - * This should be a memory location in a cache line which is - * unlikely to be touched by other processors. The actual - * content is immaterial as it is not actually modified in any way. - */ - mwait_ptr =3D ¤t_thread_info()->flags; - wbinvd(); =20 while (1) { @@ -1814,9 +1818,9 @@ static inline void mwait_play_dead(void) * case where we return around the loop. */ mb(); - clflush(mwait_ptr); + clflush(md); mb(); - __monitor(mwait_ptr, 0, 0); + __monitor(md, 0, 0); mb(); __mwait(eax, 0); From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAA95EB64D9 for ; Thu, 15 Jun 2023 20:34:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237797AbjFOUeP (ORCPT ); Thu, 15 Jun 2023 16:34:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232432AbjFOUeA (ORCPT ); Thu, 15 Jun 2023 16:34:00 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EE442711 for ; Thu, 15 Jun 2023 13:33:59 -0700 (PDT) Message-ID: <20230615193330.492257119@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861238; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ppl1FumG424u3l+T6Q6nu0slvzTBObUWClLNeDTKQKA=; b=odD/+50NSkGByh44uwDD6QThazZcPdybIJgp2Uv4cOtnRkwgZuG71vAasn6/CrNOn0nByL +7Oai2YpPksaVNLGbB8pEDs1uI8eAkrkWQQ3RGHHYJzfRA0TsTRgihju5SWgx5O5pN9SHx GvjR0XF7Y5yXYEGNrcD4EEu8U9tFD39Wd2X8jJyvZ2xvX37T0AQA8OerS8DLnkbdo32FNY Wf2FwYWBCjF0K0Y50Bi8wz+j6i3CGN05Xk29UrZrH9VHZO3YNTznlPaHswGb66yaVM6hmX /EH+S6qcRcgRkpreDhbKwAc15QT5RHUW1RU6jVPaS1GQA3jbNZCB6sKm+hYshA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861238; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ppl1FumG424u3l+T6Q6nu0slvzTBObUWClLNeDTKQKA=; b=ceE0mbKkYq3Pr9yibPe0kAXssFZhAK9gTajGDqcIKwKg5Pz6004wj82y/RKSklnJyCUJ2c tfMm0OZTV9GlmNAA== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman , Ashok Raj Subject: [patch v3 5/7] x86/smp: Cure kexec() vs. mwait_play_dead() breakage References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:33:57 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" TLDR: It's a mess. When kexec() is executed on a system with "offline" CPUs, which are parked in mwait_play_dead() it can end up in a triple fault during the bootup of the kexec kernel or cause hard to diagnose data corruption. The reason is that kexec() eventually overwrites the previous kernels text, page tables, data and stack, If it writes to the cache line which is monitored by an previously offlined CPU, MWAIT resumes execution and ends up executing the wrong text, dereferencing overwritten page tables or corrupting the kexec kernels data. Cure this by bringing the offline CPUs out of MWAIT into HLT. Write to the monitored cache line of each offline CPU, which makes MWAIT resume execution. The written control word tells the offline CPUs to issue HLT, which does not have the MWAIT problem. That does not help, if a stray NMI, MCE or SMI hits the offline CPUs as those make it come out of HLT. A follow up change will put them into INIT, which protects at least against NMI and SMI. Fixes: ea53069231f9 ("x86, hotplug: Use mwait to offline a processor, fix t= he legacy case") Reported-by: Ashok Raj Signed-off-by: Thomas Gleixner Tested-by: Ashok Raj Reviewed-by: Ashok Raj --- arch/x86/include/asm/smp.h | 2 + arch/x86/kernel/smp.c | 5 +++ arch/x86/kernel/smpboot.c | 59 ++++++++++++++++++++++++++++++++++++++++= +++++ 3 files changed, 66 insertions(+) --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -132,6 +132,8 @@ void wbinvd_on_cpu(int cpu); int wbinvd_on_all_cpus(void); void cond_wakeup_cpu0(void); =20 +void smp_kick_mwait_play_dead(void); + void native_smp_send_reschedule(int cpu); void native_send_call_func_ipi(const struct cpumask *mask); void native_send_call_func_single_ipi(int cpu); --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -21,6 +21,7 @@ #include #include #include +#include =20 #include #include @@ -157,6 +158,10 @@ static void native_stop_other_cpus(int w if (atomic_cmpxchg(&stopping_cpu, -1, cpu) !=3D -1) return; =20 + /* For kexec, ensure that offline CPUs are out of MWAIT and in HLT */ + if (kexec_in_progress) + smp_kick_mwait_play_dead(); + /* * 1) Send an IPI on the reboot vector to all other CPUs. * --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include #include @@ -106,6 +107,9 @@ struct mwait_cpu_dead { unsigned int status; }; =20 +#define CPUDEAD_MWAIT_WAIT 0xDEADBEEF +#define CPUDEAD_MWAIT_KEXEC_HLT 0x4A17DEAD + /* * Cache line aligned data for mwait_play_dead(). Separate on purpose so * that it's unlikely to be touched by other CPUs. @@ -173,6 +177,10 @@ static void smp_callin(void) { int cpuid; =20 + /* Mop up eventual mwait_play_dead() wreckage */ + this_cpu_write(mwait_cpu_dead.status, 0); + this_cpu_write(mwait_cpu_dead.control, 0); + /* * If waken up by an INIT in an 82489DX configuration * cpu_callout_mask guarantees we don't get here before @@ -1807,6 +1815,10 @@ static inline void mwait_play_dead(void) (highest_subcstate - 1); } =20 + /* Set up state for the kexec() hack below */ + md->status =3D CPUDEAD_MWAIT_WAIT; + md->control =3D CPUDEAD_MWAIT_WAIT; + wbinvd(); =20 while (1) { @@ -1824,10 +1836,57 @@ static inline void mwait_play_dead(void) mb(); __mwait(eax, 0); =20 + if (READ_ONCE(md->control) =3D=3D CPUDEAD_MWAIT_KEXEC_HLT) { + /* + * Kexec is about to happen. Don't go back into mwait() as + * the kexec kernel might overwrite text and data including + * page tables and stack. So mwait() would resume when the + * monitor cache line is written to and then the CPU goes + * south due to overwritten text, page tables and stack. + * + * Note: This does _NOT_ protect against a stray MCE, NMI, + * SMI. They will resume execution at the instruction + * following the HLT instruction and run into the problem + * which this is trying to prevent. + */ + WRITE_ONCE(md->status, CPUDEAD_MWAIT_KEXEC_HLT); + while(1) + native_halt(); + } + cond_wakeup_cpu0(); } } =20 +/* + * Kick all "offline" CPUs out of mwait on kexec(). See comment in + * mwait_play_dead(). + */ +void smp_kick_mwait_play_dead(void) +{ + u32 newstate =3D CPUDEAD_MWAIT_KEXEC_HLT; + struct mwait_cpu_dead *md; + unsigned int cpu, i; + + for_each_cpu_andnot(cpu, cpu_present_mask, cpu_online_mask) { + md =3D per_cpu_ptr(&mwait_cpu_dead, cpu); + + /* Does it sit in mwait_play_dead() ? */ + if (READ_ONCE(md->status) !=3D CPUDEAD_MWAIT_WAIT) + continue; + + /* Wait maximal 5ms */ + for (i =3D 0; READ_ONCE(md->status) !=3D newstate && i < 1000; i++) { + /* Bring it out of mwait */ + WRITE_ONCE(md->control, newstate); + udelay(5); + } + + if (READ_ONCE(md->status) !=3D newstate) + pr_err("CPU%u is stuck in mwait_play_dead()\n", cpu); + } +} + void __noreturn hlt_play_dead(void) { if (__this_cpu_read(cpu_info.x86) >=3D 4) From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D15FEB64D9 for ; Thu, 15 Jun 2023 20:34:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232553AbjFOUeS (ORCPT ); Thu, 15 Jun 2023 16:34:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231320AbjFOUeB (ORCPT ); Thu, 15 Jun 2023 16:34:01 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A90D5271E for ; Thu, 15 Jun 2023 13:34:00 -0700 (PDT) Message-ID: <20230615193330.551157083@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=TBdYSJyouEMeMZNGX8gFG7/P16NsNnlIxKFRjUwkdQA=; b=j1c6y+H+ghd90bqWcOYaeek1KPZwPr4s786lnbW2HLa9/lStXSxPNBQqGR0eEhv/o48TtW wqT2ySFUmcaquU8eMXIhz8tgLomGDQ9JQOQ52Ghn25c3LticxOcEJR/Wv5UAAepZgdTFlN TGd2NoHF6fKAJDer7Tj+kaa28gMim6jnyzPROFw+0E+V3xH3Zp7LEcxLMjxkBMTaw0KjOn hNNYeKGS0f8CrJ1OOwKaE88DuSgFJf+IQlq9WmM2NTG2A/Dind5UQtCBtuoIaHFWFn6G6b HF5UifVUhVl5Suj1M0O0ixJfB5So51YX2PITXHx09CdOkIUEsrTe8MqQjB+yHg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=TBdYSJyouEMeMZNGX8gFG7/P16NsNnlIxKFRjUwkdQA=; b=SHOg7nOYSiQk/qkaMe8xz62FLyXEHmClkNZJcz0GMKAMPhXPe3oCHuJlUg2i5gXFVBRohk ZKM55SH+d+K7WgDw== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman , Ashok Raj Subject: [patch v3 6/7] x86/smp: Split sending INIT IPI out into a helper function References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:33:58 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Putting CPUs into INIT is a safer place during kexec() to park CPUs. Split the INIT assert/deassert sequence out so it can be reused. Signed-off-by: Thomas Gleixner Reviewed-by: Ashok Raj --- V2: Fix rebase screwup --- arch/x86/kernel/smpboot.c | 49 ++++++++++++++++++-----------------------= ----- 1 file changed, 20 insertions(+), 29 deletions(-) --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -853,47 +853,38 @@ wakeup_secondary_cpu_via_nmi(int apicid, return (send_status | accept_status); } =20 -static int -wakeup_secondary_cpu_via_init(int phys_apicid, unsigned long start_eip) +static void send_init_sequence(int phys_apicid) { - unsigned long send_status =3D 0, accept_status =3D 0; - int maxlvt, num_starts, j; - - maxlvt =3D lapic_get_maxlvt(); + int maxlvt =3D lapic_get_maxlvt(); =20 - /* - * Be paranoid about clearing APIC errors. - */ + /* Be paranoid about clearing APIC errors. */ if (APIC_INTEGRATED(boot_cpu_apic_version)) { - if (maxlvt > 3) /* Due to the Pentium erratum 3AP. */ + /* Due to the Pentium erratum 3AP. */ + if (maxlvt > 3) apic_write(APIC_ESR, 0); apic_read(APIC_ESR); } =20 - pr_debug("Asserting INIT\n"); - - /* - * Turn INIT on target chip - */ - /* - * Send IPI - */ - apic_icr_write(APIC_INT_LEVELTRIG | APIC_INT_ASSERT | APIC_DM_INIT, - phys_apicid); - - pr_debug("Waiting for send to finish...\n"); - send_status =3D safe_apic_wait_icr_idle(); + /* Assert INIT on the target CPU */ + apic_icr_write(APIC_INT_LEVELTRIG | APIC_INT_ASSERT | APIC_DM_INIT, phys_= apicid); + safe_apic_wait_icr_idle(); =20 udelay(init_udelay); =20 - pr_debug("Deasserting INIT\n"); - - /* Target chip */ - /* Send IPI */ + /* Deassert INIT on the target CPU */ apic_icr_write(APIC_INT_LEVELTRIG | APIC_DM_INIT, phys_apicid); + safe_apic_wait_icr_idle(); +} + +/* + * Wake up AP by INIT, INIT, STARTUP sequence. + */ +static int wakeup_secondary_cpu_via_init(int phys_apicid, unsigned long st= art_eip) +{ + unsigned long send_status =3D 0, accept_status =3D 0; + int num_starts, j, maxlvt =3D lapic_get_maxlvt(); =20 - pr_debug("Waiting for send to finish...\n"); - send_status =3D safe_apic_wait_icr_idle(); + send_init_sequence(phys_apicid); =20 mb(); From nobody Sun Feb 8 09:10:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 826FDEB64DB for ; Thu, 15 Jun 2023 20:34:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238023AbjFOUeV (ORCPT ); Thu, 15 Jun 2023 16:34:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230219AbjFOUeE (ORCPT ); Thu, 15 Jun 2023 16:34:04 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DDA32726 for ; Thu, 15 Jun 2023 13:34:02 -0700 (PDT) Message-ID: <20230615193330.608657211@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1686861241; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=sB20Kb5T4u+9i61NvPvEsJiORwRSTA74S2zqtl199PY=; b=BaHhWLU2EShAROun32+ROuKfZitFcUI5v3U7bROdAZZ+ckMrySgLc6wkjRD9eMH/aRZymJ Nqhwos4Mm+AzCezgDOxqtjrLZIkwk77SeZceGfXdBeB5Bz2owgmnkxeIZZKR5CRUaktjRx vsdx3FEXH1wleZ9nHyTAdZsKxEe1h46OHVvifivtS7H2ZxNimLANOPFDsNz7r5ofpXafva 7k4HcrkVQim8S8pJbI7EoESkdShYAkJlZ0AWnrv2zoNfSvD6T4nb0At6k20gC/M0YRHm4m 32SoFhkjwOM6qjDr+NYT0+FJ2l+yV+udh3/8REXIkmtiCr68V4eOGYfHhx2JJQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1686861241; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=sB20Kb5T4u+9i61NvPvEsJiORwRSTA74S2zqtl199PY=; b=g4FohfePyTHWmZB41seYYWB41AN9TC1rgqtGNVATbK2B6nrMKeBT3NOlKk72yoPJY5Y5H5 fHJgAOsNIAGRrmAA== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Mario Limonciello , Tom Lendacky , Tony Battersby , Ashok Raj , Tony Luck , Arjan van de Veen , Eric Biederman , Ashok Raj Subject: [patch v3 7/7] x86/smp: Put CPUs into INIT on shutdown if possible References: <20230615190036.898273129@linutronix.de> MIME-Version: 1.0 Date: Thu, 15 Jun 2023 22:34:00 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Parking CPUs in a HLT loop is not completely safe vs. kexec() as HLT can resume execution due to NMI, SMI and MCE, which has the same issue as the MWAIT loop. Kicking the secondary CPUs into INIT makes this safe against NMI and SMI. A broadcast MCE will take the machine down, but a broadcast MCE which makes HLT resume and execute overwritten text, pagetables or data will end up in a disaster too. So chose the lesser of two evils and kick the secondary CPUs into INIT unless the system has installed special wakeup mechanisms which are not using INIT. Signed-off-by: Thomas Gleixner Reviewed-by: Ashok Raj Reviewed-by: Borislav Petkov (AMD) --- V3: Renamed the function to smp_park_other_cpus_in_init() so it can be reused for crash eventually. --- arch/x86/include/asm/smp.h | 2 ++ arch/x86/kernel/smp.c | 39 ++++++++++++++++++++++++++++++++------- arch/x86/kernel/smpboot.c | 19 +++++++++++++++++++ 3 files changed, 53 insertions(+), 7 deletions(-) --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -139,6 +139,8 @@ void native_send_call_func_ipi(const str void native_send_call_func_single_ipi(int cpu); void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle); =20 +bool smp_park_other_cpus_in_init(void); + void smp_store_boot_cpu_info(void); void smp_store_cpu_info(int id); =20 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -131,7 +131,7 @@ static int smp_stop_nmi_callback(unsigne } =20 /* - * this function calls the 'stop' function on all other CPUs in the system. + * Disable virtualization, APIC etc. and park the CPU in a HLT loop */ DEFINE_IDTENTRY_SYSVEC(sysvec_reboot) { @@ -172,13 +172,17 @@ static void native_stop_other_cpus(int w * 2) Wait for all other CPUs to report that they reached the * HLT loop in stop_this_cpu() * - * 3) If #2 timed out send an NMI to the CPUs which did not - * yet report + * 3) If the system uses INIT/STARTUP for CPU bringup, then + * send all present CPUs an INIT vector, which brings them + * completely out of the way. * - * 4) Wait for all other CPUs to report that they reached the + * 4) If #3 is not possible and #2 timed out send an NMI to the + * CPUs which did not yet report + * + * 5) Wait for all other CPUs to report that they reached the * HLT loop in stop_this_cpu() * - * #3 can obviously race against a CPU reaching the HLT loop late. + * #4 can obviously race against a CPU reaching the HLT loop late. * That CPU will have reported already and the "have all CPUs * reached HLT" condition will be true despite the fact that the * other CPU is still handling the NMI. Again, there is no @@ -194,7 +198,7 @@ static void native_stop_other_cpus(int w /* * Don't wait longer than a second for IPI completion. The * wait request is not checked here because that would - * prevent an NMI shutdown attempt in case that not all + * prevent an NMI/INIT shutdown in case that not all * CPUs reach shutdown state. */ timeout =3D USEC_PER_SEC; @@ -202,7 +206,27 @@ static void native_stop_other_cpus(int w udelay(1); } =20 - /* if the REBOOT_VECTOR didn't work, try with the NMI */ + /* + * Park all other CPUs in INIT including "offline" CPUs, if + * possible. That's a safe place where they can't resume execution + * of HLT and then execute the HLT loop from overwritten text or + * page tables. + * + * The only downside is a broadcast MCE, but up to the point where + * the kexec() kernel brought all APs online again an MCE will just + * make HLT resume and handle the MCE. The machine crashs and burns + * due to overwritten text, page tables and data. So there is a + * choice between fire and frying pan. The result is pretty much + * the same. Chose frying pan until x86 provides a sane mechanism + * to park a CPU. + */ + if (smp_park_other_cpus_in_init()) + goto done; + + /* + * If park with INIT was not possible and the REBOOT_VECTOR didn't + * take all secondary CPUs offline, try with the NMI. + */ if (!cpumask_empty(&cpus_stop_mask)) { /* * If NMI IPI is enabled, try to register the stop handler @@ -234,6 +258,7 @@ static void native_stop_other_cpus(int w udelay(1); } =20 +done: local_irq_save(flags); disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1465,6 +1465,25 @@ void arch_thaw_secondary_cpus_end(void) cache_aps_init(); } =20 +bool smp_park_other_cpus_in_init(void) +{ + unsigned int cpu, this_cpu =3D smp_processor_id(); + unsigned int apicid; + + if (apic->wakeup_secondary_cpu_64 || apic->wakeup_secondary_cpu) + return false; + + for_each_present_cpu(cpu) { + if (cpu =3D=3D this_cpu) + continue; + apicid =3D apic->cpu_present_to_apicid(cpu); + if (apicid =3D=3D BAD_APICID) + continue; + send_init_sequence(apicid); + } + return true; +} + /* * Early setup to make printk work. */