From nobody Mon Apr 6 09:43:47 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98A0EC54EE9 for ; Thu, 8 Sep 2022 23:26:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230246AbiIHX0l (ORCPT ); Thu, 8 Sep 2022 19:26:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229950AbiIHX0J (ORCPT ); Thu, 8 Sep 2022 19:26:09 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA9E5E72FF; Thu, 8 Sep 2022 16:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662679567; x=1694215567; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bjjM/7k9H5/EDjEZ2WY39fcLYpzeVRwTDof3/tbSHpk=; b=IXTAtLe4dySUGgYhICka1Zd0CYknqP/g3qCnkg6L7Q73Osa3oElQWqNW D4C/ASiord0eeNIohsrtgXsfruPWbfstQK6EmLDuHoZHpPDQ0GMoSPUct DNe9+277v54ePSkuMcg4le9XWlNl9nsxp8dP2hDbu4t/BfQP+XpRIlNPh vHO0yvQPRfO19CAyxJ+Ghj4yyVIgkn4nRAV/5sn8ODVviZdFhrGsdctR6 lIN6g2GuXE5kmQRViNaSuAEhA4YEtottCrrJTTXQHFRF6+ch8+tIpt20k RjgV2KXKK3wXEg7dbeKniPnpk8CYav0N0DXf6ooc54j+wWUTo6XWqu/cz A==; X-IronPort-AV: E=McAfee;i="6500,9779,10464"; a="298686991" X-IronPort-AV: E=Sophos;i="5.93,300,1654585200"; d="scan'208";a="298686991" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Sep 2022 16:26:06 -0700 X-IronPort-AV: E=Sophos;i="5.93,300,1654585200"; d="scan'208";a="610863180" Received: from ls.sc.intel.com (HELO localhost) ([143.183.96.54]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Sep 2022 16:26:06 -0700 From: isaku.yamahata@intel.com To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Paolo Bonzini , Sean Christopherson , Marc Zyngier , Will Deacon , Yuan Yao Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Kai Huang , Chao Gao , Atish Patra , Shaokun Zhang , Qi Liu , John Garry , Daniel Lezcano , Huang Ying , Huacai Chen , Thomas Gleixner , Yuan Yao Subject: [PATCH v4 07/26] KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section Date: Thu, 8 Sep 2022 16:25:23 -0700 Message-Id: <1d7b1068e05e4fbbeeac009fe1795c4c0170db21.1662679124.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Chao Gao The CPU STARTING section doesn't allow callbacks to fail. Move KVM's hotplug callback to ONLINE section so that it can abort onlining a CPU in certain cases to avoid potentially breaking VMs running on existing CPUs. For example, when kvm fails to enable hardware virtualization on the hotplugged CPU. Place KVM's hotplug state before CPUHP_AP_SCHED_WAIT_EMPTY as it ensures when offlining a CPU, all user tasks and non-pinned kernel tasks have left the CPU, i.e. there cannot be a vCPU task around. So, it is safe for KVM's CPU offline callback to disable hardware virtualization at that point. Likewise, KVM's online callback can enable hardware virtualization before any vCPU task gets a chance to run on hotplugged CPUs. KVM's CPU hotplug callbacks are renamed as well. Suggested-by: Thomas Gleixner Signed-off-by: Chao Gao Link: https://lore.kernel.org/r/20220216031528.92558-6-chao.gao@intel.com Reviewed-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Yuan Yao --- include/linux/cpuhotplug.h | 2 +- virt/kvm/kvm_main.c | 30 ++++++++++++++++++++++-------- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 7337414e4947..de45be38dd27 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -185,7 +185,6 @@ enum cpuhp_state { CPUHP_AP_CSKY_TIMER_STARTING, CPUHP_AP_TI_GP_TIMER_STARTING, CPUHP_AP_HYPERV_TIMER_STARTING, - CPUHP_AP_KVM_STARTING, /* Must be the last timer callback */ CPUHP_AP_DUMMY_TIMER_STARTING, CPUHP_AP_ARM_XEN_STARTING, @@ -200,6 +199,7 @@ enum cpuhp_state { =20 /* Online section invoked on the hotplugged CPU from the hotplug thread */ CPUHP_AP_ONLINE_IDLE, + CPUHP_AP_KVM_ONLINE, CPUHP_AP_SCHED_WAIT_EMPTY, CPUHP_AP_SMPBOOT_THREADS, CPUHP_AP_X86_VDSO_VMA_ONLINE, diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 278eb6cc7cbe..db1303e2abc9 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5011,13 +5011,27 @@ static void hardware_enable_nolock(void *caller_nam= e) } } =20 -static int kvm_starting_cpu(unsigned int cpu) +static int kvm_online_cpu(unsigned int cpu) { + int ret =3D 0; + raw_spin_lock(&kvm_count_lock); - if (kvm_usage_count) + /* + * Abort the CPU online process if hardware virtualization cannot + * be enabled. Otherwise running VMs would encounter unrecoverable + * errors when scheduled to this CPU. + */ + if (kvm_usage_count) { + WARN_ON_ONCE(atomic_read(&hardware_enable_failed)); + hardware_enable_nolock((void *)__func__); + if (atomic_read(&hardware_enable_failed)) { + atomic_set(&hardware_enable_failed, 0); + ret =3D -EIO; + } + } raw_spin_unlock(&kvm_count_lock); - return 0; + return ret; } =20 static void hardware_disable_nolock(void *junk) @@ -5030,7 +5044,7 @@ static void hardware_disable_nolock(void *junk) kvm_arch_hardware_disable(); } =20 -static int kvm_dying_cpu(unsigned int cpu) +static int kvm_offline_cpu(unsigned int cpu) { raw_spin_lock(&kvm_count_lock); if (kvm_usage_count) @@ -5841,8 +5855,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsign= ed vcpu_align, goto out_free_2; } =20 - r =3D cpuhp_setup_state_nocalls(CPUHP_AP_KVM_STARTING, "kvm/cpu:starting", - kvm_starting_cpu, kvm_dying_cpu); + r =3D cpuhp_setup_state_nocalls(CPUHP_AP_KVM_ONLINE, "kvm/cpu:online", + kvm_online_cpu, kvm_offline_cpu); if (r) goto out_free_2; register_reboot_notifier(&kvm_reboot_notifier); @@ -5903,7 +5917,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsign= ed vcpu_align, kmem_cache_destroy(kvm_vcpu_cache); out_free_3: unregister_reboot_notifier(&kvm_reboot_notifier); - cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING); + cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE); out_free_2: kvm_arch_hardware_unsetup(); out_free_1: @@ -5929,7 +5943,7 @@ void kvm_exit(void) kvm_async_pf_deinit(); unregister_syscore_ops(&kvm_syscore_ops); unregister_reboot_notifier(&kvm_reboot_notifier); - cpuhp_remove_state_nocalls(CPUHP_AP_KVM_STARTING); + cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE); on_each_cpu(hardware_disable_nolock, NULL, 1); kvm_arch_hardware_unsetup(); kvm_arch_exit(); --=20 2.25.1