Register KVM's cpuhp and syscore callbacks when enabling virtualization in
hardware, as the sole purpose of said callbacks is to disable and re-enable
virtualization as needed.

The primary motivation for this series is to simplify dealing with enabling
virtualization for Intel's TDX, which needs to enable virtualization
when kvm-intel.ko is loaded, i.e. long before the first VM is created.  TDX
doesn't _need_ to keep virtualization enabled, but doing so is much simpler
for KVM (see patch 3).

That said, this is a nice cleanup on its own, assuming I haven't broken
something.  By registering the callbacks on-demand, the callbacks themselves
don't need to check kvm_usage_count, because their very existence implies a
non-zero count.

The meat is in patch 1.  Patches 2 renames the helpers so that patch 3 is
less awkward.  Patch 3 adds a module param to enable virtualization when KVM
is loaded.  Patches 4-6 are tangentially related x86 cleanups to registers
KVM's "emergency disable" callback on-demand, same as the syscore callbacks.

The suspend/resume and cphup paths still need to be fully tested, as do
non-x86 architectures.

v3:
 - Collect reviews/acks.
 - Switch to kvm_usage_lock in a dedicated patch, Cc'd for stable@. [Chao]
 - Enable virt at load by default. [Chao]
 - Add comments to document how kvm_arch_{en,dis}able_virtualization() fit
   into the overall flow. [Kai]

v2:
 - https://lore.kernel.org/all/20240522022827.1690416-1-seanjc@google.com
 - Use a dedicated mutex to avoid lock inversion issues between kvm_lock and
   the cpuhp lock.
 - Register emergency disable callbacks on-demand. [Kai]
 - Drop an unintended s/junk/ign rename. [Kai]
 - Decrement kvm_usage_count on failure. [Chao]

v1: https://lore.kernel.org/all/20240425233951.3344485-1-seanjc@google.com

Sean Christopherson (8):
  KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
  KVM: Register cpuhp and syscore callbacks when enabling hardware
  KVM: Rename functions related to enabling virtualization hardware
  KVM: Add a module param to allow enabling virtualization when KVM is
    loaded
  KVM: Add arch hooks for enabling/disabling virtualization
  x86/reboot: Unconditionally define cpu_emergency_virt_cb typedef
  KVM: x86: Register "emergency disable" callbacks when virt is enabled
  KVM: Enable virtualization at load/initialization by default

 Documentation/virt/kvm/locking.rst |  19 ++-
 arch/x86/include/asm/kvm_host.h    |   3 +
 arch/x86/include/asm/reboot.h      |   2 +-
 arch/x86/kvm/svm/svm.c             |   5 +-
 arch/x86/kvm/vmx/main.c            |   2 +
 arch/x86/kvm/vmx/vmx.c             |   6 +-
 arch/x86/kvm/vmx/x86_ops.h         |   1 +
 arch/x86/kvm/x86.c                 |  10 ++
 include/linux/kvm_host.h           |  14 ++
 virt/kvm/kvm_main.c                | 258 ++++++++++++++---------------
 10 files changed, 175 insertions(+), 145 deletions(-)


base-commit: af0903ab52ee6d6f0f63af67fa73d5eb00f79b9a
-- 
2.45.2.505.gda0bf45e8d-goog

On Sat, Jun 8, 2024 at 2:06 AM Sean Christopherson <seanjc@google.com> wrote:
> The suspend/resume and cphup paths still need to be fully tested, as do
> non-x86 architectures.

You can add

Tested-by: Farrah Chen <farrah.chen@intel.com>

> For CPU hotplug we tested this:
>
> 1) offline some CPUs;
> 2) load kvm-intel.ko;
> 3) run a VM;
> 4) online those offlined CPUs;
> 5) offline those CPUs again;
> 6) online those CPUs again;
>
> All steps can be done successfully, and the VM run as expected during
> step 4) to 6).
>
> For suspend/resume we tested:
>
> 1) load kvm-intel.ko and run a VM;
> 2) suspend host
> 3) resume the host back (using the IPMI KVM console)
>
> All steps worked successfully, and the VM still run fine after resume.

Thanks Kai and Farrah :)

Paolo

On 6/8/24 02:06, Sean Christopherson wrote:
> Register KVM's cpuhp and syscore callbacks when enabling virtualization in
> hardware, as the sole purpose of said callbacks is to disable and re-enable
> virtualization as needed.
> 
> The primary motivation for this series is to simplify dealing with enabling
> virtualization for Intel's TDX, which needs to enable virtualization
> when kvm-intel.ko is loaded, i.e. long before the first VM is created.  TDX
> doesn't _need_ to keep virtualization enabled, but doing so is much simpler
> for KVM (see patch 3).
> 
> That said, this is a nice cleanup on its own, assuming I haven't broken
> something.  By registering the callbacks on-demand, the callbacks themselves
> don't need to check kvm_usage_count, because their very existence implies a
> non-zero count.
> 
> The meat is in patch 1.  Patches 2 renames the helpers so that patch 3 is
> less awkward.  Patch 3 adds a module param to enable virtualization when KVM
> is loaded.  Patches 4-6 are tangentially related x86 cleanups to registers
> KVM's "emergency disable" callback on-demand, same as the syscore callbacks.
> 
> The suspend/resume and cphup paths still need to be fully tested, as do
> non-x86 architectures.

Also placed in kvm/queue, mostly as a reminder to myself, and added 
other maintainers for testing on ARM, RISC-V and LoongArch.  The changes 
from v3 to v4 should be mostly nits, documentation and organization of 
the series.

Thanks,

Paolo

> v3:
>   - Collect reviews/acks.
>   - Switch to kvm_usage_lock in a dedicated patch, Cc'd for stable@. [Chao]
>   - Enable virt at load by default. [Chao]
>   - Add comments to document how kvm_arch_{en,dis}able_virtualization() fit
>     into the overall flow. [Kai]
> 
> v2:
>   - https://lore.kernel.org/all/20240522022827.1690416-1-seanjc@google.com
>   - Use a dedicated mutex to avoid lock inversion issues between kvm_lock and
>     the cpuhp lock.
>   - Register emergency disable callbacks on-demand. [Kai]
>   - Drop an unintended s/junk/ign rename. [Kai]
>   - Decrement kvm_usage_count on failure. [Chao]
> 
> v1: https://lore.kernel.org/all/20240425233951.3344485-1-seanjc@google.com
> 
> Sean Christopherson (8):
>    KVM: Use dedicated mutex to protect kvm_usage_count to avoid deadlock
>    KVM: Register cpuhp and syscore callbacks when enabling hardware
>    KVM: Rename functions related to enabling virtualization hardware
>    KVM: Add a module param to allow enabling virtualization when KVM is
>      loaded
>    KVM: Add arch hooks for enabling/disabling virtualization
>    x86/reboot: Unconditionally define cpu_emergency_virt_cb typedef
>    KVM: x86: Register "emergency disable" callbacks when virt is enabled
>    KVM: Enable virtualization at load/initialization by default
> 
>   Documentation/virt/kvm/locking.rst |  19 ++-
>   arch/x86/include/asm/kvm_host.h    |   3 +
>   arch/x86/include/asm/reboot.h      |   2 +-
>   arch/x86/kvm/svm/svm.c             |   5 +-
>   arch/x86/kvm/vmx/main.c            |   2 +
>   arch/x86/kvm/vmx/vmx.c             |   6 +-
>   arch/x86/kvm/vmx/x86_ops.h         |   1 +
>   arch/x86/kvm/x86.c                 |  10 ++
>   include/linux/kvm_host.h           |  14 ++
>   virt/kvm/kvm_main.c                | 258 ++++++++++++++---------------
>   10 files changed, 175 insertions(+), 145 deletions(-)
> 
> 
> base-commit: af0903ab52ee6d6f0f63af67fa73d5eb00f79b9a

> Also placed in kvm/queue, mostly as a reminder to myself, and added 
> other maintainers for testing on ARM, RISC-V and LoongArch.  The changes 
> from v3 to v4 should be mostly nits, documentation and organization of 
> the series.
> 

Also another reminder:

Could you also remove the WARN_ON() in kvm_uninit_virtualization() so 
that we can allow additional kvm_disable_virtualization() after that for 
TDX?

On Thu, Aug 15, 2024, Kai Huang wrote:
> 
> > Also placed in kvm/queue, mostly as a reminder to myself, and added
> > other maintainers for testing on ARM, RISC-V and LoongArch.  The changes
> > from v3 to v4 should be mostly nits, documentation and organization of
> > the series.
> > 
> 
> Also another reminder:
> 
> Could you also remove the WARN_ON() in kvm_uninit_virtualization() so that
> we can allow additional kvm_disable_virtualization() after that for TDX?

Yeah, I'll take care of that in v4 (as above, Paolo put this in kvm/queue as a
placeholder of sorts).

On Fri, 2024-06-07 at 17:06 -0700, Sean Christopherson wrote:
> Register KVM's cpuhp and syscore callbacks when enabling virtualization in
> hardware, as the sole purpose of said callbacks is to disable and re-enable
> virtualization as needed.
> 
> The primary motivation for this series is to simplify dealing with enabling
> virtualization for Intel's TDX, which needs to enable virtualization
> when kvm-intel.ko is loaded, i.e. long before the first VM is created.  TDX
> doesn't _need_ to keep virtualization enabled, but doing so is much simpler
> for KVM (see patch 3).
> 
> That said, this is a nice cleanup on its own, assuming I haven't broken
> something.  By registering the callbacks on-demand, the callbacks themselves
> don't need to check kvm_usage_count, because their very existence implies a
> non-zero count.
> 
> 

For this series,

Acked-by: Kai Huang <kai.huang@intel.com>