[PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup

Sean Christopherson posted 16 patches 1 month, 2 weeks ago
Documentation/arch/x86/tdx.rst              |  36 +-
arch/x86/events/intel/pt.c                  |   1 -
arch/x86/include/asm/kvm_host.h             |   3 +-
arch/x86/include/asm/reboot.h               |  11 -
arch/x86/include/asm/tdx.h                  |   4 -
arch/x86/include/asm/virt.h                 |  26 ++
arch/x86/include/asm/vmx.h                  |  11 +
arch/x86/kernel/cpu/common.c                |   2 +
arch/x86/kernel/crash.c                     |   3 +-
arch/x86/kernel/reboot.c                    |  63 +---
arch/x86/kernel/smp.c                       |   5 +-
arch/x86/kvm/svm/svm.c                      |  34 +-
arch/x86/kvm/svm/vmenter.S                  |  10 +-
arch/x86/kvm/vmx/main.c                     |  19 +-
arch/x86/kvm/vmx/tdx.c                      | 210 ++----------
arch/x86/kvm/vmx/tdx.h                      |   8 +-
arch/x86/kvm/vmx/vmcs.h                     |  11 -
arch/x86/kvm/vmx/vmenter.S                  |   2 +-
arch/x86/kvm/vmx/vmx.c                      | 138 +-------
arch/x86/kvm/x86.c                          |  29 +-
arch/x86/virt/Makefile                      |   2 +
arch/x86/virt/hw.c                          | 359 ++++++++++++++++++++
arch/x86/virt/vmx/tdx/tdx.c                 | 321 +++++++++--------
arch/x86/virt/vmx/tdx/tdx.h                 |   8 -
arch/x86/virt/vmx/tdx/tdx_global_metadata.c |  10 +-
include/linux/kvm_host.h                    |  16 +-
virt/kvm/kvm_main.c                         |  31 +-
27 files changed, 717 insertions(+), 656 deletions(-)
create mode 100644 arch/x86/include/asm/virt.h
create mode 100644 arch/x86/virt/hw.c
[PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Sean Christopherson 1 month, 2 weeks ago
Assuming I didn't break anything between v2 and v3, I think this is ready to
rip.  Given the scope of the KVM changes, and that they extend outside of x86,
my preference is to take this through the KVM tree.  But a stable topic branch
in tip would work too, though I think we'd want it sooner than later so that
it can be used as a base. 

Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
while splitting up the main patch that I'm not 100% positive I didn't regress
anything relative to v2.


The idea here is to extract _only_ VMXON+VMXOFF and EFER.SVME toggling.  AFAIK
there's no second user of SVM, i.e. no equivalent to TDX, but I wanted to keep
things as symmetrical as possible.

TDX isn't a hypervisor, and isn't trying to be a hypervisor. Specifically, TDX
should _never_ have it's own VMCSes (that are visible to the host; the
TDX-Module has it's own VMCSes to do SEAMCALL/SEAMRET), and so there is simply
no reason to move that functionality out of KVM.

With that out of the way, dealing with VMXON/VMXOFF and EFER.SVME is a fairly
simple refcounting game.

v3:
 - https://lore.kernel.org/all/20251206011054.494190-1-seanjc@google.com
 - Split up the move from KVM => virt into smaller patches. [Dan]
 - Collect reviews. [Dan, Chao, Dave]
 - Update sample dmesg output and hotplug angle in docs. [Chao]
 - Add comments in kvm_arch_shutdown() to try and explain the madness. [Dave]
 - Add a largely superfluous smp_wmb() in kvm_arch_shutdown() to provide a
   convienent location for documenting the flow. [Dave]
 - Disable preemption in x86_virt_{get,put}_ref() so that changes in how
   KVM and/or TDX use the APIs doesn't result in bugs. [Xu]
 - Add a patch to drop the bogus "IRQs must be disabled" rule in
   tdx_cpu_enable().
 - Tag more TDX helpers as __init. [Chao]
 - Don't treat loading kvm-intel.ko with tdx=1 as fatal if the system doesn't
   have a TDX-Module available. [Chao]

v2:
 - Initialize the TDX-Module via subsys initcall instead of during
   tdx_init(). [Rick]
 - Isolate the __init and __ro_after_init changes. [Rick]
 - Use ida_is_empty() instead of manually tracking HKID usage. [Dan]
 - Don't do weird things with the refcounts when virt_rebooting is
   true. [Chao]
 - Drop unnecessary setting of virt_rebooting in KVM code. [Chao]
 - Rework things to have less X86_FEATURE_FOO code. [Rick]
 - Consolidate the CPU hotplug callbacks. [Chao]

v1 (RFC):
 - https://lore.kernel.org/all/20251010220403.987927-1-seanjc@google.com

Chao Gao (1):
  x86/virt/tdx: KVM: Consolidate TDX CPU hotplug handling

Sean Christopherson (15):
  KVM: x86: Move kvm_rebooting to x86
  KVM: VMX: Move architectural "vmcs" and "vmcs_hdr" structures to
    public vmx.h
  KVM: x86: Move "kvm_rebooting" to kernel as "virt_rebooting"
  KVM: VMX: Unconditionally allocate root VMCSes during boot CPU bringup
  x86/virt: Force-clear X86_FEATURE_VMX if configuring root VMCS fails
  KVM: VMX: Move core VMXON enablement to kernel
  KVM: SVM: Move core EFER.SVME enablement to kernel
  KVM: x86: Move bulk of emergency virtualizaton logic to virt subsystem
  x86/virt: Add refcounting of VMX/SVM usage to support multiple
    in-kernel users
  x86/virt/tdx: Drop the outdated requirement that TDX be enabled in IRQ
    context
  KVM: x86/tdx: Do VMXON and TDX-Module initialization during subsys
    init
  x86/virt/tdx: Tag a pile of functions as __init, and globals as
    __ro_after_init
  x86/virt/tdx: Use ida_is_empty() to detect if any TDs may be running
  KVM: Bury kvm_{en,dis}able_virtualization() in kvm_main.c once more
  KVM: TDX: Fold tdx_bringup() into tdx_hardware_setup()

 Documentation/arch/x86/tdx.rst              |  36 +-
 arch/x86/events/intel/pt.c                  |   1 -
 arch/x86/include/asm/kvm_host.h             |   3 +-
 arch/x86/include/asm/reboot.h               |  11 -
 arch/x86/include/asm/tdx.h                  |   4 -
 arch/x86/include/asm/virt.h                 |  26 ++
 arch/x86/include/asm/vmx.h                  |  11 +
 arch/x86/kernel/cpu/common.c                |   2 +
 arch/x86/kernel/crash.c                     |   3 +-
 arch/x86/kernel/reboot.c                    |  63 +---
 arch/x86/kernel/smp.c                       |   5 +-
 arch/x86/kvm/svm/svm.c                      |  34 +-
 arch/x86/kvm/svm/vmenter.S                  |  10 +-
 arch/x86/kvm/vmx/main.c                     |  19 +-
 arch/x86/kvm/vmx/tdx.c                      | 210 ++----------
 arch/x86/kvm/vmx/tdx.h                      |   8 +-
 arch/x86/kvm/vmx/vmcs.h                     |  11 -
 arch/x86/kvm/vmx/vmenter.S                  |   2 +-
 arch/x86/kvm/vmx/vmx.c                      | 138 +-------
 arch/x86/kvm/x86.c                          |  29 +-
 arch/x86/virt/Makefile                      |   2 +
 arch/x86/virt/hw.c                          | 359 ++++++++++++++++++++
 arch/x86/virt/vmx/tdx/tdx.c                 | 321 +++++++++--------
 arch/x86/virt/vmx/tdx/tdx.h                 |   8 -
 arch/x86/virt/vmx/tdx/tdx_global_metadata.c |  10 +-
 include/linux/kvm_host.h                    |  16 +-
 virt/kvm/kvm_main.c                         |  31 +-
 27 files changed, 717 insertions(+), 656 deletions(-)
 create mode 100644 arch/x86/include/asm/virt.h
 create mode 100644 arch/x86/virt/hw.c


base-commit: 183bb0ce8c77b0fd1fb25874112bc8751a461e49
-- 
2.53.0.310.g728cabbaf7-goog
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Sagi Shahar 4 weeks, 1 day ago
On Fri, Feb 13, 2026 at 7:27 PM Sean Christopherson <seanjc@google.com> wrote:
>
> Assuming I didn't break anything between v2 and v3, I think this is ready to
> rip.  Given the scope of the KVM changes, and that they extend outside of x86,
> my preference is to take this through the KVM tree.  But a stable topic branch
> in tip would work too, though I think we'd want it sooner than later so that
> it can be used as a base.
>
> Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
> while splitting up the main patch that I'm not 100% positive I didn't regress
> anything relative to v2.

Tested running TDs and TDX module update using "Runtime TDX Module
update support" patches [1]
Tested-by: Sagi Shahar <sagishah@gmail.com>

[1] https://lore.kernel.org/lkml/20260123145645.90444-1-chao.gao@intel.com/
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Sagi Shahar 4 weeks, 1 day ago
On Tue, Mar 3, 2026 at 3:39 PM Sagi Shahar <sagis@google.com> wrote:
>
> On Fri, Feb 13, 2026 at 7:27 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > Assuming I didn't break anything between v2 and v3, I think this is ready to
> > rip.  Given the scope of the KVM changes, and that they extend outside of x86,
> > my preference is to take this through the KVM tree.  But a stable topic branch
> > in tip would work too, though I think we'd want it sooner than later so that
> > it can be used as a base.
> >
> > Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
> > while splitting up the main patch that I'm not 100% positive I didn't regress
> > anything relative to v2.
>
> Tested running TDs and TDX module update using "Runtime TDX Module
> update support" patches [1]
> Tested-by: Sagi Shahar <sagishah@gmail.com>
>
> [1] https://lore.kernel.org/lkml/20260123145645.90444-1-chao.gao@intel.com/

Actually, looking at the "Runtime TDX Module update support" patches I
don't think I ran those with this version of the patches since the
"tdx_module_status" changes are incompatible. So it was just the
patches in this patchset.
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Sean Christopherson 4 weeks ago
On Fri, 13 Feb 2026 17:26:46 -0800, Sean Christopherson wrote:
> Assuming I didn't break anything between v2 and v3, I think this is ready to
> rip.  Given the scope of the KVM changes, and that they extend outside of x86,
> my preference is to take this through the KVM tree.  But a stable topic branch
> in tip would work too, though I think we'd want it sooner than later so that
> it can be used as a base.
> 
> Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
> while splitting up the main patch that I'm not 100% positive I didn't regress
> anything relative to v2.
> 
> [...]

Applied to kvm-x86 vmxon, with the minor fixups.  I'll make sure not to touch
the hashes at this point, but holler if anyone wants an "official" stable tag.

[01/16] KVM: x86: Move kvm_rebooting to x86
        https://github.com/kvm-x86/linux/commit/4059172b2a78
[02/16] KVM: VMX: Move architectural "vmcs" and "vmcs_hdr" structures to public vmx.h
        https://github.com/kvm-x86/linux/commit/3c75e6a5da3c
[03/16] KVM: x86: Move "kvm_rebooting" to kernel as "virt_rebooting"
        https://github.com/kvm-x86/linux/commit/a1450a8156c6
[04/16] KVM: VMX: Unconditionally allocate root VMCSes during boot CPU bringup
        https://github.com/kvm-x86/linux/commit/405b7c27934e
[05/16] x86/virt: Force-clear X86_FEATURE_VMX if configuring root VMCS fails
        https://github.com/kvm-x86/linux/commit/95e4adb24ff6
[06/16] KVM: VMX: Move core VMXON enablement to kernel
        https://github.com/kvm-x86/linux/commit/920da4f75519
[07/16] KVM: SVM: Move core EFER.SVME enablement to kernel
        https://github.com/kvm-x86/linux/commit/32d76cdfa122
[08/16] KVM: x86: Move bulk of emergency virtualizaton logic to virt subsystem
        https://github.com/kvm-x86/linux/commit/428afac5a8ea
[09/16] x86/virt: Add refcounting of VMX/SVM usage to support multiple in-kernel users
        https://github.com/kvm-x86/linux/commit/8528a7f9c91d
[10/16] x86/virt/tdx: Drop the outdated requirement that TDX be enabled in IRQ context
        https://github.com/kvm-x86/linux/commit/0efe5dc16169
[11/16] KVM: x86/tdx: Do VMXON and TDX-Module initialization during subsys init
        https://github.com/kvm-x86/linux/commit/165e77353831
[12/16] x86/virt/tdx: Tag a pile of functions as __init, and globals as __ro_after_init
        https://github.com/kvm-x86/linux/commit/9900400e20c0
[13/16] x86/virt/tdx: KVM: Consolidate TDX CPU hotplug handling
        https://github.com/kvm-x86/linux/commit/eac90a5ba0aa
[14/16] x86/virt/tdx: Use ida_is_empty() to detect if any TDs may be running
        https://github.com/kvm-x86/linux/commit/afe31de159bf
[15/16] KVM: Bury kvm_{en,dis}able_virtualization() in kvm_main.c once more
        https://github.com/kvm-x86/linux/commit/d30372d0b7e6
[16/16] KVM: TDX: Fold tdx_bringup() into tdx_hardware_setup()
        https://github.com/kvm-x86/linux/commit/f630de1f8d70

--
https://github.com/kvm-x86/linux/tree/next
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by dan.j.williams@intel.com 4 weeks ago
Sean Christopherson wrote:
> On Fri, 13 Feb 2026 17:26:46 -0800, Sean Christopherson wrote:
> > Assuming I didn't break anything between v2 and v3, I think this is ready to
> > rip.  Given the scope of the KVM changes, and that they extend outside of x86,
> > my preference is to take this through the KVM tree.  But a stable topic branch
> > in tip would work too, though I think we'd want it sooner than later so that
> > it can be used as a base.
> > 
> > Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
> > while splitting up the main patch that I'm not 100% positive I didn't regress
> > anything relative to v2.
> > 
> > [...]
> 
> Applied to kvm-x86 vmxon, with the minor fixups.  I'll make sure not to touch
> the hashes at this point, but holler if anyone wants an "official" stable tag.

Thanks, Sean!

Please do make an official stable tag that I can use for coordinating
the initial TDX Connect enabling series. While there is no strict
dependency I do not want it to be the case that a bisect of TDX Connect
bounces between a world where you need to load kvm_intel before the PCI
layer can do link encryption operations and keep it loaded etc.

My proposal, unless you or Dave holler, is to take the first round of
TDX Connect enabling through the tsm.git tree with acks. This round does
not have kvm entanglements, i.e. IOMMU coordination and device
assignment come later. It also does not have much in the way of core x86
entanglements beyond new seamcall exports.
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Sean Christopherson 4 weeks ago
On Thu, Mar 05, 2026, dan.j.williams@intel.com wrote:
> Sean Christopherson wrote:
> > On Fri, 13 Feb 2026 17:26:46 -0800, Sean Christopherson wrote:
> > > Assuming I didn't break anything between v2 and v3, I think this is ready to
> > > rip.  Given the scope of the KVM changes, and that they extend outside of x86,
> > > my preference is to take this through the KVM tree.  But a stable topic branch
> > > in tip would work too, though I think we'd want it sooner than later so that
> > > it can be used as a base.
> > > 
> > > Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
> > > while splitting up the main patch that I'm not 100% positive I didn't regress
> > > anything relative to v2.
> > > 
> > > [...]
> > 
> > Applied to kvm-x86 vmxon, with the minor fixups.  I'll make sure not to touch
> > the hashes at this point, but holler if anyone wants an "official" stable tag.
> 
> Thanks, Sean!
> 
> Please do make an official stable tag that I can use for coordinating
> the initial TDX Connect enabling series. While there is no strict
> dependency I do not want it to be the case that a bisect of TDX Connect
> bounces between a world where you need to load kvm_intel before the PCI
> layer can do link encryption operations and keep it loaded etc.

With a timestamp, in case fixups on top are needed:

kvm-x86-vmxon-2026.03.05
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Dave Hansen 4 weeks ago
On 3/5/26 10:50, dan.j.williams@intel.com wrote:
> My proposal, unless you or Dave holler, is to take the first round of
> TDX Connect enabling through the tsm.git tree with acks. This round does
> not have kvm entanglements, i.e. IOMMU coordination and device
> assignment come later. It also does not have much in the way of core x86
> entanglements beyond new seamcall exports.

Sounds sane to me.
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Sean Christopherson 4 weeks ago
On Thu, Mar 05, 2026, Dave Hansen wrote:
> On 3/5/26 10:50, dan.j.williams@intel.com wrote:
> > My proposal, unless you or Dave holler, is to take the first round of
> > TDX Connect enabling through the tsm.git tree with acks. This round does
> > not have kvm entanglements, i.e. IOMMU coordination and device
> > assignment come later. It also does not have much in the way of core x86
> > entanglements beyond new seamcall exports.
> 
> Sounds sane to me.

+1.  If there aren't any KVM changes, ignorance is bliss :-)
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by Chao Gao 1 month ago
On Fri, Feb 13, 2026 at 05:26:46PM -0800, Sean Christopherson wrote:
>Assuming I didn't break anything between v2 and v3, I think this is ready to
>rip.  Given the scope of the KVM changes, and that they extend outside of x86,
>my preference is to take this through the KVM tree.  But a stable topic branch
>in tip would work too, though I think we'd want it sooner than later so that
>it can be used as a base. 
>
>Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
>while splitting up the main patch that I'm not 100% positive I didn't regress
>anything relative to v2.

I tested CPU hotplug/unplug, kvm-intel.ko loading/reloading, TD launches, and
loading kvm-intel.ko with tdx=1 when the TDX Module wasn't loaded. No issues
were found with this v3.

Tested-by: Chao Gao <chao.gao@intel.com>
Re: [PATCH v3 00/16] KVM: x86/tdx: Have TDX handle VMXON during bringup
Posted by dan.j.williams@intel.com 1 month, 2 weeks ago
Sean Christopherson wrote:
> Assuming I didn't break anything between v2 and v3, I think this is ready to
> rip.  Given the scope of the KVM changes, and that they extend outside of x86,
> my preference is to take this through the KVM tree.  But a stable topic branch
> in tip would work too, though I think we'd want it sooner than later so that
> it can be used as a base. 
> 
> Chao, I deliberately omitted your Tested-by, as I shuffled things around enough
> while splitting up the main patch that I'm not 100% positive I didn't regress
> anything relative to v2.
> 
> 
> The idea here is to extract _only_ VMXON+VMXOFF and EFER.SVME toggling.  AFAIK
> there's no second user of SVM, i.e. no equivalent to TDX, but I wanted to keep
> things as symmetrical as possible.
> 
> TDX isn't a hypervisor, and isn't trying to be a hypervisor. Specifically, TDX
> should _never_ have it's own VMCSes (that are visible to the host; the
> TDX-Module has it's own VMCSes to do SEAMCALL/SEAMRET), and so there is simply
> no reason to move that functionality out of KVM.
> 
> With that out of the way, dealing with VMXON/VMXOFF and EFER.SVME is a fairly
> simple refcounting game.
> 
> v3:
>  - https://lore.kernel.org/all/20251206011054.494190-1-seanjc@google.com
>  - Split up the move from KVM => virt into smaller patches. [Dan]
>  - Collect reviews. [Dan, Chao, Dave]
>  - Update sample dmesg output and hotplug angle in docs. [Chao]
>  - Add comments in kvm_arch_shutdown() to try and explain the madness. [Dave]
>  - Add a largely superfluous smp_wmb() in kvm_arch_shutdown() to provide a
>    convienent location for documenting the flow. [Dave]
>  - Disable preemption in x86_virt_{get,put}_ref() so that changes in how
>    KVM and/or TDX use the APIs doesn't result in bugs. [Xu]
>  - Add a patch to drop the bogus "IRQs must be disabled" rule in
>    tdx_cpu_enable().
>  - Tag more TDX helpers as __init. [Chao]
>  - Don't treat loading kvm-intel.ko with tdx=1 as fatal if the system doesn't
>    have a TDX-Module available. [Chao]

I went through the rest of the patches, the finer grained splits make
sense. No significant concerns, so for the series:

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

...I expect Chao or Yilun to have a chance to offer a Tested-by per your
comment above.