From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABE47C001DF for ; Tue, 25 Jul 2023 22:16:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231931AbjGYWQc (ORCPT ); Tue, 25 Jul 2023 18:16:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230484AbjGYWPl (ORCPT ); Tue, 25 Jul 2023 18:15:41 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC248100; Tue, 25 Jul 2023 15:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323335; x=1721859335; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bp6Ts+pJyhLHxnfqrElSIroiulHYpR89fukilxboH/8=; b=YQy9/IkVqW+g0C3y30/HgQ2YyMo8dsxrKN1iBtY3VmGlinh92gyKiKzf bpx8izgTLC6QxXECd/evm26HC78Tf54ogcNcZD6yIzzq/Glxk2dCElxlJ tcp1QxtlogPa088gqskqoVVL6dMj0+0R75/ZIuXBjdSNiEsRYJKv2GnF6 kITyGaaQPLrCOYTWaf3ehB4cQdtIBeX8ico1nYu9H1JIoFEc7FKztaCKr kzwr2NaS/7v8g1hou9aMPBuL8YZw/gzY4jbhp+xKuYOJjJs1Crh4kVL5T O/cQj0JK7K+p8c+w7l1ZnjynrKIKBAfHWqlTacL67GUrJI96PoU0JRsO+ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863015" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863015" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938765" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938765" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:14 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson , Xiaoyao Li Subject: [PATCH v15 001/115] KVM: VMX: Move out vmx_x86_ops to 'main.c' to wrap VMX and TDX Date: Tue, 25 Jul 2023 15:13:12 -0700 Message-Id: <86ae27e0addd06a245a1f72aaa5d8a7d08dda03b.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson KVM accesses Virtual Machine Control Structure (VMCS) with VMX instructions to operate on VM. TDX doesn't allow VMM to operate VMCS directly. Instead, TDX has its own data structures, and TDX SEAMCALL APIs for VMM to indirectly operate those data structures. This means we must have a TDX version of kvm_x86_ops. The existing global struct kvm_x86_ops already defines an interface which fits with TDX. But kvm_x86_ops is system-wide, not per-VM structure. To allow VMX to coexist with TDs, the kvm_x86_ops callbacks will have wrappers "if (tdx) tdx_op() else vmx_op()" to switch VMX or TDX at run time. To split the runtime switch, the VMX implementation, and the TDX implementation, add main.c, and move out the vmx_x86_ops hooks in preparation for adding TDX, which can coexist with VMX, i.e. KVM can run both VMs and TDs. Use 'vt' for the naming scheme as a nod to VT-x and as a concatenation of VmxTdx. The current code looks as follows. In vmx.c static vmx_op() { ... } static struct kvm_x86_ops vmx_x86_ops =3D { .op =3D vmx_op, initialization code The eventually converted code will look like In vmx.c, keep the VMX operations. vmx_op() { ... } VMX initialization In tdx.c, define the TDX operations. tdx_op() { ... } TDX initialization In x86_ops.h, declare the VMX and TDX operations. vmx_op(); tdx_op(); In main.c, define common wrappers for VMX and TDX. static vt_ops() { if (tdx) tdx_ops() else vmx_ops() } static struct kvm_x86_ops vt_x86_ops =3D { .op =3D vt_op, initialization to call VMX and TDX initialization Opportunistically, fix the name inconsistency from vmx_create_vcpu() and vmx_free_vcpu() to vmx_vcpu_create() and vxm_vcpu_free(). Co-developed-by: Xiaoyao Li Signed-off-by: Xiaoyao Li Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/vmx/main.c | 166 +++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 373 ++++++++++--------------------------- arch/x86/kvm/vmx/x86_ops.h | 126 +++++++++++++ 4 files changed, 395 insertions(+), 272 deletions(-) create mode 100644 arch/x86/kvm/vmx/main.c create mode 100644 arch/x86/kvm/vmx/x86_ops.h diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 80e3fe184d17..0e894ae23cbc 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -23,7 +23,7 @@ kvm-$(CONFIG_KVM_XEN) +=3D xen.o kvm-$(CONFIG_KVM_SMM) +=3D smm.o =20 kvm-intel-y +=3D vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ - vmx/hyperv.o vmx/nested.o vmx/posted_intr.o + vmx/hyperv.o vmx/nested.o vmx/posted_intr.o vmx/main.o kvm-intel-$(CONFIG_X86_SGX_KVM) +=3D vmx/sgx.o =20 kvm-amd-y +=3D svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \ diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c new file mode 100644 index 000000000000..a738ae96ca24 --- /dev/null +++ b/arch/x86/kvm/vmx/main.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +#include "x86_ops.h" +#include "vmx.h" +#include "nested.h" +#include "pmu.h" + +#define VMX_REQUIRED_APICV_INHIBITS \ + (BIT(APICV_INHIBIT_REASON_DISABLE)| \ + BIT(APICV_INHIBIT_REASON_ABSENT) | \ + BIT(APICV_INHIBIT_REASON_HYPERV) | \ + BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \ + BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \ + BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \ + BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED)) + +struct kvm_x86_ops vt_x86_ops __initdata =3D { + .name =3D KBUILD_MODNAME, + + .check_processor_compatibility =3D vmx_check_processor_compat, + + .hardware_unsetup =3D vmx_hardware_unsetup, + + .hardware_enable =3D vmx_hardware_enable, + .hardware_disable =3D vmx_hardware_disable, + .has_emulated_msr =3D vmx_has_emulated_msr, + + .is_vm_type_supported =3D vmx_is_vm_type_supported, + .vm_size =3D sizeof(struct kvm_vmx), + .vm_init =3D vmx_vm_init, + .vm_destroy =3D vmx_vm_destroy, + + .vcpu_precreate =3D vmx_vcpu_precreate, + .vcpu_create =3D vmx_vcpu_create, + .vcpu_free =3D vmx_vcpu_free, + .vcpu_reset =3D vmx_vcpu_reset, + + .prepare_switch_to_guest =3D vmx_prepare_switch_to_guest, + .vcpu_load =3D vmx_vcpu_load, + .vcpu_put =3D vmx_vcpu_put, + + .update_exception_bitmap =3D vmx_update_exception_bitmap, + .get_msr_feature =3D vmx_get_msr_feature, + .get_msr =3D vmx_get_msr, + .set_msr =3D vmx_set_msr, + .get_segment_base =3D vmx_get_segment_base, + .get_segment =3D vmx_get_segment, + .set_segment =3D vmx_set_segment, + .get_cpl =3D vmx_get_cpl, + .get_cs_db_l_bits =3D vmx_get_cs_db_l_bits, + .set_cr0 =3D vmx_set_cr0, + .is_valid_cr4 =3D vmx_is_valid_cr4, + .set_cr4 =3D vmx_set_cr4, + .set_efer =3D vmx_set_efer, + .get_idt =3D vmx_get_idt, + .set_idt =3D vmx_set_idt, + .get_gdt =3D vmx_get_gdt, + .set_gdt =3D vmx_set_gdt, + .set_dr7 =3D vmx_set_dr7, + .sync_dirty_debug_regs =3D vmx_sync_dirty_debug_regs, + .cache_reg =3D vmx_cache_reg, + .get_rflags =3D vmx_get_rflags, + .set_rflags =3D vmx_set_rflags, + .get_if_flag =3D vmx_get_if_flag, + + .flush_tlb_all =3D vmx_flush_tlb_all, + .flush_tlb_current =3D vmx_flush_tlb_current, + .flush_tlb_gva =3D vmx_flush_tlb_gva, + .flush_tlb_guest =3D vmx_flush_tlb_guest, + + .vcpu_pre_run =3D vmx_vcpu_pre_run, + .vcpu_run =3D vmx_vcpu_run, + .handle_exit =3D vmx_handle_exit, + .skip_emulated_instruction =3D vmx_skip_emulated_instruction, + .update_emulated_instruction =3D vmx_update_emulated_instruction, + .set_interrupt_shadow =3D vmx_set_interrupt_shadow, + .get_interrupt_shadow =3D vmx_get_interrupt_shadow, + .patch_hypercall =3D vmx_patch_hypercall, + .inject_irq =3D vmx_inject_irq, + .inject_nmi =3D vmx_inject_nmi, + .inject_exception =3D vmx_inject_exception, + .cancel_injection =3D vmx_cancel_injection, + .interrupt_allowed =3D vmx_interrupt_allowed, + .nmi_allowed =3D vmx_nmi_allowed, + .get_nmi_mask =3D vmx_get_nmi_mask, + .set_nmi_mask =3D vmx_set_nmi_mask, + .enable_nmi_window =3D vmx_enable_nmi_window, + .enable_irq_window =3D vmx_enable_irq_window, + .update_cr8_intercept =3D vmx_update_cr8_intercept, + .set_virtual_apic_mode =3D vmx_set_virtual_apic_mode, + .set_apic_access_page_addr =3D vmx_set_apic_access_page_addr, + .refresh_apicv_exec_ctrl =3D vmx_refresh_apicv_exec_ctrl, + .load_eoi_exitmap =3D vmx_load_eoi_exitmap, + .apicv_post_state_restore =3D vmx_apicv_post_state_restore, + .required_apicv_inhibits =3D VMX_REQUIRED_APICV_INHIBITS, + .hwapic_irr_update =3D vmx_hwapic_irr_update, + .hwapic_isr_update =3D vmx_hwapic_isr_update, + .guest_apic_has_interrupt =3D vmx_guest_apic_has_interrupt, + .sync_pir_to_irr =3D vmx_sync_pir_to_irr, + .deliver_interrupt =3D vmx_deliver_interrupt, + .dy_apicv_has_pending_interrupt =3D pi_has_pending_interrupt, + + .set_tss_addr =3D vmx_set_tss_addr, + .set_identity_map_addr =3D vmx_set_identity_map_addr, + .get_mt_mask =3D vmx_get_mt_mask, + + .get_exit_info =3D vmx_get_exit_info, + + .vcpu_after_set_cpuid =3D vmx_vcpu_after_set_cpuid, + + .has_wbinvd_exit =3D cpu_has_vmx_wbinvd_exit, + + .get_l2_tsc_offset =3D vmx_get_l2_tsc_offset, + .get_l2_tsc_multiplier =3D vmx_get_l2_tsc_multiplier, + .write_tsc_offset =3D vmx_write_tsc_offset, + .write_tsc_multiplier =3D vmx_write_tsc_multiplier, + + .load_mmu_pgd =3D vmx_load_mmu_pgd, + + .check_intercept =3D vmx_check_intercept, + .handle_exit_irqoff =3D vmx_handle_exit_irqoff, + + .request_immediate_exit =3D vmx_request_immediate_exit, + + .sched_in =3D vmx_sched_in, + + .cpu_dirty_log_size =3D PML_ENTITY_NUM, + .update_cpu_dirty_logging =3D vmx_update_cpu_dirty_logging, + + .nested_ops =3D &vmx_nested_ops, + + .pi_update_irte =3D vmx_pi_update_irte, + .pi_start_assignment =3D vmx_pi_start_assignment, + +#ifdef CONFIG_X86_64 + .set_hv_timer =3D vmx_set_hv_timer, + .cancel_hv_timer =3D vmx_cancel_hv_timer, +#endif + + .setup_mce =3D vmx_setup_mce, + +#ifdef CONFIG_KVM_SMM + .smi_allowed =3D vmx_smi_allowed, + .enter_smm =3D vmx_enter_smm, + .leave_smm =3D vmx_leave_smm, + .enable_smi_window =3D vmx_enable_smi_window, +#endif + + .can_emulate_instruction =3D vmx_can_emulate_instruction, + .apic_init_signal_blocked =3D vmx_apic_init_signal_blocked, + .migrate_timers =3D vmx_migrate_timers, + + .msr_filter_changed =3D vmx_msr_filter_changed, + .complete_emulated_msr =3D kvm_complete_insn_gp, + + .vcpu_deliver_sipi_vector =3D kvm_vcpu_deliver_sipi_vector, +}; + +struct kvm_x86_init_ops vt_init_ops __initdata =3D { + .hardware_setup =3D vmx_hardware_setup, + .handle_intel_pt_intr =3D NULL, + + .runtime_ops =3D &vt_x86_ops, + .pmu_ops =3D &intel_pmu_ops, +}; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 693f07b80966..4d8655a905c4 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -66,6 +66,7 @@ #include "vmcs12.h" #include "vmx.h" #include "x86.h" +#include "x86_ops.h" #include "smm.h" =20 MODULE_AUTHOR("Qumranet"); @@ -525,8 +526,6 @@ static inline void vmx_segment_cache_clear(struct vcpu_= vmx *vmx) static unsigned long host_idt_base; =20 #if IS_ENABLED(CONFIG_HYPERV) -static struct kvm_x86_ops vmx_x86_ops __initdata; - static bool __read_mostly enlightened_vmcs =3D true; module_param(enlightened_vmcs, bool, 0444); =20 @@ -584,9 +583,8 @@ static __init void hv_init_evmcs(void) } =20 if (ms_hyperv.nested_features & HV_X64_NESTED_DIRECT_FLUSH) - vmx_x86_ops.enable_l2_tlb_flush + vt_x86_ops.enable_l2_tlb_flush =3D hv_enable_l2_tlb_flush; - } else { enlightened_vmcs =3D false; } @@ -1457,7 +1455,7 @@ void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cp= u, * Switches to specified vcpu, until a matching vcpu_put(), but assumes * vcpu mutex is already taken. */ -static void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) +void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -1468,7 +1466,7 @@ static void vmx_vcpu_load(struct kvm_vcpu *vcpu, int = cpu) vmx->host_debugctlmsr =3D get_debugctlmsr(); } =20 -static void vmx_vcpu_put(struct kvm_vcpu *vcpu) +void vmx_vcpu_put(struct kvm_vcpu *vcpu) { vmx_vcpu_pi_put(vcpu); =20 @@ -1522,7 +1520,7 @@ void vmx_set_rflags(struct kvm_vcpu *vcpu, unsigned l= ong rflags) vmx->emulation_required =3D vmx_emulation_required(vcpu); } =20 -static bool vmx_get_if_flag(struct kvm_vcpu *vcpu) +bool vmx_get_if_flag(struct kvm_vcpu *vcpu) { return vmx_get_rflags(vcpu) & X86_EFLAGS_IF; } @@ -1628,8 +1626,8 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, = u64 data) return 0; } =20 -static bool vmx_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_ty= pe, - void *insn, int insn_len) +bool vmx_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, + void *insn, int insn_len) { /* * Emulation of instructions in SGX enclaves is impossible as RIP does @@ -1713,7 +1711,7 @@ static int skip_emulated_instruction(struct kvm_vcpu = *vcpu) * Recognizes a pending MTF VM-exit and records the nested state for later * delivery. */ -static void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu) +void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -1744,7 +1742,7 @@ static void vmx_update_emulated_instruction(struct kv= m_vcpu *vcpu) } } =20 -static int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu) +int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu) { vmx_update_emulated_instruction(vcpu); return skip_emulated_instruction(vcpu); @@ -1763,7 +1761,7 @@ static void vmx_clear_hlt(struct kvm_vcpu *vcpu) vmcs_write32(GUEST_ACTIVITY_STATE, GUEST_ACTIVITY_ACTIVE); } =20 -static void vmx_inject_exception(struct kvm_vcpu *vcpu) +void vmx_inject_exception(struct kvm_vcpu *vcpu) { struct kvm_queued_exception *ex =3D &vcpu->arch.exception; u32 intr_info =3D ex->vector | INTR_INFO_VALID_MASK; @@ -1884,12 +1882,12 @@ u64 vmx_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu) return kvm_caps.default_tsc_scaling_ratio; } =20 -static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset) +void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset) { vmcs_write64(TSC_OFFSET, offset); } =20 -static void vmx_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 multiplier) +void vmx_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 multiplier) { vmcs_write64(TSC_MULTIPLIER, multiplier); } @@ -1943,7 +1941,7 @@ static inline bool is_vmx_feature_control_msr_valid(s= truct vcpu_vmx *vmx, return !(msr->data & ~valid_bits); } =20 -static int vmx_get_msr_feature(struct kvm_msr_entry *msr) +int vmx_get_msr_feature(struct kvm_msr_entry *msr) { switch (msr->index) { case KVM_FIRST_EMULATED_VMX_MSR ... KVM_LAST_EMULATED_VMX_MSR: @@ -1960,7 +1958,7 @@ static int vmx_get_msr_feature(struct kvm_msr_entry *= msr) * Returns 0 on success, non-0 otherwise. * Assumes vcpu_load() was already called. */ -static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct vmx_uret_msr *msr; @@ -2139,7 +2137,7 @@ static u64 vmx_get_supported_debugctl(struct kvm_vcpu= *vcpu, bool host_initiated * Returns 0 on success, non-0 otherwise. * Assumes vcpu_load() was already called. */ -static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct vmx_uret_msr *msr; @@ -2442,7 +2440,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct = msr_data *msr_info) return ret; } =20 -static void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) +void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) { unsigned long guest_owned_bits; =20 @@ -2732,7 +2730,7 @@ static bool kvm_is_vmx_supported(void) return true; } =20 -static int vmx_check_processor_compat(void) +int vmx_check_processor_compat(void) { int cpu =3D raw_smp_processor_id(); struct vmcs_config vmcs_conf; @@ -2774,7 +2772,7 @@ static int kvm_cpu_vmxon(u64 vmxon_pointer) return -EFAULT; } =20 -static int vmx_hardware_enable(void) +int vmx_hardware_enable(void) { int cpu =3D raw_smp_processor_id(); u64 phys_addr =3D __pa(per_cpu(vmxarea, cpu)); @@ -2814,7 +2812,7 @@ static void vmclear_local_loaded_vmcss(void) __loaded_vmcs_clear(v); } =20 -static void vmx_hardware_disable(void) +void vmx_hardware_disable(void) { vmclear_local_loaded_vmcss(); =20 @@ -3126,7 +3124,7 @@ static void exit_lmode(struct kvm_vcpu *vcpu) =20 #endif =20 -static void vmx_flush_tlb_all(struct kvm_vcpu *vcpu) +void vmx_flush_tlb_all(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -3156,7 +3154,7 @@ static inline int vmx_get_current_vpid(struct kvm_vcp= u *vcpu) return to_vmx(vcpu)->vpid; } =20 -static void vmx_flush_tlb_current(struct kvm_vcpu *vcpu) +void vmx_flush_tlb_current(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; u64 root_hpa =3D mmu->root.hpa; @@ -3172,7 +3170,7 @@ static void vmx_flush_tlb_current(struct kvm_vcpu *vc= pu) vpid_sync_context(vmx_get_current_vpid(vcpu)); } =20 -static void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr) +void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr) { /* * vpid_sync_vcpu_addr() is a nop if vpid=3D=3D0, see the comment in @@ -3181,7 +3179,7 @@ static void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, = gva_t addr) vpid_sync_vcpu_addr(vmx_get_current_vpid(vcpu), addr); } =20 -static void vmx_flush_tlb_guest(struct kvm_vcpu *vcpu) +void vmx_flush_tlb_guest(struct kvm_vcpu *vcpu) { /* * vpid_sync_context() is a nop if vpid=3D=3D0, e.g. if enable_vpid=3D=3D= 0 or a @@ -3336,8 +3334,7 @@ u64 construct_eptp(struct kvm_vcpu *vcpu, hpa_t root_= hpa, int root_level) return eptp; } =20 -static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, - int root_level) +void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_leve= l) { struct kvm *kvm =3D vcpu->kvm; bool update_guest_cr3 =3D true; @@ -3365,8 +3362,7 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, h= pa_t root_hpa, vmcs_writel(GUEST_CR3, guest_cr3); } =20 - -static bool vmx_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +bool vmx_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) { /* * We operate under the default treatment of SMM, so VMX cannot be @@ -3482,7 +3478,7 @@ void vmx_get_segment(struct kvm_vcpu *vcpu, struct kv= m_segment *var, int seg) var->g =3D (ar >> 15) & 1; } =20 -static u64 vmx_get_segment_base(struct kvm_vcpu *vcpu, int seg) +u64 vmx_get_segment_base(struct kvm_vcpu *vcpu, int seg) { struct kvm_segment s; =20 @@ -3559,14 +3555,14 @@ void __vmx_set_segment(struct kvm_vcpu *vcpu, struc= t kvm_segment *var, int seg) vmcs_write32(sf->ar_bytes, vmx_segment_access_rights(var)); } =20 -static void vmx_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var= , int seg) +void vmx_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg) { __vmx_set_segment(vcpu, var, seg); =20 to_vmx(vcpu)->emulation_required =3D vmx_emulation_required(vcpu); } =20 -static void vmx_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l) +void vmx_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l) { u32 ar =3D vmx_read_guest_seg_ar(to_vmx(vcpu), VCPU_SREG_CS); =20 @@ -3574,25 +3570,25 @@ static void vmx_get_cs_db_l_bits(struct kvm_vcpu *v= cpu, int *db, int *l) *l =3D (ar >> 13) & 1; } =20 -static void vmx_get_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +void vmx_get_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) { dt->size =3D vmcs_read32(GUEST_IDTR_LIMIT); dt->address =3D vmcs_readl(GUEST_IDTR_BASE); } =20 -static void vmx_set_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +void vmx_set_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) { vmcs_write32(GUEST_IDTR_LIMIT, dt->size); vmcs_writel(GUEST_IDTR_BASE, dt->address); } =20 -static void vmx_get_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +void vmx_get_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) { dt->size =3D vmcs_read32(GUEST_GDTR_LIMIT); dt->address =3D vmcs_readl(GUEST_GDTR_BASE); } =20 -static void vmx_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +void vmx_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) { vmcs_write32(GUEST_GDTR_LIMIT, dt->size); vmcs_writel(GUEST_GDTR_BASE, dt->address); @@ -4064,7 +4060,7 @@ void pt_update_intercept_for_msr(struct kvm_vcpu *vcp= u) } } =20 -static bool vmx_guest_apic_has_interrupt(struct kvm_vcpu *vcpu) +bool vmx_guest_apic_has_interrupt(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); void *vapic_page; @@ -4084,7 +4080,7 @@ static bool vmx_guest_apic_has_interrupt(struct kvm_v= cpu *vcpu) return ((rvi & 0xf0) > (vppr & 0xf0)); } =20 -static void vmx_msr_filter_changed(struct kvm_vcpu *vcpu) +void vmx_msr_filter_changed(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); u32 i; @@ -4225,8 +4221,8 @@ static int vmx_deliver_posted_interrupt(struct kvm_vc= pu *vcpu, int vector) return 0; } =20 -static void vmx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mod= e, - int trig_mode, int vector) +void vmx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, + int trig_mode, int vector) { struct kvm_vcpu *vcpu =3D apic->vcpu; =20 @@ -4388,7 +4384,7 @@ static u32 vmx_vmexit_ctrl(void) ~(VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | VM_EXIT_LOAD_IA32_EFER); } =20 -static void vmx_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu) +void vmx_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -4659,7 +4655,7 @@ static int vmx_alloc_ipiv_pid_table(struct kvm *kvm) return 0; } =20 -static int vmx_vcpu_precreate(struct kvm *kvm) +int vmx_vcpu_precreate(struct kvm *kvm) { return vmx_alloc_ipiv_pid_table(kvm); } @@ -4811,7 +4807,7 @@ static void __vmx_vcpu_reset(struct kvm_vcpu *vcpu) vmx->pi_desc.sn =3D 1; } =20 -static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) +void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -4870,12 +4866,12 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, b= ool init_event) vmx_update_fb_clear_dis(vcpu, vmx); } =20 -static void vmx_enable_irq_window(struct kvm_vcpu *vcpu) +void vmx_enable_irq_window(struct kvm_vcpu *vcpu) { exec_controls_setbit(to_vmx(vcpu), CPU_BASED_INTR_WINDOW_EXITING); } =20 -static void vmx_enable_nmi_window(struct kvm_vcpu *vcpu) +void vmx_enable_nmi_window(struct kvm_vcpu *vcpu) { if (!enable_vnmi || vmcs_read32(GUEST_INTERRUPTIBILITY_INFO) & GUEST_INTR_STATE_STI) { @@ -4886,7 +4882,7 @@ static void vmx_enable_nmi_window(struct kvm_vcpu *vc= pu) exec_controls_setbit(to_vmx(vcpu), CPU_BASED_NMI_WINDOW_EXITING); } =20 -static void vmx_inject_irq(struct kvm_vcpu *vcpu, bool reinjected) +void vmx_inject_irq(struct kvm_vcpu *vcpu, bool reinjected) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); uint32_t intr; @@ -4914,7 +4910,7 @@ static void vmx_inject_irq(struct kvm_vcpu *vcpu, boo= l reinjected) vmx_clear_hlt(vcpu); } =20 -static void vmx_inject_nmi(struct kvm_vcpu *vcpu) +void vmx_inject_nmi(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -4992,7 +4988,7 @@ bool vmx_nmi_blocked(struct kvm_vcpu *vcpu) GUEST_INTR_STATE_NMI)); } =20 -static int vmx_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection) +int vmx_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection) { if (to_vmx(vcpu)->nested.nested_run_pending) return -EBUSY; @@ -5014,7 +5010,7 @@ bool vmx_interrupt_blocked(struct kvm_vcpu *vcpu) (GUEST_INTR_STATE_STI | GUEST_INTR_STATE_MOV_SS)); } =20 -static int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection) +int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection) { if (to_vmx(vcpu)->nested.nested_run_pending) return -EBUSY; @@ -5029,7 +5025,7 @@ static int vmx_interrupt_allowed(struct kvm_vcpu *vcp= u, bool for_injection) return !vmx_interrupt_blocked(vcpu); } =20 -static int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr) +int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr) { void __user *ret; =20 @@ -5049,7 +5045,7 @@ static int vmx_set_tss_addr(struct kvm *kvm, unsigned= int addr) return init_rmode_tss(kvm, ret); } =20 -static int vmx_set_identity_map_addr(struct kvm *kvm, u64 ident_addr) +int vmx_set_identity_map_addr(struct kvm *kvm, u64 ident_addr) { to_kvm_vmx(kvm)->ept_identity_map_addr =3D ident_addr; return 0; @@ -5335,8 +5331,7 @@ static int handle_io(struct kvm_vcpu *vcpu) return kvm_fast_pio(vcpu, size, port, in); } =20 -static void -vmx_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall) +void vmx_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall) { /* * Patch in the VMCALL instruction: @@ -5552,7 +5547,7 @@ static int handle_dr(struct kvm_vcpu *vcpu) return kvm_complete_insn_gp(vcpu, err); } =20 -static void vmx_sync_dirty_debug_regs(struct kvm_vcpu *vcpu) +void vmx_sync_dirty_debug_regs(struct kvm_vcpu *vcpu) { get_debugreg(vcpu->arch.db[0], 0); get_debugreg(vcpu->arch.db[1], 1); @@ -5571,7 +5566,7 @@ static void vmx_sync_dirty_debug_regs(struct kvm_vcpu= *vcpu) set_debugreg(DR6_RESERVED, 6); } =20 -static void vmx_set_dr7(struct kvm_vcpu *vcpu, unsigned long val) +void vmx_set_dr7(struct kvm_vcpu *vcpu, unsigned long val) { vmcs_writel(GUEST_DR7, val); } @@ -5842,7 +5837,7 @@ static int handle_invalid_guest_state(struct kvm_vcpu= *vcpu) return 1; } =20 -static int vmx_vcpu_pre_run(struct kvm_vcpu *vcpu) +int vmx_vcpu_pre_run(struct kvm_vcpu *vcpu) { if (vmx_emulation_required_with_pending_exception(vcpu)) { kvm_prepare_emulation_failure_exit(vcpu); @@ -6106,9 +6101,8 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu= *vcpu) =3D { static const int kvm_vmx_max_exit_handlers =3D ARRAY_SIZE(kvm_vmx_exit_handlers); =20 -static void vmx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, - u64 *info1, u64 *info2, - u32 *intr_info, u32 *error_code) +void vmx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, + u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -6551,7 +6545,7 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, f= astpath_t exit_fastpath) return 0; } =20 -static int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) +int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) { int ret =3D __vmx_handle_exit(vcpu, exit_fastpath); =20 @@ -6639,7 +6633,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vc= pu) : "eax", "ebx", "ecx", "edx"); } =20 -static void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int i= rr) +void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); int tpr_threshold; @@ -6709,7 +6703,7 @@ void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu) vmx_update_msr_bitmap_x2apic(vcpu); } =20 -static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) +void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) { const gfn_t gfn =3D APIC_DEFAULT_PHYS_BASE >> PAGE_SHIFT; struct kvm *kvm =3D vcpu->kvm; @@ -6776,7 +6770,7 @@ static void vmx_set_apic_access_page_addr(struct kvm_= vcpu *vcpu) kvm_release_pfn_clean(pfn); } =20 -static void vmx_hwapic_isr_update(int max_isr) +void vmx_hwapic_isr_update(int max_isr) { u16 status; u8 old; @@ -6810,7 +6804,7 @@ static void vmx_set_rvi(int vector) } } =20 -static void vmx_hwapic_irr_update(struct kvm_vcpu *vcpu, int max_irr) +void vmx_hwapic_irr_update(struct kvm_vcpu *vcpu, int max_irr) { /* * When running L2, updating RVI is only relevant when @@ -6824,7 +6818,7 @@ static void vmx_hwapic_irr_update(struct kvm_vcpu *vc= pu, int max_irr) vmx_set_rvi(max_irr); } =20 -static int vmx_sync_pir_to_irr(struct kvm_vcpu *vcpu) +int vmx_sync_pir_to_irr(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); int max_irr; @@ -6870,7 +6864,7 @@ static int vmx_sync_pir_to_irr(struct kvm_vcpu *vcpu) return max_irr; } =20 -static void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitm= ap) +void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap) { if (!kvm_vcpu_apicv_active(vcpu)) return; @@ -6881,7 +6875,7 @@ static void vmx_load_eoi_exitmap(struct kvm_vcpu *vcp= u, u64 *eoi_exit_bitmap) vmcs_write64(EOI_EXIT_BITMAP3, eoi_exit_bitmap[3]); } =20 -static void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) +void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -6944,7 +6938,7 @@ static void handle_external_interrupt_irqoff(struct k= vm_vcpu *vcpu) vcpu->arch.at_instruction_boundary =3D true; } =20 -static void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) +void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -6961,7 +6955,7 @@ static void vmx_handle_exit_irqoff(struct kvm_vcpu *v= cpu) * The kvm parameter can be NULL (module initialization, or invocation bef= ore * VM creation). Be sure to check the kvm parameter before using it. */ -static bool vmx_has_emulated_msr(struct kvm *kvm, u32 index) +bool vmx_has_emulated_msr(struct kvm *kvm, u32 index) { switch (index) { case MSR_IA32_SMBASE: @@ -7084,7 +7078,7 @@ static void vmx_complete_interrupts(struct vcpu_vmx *= vmx) IDT_VECTORING_ERROR_CODE); } =20 -static void vmx_cancel_injection(struct kvm_vcpu *vcpu) +void vmx_cancel_injection(struct kvm_vcpu *vcpu) { __vmx_complete_interrupts(vcpu, vmcs_read32(VM_ENTRY_INTR_INFO_FIELD), @@ -7231,7 +7225,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, guest_state_exit_irqoff(); } =20 -static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) +fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); unsigned long cr3, cr4; @@ -7394,7 +7388,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) return vmx_exit_handlers_fastpath(vcpu); } =20 -static void vmx_vcpu_free(struct kvm_vcpu *vcpu) +void vmx_vcpu_free(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -7405,7 +7399,7 @@ static void vmx_vcpu_free(struct kvm_vcpu *vcpu) free_loaded_vmcs(vmx->loaded_vmcs); } =20 -static int vmx_vcpu_create(struct kvm_vcpu *vcpu) +int vmx_vcpu_create(struct kvm_vcpu *vcpu) { struct vmx_uret_msr *tsx_ctrl; struct vcpu_vmx *vmx; @@ -7511,7 +7505,7 @@ static int vmx_vcpu_create(struct kvm_vcpu *vcpu) return err; } =20 -static bool vmx_is_vm_type_supported(unsigned long type) +bool vmx_is_vm_type_supported(unsigned long type) { /* TODO: Check if TDX is supported. */ return __kvm_is_vm_type_supported(type); @@ -7520,7 +7514,7 @@ static bool vmx_is_vm_type_supported(unsigned long ty= pe) #define L1TF_MSG_SMT "L1TF CPU bug present and SMT on, data leak possible.= See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/h= w-vuln/l1tf.html for details.\n" #define L1TF_MSG_L1D "L1TF CPU bug present and virtualization mitigation d= isabled, data leak possible. See CVE-2018-3646 and https://www.kernel.org/d= oc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n" =20 -static int vmx_vm_init(struct kvm *kvm) +int vmx_vm_init(struct kvm *kvm) { if (!ple_gap) kvm->arch.pause_in_guest =3D true; @@ -7551,7 +7545,7 @@ static int vmx_vm_init(struct kvm *kvm) return 0; } =20 -static u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) +u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { u8 cache; =20 @@ -7723,7 +7717,7 @@ static void update_intel_pt_cfg(struct kvm_vcpu *vcpu) vmx->pt_desc.ctl_bitmask &=3D ~(0xfULL << (32 + i * 4)); } =20 -static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) +void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -7868,7 +7862,7 @@ static __init void vmx_set_cpu_caps(void) kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG); } =20 -static void vmx_request_immediate_exit(struct kvm_vcpu *vcpu) +void vmx_request_immediate_exit(struct kvm_vcpu *vcpu) { to_vmx(vcpu)->req_immediate_exit =3D true; } @@ -7907,10 +7901,10 @@ static int vmx_check_intercept_io(struct kvm_vcpu *= vcpu, return intercept ? X86EMUL_UNHANDLEABLE : X86EMUL_CONTINUE; } =20 -static int vmx_check_intercept(struct kvm_vcpu *vcpu, - struct x86_instruction_info *info, - enum x86_intercept_stage stage, - struct x86_exception *exception) +int vmx_check_intercept(struct kvm_vcpu *vcpu, + struct x86_instruction_info *info, + enum x86_intercept_stage stage, + struct x86_exception *exception) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); =20 @@ -7990,8 +7984,8 @@ static inline int u64_shl_div_u64(u64 a, unsigned int= shift, return 0; } =20 -static int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, - bool *expired) +int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, + bool *expired) { struct vcpu_vmx *vmx; u64 tscl, guest_tscl, delta_tsc, lapic_timer_advance_cycles; @@ -8030,13 +8024,13 @@ static int vmx_set_hv_timer(struct kvm_vcpu *vcpu, = u64 guest_deadline_tsc, return 0; } =20 -static void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu) +void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu) { to_vmx(vcpu)->hv_deadline_tsc =3D -1; } #endif =20 -static void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu) +void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu) { if (!kvm_pause_in_guest(vcpu->kvm)) shrink_ple_window(vcpu); @@ -8065,7 +8059,7 @@ void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vc= pu) secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_ENABLE_PML); } =20 -static void vmx_setup_mce(struct kvm_vcpu *vcpu) +void vmx_setup_mce(struct kvm_vcpu *vcpu) { if (vcpu->arch.mcg_cap & MCG_LMCE_P) to_vmx(vcpu)->msr_ia32_feature_control_valid_bits |=3D @@ -8076,7 +8070,7 @@ static void vmx_setup_mce(struct kvm_vcpu *vcpu) } =20 #ifdef CONFIG_KVM_SMM -static int vmx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection) +int vmx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection) { /* we need a nested vmexit to enter SMM, postpone if run is pending */ if (to_vmx(vcpu)->nested.nested_run_pending) @@ -8084,7 +8078,7 @@ static int vmx_smi_allowed(struct kvm_vcpu *vcpu, boo= l for_injection) return !is_smm(vcpu); } =20 -static int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) +int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -8105,7 +8099,7 @@ static int vmx_enter_smm(struct kvm_vcpu *vcpu, union= kvm_smram *smram) return 0; } =20 -static int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smr= am) +int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); int ret; @@ -8126,18 +8120,18 @@ static int vmx_leave_smm(struct kvm_vcpu *vcpu, con= st union kvm_smram *smram) return 0; } =20 -static void vmx_enable_smi_window(struct kvm_vcpu *vcpu) +void vmx_enable_smi_window(struct kvm_vcpu *vcpu) { /* RSM will cause a vmexit anyway. */ } #endif =20 -static bool vmx_apic_init_signal_blocked(struct kvm_vcpu *vcpu) +bool vmx_apic_init_signal_blocked(struct kvm_vcpu *vcpu) { return to_vmx(vcpu)->nested.vmxon && !is_guest_mode(vcpu); } =20 -static void vmx_migrate_timers(struct kvm_vcpu *vcpu) +void vmx_migrate_timers(struct kvm_vcpu *vcpu) { if (is_guest_mode(vcpu)) { struct hrtimer *timer =3D &to_vmx(vcpu)->nested.preemption_timer; @@ -8147,7 +8141,7 @@ static void vmx_migrate_timers(struct kvm_vcpu *vcpu) } } =20 -static void vmx_hardware_unsetup(void) +void vmx_hardware_unsetup(void) { kvm_set_posted_intr_wakeup_handler(NULL); =20 @@ -8157,166 +8151,13 @@ static void vmx_hardware_unsetup(void) free_kvm_area(); } =20 -#define VMX_REQUIRED_APICV_INHIBITS \ -( \ - BIT(APICV_INHIBIT_REASON_DISABLE)| \ - BIT(APICV_INHIBIT_REASON_ABSENT) | \ - BIT(APICV_INHIBIT_REASON_HYPERV) | \ - BIT(APICV_INHIBIT_REASON_BLOCKIRQ) | \ - BIT(APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED) | \ - BIT(APICV_INHIBIT_REASON_APIC_ID_MODIFIED) | \ - BIT(APICV_INHIBIT_REASON_APIC_BASE_MODIFIED) \ -) - -static void vmx_vm_destroy(struct kvm *kvm) +void vmx_vm_destroy(struct kvm *kvm) { struct kvm_vmx *kvm_vmx =3D to_kvm_vmx(kvm); =20 free_pages((unsigned long)kvm_vmx->pid_table, vmx_get_pid_table_order(kvm= )); } =20 -static struct kvm_x86_ops vmx_x86_ops __initdata =3D { - .name =3D KBUILD_MODNAME, - - .check_processor_compatibility =3D vmx_check_processor_compat, - - .hardware_unsetup =3D vmx_hardware_unsetup, - - .hardware_enable =3D vmx_hardware_enable, - .hardware_disable =3D vmx_hardware_disable, - .has_emulated_msr =3D vmx_has_emulated_msr, - - .is_vm_type_supported =3D vmx_is_vm_type_supported, - .vm_size =3D sizeof(struct kvm_vmx), - .vm_init =3D vmx_vm_init, - .vm_destroy =3D vmx_vm_destroy, - - .vcpu_precreate =3D vmx_vcpu_precreate, - .vcpu_create =3D vmx_vcpu_create, - .vcpu_free =3D vmx_vcpu_free, - .vcpu_reset =3D vmx_vcpu_reset, - - .prepare_switch_to_guest =3D vmx_prepare_switch_to_guest, - .vcpu_load =3D vmx_vcpu_load, - .vcpu_put =3D vmx_vcpu_put, - - .update_exception_bitmap =3D vmx_update_exception_bitmap, - .get_msr_feature =3D vmx_get_msr_feature, - .get_msr =3D vmx_get_msr, - .set_msr =3D vmx_set_msr, - .get_segment_base =3D vmx_get_segment_base, - .get_segment =3D vmx_get_segment, - .set_segment =3D vmx_set_segment, - .get_cpl =3D vmx_get_cpl, - .get_cs_db_l_bits =3D vmx_get_cs_db_l_bits, - .set_cr0 =3D vmx_set_cr0, - .is_valid_cr4 =3D vmx_is_valid_cr4, - .set_cr4 =3D vmx_set_cr4, - .set_efer =3D vmx_set_efer, - .get_idt =3D vmx_get_idt, - .set_idt =3D vmx_set_idt, - .get_gdt =3D vmx_get_gdt, - .set_gdt =3D vmx_set_gdt, - .set_dr7 =3D vmx_set_dr7, - .sync_dirty_debug_regs =3D vmx_sync_dirty_debug_regs, - .cache_reg =3D vmx_cache_reg, - .get_rflags =3D vmx_get_rflags, - .set_rflags =3D vmx_set_rflags, - .get_if_flag =3D vmx_get_if_flag, - - .flush_tlb_all =3D vmx_flush_tlb_all, - .flush_tlb_current =3D vmx_flush_tlb_current, - .flush_tlb_gva =3D vmx_flush_tlb_gva, - .flush_tlb_guest =3D vmx_flush_tlb_guest, - - .vcpu_pre_run =3D vmx_vcpu_pre_run, - .vcpu_run =3D vmx_vcpu_run, - .handle_exit =3D vmx_handle_exit, - .skip_emulated_instruction =3D vmx_skip_emulated_instruction, - .update_emulated_instruction =3D vmx_update_emulated_instruction, - .set_interrupt_shadow =3D vmx_set_interrupt_shadow, - .get_interrupt_shadow =3D vmx_get_interrupt_shadow, - .patch_hypercall =3D vmx_patch_hypercall, - .inject_irq =3D vmx_inject_irq, - .inject_nmi =3D vmx_inject_nmi, - .inject_exception =3D vmx_inject_exception, - .cancel_injection =3D vmx_cancel_injection, - .interrupt_allowed =3D vmx_interrupt_allowed, - .nmi_allowed =3D vmx_nmi_allowed, - .get_nmi_mask =3D vmx_get_nmi_mask, - .set_nmi_mask =3D vmx_set_nmi_mask, - .enable_nmi_window =3D vmx_enable_nmi_window, - .enable_irq_window =3D vmx_enable_irq_window, - .update_cr8_intercept =3D vmx_update_cr8_intercept, - .set_virtual_apic_mode =3D vmx_set_virtual_apic_mode, - .set_apic_access_page_addr =3D vmx_set_apic_access_page_addr, - .refresh_apicv_exec_ctrl =3D vmx_refresh_apicv_exec_ctrl, - .load_eoi_exitmap =3D vmx_load_eoi_exitmap, - .apicv_post_state_restore =3D vmx_apicv_post_state_restore, - .required_apicv_inhibits =3D VMX_REQUIRED_APICV_INHIBITS, - .hwapic_irr_update =3D vmx_hwapic_irr_update, - .hwapic_isr_update =3D vmx_hwapic_isr_update, - .guest_apic_has_interrupt =3D vmx_guest_apic_has_interrupt, - .sync_pir_to_irr =3D vmx_sync_pir_to_irr, - .deliver_interrupt =3D vmx_deliver_interrupt, - .dy_apicv_has_pending_interrupt =3D pi_has_pending_interrupt, - - .set_tss_addr =3D vmx_set_tss_addr, - .set_identity_map_addr =3D vmx_set_identity_map_addr, - .get_mt_mask =3D vmx_get_mt_mask, - - .get_exit_info =3D vmx_get_exit_info, - - .vcpu_after_set_cpuid =3D vmx_vcpu_after_set_cpuid, - - .has_wbinvd_exit =3D cpu_has_vmx_wbinvd_exit, - - .get_l2_tsc_offset =3D vmx_get_l2_tsc_offset, - .get_l2_tsc_multiplier =3D vmx_get_l2_tsc_multiplier, - .write_tsc_offset =3D vmx_write_tsc_offset, - .write_tsc_multiplier =3D vmx_write_tsc_multiplier, - - .load_mmu_pgd =3D vmx_load_mmu_pgd, - - .check_intercept =3D vmx_check_intercept, - .handle_exit_irqoff =3D vmx_handle_exit_irqoff, - - .request_immediate_exit =3D vmx_request_immediate_exit, - - .sched_in =3D vmx_sched_in, - - .cpu_dirty_log_size =3D PML_ENTITY_NUM, - .update_cpu_dirty_logging =3D vmx_update_cpu_dirty_logging, - - .nested_ops =3D &vmx_nested_ops, - - .pi_update_irte =3D vmx_pi_update_irte, - .pi_start_assignment =3D vmx_pi_start_assignment, - -#ifdef CONFIG_X86_64 - .set_hv_timer =3D vmx_set_hv_timer, - .cancel_hv_timer =3D vmx_cancel_hv_timer, -#endif - - .setup_mce =3D vmx_setup_mce, - -#ifdef CONFIG_KVM_SMM - .smi_allowed =3D vmx_smi_allowed, - .enter_smm =3D vmx_enter_smm, - .leave_smm =3D vmx_leave_smm, - .enable_smi_window =3D vmx_enable_smi_window, -#endif - - .can_emulate_instruction =3D vmx_can_emulate_instruction, - .apic_init_signal_blocked =3D vmx_apic_init_signal_blocked, - .migrate_timers =3D vmx_migrate_timers, - - .msr_filter_changed =3D vmx_msr_filter_changed, - .complete_emulated_msr =3D kvm_complete_insn_gp, - - .vcpu_deliver_sipi_vector =3D kvm_vcpu_deliver_sipi_vector, -}; - static unsigned int vmx_handle_intel_pt_intr(void) { struct kvm_vcpu *vcpu =3D kvm_get_running_vcpu(); @@ -8382,9 +8223,7 @@ static void __init vmx_setup_me_spte_mask(void) kvm_mmu_set_me_spte_mask(0, me_mask); } =20 -static struct kvm_x86_init_ops vmx_init_ops __initdata; - -static __init int hardware_setup(void) +__init int vmx_hardware_setup(void) { unsigned long host_bndcfgs; struct desc_ptr dt; @@ -8453,16 +8292,16 @@ static __init int hardware_setup(void) * using the APIC_ACCESS_ADDR VMCS field. */ if (!flexpriority_enabled) - vmx_x86_ops.set_apic_access_page_addr =3D NULL; + vt_x86_ops.set_apic_access_page_addr =3D NULL; =20 if (!cpu_has_vmx_tpr_shadow()) - vmx_x86_ops.update_cr8_intercept =3D NULL; + vt_x86_ops.update_cr8_intercept =3D NULL; =20 #if IS_ENABLED(CONFIG_HYPERV) if (ms_hyperv.nested_features & HV_X64_NESTED_GUEST_MAPPING_FLUSH && enable_ept) { - vmx_x86_ops.flush_remote_tlbs =3D hv_flush_remote_tlbs; - vmx_x86_ops.flush_remote_tlbs_range =3D hv_flush_remote_tlbs_range; + vt_x86_ops.flush_remote_tlbs =3D hv_flush_remote_tlbs; + vt_x86_ops.flush_remote_tlbs_range =3D hv_flush_remote_tlbs_range; } #endif =20 @@ -8477,7 +8316,7 @@ static __init int hardware_setup(void) if (!cpu_has_vmx_apicv()) enable_apicv =3D 0; if (!enable_apicv) - vmx_x86_ops.sync_pir_to_irr =3D NULL; + vt_x86_ops.sync_pir_to_irr =3D NULL; =20 if (!enable_apicv || !cpu_has_vmx_ipiv()) enable_ipiv =3D false; @@ -8513,7 +8352,7 @@ static __init int hardware_setup(void) enable_pml =3D 0; =20 if (!enable_pml) - vmx_x86_ops.cpu_dirty_log_size =3D 0; + vt_x86_ops.cpu_dirty_log_size =3D 0; =20 if (!cpu_has_vmx_preemption_timer()) enable_preemption_timer =3D false; @@ -8538,9 +8377,9 @@ static __init int hardware_setup(void) } =20 if (!enable_preemption_timer) { - vmx_x86_ops.set_hv_timer =3D NULL; - vmx_x86_ops.cancel_hv_timer =3D NULL; - vmx_x86_ops.request_immediate_exit =3D __kvm_request_immediate_exit; + vt_x86_ops.set_hv_timer =3D NULL; + vt_x86_ops.cancel_hv_timer =3D NULL; + vt_x86_ops.request_immediate_exit =3D __kvm_request_immediate_exit; } =20 kvm_caps.supported_mce_cap |=3D MCG_LMCE_P; @@ -8551,9 +8390,9 @@ static __init int hardware_setup(void) if (!enable_ept || !enable_pmu || !cpu_has_vmx_intel_pt()) pt_mode =3D PT_MODE_SYSTEM; if (pt_mode =3D=3D PT_MODE_HOST_GUEST) - vmx_init_ops.handle_intel_pt_intr =3D vmx_handle_intel_pt_intr; + vt_init_ops.handle_intel_pt_intr =3D vmx_handle_intel_pt_intr; else - vmx_init_ops.handle_intel_pt_intr =3D NULL; + vt_init_ops.handle_intel_pt_intr =3D NULL; =20 setup_default_sgx_lepubkeyhash(); =20 @@ -8576,14 +8415,6 @@ static __init int hardware_setup(void) return r; } =20 -static struct kvm_x86_init_ops vmx_init_ops __initdata =3D { - .hardware_setup =3D hardware_setup, - .handle_intel_pt_intr =3D NULL, - - .runtime_ops =3D &vmx_x86_ops, - .pmu_ops =3D &intel_pmu_ops, -}; - static void vmx_cleanup_l1d_flush(void) { if (vmx_l1d_flush_pages) { @@ -8627,7 +8458,7 @@ static int __init vmx_init(void) */ hv_init_evmcs(); =20 - r =3D kvm_x86_vendor_init(&vmx_init_ops); + r =3D kvm_x86_vendor_init(&vt_init_ops); if (r) return r; =20 diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h new file mode 100644 index 000000000000..c15c64e99de3 --- /dev/null +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -0,0 +1,126 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMX_X86_OPS_H +#define __KVM_X86_VMX_X86_OPS_H + +#include + +#include + +#include "x86.h" + +__init int vmx_hardware_setup(void); + +extern struct kvm_x86_ops vt_x86_ops __initdata; +extern struct kvm_x86_init_ops vt_init_ops __initdata; + +void vmx_hardware_unsetup(void); +int vmx_check_processor_compat(void); +int vmx_hardware_enable(void); +void vmx_hardware_disable(void); +bool vmx_is_vm_type_supported(unsigned long type); +int vmx_vm_init(struct kvm *kvm); +void vmx_vm_destroy(struct kvm *kvm); +int vmx_vcpu_precreate(struct kvm *kvm); +int vmx_vcpu_create(struct kvm_vcpu *vcpu); +int vmx_vcpu_pre_run(struct kvm_vcpu *vcpu); +fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu); +void vmx_vcpu_free(struct kvm_vcpu *vcpu); +void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); +void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); +void vmx_vcpu_put(struct kvm_vcpu *vcpu); +int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath); +void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu); +int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu); +void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu); +int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info); +#ifdef CONFIG_KVM_SMM +int vmx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection); +int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram); +int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram); +void vmx_enable_smi_window(struct kvm_vcpu *vcpu); +#endif +bool vmx_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, + void *insn, int insn_len); +int vmx_check_intercept(struct kvm_vcpu *vcpu, + struct x86_instruction_info *info, + enum x86_intercept_stage stage, + struct x86_exception *exception); +bool vmx_apic_init_signal_blocked(struct kvm_vcpu *vcpu); +void vmx_migrate_timers(struct kvm_vcpu *vcpu); +void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu); +void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu); +bool vmx_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason); +void vmx_hwapic_irr_update(struct kvm_vcpu *vcpu, int max_irr); +void vmx_hwapic_isr_update(int max_isr); +bool vmx_guest_apic_has_interrupt(struct kvm_vcpu *vcpu); +int vmx_sync_pir_to_irr(struct kvm_vcpu *vcpu); +void vmx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, + int trig_mode, int vector); +void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu); +bool vmx_has_emulated_msr(struct kvm *kvm, u32 index); +void vmx_msr_filter_changed(struct kvm_vcpu *vcpu); +void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); +void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu); +int vmx_get_msr_feature(struct kvm_msr_entry *msr); +int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info); +u64 vmx_get_segment_base(struct kvm_vcpu *vcpu, int seg); +void vmx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg); +void vmx_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg); +int vmx_get_cpl(struct kvm_vcpu *vcpu); +void vmx_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l); +void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); +void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_leve= l); +void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); +bool vmx_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); +int vmx_set_efer(struct kvm_vcpu *vcpu, u64 efer); +void vmx_get_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt); +void vmx_set_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt); +void vmx_get_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt); +void vmx_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt); +void vmx_set_dr7(struct kvm_vcpu *vcpu, unsigned long val); +void vmx_sync_dirty_debug_regs(struct kvm_vcpu *vcpu); +void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg); +unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu); +void vmx_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); +bool vmx_get_if_flag(struct kvm_vcpu *vcpu); +void vmx_flush_tlb_all(struct kvm_vcpu *vcpu); +void vmx_flush_tlb_current(struct kvm_vcpu *vcpu); +void vmx_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr); +void vmx_flush_tlb_guest(struct kvm_vcpu *vcpu); +void vmx_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask); +u32 vmx_get_interrupt_shadow(struct kvm_vcpu *vcpu); +void vmx_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall); +void vmx_inject_irq(struct kvm_vcpu *vcpu, bool reinjected); +void vmx_inject_nmi(struct kvm_vcpu *vcpu); +void vmx_inject_exception(struct kvm_vcpu *vcpu); +void vmx_cancel_injection(struct kvm_vcpu *vcpu); +int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection); +int vmx_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection); +bool vmx_get_nmi_mask(struct kvm_vcpu *vcpu); +void vmx_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked); +void vmx_enable_nmi_window(struct kvm_vcpu *vcpu); +void vmx_enable_irq_window(struct kvm_vcpu *vcpu); +void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr); +void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu); +void vmx_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu); +void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap); +int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr); +int vmx_set_identity_map_addr(struct kvm *kvm, u64 ident_addr); +u8 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); +void vmx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, + u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code); +u64 vmx_get_l2_tsc_offset(struct kvm_vcpu *vcpu); +u64 vmx_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu); +void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset); +void vmx_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 multiplier); +void vmx_request_immediate_exit(struct kvm_vcpu *vcpu); +void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu); +void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu); +#ifdef CONFIG_X86_64 +int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, + bool *expired); +void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu); +#endif +void vmx_setup_mce(struct kvm_vcpu *vcpu); + +#endif /* __KVM_X86_VMX_X86_OPS_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 575CFC001DF for ; Tue, 25 Jul 2023 22:15:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229874AbjGYWPh (ORCPT ); Tue, 25 Jul 2023 18:15:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229445AbjGYWPf (ORCPT ); Tue, 25 Jul 2023 18:15:35 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F460B6; Tue, 25 Jul 2023 15:15:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323334; x=1721859334; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=q6K2maaUsjgfiXlRf9woadg1SoAwVb/MVIcmVsQ7T20=; b=R7Tss8A+m1rVbXQ54g1TCaRVSOujtMNkXVuOOZGFihTZrEepGdyxpYnA 5FImPgz6RG5wIPienbXnY6ZYOpLQ96Q5A4p5cxoxTQ1ZqvSNvtWzcZ8I9 cAsl6NF2GG4T9l0P1nggDDi/gm7HWToqpgkCv0i6I9Px4sjEuZeQlxnKb bzOSvhIPSsS6ooi6v2Gg+QCODSIyG8t/M5wdJrJQZQzBXvfQlJwquqFb9 F1pG0n7TNAM1yD3WC6u+e2yVbXcmkhXIuxE9yz0vtElq+ndj8hfh7//Rt jHCkqQkp5TTNtspaJmzibLTg0Ck64Nk0Cniad2yNYmXHZErYN6PK2t5WC A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863022" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863022" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938768" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938768" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:15 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 002/115] KVM: x86/vmx: initialize loaded_vmcss_on_cpu in vmx_hardware_setup() Date: Tue, 25 Jul 2023 15:13:13 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata vmx_hardware_disable() accesses loaded_vmcss_on_cpu via hardware_disable_all(). To allow hardware_enable/disable_all() before kvm_init(), initialize it in vmx_hardware_setup() so that tdx module initialization, hardware_setup method, can reference the variable. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/vmx.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 4d8655a905c4..9b035d5571fe 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -8227,8 +8227,12 @@ __init int vmx_hardware_setup(void) { unsigned long host_bndcfgs; struct desc_ptr dt; + int cpu; int r; =20 + /* vmx_hardware_disable() accesses loaded_vmcss_on_cpu. */ + for_each_possible_cpu(cpu) + INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu)); store_idt(&dt); host_idt_base =3D dt.address; =20 @@ -8475,11 +8479,8 @@ static int __init vmx_init(void) =20 vmx_setup_fb_clear_ctrl(); =20 - for_each_possible_cpu(cpu) { - INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu)); - + for_each_possible_cpu(cpu) pi_init_cpu(cpu); - } =20 #ifdef CONFIG_KEXEC_CORE rcu_assign_pointer(crash_vmclear_loaded_vmcss, --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 064F7C04FDF for ; Tue, 25 Jul 2023 22:16:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232137AbjGYWQL (ORCPT ); Tue, 25 Jul 2023 18:16:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230131AbjGYWPi (ORCPT ); Tue, 25 Jul 2023 18:15:38 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C75DE47; Tue, 25 Jul 2023 15:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323337; x=1721859337; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vZuTkNlhPpnMmzblXGgyBFUYFK4InJ2dPONAtP9F+Ko=; b=RWfIms+rr28yP/KXf3TdP9dmPAl6bSjSUC2pDeD8TaV1J/Y0tKcJJAad 9MqxnVJrE8S+03uE/SrgOYkUx/+ElMd8U8Eu3cGbD3JkL5t+LKivGM57A HZsY12AJreAID8QVS7al0MeaHQBl8v2N7FrZrE5PozglPlkHfv0oMwd+O C38zoed1+VoZ/OAeI2RkrusgKZBH9ib28SJoWIMqERjNLOlFFnDPwKTSJ yVH9YUAUJNxcCuIXOUcLi/bTIkyVUkfkN/5SFgX6yanDGoClgnTMvsgPD Kvl6eM5t9+L373wTLN1O564wmZ1X+3TDu2pZBhcUJnNaXdrVkjpowDd6d w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863026" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863026" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938772" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938772" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:15 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 003/115] KVM: x86/vmx: Refactor KVM VMX module init/exit functions Date: Tue, 25 Jul 2023 15:13:14 -0700 Message-Id: <9f7e788b0c7f90d0250ee168f0bc90fffd681415.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Currently, KVM VMX module initialization/exit functions are a single function each. Refactor KVM VMX module initialization functions into KVM common part and VMX part so that TDX specific part can be added cleanly. Opportunistically refactor module exit function as well. The current module initialization flow is, 0.) Check if VMX is supported, 1.) hyper-v specific initialization, 2.) system-wide x86 specific and vendor specific initialization, 3.) Final VMX specific system-wide initialization, 4.) calculate the sizes of VMX kvm structure and VMX vcpu structure, 5.) report those sizes to the KVM common layer and KVM common initialization Refactor the KVM VMX module initialization function into functions with a wrapper function to separate VMX logic in vmx.c from a file, main.c, common among VMX and TDX. Introduce a wrapper function for vmx_init(). The KVM architecture common layer allocates struct kvm with reported size for architecture-specific code. The KVM VMX module defines its structure as struct vmx_kvm { struct kvm; VMX specific members;} and uses it as struct vmx kvm. Similar for vcpu structure. TDX KVM patches will define TDX specific kvm and vcpu structures. The current module exit function is also a single function, a combination of VMX specific logic and common KVM logic. Refactor it into VMX specific logic and KVM common logic. This is just refactoring to keep the VMX specific logic in vmx.c from main.c. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 50 +++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 54 +++++--------------------------------- arch/x86/kvm/vmx/x86_ops.h | 13 ++++++++- 3 files changed, 68 insertions(+), 49 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index a738ae96ca24..27bfd3fcea09 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -164,3 +164,53 @@ struct kvm_x86_init_ops vt_init_ops __initdata =3D { .runtime_ops =3D &vt_x86_ops, .pmu_ops =3D &intel_pmu_ops, }; + +static int __init vt_init(void) +{ + unsigned int vcpu_size, vcpu_align; + int r; + + if (!kvm_is_vmx_supported()) + return -EOPNOTSUPP; + + /* + * Note, hv_init_evmcs() touches only VMX knobs, i.e. there's nothing + * to unwind if a later step fails. + */ + hv_init_evmcs(); + + r =3D kvm_x86_vendor_init(&vt_init_ops); + if (r) + return r; + + r =3D vmx_init(); + if (r) + goto err_vmx_init; + + /* + * Common KVM initialization _must_ come last, after this, /dev/kvm is + * exposed to userspace! + */ + vcpu_size =3D sizeof(struct vcpu_vmx); + vcpu_align =3D __alignof__(struct vcpu_vmx); + r =3D kvm_init(vcpu_size, vcpu_align, THIS_MODULE); + if (r) + goto err_kvm_init; + + return 0; + +err_kvm_init: + vmx_exit(); +err_vmx_init: + kvm_x86_vendor_exit(); + return r; +} +module_init(vt_init); + +static void vt_exit(void) +{ + kvm_exit(); + kvm_x86_vendor_exit(); + vmx_exit(); +} +module_exit(vt_exit); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 9b035d5571fe..8ff2323181fd 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -554,7 +554,7 @@ static int hv_enable_l2_tlb_flush(struct kvm_vcpu *vcpu) return 0; } =20 -static __init void hv_init_evmcs(void) +__init void hv_init_evmcs(void) { int cpu; =20 @@ -590,7 +590,7 @@ static __init void hv_init_evmcs(void) } } =20 -static void hv_reset_evmcs(void) +void hv_reset_evmcs(void) { struct hv_vp_assist_page *vp_ap; =20 @@ -614,10 +614,6 @@ static void hv_reset_evmcs(void) vp_ap->current_nested_vmcs =3D 0; vp_ap->enlighten_vmentry =3D 0; } - -#else /* IS_ENABLED(CONFIG_HYPERV) */ -static void hv_init_evmcs(void) {} -static void hv_reset_evmcs(void) {} #endif /* IS_ENABLED(CONFIG_HYPERV) */ =20 /* @@ -2712,7 +2708,7 @@ static int setup_vmcs_config(struct vmcs_config *vmcs= _conf, return 0; } =20 -static bool kvm_is_vmx_supported(void) +bool kvm_is_vmx_supported(void) { int cpu =3D raw_smp_processor_id(); =20 @@ -8429,7 +8425,7 @@ static void vmx_cleanup_l1d_flush(void) l1tf_vmx_mitigation =3D VMENTER_L1D_FLUSH_AUTO; } =20 -static void __vmx_exit(void) +void vmx_exit(void) { allow_smaller_maxphyaddr =3D false; =20 @@ -8440,32 +8436,10 @@ static void __vmx_exit(void) vmx_cleanup_l1d_flush(); } =20 -static void vmx_exit(void) -{ - kvm_exit(); - kvm_x86_vendor_exit(); - - __vmx_exit(); -} -module_exit(vmx_exit); - -static int __init vmx_init(void) +int __init vmx_init(void) { int r, cpu; =20 - if (!kvm_is_vmx_supported()) - return -EOPNOTSUPP; - - /* - * Note, hv_init_evmcs() touches only VMX knobs, i.e. there's nothing - * to unwind if a later step fails. - */ - hv_init_evmcs(); - - r =3D kvm_x86_vendor_init(&vt_init_ops); - if (r) - return r; - /* * Must be called after common x86 init so enable_ept is properly set * up. Hand the parameter mitigation value in which was stored in @@ -8475,7 +8449,7 @@ static int __init vmx_init(void) */ r =3D vmx_setup_l1d_flush(vmentry_l1d_flush_param); if (r) - goto err_l1d_flush; + return r; =20 vmx_setup_fb_clear_ctrl(); =20 @@ -8496,21 +8470,5 @@ static int __init vmx_init(void) if (!enable_ept) allow_smaller_maxphyaddr =3D true; =20 - /* - * Common KVM initialization _must_ come last, after this, /dev/kvm is - * exposed to userspace! - */ - r =3D kvm_init(sizeof(struct vcpu_vmx), __alignof__(struct vcpu_vmx), - THIS_MODULE); - if (r) - goto err_kvm_init; - return 0; - -err_kvm_init: - __vmx_exit(); -err_l1d_flush: - kvm_x86_vendor_exit(); - return r; } -module_init(vmx_init); diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index c15c64e99de3..41ae943c62cb 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -8,11 +8,22 @@ =20 #include "x86.h" =20 -__init int vmx_hardware_setup(void); +#if IS_ENABLED(CONFIG_HYPERV) +__init void hv_init_evmcs(void); +void hv_reset_evmcs(void); +#else /* IS_ENABLED(CONFIG_HYPERV) */ +static inline void hv_init_evmcs(void) {} +static inline void hv_reset_evmcs(void) {} +#endif /* IS_ENABLED(CONFIG_HYPERV) */ + +bool kvm_is_vmx_supported(void); +int __init vmx_init(void); +void vmx_exit(void); =20 extern struct kvm_x86_ops vt_x86_ops __initdata; extern struct kvm_x86_init_ops vt_init_ops __initdata; =20 +__init int vmx_hardware_setup(void); void vmx_hardware_unsetup(void); int vmx_check_processor_compat(void); int vmx_hardware_enable(void); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA0DBC04A94 for ; Tue, 25 Jul 2023 22:15:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230334AbjGYWPk (ORCPT ); Tue, 25 Jul 2023 18:15:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229762AbjGYWPh (ORCPT ); Tue, 25 Jul 2023 18:15:37 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59D97F5; Tue, 25 Jul 2023 15:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323335; x=1721859335; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pKZCqPO3oZUeDR6rPcORqpsgOwSf9m2DrMDdZUUZCs4=; b=PpqnQugjP62DkT/M7HKVRh2Oa1JTv6OYXBYbtJBD9bhijec1nJFHtmH2 Z+qTw/aokptNSnXadTrOYXwECYg6tpvuo7d8gYHAj1And3/CL3x2KTDL0 c5JEs0yTyGl8LEdBKoLMekswlqdO5HxXFu4tWHiVphQuMQrljG8odf2XD zSpXUz7Nr780t4jX+so8sNJ6DByHEPPvgOEpxQqn01MTxIzTHYPt9rBmk ai64JJNhRwxgiBjfs69xF+XI/jWUFPgdFDEzcuz/mTjumAySbiN3iRdWp z9YbHqUXq2X8Kk/YKI5kKFJMVg2bePv8bgeG4vQP4xYvLLMQWYkMgxc56 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863031" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863031" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938775" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938775" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:16 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 004/115] KVM: VMX: Reorder vmx initialization with kvm vendor initialization Date: Tue, 25 Jul 2023 15:13:15 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata To match vmx_exit cleanup. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 27bfd3fcea09..deaba44c6bdf 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -179,11 +179,11 @@ static int __init vt_init(void) */ hv_init_evmcs(); =20 - r =3D kvm_x86_vendor_init(&vt_init_ops); + r =3D vmx_init(); if (r) - return r; + goto err_vmx_init; =20 - r =3D vmx_init(); + r =3D kvm_x86_vendor_init(&vt_init_ops); if (r) goto err_vmx_init; =20 @@ -200,9 +200,9 @@ static int __init vt_init(void) return 0; =20 err_kvm_init: - vmx_exit(); -err_vmx_init: kvm_x86_vendor_exit(); +err_vmx_init: + vmx_exit(); return r; } module_init(vt_init); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 165D2C05051 for ; Tue, 25 Jul 2023 22:16:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232217AbjGYWQO (ORCPT ); Tue, 25 Jul 2023 18:16:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230293AbjGYWPj (ORCPT ); Tue, 25 Jul 2023 18:15:39 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7592FE63; Tue, 25 Jul 2023 15:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323337; x=1721859337; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5a2WKImnOiYyHHrWolihB7s+ZRh5ja5wue/MjhxUwd0=; b=Gj952pRt8vOYTci5SPjYrU8ImY6I4e/FI42/hjXUukuAVzaMTnRZncKr BTPKwFulon3j7SvYZeJ5aXngDrDGG/+wA08TN34X9ul4oZiKElqV1EIQI MOOIPJzYbz+puXkxtKaLczR53OAAaDMeSHrz9MrXg7mkjCroBu8r3qHte ZEcFSmBR2mB32fu9lqDP2SQ3q4+U7KuF5iAbDS4LVK4UONBDKDh+Jls+n SRsTsLQbpapw7HJfBy/Hx0svOyDftss0xvFLB669q8D5hrHIKaeDftFyZ ysBV1mK6gb9SJEvOMD8VRYl8TNZCkrdyoOPxQMA+UNFi9rxXSW4ibjN3y A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863036" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863036" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938778" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938778" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:16 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 005/115] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module Date: Tue, 25 Jul 2023 15:13:16 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX requires several initialization steps for KVM to create guest TDs. Detect CPU feature, enable VMX (TDX is based on VMX) on all online CPUs, detect the TDX module availability, initialize it and disable VMX. To enable/disable VMX on all online CPUs, utilize vmx_hardware_enable/disable(). The method also initializes each CPU for TDX. TDX requires calling a TDX initialization function per logical processor (LP) before the LP uses TDX. When the CPU is becoming online, call the TDX LP initialization API. If it fails to initialize TDX, refuse CPU online for simplicity instead of TDX avoiding the failed LP. There are several options on when to initialize the TDX module. A.) kernel module loading time, B.) the first guest TD creation time. A.) was chosen. With B.), a user may hit an error of the TDX initialization when trying to create the first guest TD. The machine that fails to initialize the TDX module can't boot any guest TD further. Such failure is undesirable and a surprise because the user expects that the machine can accommodate guest TD, but not. So A.) is better than B.). Introduce a module parameter, kvm_intel.tdx, to explicitly enable TDX KVM support. It's off by default to keep the same behavior for those who don't use TDX. Implement hardware_setup method to detect TDX feature of CPU and initialize TDX module. Suggested-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/Makefile | 1 + arch/x86/kvm/vmx/main.c | 34 ++++++++++++++- arch/x86/kvm/vmx/tdx.c | 84 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 8 ++++ 4 files changed, 125 insertions(+), 2 deletions(-) create mode 100644 arch/x86/kvm/vmx/tdx.c diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 0e894ae23cbc..4b01ab842ab7 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -25,6 +25,7 @@ kvm-$(CONFIG_KVM_SMM) +=3D smm.o kvm-intel-y +=3D vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ vmx/hyperv.o vmx/nested.o vmx/posted_intr.o vmx/main.o kvm-intel-$(CONFIG_X86_SGX_KVM) +=3D vmx/sgx.o +kvm-intel-$(CONFIG_INTEL_TDX_HOST) +=3D vmx/tdx.o =20 kvm-amd-y +=3D svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \ svm/sev.o svm/hyperv.o diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index deaba44c6bdf..8eb5b77d3043 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -6,6 +6,36 @@ #include "nested.h" #include "pmu.h" =20 +static bool enable_tdx __ro_after_init; +module_param_named(tdx, enable_tdx, bool, 0444); + +static int vt_hardware_enable(void) +{ + int ret; + + ret =3D vmx_hardware_enable(); + if (ret || !enable_tdx) + return ret; + + ret =3D tdx_cpu_enable(); + if (ret) + vmx_hardware_disable(); + return ret; +} + +static __init int vt_hardware_setup(void) +{ + int ret; + + ret =3D vmx_hardware_setup(); + if (ret) + return ret; + + enable_tdx =3D enable_tdx && !tdx_hardware_setup(&vt_x86_ops); + + return 0; +} + #define VMX_REQUIRED_APICV_INHIBITS \ (BIT(APICV_INHIBIT_REASON_DISABLE)| \ BIT(APICV_INHIBIT_REASON_ABSENT) | \ @@ -22,7 +52,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .hardware_unsetup =3D vmx_hardware_unsetup, =20 - .hardware_enable =3D vmx_hardware_enable, + .hardware_enable =3D vt_hardware_enable, .hardware_disable =3D vmx_hardware_disable, .has_emulated_msr =3D vmx_has_emulated_msr, =20 @@ -158,7 +188,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { }; =20 struct kvm_x86_init_ops vt_init_ops __initdata =3D { - .hardware_setup =3D vmx_hardware_setup, + .hardware_setup =3D vt_hardware_setup, .handle_intel_pt_intr =3D NULL, =20 .runtime_ops =3D &vt_x86_ops, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c new file mode 100644 index 000000000000..8a378fb6f1d4 --- /dev/null +++ b/arch/x86/kvm/vmx/tdx.c @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +#include + +#include "capabilities.h" +#include "x86_ops.h" +#include "x86.h" + +#undef pr_fmt +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +static int __init tdx_module_setup(void) +{ + int ret; + + ret =3D tdx_enable(); + if (ret) { + pr_info("Failed to initialize TDX module.\n"); + return ret; + } + + return 0; +} + +struct vmx_tdx_enabled { + cpumask_var_t vmx_enabled; + atomic_t err; +}; + +static void __init vmx_tdx_on(void *_vmx_tdx) +{ + struct vmx_tdx_enabled *vmx_tdx =3D _vmx_tdx; + int r; + + r =3D vmx_hardware_enable(); + if (!r) { + cpumask_set_cpu(smp_processor_id(), vmx_tdx->vmx_enabled); + r =3D tdx_cpu_enable(); + } + if (r) + atomic_set(&vmx_tdx->err, r); +} + +static void __init vmx_off(void *_vmx_enabled) +{ + cpumask_var_t *vmx_enabled =3D (cpumask_var_t *)_vmx_enabled; + + if (cpumask_test_cpu(smp_processor_id(), *vmx_enabled)) + vmx_hardware_disable(); +} + +int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops) +{ + struct vmx_tdx_enabled vmx_tdx =3D { + .err =3D ATOMIC_INIT(0), + }; + int r =3D 0; + + if (!enable_ept) { + pr_warn("Cannot enable TDX with EPT disabled\n"); + return -EINVAL; + } + + if (!zalloc_cpumask_var(&vmx_tdx.vmx_enabled, GFP_KERNEL)) { + r =3D -ENOMEM; + goto out; + } + + /* tdx_enable() in tdx_module_setup() requires cpus lock. */ + cpus_read_lock(); + on_each_cpu(vmx_tdx_on, &vmx_tdx, true); /* TDX requires vmxon. */ + r =3D atomic_read(&vmx_tdx.err); + if (!r) + r =3D tdx_module_setup(); + else + r =3D -EIO; + on_each_cpu(vmx_off, &vmx_tdx.vmx_enabled, true); + cpus_read_unlock(); + free_cpumask_var(vmx_tdx.vmx_enabled); + +out: + return r; +} diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 41ae943c62cb..ab1b50dcf178 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -20,6 +20,8 @@ bool kvm_is_vmx_supported(void); int __init vmx_init(void); void vmx_exit(void); =20 +__init int vmx_hardware_setup(void); + extern struct kvm_x86_ops vt_x86_ops __initdata; extern struct kvm_x86_init_ops vt_init_ops __initdata; =20 @@ -134,4 +136,10 @@ void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu); #endif void vmx_setup_mce(struct kvm_vcpu *vcpu); =20 +#ifdef CONFIG_INTEL_TDX_HOST +int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); +#else +static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } +#endif + #endif /* __KVM_X86_VMX_X86_OPS_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8237C04FE0 for ; Tue, 25 Jul 2023 22:16:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231419AbjGYWQV (ORCPT ); Tue, 25 Jul 2023 18:16:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230310AbjGYWPk (ORCPT ); Tue, 25 Jul 2023 18:15:40 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1429B6; Tue, 25 Jul 2023 15:15:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323338; x=1721859338; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Lc1h0/T67bD0JuzjGDOrH1LLcY2dcJag5qN+PDL2L3M=; b=REcnl0/N+N76CMOuLA41yK5IrtXoAP+kt+So44Wz6kcOH+r11/4HGO7Q HVHA14Ako8Gtt4H2rFM7OLO3wkw1Y7PVDURhvaXFBagLRTtvXzNQw3FKh l/mtbT3jvziOLJMjv0zkNoekFR68l/P+XUJad7ftPbQqhUYvvqFts1Jjx DpMSy8U93UJLPBIwMNvcGjexUpzRQg7RlopIupiRyEosHwyFfRaraRWM4 pXusyBuwUIrfAukxl5ys3sb8+sWWkuK0Zqe7HCykXXCb8uYQPqgbI7x/Z NxZs0lJkj7kdA+Xpm6IzKhxA6ooU54+XWhYkuFzdyxuK4GajyQ15fcj+B g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863044" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863044" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938782" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938782" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:17 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 006/115] KVM: TDX: Add placeholders for TDX VM/vcpu structure Date: Tue, 25 Jul 2023 15:13:17 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Add placeholders TDX VM/vcpu structure that overlays with VMX VM/vcpu structures. Initialize VM structure size and vcpu size/align so that x86 KVM common code knows those size irrespective of VMX or TDX. Those structures will be populated as guest creation logic develops. Add helper functions to check if the VM is guest TD and add conversion functions between KVM VM/VCPU and TDX VM/VCPU. Signed-off-by: Isaku Yamahata --- v14 -> v15: - use KVM_X86_TDX_VM Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 18 +++++++++++++-- arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/vmx/tdx.h | 50 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) create mode 100644 arch/x86/kvm/vmx/tdx.h diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 8eb5b77d3043..11ecc231f9c4 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -5,6 +5,7 @@ #include "vmx.h" #include "nested.h" #include "pmu.h" +#include "tdx.h" =20 static bool enable_tdx __ro_after_init; module_param_named(tdx, enable_tdx, bool, 0444); @@ -209,6 +210,21 @@ static int __init vt_init(void) */ hv_init_evmcs(); =20 + /* + * kvm_x86_ops is updated with vt_x86_ops. vt_x86_ops.vm_size must + * be set before kvm_x86_vendor_init(). + */ + vcpu_size =3D sizeof(struct vcpu_vmx); + vcpu_align =3D __alignof__(struct vcpu_vmx); + if (enable_tdx) { + vt_x86_ops.vm_size =3D max_t(unsigned int, vt_x86_ops.vm_size, + sizeof(struct kvm_tdx)); + vcpu_size =3D max_t(unsigned int, vcpu_size, + sizeof(struct vcpu_tdx)); + vcpu_align =3D max_t(unsigned int, vcpu_align, + __alignof__(struct vcpu_tdx)); + } + r =3D vmx_init(); if (r) goto err_vmx_init; @@ -221,8 +237,6 @@ static int __init vt_init(void) * Common KVM initialization _must_ come last, after this, /dev/kvm is * exposed to userspace! */ - vcpu_size =3D sizeof(struct vcpu_vmx); - vcpu_align =3D __alignof__(struct vcpu_vmx); r =3D kvm_init(vcpu_size, vcpu_align, THIS_MODULE); if (r) goto err_kvm_init; diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 8a378fb6f1d4..1c9884164566 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -6,6 +6,7 @@ #include "capabilities.h" #include "x86_ops.h" #include "x86.h" +#include "tdx.h" =20 #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h new file mode 100644 index 000000000000..473013265bd8 --- /dev/null +++ b/arch/x86/kvm/vmx/tdx.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_TDX_H +#define __KVM_X86_TDX_H + +#ifdef CONFIG_INTEL_TDX_HOST +struct kvm_tdx { + struct kvm kvm; + /* TDX specific members follow. */ +}; + +struct vcpu_tdx { + struct kvm_vcpu vcpu; + /* TDX specific members follow. */ +}; + +static inline bool is_td(struct kvm *kvm) +{ + return kvm->arch.vm_type =3D=3D KVM_X86_TDX_VM; +} + +static inline bool is_td_vcpu(struct kvm_vcpu *vcpu) +{ + return is_td(vcpu->kvm); +} + +static inline struct kvm_tdx *to_kvm_tdx(struct kvm *kvm) +{ + return container_of(kvm, struct kvm_tdx, kvm); +} + +static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) +{ + return container_of(vcpu, struct vcpu_tdx, vcpu); +} +#else +struct kvm_tdx { + struct kvm kvm; +}; + +struct vcpu_tdx { + struct kvm_vcpu vcpu; +}; + +static inline bool is_td(struct kvm *kvm) { return false; } +static inline bool is_td_vcpu(struct kvm_vcpu *vcpu) { return false; } +static inline struct kvm_tdx *to_kvm_tdx(struct kvm *kvm) { return NULL; } +static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *vcpu) { return NULL= ; } +#endif /* CONFIG_INTEL_TDX_HOST */ + +#endif /* __KVM_X86_TDX_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 559AEC04A6A for ; Tue, 25 Jul 2023 22:16:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231994AbjGYWQf (ORCPT ); Tue, 25 Jul 2023 18:16:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230461AbjGYWPk (ORCPT ); Tue, 25 Jul 2023 18:15:40 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F78EE0; Tue, 25 Jul 2023 15:15:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323339; x=1721859339; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iYKlzEZfVdp6EBGW2nCCTrJYsg848Ln6Mn5S9XVlZdg=; b=MLhW9l8bK+gefx1chxg6jQG34Qpg1jZGFBoK/gHMwajuPD0h208DQlW9 ICLYB9a4OOVRNKtb4C2bWqUy8I3GnbT/jtZ1hZxh6TwHRPKbrPrb4YyEB N7tc/+oYmEJ+UkGoKoYTc5amO80fxr7SdNptpinOzNfsG0Rdu3W0gvcpa HNwq9Z2bn8frqXGMdHxSqbMtz9pKG9khj+q5D/o7AIteJaONUOwWmqaJM //U3N4/qn2oEF+OYWOELPjCMx2IqQ68xLj+eWK6GWeQPkTTIP0kdN4NVR vCIsodhTLTdN7OglYBnU1fdNGz4hfS1jGUpz1bDq+t84xpQ/SJ3nODg9M g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863055" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863055" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938785" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938785" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:17 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 007/115] KVM: TDX: Make TDX VM type supported Date: Tue, 25 Jul 2023 15:13:18 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata NOTE: This patch is in position of the patch series for developers to be able to test codes during the middle of the patch series although this patch series doesn't provide functional features until the all the patches of this patch series. When merging this patch series, this patch can be moved to the end. As first step TDX VM support, return that TDX VM type supported to device model, e.g. qemu. The callback to create guest TD is vm_init callback for KVM_CREATE_VM. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 18 ++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 6 ++++++ arch/x86/kvm/vmx/vmx.c | 6 ------ arch/x86/kvm/vmx/x86_ops.h | 3 ++- 4 files changed, 24 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 11ecc231f9c4..9619473fba01 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -10,6 +10,12 @@ static bool enable_tdx __ro_after_init; module_param_named(tdx, enable_tdx, bool, 0444); =20 +static bool vt_is_vm_type_supported(unsigned long type) +{ + return __kvm_is_vm_type_supported(type) || + (enable_tdx && tdx_is_vm_type_supported(type)); +} + static int vt_hardware_enable(void) { int ret; @@ -37,6 +43,14 @@ static __init int vt_hardware_setup(void) return 0; } =20 +static int vt_vm_init(struct kvm *kvm) +{ + if (is_td(kvm)) + return -EOPNOTSUPP; /* Not ready to create guest TD yet. */ + + return vmx_vm_init(kvm); +} + #define VMX_REQUIRED_APICV_INHIBITS \ (BIT(APICV_INHIBIT_REASON_DISABLE)| \ BIT(APICV_INHIBIT_REASON_ABSENT) | \ @@ -57,9 +71,9 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .hardware_disable =3D vmx_hardware_disable, .has_emulated_msr =3D vmx_has_emulated_msr, =20 - .is_vm_type_supported =3D vmx_is_vm_type_supported, + .is_vm_type_supported =3D vt_is_vm_type_supported, .vm_size =3D sizeof(struct kvm_vmx), - .vm_init =3D vmx_vm_init, + .vm_init =3D vt_vm_init, .vm_destroy =3D vmx_vm_destroy, =20 .vcpu_precreate =3D vmx_vcpu_precreate, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 1c9884164566..9d3f593eacb8 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -24,6 +24,12 @@ static int __init tdx_module_setup(void) return 0; } =20 +bool tdx_is_vm_type_supported(unsigned long type) +{ + /* enable_tdx check is done by the caller. */ + return type =3D=3D KVM_X86_TDX_VM; +} + struct vmx_tdx_enabled { cpumask_var_t vmx_enabled; atomic_t err; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 8ff2323181fd..76e444c3e865 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7501,12 +7501,6 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu) return err; } =20 -bool vmx_is_vm_type_supported(unsigned long type) -{ - /* TODO: Check if TDX is supported. */ - return __kvm_is_vm_type_supported(type); -} - #define L1TF_MSG_SMT "L1TF CPU bug present and SMT on, data leak possible.= See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/h= w-vuln/l1tf.html for details.\n" #define L1TF_MSG_L1D "L1TF CPU bug present and virtualization mitigation d= isabled, data leak possible. See CVE-2018-3646 and https://www.kernel.org/d= oc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n" =20 diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index ab1b50dcf178..32a5c2629145 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -30,7 +30,6 @@ void vmx_hardware_unsetup(void); int vmx_check_processor_compat(void); int vmx_hardware_enable(void); void vmx_hardware_disable(void); -bool vmx_is_vm_type_supported(unsigned long type); int vmx_vm_init(struct kvm *kvm); void vmx_vm_destroy(struct kvm *kvm); int vmx_vcpu_precreate(struct kvm *kvm); @@ -138,8 +137,10 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu); =20 #ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); +bool tdx_is_vm_type_supported(unsigned long type); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } +static inline bool tdx_is_vm_type_supported(unsigned long type) { return f= alse; } #endif =20 #endif /* __KVM_X86_VMX_X86_OPS_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EBCAC04A6A for ; Tue, 25 Jul 2023 22:16:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229762AbjGYWQR (ORCPT ); Tue, 25 Jul 2023 18:16:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230302AbjGYWPk (ORCPT ); Tue, 25 Jul 2023 18:15:40 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E774CBC; Tue, 25 Jul 2023 15:15:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323339; x=1721859339; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gZq/xU5HPMr+7bL2m3UNHQGfAGkvgf+PBtN9vzhZEYs=; b=bbAnykT58VlRFXD557fX+Ty+UylOIROmYHgWGUEIwJwhw2QoN33BbUMp flwVloR5yCC9O0ptDLqTyHjoXk1DIC4VidF8FTRy0yv33BU5Rj/I04uv9 6838r1pGgQzaZJrTUsFNi0GDZt2JsA+F9lim772ceYg9fWaKTZabPvcM7 jfO9SwRPLUjgT9bqGlr7MY+dipu4QuF+iIY3szsEPvmYdy4BlzTGuCwOz do/BfkDxRQmBuPz8wdy4RnWwd27tefIX1C9+9nr97sSfGsTF8vX9nc8hp fzCCtq8qk/kTnHSc0SAyoslkaSdsh6TtSB9291XXpdY+rfTCOKgBthuAP A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863063" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863063" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938788" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938788" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:18 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 008/115] [MARKER] The start of TDX KVM patch series: TDX architectural definitions Date: Tue, 25 Jul 2023 15:13:19 -0700 Message-Id: <9419e35182ec6a9403a632e1bba919a01a3b4cc4.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TDX architectural definitions. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/index.rst | 2 ++ .../virt/kvm/intel-tdx-layer-status.rst | 29 +++++++++++++++++++ 2 files changed, 31 insertions(+) create mode 100644 Documentation/virt/kvm/intel-tdx-layer-status.rst diff --git a/Documentation/virt/kvm/index.rst b/Documentation/virt/kvm/inde= x.rst index ad13ec55ddfe..ccff56dca2b1 100644 --- a/Documentation/virt/kvm/index.rst +++ b/Documentation/virt/kvm/index.rst @@ -19,3 +19,5 @@ KVM vcpu-requests halt-polling review-checklist + + intel-tdx-layer-status diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst new file mode 100644 index 000000000000..f11ea701dc19 --- /dev/null +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Intel Trust Dodmain Extensions(TDX) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Layer status +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +What qemu can do +---------------- +- TDX VM TYPE is exposed to Qemu. +- Qemu can try to create VM of TDX VM type and then fails. + +Patch Layer status +------------------ + Patch layer Status + +* TDX, VMX coexistence: Applied +* TDX architectural definitions: Applying +* TD VM creation/destruction: Not yet +* TD vcpu creation/destruction: Not yet +* TDX EPT violation: Not yet +* TD finalization: Not yet +* TD vcpu enter/exit: Not yet +* TD vcpu interrupts/exit/hypercall: Not yet + +* KVM MMU GPA shared bits: Not yet +* KVM TDP refactoring for TDX: Not yet +* KVM TDP MMU hooks: Not yet --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93D97C001DF for ; Tue, 25 Jul 2023 22:16:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231663AbjGYWQ1 (ORCPT ); Tue, 25 Jul 2023 18:16:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbjGYWPl (ORCPT ); Tue, 25 Jul 2023 18:15:41 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07CCB137; Tue, 25 Jul 2023 15:15:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323340; x=1721859340; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XWf25yfdMbsBdduhmI3HG8soYrr5G48MYSQTf93wnMU=; b=I//q6vbE08Wv/VpBlZMnlE30qwSrIeUHQgdNLLFQXl1QFtd2SZ1e6Ptj PnNBoy4TcC9DWsxHRQdHxAOy5X22fPOHth1Wez8WZGbhXh7TBHJw1/Vc/ OJPoHAhhvnoQzegZOELK08pJTk1uRCR2ZtaQQjruzMmIRlF+T00jRm4nR ctT78HK3YOWOsPeHmGsrYUQ9NLdP4WHI3URoCJPQFiOCszWcUDXLmTg/0 /lO0dq1r6mWXXRCvqd+ndpYU0ux5tp1dolFe9i+kX35kzgJQuQMLCrmuD k7/UDN7cn5Ez+A/IUpuToitffSWMBNsos+bvw21+jADv65X7GupQbD+pd g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863070" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863070" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938791" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938791" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:18 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 009/115] KVM: TDX: Define TDX architectural definitions Date: Tue, 25 Jul 2023 15:13:20 -0700 Message-Id: <881e01c7f11d2c56a3c122a744f5da41bfe27ad7.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Define architectural definitions for KVM to issue the TDX SEAMCALLs. Structures and values that are architecturally defined in the TDX module specifications the chapter of ABI Reference. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx_arch.h | 168 ++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 arch/x86/kvm/vmx/tdx_arch.h diff --git a/arch/x86/kvm/vmx/tdx_arch.h b/arch/x86/kvm/vmx/tdx_arch.h new file mode 100644 index 000000000000..942a0e561a7b --- /dev/null +++ b/arch/x86/kvm/vmx/tdx_arch.h @@ -0,0 +1,168 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* architectural constants/data definitions for TDX SEAMCALLs */ + +#ifndef __KVM_X86_TDX_ARCH_H +#define __KVM_X86_TDX_ARCH_H + +#include + +/* + * TDX SEAMCALL API function leaves + */ +#define TDH_VP_ENTER 0 +#define TDH_MNG_ADDCX 1 +#define TDH_MEM_PAGE_ADD 2 +#define TDH_MEM_SEPT_ADD 3 +#define TDH_VP_ADDCX 4 +#define TDH_MEM_PAGE_RELOCATE 5 +#define TDH_MEM_PAGE_AUG 6 +#define TDH_MEM_RANGE_BLOCK 7 +#define TDH_MNG_KEY_CONFIG 8 +#define TDH_MNG_CREATE 9 +#define TDH_VP_CREATE 10 +#define TDH_MNG_RD 11 +#define TDH_MR_EXTEND 16 +#define TDH_MR_FINALIZE 17 +#define TDH_VP_FLUSH 18 +#define TDH_MNG_VPFLUSHDONE 19 +#define TDH_MNG_KEY_FREEID 20 +#define TDH_MNG_INIT 21 +#define TDH_VP_INIT 22 +#define TDH_VP_RD 26 +#define TDH_MNG_KEY_RECLAIMID 27 +#define TDH_PHYMEM_PAGE_RECLAIM 28 +#define TDH_MEM_PAGE_REMOVE 29 +#define TDH_MEM_SEPT_REMOVE 30 +#define TDH_MEM_TRACK 38 +#define TDH_MEM_RANGE_UNBLOCK 39 +#define TDH_PHYMEM_CACHE_WB 40 +#define TDH_PHYMEM_PAGE_WBINVD 41 +#define TDH_VP_WR 43 +#define TDH_SYS_LP_SHUTDOWN 44 + +#define TDG_VP_VMCALL_GET_TD_VM_CALL_INFO 0x10000 +#define TDG_VP_VMCALL_MAP_GPA 0x10001 +#define TDG_VP_VMCALL_GET_QUOTE 0x10002 +#define TDG_VP_VMCALL_REPORT_FATAL_ERROR 0x10003 +#define TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT 0x10004 + +/* TDX control structure (TDR/TDCS/TDVPS) field access codes */ +#define TDX_NON_ARCH BIT_ULL(63) +#define TDX_CLASS_SHIFT 56 +#define TDX_FIELD_MASK GENMASK_ULL(31, 0) + +#define __BUILD_TDX_FIELD(non_arch, class, field) \ + (((non_arch) ? TDX_NON_ARCH : 0) | \ + ((u64)(class) << TDX_CLASS_SHIFT) | \ + ((u64)(field) & TDX_FIELD_MASK)) + +#define BUILD_TDX_FIELD(class, field) \ + __BUILD_TDX_FIELD(false, (class), (field)) + +#define BUILD_TDX_FIELD_NON_ARCH(class, field) \ + __BUILD_TDX_FIELD(true, (class), (field)) + + +/* Class code for TD */ +#define TD_CLASS_EXECUTION_CONTROLS 17ULL + +/* Class code for TDVPS */ +#define TDVPS_CLASS_VMCS 0ULL +#define TDVPS_CLASS_GUEST_GPR 16ULL +#define TDVPS_CLASS_OTHER_GUEST 17ULL +#define TDVPS_CLASS_MANAGEMENT 32ULL + +enum tdx_tdcs_execution_control { + TD_TDCS_EXEC_TSC_OFFSET =3D 10, +}; + +/* @field is any of enum tdx_tdcs_execution_control */ +#define TDCS_EXEC(field) BUILD_TDX_FIELD(TD_CLASS_EXECUTION_CONTROLS, (fi= eld)) + +/* @field is the VMCS field encoding */ +#define TDVPS_VMCS(field) BUILD_TDX_FIELD(TDVPS_CLASS_VMCS, (field)) + +enum tdx_vcpu_guest_other_state { + TD_VCPU_STATE_DETAILS_NON_ARCH =3D 0x100, +}; + +union tdx_vcpu_state_details { + struct { + u64 vmxip : 1; + u64 reserved : 63; + }; + u64 full; +}; + +/* @field is any of enum tdx_guest_other_state */ +#define TDVPS_STATE(field) BUILD_TDX_FIELD(TDVPS_CLASS_OTHER_GUEST, (fiel= d)) +#define TDVPS_STATE_NON_ARCH(field) BUILD_TDX_FIELD_NON_ARCH(TDVPS_CLASS_O= THER_GUEST, (field)) + +/* Management class fields */ +enum tdx_vcpu_guest_management { + TD_VCPU_PEND_NMI =3D 11, +}; + +/* @field is any of enum tdx_vcpu_guest_management */ +#define TDVPS_MANAGEMENT(field) BUILD_TDX_FIELD(TDVPS_CLASS_MANAGEMENT, (= field)) + +#define TDX_EXTENDMR_CHUNKSIZE 256 + +struct tdx_cpuid_value { + u32 eax; + u32 ebx; + u32 ecx; + u32 edx; +} __packed; + +#define TDX_TD_ATTRIBUTE_DEBUG BIT_ULL(0) +#define TDX_TD_ATTRIBUTE_PKS BIT_ULL(30) +#define TDX_TD_ATTRIBUTE_KL BIT_ULL(31) +#define TDX_TD_ATTRIBUTE_PERFMON BIT_ULL(63) + +/* + * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is= 1024B. + */ +#define TDX_MAX_VCPUS (~(u16)0) + +struct td_params { + u64 attributes; + u64 xfam; + u16 max_vcpus; + u8 reserved0[6]; + + u64 eptp_controls; + u64 exec_controls; + u16 tsc_frequency; + u8 reserved1[38]; + + u64 mrconfigid[6]; + u64 mrowner[6]; + u64 mrownerconfig[6]; + u64 reserved2[4]; + + union { + struct tdx_cpuid_value cpuid_values[0]; + u8 reserved3[768]; + }; +} __packed __aligned(1024); + +/* + * Guest uses MAX_PA for GPAW when set. + * 0: GPA.SHARED bit is GPA[47] + * 1: GPA.SHARED bit is GPA[51] + */ +#define TDX_EXEC_CONTROL_MAX_GPAW BIT_ULL(0) + +/* + * TDX requires the frequency to be defined in units of 25MHz, which is the + * frequency of the core crystal clock on TDX-capable platforms, i.e. the = TDX + * module can only program frequencies that are multiples of 25MHz. The + * frequency must be between 100mhz and 10ghz (inclusive). + */ +#define TDX_TSC_KHZ_TO_25MHZ(tsc_in_khz) ((tsc_in_khz) / (25 * 1000)) +#define TDX_TSC_25MHZ_TO_KHZ(tsc_in_25mhz) ((tsc_in_25mhz) * (25 * 1000)) +#define TDX_MIN_TSC_FREQUENCY_KHZ (100 * 1000) +#define TDX_MAX_TSC_FREQUENCY_KHZ (10 * 1000 * 1000) + +#endif /* __KVM_X86_TDX_ARCH_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68D3CC001E0 for ; Tue, 25 Jul 2023 22:16:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231739AbjGYWQX (ORCPT ); Tue, 25 Jul 2023 18:16:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231324AbjGYWPm (ORCPT ); Tue, 25 Jul 2023 18:15:42 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB30F10D4; Tue, 25 Jul 2023 15:15:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323341; x=1721859341; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jsnC5w7xoBmZgcR+lzWktph1m0Tek+3YysP7GWy/DeA=; b=BIkVv5+yCqEu8I9XxI9NuM3GwSyLlv8hN6n+XGYVcxp/Ak0+D8lPh1GQ JOkFg1G1kU+E7B2Tpxn4fQWPlVKcm7fltPSMQ2cqr06Hpad+lx+TTA03J r7nKPVA3SnjDH3RJacbOawREOkbvw4P4zZnD6orb/aEol1RpFrgfemfjA GpsROJ/eK6tMbtAeyDWw9D+2XLIIETZaCh0spFANrG6lKcyrAXy/FRNz1 ColuXk+o027o6wpipM3ozzWXGVDAcisUwBCvehZ+H8DPggwchuChVc/NK 6c+z/lVXsocLKjWRuymjjs1rOTrsNtEhoBb9UhSBg3GZGsHG/bnW2e2FL Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863074" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863074" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938794" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938794" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:19 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 010/115] KVM: TDX: Add TDX "architectural" error codes Date: Tue, 25 Jul 2023 15:13:21 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Add error codes for the TDX SEAMCALLs both for TDX VMM side for TDH SEAMCALL and TDX guest side for TDG.VP.VMCALL. KVM issues the TDX SEAMCALLs and checks its error code. KVM handles hypercall from the TDX guest and may return an error. So error code for the TDX guest is also needed. TDX SEAMCALL uses bits 31:0 to return more information, so these error codes will only exactly match RAX[63:32]. Error codes for TDG.VP.VMCALL is defined by TDX Guest-Host-Communication interface spec. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx_errno.h | 40 ++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 arch/x86/kvm/vmx/tdx_errno.h diff --git a/arch/x86/kvm/vmx/tdx_errno.h b/arch/x86/kvm/vmx/tdx_errno.h new file mode 100644 index 000000000000..56cfd2f558fa --- /dev/null +++ b/arch/x86/kvm/vmx/tdx_errno.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* architectural status code for SEAMCALL */ + +#ifndef __KVM_X86_TDX_ERRNO_H +#define __KVM_X86_TDX_ERRNO_H + +#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL + +/* + * TDX SEAMCALL Status Codes (returned in RAX) + */ +#define TDX_SUCCESS 0x0000000000000000ULL +#define TDX_NON_RECOVERABLE_VCPU 0x4000000100000000ULL +#define TDX_INTERRUPTED_RESUMABLE 0x8000000300000000ULL +#define TDX_OPERAND_INVALID 0xC000010000000000ULL +#define TDX_OPERAND_BUSY 0x8000020000000000ULL +#define TDX_VCPU_NOT_ASSOCIATED 0x8000070200000000ULL +#define TDX_KEY_GENERATION_FAILED 0x8000080000000000ULL +#define TDX_KEY_STATE_INCORRECT 0xC000081100000000ULL +#define TDX_KEY_CONFIGURED 0x0000081500000000ULL +#define TDX_NO_HKID_READY_TO_WBCACHE 0x0000082100000000ULL +#define TDX_EPT_WALK_FAILED 0xC0000B0000000000ULL + +/* + * TDG.VP.VMCALL Status Codes (returned in R10) + */ +#define TDG_VP_VMCALL_SUCCESS 0x0000000000000000ULL +#define TDG_VP_VMCALL_RETRY 0x0000000000000001ULL +#define TDG_VP_VMCALL_INVALID_OPERAND 0x8000000000000000ULL +#define TDG_VP_VMCALL_TDREPORT_FAILED 0x8000000000000001ULL + +/* + * TDX module operand ID, appears in 31:0 part of error code as + * detail information + */ +#define TDX_OPERAND_ID_RCX 0x01 +#define TDX_OPERAND_ID_SEPT 0x92 +#define TDX_OPERAND_ID_TD_EPOCH 0xa9 + +#endif /* __KVM_X86_TDX_ERRNO_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D14C1C0015E for ; Tue, 25 Jul 2023 22:16:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232131AbjGYWQr (ORCPT ); Tue, 25 Jul 2023 18:16:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231409AbjGYWPn (ORCPT ); Tue, 25 Jul 2023 18:15:43 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76EE31733; Tue, 25 Jul 2023 15:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323341; x=1721859341; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XCdxkHSNfG3a+eFTqVYn3k3O1d86etbzM4CeTXyKbwM=; b=cWrvujUUuIqHgCnORIEMpI8T7u8HfvNEdrWNkJ8r22h3lySGkpnTGrLH xfxXuLDnsKPHOsjJW42hBq4F7o5pyslM4MnLa+gEPymlKIqIzGY4w7etR 73NOVpzneJzem9cr63L8Zyn9vxprTbd83k76H0y8KO1G2onr0P2bEfCp4 TsY1ZnNQ04/SGvXKMmNk4Xp3x+kRGFVE1JWkwoTg/XcgNtflY0djVxYSL sceKVPBIhJZZwm5kKzpANqEwlQfO7IF+llKwR1G1KHeeXXkZR500VdFvi BTVxLI+vl7IJDo4ZF9BK0dY1pz2kBz/wftiYJsZ+nrPPB+Dlx2SPmC6zR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863080" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863080" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938797" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938797" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:19 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 011/115] KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module Date: Tue, 25 Jul 2023 15:13:22 -0700 Message-Id: <2d81a22fc1d641b3f66aec13e5d1ee13ad266857.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata A VMM interacts with the TDX module using a new instruction (SEAMCALL). For instance, a TDX VMM does not have full access to the VM control structure corresponding to VMX VMCS. Instead, a VMM induces the TDX module to act on behalf via SEAMCALLs. Export __seamcall and define C wrapper functions for SEAMCALLs for readability. Some SEAMCALL APIs donate host pages to TDX module or guest TD, and the donated pages are encrypted. Those require the VMM to flush the cache lines to avoid cache line alias. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/tdx.h | 4 + arch/x86/kvm/vmx/tdx_ops.h | 204 +++++++++++++++++++++++++++++++ arch/x86/virt/vmx/tdx/seamcall.S | 2 + arch/x86/virt/vmx/tdx/tdx.h | 3 - 4 files changed, 210 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kvm/vmx/tdx_ops.h diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index ed84211fe190..bf5324b5ea01 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -90,12 +90,16 @@ int tdx_cpu_enable(void); int tdx_enable(void); void tdx_reset_memory(void); bool tdx_is_private_mem(unsigned long phys); +u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, + struct tdx_module_output *out); #else /* !CONFIG_INTEL_TDX_HOST */ static inline bool platform_tdx_enabled(void) { return false; } static inline int tdx_cpu_enable(void) { return -ENODEV; } static inline int tdx_enable(void) { return -ENODEV; } static inline void tdx_reset_memory(void) { } static inline bool tdx_is_private_mem(unsigned long phys) { return false; } +static inline u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, + struct tdx_module_output *out) { return TDX_SEAMCALL_UD; }; #endif /* CONFIG_INTEL_TDX_HOST */ =20 #endif /* !__ASSEMBLY__ */ diff --git a/arch/x86/kvm/vmx/tdx_ops.h b/arch/x86/kvm/vmx/tdx_ops.h new file mode 100644 index 000000000000..76eddecdca12 --- /dev/null +++ b/arch/x86/kvm/vmx/tdx_ops.h @@ -0,0 +1,204 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* constants/data definitions for TDX SEAMCALLs */ + +#ifndef __KVM_X86_TDX_OPS_H +#define __KVM_X86_TDX_OPS_H + +#include + +#include +#include +#include + +#include "tdx_errno.h" +#include "tdx_arch.h" +#include "x86.h" + +static inline u64 tdx_seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, + struct tdx_module_output *out) +{ + u64 ret; + + ret =3D __seamcall(op, rcx, rdx, r8, r9, out); + if (unlikely(ret =3D=3D TDX_SEAMCALL_UD)) { + /* + * SEAMCALLs fail with TDX_SEAMCALL_UD returned when VMX is off. + * This can happen when the host gets rebooted or live + * updated. In this case, the instruction execution is ignored + * as KVM is shut down, so the error code is suppressed. Other + * than this, the error is unexpected and the execution can't + * continue as the TDX features reply on VMX to be on. + */ + kvm_spurious_fault(); + return 0; + } + return ret; +} + +static inline u64 tdh_mng_addcx(hpa_t tdr, hpa_t addr) +{ + clflush_cache_range(__va(addr), PAGE_SIZE); + return tdx_seamcall(TDH_MNG_ADDCX, addr, tdr, 0, 0, NULL); +} + +static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa, hpa_t hpa, hpa_t = source, + struct tdx_module_output *out) +{ + clflush_cache_range(__va(hpa), PAGE_SIZE); + return tdx_seamcall(TDH_MEM_PAGE_ADD, gpa, tdr, hpa, source, out); +} + +static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t = page, + struct tdx_module_output *out) +{ + clflush_cache_range(__va(page), PAGE_SIZE); + return tdx_seamcall(TDH_MEM_SEPT_ADD, gpa | level, tdr, page, 0, out); +} + +static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_MEM_SEPT_REMOVE, gpa | level, tdr, 0, 0, out); +} + +static inline u64 tdh_vp_addcx(hpa_t tdvpr, hpa_t addr) +{ + clflush_cache_range(__va(addr), PAGE_SIZE); + return tdx_seamcall(TDH_VP_ADDCX, addr, tdvpr, 0, 0, NULL); +} + +static inline u64 tdh_mem_page_relocate(hpa_t tdr, gpa_t gpa, hpa_t hpa, + struct tdx_module_output *out) +{ + clflush_cache_range(__va(hpa), PAGE_SIZE); + return tdx_seamcall(TDH_MEM_PAGE_RELOCATE, gpa, tdr, hpa, 0, out); +} + +static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gpa, hpa_t hpa, + struct tdx_module_output *out) +{ + clflush_cache_range(__va(hpa), PAGE_SIZE); + return tdx_seamcall(TDH_MEM_PAGE_AUG, gpa, tdr, hpa, 0, out); +} + +static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_MEM_RANGE_BLOCK, gpa | level, tdr, 0, 0, out); +} + +static inline u64 tdh_mng_key_config(hpa_t tdr) +{ + return tdx_seamcall(TDH_MNG_KEY_CONFIG, tdr, 0, 0, 0, NULL); +} + +static inline u64 tdh_mng_create(hpa_t tdr, int hkid) +{ + clflush_cache_range(__va(tdr), PAGE_SIZE); + return tdx_seamcall(TDH_MNG_CREATE, tdr, hkid, 0, 0, NULL); +} + +static inline u64 tdh_vp_create(hpa_t tdr, hpa_t tdvpr) +{ + clflush_cache_range(__va(tdvpr), PAGE_SIZE); + return tdx_seamcall(TDH_VP_CREATE, tdvpr, tdr, 0, 0, NULL); +} + +static inline u64 tdh_mng_rd(hpa_t tdr, u64 field, struct tdx_module_outpu= t *out) +{ + return tdx_seamcall(TDH_MNG_RD, tdr, field, 0, 0, out); +} + +static inline u64 tdh_mr_extend(hpa_t tdr, gpa_t gpa, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_MR_EXTEND, gpa, tdr, 0, 0, out); +} + +static inline u64 tdh_mr_finalize(hpa_t tdr) +{ + return tdx_seamcall(TDH_MR_FINALIZE, tdr, 0, 0, 0, NULL); +} + +static inline u64 tdh_vp_flush(hpa_t tdvpr) +{ + return tdx_seamcall(TDH_VP_FLUSH, tdvpr, 0, 0, 0, NULL); +} + +static inline u64 tdh_mng_vpflushdone(hpa_t tdr) +{ + return tdx_seamcall(TDH_MNG_VPFLUSHDONE, tdr, 0, 0, 0, NULL); +} + +static inline u64 tdh_mng_key_freeid(hpa_t tdr) +{ + return tdx_seamcall(TDH_MNG_KEY_FREEID, tdr, 0, 0, 0, NULL); +} + +static inline u64 tdh_mng_init(hpa_t tdr, hpa_t td_params, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_MNG_INIT, tdr, td_params, 0, 0, out); +} + +static inline u64 tdh_vp_init(hpa_t tdvpr, u64 rcx) +{ + return tdx_seamcall(TDH_VP_INIT, tdvpr, rcx, 0, 0, NULL); +} + +static inline u64 tdh_vp_rd(hpa_t tdvpr, u64 field, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_VP_RD, tdvpr, field, 0, 0, out); +} + +static inline u64 tdh_mng_key_reclaimid(hpa_t tdr) +{ + return tdx_seamcall(TDH_MNG_KEY_RECLAIMID, tdr, 0, 0, 0, NULL); +} + +static inline u64 tdh_phymem_page_reclaim(hpa_t page, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_PHYMEM_PAGE_RECLAIM, page, 0, 0, 0, out); +} + +static inline u64 tdh_mem_page_remove(hpa_t tdr, gpa_t gpa, int level, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_MEM_PAGE_REMOVE, gpa | level, tdr, 0, 0, out); +} + +static inline u64 tdh_sys_lp_shutdown(void) +{ + return tdx_seamcall(TDH_SYS_LP_SHUTDOWN, 0, 0, 0, 0, NULL); +} + +static inline u64 tdh_mem_track(hpa_t tdr) +{ + return tdx_seamcall(TDH_MEM_TRACK, tdr, 0, 0, 0, NULL); +} + +static inline u64 tdh_mem_range_unblock(hpa_t tdr, gpa_t gpa, int level, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_MEM_RANGE_UNBLOCK, gpa | level, tdr, 0, 0, out); +} + +static inline u64 tdh_phymem_cache_wb(bool resume) +{ + return tdx_seamcall(TDH_PHYMEM_CACHE_WB, resume ? 1 : 0, 0, 0, 0, NULL); +} + +static inline u64 tdh_phymem_page_wbinvd(hpa_t page) +{ + return tdx_seamcall(TDH_PHYMEM_PAGE_WBINVD, page, 0, 0, 0, NULL); +} + +static inline u64 tdh_vp_wr(hpa_t tdvpr, u64 field, u64 val, u64 mask, + struct tdx_module_output *out) +{ + return tdx_seamcall(TDH_VP_WR, tdvpr, field, val, mask, out); +} + +#endif /* __KVM_X86_TDX_OPS_H */ diff --git a/arch/x86/virt/vmx/tdx/seamcall.S b/arch/x86/virt/vmx/tdx/seamc= all.S index f81be6b9c133..b90a7fe05494 100644 --- a/arch/x86/virt/vmx/tdx/seamcall.S +++ b/arch/x86/virt/vmx/tdx/seamcall.S @@ -1,5 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include +#include #include =20 #include "tdxcall.S" @@ -50,3 +51,4 @@ SYM_FUNC_START(__seamcall) FRAME_END RET SYM_FUNC_END(__seamcall) +EXPORT_SYMBOL_GPL(__seamcall) diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h index 2fefd688924c..70315263d8d2 100644 --- a/arch/x86/virt/vmx/tdx/tdx.h +++ b/arch/x86/virt/vmx/tdx/tdx.h @@ -145,7 +145,4 @@ struct tdmr_info_list { int max_tdmrs; /* How many 'tdmr_info's are allocated */ }; =20 -struct tdx_module_output; -u64 __seamcall(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, - struct tdx_module_output *out); #endif --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FEAEEB64DD for ; Tue, 25 Jul 2023 22:16:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231475AbjGYWQl (ORCPT ); Tue, 25 Jul 2023 18:16:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231186AbjGYWPl (ORCPT ); Tue, 25 Jul 2023 18:15:41 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DF45E47; Tue, 25 Jul 2023 15:15:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323340; x=1721859340; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MkHfcpMwySOcGOfR6N8gvbfjq2Uw4EljfqsyyqkNNzc=; b=dKTqci6BL8cNhhwZQ/dgGUETXmdI6CyBD8rkg3FyvTH4tCMryM/DC8Pv E8ztLwB+oZ1TR1gCF+FO2lWowcLHmcD3utvRvo71KTYwUIVXgimQK3abr dSu2WiRu5RFKzZ7VvEUMZvIIDBQOHUL8ffzKvcPTMAhfV84nGvNRI03UO X3oUAAPVJQzAU6YwtP6an+YxhJOmOmzfGxEfYo86bWhOlNu8nzEkZ0wPZ Cm8IS3V67SPEVvpXBYwiJFt3B/wmsE/KwoldvNRT/aSditNRWXRaH1PXb 8apzf1KxVb/Ds8C7xdexUPumm2xDsvdB9Nmmm0gIWc/vyWdJZnwrhT1+/ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863087" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863087" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938803" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938803" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:19 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 012/115] KVM: TDX: Retry SEAMCALL on the lack of entropy error Date: Tue, 25 Jul 2023 15:13:23 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Some SEAMCALL may return TDX_RND_NO_ENTROPY error when the entropy is lacking. Retry SEAMCALL on the error following rdrand_long() to retry RDRAND_RETRY_LOOPS times. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx_errno.h | 1 + arch/x86/kvm/vmx/tdx_ops.h | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/tdx_errno.h b/arch/x86/kvm/vmx/tdx_errno.h index 56cfd2f558fa..53dc14ba9107 100644 --- a/arch/x86/kvm/vmx/tdx_errno.h +++ b/arch/x86/kvm/vmx/tdx_errno.h @@ -14,6 +14,7 @@ #define TDX_INTERRUPTED_RESUMABLE 0x8000000300000000ULL #define TDX_OPERAND_INVALID 0xC000010000000000ULL #define TDX_OPERAND_BUSY 0x8000020000000000ULL +#define TDX_RND_NO_ENTROPY 0x8000020300000000ULL #define TDX_VCPU_NOT_ASSOCIATED 0x8000070200000000ULL #define TDX_KEY_GENERATION_FAILED 0x8000080000000000ULL #define TDX_KEY_STATE_INCORRECT 0xC000081100000000ULL diff --git a/arch/x86/kvm/vmx/tdx_ops.h b/arch/x86/kvm/vmx/tdx_ops.h index 76eddecdca12..d588a5507f5a 100644 --- a/arch/x86/kvm/vmx/tdx_ops.h +++ b/arch/x86/kvm/vmx/tdx_ops.h @@ -6,6 +6,7 @@ =20 #include =20 +#include #include #include #include @@ -17,9 +18,14 @@ static inline u64 tdx_seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, struct tdx_module_output *out) { + int retry; u64 ret; =20 - ret =3D __seamcall(op, rcx, rdx, r8, r9, out); + /* Mimic the existing rdrand_long() to retry RDRAND_RETRY_LOOPS times. */ + retry =3D RDRAND_RETRY_LOOPS; + do { + ret =3D __seamcall(op, rcx, rdx, r8, r9, out); + } while (unlikely(ret =3D=3D TDX_RND_NO_ENTROPY) && --retry); if (unlikely(ret =3D=3D TDX_SEAMCALL_UD)) { /* * SEAMCALLs fail with TDX_SEAMCALL_UD returned when VMX is off. --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13530C04FDF for ; Tue, 25 Jul 2023 22:16:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231382AbjGYWQx (ORCPT ); Tue, 25 Jul 2023 18:16:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231414AbjGYWPn (ORCPT ); Tue, 25 Jul 2023 18:15:43 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DDCBBC; Tue, 25 Jul 2023 15:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323341; x=1721859341; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JMtVpqNAMjI4Ci0bJaSt8eNsqbuL48mvNaWm2+I8CxI=; b=hORLSQlRMTfETKEdi7pCzQ1BVE4mFrY8X0UkjPrN3y/Z7qQFFXqcQR58 fvOBMbBg4nuWZHJtrhQ7iaR+wk8eLJKEr+eiazN92CiPnVfKw5z9zhRD8 gMTZld0KzrworPRwCpvNjOJYCClYzYJZZ6ShBNMzdbc4Z4nsfG39LPt6M epQCEyh5TFXlJYocvVl7ZJMNrD+32Iaov5tA4iOiwW5uDo/PqZwDw6nC7 FtusNRvOe1mZWxK6RV/LztuqY+UIWwJVVZtuuCMnv20Lmm31UJJ2c/mZM bI90nwdqU5BG+pMKM+Nauw0zs9jL50YEDgf2nxAMWwtrBJz5vFT5eSZXQ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863093" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863093" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938806" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938806" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:20 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 013/115] KVM: TDX: Add helper functions to print TDX SEAMCALL error Date: Tue, 25 Jul 2023 15:13:24 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Add helper functions to print out errors from the TDX module in a uniform manner. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/vmx/tdx_error.c | 20 ++++++++++++++++++++ arch/x86/kvm/vmx/tdx_ops.h | 5 +++++ 3 files changed, 26 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kvm/vmx/tdx_error.c diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 4b01ab842ab7..e3354b784e10 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -25,7 +25,7 @@ kvm-$(CONFIG_KVM_SMM) +=3D smm.o kvm-intel-y +=3D vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ vmx/hyperv.o vmx/nested.o vmx/posted_intr.o vmx/main.o kvm-intel-$(CONFIG_X86_SGX_KVM) +=3D vmx/sgx.o -kvm-intel-$(CONFIG_INTEL_TDX_HOST) +=3D vmx/tdx.o +kvm-intel-$(CONFIG_INTEL_TDX_HOST) +=3D vmx/tdx.o vmx/tdx_error.o =20 kvm-amd-y +=3D svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \ svm/sev.o svm/hyperv.o diff --git a/arch/x86/kvm/vmx/tdx_error.c b/arch/x86/kvm/vmx/tdx_error.c new file mode 100644 index 000000000000..6459cbada713 --- /dev/null +++ b/arch/x86/kvm/vmx/tdx_error.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* functions to record TDX SEAMCALL error */ + +#include +#include + +#include "tdx_ops.h" + +void pr_tdx_error(u64 op, u64 error_code, const struct tdx_module_output *= out) +{ + if (!out) { + pr_err_ratelimited("SEAMCALL[%lld] failed: 0x%llx\n", + op, error_code); + return; + } + +#define MSG "SEAMCALL[%lld] failed: 0x%llx RCX 0x%llx RDX 0x%llx R8 0x%llx= R9 0x%llx R10 0x%llx R11 0x%llx\n" + pr_err_ratelimited(MSG, op, error_code, out->rcx, out->rdx, out->r8, + out->r9, out->r10, out->r11); +} diff --git a/arch/x86/kvm/vmx/tdx_ops.h b/arch/x86/kvm/vmx/tdx_ops.h index d588a5507f5a..9db19c0711a9 100644 --- a/arch/x86/kvm/vmx/tdx_ops.h +++ b/arch/x86/kvm/vmx/tdx_ops.h @@ -10,6 +10,7 @@ #include #include #include +#include =20 #include "tdx_errno.h" #include "tdx_arch.h" @@ -41,6 +42,10 @@ static inline u64 tdx_seamcall(u64 op, u64 rcx, u64 rdx,= u64 r8, u64 r9, return ret; } =20 +#ifdef CONFIG_INTEL_TDX_HOST +void pr_tdx_error(u64 op, u64 error_code, const struct tdx_module_output *= out); +#endif + static inline u64 tdh_mng_addcx(hpa_t tdr, hpa_t addr) { clflush_cache_range(__va(addr), PAGE_SIZE); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E520C41513 for ; Tue, 25 Jul 2023 22:16:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232086AbjGYWQo (ORCPT ); Tue, 25 Jul 2023 18:16:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231415AbjGYWPn (ORCPT ); Tue, 25 Jul 2023 18:15:43 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9565C1738; Tue, 25 Jul 2023 15:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323341; x=1721859341; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DLPNi9jXjbr4HdbjAnuHBOpBxwIVUIeOA5wtFcSAGOI=; b=UPI5p41bqGYTD1i1T5r4Y5Hc7S9u+3dL3ABRce3/xtFCEMjM4lxTu+JZ 51Boezb9j4KzD+Zkwzz+vTuR9UDl1Nho9QLJsd5PO/A+DCWedQ43EP+X/ A9vkG1s27SyFAZxJspn6bpGfSNOzcubbTZI0iFJSaDLPhQsX06Fvxf9qw g3TYmjFYujICCSKNLIdBcELkDvg7sjgFf7ViGYtfqnhOQu3oUCDpYpDfy EjWGKNbErBM83TmDfdrbozdoE6+hK3eDWlp9DbfOM7KQZX5zs/kNUDQC/ t7aPjQMwKMZJ+rnalHFsjm7a7z/DqP29MGiM6k/MOr9LtCbWWDD58+TRa g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863098" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863098" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938810" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938810" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:20 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 014/115] [MARKER] The start of TDX KVM patch series: TD VM creation/destruction Date: Tue, 25 Jul 2023 15:13:25 -0700 Message-Id: <860b9552ec678720560e6dc60123b7dae69e0642.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TD VM creation/destruction. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index f11ea701dc19..098150da6ea2 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -16,8 +16,8 @@ Patch Layer status Patch layer Status =20 * TDX, VMX coexistence: Applied -* TDX architectural definitions: Applying -* TD VM creation/destruction: Not yet +* TDX architectural definitions: Applied +* TD VM creation/destruction: Applying * TD vcpu creation/destruction: Not yet * TDX EPT violation: Not yet * TD finalization: Not yet --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FD2EEB64DD for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232116AbjGYWQz (ORCPT ); Tue, 25 Jul 2023 18:16:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231422AbjGYWPn (ORCPT ); Tue, 25 Jul 2023 18:15:43 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9574C173B; Tue, 25 Jul 2023 15:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323341; x=1721859341; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QLI5gFKdIox5w0Oi+9+VH1yeFL7uGsPBS1/CMUHOHKo=; b=jcizGMC53ibHQ5tOzaPQ4pmXDc5bZjZRWCwHaviJtBJpjeP52XPDD4Yr mxyHmMAXkwXtFAQ9qc2dxGwOA4Hm012V7JGoRemHz3/5v5BdxM/Z30FXH RICgqwQPieqMvsCbYzC0axgjmf7yOzwbQ9FQLRVOSB/ekrFv/esX2y9cb asNXg12NyqI6CmAwRpBlswfe8ee+sXlIASdYHuU7iMzJGocrMSkIgTiTI BfXdBUyc6gmqp9pFKSVw5P88w9aDTDDNqPT8Arf9yU5utwbGy+pS5H2eG JrkEmkV3E7tNJQdZZOSOpolidU17X61pVOiND9Jo0Vr14HXN8BRZTCilZ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863104" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863104" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938814" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938814" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:21 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 015/115] x86/cpu: Add helper functions to allocate/free TDX private host key id Date: Tue, 25 Jul 2023 15:13:26 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Add helper functions to allocate/free TDX private host key id (HKID), and export the global TDX HKID. The memory controller encrypts TDX memory with the assigned TDX HKIDs. The global TDX HKID is to encrypt the TDX module, its memory, and some dynamic data (TDR). The private TDX HKID is assigned to guest TD to encrypt guest memory and the related data. When VMM releases an encrypted page for reuse, the page needs a cache flush with the used HKID. VMM needs the global TDX HKID and the private TDX HKIDs to flush encrypted pages. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/tdx.h | 13 +++++++++++++ arch/x86/virt/vmx/tdx/tdx.c | 28 +++++++++++++++++++++++++++- 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index bf5324b5ea01..245c0c93cf71 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -90,6 +90,17 @@ int tdx_cpu_enable(void); int tdx_enable(void); void tdx_reset_memory(void); bool tdx_is_private_mem(unsigned long phys); + +/* + * Key id globally used by TDX module: TDX module maps TDR with this TDX g= lobal + * key id. TDR includes key id assigned to the TD. Then TDX module maps = other + * TD-related pages with the assigned key id. TDR requires this TDX globa= l key + * id for cache flush unlike other TD-related pages. + */ +extern u32 tdx_global_keyid __ro_after_init; +int tdx_guest_keyid_alloc(void); +void tdx_guest_keyid_free(int keyid); + u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, struct tdx_module_output *out); #else /* !CONFIG_INTEL_TDX_HOST */ @@ -100,6 +111,8 @@ static inline void tdx_reset_memory(void) { } static inline bool tdx_is_private_mem(unsigned long phys) { return false; } static inline u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, struct tdx_module_output *out) { return TDX_SEAMCALL_UD; }; +static inline int tdx_guest_keyid_alloc(void) { return -EOPNOTSUPP; } +static inline void tdx_guest_keyid_free(int keyid) { } #endif /* CONFIG_INTEL_TDX_HOST */ =20 #endif /* !__ASSEMBLY__ */ diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 5f96c2d866e5..ef3a1d9dcf2f 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -35,7 +35,8 @@ #include #include "tdx.h" =20 -static u32 tdx_global_keyid __ro_after_init; +u32 tdx_global_keyid __ro_after_init; +EXPORT_SYMBOL_GPL(tdx_global_keyid); static u32 tdx_guest_keyid_start __ro_after_init; static u32 tdx_nr_guest_keyids __ro_after_init; =20 @@ -53,6 +54,31 @@ static struct tdmr_info_list tdx_tdmr_list; =20 static atomic_t tdx_may_has_private_mem; =20 +/* TDX KeyID pool */ +static DEFINE_IDA(tdx_guest_keyid_pool); + +int tdx_guest_keyid_alloc(void) +{ + if (WARN_ON_ONCE(!tdx_guest_keyid_start || !tdx_nr_guest_keyids)) + return -EINVAL; + + /* The first keyID is reserved for the global key. */ + return ida_alloc_range(&tdx_guest_keyid_pool, tdx_guest_keyid_start + 1, + tdx_guest_keyid_start + tdx_nr_guest_keyids - 1, + GFP_KERNEL); +} +EXPORT_SYMBOL_GPL(tdx_guest_keyid_alloc); + +void tdx_guest_keyid_free(int keyid) +{ + /* keyid =3D 0 is reserved. */ + if (WARN_ON_ONCE(keyid <=3D 0)) + return; + + ida_free(&tdx_guest_keyid_pool, keyid); +} +EXPORT_SYMBOL_GPL(tdx_guest_keyid_free); + /* * Wrapper of __seamcall() to convert SEAMCALL leaf function error code * to kernel error code. @seamcall_ret and @out contain the SEAMCALL --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 917D2C001DF for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229572AbjGYWQ7 (ORCPT ); Tue, 25 Jul 2023 18:16:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231317AbjGYWPo (ORCPT ); Tue, 25 Jul 2023 18:15:44 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D65BE0; Tue, 25 Jul 2023 15:15:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323342; x=1721859342; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+rgqfKuT11k4dIaW4W1G9GmV5lM2K7dgjyD0BUyN94I=; b=ZC7zr0gIYW47dHz/0ykDfsdKWM3baUUVCpPfX9FRRWyOViE1I57uccLx 4/AD4G2FabbXFhsTcH07T6l/HQIvgSt1oGhJzvU4hOjo509+UZktXmjeP 90MdTOjVl9KuhXGxCyBrk+CHSYY/PyfHyvRKwHeG6JuEJc0VpfOfJo5Id jMmSoeAE2IE4HiZO2dVqkuQduvrUirddI7sX498QyozHC0AcHo6vuu64q h6dqHCCWxAeUnENTnT9ZjOnxRKUYY+tG9gM8czWJYz/xw7PZNWbJJfl5p vtBmoIPphECuY5lCWelhuaSDJE0/zQS3gRQSUrdZk4UUjA/qGN/bGudBT A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863109" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863109" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938817" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938817" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:21 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 016/115] x86/virt/tdx: Add a helper function to return system wide info about TDX module Date: Tue, 25 Jul 2023 15:13:27 -0700 Message-Id: <54648132d8e33e266d14bac3e7faec095b2fa385.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX KVM needs system-wide information about the TDX module, struct tdsysinfo_struct. Add a helper function tdx_get_sysinfo() to return it instead of KVM getting it with various error checks. Make KVM call the function and stash the info. Move out the struct definition about it to common place arch/x86/include/asm/tdx.h. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/tdx.h | 57 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.c | 15 +++++++++- arch/x86/virt/vmx/tdx/tdx.c | 25 ++++++++++------ arch/x86/virt/vmx/tdx/tdx.h | 50 -------------------------------- 4 files changed, 88 insertions(+), 59 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 245c0c93cf71..86517add595f 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -85,6 +85,61 @@ static inline long tdx_kvm_hypercall(unsigned int nr, un= signed long p1, #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */ =20 #ifdef CONFIG_INTEL_TDX_HOST +struct tdx_cpuid_config { + __struct_group(tdx_cpuid_config_leaf, leaf_sub_leaf, __packed, + u32 leaf; + u32 sub_leaf; + ); + __struct_group(tdx_cpuid_config_value, value, __packed, + u32 eax; + u32 ebx; + u32 ecx; + u32 edx; + ); +} __packed; + +#define TDSYSINFO_STRUCT_SIZE 1024 + +/* + * The size of this structure itself is flexible. The actual structure + * passed to TDH.SYS.INFO must be padded to 1024 bytes and be 1204-bytes + * aligned. + */ +struct tdsysinfo_struct { + /* TDX-SEAM Module Info */ + u32 attributes; + u32 vendor_id; + u32 build_date; + u16 build_num; + u16 minor_version; + u16 major_version; + u8 reserved0[14]; + /* Memory Info */ + u16 max_tdmrs; + u16 max_reserved_per_tdmr; + u16 pamt_entry_size; + u8 reserved1[10]; + /* Control Struct Info */ + u16 tdcs_base_size; + u8 reserved2[2]; + u16 tdvps_base_size; + u8 tdvps_xfam_dependent_size; + u8 reserved3[9]; + /* TD Capabilities */ + u64 attributes_fixed0; + u64 attributes_fixed1; + u64 xfam_fixed0; + u64 xfam_fixed1; + u8 reserved4[32]; + u32 num_cpuid_config; + /* + * The actual number of CPUID_CONFIG depends on above + * 'num_cpuid_config'. + */ + DECLARE_FLEX_ARRAY(struct tdx_cpuid_config, cpuid_configs); +} __packed; + +const struct tdsysinfo_struct *tdx_get_sysinfo(void); bool platform_tdx_enabled(void); int tdx_cpu_enable(void); int tdx_enable(void); @@ -104,6 +159,8 @@ void tdx_guest_keyid_free(int keyid); u64 __seamcall(u64 op, u64 rcx, u64 rdx, u64 r8, u64 r9, struct tdx_module_output *out); #else /* !CONFIG_INTEL_TDX_HOST */ +struct tdsysinfo_struct; +static inline const struct tdsysinfo_struct *tdx_get_sysinfo(void) { retur= n NULL; } static inline bool platform_tdx_enabled(void) { return false; } static inline int tdx_cpu_enable(void) { return -ENODEV; } static inline int tdx_enable(void) { return -ENODEV; } diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 9d3f593eacb8..b0e3409da5a8 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -11,9 +11,18 @@ #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt =20 +#define TDX_MAX_NR_CPUID_CONFIGS \ + ((TDSYSINFO_STRUCT_SIZE - \ + offsetof(struct tdsysinfo_struct, cpuid_configs)) \ + / sizeof(struct tdx_cpuid_config)) + static int __init tdx_module_setup(void) { - int ret; + const struct tdsysinfo_struct *tdsysinfo; + int ret =3D 0; + + BUILD_BUG_ON(sizeof(*tdsysinfo) > TDSYSINFO_STRUCT_SIZE); + BUILD_BUG_ON(TDX_MAX_NR_CPUID_CONFIGS !=3D 37); =20 ret =3D tdx_enable(); if (ret) { @@ -21,6 +30,10 @@ static int __init tdx_module_setup(void) return ret; } =20 + /* Sanitary check just in case. */ + tdsysinfo =3D tdx_get_sysinfo(); + WARN_ON(tdsysinfo->num_cpuid_config > TDX_MAX_NR_CPUID_CONFIGS); + return 0; } =20 diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index ef3a1d9dcf2f..f49a89cd2f34 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -230,7 +230,7 @@ static void print_cmrs(struct cmr_info *cmr_array, int = nr_cmrs) } } =20 -static int tdx_get_sysinfo(struct tdsysinfo_struct *sysinfo, +static int __tdx_get_sysinfo(struct tdsysinfo_struct *sysinfo, struct cmr_info *cmr_array) { struct tdx_module_output out; @@ -255,6 +255,20 @@ static int tdx_get_sysinfo(struct tdsysinfo_struct *sy= sinfo, return 0; } =20 +static struct tdsysinfo_struct *sysinfo; + +const struct tdsysinfo_struct *tdx_get_sysinfo(void) +{ + const struct tdsysinfo_struct *r =3D NULL; + + mutex_lock(&tdx_module_lock); + if (tdx_module_status =3D=3D TDX_MODULE_INITIALIZED) + r =3D sysinfo; + mutex_unlock(&tdx_module_lock); + return r; +} +EXPORT_SYMBOL_GPL(tdx_get_sysinfo); + /* * Add a memory region as a TDX memory block. The caller must make sure * all memory regions are added in address ascending order and don't @@ -1083,7 +1097,6 @@ static int init_tdmrs(struct tdmr_info_list *tdmr_lis= t) =20 static int init_tdx_module(void) { - struct tdsysinfo_struct *sysinfo; struct cmr_info *cmr_array; int ret; =20 @@ -1103,7 +1116,7 @@ static int init_tdx_module(void) BUILD_BUG_ON(PAGE_SIZE / 2 < TDSYSINFO_STRUCT_SIZE); BUILD_BUG_ON(PAGE_SIZE / 2 < sizeof(struct cmr_info) * MAX_CMRS); =20 - ret =3D tdx_get_sysinfo(sysinfo, cmr_array); + ret =3D __tdx_get_sysinfo(sysinfo, cmr_array); if (ret) goto out; =20 @@ -1177,11 +1190,6 @@ static int init_tdx_module(void) * Lock out memory hotplug code while building it. */ put_online_mems(); - /* - * For now both @sysinfo and @cmr_array are only used during - * module initialization, so always free them. - */ - free_page((unsigned long)sysinfo); =20 return 0; out_reset_pamts: @@ -1219,6 +1227,7 @@ static int init_tdx_module(void) put_online_mems(); out: free_page((unsigned long)sysinfo); + sysinfo =3D NULL; return ret; } =20 diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h index 70315263d8d2..91086576651b 100644 --- a/arch/x86/virt/vmx/tdx/tdx.h +++ b/arch/x86/virt/vmx/tdx/tdx.h @@ -40,56 +40,6 @@ struct cmr_info { =20 #define MAX_CMRS 32 =20 -struct cpuid_config { - u32 leaf; - u32 sub_leaf; - u32 eax; - u32 ebx; - u32 ecx; - u32 edx; -} __packed; - -#define TDSYSINFO_STRUCT_SIZE 1024 - -/* - * The size of this structure itself is flexible. The actual structure - * passed to TDH.SYS.INFO must be padded to 1024 bytes and be 1204-bytes - * aligned. - */ -struct tdsysinfo_struct { - /* TDX-SEAM Module Info */ - u32 attributes; - u32 vendor_id; - u32 build_date; - u16 build_num; - u16 minor_version; - u16 major_version; - u8 reserved0[14]; - /* Memory Info */ - u16 max_tdmrs; - u16 max_reserved_per_tdmr; - u16 pamt_entry_size; - u8 reserved1[10]; - /* Control Struct Info */ - u16 tdcs_base_size; - u8 reserved2[2]; - u16 tdvps_base_size; - u8 tdvps_xfam_dependent_size; - u8 reserved3[9]; - /* TD Capabilities */ - u64 attributes_fixed0; - u64 attributes_fixed1; - u64 xfam_fixed0; - u64 xfam_fixed1; - u8 reserved4[32]; - u32 num_cpuid_config; - /* - * The actual number of CPUID_CONFIG depends on above - * 'num_cpuid_config'. - */ - DECLARE_FLEX_ARRAY(struct cpuid_config, cpuid_configs); -} __packed; - struct tdmr_reserved_area { u64 offset; u64 size; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A58C0015E for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231286AbjGYWRE (ORCPT ); Tue, 25 Jul 2023 18:17:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231759AbjGYWPx (ORCPT ); Tue, 25 Jul 2023 18:15:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7ECCE47; Tue, 25 Jul 2023 15:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323343; x=1721859343; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=R61O3xb/EGPLdCFHSDNrCquLJQTQOLujl1PSSzI03OY=; b=EBKq/QaWNIyE0XIOumPqcP79p6L+OJbDFAbEOgNsbGHLllbY1rQQQeTa RnHtajIywifW0Ky1rwbUkdLS4vCU5JT35DIruPBeW31V+2b0Wh0BWDvxK LmIZu0bD7NBT6GWfdZK0by8zCP58peb/FUH5J3QTTO5s98L+TZoA7M1aV P//vkDF1+mH1gzTmmZMoIFG1DQbGkw+WjloUZAmLRwzaxUrhKSfHzH/jB 3FbxmxdNg6rbCzHfVzA44QmzwtZfuKlzt59gPuf+Jhcfm93lKDZ13NvVD nQBVJxFC6LV94roTzuJ7bH3Fr2zAZx1GvUh2XbXONFHECWmf5Inr1IBT8 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863114" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863114" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938820" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938820" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:21 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 017/115] KVM: TDX: Add place holder for TDX VM specific mem_enc_op ioctl Date: Tue, 25 Jul 2023 15:13:28 -0700 Message-Id: <5b0a4c53a12fc6d5e98c5bf10e16fff44a29eb26.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata KVM_MEMORY_ENCRYPT_OP was introduced for VM-scoped operations specific for guest state-protected VM. It defined subcommands for technology-specific operations under KVM_MEMORY_ENCRYPT_OP. Despite its name, the subcommands are not limited to memory encryption, but various technology-specific operations are defined. It's natural to repurpose KVM_MEMORY_ENCRYPT_OP for TDX specific operations and define subcommands. TDX requires VM-scoped TDX-specific operations for device model, for example, qemu. Getting system-wide parameters, TDX-specific VM initialization. Add a place holder function for TDX specific VM-scoped ioctl as mem_enc_op. TDX specific sub-commands will be added to retrieve/pass TDX specific parameters. Make mem_enc_ioctl non-optional as it's always filled. Signed-off-by: Isaku Yamahata --- v15: - change struct kvm_tdx_cmd to drop unused member. --- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/uapi/asm/kvm.h | 26 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.c | 26 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ arch/x86/kvm/x86.c | 4 ---- 6 files changed, 67 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index d520c6370cd6..7b22bf8d6686 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -119,7 +119,7 @@ KVM_X86_OP(enter_smm) KVM_X86_OP(leave_smm) KVM_X86_OP(enable_smi_window) #endif -KVM_X86_OP_OPTIONAL(mem_enc_ioctl) +KVM_X86_OP(mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_register_region) KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index aa7a56a47564..615fb60b3717 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -567,4 +567,30 @@ struct kvm_pmu_event_filter { #define KVM_X86_TDX_VM 2 #define KVM_X86_SNP_VM 3 =20 +/* Trust Domain eXtension sub-ioctl() commands. */ +enum kvm_tdx_cmd_id { + KVM_TDX_CAPABILITIES =3D 0, + + KVM_TDX_CMD_NR_MAX, +}; + +struct kvm_tdx_cmd { + /* enum kvm_tdx_cmd_id */ + __u32 id; + /* flags for sub-commend. If sub-command doesn't use this, set zero. */ + __u32 flags; + /* + * data for each sub-command. An immediate or a pointer to the actual + * data in process virtual address. If sub-command doesn't use it, + * set zero. + */ + __u64 data; + /* + * Auxiliary error code. The sub-command may return TDX SEAMCALL + * status code in addition to -Exxx. + * Defined for consistency with struct kvm_sev_cmd. + */ + __u64 error; +}; + #endif /* _ASM_X86_KVM_H */ diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 9619473fba01..fcd2516088ce 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -51,6 +51,14 @@ static int vt_vm_init(struct kvm *kvm) return vmx_vm_init(kvm); } =20 +static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) +{ + if (!is_td(kvm)) + return -ENOTTY; + + return tdx_vm_ioctl(kvm, argp); +} + #define VMX_REQUIRED_APICV_INHIBITS \ (BIT(APICV_INHIBIT_REASON_DISABLE)| \ BIT(APICV_INHIBIT_REASON_ABSENT) | \ @@ -200,6 +208,8 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .complete_emulated_msr =3D kvm_complete_insn_gp, =20 .vcpu_deliver_sipi_vector =3D kvm_vcpu_deliver_sipi_vector, + + .mem_enc_ioctl =3D vt_mem_enc_ioctl, }; =20 struct kvm_x86_init_ops vt_init_ops __initdata =3D { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b0e3409da5a8..ead229e34813 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -16,6 +16,32 @@ offsetof(struct tdsysinfo_struct, cpuid_configs)) \ / sizeof(struct tdx_cpuid_config)) =20 +int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) +{ + struct kvm_tdx_cmd tdx_cmd; + int r; + + if (copy_from_user(&tdx_cmd, argp, sizeof(struct kvm_tdx_cmd))) + return -EFAULT; + if (tdx_cmd.error) + return -EINVAL; + + mutex_lock(&kvm->lock); + + switch (tdx_cmd.id) { + default: + r =3D -EINVAL; + goto out; + } + + if (copy_to_user(argp, &tdx_cmd, sizeof(struct kvm_tdx_cmd))) + r =3D -EFAULT; + +out: + mutex_unlock(&kvm->lock); + return r; +} + static int __init tdx_module_setup(void) { const struct tdsysinfo_struct *tdsysinfo; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 32a5c2629145..1a6bf336ca60 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -138,9 +138,13 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu); #ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); bool tdx_is_vm_type_supported(unsigned long type); + +int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline bool tdx_is_vm_type_supported(unsigned long type) { return f= alse; } + +static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { retur= n -EOPNOTSUPP; } #endif =20 #endif /* __KVM_X86_VMX_X86_OPS_H */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 2ae40fa8e178..d700da8ff4f2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7040,10 +7040,6 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned in= t ioctl, unsigned long arg) goto out; } case KVM_MEMORY_ENCRYPT_OP: { - r =3D -ENOTTY; - if (!kvm_x86_ops.mem_enc_ioctl) - goto out; - r =3D static_call(kvm_x86_mem_enc_ioctl)(kvm, argp); break; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA8FBC04A94 for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230523AbjGYWRN (ORCPT ); Tue, 25 Jul 2023 18:17:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231862AbjGYWQA (ORCPT ); Tue, 25 Jul 2023 18:16:00 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A85E210F7; Tue, 25 Jul 2023 15:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323343; x=1721859343; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FS2KQf5q2nLsELGXmMGvnBKPdHJQHT10RFv7m7kWMKA=; b=CGYift4CD17OEvYMWHfGuZO3zSRAdbahGzesAPdVaQnoUsIDNmKp6QpI n3bMixmIVIWsBirl5c1GHHVR4V0eoI3Z+CclRTHSZH7j456PE+MQmPWZ3 WI6RnlbPH8QKrp6F6d4Cc3+pP46fSizHSUAEBjGKaOH8pOrfff9rlAyId xSHtaSsgcaVZYaHt/Q6AJ8XOjbXPs16m05wr6VcsgUZZ8NQMHhVrU83Qx OEKihaZZWufvh2ocVehQ2YS84tM81cLANRxbuoM7gu0q1ekRYoiRyKm33 B9o4IWwSnHyveR2K6qKdWQwqog/6sytzXNAal3MrgecYGUeUHqjodCmfL g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863124" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863124" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938825" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938825" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:22 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 018/115] KVM: TDX: x86: Add ioctl to get TDX systemwide parameters Date: Tue, 25 Jul 2023 15:13:29 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Implement an ioctl to get system-wide parameters for TDX. Although the function is systemwide, vm scoped mem_enc ioctl works for userspace VMM like qemu and device scoped version is not define, re-use vm scoped mem_enc. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- v14 -> v15: - ABI change: added supported_gpaw and reserved area, --- arch/x86/include/uapi/asm/kvm.h | 24 ++++++++++ arch/x86/kvm/vmx/tdx.c | 64 +++++++++++++++++++++++++++ tools/arch/x86/include/uapi/asm/kvm.h | 52 ++++++++++++++++++++++ 3 files changed, 140 insertions(+) diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index 615fb60b3717..3fbd43d5177b 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -593,4 +593,28 @@ struct kvm_tdx_cmd { __u64 error; }; =20 +struct kvm_tdx_cpuid_config { + __u32 leaf; + __u32 sub_leaf; + __u32 eax; + __u32 ebx; + __u32 ecx; + __u32 edx; +}; + +struct kvm_tdx_capabilities { + __u64 attrs_fixed0; + __u64 attrs_fixed1; + __u64 xfam_fixed0; + __u64 xfam_fixed1; +#define TDX_CAP_GPAW_48 (1 << 0) +#define TDX_CAP_GPAW_52 (1 << 1) + __u32 supported_gpaw; + __u32 padding; + __u64 reserved[251]; + + __u32 nr_cpuid_configs; + struct kvm_tdx_cpuid_config cpuid_configs[]; +}; + #endif /* _ASM_X86_KVM_H */ diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index ead229e34813..229c079d7686 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -6,6 +6,7 @@ #include "capabilities.h" #include "x86_ops.h" #include "x86.h" +#include "mmu.h" #include "tdx.h" =20 #undef pr_fmt @@ -16,6 +17,66 @@ offsetof(struct tdsysinfo_struct, cpuid_configs)) \ / sizeof(struct tdx_cpuid_config)) =20 +static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) +{ + struct kvm_tdx_capabilities __user *user_caps; + const struct tdsysinfo_struct *tdsysinfo; + struct kvm_tdx_capabilities *caps =3D NULL; + int ret; + + BUILD_BUG_ON(sizeof(struct kvm_tdx_cpuid_config) !=3D + sizeof(struct tdx_cpuid_config)); + + if (cmd->flags) + return -EINVAL; + + tdsysinfo =3D tdx_get_sysinfo(); + if (!tdsysinfo) + return -EOPNOTSUPP; + + caps =3D kmalloc(sizeof(*caps), GFP_KERNEL); + if (!caps) + return -ENOMEM; + + user_caps =3D (void __user *)cmd->data; + if (copy_from_user(caps, user_caps, sizeof(*caps))) { + ret =3D -EFAULT; + goto out; + } + + if (caps->nr_cpuid_configs < tdsysinfo->num_cpuid_config) { + ret =3D -E2BIG; + goto out; + } + + *caps =3D (struct kvm_tdx_capabilities) { + .attrs_fixed0 =3D tdsysinfo->attributes_fixed0, + .attrs_fixed1 =3D tdsysinfo->attributes_fixed1, + .xfam_fixed0 =3D tdsysinfo->xfam_fixed0, + .xfam_fixed1 =3D tdsysinfo->xfam_fixed1, + .supported_gpaw =3D TDX_CAP_GPAW_48 | + (kvm_get_shadow_phys_bits() >=3D 52 && + cpu_has_vmx_ept_5levels()) ? TDX_CAP_GPAW_52 : 0, + .nr_cpuid_configs =3D tdsysinfo->num_cpuid_config, + .padding =3D 0, + }; + + if (copy_to_user(user_caps, caps, sizeof(*caps))) { + ret =3D -EFAULT; + goto out; + } + if (copy_to_user(user_caps->cpuid_configs, &tdsysinfo->cpuid_configs, + tdsysinfo->num_cpuid_config * + sizeof(struct tdx_cpuid_config))) { + ret =3D -EFAULT; + } + +out: + /* kfree() accepts NULL. */ + kfree(caps); + return ret; +} + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { struct kvm_tdx_cmd tdx_cmd; @@ -29,6 +90,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) mutex_lock(&kvm->lock); =20 switch (tdx_cmd.id) { + case KVM_TDX_CAPABILITIES: + r =3D tdx_get_capabilities(&tdx_cmd); + break; default: r =3D -EINVAL; goto out; diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include= /uapi/asm/kvm.h index 1a6a1f987949..7a08723e99e2 100644 --- a/tools/arch/x86/include/uapi/asm/kvm.h +++ b/tools/arch/x86/include/uapi/asm/kvm.h @@ -562,4 +562,56 @@ struct kvm_pmu_event_filter { /* x86-specific KVM_EXIT_HYPERCALL flags. */ #define KVM_EXIT_HYPERCALL_LONG_MODE BIT(0) =20 +/* Trust Domain eXtension sub-ioctl() commands. */ +enum kvm_tdx_cmd_id { + KVM_TDX_CAPABILITIES =3D 0, + + KVM_TDX_CMD_NR_MAX, +}; + +struct kvm_tdx_cmd { + /* enum kvm_tdx_cmd_id */ + __u32 id; + /* flags for sub-commend. If sub-command doesn't use this, set zero. */ + __u32 flags; + /* + * data for each sub-command. An immediate or a pointer to the actual + * data in process virtual address. If sub-command doesn't use it, + * set zero. + */ + __u64 data; + /* + * Auxiliary error code. The sub-command may return TDX SEAMCALL + * status code in addition to -Exxx. + * Defined for consistency with struct kvm_sev_cmd. + */ + __u64 error; + /* Reserved: Defined for consistency with struct kvm_sev_cmd. */ + __u64 unused; +}; + +struct kvm_tdx_cpuid_config { + __u32 leaf; + __u32 sub_leaf; + __u32 eax; + __u32 ebx; + __u32 ecx; + __u32 edx; +}; + +struct kvm_tdx_capabilities { + __u64 attrs_fixed0; + __u64 attrs_fixed1; + __u64 xfam_fixed0; + __u64 xfam_fixed1; +#define TDX_CAP_GPAW_48 (1 << 0) +#define TDX_CAP_GPAW_52 (1 << 1) + __u32 supported_gpaw; + __u32 padding; + __u64 reserved[251]; + + __u32 nr_cpuid_configs; + struct kvm_tdx_cpuid_config cpuid_configs[]; +}; + #endif /* _ASM_X86_KVM_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C615CC001E0 for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232278AbjGYWRH (ORCPT ); Tue, 25 Jul 2023 18:17:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231712AbjGYWPw (ORCPT ); Tue, 25 Jul 2023 18:15:52 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7DCD137; Tue, 25 Jul 2023 15:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323343; x=1721859343; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=G7Cn342NX8Ka6m0bqsWTxVCzoSZJtnTp13GnLbmBCJU=; b=nqpD7aHxZzsyNCyHg48ZIFXCM0FFZ3CXvru23js9jRF+5S/CByDXiSKi pu8X3AK+87hmTqDV2U6lMj/tJHy7GQDJyxcj+6pKs2XoX9cI3Oolt2c3Z 2kw9AumrBbcOIJRMC9luXC2GY9vb3ErPuhvlISTPlDq8Xg0feu47aVbyM sbC23KJL1z/ZI7mb4uwMTkS4t/r2t1ZszOAuw70Lk+D1H4wilg8eGAPql zb9R0Le2zWiTPy/sD55e14U3/S8HvcIe+cRpsKG/7GG3x4Sbur2DZD8EK 83VbOjgyz1G74XhZwwxcYMxvFZekPPEpYhULOvpChvxKVzcsWuiBsBE/7 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863131" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863131" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938828" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938828" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:22 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 019/115] KVM: x86, tdx: Make KVM_CAP_MAX_VCPUS backend specific Date: Tue, 25 Jul 2023 15:13:30 -0700 Message-Id: <832eefa4e73f08f196ef009263a145d1fb1d7363.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX has its own limitation on the maximum number of vcpus that the guest can accommodate. Allow x86 kvm backend to implement its own KVM_ENABLE_CAP handler and implement TDX backend for KVM_CAP_MAX_VCPUS. user space VMM, e.g. qemu, can specify its value instead of KVM_MAX_VCPUS. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 2 ++ arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/vmx/main.c | 22 ++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.c | 30 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 3 +++ arch/x86/kvm/vmx/x86_ops.h | 5 +++++ arch/x86/kvm/x86.c | 4 ++++ 7 files changed, 68 insertions(+) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index 7b22bf8d6686..ba79b97b2455 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -21,6 +21,8 @@ KVM_X86_OP(hardware_unsetup) KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported) +KVM_X86_OP_OPTIONAL(max_vcpus); +KVM_X86_OP_OPTIONAL(vm_enable_cap) KVM_X86_OP(vm_init) KVM_X86_OP_OPTIONAL(vm_destroy) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index dc83bb543307..f2eefc322d42 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1546,7 +1546,9 @@ struct kvm_x86_ops { void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu); =20 bool (*is_vm_type_supported)(unsigned long vm_type); + int (*max_vcpus)(struct kvm *kvm); unsigned int vm_size; + int (*vm_enable_cap)(struct kvm *kvm, struct kvm_enable_cap *cap); int (*vm_init)(struct kvm *kvm); void (*vm_destroy)(struct kvm *kvm); =20 diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index fcd2516088ce..76ea59374ad0 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -6,6 +6,7 @@ #include "nested.h" #include "pmu.h" #include "tdx.h" +#include "tdx_arch.h" =20 static bool enable_tdx __ro_after_init; module_param_named(tdx, enable_tdx, bool, 0444); @@ -16,6 +17,17 @@ static bool vt_is_vm_type_supported(unsigned long type) (enable_tdx && tdx_is_vm_type_supported(type)); } =20 +static int vt_max_vcpus(struct kvm *kvm) +{ + if (!kvm) + return KVM_MAX_VCPUS; + + if (is_td(kvm)) + return min3(kvm->max_vcpus, KVM_MAX_VCPUS, TDX_MAX_VCPUS); + + return kvm->max_vcpus; +} + static int vt_hardware_enable(void) { int ret; @@ -43,6 +55,14 @@ static __init int vt_hardware_setup(void) return 0; } =20 +static int vt_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) +{ + if (is_td(kvm)) + return tdx_vm_enable_cap(kvm, cap); + + return -EINVAL; +} + static int vt_vm_init(struct kvm *kvm) { if (is_td(kvm)) @@ -80,7 +100,9 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .has_emulated_msr =3D vmx_has_emulated_msr, =20 .is_vm_type_supported =3D vt_is_vm_type_supported, + .max_vcpus =3D vt_max_vcpus, .vm_size =3D sizeof(struct kvm_vmx), + .vm_enable_cap =3D vt_vm_enable_cap, .vm_init =3D vt_vm_init, .vm_destroy =3D vmx_vm_destroy, =20 diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 229c079d7686..8901ae86c9da 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -17,6 +17,36 @@ offsetof(struct tdsysinfo_struct, cpuid_configs)) \ / sizeof(struct tdx_cpuid_config)) =20 +int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) +{ + int r; + + switch (cap->cap) { + case KVM_CAP_MAX_VCPUS: { + if (cap->flags || cap->args[0] =3D=3D 0) + return -EINVAL; + if (cap->args[0] > KVM_MAX_VCPUS) + return -E2BIG; + if (cap->args[0] > TDX_MAX_VCPUS) + return -E2BIG; + + mutex_lock(&kvm->lock); + if (kvm->created_vcpus) + r =3D -EBUSY; + else { + kvm->max_vcpus =3D cap->args[0]; + r =3D 0; + } + mutex_unlock(&kvm->lock); + break; + } + default: + r =3D -EINVAL; + break; + } + return r; +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 473013265bd8..22c0b57f69ca 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -3,6 +3,9 @@ #define __KVM_X86_TDX_H =20 #ifdef CONFIG_INTEL_TDX_HOST + +#include "tdx_ops.h" + struct kvm_tdx { struct kvm kvm; /* TDX specific members follow. */ diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 1a6bf336ca60..cb96a9af9e79 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -139,11 +139,16 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu); int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); bool tdx_is_vm_type_supported(unsigned long type); =20 +int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap); int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline bool tdx_is_vm_type_supported(unsigned long type) { return f= alse; } =20 +static inline int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap= *cap) +{ + return -EINVAL; +}; static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { retur= n -EOPNOTSUPP; } #endif =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index d700da8ff4f2..6970e6198608 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -4572,6 +4572,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, lon= g ext) break; case KVM_CAP_MAX_VCPUS: r =3D KVM_MAX_VCPUS; + if (kvm_x86_ops.max_vcpus) + r =3D static_call(kvm_x86_max_vcpus)(kvm); break; case KVM_CAP_MAX_VCPU_ID: r =3D KVM_MAX_VCPU_IDS; @@ -6505,6 +6507,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, break; default: r =3D -EINVAL; + if (kvm_x86_ops.vm_enable_cap) + r =3D static_call(kvm_x86_vm_enable_cap)(kvm, cap); break; } return r; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EABA6C04A6A for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229844AbjGYWRQ (ORCPT ); Tue, 25 Jul 2023 18:17:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231768AbjGYWPx (ORCPT ); Tue, 25 Jul 2023 18:15:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A802CE63; Tue, 25 Jul 2023 15:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323343; x=1721859343; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SQSubMCKs0o2a0uF2xYWHDpEMD5OL7IpcVOA7CmDi9c=; b=ZVS8JqbfnUpDdW06Z6IrBM5jmLRESQEYJueLo8dnb91jEkzGdantS16S SV8gQJzGF8EgPMg9+3l5QHwvFwLnsdqXnsQ+aANyEQ6jycn8xwk7fn6V6 LVF46x5qA/YIr7NB6E0bfU/2tfvHP7Sls87dvl0ZNfRysolZkS32Ve4k2 2VQcA0D3TuHULYgRjaoB5Gp97ZXbYOsAtYGXDzyR+3/ML9Be72MEEcewV 35K78mKCXjWpOCEFYt0pU28PiAM3sRMZ5KdGc9lUuoHT29Yk+W91i0PsP +/NGX2LTJIMKWmcIc6bd+VD5IlNvAEvatgVrc5mnsx5j/4p7hOxgAbBNz Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863136" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863136" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938831" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938831" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:23 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 020/115] KVM: TDX: create/destroy VM structure Date: Tue, 25 Jul 2023 15:13:31 -0700 Message-Id: <265c9c5821d020ffa28bf7734d73380f56a23722.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata As the first step to create TDX guest, create/destroy VM struct. Assign TDX private Host Key ID (HKID) to the TDX guest for memory encryption and allocate extra pages for the TDX guest. On destruction, free allocated pages, and HKID. Before tearing down private page tables, TDX requires some resources of the guest TD to be destroyed (i.e. HKID must have been reclaimed, etc). Add flush_shadow_all_private callback before tearing down private page tables for it. Add vm_free() of kvm_x86_ops hook at the end of kvm_arch_destroy_vm() because some per-VM TDX resources, e.g. TDR, need to be freed after other TDX resources, e.g. HKID, were freed. Co-developed-by: Kai Huang Signed-off-by: Kai Huang Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 2 + arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/Kconfig | 2 + arch/x86/kvm/vmx/main.c | 35 ++- arch/x86/kvm/vmx/tdx.c | 452 ++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 6 +- arch/x86/kvm/vmx/x86_ops.h | 8 + arch/x86/kvm/x86.c | 8 + 8 files changed, 509 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index ba79b97b2455..a574e7eb04f3 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -24,7 +24,9 @@ KVM_X86_OP(is_vm_type_supported) KVM_X86_OP_OPTIONAL(max_vcpus); KVM_X86_OP_OPTIONAL(vm_enable_cap) KVM_X86_OP(vm_init) +KVM_X86_OP_OPTIONAL(flush_shadow_all_private) KVM_X86_OP_OPTIONAL(vm_destroy) +KVM_X86_OP_OPTIONAL(vm_free) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate) KVM_X86_OP(vcpu_create) KVM_X86_OP(vcpu_free) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index f2eefc322d42..6ce2f512458e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1550,7 +1550,9 @@ struct kvm_x86_ops { unsigned int vm_size; int (*vm_enable_cap)(struct kvm *kvm, struct kvm_enable_cap *cap); int (*vm_init)(struct kvm *kvm); + void (*flush_shadow_all_private)(struct kvm *kvm); void (*vm_destroy)(struct kvm *kvm); + void (*vm_free)(struct kvm *kvm); =20 /* Create, but do not attach this VCPU */ int (*vcpu_precreate)(struct kvm *kvm); diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 029c76bcd1a5..8d09fc955972 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -92,6 +92,8 @@ config KVM_SW_PROTECTED_VM config KVM_INTEL tristate "KVM for Intel (and compatible) processors support" depends on KVM && IA32_FEAT_CTL + select KVM_SW_PROTECTED_VM if INTEL_TDX_HOST + select KVM_PRIVATE_MEM if INTEL_TDX_HOST help Provides support for KVM on processors equipped with Intel's VT extensions, a.k.a. Virtual Machine Extensions (VMX). diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 76ea59374ad0..ef08a46b04b3 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -63,14 +63,41 @@ static int vt_vm_enable_cap(struct kvm *kvm, struct kvm= _enable_cap *cap) return -EINVAL; } =20 +static void vt_hardware_unsetup(void) +{ + if (enable_tdx) + tdx_hardware_unsetup(); + vmx_hardware_unsetup(); +} + static int vt_vm_init(struct kvm *kvm) { if (is_td(kvm)) - return -EOPNOTSUPP; /* Not ready to create guest TD yet. */ + return tdx_vm_init(kvm); =20 return vmx_vm_init(kvm); } =20 +static void vt_flush_shadow_all_private(struct kvm *kvm) +{ + if (is_td(kvm)) + tdx_mmu_release_hkid(kvm); +} + +static void vt_vm_destroy(struct kvm *kvm) +{ + if (is_td(kvm)) + return; + + vmx_vm_destroy(kvm); +} + +static void vt_vm_free(struct kvm *kvm) +{ + if (is_td(kvm)) + tdx_vm_free(kvm); +} + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -93,7 +120,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .check_processor_compatibility =3D vmx_check_processor_compat, =20 - .hardware_unsetup =3D vmx_hardware_unsetup, + .hardware_unsetup =3D vt_hardware_unsetup, =20 .hardware_enable =3D vt_hardware_enable, .hardware_disable =3D vmx_hardware_disable, @@ -104,7 +131,9 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .vm_size =3D sizeof(struct kvm_vmx), .vm_enable_cap =3D vt_vm_enable_cap, .vm_init =3D vt_vm_init, - .vm_destroy =3D vmx_vm_destroy, + .flush_shadow_all_private =3D vt_flush_shadow_all_private, + .vm_destroy =3D vt_vm_destroy, + .vm_free =3D vt_vm_free, =20 .vcpu_precreate =3D vmx_vcpu_precreate, .vcpu_create =3D vmx_vcpu_create, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 8901ae86c9da..af8c92f24f0f 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -5,9 +5,10 @@ =20 #include "capabilities.h" #include "x86_ops.h" -#include "x86.h" #include "mmu.h" #include "tdx.h" +#include "tdx_ops.h" +#include "x86.h" =20 #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt @@ -47,6 +48,270 @@ int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enabl= e_cap *cap) return r; } =20 +struct tdx_info { + u8 nr_tdcs_pages; +}; + +/* Info about the TDX module. */ +static struct tdx_info tdx_info __ro_after_init; + +/* + * Some TDX SEAMCALLs (TDH.MNG.CREATE, TDH.PHYMEM.CACHE.WB, + * TDH.MNG.KEY.RECLAIMID, TDH.MNG.KEY.FREEID etc) tries to acquire a globa= l lock + * internally in TDX module. If failed, TDX_OPERAND_BUSY is returned with= out + * spinning or waiting due to a constraint on execution time. It's caller= 's + * responsibility to avoid race (or retry on TDX_OPERAND_BUSY). Use this = mutex + * to avoid race in TDX module because the kernel knows better about sched= uling. + */ +static DEFINE_MUTEX(tdx_lock); +static struct mutex *tdx_mng_key_config_lock; + +static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid) +{ + return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits); +} + +static inline bool is_td_created(struct kvm_tdx *kvm_tdx) +{ + return kvm_tdx->tdr_pa; +} + +static inline void tdx_hkid_free(struct kvm_tdx *kvm_tdx) +{ + tdx_guest_keyid_free(kvm_tdx->hkid); + kvm_tdx->hkid =3D 0; +} + +static inline bool is_hkid_assigned(struct kvm_tdx *kvm_tdx) +{ + return kvm_tdx->hkid > 0; +} + +static void tdx_clear_page(unsigned long page_pa) +{ + const void *zero_page =3D (const void *) __va(page_to_phys(ZERO_PAGE(0))); + void *page =3D __va(page_pa); + unsigned long i; + + /* + * When re-assign one page from old keyid to a new keyid, MOVDIR64B is + * required to clear/write the page with new keyid to prevent integrity + * error when read on the page with new keyid. + * + * clflush doesn't flush cache with HKID set. The cache line could be + * poisoned (even without MKTME-i), clear the poison bit. + */ + for (i =3D 0; i < PAGE_SIZE; i +=3D 64) + movdir64b(page + i, zero_page); + /* + * MOVDIR64B store uses WC buffer. Prevent following memory reads + * from seeing potentially poisoned cache. + */ + __mb(); +} + +static int tdx_reclaim_page(hpa_t pa, bool do_wb, u16 hkid) +{ + struct tdx_module_output out; + u64 err; + + do { + err =3D tdh_phymem_page_reclaim(pa, &out); + /* + * TDH.PHYMEM.PAGE.RECLAIM is allowed only when TD is shutdown. + * state. i.e. destructing TD. + * TDH.PHYMEM.PAGE.RECLAIM requires TDR and target page. + * Because we're destructing TD, it's rare to contend with TDR. + */ + } while (unlikely(err =3D=3D (TDX_OPERAND_BUSY | TDX_OPERAND_ID_RCX))); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_PHYMEM_PAGE_RECLAIM, err, &out); + return -EIO; + } + + if (do_wb) { + /* + * Only TDR page gets into this path. No contention is expected + * because of the last page of TD. + */ + err =3D tdh_phymem_page_wbinvd(set_hkid_to_hpa(pa, hkid)); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err, NULL); + return -EIO; + } + } + + tdx_clear_page(pa); + return 0; +} + +static void tdx_reclaim_td_page(unsigned long td_page_pa) +{ + WARN_ON_ONCE(!td_page_pa); + + /* + * TDCX are being reclaimed. TDX module maps TDCX with HKID + * assigned to the TD. Here the cache associated to the TD + * was already flushed by TDH.PHYMEM.CACHE.WB before here, So + * cache doesn't need to be flushed again. + */ + if (tdx_reclaim_page(td_page_pa, false, 0)) + /* + * Leak the page on failure: + * tdx_reclaim_page() returns an error if and only if there's an + * unexpected, fatal error, e.g. a SEAMCALL with bad params, + * incorrect concurrency in KVM, a TDX Module bug, etc. + * Retrying at a later point is highly unlikely to be + * successful. + * No log here as tdx_reclaim_page() already did. + */ + return; + free_page((unsigned long)__va(td_page_pa)); +} + +static int tdx_do_tdh_phymem_cache_wb(void *param) +{ + u64 err =3D 0; + + do { + err =3D tdh_phymem_cache_wb(!!err); + } while (err =3D=3D TDX_INTERRUPTED_RESUMABLE); + + /* Other thread may have done for us. */ + if (err =3D=3D TDX_NO_HKID_READY_TO_WBCACHE) + err =3D TDX_SUCCESS; + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_PHYMEM_CACHE_WB, err, NULL); + return -EIO; + } + + return 0; +} + +void tdx_mmu_release_hkid(struct kvm *kvm) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + cpumask_var_t packages; + bool cpumask_allocated; + u64 err; + int ret; + int i; + + if (!is_hkid_assigned(kvm_tdx)) + return; + + if (!is_td_created(kvm_tdx)) + goto free_hkid; + + cpumask_allocated =3D zalloc_cpumask_var(&packages, GFP_KERNEL); + cpus_read_lock(); + for_each_online_cpu(i) { + if (cpumask_allocated && + cpumask_test_and_set_cpu(topology_physical_package_id(i), + packages)) + continue; + + /* + * We can destroy multiple the guest TDs simultaneously. + * Prevent tdh_phymem_cache_wb from returning TDX_BUSY by + * serialization. + */ + mutex_lock(&tdx_lock); + ret =3D smp_call_on_cpu(i, tdx_do_tdh_phymem_cache_wb, NULL, 1); + mutex_unlock(&tdx_lock); + if (ret) + break; + } + cpus_read_unlock(); + free_cpumask_var(packages); + + mutex_lock(&tdx_lock); + err =3D tdh_mng_key_freeid(kvm_tdx->tdr_pa); + mutex_unlock(&tdx_lock); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MNG_KEY_FREEID, err, NULL); + pr_err("tdh_mng_key_freeid failed. HKID %d is leaked.\n", + kvm_tdx->hkid); + return; + } + +free_hkid: + tdx_hkid_free(kvm_tdx); +} + +void tdx_vm_free(struct kvm *kvm) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + int i; + + /* + * tdx_mmu_release_hkid() failed to reclaim HKID. Something went wrong + * heavily with TDX module. Give up freeing TD pages. As the function + * already warned, don't warn it again. + */ + if (is_hkid_assigned(kvm_tdx)) + return; + + if (kvm_tdx->tdcs_pa) { + for (i =3D 0; i < tdx_info.nr_tdcs_pages; i++) { + if (kvm_tdx->tdcs_pa[i]) + tdx_reclaim_td_page(kvm_tdx->tdcs_pa[i]); + } + kfree(kvm_tdx->tdcs_pa); + kvm_tdx->tdcs_pa =3D NULL; + } + + if (!kvm_tdx->tdr_pa) + return; + /* + * TDX module maps TDR with TDX global HKID. TDX module may access TDR + * while operating on TD (Especially reclaiming TDCS). Cache flush with + * TDX global HKID is needed. + */ + if (tdx_reclaim_page(kvm_tdx->tdr_pa, true, tdx_global_keyid)) + return; + + free_page((unsigned long)__va(kvm_tdx->tdr_pa)); + kvm_tdx->tdr_pa =3D 0; +} + +static int tdx_do_tdh_mng_key_config(void *param) +{ + hpa_t *tdr_p =3D param; + u64 err; + + do { + err =3D tdh_mng_key_config(*tdr_p); + + /* + * If it failed to generate a random key, retry it because this + * is typically caused by an entropy error of the CPU's random + * number generator. + */ + } while (err =3D=3D TDX_KEY_GENERATION_FAILED); + + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MNG_KEY_CONFIG, err, NULL); + return -EIO; + } + + return 0; +} + +static int __tdx_td_init(struct kvm *kvm); + +int tdx_vm_init(struct kvm *kvm) +{ + /* + * TDX has its own limit of the number of vcpus in addition to + * KVM_MAX_VCPUS. + */ + kvm->max_vcpus =3D min(kvm->max_vcpus, TDX_MAX_VCPUS); + + /* Place holder for TDX specific logic. */ + return __tdx_td_init(kvm); +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; @@ -107,6 +372,167 @@ static int tdx_get_capabilities(struct kvm_tdx_cmd *c= md) return ret; } =20 +static int __tdx_td_init(struct kvm *kvm) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + cpumask_var_t packages; + unsigned long *tdcs_pa =3D NULL; + unsigned long tdr_pa =3D 0; + unsigned long va; + int ret, i; + u64 err; + + ret =3D tdx_guest_keyid_alloc(); + if (ret < 0) + return ret; + kvm_tdx->hkid =3D ret; + + va =3D __get_free_page(GFP_KERNEL_ACCOUNT); + if (!va) + goto free_hkid; + tdr_pa =3D __pa(va); + + tdcs_pa =3D kcalloc(tdx_info.nr_tdcs_pages, sizeof(*kvm_tdx->tdcs_pa), + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (!tdcs_pa) + goto free_tdr; + for (i =3D 0; i < tdx_info.nr_tdcs_pages; i++) { + va =3D __get_free_page(GFP_KERNEL_ACCOUNT); + if (!va) + goto free_tdcs; + tdcs_pa[i] =3D __pa(va); + } + + if (!zalloc_cpumask_var(&packages, GFP_KERNEL)) { + ret =3D -ENOMEM; + goto free_tdcs; + } + cpus_read_lock(); + /* + * Need at least one CPU of the package to be online in order to + * program all packages for host key id. Check it. + */ + for_each_present_cpu(i) + cpumask_set_cpu(topology_physical_package_id(i), packages); + for_each_online_cpu(i) + cpumask_clear_cpu(topology_physical_package_id(i), packages); + if (!cpumask_empty(packages)) { + ret =3D -EIO; + /* + * Because it's hard for human operator to figure out the + * reason, warn it. + */ +#define MSG_ALLPKG "All packages need to have online CPU to create TD. Onl= ine CPU and retry.\n" + pr_warn_ratelimited(MSG_ALLPKG); + goto free_packages; + } + + /* + * Acquire global lock to avoid TDX_OPERAND_BUSY: + * TDH.MNG.CREATE and other APIs try to lock the global Key Owner + * Table (KOT) to track the assigned TDX private HKID. It doesn't spin + * to acquire the lock, returns TDX_OPERAND_BUSY instead, and let the + * caller to handle the contention. This is because of time limitation + * usable inside the TDX module and OS/VMM knows better about process + * scheduling. + * + * APIs to acquire the lock of KOT: + * TDH.MNG.CREATE, TDH.MNG.KEY.FREEID, TDH.MNG.VPFLUSHDONE, and + * TDH.PHYMEM.CACHE.WB. + */ + mutex_lock(&tdx_lock); + err =3D tdh_mng_create(tdr_pa, kvm_tdx->hkid); + mutex_unlock(&tdx_lock); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MNG_CREATE, err, NULL); + ret =3D -EIO; + goto free_packages; + } + kvm_tdx->tdr_pa =3D tdr_pa; + + for_each_online_cpu(i) { + int pkg =3D topology_physical_package_id(i); + + if (cpumask_test_and_set_cpu(pkg, packages)) + continue; + + /* + * Program the memory controller in the package with an + * encryption key associated to a TDX private host key id + * assigned to this TDR. Concurrent operations on same memory + * controller results in TDX_OPERAND_BUSY. Avoid this race by + * mutex. + */ + mutex_lock(&tdx_mng_key_config_lock[pkg]); + ret =3D smp_call_on_cpu(i, tdx_do_tdh_mng_key_config, + &kvm_tdx->tdr_pa, true); + mutex_unlock(&tdx_mng_key_config_lock[pkg]); + if (ret) + break; + } + cpus_read_unlock(); + free_cpumask_var(packages); + if (ret) { + i =3D 0; + goto teardown; + } + + kvm_tdx->tdcs_pa =3D tdcs_pa; + for (i =3D 0; i < tdx_info.nr_tdcs_pages; i++) { + err =3D tdh_mng_addcx(kvm_tdx->tdr_pa, tdcs_pa[i]); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MNG_ADDCX, err, NULL); + ret =3D -EIO; + goto teardown; + } + } + + /* + * Note, TDH_MNG_INIT cannot be invoked here. TDH_MNG_INIT requires a de= dicated + * ioctl() to define the configure CPUID values for the TD. + */ + return 0; + + /* + * The sequence for freeing resources from a partially initialized TD + * varies based on where in the initialization flow failure occurred. + * Simply use the full teardown and destroy, which naturally play nice + * with partial initialization. + */ +teardown: + for (; i < tdx_info.nr_tdcs_pages; i++) { + if (tdcs_pa[i]) { + free_page((unsigned long)__va(tdcs_pa[i])); + tdcs_pa[i] =3D 0; + } + } + if (!kvm_tdx->tdcs_pa) + kfree(tdcs_pa); + tdx_mmu_release_hkid(kvm); + tdx_vm_free(kvm); + return ret; + +free_packages: + cpus_read_unlock(); + free_cpumask_var(packages); +free_tdcs: + for (i =3D 0; i < tdx_info.nr_tdcs_pages; i++) { + if (tdcs_pa[i]) + free_page((unsigned long)__va(tdcs_pa[i])); + } + kfree(tdcs_pa); + kvm_tdx->tdcs_pa =3D NULL; + +free_tdr: + if (tdr_pa) + free_page((unsigned long)__va(tdr_pa)); + kvm_tdx->tdr_pa =3D 0; +free_hkid: + if (is_hkid_assigned(kvm_tdx)) + tdx_hkid_free(kvm_tdx); + return ret; +} + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { struct kvm_tdx_cmd tdx_cmd; @@ -150,9 +576,11 @@ static int __init tdx_module_setup(void) return ret; } =20 - /* Sanitary check just in case. */ tdsysinfo =3D tdx_get_sysinfo(); WARN_ON(tdsysinfo->num_cpuid_config > TDX_MAX_NR_CPUID_CONFIGS); + tdx_info =3D (struct tdx_info) { + .nr_tdcs_pages =3D tdsysinfo->tdcs_base_size / PAGE_SIZE, + }; =20 return 0; } @@ -195,13 +623,27 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86= _ops) struct vmx_tdx_enabled vmx_tdx =3D { .err =3D ATOMIC_INIT(0), }; + int max_pkgs; int r =3D 0; + int i; =20 + if (!cpu_feature_enabled(X86_FEATURE_MOVDIR64B)) { + pr_warn("MOVDIR64B is reqiured for TDX\n"); + return -EOPNOTSUPP; + } if (!enable_ept) { pr_warn("Cannot enable TDX with EPT disabled\n"); return -EINVAL; } =20 + max_pkgs =3D topology_max_packages(); + tdx_mng_key_config_lock =3D kcalloc(max_pkgs, sizeof(*tdx_mng_key_config_= lock), + GFP_KERNEL); + if (!tdx_mng_key_config_lock) + return -ENOMEM; + for (i =3D 0; i < max_pkgs; i++) + mutex_init(&tdx_mng_key_config_lock[i]); + if (!zalloc_cpumask_var(&vmx_tdx.vmx_enabled, GFP_KERNEL)) { r =3D -ENOMEM; goto out; @@ -222,3 +664,9 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x86_o= ps) out: return r; } + +void tdx_hardware_unsetup(void) +{ + /* kfree accepts NULL. */ + kfree(tdx_mng_key_config_lock); +} diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 22c0b57f69ca..ae117f864cfb 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -8,7 +8,11 @@ =20 struct kvm_tdx { struct kvm kvm; - /* TDX specific members follow. */ + + unsigned long tdr_pa; + unsigned long *tdcs_pa; + + int hkid; }; =20 struct vcpu_tdx { diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index cb96a9af9e79..fc5348dd20da 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -137,18 +137,26 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu); =20 #ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); +void tdx_hardware_unsetup(void); bool tdx_is_vm_type_supported(unsigned long type); =20 int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap); +int tdx_vm_init(struct kvm *kvm); +void tdx_mmu_release_hkid(struct kvm *kvm); +void tdx_vm_free(struct kvm *kvm); int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } +static inline void tdx_hardware_unsetup(void) {} static inline bool tdx_is_vm_type_supported(unsigned long type) { return f= alse; } =20 static inline int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap= *cap) { return -EINVAL; }; +static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; } +static inline void tdx_mmu_release_hkid(struct kvm *kvm) {} +static inline void tdx_vm_free(struct kvm *kvm) {} static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { retur= n -EOPNOTSUPP; } #endif =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 6970e6198608..bc7cdd8cbbb0 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -12496,6 +12496,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm) kvm_page_track_cleanup(kvm); kvm_xen_destroy_vm(kvm); kvm_hv_destroy_vm(kvm); + static_call_cond(kvm_x86_vm_free)(kvm); } =20 static void memslot_rmap_free(struct kvm_memory_slot *slot) @@ -12806,6 +12807,13 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, =20 void kvm_arch_flush_shadow_all(struct kvm *kvm) { + /* + * kvm_mmu_zap_all() zaps both private and shared page tables. Before + * tearing down private page tables, TDX requires some TD resources to + * be destroyed (i.e. keyID must have been reclaimed, etc). Invoke + * kvm_x86_flush_shadow_all_private() for this. + */ + static_call_cond(kvm_x86_flush_shadow_all_private)(kvm); kvm_mmu_zap_all(kvm); } =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21D78C04E69 for ; Tue, 25 Jul 2023 22:17:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232285AbjGYWRW (ORCPT ); Tue, 25 Jul 2023 18:17:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231991AbjGYWQI (ORCPT ); Tue, 25 Jul 2023 18:16:08 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D39DC2685; Tue, 25 Jul 2023 15:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323345; x=1721859345; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=m9QLwYt1HBfAN8/8K2Uxblrc7k83cU402kZebHWmJuo=; b=AVxxmKeiBZwk93ikrbCaCLkwWM2PLrTd0UqJW9GderaqsMRTZWJi2Ioj 4/XIwKyz4kTm9Xheps8VIK7L/aRBfPVil6Tk4ujoayWK3MeVkR6HAw8kj 1qol3vfEvjL8uPbx4iafxpui94gHT3G/1a0kNxL8thTAHsnUUYPRLE3F3 5VY97Dv4HF3juodIfRnvJjtBjfZfMg9d8pfmeySF6W0nXuJwrkIpKgzUp 8LL6MPR0TADBPJoTbZOR0qWN1h6CXFSJLy8jp8XESWfjhTokpjm9M0DAM VSajHoBOKBQ99IF2srKzBYUNBygIV2egAPL4C9pw/HUDSNG6dzY0Y8hkh Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863141" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863141" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938834" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938834" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:23 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Xiaoyao Li Subject: [PATCH v15 021/115] KVM: TDX: initialize VM with TDX specific parameters Date: Tue, 25 Jul 2023 15:13:32 -0700 Message-Id: <4819838d73f33f4f9d4028df408bf23aab57e064.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX requires additional parameters for TDX VM for confidential execution to protect the confidentiality of its memory contents and CPU state from any other software, including VMM. When creating a guest TD VM before creating vcpu, the number of vcpu, TSC frequency (the values are the same among vcpus, and it can't change.) CPUIDs which the TDX module emulates. Guest TDs can trust those CPUIDs and sha384 values for measurement. Add a new subcommand, KVM_TDX_INIT_VM, to pass parameters for the TDX guest. It assigns an encryption key to the TDX guest for memory encryption. TDX encrypts memory per guest basis. The device model, say qemu, passes per-VM parameters for the TDX guest. The maximum number of vcpus, TSC frequency (TDX guest has fixed VM-wide TSC frequency, not per vcpu. The TDX guest can not change it.), attributes (production or debug), available extended features (which configure guest XCR0, IA32_XSS MSR), CPUIDs, sha384 measurements, etc. Call this subcommand before creating vcpu and KVM_SET_CPUID2, i.e. CPUID configurations aren't available yet. So CPUIDs configuration values need to be passed in struct kvm_tdx_init_vm. The device model's responsibility to make this CPUID config for KVM_TDX_INIT_VM and KVM_SET_CPUID2. Signed-off-by: Xiaoyao Li Signed-off-by: Isaku Yamahata --- v14 -> v15: - add check if the reserved area of init_vm is zero Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/tdx.h | 3 + arch/x86/include/uapi/asm/kvm.h | 27 +++ arch/x86/kvm/cpuid.c | 7 + arch/x86/kvm/cpuid.h | 2 + arch/x86/kvm/vmx/tdx.c | 271 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 18 ++ tools/arch/x86/include/uapi/asm/kvm.h | 33 ++++ 7 files changed, 351 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 86517add595f..97b23325ba5e 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -85,6 +85,9 @@ static inline long tdx_kvm_hypercall(unsigned int nr, uns= igned long p1, #endif /* CONFIG_INTEL_TDX_GUEST && CONFIG_KVM_GUEST */ =20 #ifdef CONFIG_INTEL_TDX_HOST + +/* -1 indicates CPUID leaf with no sub-leaves. */ +#define TDX_CPUID_NO_SUBLEAF ((u32)-1) struct tdx_cpuid_config { __struct_group(tdx_cpuid_config_leaf, leaf_sub_leaf, __packed, u32 leaf; diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index 3fbd43d5177b..7112546bd1d0 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -570,6 +570,7 @@ struct kvm_pmu_event_filter { /* Trust Domain eXtension sub-ioctl() commands. */ enum kvm_tdx_cmd_id { KVM_TDX_CAPABILITIES =3D 0, + KVM_TDX_INIT_VM, =20 KVM_TDX_CMD_NR_MAX, }; @@ -617,4 +618,30 @@ struct kvm_tdx_capabilities { struct kvm_tdx_cpuid_config cpuid_configs[]; }; =20 +struct kvm_tdx_init_vm { + __u64 attributes; + __u64 mrconfigid[6]; /* sha384 digest */ + __u64 mrowner[6]; /* sha384 digest */ + __u64 mrownerconfig[6]; /* sha348 digest */ + /* + * For future extensibility to make sizeof(struct kvm_tdx_init_vm) =3D 8K= B. + * This should be enough given sizeof(TD_PARAMS) =3D 1024. + * 8KB was chosen given because + * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(=3D256) =3D 8K= B. + */ + __u64 reserved[1004]; + + /* + * Call KVM_TDX_INIT_VM before vcpu creation, thus before + * KVM_SET_CPUID2. + * This configuration supersedes KVM_SET_CPUID2s for VCPUs because the + * TDX module directly virtualizes those CPUIDs without VMM. The user + * space VMM, e.g. qemu, should make KVM_SET_CPUID2 consistent with + * those values. If it doesn't, KVM may have wrong idea of vCPUIDs of + * the guest, and KVM may wrongly emulate CPUIDs or MSRs that the TDX + * module doesn't virtualize. + */ + struct kvm_cpuid2 cpuid; +}; + #endif /* _ASM_X86_KVM_H */ diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 7f4d13383cf2..09b83f7c228d 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -1385,6 +1385,13 @@ int kvm_dev_ioctl_get_cpuid(struct kvm_cpuid2 *cpuid, return r; } =20 +struct kvm_cpuid_entry2 *kvm_find_cpuid_entry2( + struct kvm_cpuid_entry2 *entries, int nent, u32 function, u64 index) +{ + return cpuid_entry2_find(entries, nent, function, index); +} +EXPORT_SYMBOL_GPL(kvm_find_cpuid_entry2); + struct kvm_cpuid_entry2 *kvm_find_cpuid_entry_index(struct kvm_vcpu *vcpu, u32 function, u32 index) { diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h index b1658c0de847..6d94d852af9d 100644 --- a/arch/x86/kvm/cpuid.h +++ b/arch/x86/kvm/cpuid.h @@ -13,6 +13,8 @@ void kvm_set_cpu_caps(void); =20 void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu); void kvm_update_pv_runtime(struct kvm_vcpu *vcpu); +struct kvm_cpuid_entry2 *kvm_find_cpuid_entry2(struct kvm_cpuid_entry2 *en= tries, + int nent, u32 function, u64 index); struct kvm_cpuid_entry2 *kvm_find_cpuid_entry_index(struct kvm_vcpu *vcpu, u32 function, u32 index); struct kvm_cpuid_entry2 *kvm_find_cpuid_entry(struct kvm_vcpu *vcpu, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index af8c92f24f0f..eb94572631aa 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -7,7 +7,6 @@ #include "x86_ops.h" #include "mmu.h" #include "tdx.h" -#include "tdx_ops.h" #include "x86.h" =20 #undef pr_fmt @@ -298,18 +297,21 @@ static int tdx_do_tdh_mng_key_config(void *param) return 0; } =20 -static int __tdx_td_init(struct kvm *kvm); - int tdx_vm_init(struct kvm *kvm) { + /* + * This function initializes only KVM software construct. It doesn't + * initialize TDX stuff, e.g. TDCS, TDR, TDCX, HKID etc. + * It is handled by KVM_TDX_INIT_VM, __tdx_td_init(). + */ + /* * TDX has its own limit of the number of vcpus in addition to * KVM_MAX_VCPUS. */ kvm->max_vcpus =3D min(kvm->max_vcpus, TDX_MAX_VCPUS); =20 - /* Place holder for TDX specific logic. */ - return __tdx_td_init(kvm); + return 0; } =20 static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) @@ -372,9 +374,171 @@ static int tdx_get_capabilities(struct kvm_tdx_cmd *c= md) return ret; } =20 -static int __tdx_td_init(struct kvm *kvm) +static int setup_tdparams_eptp_controls(struct kvm_cpuid2 *cpuid, + struct td_params *td_params) +{ + const struct kvm_cpuid_entry2 *entry; + int max_pa =3D 36; + + entry =3D kvm_find_cpuid_entry2(cpuid->entries, cpuid->nent, 0x80000008, = 0); + if (entry) + max_pa =3D entry->eax & 0xff; + + td_params->eptp_controls =3D VMX_EPTP_MT_WB; + /* + * No CPU supports 4-level && max_pa > 48. + * "5-level paging and 5-level EPT" section 4.1 4-level EPT + * "4-level EPT is limited to translating 48-bit guest-physical + * addresses." + * cpu_has_vmx_ept_5levels() check is just in case. + */ + if (!cpu_has_vmx_ept_5levels() && max_pa > 48) + return -EINVAL; + if (cpu_has_vmx_ept_5levels() && max_pa > 48) { + td_params->eptp_controls |=3D VMX_EPTP_PWL_5; + td_params->exec_controls |=3D TDX_EXEC_CONTROL_MAX_GPAW; + } else { + td_params->eptp_controls |=3D VMX_EPTP_PWL_4; + } + + return 0; +} + +static void setup_tdparams_cpuids(const struct tdsysinfo_struct *tdsysinfo, + struct kvm_cpuid2 *cpuid, + struct td_params *td_params) +{ + int i; + + /* + * td_params.cpuid_values: The number and the order of cpuid_value must + * be same to the one of struct tdsysinfo.{num_cpuid_config, cpuid_config= s} + * It's assumed that td_params was zeroed. + */ + for (i =3D 0; i < tdsysinfo->num_cpuid_config; i++) { + const struct tdx_cpuid_config *config =3D &tdsysinfo->cpuid_configs[i]; + /* TDX_CPUID_NO_SUBLEAF in TDX CPUID_CONFIG means index =3D 0. */ + u32 index =3D config->sub_leaf =3D=3D TDX_CPUID_NO_SUBLEAF ? 0 : config-= >sub_leaf; + const struct kvm_cpuid_entry2 *entry =3D + kvm_find_cpuid_entry2(cpuid->entries, cpuid->nent, + config->leaf, index); + struct tdx_cpuid_value *value =3D &td_params->cpuid_values[i]; + + if (!entry) + continue; + + /* + * tdsysinfo.cpuid_configs[].{eax, ebx, ecx, edx} + * bit 1 means it can be configured to zero or one. + * bit 0 means it must be zero. + * Mask out non-configurable bits. + */ + value->eax =3D entry->eax & config->eax; + value->ebx =3D entry->ebx & config->ebx; + value->ecx =3D entry->ecx & config->ecx; + value->edx =3D entry->edx & config->edx; + } +} + +static int setup_tdparams_xfam(struct kvm_cpuid2 *cpuid, struct td_params = *td_params) +{ + const struct kvm_cpuid_entry2 *entry; + u64 guest_supported_xcr0; + u64 guest_supported_xss; + + /* Setup td_params.xfam */ + entry =3D kvm_find_cpuid_entry2(cpuid->entries, cpuid->nent, 0xd, 0); + if (entry) + guest_supported_xcr0 =3D (entry->eax | ((u64)entry->edx << 32)); + else + guest_supported_xcr0 =3D 0; + guest_supported_xcr0 &=3D kvm_caps.supported_xcr0; + + entry =3D kvm_find_cpuid_entry2(cpuid->entries, cpuid->nent, 0xd, 1); + if (entry) + guest_supported_xss =3D (entry->ecx | ((u64)entry->edx << 32)); + else + guest_supported_xss =3D 0; + /* PT can be exposed to TD guest regardless of KVM's XSS support */ + guest_supported_xss &=3D (kvm_caps.supported_xss | XFEATURE_MASK_PT); + + td_params->xfam =3D guest_supported_xcr0 | guest_supported_xss; + if (td_params->xfam & XFEATURE_MASK_LBR) { + /* + * TODO: once KVM supports LBR(save/restore LBR related + * registers around TDENTER), remove this guard. + */ +#define MSG_LBR "TD doesn't support LBR yet. KVM needs to save/restore IA3= 2_LBR_DEPTH properly.\n" + pr_warn(MSG_LBR); + return -EOPNOTSUPP; + } + + if (td_params->xfam & XFEATURE_MASK_XTILE) { + /* + * TODO: once KVM supports AMX(save/restore AMX related + * registers around TDENTER), remove this guard. + */ +#define MSG_AMX "TD doesn't support AMX yet. KVM needs to save/restore IA3= 2_XFD, IA32_XFD_ERR properly.\n" + pr_warn(MSG_AMX); + return -EOPNOTSUPP; + } + + return 0; +} + +static int setup_tdparams(struct kvm *kvm, struct td_params *td_params, + struct kvm_tdx_init_vm *init_vm) +{ + struct kvm_cpuid2 *cpuid =3D &init_vm->cpuid; + const struct tdsysinfo_struct *tdsysinfo; + int ret; + + tdsysinfo =3D tdx_get_sysinfo(); + if (!tdsysinfo) + return -EOPNOTSUPP; + if (kvm->created_vcpus) + return -EBUSY; + + if (td_params->attributes & TDX_TD_ATTRIBUTE_PERFMON) { + /* + * TODO: save/restore PMU related registers around TDENTER. + * Once it's done, remove this guard. + */ +#define MSG_PERFMON "TD doesn't support perfmon yet. KVM needs to save/res= tore host perf registers properly.\n" + pr_warn(MSG_PERFMON); + return -EOPNOTSUPP; + } + + td_params->max_vcpus =3D kvm->max_vcpus; + td_params->attributes =3D init_vm->attributes; + td_params->tsc_frequency =3D TDX_TSC_KHZ_TO_25MHZ(kvm->arch.default_tsc_k= hz); + + ret =3D setup_tdparams_eptp_controls(cpuid, td_params); + if (ret) + return ret; + setup_tdparams_cpuids(tdsysinfo, cpuid, td_params); + ret =3D setup_tdparams_xfam(cpuid, td_params); + if (ret) + return ret; + +#define MEMCPY_SAME_SIZE(dst, src) \ + do { \ + BUILD_BUG_ON(sizeof(dst) !=3D sizeof(src)); \ + memcpy((dst), (src), sizeof(dst)); \ + } while (0) + + MEMCPY_SAME_SIZE(td_params->mrconfigid, init_vm->mrconfigid); + MEMCPY_SAME_SIZE(td_params->mrowner, init_vm->mrowner); + MEMCPY_SAME_SIZE(td_params->mrownerconfig, init_vm->mrownerconfig); + + return 0; +} + +static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, + u64 *seamcall_err) { struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + struct tdx_module_output out; cpumask_var_t packages; unsigned long *tdcs_pa =3D NULL; unsigned long tdr_pa =3D 0; @@ -382,6 +546,7 @@ static int __tdx_td_init(struct kvm *kvm) int ret, i; u64 err; =20 + *seamcall_err =3D 0; ret =3D tdx_guest_keyid_alloc(); if (ret < 0) return ret; @@ -487,10 +652,23 @@ static int __tdx_td_init(struct kvm *kvm) } } =20 - /* - * Note, TDH_MNG_INIT cannot be invoked here. TDH_MNG_INIT requires a de= dicated - * ioctl() to define the configure CPUID values for the TD. - */ + err =3D tdh_mng_init(kvm_tdx->tdr_pa, __pa(td_params), &out); + if ((err & TDX_SEAMCALL_STATUS_MASK) =3D=3D TDX_OPERAND_INVALID) { + /* + * Because a user gives operands, don't warn. + * Return a hint to the user because it's sometimes hard for the + * user to figure out which operand is invalid. SEAMCALL status + * code includes which operand caused invalid operand error. + */ + *seamcall_err =3D err; + ret =3D -EINVAL; + goto teardown; + } else if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MNG_INIT, err, &out); + ret =3D -EIO; + goto teardown; + } + return 0; =20 /* @@ -533,6 +711,76 @@ static int __tdx_td_init(struct kvm *kvm) return ret; } =20 +static int tdx_td_init(struct kvm *kvm, struct kvm_tdx_cmd *cmd) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + struct kvm_tdx_init_vm *init_vm =3D NULL; + struct td_params *td_params =3D NULL; + int ret; + + BUILD_BUG_ON(sizeof(*init_vm) !=3D 8 * 1024); + BUILD_BUG_ON(sizeof(struct td_params) !=3D 1024); + + if (is_hkid_assigned(kvm_tdx)) + return -EINVAL; + + if (cmd->flags) + return -EINVAL; + + init_vm =3D kzalloc(sizeof(*init_vm) + + sizeof(init_vm->cpuid.entries[0]) * KVM_MAX_CPUID_ENTRIES, + GFP_KERNEL); + if (!init_vm) + return -ENOMEM; + if (copy_from_user(init_vm, (void __user *)cmd->data, sizeof(*init_vm))) { + ret =3D -EFAULT; + goto out; + } + if (init_vm->cpuid.nent > KVM_MAX_CPUID_ENTRIES) { + ret =3D -E2BIG; + goto out; + } + if (copy_from_user(init_vm->cpuid.entries, + (void __user *)cmd->data + sizeof(*init_vm), + sizeof(init_vm->cpuid.entries[0]) * init_vm->cpuid.nent)) { + ret =3D -EFAULT; + goto out; + } + + if (memchr_inv(init_vm->reserved, 0, sizeof(init_vm->reserved))) { + ret =3D -EINVAL; + goto out; + } + if (init_vm->cpuid.padding) { + ret =3D -EINVAL; + goto out; + } + + td_params =3D kzalloc(sizeof(struct td_params), GFP_KERNEL); + if (!td_params) { + ret =3D -ENOMEM; + goto out; + } + + ret =3D setup_tdparams(kvm, td_params, init_vm); + if (ret) + goto out; + + ret =3D __tdx_td_init(kvm, td_params, &cmd->error); + if (ret) + goto out; + + kvm_tdx->tsc_offset =3D td_tdcs_exec_read64(kvm_tdx, TD_TDCS_EXEC_TSC_OFF= SET); + kvm_tdx->attributes =3D td_params->attributes; + kvm_tdx->xfam =3D td_params->xfam; + +out: + /* kfree() accepts NULL. */ + kfree(init_vm); + kfree(td_params); + return ret; +} + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { struct kvm_tdx_cmd tdx_cmd; @@ -549,6 +797,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) case KVM_TDX_CAPABILITIES: r =3D tdx_get_capabilities(&tdx_cmd); break; + case KVM_TDX_INIT_VM: + r =3D tdx_td_init(kvm, &tdx_cmd); + break; default: r =3D -EINVAL; goto out; diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index ae117f864cfb..646989eac5e3 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -12,7 +12,11 @@ struct kvm_tdx { unsigned long tdr_pa; unsigned long *tdcs_pa; =20 + u64 attributes; + u64 xfam; int hkid; + + u64 tsc_offset; }; =20 struct vcpu_tdx { @@ -39,6 +43,20 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *v= cpu) { return container_of(vcpu, struct vcpu_tdx, vcpu); } + +static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u3= 2 field) +{ + struct tdx_module_output out; + u64 err; + + err =3D tdh_mng_rd(kvm_tdx->tdr_pa, TDCS_EXEC(field), &out); + if (unlikely(err)) { + pr_err("TDH_MNG_RD[EXEC.0x%x] failed: 0x%llx\n", field, err); + return 0; + } + return out.r8; +} + #else struct kvm_tdx { struct kvm kvm; diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include= /uapi/asm/kvm.h index 7a08723e99e2..61ce7d174fcf 100644 --- a/tools/arch/x86/include/uapi/asm/kvm.h +++ b/tools/arch/x86/include/uapi/asm/kvm.h @@ -565,6 +565,7 @@ struct kvm_pmu_event_filter { /* Trust Domain eXtension sub-ioctl() commands. */ enum kvm_tdx_cmd_id { KVM_TDX_CAPABILITIES =3D 0, + KVM_TDX_INIT_VM, =20 KVM_TDX_CMD_NR_MAX, }; @@ -614,4 +615,36 @@ struct kvm_tdx_capabilities { struct kvm_tdx_cpuid_config cpuid_configs[]; }; =20 +struct kvm_tdx_init_vm { + __u64 attributes; + __u32 max_vcpus; + __u32 padding; + __u64 mrconfigid[6]; /* sha384 digest */ + __u64 mrowner[6]; /* sha384 digest */ + __u64 mrownerconfig[6]; /* sha348 digest */ + union { + /* + * KVM_TDX_INIT_VM is called before vcpu creation, thus before + * KVM_SET_CPUID2. CPUID configurations needs to be passed. + * + * This configuration supersedes KVM_SET_CPUID{,2}. + * The user space VMM, e.g. qemu, should make them consistent + * with this values. + * sizeof(struct kvm_cpuid_entry2) * KVM_MAX_CPUID_ENTRIES(256) + * =3D 8KB. + */ + struct { + struct kvm_cpuid2 cpuid; + /* 8KB with KVM_MAX_CPUID_ENTRIES. */ + struct kvm_cpuid_entry2 entries[]; + }; + /* + * For future extensibility. + * The size(struct kvm_tdx_init_vm) =3D 16KB. + * This should be enough given sizeof(TD_PARAMS) =3D 1024 + */ + __u64 reserved[2028]; + }; +}; + #endif /* _ASM_X86_KVM_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6DB8C41513 for ; Tue, 25 Jul 2023 22:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231345AbjGYWRC (ORCPT ); Tue, 25 Jul 2023 18:17:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231743AbjGYWPx (ORCPT ); Tue, 25 Jul 2023 18:15:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A81BF10F1; Tue, 25 Jul 2023 15:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323343; x=1721859343; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PnACD10nMFjfy7BWoTwQP9iGyvXHJreWsftSNyiBiNY=; b=LZWmvkXOvdniv2Jdm6c3OFKoowf/dncj1vFZGQdrW0FMQAElWkX0q5LG yTSCyFHi2h+6VS+wZhXza4C7OJYhBaJ9QUHPPcmPSnNurutPSrgoE6fsV B2pAg2gSYFCpmu2pM+49l5ipKI11wXneKrzIiZRow8GU/SEZrC2q3Cptr UOYl3SedJs0Ags3ppuCPRnEd1rydMglkQITuu7ScGvuZ8E9Jn4RnSNl8w tmtrUr917IVTTnOWhhPEJe8hV75am8Zzu5apVpuuAhUZs5Ji3NQdSJsVv ulKXk0w7BEuo+45xcHOul4DeOjy+OcUV7foCfMA94f+NqFjcssrOypne0 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863145" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863145" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938837" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938837" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:23 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 022/115] KVM: TDX: Make pmu_intel.c ignore guest TD case Date: Tue, 25 Jul 2023 15:13:33 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because TDX KVM doesn't support PMU yet (it's future work of TDX KVM support as another patch series) and pmu_intel.c touches vmx specific structure in vcpu initialization, as workaround add dummy structure to struct vcpu_tdx and pmu_intel.c can ignore TDX case. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/pmu_intel.c | 46 +++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/pmu_intel.h | 28 ++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 8 ++++++- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/vmx/vmx.h | 32 +------------------------ 5 files changed, 82 insertions(+), 34 deletions(-) create mode 100644 arch/x86/kvm/vmx/pmu_intel.h diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index 80c769c58a87..7f9d6eba77b6 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -19,6 +19,7 @@ #include "lapic.h" #include "nested.h" #include "pmu.h" +#include "tdx.h" =20 #define MSR_PMC_FULL_WIDTH_BIT (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0) =20 @@ -40,6 +41,26 @@ static struct { /* mapping between fixed pmc index and intel_arch_events array */ static int fixed_pmc_events[] =3D {1, 0, 7}; =20 +struct lbr_desc *vcpu_to_lbr_desc(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_INTEL_TDX_HOST + if (is_td_vcpu(vcpu)) + return &to_tdx(vcpu)->lbr_desc; +#endif + + return &to_vmx(vcpu)->lbr_desc; +} + +struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_INTEL_TDX_HOST + if (is_td_vcpu(vcpu)) + return &to_tdx(vcpu)->lbr_desc.records; +#endif + + return &to_vmx(vcpu)->lbr_desc.records; +} + static void reprogram_fixed_counters(struct kvm_pmu *pmu, u64 data) { struct kvm_pmc *pmc; @@ -149,6 +170,23 @@ static inline struct kvm_pmc *get_fw_gp_pmc(struct kvm= _pmu *pmu, u32 msr) return get_gp_pmc(pmu, msr, MSR_IA32_PMC0); } =20 +bool intel_pmu_lbr_is_compatible(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return false; + return cpuid_model_is_consistent(vcpu); +} + +bool intel_pmu_lbr_is_enabled(struct kvm_vcpu *vcpu) +{ + struct x86_pmu_lbr *lbr =3D vcpu_to_lbr_records(vcpu); + + if (is_td_vcpu(vcpu)) + return false; + + return lbr->nr && (vcpu_get_perf_capabilities(vcpu) & PMU_CAP_LBR_FMT); +} + static bool intel_pmu_is_valid_lbr_msr(struct kvm_vcpu *vcpu, u32 index) { struct x86_pmu_lbr *records =3D vcpu_to_lbr_records(vcpu); @@ -255,6 +293,9 @@ int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *v= cpu) PERF_SAMPLE_BRANCH_USER, }; =20 + if (WARN_ON_ONCE(is_td_vcpu(vcpu))) + return 0; + if (unlikely(lbr_desc->event)) { __set_bit(INTEL_PMC_IDX_FIXED_VLBR, pmu->pmc_in_use); return 0; @@ -551,7 +592,7 @@ static void intel_pmu_refresh(struct kvm_vcpu *vcpu) INTEL_PMC_MAX_GENERIC, pmu->nr_arch_fixed_counters); =20 perf_capabilities =3D vcpu_get_perf_capabilities(vcpu); - if (cpuid_model_is_consistent(vcpu) && + if (intel_pmu_lbr_is_compatible(vcpu) && (perf_capabilities & PMU_CAP_LBR_FMT)) x86_perf_get_lbr(&lbr_desc->records); else @@ -607,6 +648,9 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu) struct kvm_pmc *pmc =3D NULL; int i; =20 + if (is_td_vcpu(vcpu)) + return; + for (i =3D 0; i < KVM_INTEL_PMC_MAX_GENERIC; i++) { pmc =3D &pmu->gp_counters[i]; =20 diff --git a/arch/x86/kvm/vmx/pmu_intel.h b/arch/x86/kvm/vmx/pmu_intel.h new file mode 100644 index 000000000000..66bba47c1269 --- /dev/null +++ b/arch/x86/kvm/vmx/pmu_intel.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMX_PMU_INTEL_H +#define __KVM_X86_VMX_PMU_INTEL_H + +struct lbr_desc *vcpu_to_lbr_desc(struct kvm_vcpu *vcpu); +struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcpu); + +bool intel_pmu_lbr_is_compatible(struct kvm_vcpu *vcpu); +bool intel_pmu_lbr_is_enabled(struct kvm_vcpu *vcpu); +int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu); + +struct lbr_desc { + /* Basic info about guest LBR records. */ + struct x86_pmu_lbr records; + + /* + * Emulate LBR feature via passthrough LBR registers when the + * per-vcpu guest LBR event is scheduled on the current pcpu. + * + * The records may be inaccurate if the host reclaims the LBR. + */ + struct perf_event *event; + + /* True if LBRs are marked as not intercepted in the MSR bitmap */ + bool msr_passthrough; +}; + +#endif /* __KVM_X86_VMX_PMU_INTEL_H */ diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 646989eac5e3..af7fdc1516d5 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -4,6 +4,7 @@ =20 #ifdef CONFIG_INTEL_TDX_HOST =20 +#include "pmu_intel.h" #include "tdx_ops.h" =20 struct kvm_tdx { @@ -21,7 +22,12 @@ struct kvm_tdx { =20 struct vcpu_tdx { struct kvm_vcpu vcpu; - /* TDX specific members follow. */ + + /* + * Dummy to make pmu_intel not corrupt memory. + * TODO: Support PMU for TDX. Future work. + */ + struct lbr_desc lbr_desc; }; =20 static inline bool is_td(struct kvm *kvm) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 76e444c3e865..540674f1ef2f 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -2403,7 +2403,7 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_dat= a *msr_info) if ((data & PMU_CAP_LBR_FMT) !=3D (kvm_caps.supported_perf_cap & PMU_CAP_LBR_FMT)) return 1; - if (!cpuid_model_is_consistent(vcpu)) + if (!intel_pmu_lbr_is_compatible(vcpu)) return 1; } if (data & PERF_CAP_PEBS_FORMAT) { diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 32384ba38499..016a9499b577 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -11,6 +11,7 @@ #include "capabilities.h" #include "../kvm_cache_regs.h" #include "posted_intr.h" +#include "pmu_intel.h" #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h" @@ -93,22 +94,6 @@ union vmx_exit_reason { u32 full; }; =20 -struct lbr_desc { - /* Basic info about guest LBR records. */ - struct x86_pmu_lbr records; - - /* - * Emulate LBR feature via passthrough LBR registers when the - * per-vcpu guest LBR event is scheduled on the current pcpu. - * - * The records may be inaccurate if the host reclaims the LBR. - */ - struct perf_event *event; - - /* True if LBRs are marked as not intercepted in the MSR bitmap */ - bool msr_passthrough; -}; - /* * The nested_vmx structure is part of vcpu_vmx, and holds information we = need * for correct emulation of VMX (i.e., nested VMX) on this vcpu. @@ -656,21 +641,6 @@ static __always_inline struct vcpu_vmx *to_vmx(struct = kvm_vcpu *vcpu) return container_of(vcpu, struct vcpu_vmx, vcpu); } =20 -static inline struct lbr_desc *vcpu_to_lbr_desc(struct kvm_vcpu *vcpu) -{ - return &to_vmx(vcpu)->lbr_desc; -} - -static inline struct x86_pmu_lbr *vcpu_to_lbr_records(struct kvm_vcpu *vcp= u) -{ - return &vcpu_to_lbr_desc(vcpu)->records; -} - -static inline bool intel_pmu_lbr_is_enabled(struct kvm_vcpu *vcpu) -{ - return !!vcpu_to_lbr_records(vcpu)->nr; -} - void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu); int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu); void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CFAFC04FDF for ; Tue, 25 Jul 2023 22:17:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231675AbjGYWRU (ORCPT ); Tue, 25 Jul 2023 18:17:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231978AbjGYWQI (ORCPT ); Tue, 25 Jul 2023 18:16:08 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E1FC2696; Tue, 25 Jul 2023 15:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323346; x=1721859346; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lnuf3/jjPZZS0jUzvhyB2DBfnZhKXUEO0TV4zg7LKf8=; b=gssJLtyH6IXneR2b3bsBjQgEePfZKbZa/B8HPmsQbQB1cX3BuGYbeZ6W oEjrFxx8lDm9w5Wi68xmEwXUroS6yJA3GRK92+gk0/n2CpwARZv7iU9nv fbjYkIJJxwSFRf8j0s4zfSL9KfOmvTDZWQyHXEM1SrCYf69ZwVwpon6QN tDhFD7H5bVEVScn0e5LLFXzAPkcuHIQrsv+lrcKUPV1qD5RKqTP0RaZ1h 9Z02i7nXXXIW5mLAZqu6WHLSyq+fg59inhsOHH8XCfSwO8UGmCTwN7Z+f 9cDbwXnUOn6SrDb3xMY+lBa6M8AZPksWn58sM1RhTi8jhDB91gkp9m81e g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863151" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863151" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938841" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938841" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:24 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 023/115] KVM: TDX: Refuse to unplug the last cpu on the package Date: Tue, 25 Jul 2023 15:13:34 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata In order to reclaim TDX HKID, (i.e. when deleting guest TD), needs to call TDH.PHYMEM.PAGE.WBINVD on all packages. If we have active TDX HKID, refuse to offline the last online cpu to guarantee at least one CPU online per package. Add arch callback for cpu offline. Because TDX doesn't support suspend by the TDX 1.0 spec, this also refuses suspend if TDs are running. If no TD is running, suspend is allowed. Suggested-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/main.c | 1 + arch/x86/kvm/vmx/tdx.c | 44 +++++++++++++++++++++++++++++- arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 5 ++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 12 ++++++-- 8 files changed, 64 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index a574e7eb04f3..d711829fb26a 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -18,6 +18,7 @@ KVM_X86_OP(check_processor_compatibility) KVM_X86_OP(hardware_enable) KVM_X86_OP(hardware_disable) KVM_X86_OP(hardware_unsetup) +KVM_X86_OP_OPTIONAL_RET0(offline_cpu) KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 6ce2f512458e..5deb39828820 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1542,6 +1542,7 @@ struct kvm_x86_ops { int (*hardware_enable)(void); void (*hardware_disable)(void); void (*hardware_unsetup)(void); + int (*offline_cpu)(void); bool (*has_emulated_msr)(struct kvm *kvm, u32 index); void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu); =20 diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index ef08a46b04b3..d9c8becfe749 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -121,6 +121,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .check_processor_compatibility =3D vmx_check_processor_compat, =20 .hardware_unsetup =3D vt_hardware_unsetup, + .offline_cpu =3D tdx_offline_cpu, =20 .hardware_enable =3D vt_hardware_enable, .hardware_disable =3D vmx_hardware_disable, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index eb94572631aa..36d687e7c3f3 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -64,6 +64,7 @@ static struct tdx_info tdx_info __ro_after_init; */ static DEFINE_MUTEX(tdx_lock); static struct mutex *tdx_mng_key_config_lock; +static atomic_t nr_configured_hkid; =20 static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid) { @@ -232,7 +233,8 @@ void tdx_mmu_release_hkid(struct kvm *kvm) pr_err("tdh_mng_key_freeid failed. HKID %d is leaked.\n", kvm_tdx->hkid); return; - } + } else + atomic_dec(&nr_configured_hkid); =20 free_hkid: tdx_hkid_free(kvm_tdx); @@ -635,6 +637,8 @@ static int __tdx_td_init(struct kvm *kvm, struct td_par= ams *td_params, if (ret) break; } + if (!ret) + atomic_inc(&nr_configured_hkid); cpus_read_unlock(); free_cpumask_var(packages); if (ret) { @@ -921,3 +925,41 @@ void tdx_hardware_unsetup(void) /* kfree accepts NULL. */ kfree(tdx_mng_key_config_lock); } + +int tdx_offline_cpu(void) +{ + int curr_cpu =3D smp_processor_id(); + cpumask_var_t packages; + int ret =3D 0; + int i; + + /* No TD is running. Allow any cpu to be offline. */ + if (!atomic_read(&nr_configured_hkid)) + return 0; + + /* + * In order to reclaim TDX HKID, (i.e. when deleting guest TD), need to + * call TDH.PHYMEM.PAGE.WBINVD on all packages to program all memory + * controller with pconfig. If we have active TDX HKID, refuse to + * offline the last online cpu. + */ + if (!zalloc_cpumask_var(&packages, GFP_KERNEL)) + return -ENOMEM; + for_each_online_cpu(i) { + if (i !=3D curr_cpu) + cpumask_set_cpu(topology_physical_package_id(i), packages); + } + /* Check if this cpu is the last online cpu of this package. */ + if (!cpumask_test_cpu(topology_physical_package_id(curr_cpu), packages)) + ret =3D -EBUSY; + free_cpumask_var(packages); + if (ret) + /* + * Because it's hard for human operator to understand the + * reason, warn it. + */ +#define MSG_ALLPKG_ONLINE \ + "TDX requires all packages to have an online CPU. Delete all TDs in order= to offline all CPUs of a package.\n" + pr_warn_ratelimited(MSG_ALLPKG_ONLINE); + return ret; +} diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index fc5348dd20da..9394a7148c5e 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -139,6 +139,7 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu); int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); void tdx_hardware_unsetup(void); bool tdx_is_vm_type_supported(unsigned long type); +int tdx_offline_cpu(void); =20 int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap); int tdx_vm_init(struct kvm *kvm); @@ -149,6 +150,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline void tdx_hardware_unsetup(void) {} static inline bool tdx_is_vm_type_supported(unsigned long type) { return f= alse; } +static inline int tdx_offline_cpu(void) { return 0; } =20 static inline int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_cap= *cap) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index bc7cdd8cbbb0..29a71f722fbb 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -12259,6 +12259,11 @@ void kvm_arch_hardware_disable(void) drop_user_return_notifiers(); } =20 +int kvm_arch_offline_cpu(unsigned int cpu) +{ + return static_call(kvm_x86_offline_cpu)(); +} + bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu) { return vcpu->kvm->arch.bsp_vcpu_id =3D=3D vcpu->vcpu_id; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 42b5a2ccc9d1..e8770afce5cf 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1486,6 +1486,7 @@ static inline void kvm_create_vcpu_debugfs(struct kvm= _vcpu *vcpu) {} int kvm_arch_hardware_enable(void); void kvm_arch_hardware_disable(void); #endif +int kvm_arch_offline_cpu(unsigned int cpu); int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu); bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu); int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index ec2e879bb3f2..60ed0f613bce 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5441,13 +5441,21 @@ static void hardware_disable_nolock(void *junk) __this_cpu_write(hardware_enabled, false); } =20 +__weak int kvm_arch_offline_cpu(unsigned int cpu) +{ + return 0; +} + static int kvm_offline_cpu(unsigned int cpu) { + int r =3D 0; + mutex_lock(&kvm_lock); - if (kvm_usage_count) + r =3D kvm_arch_offline_cpu(cpu); + if (!r && kvm_usage_count) hardware_disable_nolock(NULL); mutex_unlock(&kvm_lock); - return 0; + return r; } =20 static void hardware_disable_all_nolock(void) --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A145C001DF for ; Tue, 25 Jul 2023 22:17:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231195AbjGYWRb (ORCPT ); Tue, 25 Jul 2023 18:17:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232085AbjGYWQK (ORCPT ); Tue, 25 Jul 2023 18:16:10 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C994E74; Tue, 25 Jul 2023 15:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323346; x=1721859346; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=i0N39TZmKLiC6CILEgWJyLqxHNkrsso5GLu5HE01vt8=; b=EjB+E/Go1g8ebtHOoFQoq/9MgmOunj6EL9A3+ogrnsVTqF7qrckcCR8l frZYOh07f20KIo9zRqKgh7mWHGLTEvIfXqu04z7ZhsFPhoMYdxf1TyLma 1dxONIuTTU7S4iOCpFYAdXwbxSqKA44O0aIXIixYhptReaL6ExKusK/BU EBglHIlNEKQqKnhhtTgZgnNq0TTQ2DkkFpIPs6beJ/Q0BV68mMzL9f/kt N0+bf78F8x8PzieQVGx79UI87PzGDZQ5au/kdlYzyHlP1CVf1iEdV68PD UfyYeglhoJweX1E0Bgqi6g/3mfnndIid5DHwOfnOCCAbWU06o/7Fc+b2T A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863157" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863157" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938844" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938844" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:24 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 024/115] [MARKER] The start of TDX KVM patch series: TD vcpu creation/destruction Date: Tue, 25 Jul 2023 15:13:35 -0700 Message-Id: <0f60471679e2f3e80caf4134114a1a0092e554c1.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TD vcpu creation/destruction. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index 098150da6ea2..25082e9c0b20 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -9,7 +9,7 @@ Layer status What qemu can do ---------------- - TDX VM TYPE is exposed to Qemu. -- Qemu can try to create VM of TDX VM type and then fails. +- Qemu can create/destroy guest of TDX vm type. =20 Patch Layer status ------------------ @@ -17,8 +17,8 @@ Patch Layer status =20 * TDX, VMX coexistence: Applied * TDX architectural definitions: Applied -* TD VM creation/destruction: Applying -* TD vcpu creation/destruction: Not yet +* TD VM creation/destruction: Applied +* TD vcpu creation/destruction: Applying * TDX EPT violation: Not yet * TD finalization: Not yet * TD vcpu enter/exit: Not yet --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00CB1C04FDF for ; Tue, 25 Jul 2023 22:17:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232149AbjGYWRi (ORCPT ); Tue, 25 Jul 2023 18:17:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232129AbjGYWQK (ORCPT ); Tue, 25 Jul 2023 18:16:10 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CA7726BB; Tue, 25 Jul 2023 15:15:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323347; x=1721859347; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VpOzptfsxF3R9fDSCTV2aFECXUdM5I95VrP/+dTQHbU=; b=V6OqX24zGAFU9EIsh8MFqWrcn/cEcCMnA7ZjT5BK9BI+8f2u39/yTw/8 HER1BdhT5msl91iDVKEJl/2wcVX55O6rhW1skuwYR3JunHXUfnOqkUdLt EqMq01d6EAMLmI4zPlh4xK8uAbmhLk3K8rSOJ1Z10vmGsGuArQB6pT+qS Cm5XGVjr/2BHiQTZl9Hh2ginrLwwVRo2KmhMqT2cagZtUomt+SF2UrPRF v29e7DARJD+eQ0GAkKN0p+QITppnlt27S5XctT7O46+0SblRAGqRyH3GW sjAL6XHMq5l5WaqnR2yL7fiOFn+QRTdQiYPcThgohbWxtik0HgUc1aoCf Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863161" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863161" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938848" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938848" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:25 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 025/115] KVM: TDX: allocate/free TDX vcpu structure Date: Tue, 25 Jul 2023 15:13:36 -0700 Message-Id: <8e54ce03a43838906ceae5866cc0c57d0e112eee.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata The next step of TDX guest creation is to create vcpu. Allocate TDX vcpu structures, initialize it that doesn't require TDX SEAMCALL. TDX specific vcpu initialization will be implemented as independent KVM_TDX_INIT_VCPU so that when error occurs it's easy to determine which component has the issue, KVM or TDX. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 44 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 44 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 10 +++++++++ arch/x86/kvm/x86.c | 2 ++ 4 files changed, 96 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index d9c8becfe749..6ed9116f1b5c 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -98,6 +98,42 @@ static void vt_vm_free(struct kvm *kvm) tdx_vm_free(kvm); } =20 +static int vt_vcpu_precreate(struct kvm *kvm) +{ + if (is_td(kvm)) + return 0; + + return vmx_vcpu_precreate(kvm); +} + +static int vt_vcpu_create(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return tdx_vcpu_create(vcpu); + + return vmx_vcpu_create(vcpu); +} + +static void vt_vcpu_free(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_vcpu_free(vcpu); + return; + } + + vmx_vcpu_free(vcpu); +} + +static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) +{ + if (is_td_vcpu(vcpu)) { + tdx_vcpu_reset(vcpu, init_event); + return; + } + + vmx_vcpu_reset(vcpu, init_event); +} + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -136,10 +172,10 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .vm_destroy =3D vt_vm_destroy, .vm_free =3D vt_vm_free, =20 - .vcpu_precreate =3D vmx_vcpu_precreate, - .vcpu_create =3D vmx_vcpu_create, - .vcpu_free =3D vmx_vcpu_free, - .vcpu_reset =3D vmx_vcpu_reset, + .vcpu_precreate =3D vt_vcpu_precreate, + .vcpu_create =3D vt_vcpu_create, + .vcpu_free =3D vt_vcpu_free, + .vcpu_reset =3D vt_vcpu_reset, =20 .prepare_switch_to_guest =3D vmx_prepare_switch_to_guest, .vcpu_load =3D vmx_vcpu_load, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 36d687e7c3f3..5f5e451b201a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -316,6 +316,50 @@ int tdx_vm_init(struct kvm *kvm) return 0; } =20 +int tdx_vcpu_create(struct kvm_vcpu *vcpu) +{ + /* + * On cpu creation, cpuid entry is blank. Forcibly enable + * X2APIC feature to allow X2APIC. + * Because vcpu_reset() can't return error, allocation is done here. + */ + WARN_ON_ONCE(vcpu->arch.cpuid_entries); + WARN_ON_ONCE(vcpu->arch.cpuid_nent); + + /* TDX only supports x2APIC, which requires an in-kernel local APIC. */ + if (!vcpu->arch.apic) + return -EINVAL; + + fpstate_set_confidential(&vcpu->arch.guest_fpu); + + vcpu->arch.efer =3D EFER_SCE | EFER_LME | EFER_LMA | EFER_NX; + + vcpu->arch.cr0_guest_owned_bits =3D -1ul; + vcpu->arch.cr4_guest_owned_bits =3D -1ul; + + vcpu->arch.tsc_offset =3D to_kvm_tdx(vcpu->kvm)->tsc_offset; + vcpu->arch.l1_tsc_offset =3D vcpu->arch.tsc_offset; + vcpu->arch.guest_state_protected =3D + !(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG); + + return 0; +} + +void tdx_vcpu_free(struct kvm_vcpu *vcpu) +{ + /* This is stub for now. More logic will come. */ +} + +void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) +{ + + /* Ignore INIT silently because TDX doesn't support INIT event. */ + if (init_event) + return; + + /* This is stub for now. More logic will come here. */ +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 9394a7148c5e..aaa419363276 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -145,7 +145,12 @@ int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enab= le_cap *cap); int tdx_vm_init(struct kvm *kvm); void tdx_mmu_release_hkid(struct kvm *kvm); void tdx_vm_free(struct kvm *kvm); + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); + +int tdx_vcpu_create(struct kvm_vcpu *vcpu); +void tdx_vcpu_free(struct kvm_vcpu *vcpu); +void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline void tdx_hardware_unsetup(void) {} @@ -159,7 +164,12 @@ static inline int tdx_vm_enable_cap(struct kvm *kvm, s= truct kvm_enable_cap *cap) static inline int tdx_vm_init(struct kvm *kvm) { return -EOPNOTSUPP; } static inline void tdx_mmu_release_hkid(struct kvm *kvm) {} static inline void tdx_vm_free(struct kvm *kvm) {} + static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { retur= n -EOPNOTSUPP; } + +static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTS= UPP; } +static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) = {} #endif =20 #endif /* __KVM_X86_VMX_X86_OPS_H */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 29a71f722fbb..4afe24f50dcb 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -498,6 +498,7 @@ int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr= _data *msr_info) kvm_recalculate_apic_map(vcpu->kvm); return 0; } +EXPORT_SYMBOL_GPL(kvm_set_apic_base); =20 /* * Handle a fault on a hardware virtualization (VMX or SVM) instruction. @@ -12268,6 +12269,7 @@ bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu) { return vcpu->kvm->arch.bsp_vcpu_id =3D=3D vcpu->vcpu_id; } +EXPORT_SYMBOL_GPL(kvm_vcpu_is_reset_bsp); =20 bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu) { --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32C56EB64DD for ; Tue, 25 Jul 2023 22:17:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229626AbjGYWR2 (ORCPT ); Tue, 25 Jul 2023 18:17:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232046AbjGYWQJ (ORCPT ); Tue, 25 Jul 2023 18:16:09 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67DD726B3; Tue, 25 Jul 2023 15:15:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323347; x=1721859347; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=r109XFmiz2M3uwwwWEqxuGrtTBMgiFtxdMzYHmYMvu0=; b=j8qmrK6QcQaeD2XqPHozCIhx3Ver/+lG5CAa/25+O3zrOyfPnNPtHu0A ZUMcV+lY7FwK0bBEW5LSV2dFyzjUktWrxlg4fhbTbl1ogEfDWbVO1d0UT zeI40yNarS5uhL26koPmbeGQ2fmYLaHrQ2u5alHKv7F5YgUtBlAsoZ7y9 fRSLc/vFACRTIyJP0rL4mWxz7BKIkVChfwiPU8EUPtQeA7b56i0A9HCtA J2g1q7qu/+E5OtQm8Ejd555q2UYVMvD7sPpQV/J4KMS0mnf/aNZJeWk1r t/NlUWFQw5GyJISmpR00DzdNv2WWp0EOj56OSF5Up6gXztCGNuoObuNaz w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863168" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863168" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938851" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938851" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:25 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 026/115] KVM: TDX: Do TDX specific vcpu initialization Date: Tue, 25 Jul 2023 15:13:37 -0700 Message-Id: <32406a6ab6208eb33ed24f67696a71e1e80938ae.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TD guest vcpu needs TDX specific initialization before running. Repurpose KVM_MEMORY_ENCRYPT_OP to vcpu-scope, add a new sub-command KVM_TDX_INIT_VCPU, and implement the callback for it. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/main.c | 9 ++ arch/x86/kvm/vmx/tdx.c | 180 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 7 + arch/x86/kvm/vmx/x86_ops.h | 4 + arch/x86/kvm/x86.c | 6 + tools/arch/x86/include/uapi/asm/kvm.h | 1 + 9 files changed, 208 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index d711829fb26a..bcf04a75b506 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -125,6 +125,7 @@ KVM_X86_OP(leave_smm) KVM_X86_OP(enable_smi_window) #endif KVM_X86_OP(mem_enc_ioctl) +KVM_X86_OP_OPTIONAL(vcpu_mem_enc_ioctl) KVM_X86_OP_OPTIONAL(mem_enc_register_region) KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 5deb39828820..b265e4507a1e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1716,6 +1716,7 @@ struct kvm_x86_ops { #endif =20 int (*mem_enc_ioctl)(struct kvm *kvm, void __user *argp); + int (*vcpu_mem_enc_ioctl)(struct kvm_vcpu *vcpu, void __user *argp); int (*mem_enc_register_region)(struct kvm *kvm, struct kvm_enc_region *ar= gp); int (*mem_enc_unregister_region)(struct kvm *kvm, struct kvm_enc_region *= argp); int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd); diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index 7112546bd1d0..311a7894b712 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -571,6 +571,7 @@ struct kvm_pmu_event_filter { enum kvm_tdx_cmd_id { KVM_TDX_CAPABILITIES =3D 0, KVM_TDX_INIT_VM, + KVM_TDX_INIT_VCPU, =20 KVM_TDX_CMD_NR_MAX, }; diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 6ed9116f1b5c..8bb38db4323d 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -142,6 +142,14 @@ static int vt_mem_enc_ioctl(struct kvm *kvm, void __us= er *argp) return tdx_vm_ioctl(kvm, argp); } =20 +static int vt_vcpu_mem_enc_ioctl(struct kvm_vcpu *vcpu, void __user *argp) +{ + if (!is_td_vcpu(vcpu)) + return -EINVAL; + + return tdx_vcpu_ioctl(vcpu, argp); +} + #define VMX_REQUIRED_APICV_INHIBITS \ (BIT(APICV_INHIBIT_REASON_DISABLE)| \ BIT(APICV_INHIBIT_REASON_ABSENT) | \ @@ -298,6 +306,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .vcpu_deliver_sipi_vector =3D kvm_vcpu_deliver_sipi_vector, =20 .mem_enc_ioctl =3D vt_mem_enc_ioctl, + .vcpu_mem_enc_ioctl =3D vt_vcpu_mem_enc_ioctl, }; =20 struct kvm_x86_init_ops vt_init_ops __initdata =3D { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 5f5e451b201a..488fefad1833 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -49,6 +49,7 @@ int tdx_vm_enable_cap(struct kvm *kvm, struct kvm_enable_= cap *cap) =20 struct tdx_info { u8 nr_tdcs_pages; + u8 nr_tdvpx_pages; }; =20 /* Info about the TDX module. */ @@ -71,6 +72,11 @@ static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u= 16 hkid) return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits); } =20 +static inline bool is_td_vcpu_created(struct vcpu_tdx *tdx) +{ + return tdx->tdvpr_pa; +} + static inline bool is_td_created(struct kvm_tdx *kvm_tdx) { return kvm_tdx->tdr_pa; @@ -87,6 +93,11 @@ static inline bool is_hkid_assigned(struct kvm_tdx *kvm_= tdx) return kvm_tdx->hkid > 0; } =20 +static inline bool is_td_finalized(struct kvm_tdx *kvm_tdx) +{ + return kvm_tdx->finalized; +} + static void tdx_clear_page(unsigned long page_pa) { const void *zero_page =3D (const void *) __va(page_to_phys(ZERO_PAGE(0))); @@ -347,7 +358,32 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) =20 void tdx_vcpu_free(struct kvm_vcpu *vcpu) { - /* This is stub for now. More logic will come. */ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + int i; + + /* + * This methods can be called when vcpu allocation/initialization + * failed. So it's possible that hkid, tdvpx and tdvpr are not assigned + * yet. + */ + if (is_hkid_assigned(to_kvm_tdx(vcpu->kvm))) { + WARN_ON_ONCE(tdx->tdvpx_pa); + WARN_ON_ONCE(tdx->tdvpr_pa); + return; + } + + if (tdx->tdvpx_pa) { + for (i =3D 0; i < tdx_info.nr_tdvpx_pages; i++) { + if (tdx->tdvpx_pa[i]) + tdx_reclaim_td_page(tdx->tdvpx_pa[i]); + } + kfree(tdx->tdvpx_pa); + tdx->tdvpx_pa =3D NULL; + } + if (tdx->tdvpr_pa) { + tdx_reclaim_td_page(tdx->tdvpr_pa); + tdx->tdvpr_pa =3D 0; + } } =20 void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) @@ -356,8 +392,13 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_e= vent) /* Ignore INIT silently because TDX doesn't support INIT event. */ if (init_event) return; + if (KVM_BUG_ON(is_td_vcpu_created(to_tdx(vcpu)), vcpu->kvm)) + return; =20 - /* This is stub for now. More logic will come here. */ + /* + * Don't update mp_state to runnable because more initialization + * is needed by TDX_VCPU_INIT. + */ } =20 static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) @@ -861,6 +902,136 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) return r; } =20 +/* VMM can pass one 64bit auxiliary data to vcpu via RCX for guest BIOS. */ +static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(vcpu->kvm); + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + unsigned long *tdvpx_pa =3D NULL; + unsigned long tdvpr_pa; + unsigned long va; + int ret, i; + u64 err; + + if (is_td_vcpu_created(tdx)) + return -EINVAL; + + /* + * vcpu_free method frees allocated pages. Avoid partial setup so + * that the method can't handle it. + */ + va =3D __get_free_page(GFP_KERNEL_ACCOUNT); + if (!va) + return -ENOMEM; + tdvpr_pa =3D __pa(va); + + tdvpx_pa =3D kcalloc(tdx_info.nr_tdvpx_pages, sizeof(*tdx->tdvpx_pa), + GFP_KERNEL_ACCOUNT); + if (!tdvpx_pa) { + ret =3D -ENOMEM; + goto free_tdvpr; + } + for (i =3D 0; i < tdx_info.nr_tdvpx_pages; i++) { + va =3D __get_free_page(GFP_KERNEL_ACCOUNT); + if (!va) { + ret =3D -ENOMEM; + goto free_tdvpx; + } + tdvpx_pa[i] =3D __pa(va); + } + + err =3D tdh_vp_create(kvm_tdx->tdr_pa, tdvpr_pa); + if (KVM_BUG_ON(err, vcpu->kvm)) { + ret =3D -EIO; + pr_tdx_error(TDH_VP_CREATE, err, NULL); + goto free_tdvpx; + } + tdx->tdvpr_pa =3D tdvpr_pa; + + tdx->tdvpx_pa =3D tdvpx_pa; + for (i =3D 0; i < tdx_info.nr_tdvpx_pages; i++) { + err =3D tdh_vp_addcx(tdx->tdvpr_pa, tdvpx_pa[i]); + if (KVM_BUG_ON(err, vcpu->kvm)) { + pr_tdx_error(TDH_VP_ADDCX, err, NULL); + for (; i < tdx_info.nr_tdvpx_pages; i++) { + free_page((unsigned long)__va(tdvpx_pa[i])); + tdvpx_pa[i] =3D 0; + } + /* vcpu_free method frees TDVPX and TDR donated to TDX */ + return -EIO; + } + } + + err =3D tdh_vp_init(tdx->tdvpr_pa, vcpu_rcx); + if (KVM_BUG_ON(err, vcpu->kvm)) { + pr_tdx_error(TDH_VP_INIT, err, NULL); + return -EIO; + } + + vcpu->arch.mp_state =3D KVM_MP_STATE_RUNNABLE; + return 0; + +free_tdvpx: + for (i =3D 0; i < tdx_info.nr_tdvpx_pages; i++) { + if (tdvpx_pa[i]) + free_page((unsigned long)__va(tdvpx_pa[i])); + tdvpx_pa[i] =3D 0; + } + kfree(tdvpx_pa); + tdx->tdvpx_pa =3D NULL; +free_tdvpr: + if (tdvpr_pa) + free_page((unsigned long)__va(tdvpr_pa)); + tdx->tdvpr_pa =3D 0; + + return ret; +} + +int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) +{ + struct msr_data apic_base_msr; + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(vcpu->kvm); + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + struct kvm_tdx_cmd cmd; + int ret; + + if (tdx->initialized) + return -EINVAL; + + if (!is_hkid_assigned(kvm_tdx) || is_td_finalized(kvm_tdx)) + return -EINVAL; + + if (copy_from_user(&cmd, argp, sizeof(cmd))) + return -EFAULT; + + if (cmd.error) + return -EINVAL; + + /* Currently only KVM_TDX_INTI_VCPU is defined for vcpu operation. */ + if (cmd.flags || cmd.id !=3D KVM_TDX_INIT_VCPU) + return -EINVAL; + + /* + * As TDX requires X2APIC, set local apic mode to X2APIC. User space + * VMM, e.g. qemu, is required to set CPUID[0x1].ecx.X2APIC=3D1 by + * KVM_SET_CPUID2. Otherwise kvm_set_apic_base() will fail. + */ + apic_base_msr =3D (struct msr_data) { + .host_initiated =3D true, + .data =3D APIC_DEFAULT_PHYS_BASE | LAPIC_MODE_X2APIC | + (kvm_vcpu_is_reset_bsp(vcpu) ? MSR_IA32_APICBASE_BSP : 0), + }; + if (kvm_set_apic_base(vcpu, &apic_base_msr)) + return -EINVAL; + + ret =3D tdx_td_vcpu_init(vcpu, (u64)cmd.data); + if (ret) + return ret; + + tdx->initialized =3D true; + return 0; +} + static int __init tdx_module_setup(void) { const struct tdsysinfo_struct *tdsysinfo; @@ -879,6 +1050,11 @@ static int __init tdx_module_setup(void) WARN_ON(tdsysinfo->num_cpuid_config > TDX_MAX_NR_CPUID_CONFIGS); tdx_info =3D (struct tdx_info) { .nr_tdcs_pages =3D tdsysinfo->tdcs_base_size / PAGE_SIZE, + /* + * TDVPS =3D TDVPR(4K page) + TDVPX(multiple 4K pages). + * -1 for TDVPR. + */ + .nr_tdvpx_pages =3D tdsysinfo->tdvps_base_size / PAGE_SIZE - 1, }; =20 return 0; diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index af7fdc1516d5..c39d866e0653 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -17,12 +17,19 @@ struct kvm_tdx { u64 xfam; int hkid; =20 + bool finalized; + u64 tsc_offset; }; =20 struct vcpu_tdx { struct kvm_vcpu vcpu; =20 + unsigned long tdvpr_pa; + unsigned long *tdvpx_pa; + + bool initialized; + /* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work. diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index aaa419363276..8a7e256b44ac 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -151,6 +151,8 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); int tdx_vcpu_create(struct kvm_vcpu *vcpu); void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); + +int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline void tdx_hardware_unsetup(void) {} @@ -170,6 +172,8 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __= user *argp) { return -EOP static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTS= UPP; } static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) = {} + +static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } #endif =20 #endif /* __KVM_X86_VMX_X86_OPS_H */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4afe24f50dcb..2922c4a69a6e 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -6073,6 +6073,12 @@ long kvm_arch_vcpu_ioctl(struct file *filp, case KVM_SET_DEVICE_ATTR: r =3D kvm_vcpu_ioctl_device_attr(vcpu, ioctl, argp); break; + case KVM_MEMORY_ENCRYPT_OP: + r =3D -ENOTTY; + if (!kvm_x86_ops.vcpu_mem_enc_ioctl) + goto out; + r =3D kvm_x86_ops.vcpu_mem_enc_ioctl(vcpu, argp); + break; default: r =3D -EINVAL; } diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include= /uapi/asm/kvm.h index 61ce7d174fcf..83bd9e3118d1 100644 --- a/tools/arch/x86/include/uapi/asm/kvm.h +++ b/tools/arch/x86/include/uapi/asm/kvm.h @@ -566,6 +566,7 @@ struct kvm_pmu_event_filter { enum kvm_tdx_cmd_id { KVM_TDX_CAPABILITIES =3D 0, KVM_TDX_INIT_VM, + KVM_TDX_INIT_VCPU, =20 KVM_TDX_CMD_NR_MAX, }; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EF39C04E69 for ; Tue, 25 Jul 2023 22:17:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231782AbjGYWRf (ORCPT ); Tue, 25 Jul 2023 18:17:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232098AbjGYWQK (ORCPT ); Tue, 25 Jul 2023 18:16:10 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8338026B7; Tue, 25 Jul 2023 15:15:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323347; x=1721859347; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ESlsFtK7TCNNodzGvNFsc1Y94K335DvZHywlXfgF6Fc=; b=BaMLrWFNR6kD64F3nyDb4XvMcItu4N94tmey2FD1pyYAr1kvZTf3GIs2 D3qxOVEPGudBEjimm5l2oTgXl0OTEOUO90CitRRMCL+Pw4AqkQq62cWON +AUpmLlGlwQQw9VRHjXxRVeYzpQLYl0aBCs4m7GRsNHa1VHRatiSp8fBY WpVBU52bKrgGQn8bKv7EDNa2GSMRar9bz9ySqSBKiImyuBm2yA55JzWMb WNurJszMeFG21LgIDEF3Kh9BaaIZ/8LCl+KoFJXEUBBc1sHIKISv2d60k NWKovGiyd/3ZniLh6DUg0KS11J7kI3F+8547eSnGl/Oso3ChM3I70KHfz g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863170" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863170" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938854" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938854" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:26 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 027/115] [MARKER] The start of TDX KVM patch series: KVM MMU GPA shared bits Date: Tue, 25 Jul 2023 15:13:38 -0700 Message-Id: <56177f3e75a37eac675fc717da31ee6714bf4343.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of KVM MMU GPA shared bits. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index 25082e9c0b20..8b8186e7bfeb 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -10,6 +10,7 @@ What qemu can do ---------------- - TDX VM TYPE is exposed to Qemu. - Qemu can create/destroy guest of TDX vm type. +- Qemu can create/destroy vcpu of TDX vm type. =20 Patch Layer status ------------------ @@ -18,12 +19,12 @@ Patch Layer status * TDX, VMX coexistence: Applied * TDX architectural definitions: Applied * TD VM creation/destruction: Applied -* TD vcpu creation/destruction: Applying +* TD vcpu creation/destruction: Applied * TDX EPT violation: Not yet * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet =20 -* KVM MMU GPA shared bits: Not yet +* KVM MMU GPA shared bits: Applying * KVM TDP refactoring for TDX: Not yet * KVM TDP MMU hooks: Not yet --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E9AEC04A94 for ; Tue, 25 Jul 2023 22:18:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231562AbjGYWRp (ORCPT ); Tue, 25 Jul 2023 18:17:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232237AbjGYWQO (ORCPT ); Tue, 25 Jul 2023 18:16:14 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0F0E30E7; Tue, 25 Jul 2023 15:15:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323351; x=1721859351; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CnABhAX+vjImDSiz3yRh4rdSKa5hBkboE8/7lwMKZ9s=; b=BRJJ/i+NNg7C+RvMI8VU9EBv/FAkUwGF7Hj7bVZT1a2MOamm5eEz0UCG yVWspqWYjrzRflTmLfSUazZDnzE0KJ0YVuief1P25AeY1JHTMTy56gxZc b4i+hehbgh38HlBdFuXialOD/Yc4CSvbPHyqphQYr5X6iuAlvP62nduhR olVGfIv/QeAZ+/U7xy8V+WiL8bsVUxY5Ij3/ZPaKaaBOuvZ7pPh7+9osF 4KHTy0VAh1GGcIWUdI5YOy1KrorYH4Yo9XF6T0LeYQ1uGcUugQ2MgSHnp wxJFmbH8e/YTZjmakXnF8mGhQT4+RxvwNfVEq9GWLGGG7QDUqCRLZ8sfJ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863193" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863193" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938869" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938869" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:31 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 028/115] KVM: x86/mmu: introduce config for PRIVATE KVM MMU Date: Tue, 25 Jul 2023 15:13:39 -0700 Message-Id: <13bb1acd32fe2056e61c6dc3dc4863942d8333f1.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata To keep the case of non TDX intact, introduce a new config option for private KVM MMU support. At the moment, this is synonym for CONFIG_INTEL_TDX_HOST && CONFIG_KVM_INTEL. The config makes it clear that the config is only for x86 KVM MMU. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/Kconfig | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 8d09fc955972..c7cb060c4ddc 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -155,4 +155,8 @@ config KVM_XEN config KVM_EXTERNAL_WRITE_TRACKING bool =20 +config KVM_MMU_PRIVATE + def_bool y + depends on INTEL_TDX_HOST && KVM_INTEL + endif # VIRTUALIZATION --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 345BDC05051 for ; Tue, 25 Jul 2023 22:18:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232177AbjGYWRv (ORCPT ); Tue, 25 Jul 2023 18:17:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232236AbjGYWQO (ORCPT ); Tue, 25 Jul 2023 18:16:14 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5072030ED; Tue, 25 Jul 2023 15:15:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323351; x=1721859351; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zCAgGQEEML4CMrA+OwifaNyRtVtPmIPEwrZLmmByLZ8=; b=ksa2D0W9zgaSUqQI/WfFgMBFj8GsYsdKCTFrJFUJPZLCvtZ07AcgPHiT SM0DKskxm2tWFsxcrqGBDUA4ulpjSYOlwayGGW4nzQG50oBOTx6CXgcFi hdoOO+jgPIciY4IAn/HwLxn9LafmExns/GK3pUHerUuAGHVLIc5Xi1/Zq ojjS1q1U1soleFuMK21XV9IYPpGWUm3jI+tdTIRwtb+0tX7kJx35RzG93 TOv8Jqd9ENI58MZOao7Oe8WPLKI1uoSXsOXqVD2qfSRg5FZDNwDlxgXtF w3l31JHJlf8YL3IO3fCKESh9+2BZIzq1TVYPnskJtw1uR795lY3q8Dpmn w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863198" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863198" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938873" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938873" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:31 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Rick Edgecombe Subject: [PATCH v15 029/115] KVM: x86/mmu: Add address conversion functions for TDX shared bit of GPA Date: Tue, 25 Jul 2023 15:13:40 -0700 Message-Id: <8f3242ea8a9f825f56ce31e6015799866d3dfe58.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX repurposes one GPA bit (51 bit or 47 bit based on configuration) to indicate the GPA is private(if cleared) or shared (if set) with VMM. If GPA.shared is set, GPA is covered by the existing conventional EPT pointed by EPTP. If GPA.shared bit is cleared, GPA is covered by TDX module. VMM has to issue SEAMCALLs to operate. Add a member to remember GPA shared bit for each guest TDs, add address conversion functions between private GPA and shared GPA and test if GPA is private. Because struct kvm_arch (or struct kvm which includes struct kvm_arch. See kvm_arch_alloc_vm() that passes __GPF_ZERO) is zero-cleared when allocated, the new member to remember GPA shared bit is guaranteed to be zero with this patch unless it's initialized explicitly. Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/mmu.h | 27 +++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.c | 5 +++++ 3 files changed, 36 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index b265e4507a1e..a39d88d2f6fc 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1447,6 +1447,10 @@ struct kvm_arch { */ #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1) struct kvm_mmu_memory_cache split_desc_cache; + +#ifdef CONFIG_KVM_MMU_PRIVATE + gfn_t gfn_shared_mask; +#endif }; =20 struct kvm_vm_stat { diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 963c734642f6..919fa5109e8c 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -300,4 +300,31 @@ static inline gpa_t kvm_translate_gpa(struct kvm_vcpu = *vcpu, return gpa; return translate_nested_gpa(vcpu, gpa, access, exception); } + +static inline gfn_t kvm_gfn_shared_mask(const struct kvm *kvm) +{ +#ifdef CONFIG_KVM_MMU_PRIVATE + return kvm->arch.gfn_shared_mask; +#else + return 0; +#endif +} + +static inline gfn_t kvm_gfn_to_shared(const struct kvm *kvm, gfn_t gfn) +{ + return gfn | kvm_gfn_shared_mask(kvm); +} + +static inline gfn_t kvm_gfn_to_private(const struct kvm *kvm, gfn_t gfn) +{ + return gfn & ~kvm_gfn_shared_mask(kvm); +} + +static inline bool kvm_is_private_gpa(const struct kvm *kvm, gpa_t gpa) +{ + gfn_t mask =3D kvm_gfn_shared_mask(kvm); + + return mask && !(gpa_to_gfn(gpa) & mask); +} + #endif diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 488fefad1833..a10caf87e4fb 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -863,6 +863,11 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_tdx= _cmd *cmd) kvm_tdx->attributes =3D td_params->attributes; kvm_tdx->xfam =3D td_params->xfam; =20 + if (td_params->exec_controls & TDX_EXEC_CONTROL_MAX_GPAW) + kvm->arch.gfn_shared_mask =3D gpa_to_gfn(BIT_ULL(51)); + else + kvm->arch.gfn_shared_mask =3D gpa_to_gfn(BIT_ULL(47)); + out: /* kfree() accepts NULL. */ kfree(init_vm); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E8B0C001E0 for ; Tue, 25 Jul 2023 22:18:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232191AbjGYWSI (ORCPT ); Tue, 25 Jul 2023 18:18:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231906AbjGYWQa (ORCPT ); Tue, 25 Jul 2023 18:16:30 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63F86359C; Tue, 25 Jul 2023 15:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323354; x=1721859354; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zdJ2GT4ARrSNbJp3F6eaxutpQ4DdrmPtIT6M7sepMRw=; b=Etl0ney8n7SyAZmYoZT4xWEByy3ZhcFakDCeOGb7MCX3oJgQn4QK/kg5 05dCwGnYeXfDTA09jwPoKzy7gNY03c2ZuZDJmzcs3AML6l2RUyh38m/Tv XLSEXfbAy3XYKo5s8Y/lRogqrZTRC+r4nHf2q9kHrq4VnJq9rdDReKLsJ kj7nbPGJN3Bj10EL3vckemdVCPJszR7zAeuA+tOha5B8ihZHkIAv2MESi ACTzA90/SYpzvxxf6VpiE0CAi75iXzjqZAhBSgXZe16XV9bstx8RCVD81 aJzLbKefJnqUXJcyG5RvD68f5HxKB4uNhCMbSoAVhLyuOgtxW2TNjS9L6 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863202" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863202" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938878" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938878" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:32 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 030/115] [MARKER] The start of TDX KVM patch series: KVM TDP refactoring for TDX Date: Tue, 25 Jul 2023 15:13:41 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of KVM TDP refactoring for TDX. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index 8b8186e7bfeb..e893a3d714c7 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -25,6 +25,6 @@ Patch Layer status * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet =20 -* KVM MMU GPA shared bits: Applying -* KVM TDP refactoring for TDX: Not yet +* KVM MMU GPA shared bits: Applied +* KVM TDP refactoring for TDX: Applying * KVM TDP MMU hooks: Not yet --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EC6CC04A6A for ; Tue, 25 Jul 2023 22:19:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232279AbjGYWTN (ORCPT ); Tue, 25 Jul 2023 18:19:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229998AbjGYWR2 (ORCPT ); Tue, 25 Jul 2023 18:17:28 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D16641FEC; Tue, 25 Jul 2023 15:16:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323369; x=1721859369; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OIrsaq2bp9o27ZsGmX41LNMKR7WNeny0bQMTCvxkndI=; b=gXBZmvjenWwuH0+/LXc2VRWS9o1uUSHoQ7hZ0euR20wzddZXyZcqI8vx 2UfFRyzwOvHXAnlCdlh1TeW25Id2w7Js/kp/YrgZrcs7GsZmX4fJx538f Zd267X9e/r49rvqwdBuxwk1z+mNWR4p0I0k4PKX/5xdZkwfwShISqS2eX htveO2CCn/pEuWIIeZvYxpZAJ1xbgqKheWePiusrQOlUGAg6AB9Rn3bdg CXYDI4NXiZT9wO+C9t49qaL6z3hfgyRbBcstCeigK5kcPFke7kJDSmqAh +OjUd2ZWNTVF6QIN02P32G2xyLdE6JHPF7L3f0ftBab1qDL9VO2mRfrc+ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863207" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863207" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938882" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938882" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:32 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 031/115] KVM: Allow page-sized MMU caches to be initialized with custom 64-bit values Date: Tue, 25 Jul 2023 15:13:42 -0700 Message-Id: <98d098febd95c11ea5e27cb651a634926eb38f31.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Add support to MMU caches for initializing a page with a custom 64-bit value, e.g. to pre-fill an entire page table with non-zero PTE values. The functionality will be used by x86 to support Intel's TDX, which needs to set bit 63 in all non-present PTEs in order to prevent !PRESENT page faults from getting reflected into the guest (Intel's EPT Violation #VE architecture made the less than brilliant decision of having the per-PTE behavior be opt-out instead of opt-in). Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index 6f4737d5046a..4932bc90a0a0 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -93,6 +93,7 @@ struct gfn_to_pfn_cache { struct kvm_mmu_memory_cache { gfp_t gfp_zero; gfp_t gfp_custom; + u64 init_value; struct kmem_cache *kmem_cache; int capacity; int nobjs; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 60ed0f613bce..14b1fa9fe644 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -378,12 +378,17 @@ static void kvm_flush_shadow_all(struct kvm *kvm) static inline void *mmu_memory_cache_alloc_obj(struct kvm_mmu_memory_cache= *mc, gfp_t gfp_flags) { + void *page; + gfp_flags |=3D mc->gfp_zero; =20 if (mc->kmem_cache) return kmem_cache_alloc(mc->kmem_cache, gfp_flags); - else - return (void *)__get_free_page(gfp_flags); + + page =3D (void *)__get_free_page(gfp_flags); + if (page && mc->init_value) + memset64(page, mc->init_value, PAGE_SIZE / sizeof(mc->init_value)); + return page; } =20 int __kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int capa= city, int min) @@ -398,6 +403,13 @@ int __kvm_mmu_topup_memory_cache(struct kvm_mmu_memory= _cache *mc, int capacity, if (WARN_ON_ONCE(!capacity)) return -EIO; =20 + /* + * Custom init values can be used only for page allocations, + * and obviously conflict with __GFP_ZERO. + */ + if (WARN_ON_ONCE(mc->init_value && (mc->kmem_cache || mc->gfp_zero))) + return -EIO; + mc->objects =3D kvmalloc_array(sizeof(void *), capacity, gfp); if (!mc->objects) return -ENOMEM; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5B75C04A6A for ; Tue, 25 Jul 2023 22:19:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232140AbjGYWSq (ORCPT ); Tue, 25 Jul 2023 18:18:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230172AbjGYWRB (ORCPT ); Tue, 25 Jul 2023 18:17:01 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31A4735AE; Tue, 25 Jul 2023 15:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323355; x=1721859355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hj/RhaVxVsjToTdYzzQ8a7zYXVh5dwSrMRchhmWHVrU=; b=FC1r0XgkUEpEYCLS6W5+Mq8mTAr4J5KfyV0dPweclPYaEbqbFEGJHAYk Ote/s56dmollY/XTmElZfNurSSCCguqqmSlkHvOgaDTXK97mKeDkXzS8j NTVr5G6ZYGf2yLOTW45XURhVmX2iaGCjxUME2+CVfpES5mDHtc1llR9e5 eHCGTffEzUipxNik3KR2Jr0ONwMVrKJDavFv8ce8PPVKCRobsAePl+Hg1 Bxiz+AlWPCDD6WtUyOZitG82/kGcUzNILyhaVGP5TiHsTLe7wdWmTs2Zm YqciRevjDNeAyD3AIZa40jDW8ZEg2vJajM8s932tAScnzRcyrv5f+829f w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863212" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863212" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938889" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938889" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:32 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 032/115] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE Date: Tue, 25 Jul 2023 15:13:43 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata The TDX support will need the "suppress #VE" bit (bit 63) set as the initial value for SPTE. To reduce code change size, introduce a new macro SHADOW_NONPRESENT_VALUE for the initial value for the shadow page table entry (SPTE) and replace hard-coded value 0 for it. Initialize shadow page tables with their value. The plan is to unconditionally set the "suppress #VE" bit for both AMD and Intel as: 1) AMD hardware uses the bit 63 as NX for present SPTE and ignored for non-present SPTE; 2) for conventional VMX guests, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/mmu.c | 20 +++++++++++++++----- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/mmu/spte.h | 2 ++ arch/x86/kvm/mmu/tdp_mmu.c | 14 +++++++------- 4 files changed, 25 insertions(+), 13 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f76cf14acb7f..8183b52d7a19 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -579,9 +579,9 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u= 64 *sptep) =20 if (!is_shadow_present_pte(old_spte) || !spte_has_volatile_bits(old_spte)) - __update_clear_spte_fast(sptep, 0ull); + __update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE); else - old_spte =3D __update_clear_spte_slow(sptep, 0ull); + old_spte =3D __update_clear_spte_slow(sptep, SHADOW_NONPRESENT_VALUE); =20 if (!is_shadow_present_pte(old_spte)) return old_spte; @@ -615,7 +615,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u= 64 *sptep) */ static void mmu_spte_clear_no_track(u64 *sptep) { - __update_clear_spte_fast(sptep, 0ull); + __update_clear_spte_fast(sptep, SHADOW_NONPRESENT_VALUE); } =20 static u64 mmu_spte_get_lockless(u64 *sptep) @@ -1976,7 +1976,8 @@ static bool kvm_sync_page_check(struct kvm_vcpu *vcpu= , struct kvm_mmu_page *sp) =20 static int kvm_sync_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, i= nt i) { - if (!sp->spt[i]) + /* sp->spt[i] has initial value of shadow page table allocation */ + if (sp->spt[i] =3D=3D SHADOW_NONPRESENT_VALUE) return 0; =20 return vcpu->arch.mmu->sync_spte(vcpu, sp, i); @@ -6173,7 +6174,16 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu) vcpu->arch.mmu_page_header_cache.kmem_cache =3D mmu_page_header_cache; vcpu->arch.mmu_page_header_cache.gfp_zero =3D __GFP_ZERO; =20 - vcpu->arch.mmu_shadow_page_cache.gfp_zero =3D __GFP_ZERO; + /* + * When X86_64, initial SEPT entries are initialized with + * SHADOW_NONPRESENT_VALUE. Otherwise zeroed. See + * mmu_memory_cache_alloc_obj(). + */ + if (IS_ENABLED(CONFIG_X86_64)) + vcpu->arch.mmu_shadow_page_cache.init_value =3D + SHADOW_NONPRESENT_VALUE; + if (!vcpu->arch.mmu_shadow_page_cache.init_value) + vcpu->arch.mmu_shadow_page_cache.gfp_zero =3D __GFP_ZERO; =20 vcpu->arch.mmu =3D &vcpu->arch.root_mmu; vcpu->arch.walk_mmu =3D &vcpu->arch.root_mmu; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 42d48b1ec7b3..e616a7a781a4 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -892,7 +892,7 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, stru= ct kvm_mmu_page *sp, int gpa_t pte_gpa; gfn_t gfn; =20 - if (WARN_ON_ONCE(!sp->spt[i])) + if (WARN_ON_ONCE(sp->spt[i] =3D=3D SHADOW_NONPRESENT_VALUE)) return 0; =20 first_pte_gpa =3D FNAME(get_level1_sp_gpa)(sp); diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 1279db2eab44..a99eb7d4ae5d 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -148,6 +148,8 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS =3D=3D 8 && MMIO_S= PTE_GEN_HIGH_BITS =3D=3D 11); =20 #define MMIO_SPTE_GEN_MASK GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE= _GEN_HIGH_BITS - 1, 0) =20 +#define SHADOW_NONPRESENT_VALUE 0ULL + extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 70052f59cfdf..465bb01c16a1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -630,7 +630,7 @@ static inline int tdp_mmu_zap_spte_atomic(struct kvm *k= vm, * here since the SPTE is going from non-present to non-present. Use * the raw write helper to avoid an unnecessary check on volatile bits. */ - __kvm_tdp_mmu_write_spte(iter->sptep, 0); + __kvm_tdp_mmu_write_spte(iter->sptep, SHADOW_NONPRESENT_VALUE); =20 return 0; } @@ -767,8 +767,8 @@ static void __tdp_mmu_zap_root(struct kvm *kvm, struct = kvm_mmu_page *root, continue; =20 if (!shared) - tdp_mmu_iter_set_spte(kvm, &iter, 0); - else if (tdp_mmu_set_spte_atomic(kvm, &iter, 0)) + tdp_mmu_iter_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE); + else if (tdp_mmu_set_spte_atomic(kvm, &iter, SHADOW_NONPRESENT_VALUE)) goto retry; } } @@ -824,8 +824,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu= _page *sp) if (WARN_ON_ONCE(!is_shadow_present_pte(old_spte))) return false; =20 - tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, 0, - sp->gfn, sp->role.level + 1); + tdp_mmu_set_spte(kvm, kvm_mmu_page_as_id(sp), sp->ptep, old_spte, + SHADOW_NONPRESENT_VALUE, sp->gfn, sp->role.level + 1); =20 return true; } @@ -859,7 +859,7 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct k= vm_mmu_page *root, !is_last_spte(iter.old_spte, iter.level)) continue; =20 - tdp_mmu_iter_set_spte(kvm, &iter, 0); + tdp_mmu_iter_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE); flush =3D true; } =20 @@ -1253,7 +1253,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_= iter *iter, * invariant that the PFN of a present * leaf SPTE can never change. * See handle_changed_spte(). */ - tdp_mmu_iter_set_spte(kvm, iter, 0); + tdp_mmu_iter_set_spte(kvm, iter, SHADOW_NONPRESENT_VALUE); =20 if (!pte_write(range->arg.pte)) { new_spte =3D kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte, --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A622DC41513 for ; Tue, 25 Jul 2023 22:19:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231629AbjGYWSe (ORCPT ); Tue, 25 Jul 2023 18:18:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232050AbjGYWQn (ORCPT ); Tue, 25 Jul 2023 18:16:43 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BE1935BD; Tue, 25 Jul 2023 15:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323355; x=1721859355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CoQnqLun9XBJdTIBa8S7YePe8TwiMfxkj+IV5w5Hkm4=; b=VvjjkI4yYQQWEGtLjOcINVEiRtKRQGIxMYFIaR/3vvfe+LG4Emd0Rj0Q xSqa/DLHyemMFEnZ8jY33BViMXjlAwOowHa1a50pM8MFEw62Yr68SbeZq T5iSozfutT5tFNac/0JBMM7EJHki7QDWiyzCALO7NdawlBw6Z6kfbfVyJ amN+x4kbZtRUIGwJCoKDwzB5/Ktad2UROonb4OyNvigoVoh42bridXFPe 2/M9NFPmNFMPghXu+p4NmuvkBWRx94sjiVZ3h3TmoVfs+U8Cr/omw4rgM ufSlGHfhiRO3/2jMtXZpLIdubZb8BgISx1o5MtC4yrwBAVNY2PUpfZiYU w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863217" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863217" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938892" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938892" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:33 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 033/115] KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE Date: Tue, 25 Jul 2023 15:13:44 -0700 Message-Id: <6369eb0081da9d9b3b6b71a0a1b5ece669586785.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson For TD guest, the current way to emulate MMIO doesn't work any more, as KVM is not able to access the private memory of TD guest and do the emulation. Instead, TD guest expects to receive #VE when it accesses the MMIO and then it can explicitly make hypercall to KVM to get the expected information. To achieve this, the TDX module always enables "EPT-violation #VE" in the VMCS control. And accordingly, for the MMIO spte for the shared GPA, 1. KVM needs to set "suppress #VE" bit for the non-present SPTE so that EPT violation happens on TD accessing MMIO range. 2. On EPT violation, KVM sets the MMIO spte to clear "suppress #VE" bit so the TD guest can receive the #VE instead of EPT misconfigration unlike VMX case. For the shared GPA that is not populated yet, EPT violation need to be triggered when TD guest accesses such shared GPA. The non-present SPTE value for shared GPA should set "suppress #VE" bit. Add "suppress #VE" bit (bit 63) to SHADOW_NONPRESENT_VALUE and REMOVED_SPTE. Unconditionally set the "suppress #VE" bit (which is bit 63) for both AMD and Intel as: 1) AMD hardware doesn't use this bit when present bit is off; 2) for normal VMX guest, KVM never enables the "EPT-violation #VE" in VMCS control and "suppress #VE" bit is ignored by hardware. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/spte.h | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index a99eb7d4ae5d..a57667810344 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -148,7 +148,20 @@ static_assert(MMIO_SPTE_GEN_LOW_BITS =3D=3D 8 && MMIO_= SPTE_GEN_HIGH_BITS =3D=3D 11); =20 #define MMIO_SPTE_GEN_MASK GENMASK_ULL(MMIO_SPTE_GEN_LOW_BITS + MMIO_SPTE= _GEN_HIGH_BITS - 1, 0) =20 +/* + * Non-present SPTE value for both VMX and SVM for TDP MMU. + * For SVM NPT, for non-present spte (bit 0 =3D 0), other bits are ignored. + * For VMX EPT, bit 63 is ignored if #VE is disabled. (EPT_VIOLATION_VE=3D= 0) + * bit 63 is #VE suppress if #VE is enabled. (EPT_VIOLATION_V= E=3D1) + * For TDX: + * TDX module sets EPT_VIOLATION_VE for Secure-EPT and conventional EPT + */ +#ifdef CONFIG_X86_64 +#define SHADOW_NONPRESENT_VALUE BIT_ULL(63) +static_assert(!(SHADOW_NONPRESENT_VALUE & SPTE_MMU_PRESENT_MASK)); +#else #define SHADOW_NONPRESENT_VALUE 0ULL +#endif =20 extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; @@ -195,7 +208,7 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_mask; * * Only used by the TDP MMU. */ -#define REMOVED_SPTE 0x5a0ULL +#define REMOVED_SPTE (SHADOW_NONPRESENT_VALUE | 0x5a0ULL) =20 /* Removed SPTEs must not be misconstrued as shadow present PTEs. */ static_assert(!(REMOVED_SPTE & SPTE_MMU_PRESENT_MASK)); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2053BC04A94 for ; Tue, 25 Jul 2023 22:19:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232240AbjGYWSz (ORCPT ); Tue, 25 Jul 2023 18:18:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232260AbjGYWQs (ORCPT ); Tue, 25 Jul 2023 18:16:48 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 524283A89; Tue, 25 Jul 2023 15:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323356; x=1721859356; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rMN2DEXHYJg4BrG9HywFW+dJdkWtkQ+SIXgNGj4Zeac=; b=WMUMRw7NWl7j58VdYCFtdDJcAE8WNVU87dmqsu4mv3z/I50vIZYhAbXi ldBHBW2J8YSALFajKsjnR5xnfB44S0DzTW+dWr1V+wo6Xyjw5l/48lK80 lna6HYAYnE8FHo91ZPEK5bFDsHXk9ApiEDawNOR+lGZs1goH3vMDyBgva aDVJNgrBoRAXB49TtIUaV6U3V1MsFdnWGoemVhgQQNQcBmmeJOJlZ5FxO teThQYzeuloaaV/oY2heNJvHPaoUaueIjKLL6eoEAXBBJgLOzIZ96Bw6A +SS8gAIaKn4sLQXSigRUuPQM2oEWBEaLssSQgd2d5va6ssfXO+meU9SPe A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863223" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863223" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938895" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938895" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:33 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 034/115] KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_mask/shadow_present_mask Date: Tue, 25 Jul 2023 15:13:45 -0700 Message-Id: <0c46520f8f0ec9e3867cc79d61e49550e0043f12.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata To make use of the same value of shadow_mmio_mask and shadow_present_mask for TDX and VMX, add Suppress-VE bit to shadow_mmio_mask and shadow_present_mask so that they can be common for both VMX and TDX. TDX will require shadow_mmio_mask and shadow_present_mask to include VMX_SUPPRESS_VE for shared GPA so that EPT violation is triggered for shared GPA. For VMX, VMX_SUPPRESS_VE doesn't matter for MMIO because the spte value is required to cause EPT misconfig. the additional bit doesn't affect VMX logic to add the bit to shadow_mmio_{value, mask}. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/mmu/spte.c | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 0d02c4aafa6f..3066ca5ca246 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -513,6 +513,7 @@ enum vmcs_field { #define VMX_EPT_IPAT_BIT (1ull << 6) #define VMX_EPT_ACCESS_BIT (1ull << 8) #define VMX_EPT_DIRTY_BIT (1ull << 9) +#define VMX_EPT_SUPPRESS_VE_BIT (1ull << 63) #define VMX_EPT_RWX_MASK (VMX_EPT_READABLE_MASK | = \ VMX_EPT_WRITABLE_MASK | \ VMX_EPT_EXECUTABLE_MASK) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index cf2c6426a6fc..778fbaec1887 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -431,7 +431,9 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_e= xec_only) shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; shadow_nx_mask =3D 0ull; shadow_x_mask =3D VMX_EPT_EXECUTABLE_MASK; - shadow_present_mask =3D has_exec_only ? 0ull : VMX_EPT_READABLE_MASK; + /* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */ + shadow_present_mask =3D + (has_exec_only ? 0ull : VMX_EPT_READABLE_MASK) | VMX_EPT_SUPPRESS_VE_BIT; /* * EPT overrides the host MTRRs, and so KVM must program the desired * memtype directly into the SPTEs. Note, this mask is just the mask @@ -448,7 +450,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_e= xec_only) * of an EPT paging-structure entry is 110b (write/execute). */ kvm_mmu_set_mmio_spte_mask(VMX_EPT_MISCONFIG_WX_VALUE, - VMX_EPT_RWX_MASK, 0); + VMX_EPT_RWX_MASK | VMX_EPT_SUPPRESS_VE_BIT, 0); } EXPORT_SYMBOL_GPL(kvm_mmu_set_ept_masks); =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8444BC41513 for ; Tue, 25 Jul 2023 22:20:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231791AbjGYWTj (ORCPT ); Tue, 25 Jul 2023 18:19:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230148AbjGYWSR (ORCPT ); Tue, 25 Jul 2023 18:18:17 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D14B6E63; Tue, 25 Jul 2023 15:16:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323369; x=1721859369; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gKqW/UX5njbBa1LnKjGGLK3AnsIb3N3p02RzFIMgpK8=; b=JqTaJQxPF3akN8TpmsdhcrEsmp1IEQTg83LU2nbnXE4dqESbONuTbHu9 tTnKgxN8gVitaI3weAqrB1DpXxHxD1jpmp9qTCNS1wvEcQzL+ffHc0qUp cleGI50gY3meQBtXWtN/qDhneN0OIbIdjU8rvf9gp4FhUV/dTEKelS6gS Q9SE+Su2+QA7zmygffRpzlxIGLTXb+mnIdfe8YFBP6wKzBuJhZU43piaQ ns7+e2L8+2K2C9CIu+cXY6AFg7OgF/ZGcZWMnDwZPQ2F/6GRPcV1kwQ88 ElW+q87wkcyBKNW/O+vDeRQsUx3jESBtXHgT1yJiLe3PM4SR17RYUISeH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863229" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863229" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938899" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938899" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:34 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 035/115] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis Date: Tue, 25 Jul 2023 15:13:46 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX will use a different shadow PTE entry value for MMIO from VMX. Add members to kvm_arch and track value for MMIO per-VM instead of global variables. By using the per-VM EPT entry value for MMIO, the existing VMX logic is kept working. Introduce a separate setter function so that guest TD can override later. Also require mmio spte cachcing for TDX. Actually this is true case because TDX require EPT and KVM EPT allows mmio spte caching. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 7 ++++--- arch/x86/kvm/mmu/spte.c | 10 ++++++++-- arch/x86/kvm/mmu/spte.h | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 6 +++--- 6 files changed, 20 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index a39d88d2f6fc..07b47398f68e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1260,6 +1260,8 @@ struct kvm_arch { */ spinlock_t mmu_unsync_pages_lock; =20 + u64 shadow_mmio_value; + struct list_head assigned_dev_head; struct iommu_domain *iommu_domain; bool iommu_noncoherent; diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 919fa5109e8c..801e3d6b572d 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -101,6 +101,7 @@ static inline u8 kvm_get_shadow_phys_bits(void) } =20 void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_= mask); +void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value); void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask); void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only); =20 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 8183b52d7a19..f0f8166a2b1d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2541,7 +2541,7 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct k= vm_mmu_page *sp, return kvm_mmu_prepare_zap_page(kvm, child, invalid_list); } - } else if (is_mmio_spte(pte)) { + } else if (is_mmio_spte(kvm, pte)) { mmu_spte_clear_no_track(spte); } return 0; @@ -4223,7 +4223,7 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vc= pu, u64 addr, bool direct) if (WARN_ON(reserved)) return -EINVAL; =20 - if (is_mmio_spte(spte)) { + if (is_mmio_spte(vcpu->kvm, spte)) { gfn_t gfn =3D get_mmio_spte_gfn(spte); unsigned int access =3D get_mmio_spte_access(spte); =20 @@ -4788,7 +4788,7 @@ EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) { - if (unlikely(is_mmio_spte(*sptep))) { + if (unlikely(is_mmio_spte(vcpu->kvm, *sptep))) { if (gfn !=3D get_mmio_spte_gfn(*sptep)) { mmu_spte_clear_no_track(sptep); return true; @@ -6336,6 +6336,7 @@ int kvm_mmu_init_vm(struct kvm *kvm) struct kvm_page_track_notifier_node *node =3D &kvm->arch.mmu_sp_tracker; int r; =20 + kvm->arch.shadow_mmio_value =3D shadow_mmio_value; INIT_LIST_HEAD(&kvm->arch.active_mmu_pages); INIT_LIST_HEAD(&kvm->arch.zapped_obsolete_pages); INIT_LIST_HEAD(&kvm->arch.possible_nx_huge_pages); diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 778fbaec1887..a1f332eb3b59 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -74,10 +74,10 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsi= gned int access) u64 spte =3D generation_mmio_spte_mask(gen); u64 gpa =3D gfn << PAGE_SHIFT; =20 - WARN_ON_ONCE(!shadow_mmio_value); + WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value); =20 access &=3D shadow_mmio_access_mask; - spte |=3D shadow_mmio_value | access; + spte |=3D vcpu->kvm->arch.shadow_mmio_value | access; spte |=3D gpa | shadow_nonpresent_or_rsvd_mask; spte |=3D (gpa & shadow_nonpresent_or_rsvd_mask) << SHADOW_NONPRESENT_OR_RSVD_MASK_LEN; @@ -413,6 +413,12 @@ void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mm= io_mask, u64 access_mask) } EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_mask); =20 +void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value) +{ + kvm->arch.shadow_mmio_value =3D mmio_value; +} +EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_value); + void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask) { /* shadow_me_value must be a subset of shadow_me_mask */ diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index a57667810344..a8418fd8ae9e 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -251,9 +251,9 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *spt= ep) return to_shadow_page(__pa(sptep)); } =20 -static inline bool is_mmio_spte(u64 spte) +static inline bool is_mmio_spte(struct kvm *kvm, u64 spte) { - return (spte & shadow_mmio_mask) =3D=3D shadow_mmio_value && + return (spte & shadow_mmio_mask) =3D=3D kvm->arch.shadow_mmio_value && likely(enable_mmio_caching); } =20 diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 465bb01c16a1..4fe31a1efa9a 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -522,8 +522,8 @@ static void handle_changed_spte(struct kvm *kvm, int as= _id, gfn_t gfn, * impact the guest since both the former and current SPTEs * are nonpresent. */ - if (WARN_ON(!is_mmio_spte(old_spte) && - !is_mmio_spte(new_spte) && + if (WARN_ON(!is_mmio_spte(kvm, old_spte) && + !is_mmio_spte(kvm, new_spte) && !is_removed_spte(new_spte))) pr_err("Unexpected SPTE change! Nonpresent SPTEs\n" "should not be replaced with another,\n" @@ -1010,7 +1010,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm= _vcpu *vcpu, } =20 /* If a MMIO SPTE is installed, the MMIO will need to be emulated. */ - if (unlikely(is_mmio_spte(new_spte))) { + if (unlikely(is_mmio_spte(vcpu->kvm, new_spte))) { vcpu->stat.pf_mmio_spte_created++; trace_mark_mmio_spte(rcu_dereference(iter->sptep), iter->gfn, new_spte); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAA3DC04A6A for ; Tue, 25 Jul 2023 22:20:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232208AbjGYWTo (ORCPT ); Tue, 25 Jul 2023 18:19:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232135AbjGYWS0 (ORCPT ); Tue, 25 Jul 2023 18:18:26 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A78B12681; Tue, 25 Jul 2023 15:16:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323372; x=1721859372; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zcDt6w72s30vbUGADhv9UguzmHhSq0mDupruji6aYgE=; b=XJ6Xbgx0/nS4o7NKUY8Z5aLkBu/0YxF9otX3pIsDPYoVse23Ec7KiEWg 5ZXQ+xvqJ23B0NOWpww/YbVJtT55fKXIKgZ+Sarc9Q7CuXY/FwAGkKwVz VV45F8OOReCDKEpeNYkJeuI+fxByxx8oKQ+E+D9J8SRlhqx9EPaa78GVc YZUr8rv1p/XhrKxzS/B6SDJlfshzW3ntMQAubNEfEXcCXMBXBegL7QOXq 4Mo+juT1LX7E+yjwZGLZ52Ag1we7JZlf4t+9sfk7/5w7UdszyZCFqARc4 QUdDjZs/q3kBu0o2BD/+Hri35h0Bw3F/wVJpl0ruF0NL1ndhEcktk4Ql9 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863232" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863232" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938904" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938904" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:34 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 036/115] KVM: x86/mmu: Disallow fast page fault on private GPA Date: Tue, 25 Jul 2023 15:13:47 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX requires TDX SEAMCALL to operate Secure EPT instead of direct memory access and TDX SEAMCALL is heavy operation. Fast page fault on private GPA doesn't make sense. Disallow fast page fault on private GPA. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f0f8166a2b1d..163ff3308091 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3375,8 +3375,16 @@ static int kvm_handle_noslot_fault(struct kvm_vcpu *= vcpu, return RET_PF_CONTINUE; } =20 -static bool page_fault_can_be_fast(struct kvm_page_fault *fault) +static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault = *fault) { + /* + * TDX private mapping doesn't support fast page fault because the EPT + * entry is read/written with TDX SEAMCALLs instead of direct memory + * access. + */ + if (kvm_is_private_gpa(kvm, fault->addr)) + return false; + /* * Page faults with reserved bits set, i.e. faults on MMIO SPTEs, only * reach the common page fault handler if the SPTE has an invalid MMIO @@ -3486,7 +3494,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, str= uct kvm_page_fault *fault) u64 *sptep =3D NULL; uint retry_count =3D 0; =20 - if (!page_fault_can_be_fast(fault)) + if (!page_fault_can_be_fast(vcpu->kvm, fault)) return ret; =20 walk_shadow_page_lockless_begin(vcpu); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF3ADC04FE0 for ; Tue, 25 Jul 2023 22:20:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231270AbjGYWTr (ORCPT ); Tue, 25 Jul 2023 18:19:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232225AbjGYWS2 (ORCPT ); Tue, 25 Jul 2023 18:18:28 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B9FE2695; Tue, 25 Jul 2023 15:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323374; x=1721859374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LcSauWHj2D9Iv4ZMWLKYaE9YEagzFr8J7NfyqVZHCFg=; b=c7reDnsZVWUEJp6RG4fjIeJF4fIVa/2EL2INn1hW3C5/JZPEh0MsipqS AB9qo57lDkCffGSP1CbYdvuMujXuM38vf9C7c++GIFtffyjheHsWqtwc2 l/DGtFqNqWUl6iq5+5XrRxno4t2sIMKK3lGy8JQAc7LNClPBgp+qCHxzY eCMSR1YB5e58Dau4ANQnTTHV1Q+34nJ8TWRdxdwjRAU1GXSdKfcLePmB1 sERclfBc3yIQ5CDDyl8d2A5ZTMU15KJVa2Qj+jFtg2+jCOvd/gOtP3ooU nNssuQeZ2vR0x/CcE1aXpK4qmL/dOgiOsWFNzGd+yIkMg1uUHSQe/i0wx Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863236" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863236" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938907" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938907" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:34 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 037/115] KVM: x86/mmu: Allow per-VM override of the TDP max page level Date: Tue, 25 Jul 2023 15:13:48 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson TDX requires special handling to support large private page. For simplicity, only support 4K page for TD guest for now. Add per-VM maximum page level support to support different maximum page sizes for TD guest and conventional VMX guest. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Acked-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/mmu/mmu_internal.h | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 07b47398f68e..0bc53c942c6c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1234,6 +1234,7 @@ struct kvm_arch { unsigned long n_requested_mmu_pages; unsigned long n_max_mmu_pages; unsigned int indirect_shadow_pages; + int tdp_max_page_level; u8 mmu_valid_gen; struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES]; struct list_head active_mmu_pages; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 163ff3308091..9bf8d05937c5 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6368,6 +6368,7 @@ int kvm_mmu_init_vm(struct kvm *kvm) kvm->arch.split_desc_cache.kmem_cache =3D pte_list_desc_cache; kvm->arch.split_desc_cache.gfp_zero =3D __GFP_ZERO; =20 + kvm->arch.tdp_max_page_level =3D KVM_MAX_HUGEPAGE_LEVEL; return 0; } =20 diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index 3a423403af01..76fa38da74f1 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -299,7 +299,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu= *vcpu, gpa_t cr2_or_gpa, .nx_huge_page_workaround_enabled =3D is_nx_huge_page_enabled(vcpu->kvm), =20 - .max_level =3D KVM_MAX_HUGEPAGE_LEVEL, + .max_level =3D vcpu->kvm->arch.tdp_max_page_level, .req_level =3D PG_LEVEL_4K, .goal_level =3D PG_LEVEL_4K, }; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 327C9C04FE1 for ; Tue, 25 Jul 2023 22:20:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232335AbjGYWT4 (ORCPT ); Tue, 25 Jul 2023 18:19:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232227AbjGYWS2 (ORCPT ); Tue, 25 Jul 2023 18:18:28 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCE6C269E; Tue, 25 Jul 2023 15:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323374; x=1721859374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/Au/4v/tNqN+iA96Ys06DdIPRld6G0dSUDZpPakmGvU=; b=MUWdr7AnBkjrIGhWwOKiqV8VC8UMTL78hSwQUZBWCSEVDT5KP8vpjAjz FdKctJb9lDAvTaOUal9zKQJQ0dRBLgCl59SHw9OhDNq4Caytl2Rzklbxd gFsyp5c56Qlp/vDUOQM3b8PXPvGx5/n2GZ0CvFskQm+Rcd+ZdqgoINMVO lwGZ76tE7qcv5LLn0V33crKsjtZNUcCEn4Z+e5MdAuQZqpJh39uY05ADz 8/IzDCHMJTthjel4HrOLA6V1C3wxivcCtG3yYS4+Vtm8KAed9oymCOdpV vhNGf8gLAv/4cQ7uuZwnFnIiIcQbKNWq+RqN6OFFJbMJS/2M2Aau+9QBI Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863240" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863240" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938910" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938910" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:35 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 038/115] KVM: VMX: Introduce test mode related to EPT violation VE Date: Tue, 25 Jul 2023 15:13:49 -0700 Message-Id: <3c4ccdfa824c1be67b6a4d48d3e740f0b14f3c1f.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata To support TDX, KVM is enhanced to operate with #VE. For TDX, KVM programs to inject #VE conditionally and set #VE suppress bit in EPT entry. For VMX case, #VE isn't used. If #VE happens for VMX, it's a bug. To be defensive (test that VMX case isn't broken), introduce option ept_violation_ve_test and when it's set, set error. Suggested-by: Paolo Bonzini Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/vmx.h | 12 +++++++ arch/x86/kvm/vmx/vmcs.h | 5 +++ arch/x86/kvm/vmx/vmx.c | 69 +++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.h | 6 +++- 4 files changed, 90 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 3066ca5ca246..56e192797742 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -70,6 +70,7 @@ #define SECONDARY_EXEC_ENCLS_EXITING VMCS_CONTROL_BIT(ENCLS_EXITING) #define SECONDARY_EXEC_RDSEED_EXITING VMCS_CONTROL_BIT(RDSEED_EXITING) #define SECONDARY_EXEC_ENABLE_PML VMCS_CONTROL_BIT(PAGE_MOD_= LOGGING) +#define SECONDARY_EXEC_EPT_VIOLATION_VE VMCS_CONTROL_BIT(EPT_VIOLATION_VE) #define SECONDARY_EXEC_PT_CONCEAL_VMX VMCS_CONTROL_BIT(PT_CONCEAL_VMX) #define SECONDARY_EXEC_XSAVES VMCS_CONTROL_BIT(XSAVES) #define SECONDARY_EXEC_MODE_BASED_EPT_EXEC VMCS_CONTROL_BIT(MODE_BASED_EPT= _EXEC) @@ -225,6 +226,8 @@ enum vmcs_field { VMREAD_BITMAP_HIGH =3D 0x00002027, VMWRITE_BITMAP =3D 0x00002028, VMWRITE_BITMAP_HIGH =3D 0x00002029, + VE_INFORMATION_ADDRESS =3D 0x0000202A, + VE_INFORMATION_ADDRESS_HIGH =3D 0x0000202B, XSS_EXIT_BITMAP =3D 0x0000202C, XSS_EXIT_BITMAP_HIGH =3D 0x0000202D, ENCLS_EXITING_BITMAP =3D 0x0000202E, @@ -630,4 +633,13 @@ enum vmx_l1d_flush_state { =20 extern enum vmx_l1d_flush_state l1tf_vmx_mitigation; =20 +struct vmx_ve_information { + u32 exit_reason; + u32 delivery; + u64 exit_qualification; + u64 guest_linear_address; + u64 guest_physical_address; + u16 eptp_index; +}; + #endif diff --git a/arch/x86/kvm/vmx/vmcs.h b/arch/x86/kvm/vmx/vmcs.h index 7c1996b433e2..b25625314658 100644 --- a/arch/x86/kvm/vmx/vmcs.h +++ b/arch/x86/kvm/vmx/vmcs.h @@ -140,6 +140,11 @@ static inline bool is_nm_fault(u32 intr_info) return is_exception_n(intr_info, NM_VECTOR); } =20 +static inline bool is_ve_fault(u32 intr_info) +{ + return is_exception_n(intr_info, VE_VECTOR); +} + /* Undocumented: icebp/int1 */ static inline bool is_icebp(u32 intr_info) { diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 540674f1ef2f..c9020e751f69 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -127,6 +127,9 @@ module_param(error_on_inconsistent_vmcs_config, bool, 0= 444); static bool __read_mostly dump_invalid_vmcs =3D 0; module_param(dump_invalid_vmcs, bool, 0644); =20 +static bool __read_mostly ept_violation_ve_test; +module_param(ept_violation_ve_test, bool, 0444); + #define MSR_BITMAP_MODE_X2APIC 1 #define MSR_BITMAP_MODE_X2APIC_APICV 2 =20 @@ -845,6 +848,13 @@ void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu) =20 eb =3D (1u << PF_VECTOR) | (1u << UD_VECTOR) | (1u << MC_VECTOR) | (1u << DB_VECTOR) | (1u << AC_VECTOR); + /* + * #VE isn't used for VMX, but for TDX. To test against unexpected + * change related to #VE for VMX, intercept unexpected #VE and warn on + * it. + */ + if (ept_violation_ve_test) + eb |=3D 1u << VE_VECTOR; /* * Guest access to VMware backdoor ports could legitimately * trigger #GP because of TSS I/O permission bitmap. @@ -2584,6 +2594,9 @@ static int setup_vmcs_config(struct vmcs_config *vmcs= _conf, &_cpu_based_2nd_exec_control)) return -EIO; } + if (!ept_violation_ve_test) + _cpu_based_2nd_exec_control &=3D ~SECONDARY_EXEC_EPT_VIOLATION_VE; + #ifndef CONFIG_X86_64 if (!(_cpu_based_2nd_exec_control & SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) @@ -2608,6 +2621,7 @@ static int setup_vmcs_config(struct vmcs_config *vmcs= _conf, return -EIO; =20 vmx_cap->ept =3D 0; + _cpu_based_2nd_exec_control &=3D ~SECONDARY_EXEC_EPT_VIOLATION_VE; } if (!(_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_VPID) && vmx_cap->vpid) { @@ -4543,6 +4557,7 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx= *vmx) exec_control &=3D ~SECONDARY_EXEC_ENABLE_VPID; if (!enable_ept) { exec_control &=3D ~SECONDARY_EXEC_ENABLE_EPT; + exec_control &=3D ~SECONDARY_EXEC_EPT_VIOLATION_VE; enable_unrestricted_guest =3D 0; } if (!enable_unrestricted_guest) @@ -4676,8 +4691,40 @@ static void init_vmcs(struct vcpu_vmx *vmx) =20 exec_controls_set(vmx, vmx_exec_control(vmx)); =20 - if (cpu_has_secondary_exec_ctrls()) + if (cpu_has_secondary_exec_ctrls()) { secondary_exec_controls_set(vmx, vmx_secondary_exec_control(vmx)); + if (secondary_exec_controls_get(vmx) & + SECONDARY_EXEC_EPT_VIOLATION_VE) { + if (!vmx->ve_info) { + /* ve_info must be page aligned. */ + struct page *page; + + BUILD_BUG_ON(sizeof(*vmx->ve_info) > PAGE_SIZE); + page =3D alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (page) + vmx->ve_info =3D page_to_virt(page); + } + if (vmx->ve_info) { + /* + * Allow #VE delivery. CPU sets this field to + * 0xFFFFFFFF on #VE delivery. Another #VE can + * occur only if software clears the field. + */ + vmx->ve_info->delivery =3D 0; + vmcs_write64(VE_INFORMATION_ADDRESS, + __pa(vmx->ve_info)); + } else { + /* + * Because SECONDARY_EXEC_EPT_VIOLATION_VE is + * used only when ept_violation_ve_test is true, + * it's okay to go with the bit disabled. + */ + pr_err("Failed to allocate ve_info. disabling EPT_VIOLATION_VE.\n"); + secondary_exec_controls_clearbit(vmx, + SECONDARY_EXEC_EPT_VIOLATION_VE); + } + } + } =20 if (cpu_has_tertiary_exec_ctrls()) tertiary_exec_controls_set(vmx, vmx_tertiary_exec_control(vmx)); @@ -5162,6 +5209,12 @@ static int handle_exception_nmi(struct kvm_vcpu *vcp= u) if (is_invalid_opcode(intr_info)) return handle_ud(vcpu); =20 + /* + * #VE isn't supposed to happen. Although vcpu can send + */ + if (KVM_BUG_ON(is_ve_fault(intr_info), vcpu->kvm)) + return -EIO; + error_code =3D 0; if (intr_info & INTR_INFO_DELIVER_CODE_MASK) error_code =3D vmcs_read32(VM_EXIT_INTR_ERROR_CODE); @@ -6356,6 +6409,18 @@ void dump_vmcs(struct kvm_vcpu *vcpu) if (secondary_exec_control & SECONDARY_EXEC_ENABLE_VPID) pr_err("Virtual processor ID =3D 0x%04x\n", vmcs_read16(VIRTUAL_PROCESSOR_ID)); + if (secondary_exec_control & SECONDARY_EXEC_EPT_VIOLATION_VE) { + struct vmx_ve_information *ve_info; + + pr_err("VE info address =3D 0x%016llx\n", + vmcs_read64(VE_INFORMATION_ADDRESS)); + ve_info =3D __va(vmcs_read64(VE_INFORMATION_ADDRESS)); + pr_err("ve_info: 0x%08x 0x%08x 0x%016llx 0x%016llx 0x%016llx 0x%04x\n", + ve_info->exit_reason, ve_info->delivery, + ve_info->exit_qualification, + ve_info->guest_linear_address, + ve_info->guest_physical_address, ve_info->eptp_index); + } } =20 /* @@ -7393,6 +7458,8 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu) free_vpid(vmx->vpid); nested_vmx_free_vcpu(vcpu); free_loaded_vmcs(vmx->loaded_vmcs); + if (vmx->ve_info) + free_page((unsigned long)vmx->ve_info); } =20 int vmx_vcpu_create(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 016a9499b577..0c97328fc3d5 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -347,6 +347,9 @@ struct vcpu_vmx { DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS); DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS); } shadow_msr_intercept; + + /* ve_info must be page aligned. */ + struct vmx_ve_information *ve_info; }; =20 struct kvm_vmx { @@ -558,7 +561,8 @@ static inline u8 vmx_get_rvi(void) SECONDARY_EXEC_ENABLE_VMFUNC | \ SECONDARY_EXEC_BUS_LOCK_DETECTION | \ SECONDARY_EXEC_NOTIFY_VM_EXITING | \ - SECONDARY_EXEC_ENCLS_EXITING) + SECONDARY_EXEC_ENCLS_EXITING | \ + SECONDARY_EXEC_EPT_VIOLATION_VE) =20 #define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0 #define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL \ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCB87C0015E for ; Tue, 25 Jul 2023 22:20:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232383AbjGYWUM (ORCPT ); Tue, 25 Jul 2023 18:20:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232297AbjGYWTU (ORCPT ); Tue, 25 Jul 2023 18:19:20 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98ECF4231; Tue, 25 Jul 2023 15:16:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323389; x=1721859389; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WkpnJmQUuLJFyL2brOxfTQdsOI1DqG1Lui1rEGPKdd4=; b=E0+G+s0iTiHl5OCRpd7xcTbLFzR1KSvGsn4dh7NxvhBahQwXdN9RJ8MP 6ZaXyPfvGBSAGVZ33Ho/9yNiQEaN92S3qKHVCdLDItkl8NQyTJs51nRRO ndwnmcCT8wZMl532TgM4RXMupdM8AARD9gRZKd9QEVgF6B3V2IR7A2fxB nvsXiMhCOMW99CXWDe7HzYpDoxkyq1SF48724q99SxghdL3bN6KCcHCyF ULbvUBXm8W/xa2vvep0biqGUXTUm8CFw8MMI7np8hV7+JPl9q99hIGmfJ x74fKgyQoxDNMZasrkzJrNABRENFzvoCkOILFENgcFTj4dHX+vW5Hq9rp Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863244" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863244" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938913" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938913" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:35 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 039/115] [MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks Date: Tue, 25 Jul 2023 15:13:50 -0700 Message-Id: <68f9096a4c184a02ca9ca43d2c1fa6b959bf8623.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of KVM TDP MMU hooks. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index e893a3d714c7..7903473abad1 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -26,5 +26,5 @@ Patch Layer status * TD vcpu interrupts/exit/hypercall: Not yet =20 * KVM MMU GPA shared bits: Applied -* KVM TDP refactoring for TDX: Applying -* KVM TDP MMU hooks: Not yet +* KVM TDP refactoring for TDX: Applied +* KVM TDP MMU hooks: Applying --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41E7EC05052 for ; Tue, 25 Jul 2023 22:20:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232468AbjGYWT6 (ORCPT ); Tue, 25 Jul 2023 18:19:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232236AbjGYWS3 (ORCPT ); Tue, 25 Jul 2023 18:18:29 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B13426A2; Tue, 25 Jul 2023 15:16:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323375; x=1721859375; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BFQ/sesQVcdvoe6xpw9/CVcgY2UBxZorT/LZeWCLqRI=; b=OESg9OJoYnKOotcBAmCjlJxS4CAZ+zmQlUXIS9+ewBvSKd9xSPMUYCfW /loH1eeiinfTCGKctMceznIlOi6GL2qX9NBOrJkYJHyxFqEl4CBxiVw4k LXuLvWR1iMgNrQ5cXuPVn9Oe8TOQOiVTD4BQJvNFelqXJeYSWS/LS6Y6d vu/hU47gp9z6bp9p1PdtcdV7ZsHdNaK/tUSBeMCwQyi7vvCdAvylIM0nE Ov/KZU9u4Wg0DmP+o7ub5GiT/BbQqQwvPrd/MG1VrTB1NZ2XqXotMtGB9 aAEcPHyxm7oUZEaVHM+BkZHMOutQNrq4SKObW44XIV81ATfxRltqqO4la A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863250" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863250" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938916" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938916" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:36 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Chao Gao Subject: [PATCH v15 040/115] KVM: x86/mmu: Assume guest MMIOs are shared Date: Tue, 25 Jul 2023 15:13:51 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Chao Gao Guest TD doesn't necessarily invoke MAP_GPA to convert the virtual MMIO range to shared before accessing it. When TD tries to access the virtual device's MMIO as shared, an EPT violation is raised first. kvm_mem_is_private() checks whether the GFN is shared or private. If MAP_GPA is not called for the GPA, KVM thinks the GPA is private and refuses shared access, and doesn't set up shared EPT entry. The guest can't make progress. Instead of requiring the guest to invoke MAP_GPA for regions of virtual MMIOs assume regions of virtual MMIOs are shared in KVM as well (i.e., GPAs either have no kvm_memory_slot or are backed by host MMIOs). So that guests can access those MMIO regions. Signed-off-by: Chao Gao Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/mmu.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 9bf8d05937c5..ffe292b3a44d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4418,7 +4418,12 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, = struct kvm_page_fault *fault return RET_PF_EMULATE; } =20 - if (fault->is_private !=3D kvm_mem_is_private(vcpu->kvm, fault->gfn)) { + /* + * !fault->slot means MMIO. Don't require explicit GPA conversion for + * MMIO because MMIO is assigned at the boot time. + */ + if (fault->slot && + fault->is_private !=3D kvm_mem_is_private(vcpu->kvm, fault->gfn)) { if (vcpu->kvm->arch.vm_type =3D=3D KVM_X86_SW_PROTECTED_VM) return RET_PF_RETRY; else --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90263C05052 for ; Tue, 25 Jul 2023 22:20:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232431AbjGYWU1 (ORCPT ); Tue, 25 Jul 2023 18:20:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232353AbjGYWTV (ORCPT ); Tue, 25 Jul 2023 18:19:21 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D27DA423C; Tue, 25 Jul 2023 15:16:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323391; x=1721859391; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PI66i/YrGN1PUdZt3jE6X7iD6PIxSsfOjRKHLyYrHqw=; b=CkrbOmG+KlDoRfp3mAb4aQuCkZbWXZE5kT8KZqbTJthLLBe3ojjSKAgK wX5wMIjPS/9sgRQ6J4JvsUWxTdmt7HSe5VW1oX6ipAX1JAXwF8QQMX7sO IWLz9kbf3OIsEnKOdz15JYt1ohlL9ZRyQX1RVuQ26nJDCo0JWr09/esqa sYyHjHp4cP015uvng5+88Fh6C5vqY0DgBg1YIv6Vj5Wd/kK9QG4wkAVpS 44wBvRsuASHj0n1lujic86UR/XzDkPr8qsx97DIshZbUnuGVzvlZXb5Aq 2lcmseorduWK1eyFZQfNHbbJwKOLfxL+6keIKHdKodN8anCSM/dwfzHz/ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863254" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863254" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938919" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938919" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:36 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 041/115] KVM: x86/tdp_mmu: Init role member of struct kvm_mmu_page at allocation Date: Tue, 25 Jul 2023 15:13:52 -0700 Message-Id: <6006f01ea54c46d0688cd307b648d7d66b26b5e6.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Refactor tdp_mmu_alloc_sp() and tdp_mmu_init_sp and eliminate tdp_mmu_init_child_sp(). Currently tdp_mmu_init_sp() (or tdp_mmu_init_child_sp()) sets kvm_mmu_page.role after tdp_mmu_alloc_sp() allocating struct kvm_mmu_page and its page table page. This patch makes tdp_mmu_alloc_sp() initialize kvm_mmu_page.role instead of tdp_mmu_init_sp(). To handle private page tables, argument of is_private needs to be passed down. Given that already page level is passed down, it would be cumbersome to add one more parameter about sp. Instead replace the level argument with union kvm_mmu_page_role. Thus the number of argument won't be increased and more info about sp can be passed down. For private sp, secure page table will be also allocated in addition to struct kvm_mmu_page and page table (spt member). The allocation functions (tdp_mmu_alloc_sp() and __tdp_mmu_alloc_sp_for_split()) need to know if the allocation is for the conventional page table or private page table. Pass union kvm_mmu_role to those functions and initialize role member of struct kvm_mmu_page. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/tdp_iter.h | 12 ++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 44 ++++++++++++++++--------------------- 2 files changed, 31 insertions(+), 25 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index fae559559a80..e1e40e3f5eb7 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -135,4 +135,16 @@ void tdp_iter_start(struct tdp_iter *iter, struct kvm_= mmu_page *root, void tdp_iter_next(struct tdp_iter *iter); void tdp_iter_restart(struct tdp_iter *iter); =20 +static inline union kvm_mmu_page_role tdp_iter_child_role(struct tdp_iter = *iter) +{ + union kvm_mmu_page_role child_role; + struct kvm_mmu_page *parent_sp; + + parent_sp =3D sptep_to_sp(rcu_dereference(iter->sptep)); + + child_role =3D parent_sp->role; + child_role.level--; + return child_role; +} + #endif /* __KVM_X86_MMU_TDP_ITER_H */ diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 4fe31a1efa9a..87c378d0677f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -227,24 +227,30 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct = kvm *kvm, kvm_mmu_page_as_id(_root) !=3D _as_id) { \ } else =20 -static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu) +static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, + union kvm_mmu_page_role role) { struct kvm_mmu_page *sp; =20 sp =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache); sp->spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); + sp->role =3D role; =20 return sp; } =20 static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, tdp_ptep_t sptep, - gfn_t gfn, union kvm_mmu_page_role role) + gfn_t gfn) { INIT_LIST_HEAD(&sp->possible_nx_huge_page_link); =20 set_page_private(virt_to_page(sp->spt), (unsigned long)sp); =20 - sp->role =3D role; + /* + * role must be set before calling this function. At least role.level + * is not 0 (PG_LEVEL_NONE). + */ + WARN_ON_ONCE(!sp->role.word); sp->gfn =3D gfn; sp->ptep =3D sptep; sp->tdp_mmu_page =3D true; @@ -252,20 +258,6 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, t= dp_ptep_t sptep, trace_kvm_mmu_get_page(sp, true); } =20 -static void tdp_mmu_init_child_sp(struct kvm_mmu_page *child_sp, - struct tdp_iter *iter) -{ - struct kvm_mmu_page *parent_sp; - union kvm_mmu_page_role role; - - parent_sp =3D sptep_to_sp(rcu_dereference(iter->sptep)); - - role =3D parent_sp->role; - role.level--; - - tdp_mmu_init_sp(child_sp, iter->sptep, iter->gfn, role); -} - hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) { union kvm_mmu_page_role role =3D vcpu->arch.mmu->root_role; @@ -284,8 +276,8 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vc= pu) goto out; } =20 - root =3D tdp_mmu_alloc_sp(vcpu); - tdp_mmu_init_sp(root, NULL, 0, role); + root =3D tdp_mmu_alloc_sp(vcpu, role); + tdp_mmu_init_sp(root, NULL, 0); =20 /* * TDP MMU roots are kept until they are explicitly invalidated, either @@ -1100,8 +1092,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm= _page_fault *fault) * The SPTE is either non-present or points to a huge page that * needs to be split. */ - sp =3D tdp_mmu_alloc_sp(vcpu); - tdp_mmu_init_child_sp(sp, &iter); + sp =3D tdp_mmu_alloc_sp(vcpu, tdp_iter_child_role(&iter)); + tdp_mmu_init_sp(sp, iter.sptep, iter.gfn); =20 sp->nx_huge_page_disallowed =3D fault->huge_page_disallowed; =20 @@ -1339,7 +1331,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, return spte_set; } =20 -static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp) +static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp, union = kvm_mmu_page_role role) { struct kvm_mmu_page *sp; =20 @@ -1349,6 +1341,7 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_sp= lit(gfp_t gfp) if (!sp) return NULL; =20 + sp->role =3D role; sp->spt =3D (void *)__get_free_page(gfp); if (!sp->spt) { kmem_cache_free(mmu_page_header_cache, sp); @@ -1362,6 +1355,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spli= t(struct kvm *kvm, struct tdp_iter *iter, bool shared) { + union kvm_mmu_page_role role =3D tdp_iter_child_role(iter); struct kvm_mmu_page *sp; =20 /* @@ -1373,7 +1367,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spli= t(struct kvm *kvm, * If this allocation fails we drop the lock and retry with reclaim * allowed. */ - sp =3D __tdp_mmu_alloc_sp_for_split(GFP_NOWAIT | __GFP_ACCOUNT); + sp =3D __tdp_mmu_alloc_sp_for_split(GFP_NOWAIT | __GFP_ACCOUNT, role); if (sp) return sp; =20 @@ -1385,7 +1379,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spli= t(struct kvm *kvm, write_unlock(&kvm->mmu_lock); =20 iter->yielded =3D true; - sp =3D __tdp_mmu_alloc_sp_for_split(GFP_KERNEL_ACCOUNT); + sp =3D __tdp_mmu_alloc_sp_for_split(GFP_KERNEL_ACCOUNT, role); =20 if (shared) read_lock(&kvm->mmu_lock); @@ -1480,7 +1474,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *= kvm, continue; } =20 - tdp_mmu_init_child_sp(sp, &iter); + tdp_mmu_init_sp(sp, iter.sptep, iter.gfn); =20 if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared)) goto retry; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A6FC07E8C for ; Tue, 25 Jul 2023 22:20:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231386AbjGYWUb (ORCPT ); Tue, 25 Jul 2023 18:20:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232358AbjGYWTV (ORCPT ); Tue, 25 Jul 2023 18:19:21 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 024C24482; Tue, 25 Jul 2023 15:16:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323392; x=1721859392; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oSpwVS/FUjL4d73DdU92utlrGySSdjlxHtDcqhGvYZw=; b=k4yM29fXIQYiIa+0yHggSdfvSE18XSyYon/D0F8QTPwMGvAtntEsFmgh 7ea81ELsvF73lm6Y7m18wb8627SjRFDfhMES2t1/SZsmsgakOS3gkZ/Sg 25WpA0cBKRjlWG2vmwJ+V8kbJV9yT2HSu5q5NdcOlapx+89Updipzg5nJ t6SOrqp31S0Ag/1OJ/KJ4dS3URoiyAk0vqZGpy9gsR0SkNqk5QaaIrN4F UcvLkxUl2JavL0sE3eK1xDfdXD6vxUSLpyA1cUm3t3tBMFJ7SKYrabFZQ HPmOiFylwuubMX7hm6pMUyVf7iKtB6drpSij4W1vMLaUf6Yj6SoKGvM8t g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863260" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863260" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938923" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938923" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:36 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 042/115] KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role Date: Tue, 25 Jul 2023 15:13:53 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because TDX support introduces private mapping, add a new member in union kvm_mmu_page_role with access functions to check the member. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 27 +++++++++++++++++++++++++++ arch/x86/kvm/mmu/mmu_internal.h | 5 +++++ arch/x86/kvm/mmu/spte.h | 6 ++++++ 3 files changed, 38 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 0bc53c942c6c..56f9297b1bb8 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -340,7 +340,12 @@ union kvm_mmu_page_role { unsigned ad_disabled:1; unsigned guest_mode:1; unsigned passthrough:1; +#ifdef CONFIG_KVM_MMU_PRIVATE + unsigned is_private:1; + unsigned :4; +#else unsigned :5; +#endif =20 /* * This is left at the top of the word so that @@ -352,6 +357,28 @@ union kvm_mmu_page_role { }; }; =20 +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline bool kvm_mmu_page_role_is_private(union kvm_mmu_page_role ro= le) +{ + return !!role.is_private; +} + +static inline void kvm_mmu_page_role_set_private(union kvm_mmu_page_role *= role) +{ + role->is_private =3D 1; +} +#else +static inline bool kvm_mmu_page_role_is_private(union kvm_mmu_page_role ro= le) +{ + return false; +} + +static inline void kvm_mmu_page_role_set_private(union kvm_mmu_page_role *= role) +{ + WARN_ON_ONCE(1); +} +#endif + /* * kvm_mmu_extended_role complements kvm_mmu_page_role, tracking properties * relevant to the current MMU configuration. When loading CR0, CR4, or = EFER, diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index 76fa38da74f1..2409c4dca208 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -143,6 +143,11 @@ static inline int kvm_mmu_page_as_id(struct kvm_mmu_pa= ge *sp) return kvm_mmu_role_as_id(sp->role); } =20 +static inline bool is_private_sp(const struct kvm_mmu_page *sp) +{ + return kvm_mmu_page_role_is_private(sp->role); +} + static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page = *sp) { /* diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index a8418fd8ae9e..41973fe6bc22 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -251,6 +251,12 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sp= tep) return to_shadow_page(__pa(sptep)); } =20 +static inline bool is_private_sptep(u64 *sptep) +{ + WARN_ON_ONCE(!sptep); + return is_private_sp(sptep_to_sp(sptep)); +} + static inline bool is_mmio_spte(struct kvm *kvm, u64 spte) { return (spte & shadow_mmio_mask) =3D=3D kvm->arch.shadow_mmio_value && --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E31B8C001E0 for ; Tue, 25 Jul 2023 22:20:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232262AbjGYWUh (ORCPT ); Tue, 25 Jul 2023 18:20:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232386AbjGYWTZ (ORCPT ); Tue, 25 Jul 2023 18:19:25 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EA0B4494; Tue, 25 Jul 2023 15:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323394; x=1721859394; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F08o8A4X3RdnEoknQIAeaSmU/wYAWbicXJ6YfnS08Ls=; b=aOdHUr2t2fk8i7ONpNUu09qh8N0ejm6L8amgRoMbb/2ZQG1p7xMhFIKt wDxI/bVA6hCtIzGzyx3bw2sIUShuQj7BpLQiewYKBwzUkn27Fz6RFVFDz xgH5MU3FzOHnlahyczIURgOUg9zdnES+XhTrUS5Cp44t1QWIllGyfs7uP FRwzd1l5zIm1CAj6BbWGhdiunwXA6eGTwz7BNitu3Pki3xmlUa3MCBZh4 EHzq7hCFWKm24IFv0dKP4YdbMVFl4YcOGpx9smPBDSLmj3AGVDVs76jda 6ZyS9OIB/O508THCwNwvJ5bvmP8VOR3Bp82ohr0WUIzc4mdoIDxExVCDh Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863265" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863265" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938927" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938927" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:37 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 043/115] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page Date: Tue, 25 Jul 2023 15:13:54 -0700 Message-Id: <1c859e69241172f9b640bcf9a47818f38a7ce605.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive. When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk private page table, allocate one more page to copy the dummy page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the private page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with private page table), and the page table with private page table is called a private page table. The relationship is depicted below. Add a private pointer to struct kvm_mmu_page for private page table and add helper functions to allocate/initialize/free a private page table page. KVM page fault | | | V | -------------+---------- | | | | V V | shared GPA private GPA | | | | V V | shared PT root dummy PT root | private PT root | | | | V V | V shared PT dummy PT ----propagate----> private PT | | | | | \-----------------+------\ | | | | | V | V V shared guest page | private guest page | non-encrypted memory | encrypted memory | PT: page table - Shared PT is visible to KVM and it is used by CPU. - Private PT is used by CPU but it is invisible to KVM. - Dummy PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual private PT which is used by CPU. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 5 ++ arch/x86/kvm/mmu/mmu.c | 7 +++ arch/x86/kvm/mmu/mmu_internal.h | 83 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 1 + 4 files changed, 92 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 56f9297b1bb8..a57c2c96ffc4 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -817,6 +817,11 @@ struct kvm_vcpu_arch { struct kvm_mmu_memory_cache mmu_shadow_page_cache; struct kvm_mmu_memory_cache mmu_shadowed_info_cache; struct kvm_mmu_memory_cache mmu_page_header_cache; + /* + * This cache is to allocate private page table. E.g. Secure-EPT used + * by the TDX module. + */ + struct kvm_mmu_memory_cache mmu_private_spt_cache; =20 /* * QEMU userspace and the guest each have their own FPU state. diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ffe292b3a44d..f30d62362667 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -697,6 +697,12 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vc= pu, bool maybe_indirect) 1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM); if (r) return r; + if (kvm_gfn_shared_mask(vcpu->kvm)) { + r =3D kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_private_spt_cache, + PT64_ROOT_MAX_LEVEL); + if (r) + return r; + } r =3D kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache, PT64_ROOT_MAX_LEVEL); if (r) @@ -716,6 +722,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcp= u) kvm_mmu_free_memory_cache(&vcpu->arch.mmu_pte_list_desc_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadow_page_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadowed_info_cache); + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_private_spt_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache); } =20 diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index 2409c4dca208..17ad9df1bb71 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -93,7 +93,23 @@ struct kvm_mmu_page { int root_count; refcount_t tdp_mmu_root_count; }; - unsigned int unsync_children; + union { + struct { + unsigned int unsync_children; + /* + * Number of writes since the last time traversal + * visited this page. + */ + atomic_t write_flooding_count; + }; +#ifdef CONFIG_KVM_MMU_PRIVATE + /* + * Associated private shadow page table, e.g. Secure-EPT page + * passed to the TDX module. + */ + void *private_spt; +#endif + }; union { struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */ tdp_ptep_t ptep; @@ -122,9 +138,6 @@ struct kvm_mmu_page { int clear_spte_count; #endif =20 - /* Number of writes since the last time traversal visited this page. */ - atomic_t write_flooding_count; - #ifdef CONFIG_X86_64 /* Used for freeing the page asynchronously if it is a TDP MMU page. */ struct rcu_head rcu_head; @@ -148,6 +161,68 @@ static inline bool is_private_sp(const struct kvm_mmu_= page *sp) return kvm_mmu_page_role_is_private(sp->role); } =20 +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline void *kvm_mmu_private_spt(struct kvm_mmu_page *sp) +{ + return sp->private_spt; +} + +static inline void kvm_mmu_init_private_spt(struct kvm_mmu_page *sp, void = *private_spt) +{ + sp->private_spt =3D private_spt; +} + +static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct= kvm_mmu_page *sp) +{ + bool is_root =3D vcpu->arch.root_mmu.root_role.level =3D=3D sp->role.leve= l; + + KVM_BUG_ON(!kvm_mmu_page_role_is_private(sp->role), vcpu->kvm); + if (is_root) + /* + * Because TDX module assigns root Secure-EPT page and set it to + * Secure-EPTP when TD vcpu is created, secure page table for + * root isn't needed. + */ + sp->private_spt =3D NULL; + else { + /* + * Because the TDX module doesn't trust VMM and initializes + * the pages itself, KVM doesn't initialize them. Allocate + * pages with garbage and give them to the TDX module. + */ + sp->private_spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_private_s= pt_cache); + /* + * Because mmu_private_spt_cache is topped up before staring kvm + * page fault resolving, the allocation above shouldn't fail. + */ + WARN_ON_ONCE(!sp->private_spt); + } +} + +static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp) +{ + if (sp->private_spt) + free_page((unsigned long)sp->private_spt); +} +#else +static inline void *kvm_mmu_private_spt(struct kvm_mmu_page *sp) +{ + return NULL; +} + +static inline void kvm_mmu_init_private_spt(struct kvm_mmu_page *sp, void = *private_spt) +{ +} + +static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct= kvm_mmu_page *sp) +{ +} + +static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp) +{ +} +#endif + static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page = *sp) { /* diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 87c378d0677f..8ca547987238 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -66,6 +66,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) =20 static void tdp_mmu_free_sp(struct kvm_mmu_page *sp) { + kvm_mmu_free_private_spt(sp); free_page((unsigned long)sp->spt); kmem_cache_free(mmu_page_header_cache, sp); } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13C8DC41513 for ; Tue, 25 Jul 2023 22:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232443AbjGYWUm (ORCPT ); Tue, 25 Jul 2023 18:20:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232391AbjGYWT0 (ORCPT ); Tue, 25 Jul 2023 18:19:26 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10FFA449A; Tue, 25 Jul 2023 15:16:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323395; x=1721859395; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dgcv4d08IZHY00QcJwZ7RSp4KjvPFjj7y6USyEPdrVI=; b=eKgdxHWSOKeZStDheKHZgQsat5N3bW6fxEVYXFnMD3HqbpeaQLZJTrw0 qM/JxE8Ibs8VoJKtwRaJslRLo+EQRIrAq7hGgPuuNHItV2m/Cn8q84Of2 s2CFYPDPUPEzHi7rVOpmwCksxKq7ZBJzOi4bYE3yvkh4dtEVI9tpv1X8I SCZ5HNaLtfk7p7PA+KKXmxUSuvRLuvqMhl2IusKRnrVWdSKLs4FGCkBXJ UMtnOgJj/ijyrqoBxfF3rFvsZczUk4wu2do0jXslczT4kcyCsNXww2FWr DoxN52Y4+yxiLsSzOrxPpAhFTWZ6IQpf6sRsREsOqu+e5Y1A98bWBVMoD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863269" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863269" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938930" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938930" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:37 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 044/115] KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases Date: Tue, 25 Jul 2023 15:13:55 -0700 Message-Id: <835d544fc82d1776c9c623cb6c52336d46955cb8.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson TDX supports only write-back(WB) memory type for private memory architecturally so that (virtualized) memory type change doesn't make sense for private memory. Also currently, page migration isn't supported for TDX yet. (TDX architecturally supports page migration. it's KVM and kernel implementation issue.) Regarding memory type change (mtrr virtualization and lapic page mapping change), pages are zapped by kvm_zap_gfn_range(). On the next KVM page fault, the SPTE entry with a new memory type for the page is populated. Regarding page migration, pages are zapped by the mmu notifier. On the next KVM page fault, the new migrated page is populated. Don't zap private pages on unmapping for those two cases. When deleting/moving a KVM memory slot, zap private pages. Typically tearing down VM. Don't invalidate private page tables. i.e. zap only leaf SPTEs for KVM mmu that has a shared bit mask. The existing kvm_tdp_mmu_invalidate_all_roots() depends on role.invalid with read-lock of mmu_lock so that other vcpu can operate on KVM mmu concurrently. It marks the root page table invalid and zaps SPTEs of the root page tables. The TDX module doesn't allow to unlink a protected root page table from the hardware and then allocate a new one for it. i.e. replacing a protected root page table. Instead, zap only leaf SPTEs for KVM mmu with a shared bit mask set. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/mmu.c | 62 ++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 5 +-- 3 files changed, 93 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f30d62362667..513083a14552 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6311,7 +6311,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) * e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield. */ if (tdp_mmu_enabled) - kvm_tdp_mmu_invalidate_all_roots(kvm); + kvm_tdp_mmu_invalidate_all_roots(kvm, true); =20 /* * Notify all vcpus to reload its shadow page table and flush TLB. @@ -6344,11 +6344,57 @@ static bool kvm_has_zapped_obsolete_pages(struct kv= m *kvm) return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages)); } =20 +static void kvm_mmu_zap_memslot(struct kvm *kvm, struct kvm_memory_slot *s= lot) +{ + bool flush =3D false; + + write_lock(&kvm->mmu_lock); + + /* + * Zapping non-leaf SPTEs, a.k.a. not-last SPTEs, isn't required, worst + * case scenario we'll have unused shadow pages lying around until they + * are recycled due to age or when the VM is destroyed. + */ + if (tdp_mmu_enabled) { + struct kvm_gfn_range range =3D { + .slot =3D slot, + .start =3D slot->base_gfn, + .end =3D slot->base_gfn + slot->npages, + .may_block =3D true, + + /* + * This handles both private gfn and shared gfn. + * All private page should be zapped on memslot deletion. + */ + .only_private =3D true, + .only_shared =3D true, + }; + + flush =3D kvm_tdp_mmu_unmap_gfn_range(kvm, &range, flush); + } else { + /* TDX supports only TDP-MMU case. */ + WARN_ON_ONCE(1); + flush =3D true; + } + if (flush) + kvm_flush_remote_tlbs(kvm); + + write_unlock(&kvm->mmu_lock); +} + static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm, struct kvm_memory_slot *slot, struct kvm_page_track_notifier_node *node) { - kvm_mmu_zap_all_fast(kvm); + if (kvm_gfn_shared_mask(kvm)) + /* + * Secure-EPT requires to release PTs from the leaf. The + * optimization to zap root PT first with child PT doesn't + * work. + */ + kvm_mmu_zap_memslot(kvm, slot); + else + kvm_mmu_zap_all_fast(kvm); } =20 int kvm_mmu_init_vm(struct kvm *kvm) @@ -6456,8 +6502,18 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_st= art, gfn_t gfn_end) =20 if (tdp_mmu_enabled) { for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) + /* + * zap_private =3D true. Zap both private/shared pages. + * + * kvm_zap_gfn_range() is used when MTRR or PAT memory + * type was changed. Later on the next kvm page fault, + * populate it with updated spte entry. + * Because only WB is supported for private pages, don't + * care of private pages. + */ flush =3D kvm_tdp_mmu_zap_leafs(kvm, i, gfn_start, - gfn_end, true, flush); + gfn_end, true, flush, + false); } =20 if (flush) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 8ca547987238..643c7c65456c 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -45,7 +45,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) * for zapping and thus puts the TDP MMU's reference to each root, i.e. * ultimately frees all roots. */ - kvm_tdp_mmu_invalidate_all_roots(kvm); + kvm_tdp_mmu_invalidate_all_roots(kvm, false); =20 /* * Destroying a workqueue also first flushes the workqueue, i.e. no @@ -831,7 +831,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu= _page *sp) * operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end, bool can_yield, bool flush) + gfn_t start, gfn_t end, bool can_yield, bool flush, + bool zap_private) { struct tdp_iter iter; =20 @@ -839,6 +840,10 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct = kvm_mmu_page *root, =20 lockdep_assert_held_write(&kvm->mmu_lock); =20 + WARN_ON_ONCE(zap_private && !is_private_sp(root)); + if (!zap_private && is_private_sp(root)) + return false; + rcu_read_lock(); =20 for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) { @@ -871,12 +876,13 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct= kvm_mmu_page *root, * more SPTEs were zapped since the MMU lock was last acquired. */ bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, gfn_t = end, - bool can_yield, bool flush) + bool can_yield, bool flush, bool zap_private) { struct kvm_mmu_page *root; =20 for_each_tdp_mmu_root_yield_safe(kvm, root, as_id) - flush =3D tdp_mmu_zap_leafs(kvm, root, start, end, can_yield, flush); + flush =3D tdp_mmu_zap_leafs(kvm, root, start, end, can_yield, flush, + zap_private && is_private_sp(root)); =20 return flush; } @@ -924,7 +930,7 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) * Note, the asynchronous worker is gifted the TDP MMU's reference. * See kvm_tdp_mmu_get_vcpu_root_hpa(). */ -void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm) +void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm, bool skip_private) { struct kvm_mmu_page *root; =20 @@ -952,6 +958,12 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm) rcu_read_lock(); =20 list_for_each_entry_rcu(root, &kvm->arch.tdp_mmu_roots, link) { + /* + * Skip private root since private page table + * is only torn down when VM is destroyed. + */ + if (skip_private && is_private_sp(root)) + continue; if (!root->role.invalid) { root->role.invalid =3D true; tdp_mmu_schedule_zap_root(kvm, root); @@ -1136,11 +1148,24 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct k= vm_page_fault *fault) return ret; } =20 +/* Used by mmu notifier via kvm_unmap_gfn_range() */ bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *ra= nge, bool flush) { + bool zap_private =3D false; + + if (kvm_gfn_shared_mask(kvm)) { + if (!range->only_private && !range->only_shared) + /* attributes change */ + zap_private =3D !(range->arg.attributes & + KVM_MEMORY_ATTRIBUTE_PRIVATE); + else + zap_private =3D range->only_private; + } + return kvm_tdp_mmu_zap_leafs(kvm, range->slot->as_id, range->start, - range->end, range->may_block, flush); + range->end, range->may_block, flush, + zap_private); } =20 typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter, diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 0a63b1afabd3..3df604352648 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -21,10 +21,11 @@ void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_m= mu_page *root, bool shared); =20 bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, int as_id, gfn_t start, - gfn_t end, bool can_yield, bool flush); + gfn_t end, bool can_yield, bool flush, + bool zap_private); bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp); void kvm_tdp_mmu_zap_all(struct kvm *kvm); -void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm); +void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm, bool skip_private); void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm); =20 int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC113EB64DD for ; Tue, 25 Jul 2023 22:20:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230095AbjGYWUq (ORCPT ); Tue, 25 Jul 2023 18:20:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232394AbjGYWT0 (ORCPT ); Tue, 25 Jul 2023 18:19:26 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FC9944A1; Tue, 25 Jul 2023 15:16:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323396; x=1721859396; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FnRel535juPIzcohaQT4gbTQVJFUznDwrWq/ooVWuIg=; b=D7AHKu1l3qfg6WWfHhI0Vpf5NjZhv1QItXmctsqwIyCLNegEAu1UIBKW scudz7saEzbup4HVVt0dC+3hLbsR/lmb/DC4XwurtB++CKu1XXPoBPhM9 dAIV8CmVeUEEXcQbEGc7P0pR/eMFYxZTHd2LCGn2t0k9B2+x6HDIljJoZ R2HVN8s1MRxe/usuh5jkw62nM7UXitF0k3Mp4Peong/hMci6G24bcMORm WoDdumsFBHFcBBEw9bVUaHcw2N1zANZnLh2PECXhAt2OwN/LSaRRpfK2B RXFwW5O7o3qF7S0hYbQ9HFVUAt11ReBi+f9shFcvJ6U7wFwgYAqTZj1j5 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863277" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863277" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938934" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938934" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:38 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 045/115] KVM: x86/tdp_mmu: Sprinkle __must_check Date: Tue, 25 Jul 2023 15:13:56 -0700 Message-Id: <4010ab10284b74d841d455d1b3e433376e8b6c4a.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDP MMU allows tdp_mmu_set_spte_atomic() and tdp_mmu_zap_spte_atomic() to return -EBUSY or -EAGAIN error. The caller must check the return value and retry. Sprinkle __must_check to guarantee it. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 643c7c65456c..d3788a414551 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -567,9 +567,9 @@ static void handle_changed_spte(struct kvm *kvm, int as= _id, gfn_t gfn, * no side-effects other than setting iter->old_spte to the last * known value of the spte. */ -static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm, - struct tdp_iter *iter, - u64 new_spte) +static inline int __must_check tdp_mmu_set_spte_atomic(struct kvm *kvm, + struct tdp_iter *iter, + u64 new_spte) { u64 *sptep =3D rcu_dereference(iter->sptep); =20 @@ -599,8 +599,8 @@ static inline int tdp_mmu_set_spte_atomic(struct kvm *k= vm, return 0; } =20 -static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, - struct tdp_iter *iter) +static inline int __must_check tdp_mmu_zap_spte_atomic(struct kvm *kvm, + struct tdp_iter *iter) { int ret; =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C4CCEB64DD for ; Tue, 25 Jul 2023 22:21:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232553AbjGYWU7 (ORCPT ); Tue, 25 Jul 2023 18:20:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232318AbjGYWTx (ORCPT ); Tue, 25 Jul 2023 18:19:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBC8346A9; Tue, 25 Jul 2023 15:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323414; x=1721859414; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TcAj6qtDbJdlu3C9bJRLWto/yTJqbDJNJ0pT5Zv4Tpc=; b=nDRxnUv3S7pKX3wDqtgv81n4JSPDuvrO0ro8C2umMlH50cIU6WfcPOb7 0XTOp6lto+YVEw52AzXASXRYEJVOYj2Ee4G/XQxcY4rqgXUGc5roKaZRc LFg7loWXR8lOEUvqDVPzS34P3LiftqvLU4cRjFMY4lD/cpJi/AaLi2M5x cBq66WQ7pIfoOSxMFHTNeCvTh4hBk7ql+ASD9vyH2W8whZU9eT0MhtKYj 0u1JFelVESW+QYqabOH4NW5BUY4M8fa14+qttBxZcXMw+H//dwb2j9xxf wlFHzoJ+TR8ybWl5AkHr6MmEsfUd4NQZDyq+GiTzey22wTANLumiqnSO/ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863282" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863282" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938938" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938938" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:38 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 046/115] KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU Date: Tue, 25 Jul 2023 15:13:57 -0700 Message-Id: <9e641e1368b37e108ff8a8b1078a73588f7f9539.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Allocate protected page table for private page table, and add hooks to operate on protected page table. This patch adds allocation/free of protected page tables and hooks. When calling hooks to update SPTE entry, freeze the entry, call hooks and unfreeze the entry to allow concurrent updates on page tables. Which is the advantage of TDP MMU. As kvm_gfn_shared_mask() returns false always, those hooks aren't called yet with this patch. When the faulting GPA is private, the KVM fault is called private. When resolving private KVM fault, allocate protected page table and call hooks to operate on protected page table. On the change of the private PTE entry, invoke kvm_x86_ops hook in __handle_changed_spte() to propagate the change to protected page table. The following depicts the relationship. private KVM page fault | | | V | private GPA | CPU protected EPTP | | | V | V private PT root | protected PT root | | | V | V private PT --hook to propagate-->protected PT | | | \--------------------+------\ | | | | | V V | private guest page | | non-encrypted memory | encrypted memory | PT: page table The existing KVM TDP MMU code uses atomic update of SPTE. On populating the EPT entry, atomically set the entry. However, it requires TLB shootdown to zap SPTE. To address it, the entry is frozen with the special SPTE value that clears the present bit. After the TLB shootdown, the entry is set to the eventual value (unfreeze). For protected page table, hooks are called to update protected page table in addition to direct access to the private SPTE. For the zapping case, it works to freeze the SPTE. It can call hooks in addition to TLB shootdown. For populating the private SPTE entry, there can be a race condition without further protection vcpu 1: populating 2M private SPTE vcpu 2: populating 4K private SPTE vcpu 2: TDX SEAMCALL to update 4K protected SPTE =3D> error vcpu 1: TDX SEAMCALL to update 2M protected SPTE To avoid the race, the frozen SPTE is utilized. Instead of atomic update of the private entry, freeze the entry, call the hook that update protected SPTE, set the entry to the final value. Support 4K page only at this stage. 2M page support can be done in future patches. Co-developed-by: Kai Huang Signed-off-by: Kai Huang Signed-off-by: Isaku Yamahata --- v14 -> v15: - Refined is_private condition check in kvm_tdp_mmu_map(). Add kvm_gfn_shared_mask() check. - catch up for struct kvm_range change Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 5 + arch/x86/include/asm/kvm_host.h | 11 ++ arch/x86/kvm/mmu/mmu.c | 13 +- arch/x86/kvm/mmu/mmu_internal.h | 19 +- arch/x86/kvm/mmu/tdp_iter.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 293 +++++++++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- virt/kvm/kvm_main.c | 1 + 8 files changed, 307 insertions(+), 39 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index bcf04a75b506..49f19cfeb11d 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -98,6 +98,11 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr) KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr) KVM_X86_OP_OPTIONAL_RET0(get_mt_mask) KVM_X86_OP(load_mmu_pgd) +KVM_X86_OP_OPTIONAL(link_private_spt) +KVM_X86_OP_OPTIONAL(free_private_spt) +KVM_X86_OP_OPTIONAL(set_private_spte) +KVM_X86_OP_OPTIONAL(remove_private_spte) +KVM_X86_OP_OPTIONAL(zap_private_spte) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index a57c2c96ffc4..9705e9f30068 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -475,6 +475,7 @@ struct kvm_mmu { int (*sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int i); struct kvm_mmu_root_info root; + hpa_t private_root_hpa; union kvm_cpu_role cpu_role; union kvm_mmu_page_role root_role; =20 @@ -1698,6 +1699,16 @@ struct kvm_x86_ops { void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_level); =20 + int (*link_private_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level, + void *private_spt); + int (*free_private_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level, + void *private_spt); + int (*set_private_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level, + kvm_pfn_t pfn); + int (*remove_private_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level leve= l, + kvm_pfn_t pfn); + int (*zap_private_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level); + bool (*has_wbinvd_exit)(void); =20 u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 513083a14552..5b48ac4a5fbc 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3757,7 +3757,12 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *v= cpu) goto out_unlock; =20 if (tdp_mmu_enabled) { - root =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu); + if (kvm_gfn_shared_mask(vcpu->kvm) && + !VALID_PAGE(mmu->private_root_hpa)) { + root =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu, true); + mmu->private_root_hpa =3D root; + } + root =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu, false); mmu->root.hpa =3D root; } else if (shadow_root_level >=3D PT64_ROOT_4LEVEL) { root =3D mmu_alloc_root(vcpu, 0, 0, shadow_root_level); @@ -4651,7 +4656,7 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct = kvm_page_fault *fault) if (shadow_memtype_mask && kvm_arch_has_noncoherent_dma(vcpu->kvm)) { for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level) { int page_num =3D KVM_PAGES_PER_HPAGE(fault->max_level); - gfn_t base =3D gfn_round_for_level(fault->gfn, + gfn_t base =3D gfn_round_for_level(gpa_to_gfn(fault->addr), fault->max_level); =20 if (kvm_mtrr_check_gfn_range_consistency(vcpu, base, page_num)) @@ -6138,6 +6143,7 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, st= ruct kvm_mmu *mmu) =20 mmu->root.hpa =3D INVALID_PAGE; mmu->root.pgd =3D 0; + mmu->private_root_hpa =3D INVALID_PAGE; for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) mmu->prev_roots[i] =3D KVM_MMU_ROOT_INFO_INVALID; =20 @@ -7206,6 +7212,9 @@ int kvm_mmu_vendor_module_init(void) void kvm_mmu_destroy(struct kvm_vcpu *vcpu) { kvm_mmu_unload(vcpu); + if (tdp_mmu_enabled) + mmu_free_root_page(vcpu->kvm, &vcpu->arch.mmu->private_root_hpa, + NULL); free_mmu_pages(&vcpu->arch.root_mmu); free_mmu_pages(&vcpu->arch.guest_mmu); mmu_free_memory_caches(vcpu); diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index 17ad9df1bb71..d65324d87a17 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -6,6 +6,8 @@ #include #include =20 +#include "mmu.h" + #undef MMU_DEBUG =20 #ifdef MMU_DEBUG @@ -204,6 +206,15 @@ static inline void kvm_mmu_free_private_spt(struct kvm= _mmu_page *sp) if (sp->private_spt) free_page((unsigned long)sp->private_spt); } + +static inline gfn_t kvm_gfn_for_root(struct kvm *kvm, struct kvm_mmu_page = *root, + gfn_t gfn) +{ + if (is_private_sp(root)) + return kvm_gfn_to_private(kvm, gfn); + else + return kvm_gfn_to_shared(kvm, gfn); +} #else static inline void *kvm_mmu_private_spt(struct kvm_mmu_page *sp) { @@ -221,6 +232,12 @@ static inline void kvm_mmu_alloc_private_spt(struct kv= m_vcpu *vcpu, struct kvm_m static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp) { } + +static inline gfn_t kvm_gfn_for_root(struct kvm *kvm, struct kvm_mmu_page = *root, + gfn_t gfn) +{ + return gfn; +} #endif =20 static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page = *sp) @@ -386,7 +403,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu= *vcpu, gpa_t cr2_or_gpa, int r; =20 if (vcpu->arch.mmu->root_role.direct) { - fault.gfn =3D fault.addr >> PAGE_SHIFT; + fault.gfn =3D gpa_to_gfn(fault.addr) & ~kvm_gfn_shared_mask(vcpu->kvm); fault.slot =3D kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn); } =20 diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index e1e40e3f5eb7..a9c9cd0db20a 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -91,7 +91,7 @@ struct tdp_iter { tdp_ptep_t pt_path[PT64_ROOT_MAX_LEVEL]; /* A pointer to the current SPTE */ tdp_ptep_t sptep; - /* The lowest GFN mapped by the current SPTE */ + /* The lowest GFN (shared bits included) mapped by the current SPTE */ gfn_t gfn; /* The level of the root page given to the iterator */ int root_level; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d3788a414551..95ba78944712 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -237,6 +237,9 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm= _vcpu *vcpu, sp->spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); sp->role =3D role; =20 + if (kvm_mmu_page_role_is_private(role)) + kvm_mmu_alloc_private_spt(vcpu, sp); + return sp; } =20 @@ -259,7 +262,8 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, td= p_ptep_t sptep, trace_kvm_mmu_get_page(sp, true); } =20 -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) +static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcp= u, + bool private) { union kvm_mmu_page_role role =3D vcpu->arch.mmu->root_role; struct kvm *kvm =3D vcpu->kvm; @@ -271,6 +275,8 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vc= pu) * Check for an existing root before allocating a new one. Note, the * role check prevents consuming an invalid root. */ + if (private) + kvm_mmu_page_role_set_private(&role); for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { if (root->role.word =3D=3D role.word && kvm_tdp_mmu_get_root(root)) @@ -294,12 +300,17 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *= vcpu) spin_unlock(&kvm->arch.tdp_mmu_pages_lock); =20 out: - return __pa(root->spt); + return root; +} + +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private) +{ + return __pa(kvm_tdp_mmu_get_vcpu_root(vcpu, private)->spt); } =20 static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level, - bool shared); + u64 old_spte, u64 new_spte, + union kvm_mmu_page_role role, bool shared); =20 static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) { @@ -436,12 +447,78 @@ static void handle_removed_pt(struct kvm *kvm, tdp_pt= ep_t pt, bool shared) REMOVED_SPTE, level); } handle_changed_spte(kvm, kvm_mmu_page_as_id(sp), gfn, - old_spte, REMOVED_SPTE, level, shared); + old_spte, REMOVED_SPTE, sp->role, + shared); + } + + if (is_private_sp(sp) && + WARN_ON(static_call(kvm_x86_free_private_spt)(kvm, sp->gfn, sp->role.= level, + kvm_mmu_private_spt(sp)))) { + /* + * Failed to unlink Secure EPT page and there is nothing to do + * further. Intentionally leak the page to prevent the kernel + * from accessing the encrypted page. + */ + kvm_mmu_init_private_spt(sp, NULL); } =20 call_rcu(&sp->rcu_head, tdp_mmu_free_sp_rcu_callback); } =20 +static void *get_private_spt(gfn_t gfn, u64 new_spte, int level) +{ + if (is_shadow_present_pte(new_spte) && !is_last_spte(new_spte, level)) { + struct kvm_mmu_page *sp =3D to_shadow_page(pfn_to_hpa(spte_to_pfn(new_sp= te))); + void *private_spt =3D kvm_mmu_private_spt(sp); + + WARN_ON_ONCE(!private_spt); + WARN_ON_ONCE(sp->role.level + 1 !=3D level); + WARN_ON_ONCE(sp->gfn !=3D gfn); + return private_spt; + } + + return NULL; +} + +static void handle_removed_private_spte(struct kvm *kvm, gfn_t gfn, + u64 old_spte, u64 new_spte, + int level) +{ + bool was_present =3D is_shadow_present_pte(old_spte); + bool is_present =3D is_shadow_present_pte(new_spte); + bool was_leaf =3D was_present && is_last_spte(old_spte, level); + bool is_leaf =3D is_present && is_last_spte(new_spte, level); + kvm_pfn_t old_pfn =3D spte_to_pfn(old_spte); + kvm_pfn_t new_pfn =3D spte_to_pfn(new_spte); + int ret; + + /* Ignore change of software only bits. e.g. host_writable */ + if (was_leaf =3D=3D is_leaf && was_present =3D=3D is_present) + return; + + /* + * Allow only leaf page to be zapped. Reclaim Non-leaf page tables at + * destroying VM. + */ + WARN_ON_ONCE(is_present); + if (!was_leaf) + return; + + /* non-present -> non-present doesn't make sense. */ + KVM_BUG_ON(!was_present, kvm); + KVM_BUG_ON(new_pfn, kvm); + + /* Zapping leaf spte is allowed only when write lock is held. */ + lockdep_assert_held_write(&kvm->mmu_lock); + ret =3D static_call(kvm_x86_zap_private_spte)(kvm, gfn, level); + /* Because write lock is held, operation should success. */ + if (KVM_BUG_ON(ret, kvm)) + return; + + ret =3D static_call(kvm_x86_remove_private_spte)(kvm, gfn, level, old_pfn= ); + KVM_BUG_ON(ret, kvm); +} + /** * handle_changed_spte - handle bookkeeping associated with an SPTE change * @kvm: kvm instance @@ -449,7 +526,7 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep= _t pt, bool shared) * @gfn: the base GFN that was mapped by the SPTE * @old_spte: The value of the SPTE before the change * @new_spte: The value of the SPTE after the change - * @level: the level of the PT the SPTE is part of in the paging structure + * @role: the role of the PT the SPTE is part of in the paging structure * @shared: This operation may not be running under the exclusive use of * the MMU lock and the operation must synchronize with other * threads that might be modifying SPTEs. @@ -459,14 +536,18 @@ static void handle_removed_pt(struct kvm *kvm, tdp_pt= ep_t pt, bool shared) * and fast_pf_fix_direct_spte()). */ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, - u64 old_spte, u64 new_spte, int level, - bool shared) + u64 old_spte, u64 new_spte, + union kvm_mmu_page_role role, bool shared) { + bool is_private =3D kvm_mmu_page_role_is_private(role); + int level =3D role.level; bool was_present =3D is_shadow_present_pte(old_spte); bool is_present =3D is_shadow_present_pte(new_spte); bool was_leaf =3D was_present && is_last_spte(old_spte, level); bool is_leaf =3D is_present && is_last_spte(new_spte, level); - bool pfn_changed =3D spte_to_pfn(old_spte) !=3D spte_to_pfn(new_spte); + kvm_pfn_t old_pfn =3D spte_to_pfn(old_spte); + kvm_pfn_t new_pfn =3D spte_to_pfn(new_spte); + bool pfn_changed =3D old_pfn !=3D new_pfn; =20 WARN_ON(level > PT64_ROOT_MAX_LEVEL); WARN_ON(level < PG_LEVEL_4K); @@ -533,7 +614,7 @@ static void handle_changed_spte(struct kvm *kvm, int as= _id, gfn_t gfn, =20 if (was_leaf && is_dirty_spte(old_spte) && (!is_present || !is_dirty_spte(new_spte) || pfn_changed)) - kvm_set_pfn_dirty(spte_to_pfn(old_spte)); + kvm_set_pfn_dirty(old_pfn); =20 /* * Recursively handle child PTs if the change removed a subtree from @@ -542,14 +623,82 @@ static void handle_changed_spte(struct kvm *kvm, int = as_id, gfn_t gfn, * pages are kernel allocations and should never be migrated. */ if (was_present && !was_leaf && - (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed))) + (is_leaf || !is_present || WARN_ON_ONCE(pfn_changed))) { + KVM_BUG_ON(is_private !=3D is_private_sptep(spte_to_child_pt(old_spte, l= evel)), + kvm); handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared); + } + + /* + * Secure-EPT requires to remove Secure-EPT tables after removing + * children. hooks after handling lower page table by above + * handle_remove_pt(). + */ + if (is_private && !is_present) + handle_removed_private_spte(kvm, gfn, old_spte, new_spte, role.level); =20 if (was_leaf && is_accessed_spte(old_spte) && (!is_present || !is_accessed_spte(new_spte) || pfn_changed)) kvm_set_pfn_accessed(spte_to_pfn(old_spte)); } =20 +static int __must_check __set_private_spte_present(struct kvm *kvm, tdp_pt= ep_t sptep, + gfn_t gfn, u64 old_spte, + u64 new_spte, int level) +{ + bool was_present =3D is_shadow_present_pte(old_spte); + bool is_present =3D is_shadow_present_pte(new_spte); + bool is_leaf =3D is_present && is_last_spte(new_spte, level); + kvm_pfn_t new_pfn =3D spte_to_pfn(new_spte); + int ret =3D 0; + + lockdep_assert_held(&kvm->mmu_lock); + /* TDP MMU doesn't change present -> present */ + KVM_BUG_ON(was_present, kvm); + + /* + * Use different call to either set up middle level + * private page table, or leaf. + */ + if (is_leaf) + ret =3D static_call(kvm_x86_set_private_spte)(kvm, gfn, level, new_pfn); + else { + void *private_spt =3D get_private_spt(gfn, new_spte, level); + + KVM_BUG_ON(!private_spt, kvm); + ret =3D static_call(kvm_x86_link_private_spt)(kvm, gfn, level, private_s= pt); + } + + return ret; +} + +static int __must_check set_private_spte_present(struct kvm *kvm, tdp_ptep= _t sptep, + gfn_t gfn, u64 old_spte, + u64 new_spte, int level) +{ + int ret; + + /* + * For private page table, callbacks are needed to propagate SPTE + * change into the protected page table. In order to atomically update + * both the SPTE and the protected page tables with callbacks, utilize + * freezing SPTE. + * - Freeze the SPTE. Set entry to REMOVED_SPTE. + * - Trigger callbacks for protected page tables. + * - Unfreeze the SPTE. Set the entry to new_spte. + */ + lockdep_assert_held(&kvm->mmu_lock); + if (!try_cmpxchg64(sptep, &old_spte, REMOVED_SPTE)) + return -EBUSY; + + ret =3D __set_private_spte_present(kvm, sptep, gfn, old_spte, new_spte, l= evel); + if (ret) + __kvm_tdp_mmu_write_spte(sptep, old_spte); + else + __kvm_tdp_mmu_write_spte(sptep, new_spte); + return ret; +} + /* * tdp_mmu_set_spte_atomic - Set a TDP MMU SPTE atomically * and handle the associated bookkeeping. Do not mark the page dirty @@ -572,6 +721,7 @@ static inline int __must_check tdp_mmu_set_spte_atomic(= struct kvm *kvm, u64 new_spte) { u64 *sptep =3D rcu_dereference(iter->sptep); + bool freezed =3D false; =20 /* * The caller is responsible for ensuring the old SPTE is not a REMOVED @@ -583,19 +733,36 @@ static inline int __must_check tdp_mmu_set_spte_atomi= c(struct kvm *kvm, =20 lockdep_assert_held_read(&kvm->mmu_lock); =20 - /* - * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and - * does not hold the mmu_lock. On failure, i.e. if a different logical - * CPU modified the SPTE, try_cmpxchg64() updates iter->old_spte with - * the current value, so the caller operates on fresh data, e.g. if it - * retries tdp_mmu_set_spte_atomic() - */ - if (!try_cmpxchg64(sptep, &iter->old_spte, new_spte)) - return -EBUSY; + if (is_private_sptep(iter->sptep) && !is_removed_spte(new_spte)) { + int ret; =20 - handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, - new_spte, iter->level, true); + if (is_shadow_present_pte(new_spte)) { + ret =3D set_private_spte_present(kvm, iter->sptep, iter->gfn, + iter->old_spte, new_spte, iter->level); + if (ret) + return ret; + } else { + if (!try_cmpxchg64(sptep, &iter->old_spte, REMOVED_SPTE)) + return -EBUSY; + freezed =3D true; + } + } else { + /* + * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs + * and does not hold the mmu_lock. On failure, i.e. if a + * different logical CPU modified the SPTE, try_cmpxchg64() + * updates iter->old_spte with the current value, so the caller + * operates on fresh data, e.g. if it retries + * tdp_mmu_set_spte_atomic() + */ + if (!try_cmpxchg64(sptep, &iter->old_spte, new_spte)) + return -EBUSY; + } =20 + handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, + new_spte, sptep_to_sp(sptep)->role, true); + if (freezed) + __kvm_tdp_mmu_write_spte(sptep, new_spte); return 0; } =20 @@ -645,6 +812,8 @@ static inline int __must_check tdp_mmu_zap_spte_atomic(= struct kvm *kvm, static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, u64 old_spte, u64 new_spte, gfn_t gfn, int level) { + union kvm_mmu_page_role role; + lockdep_assert_held_write(&kvm->mmu_lock); =20 /* @@ -657,8 +826,17 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id= , tdp_ptep_t sptep, WARN_ON(is_removed_spte(old_spte) || is_removed_spte(new_spte)); =20 old_spte =3D kvm_tdp_mmu_write_spte(sptep, old_spte, new_spte, level); + if (is_private_sptep(sptep) && !is_removed_spte(new_spte) && + is_shadow_present_pte(new_spte)) { + lockdep_assert_held_write(&kvm->mmu_lock); + /* Because write spin lock is held, no race. It should success. */ + KVM_BUG_ON(__set_private_spte_present(kvm, sptep, gfn, old_spte, + new_spte, level), kvm); + } =20 - handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); + role =3D sptep_to_sp(sptep)->role; + role.level =3D level; + handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, role, false); return old_spte; } =20 @@ -681,8 +859,11 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *k= vm, struct tdp_iter *iter, continue; \ else =20 -#define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, to_shadow_page(_mmu->root.hpa), _start, _end) +#define tdp_mmu_for_each_pte(_iter, _mmu, _private, _start, _end) \ + for_each_tdp_pte(_iter, \ + to_shadow_page((_private) ? _mmu->private_root_hpa : \ + _mmu->root.hpa), \ + _start, _end) =20 /* * Yield if the MMU lock is contended or this thread needs to return contr= ol @@ -844,6 +1025,14 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct= kvm_mmu_page *root, if (!zap_private && is_private_sp(root)) return false; =20 + /* + * start and end doesn't have GFN shared bit. This function zaps + * a region including alias. Adjust shared bit of [start, end) if the + * root is shared. + */ + start =3D kvm_gfn_for_root(kvm, root, start); + end =3D kvm_gfn_for_root(kvm, root, end); + rcu_read_lock(); =20 for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) { @@ -991,10 +1180,19 @@ static int tdp_mmu_map_handle_target_level(struct kv= m_vcpu *vcpu, =20 if (unlikely(!fault->slot)) new_spte =3D make_mmio_spte(vcpu, iter->gfn, ACC_ALL); - else - wrprot =3D make_spte(vcpu, sp, fault->slot, ACC_ALL, iter->gfn, - fault->pfn, iter->old_spte, fault->prefetch, true, - fault->map_writable, &new_spte); + else { + unsigned long pte_access =3D ACC_ALL; + + /* TDX shared GPAs are no executable, enforce this for the SDV. */ + if (kvm_gfn_shared_mask(vcpu->kvm) && !fault->is_private) + pte_access &=3D ~ACC_EXEC_MASK; + + wrprot =3D make_spte(vcpu, sp, fault->slot, pte_access, + gpa_to_gfn(fault->addr)/* include shared bit */, + fault->pfn, iter->old_spte, + fault->prefetch, true, fault->map_writable, + &new_spte); + } =20 if (new_spte =3D=3D iter->old_spte) ret =3D RET_PF_SPURIOUS; @@ -1072,6 +1270,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm= _page_fault *fault) struct kvm *kvm =3D vcpu->kvm; struct tdp_iter iter; struct kvm_mmu_page *sp; + gfn_t raw_gfn; + bool is_private =3D fault->is_private && kvm_gfn_shared_mask(kvm); int ret =3D RET_PF_RETRY; =20 kvm_mmu_hugepage_adjust(vcpu, fault); @@ -1080,7 +1280,17 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kv= m_page_fault *fault) =20 rcu_read_lock(); =20 - tdp_mmu_for_each_pte(iter, mmu, fault->gfn, fault->gfn + 1) { + raw_gfn =3D gpa_to_gfn(fault->addr); + + if (is_error_noslot_pfn(fault->pfn) || + !kvm_pfn_to_refcounted_page(fault->pfn)) { + if (is_private) { + rcu_read_unlock(); + return -EFAULT; + } + } + + tdp_mmu_for_each_pte(iter, mmu, is_private, raw_gfn, raw_gfn + 1) { int r; =20 if (fault->nx_huge_page_workaround_enabled) @@ -1110,9 +1320,14 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kv= m_page_fault *fault) =20 sp->nx_huge_page_disallowed =3D fault->huge_page_disallowed; =20 - if (is_shadow_present_pte(iter.old_spte)) + if (is_shadow_present_pte(iter.old_spte)) { + /* + * TODO: large page support. + * Doesn't support large page for TDX now + */ + KVM_BUG_ON(is_private_sptep(iter.sptep), vcpu->kvm); r =3D tdp_mmu_split_huge_page(kvm, &iter, sp, true); - else + } else r =3D tdp_mmu_link_sp(kvm, &iter, sp, true); =20 /* @@ -1369,6 +1584,8 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_sp= lit(gfp_t gfp, union kvm_mm =20 sp->role =3D role; sp->spt =3D (void *)__get_free_page(gfp); + /* TODO: large page support for private GPA. */ + WARN_ON_ONCE(kvm_mmu_page_role_is_private(role)); if (!sp->spt) { kmem_cache_free(mmu_page_header_cache, sp); return NULL; @@ -1384,6 +1601,11 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_spl= it(struct kvm *kvm, union kvm_mmu_page_role role =3D tdp_iter_child_role(iter); struct kvm_mmu_page *sp; =20 + KVM_BUG_ON(kvm_mmu_page_role_is_private(role) !=3D + is_private_sptep(iter->sptep), kvm); + /* TODO: Large page isn't supported for private SPTE yet. */ + KVM_BUG_ON(kvm_mmu_page_role_is_private(role), kvm); + /* * Since we are allocating while under the MMU lock we have to be * careful about GFP flags. Use GFP_NOWAIT to avoid blocking on direct @@ -1808,7 +2030,7 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 a= ddr, u64 *sptes, =20 *root_level =3D vcpu->arch.mmu->root_role.level; =20 - tdp_mmu_for_each_pte(iter, mmu, gfn, gfn + 1) { + tdp_mmu_for_each_pte(iter, mmu, false, gfn, gfn + 1) { leaf =3D iter.level; sptes[leaf] =3D iter.old_spte; } @@ -1835,7 +2057,10 @@ u64 *kvm_tdp_mmu_fast_pf_get_last_sptep(struct kvm_v= cpu *vcpu, u64 addr, gfn_t gfn =3D addr >> PAGE_SHIFT; tdp_ptep_t sptep =3D NULL; =20 - tdp_mmu_for_each_pte(iter, mmu, gfn, gfn + 1) { + /* fast page fault for private GPA isn't supported. */ + WARN_ON_ONCE(kvm_is_private_gpa(vcpu->kvm, addr)); + + tdp_mmu_for_each_pte(iter, mmu, false, gfn, gfn + 1) { *spte =3D iter.old_spte; sptep =3D iter.sptep; } diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3df604352648..6ae311b5e988 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -10,7 +10,7 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm); void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm); =20 -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private); =20 __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *= root) { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 14b1fa9fe644..0c277e1f5f12 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -203,6 +203,7 @@ struct page *kvm_pfn_to_refcounted_page(kvm_pfn_t pfn) =20 return NULL; } +EXPORT_SYMBOL_GPL(kvm_pfn_to_refcounted_page); =20 /* * Switches to specified vcpu, until a matching vcpu_put() --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 273E4EB64DD for ; Tue, 25 Jul 2023 22:21:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231560AbjGYWVM (ORCPT ); Tue, 25 Jul 2023 18:21:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232340AbjGYWT4 (ORCPT ); Tue, 25 Jul 2023 18:19:56 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B1A049C7; Tue, 25 Jul 2023 15:16:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323417; x=1721859417; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VWeErb8XXD2SBqMy8L7JRBcxtOCdpKU+Rt6/TTjVA8Q=; b=YG9ZFiQwdtkwZxzByuA2R3hhyVJoshcRn+xOl2mK88WSQCYJW+f/yEZ6 mlMaxb06Wd1CPxXTwW4+Ql4a9HtzGv6r36bnphVW5uI02D3tD9rtR+w7s FkidBdODQPzhQX2LFlL1KNXNxYF7ULTXrRRfT6ir9WFGLmbIN47GZArtS GraB77uDlFdOiV6hcMIyWN6+yqqnjBCsrkkw4AWJCy1/PyVAfoLPFtTLd kcbSw8AeKwKfZ6tQvXSGU+c0Ly+2X4zL3Z2Lg4B7iVtxbjlZrAku4SmF7 ykqohesKYnx0egXxZU2QpAbXa48vYTRnfgrz9tXHpsk6E4tIpcBTfBR+P A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863286" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863286" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938941" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938941" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:39 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 047/115] [MARKER] The start of TDX KVM patch series: TDX EPT violation Date: Tue, 25 Jul 2023 15:13:58 -0700 Message-Id: <79ccdd541412746308432432baede77a635dfa44.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TDX EPT violation. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index 7903473abad1..c4d67dd9ddf8 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -20,11 +20,11 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Not yet +* TDX EPT violation: Applying * TD finalization: Not yet * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet =20 * KVM MMU GPA shared bits: Applied * KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applying +* KVM TDP MMU hooks: Applied --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD01AEB64DD for ; Tue, 25 Jul 2023 22:21:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231419AbjGYWVH (ORCPT ); Tue, 25 Jul 2023 18:21:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232336AbjGYWT4 (ORCPT ); Tue, 25 Jul 2023 18:19:56 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8050A49C2; Tue, 25 Jul 2023 15:16:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323416; x=1721859416; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fDJTlB5bV3N6Gz25H1kwqBta18u/BfAiuqVVM3y2CYw=; b=QdM6EST1W/fHG47pnIcCSQKzD5tL/6Vanr/2RnvrMo/aEYSOPVizVVjQ pj4DmctmGSv6rRZ4eay7m37HyjGU9Es2zZ42Lx+YQjJzAIkFykM50A9rB MbtR2sxr/K1G8DwMWEX2FuRMOqXT38zG4vYP3QTfcfMnMD+PcCyo66PWI XAjqerJyQXGGm+CTbMUt1cx6zRcWABVHJ+G3i/KlNfTkHqQQH9eA9JZhb kHkg37vvATpD+Tb6qC9I2mOPVc6rKKxyA8F3UBp4egnmI7s3eFItK0xJQ 0GTTz1ALDLAhjg6QIXcEYqhYaHfsHmn/rErmD4jMfXSWNVtSX+q7pS9FN w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863290" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863290" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938944" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938944" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:39 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Yan Zhao , Yuan Yao Subject: [PATCH v15 048/115] KVM: x86/mmu: TDX: Do not enable page track for TD guest Date: Tue, 25 Jul 2023 15:13:59 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yan Zhao TDX does not support write protection and hence page track. Though !tdp_enabled and kvm_shadow_root_allocated(kvm) are always false for TD guest, should also return false when external write tracking is enabled. Cc: Yuan Yao Signed-off-by: Yan Zhao Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/page_track.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index 0a2ac438d647..571c2c40004a 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -22,6 +22,9 @@ =20 bool kvm_page_track_write_tracking_enabled(struct kvm *kvm) { + if (kvm->arch.vm_type =3D=3D KVM_X86_TDX_VM) + return false; + return IS_ENABLED(CONFIG_KVM_EXTERNAL_WRITE_TRACKING) || !tdp_enabled || kvm_shadow_root_allocated(kvm); } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C015C0015E for ; Tue, 25 Jul 2023 22:21:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232596AbjGYWVk (ORCPT ); Tue, 25 Jul 2023 18:21:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232502AbjGYWUD (ORCPT ); Tue, 25 Jul 2023 18:20:03 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DB4049D3; Tue, 25 Jul 2023 15:17:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323426; x=1721859426; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bIvss2mZaMxkdM4kifawdK2jvwpOiCxUFAgl88Gf7Bc=; b=R66V2WtgvH4LWqv01e1tsgbEHQyIUMuYKtTd8EUxE/+gF1q0GFUyWZtP WhlmQfirTjNzSmwDFQ/f/1IpGWFpJLwd42V6Rn4KpHHSQ1YPqZh4K7NIv NP+94YqC/AsdW9IB4fK0C26oSlJEf73HDJOqgwcXBHeCP3CjC/dptne4g 3YuhanMoyRIyRmV5f0N40R+DEIlPaEnKa30Da+Nf/tZLOmxY0LQet4dur pPE85eIVzniM+4QDn7olX256CossQM7KT54tMMCv8ATTcufiYz+XaB1bq DWETeCCM8Y1qARAYEadnsNsApMI7Fg6B1HZzFgvz1cvowHCledbNwGFKc g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863293" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863293" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938947" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938947" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:40 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 049/115] KVM: VMX: Split out guts of EPT violation to common/exposed function Date: Tue, 25 Jul 2023 15:14:00 -0700 Message-Id: <1a2e839aeabc0663551504c93a5513bf09f0e953.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson The difference of TDX EPT violation is how to retrieve information, GPA, and exit qualification. To share the code to handle EPT violation, split out the guts of EPT violation handler so that VMX/TDX exit handler can call it after retrieving GPA and exit qualification. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini Reviewed-by: Kai Huang --- arch/x86/kvm/vmx/common.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 25 +++---------------------- 2 files changed, 36 insertions(+), 22 deletions(-) create mode 100644 arch/x86/kvm/vmx/common.h diff --git a/arch/x86/kvm/vmx/common.h b/arch/x86/kvm/vmx/common.h new file mode 100644 index 000000000000..235908f3e044 --- /dev/null +++ b/arch/x86/kvm/vmx/common.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_X86_VMX_COMMON_H +#define __KVM_X86_VMX_COMMON_H + +#include + +#include "mmu.h" + +static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t = gpa, + unsigned long exit_qualification) +{ + u64 error_code; + + /* Is it a read fault? */ + error_code =3D (exit_qualification & EPT_VIOLATION_ACC_READ) + ? PFERR_USER_MASK : 0; + /* Is it a write fault? */ + error_code |=3D (exit_qualification & EPT_VIOLATION_ACC_WRITE) + ? PFERR_WRITE_MASK : 0; + /* Is it a fetch fault? */ + error_code |=3D (exit_qualification & EPT_VIOLATION_ACC_INSTR) + ? PFERR_FETCH_MASK : 0; + /* ept page table entry is present? */ + error_code |=3D (exit_qualification & EPT_VIOLATION_RWX_MASK) + ? PFERR_PRESENT_MASK : 0; + + error_code |=3D (exit_qualification & EPT_VIOLATION_GVA_TRANSLATED) !=3D = 0 ? + PFERR_GUEST_FINAL_MASK : PFERR_GUEST_PAGE_MASK; + + return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); +} + +#endif /* __KVM_X86_VMX_COMMON_H */ diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c9020e751f69..408c155f8566 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -51,6 +51,7 @@ #include =20 #include "capabilities.h" +#include "common.h" #include "cpuid.h" #include "hyperv.h" #include "kvm_onhyperv.h" @@ -5752,11 +5753,8 @@ static int handle_task_switch(struct kvm_vcpu *vcpu) =20 static int handle_ept_violation(struct kvm_vcpu *vcpu) { - unsigned long exit_qualification; + unsigned long exit_qualification =3D vmx_get_exit_qual(vcpu); gpa_t gpa; - u64 error_code; - - exit_qualification =3D vmx_get_exit_qual(vcpu); =20 /* * EPT violation happened while executing iret from NMI, @@ -5771,23 +5769,6 @@ static int handle_ept_violation(struct kvm_vcpu *vcp= u) =20 gpa =3D vmcs_read64(GUEST_PHYSICAL_ADDRESS); trace_kvm_page_fault(vcpu, gpa, exit_qualification); - - /* Is it a read fault? */ - error_code =3D (exit_qualification & EPT_VIOLATION_ACC_READ) - ? PFERR_USER_MASK : 0; - /* Is it a write fault? */ - error_code |=3D (exit_qualification & EPT_VIOLATION_ACC_WRITE) - ? PFERR_WRITE_MASK : 0; - /* Is it a fetch fault? */ - error_code |=3D (exit_qualification & EPT_VIOLATION_ACC_INSTR) - ? PFERR_FETCH_MASK : 0; - /* ept page table entry is present? */ - error_code |=3D (exit_qualification & EPT_VIOLATION_RWX_MASK) - ? PFERR_PRESENT_MASK : 0; - - error_code |=3D (exit_qualification & EPT_VIOLATION_GVA_TRANSLATED) !=3D = 0 ? - PFERR_GUEST_FINAL_MASK : PFERR_GUEST_PAGE_MASK; - vcpu->arch.exit_qualification =3D exit_qualification; =20 /* @@ -5801,7 +5782,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) if (unlikely(allow_smaller_maxphyaddr && kvm_vcpu_is_illegal_gpa(vcpu, gp= a))) return kvm_emulate_instruction(vcpu, 0); =20 - return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); + return __vmx_handle_ept_violation(vcpu, gpa, exit_qualification); } =20 static int handle_ept_misconfig(struct kvm_vcpu *vcpu) --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71D06C001DF for ; Tue, 25 Jul 2023 22:21:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232584AbjGYWVg (ORCPT ); Tue, 25 Jul 2023 18:21:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232503AbjGYWUD (ORCPT ); Tue, 25 Jul 2023 18:20:03 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9B1649D7; Tue, 25 Jul 2023 15:17:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323427; x=1721859427; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=S7tfUygsfsO9iaqRrMz1ajmnxRELIU1u0wIlIO3T+fQ=; b=Gnpq/jCnkUhdX/V4Dw7YygLlIpA/Kz4S7PN/loEV9lDoyUkmxahKGzTz zfpFc/Q+JfUGjb+kdA4KbGzmh6lB6BdiVZ+4PQD5DEQ5zJI0W8gq2caGO y0HPPVrPgJRu87+n2KjyJFSRS1ZV46E3fGnakz07AA12mQB5Uo59OLme/ Y1WKZXF7VoC1yF9IAfksQ/QbzIlzOzygHk/j8ordaX0CYALono5qPfFPs ZZnfFbOfIHxvpAuPWbWBVvs6HEsS+o+DyGQfzCrLpgbup2gYBhYaW6OJH 7JYbFw8RD9cdjGNWTI1WzsFhQCldLFMIRlYfopGo2rtYy9oxCypJaylAy g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863297" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863297" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938950" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938950" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:40 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 050/115] KVM: VMX: Move setting of EPT MMU masks to common VT-x code Date: Tue, 25 Jul 2023 15:14:01 -0700 Message-Id: <1e5d9f4a8566a5de16422b826ca6c647d43015bd.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson EPT MMU masks are used commonly for VMX and TDX. The value needs to be initialized in common code before both VMX/TDX-specific initialization code. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 4 ---- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 8bb38db4323d..59a53a8cc475 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -4,6 +4,7 @@ #include "x86_ops.h" #include "vmx.h" #include "nested.h" +#include "mmu.h" #include "pmu.h" #include "tdx.h" #include "tdx_arch.h" @@ -50,6 +51,14 @@ static __init int vt_hardware_setup(void) if (ret) return ret; =20 + /* + * As kvm_mmu_set_ept_masks() updates enable_mmio_caching, call it + * before checking enable_mmio_caching. + */ + if (enable_ept) + kvm_mmu_set_ept_masks(enable_ept_ad_bits, + cpu_has_vmx_ept_execute_only()); + enable_tdx =3D enable_tdx && !tdx_hardware_setup(&vt_x86_ops); =20 return 0; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 408c155f8566..26a762df2c23 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -8373,10 +8373,6 @@ __init int vmx_hardware_setup(void) =20 set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */ =20 - if (enable_ept) - kvm_mmu_set_ept_masks(enable_ept_ad_bits, - cpu_has_vmx_ept_execute_only()); - /* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID * bits to shadow_zero_check. --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F5DBEB64DD for ; Tue, 25 Jul 2023 22:21:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232605AbjGYWVo (ORCPT ); Tue, 25 Jul 2023 18:21:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232505AbjGYWUD (ORCPT ); Tue, 25 Jul 2023 18:20:03 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9CC149D8; Tue, 25 Jul 2023 15:17:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323427; x=1721859427; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iVI+/47YGQn0kb5cDtHDjuy5q76Jn46TOv3yVMjSY4M=; b=S6gGtQLoH9UMyt6GJ63wb6A6Ps0SaofZ4Imyh0S8g06xdYYxeNrfZ59x HRcYbasIn/HDRZ1FP9tXjXdqSzb6MiYzdMB2lPP2/ngOl8xkeq9j3fvK+ W53Wu6rne3oKmoq+tNvqMAuzemssQMZ7xiSFJYNIoRVS3RSB2LpV9OFMa UplCjB0m3rI/z/d9bG7jOCg1Sx2mBIdzPorBb39KGdV4MMMUIwFINBT9b WKqVeMXxevQj8GZnmkOU2ylc5uG7XaVK5jsBMpaQJN1bCsgswJ+iGNeAG a2mknJlHFM2Evefj5Ad+u7OW0+OLOoHY/N2NZ7kiI4sexkry5/3dUC5lE g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863305" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863305" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938954" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938954" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:40 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 051/115] KVM: TDX: Add accessors VMX VMCS helpers Date: Tue, 25 Jul 2023 15:14:02 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX defines SEAMCALL APIs to access TDX control structures corresponding to the VMX VMCS. Introduce helper accessors to hide its SEAMCALL ABI details. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.h | 95 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index c39d866e0653..a0faa9942714 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -57,6 +57,101 @@ static inline struct vcpu_tdx *to_tdx(struct kvm_vcpu *= vcpu) return container_of(vcpu, struct vcpu_tdx, vcpu); } =20 +static __always_inline void tdvps_vmcs_check(u32 field, u8 bits) +{ +#define VMCS_ENC_ACCESS_TYPE_MASK 0x1UL +#define VMCS_ENC_ACCESS_TYPE_FULL 0x0UL +#define VMCS_ENC_ACCESS_TYPE_HIGH 0x1UL +#define VMCS_ENC_ACCESS_TYPE(field) ((field) & VMCS_ENC_ACCESS_TYPE_MASK) + + /* TDX is 64bit only. HIGH field isn't supported. */ + BUILD_BUG_ON_MSG(__builtin_constant_p(field) && + VMCS_ENC_ACCESS_TYPE(field) =3D=3D VMCS_ENC_ACCESS_TYPE_HIGH, + "Read/Write to TD VMCS *_HIGH fields not supported"); + + BUILD_BUG_ON(bits !=3D 16 && bits !=3D 32 && bits !=3D 64); + +#define VMCS_ENC_WIDTH_MASK GENMASK(14, 13) +#define VMCS_ENC_WIDTH_16BIT (0UL << 13) +#define VMCS_ENC_WIDTH_64BIT (1UL << 13) +#define VMCS_ENC_WIDTH_32BIT (2UL << 13) +#define VMCS_ENC_WIDTH_NATURAL (3UL << 13) +#define VMCS_ENC_WIDTH(field) ((field) & VMCS_ENC_WIDTH_MASK) + + /* TDX is 64bit only. i.e. natural width =3D 64bit. */ + BUILD_BUG_ON_MSG(bits !=3D 64 && __builtin_constant_p(field) && + (VMCS_ENC_WIDTH(field) =3D=3D VMCS_ENC_WIDTH_64BIT || + VMCS_ENC_WIDTH(field) =3D=3D VMCS_ENC_WIDTH_NATURAL), + "Invalid TD VMCS access for 64-bit field"); + BUILD_BUG_ON_MSG(bits !=3D 32 && __builtin_constant_p(field) && + VMCS_ENC_WIDTH(field) =3D=3D VMCS_ENC_WIDTH_32BIT, + "Invalid TD VMCS access for 32-bit field"); + BUILD_BUG_ON_MSG(bits !=3D 16 && __builtin_constant_p(field) && + VMCS_ENC_WIDTH(field) =3D=3D VMCS_ENC_WIDTH_16BIT, + "Invalid TD VMCS access for 16-bit field"); +} + +static __always_inline void tdvps_state_non_arch_check(u64 field, u8 bits)= {} +static __always_inline void tdvps_management_check(u64 field, u8 bits) {} + +#define TDX_BUILD_TDVPS_ACCESSORS(bits, uclass, lclass) \ +static __always_inline u##bits td_##lclass##_read##bits(struct vcpu_tdx *t= dx, \ + u32 field) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err =3D tdh_vp_rd(tdx->tdvpr_pa, TDVPS_##uclass(field), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) { \ + pr_err("TDH_VP_RD["#uclass".0x%x] failed: 0x%llx\n", \ + field, err); \ + return 0; \ + } \ + return (u##bits)out.r8; \ +} \ +static __always_inline void td_##lclass##_write##bits(struct vcpu_tdx *tdx= , \ + u32 field, u##bits val) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err =3D tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), val, \ + GENMASK_ULL(bits - 1, 0), &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] =3D 0x%llx failed: 0x%llx\n", \ + field, (u64)val, err); \ +} \ +static __always_inline void td_##lclass##_setbit##bits(struct vcpu_tdx *td= x, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err =3D tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), bit, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] |=3D 0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} \ +static __always_inline void td_##lclass##_clearbit##bits(struct vcpu_tdx *= tdx, \ + u32 field, u64 bit) \ +{ \ + struct tdx_module_output out; \ + u64 err; \ + \ + tdvps_##lclass##_check(field, bits); \ + err =3D tdh_vp_wr(tdx->tdvpr_pa, TDVPS_##uclass(field), 0, bit, &out); \ + if (KVM_BUG_ON(err, tdx->vcpu.kvm)) \ + pr_err("TDH_VP_WR["#uclass".0x%x] &=3D ~0x%llx failed: 0x%llx\n", \ + field, bit, err); \ +} + +TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs); +TDX_BUILD_TDVPS_ACCESSORS(32, VMCS, vmcs); +TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u3= 2 field) { struct tdx_module_output out; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 137EAC0015E for ; Tue, 25 Jul 2023 22:22:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232712AbjGYWWX (ORCPT ); Tue, 25 Jul 2023 18:22:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232236AbjGYWV3 (ORCPT ); Tue, 25 Jul 2023 18:21:29 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33D804C0A; Tue, 25 Jul 2023 15:17:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323441; x=1721859441; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=B4b+/3mQAJMlRyxqeOqx7XGaj976+2I6aQYaqoaptX8=; b=JSf6xUxnH+O4NorgUPUjdzs3Qe00AjqlJ++H3LHdkAPSFV57iguBele8 nf/YVLhr3YrItfuwDP9TsONYsvRYTKfqYhjPhZzqEYJqQoUxd2lSED1YZ 6tku2fgEt0J7Ur+sq5dfx9EDcwph7T+ONDyI1ODKH7fbrxHmU/dID2CIu IbDr4D0bjFccX6nkEKX772kWp75/lzv+Xe9/HncIKsWf8U1WbVeub8PTO ZM1hS1XY/6dm8/SlwYK/szo1jHq5svq8xx+ybdfywtk13Be88byYSBTt5 59WZDnxH5J2tllYNI8GyYEdAiZnmKgwmkzf5sjyUgUN1/PUcdoXcZHZCe w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863312" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863312" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938957" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938957" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:41 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 052/115] KVM: TDX: Add load_mmu_pgd method for TDX Date: Tue, 25 Jul 2023 15:14:03 -0700 Message-Id: <650688863a77b11af9a0903bd7a3b40b0d4b984d.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson For virtual IO, the guest TD shares guest pages with VMM without encryption. Shared EPT is used to map guest pages in unprotected way. Add the VMCS field encoding for the shared EPTP, which will be used by TDX to have separate EPT walks for private GPAs (existing EPTP) versus shared GPAs (new shared EPTP). Set shared EPT pointer value for the TDX guest to initialize TDX MMU. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/include/asm/vmx.h | 1 + arch/x86/kvm/vmx/main.c | 13 ++++++++++++- arch/x86/kvm/vmx/tdx.c | 5 +++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 56e192797742..cba8c9690abb 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -236,6 +236,7 @@ enum vmcs_field { TSC_MULTIPLIER_HIGH =3D 0x00002033, TERTIARY_VM_EXEC_CONTROL =3D 0x00002034, TERTIARY_VM_EXEC_CONTROL_HIGH =3D 0x00002035, + SHARED_EPT_POINTER =3D 0x0000203C, PID_POINTER_TABLE =3D 0x00002042, PID_POINTER_TABLE_HIGH =3D 0x00002043, GUEST_PHYSICAL_ADDRESS =3D 0x00002400, diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 59a53a8cc475..c4cf88987b00 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -143,6 +143,17 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool = init_event) vmx_vcpu_reset(vcpu, init_event); } =20 +static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, + int pgd_level) +{ + if (is_td_vcpu(vcpu)) { + tdx_load_mmu_pgd(vcpu, root_hpa, pgd_level); + return; + } + + vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level); +} + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -274,7 +285,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .write_tsc_offset =3D vmx_write_tsc_offset, .write_tsc_multiplier =3D vmx_write_tsc_multiplier, =20 - .load_mmu_pgd =3D vmx_load_mmu_pgd, + .load_mmu_pgd =3D vt_load_mmu_pgd, =20 .check_intercept =3D vmx_check_intercept, .handle_exit_irqoff =3D vmx_handle_exit_irqoff, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index a10caf87e4fb..f0d138cbe507 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -401,6 +401,11 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_e= vent) */ } =20 +void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) +{ + td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 8a7e256b44ac..258bafec576a 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -153,6 +153,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); + +void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_leve= l); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline void tdx_hardware_unsetup(void) {} @@ -174,6 +176,8 @@ static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu)= {} static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) = {} =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } + +static inline void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,= int root_level) {} #endif =20 #endif /* __KVM_X86_VMX_X86_OPS_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A38EFC001E0 for ; Tue, 25 Jul 2023 22:22:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232578AbjGYWW2 (ORCPT ); Tue, 25 Jul 2023 18:22:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232562AbjGYWVa (ORCPT ); Tue, 25 Jul 2023 18:21:30 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 346EC2D79; Tue, 25 Jul 2023 15:17:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323443; x=1721859443; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HSYqTrf4S+axzOfvfShXkPansWqRKfM/KCR4xdDb8/g=; b=TuPusrrGOvEKf0jLrmeRwzPWAJS30qpg8O3qXLcmRPs4BjwZF7wadHQf v7hBEuBhF+bzb4o5DZXKSimVxQEQ5FYUGx+198H+KOCilKjeQDBmIvg1i p6CrrJNcXCXnbvBVlB+RdB6PPJD7EDXuPdrDAhxaFQ3eTNqrHB64UnhkK txEbKh7Qdj5G/h+tbz7FscSzdfpje3Lk6QaqGdYF++FrFD3q4yvZItXqX MdcRxiff0yp7oplR5a6qgnRaFNiLisYmVFmQ+TBJ5q1wT/rx0WKKUUyTu x72nibrTCeEFNdKTCWxapdm+/oF+mEZvsbKcpGYERr59PEb3bqT88peXJ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863316" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863316" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938960" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938960" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:41 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Yuan Yao Subject: [PATCH v15 053/115] KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT Date: Tue, 25 Jul 2023 15:14:04 -0700 Message-Id: <56def5772292451d8dd002c388c07ff42c3b4414.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yuan Yao TDX module internally uses locks to protect internal resources. It tries to acquire the locks. If it fails to obtain the lock, it returns TDX_OPERAND_BUSY error without spin because its execution time limitation. TDX SEAMCALL API reference describes what resources are used. It's known which TDX SEAMCALL can cause contention with which resources. VMM can avoid contention inside the TDX module by avoiding contentious TDX SEAMCALL with, for example, spinlock. Because OS knows better its process scheduling and its scalability, a lock at OS/VMM layer would work better than simply retrying TDX SEAMCALLs. TDH.MEM.* API except for TDH.MEM.TRACK operates on a secure EPT tree and the TDX module internally tries to acquire the lock of the secure EPT tree. They return TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT in case of failure to get the lock. TDX KVM allows sept callbacks to return error so that TDP MMU layer can retry. TDH.VP.ENTER is an exception with zero-step attack mitigation. Normally TDH.VP.ENTER uses only TD vcpu resources and it doesn't cause contention. When a zero-step attack is suspected, it obtains a secure EPT tree lock and tracks the GPAs causing a secure EPT fault. Thus TDG.VP.ENTER may result in TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT. Also TDH.MEM.* SEAMCALLs may result in TDX_OPERAN_BUSY | TDX_OPERAND_ID_SEPT. Retry TDX TDH.MEM.* API and TDH.VP.ENTER on the error because the error is a rare event caused by zero-step attack mitigation and spinlock can not be used for TDH.VP.ENTER due to indefinite time execution. Signed-off-by: Yuan Yao Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx_ops.h | 42 ++++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx_ops.h b/arch/x86/kvm/vmx/tdx_ops.h index 9db19c0711a9..c7819abd61b0 100644 --- a/arch/x86/kvm/vmx/tdx_ops.h +++ b/arch/x86/kvm/vmx/tdx_ops.h @@ -46,6 +46,36 @@ static inline u64 tdx_seamcall(u64 op, u64 rcx, u64 rdx,= u64 r8, u64 r9, void pr_tdx_error(u64 op, u64 error_code, const struct tdx_module_output *= out); #endif =20 +/* + * TDX module acquires its internal lock for resources. It doesn't spin t= o get + * locks because of its restrictions of allowed execution time. Instead, = it + * returns TDX_OPERAND_BUSY with an operand id. + * + * Multiple VCPUs can operate on SEPT. Also with zero-step attack mitigat= ion, + * TDH.VP.ENTER may rarely acquire SEPT lock and release it when zero-step + * attack is suspected. It results in TDX_OPERAND_BUSY | TDX_OPERAND_ID_S= EPT + * with TDH.MEM.* operation. Note: TDH.MEM.TRACK is an exception. + * + * Because TDP MMU uses read lock for scalability, spin lock around SEAMCA= LL + * spoils TDP MMU effort. Retry several times with the assumption that SE= PT + * lock contention is rare. But don't loop forever to avoid lockup. Let = TDP + * MMU retry. + */ +#define TDX_ERROR_SEPT_BUSY (TDX_OPERAND_BUSY | TDX_OPERAND_ID_SEPT) + +static inline u64 tdx_seamcall_sept(u64 op, u64 rcx, u64 rdx, u64 r8, u64 = r9, + struct tdx_module_output *out) +{ +#define SEAMCALL_RETRY_MAX 16 + int retry =3D SEAMCALL_RETRY_MAX; + u64 ret; + + do { + ret =3D tdx_seamcall(op, rcx, rdx, r8, r9, out); + } while (ret =3D=3D TDX_ERROR_SEPT_BUSY && retry-- > 0); + return ret; +} + static inline u64 tdh_mng_addcx(hpa_t tdr, hpa_t addr) { clflush_cache_range(__va(addr), PAGE_SIZE); @@ -56,14 +86,14 @@ static inline u64 tdh_mem_page_add(hpa_t tdr, gpa_t gpa= , hpa_t hpa, hpa_t source struct tdx_module_output *out) { clflush_cache_range(__va(hpa), PAGE_SIZE); - return tdx_seamcall(TDH_MEM_PAGE_ADD, gpa, tdr, hpa, source, out); + return tdx_seamcall_sept(TDH_MEM_PAGE_ADD, gpa, tdr, hpa, source, out); } =20 static inline u64 tdh_mem_sept_add(hpa_t tdr, gpa_t gpa, int level, hpa_t = page, struct tdx_module_output *out) { clflush_cache_range(__va(page), PAGE_SIZE); - return tdx_seamcall(TDH_MEM_SEPT_ADD, gpa | level, tdr, page, 0, out); + return tdx_seamcall_sept(TDH_MEM_SEPT_ADD, gpa | level, tdr, page, 0, out= ); } =20 static inline u64 tdh_mem_sept_remove(hpa_t tdr, gpa_t gpa, int level, @@ -89,13 +119,13 @@ static inline u64 tdh_mem_page_aug(hpa_t tdr, gpa_t gp= a, hpa_t hpa, struct tdx_module_output *out) { clflush_cache_range(__va(hpa), PAGE_SIZE); - return tdx_seamcall(TDH_MEM_PAGE_AUG, gpa, tdr, hpa, 0, out); + return tdx_seamcall_sept(TDH_MEM_PAGE_AUG, gpa, tdr, hpa, 0, out); } =20 static inline u64 tdh_mem_range_block(hpa_t tdr, gpa_t gpa, int level, struct tdx_module_output *out) { - return tdx_seamcall(TDH_MEM_RANGE_BLOCK, gpa | level, tdr, 0, 0, out); + return tdx_seamcall_sept(TDH_MEM_RANGE_BLOCK, gpa | level, tdr, 0, 0, out= ); } =20 static inline u64 tdh_mng_key_config(hpa_t tdr) @@ -177,7 +207,7 @@ static inline u64 tdh_phymem_page_reclaim(hpa_t page, static inline u64 tdh_mem_page_remove(hpa_t tdr, gpa_t gpa, int level, struct tdx_module_output *out) { - return tdx_seamcall(TDH_MEM_PAGE_REMOVE, gpa | level, tdr, 0, 0, out); + return tdx_seamcall_sept(TDH_MEM_PAGE_REMOVE, gpa | level, tdr, 0, 0, out= ); } =20 static inline u64 tdh_sys_lp_shutdown(void) @@ -193,7 +223,7 @@ static inline u64 tdh_mem_track(hpa_t tdr) static inline u64 tdh_mem_range_unblock(hpa_t tdr, gpa_t gpa, int level, struct tdx_module_output *out) { - return tdx_seamcall(TDH_MEM_RANGE_UNBLOCK, gpa | level, tdr, 0, 0, out); + return tdx_seamcall_sept(TDH_MEM_RANGE_UNBLOCK, gpa | level, tdr, 0, 0, o= ut); } =20 static inline u64 tdh_phymem_cache_wb(bool resume) --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C02FEB64DD for ; Tue, 25 Jul 2023 22:22:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232744AbjGYWWj (ORCPT ); Tue, 25 Jul 2023 18:22:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232470AbjGYWV3 (ORCPT ); Tue, 25 Jul 2023 18:21:29 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2AA2270B; Tue, 25 Jul 2023 15:17:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323442; x=1721859442; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+Yn2JKEs6mv69w6nkl5UeCBiJCVQ70hp7WXmBsy85ZM=; b=nkbBxHtZjBYEACKsaMyWZnvERmYuwOsrMHNgswwKGsNQ2m456wyPNCon 0/nnuo0zIIqCPy4EBk3clny89hSvxDWqmC3HZlti4jVUgRtWjDhtsX8tP qa/CJrBUnnyiE9KNTDvILh/gIfYNkaH0EV504YmzZdLviP0Cz+oYX18Jr bPKutG/ZW1o6pA1mmWC73VKxt17Kg+A1V/ekCz7SFUMbx750ex1PfAv4T 6Pk8Y9a45TIs+A70X+1fPaRdGBQ75z5/zsJWmgrQs9SsN5uUcIkwezLH3 kmoRJEhM8TJJDOc7DecpukAy5dfWrwV0oSJgDyhsi/hjBduV1V6j4KfIx A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863322" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863322" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938963" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938963" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:42 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 054/115] KVM: TDX: Require TDP MMU and mmio caching for TDX Date: Tue, 25 Jul 2023 15:14:05 -0700 Message-Id: <3f9a65ef68973a718569230e958f631777ba0a0d.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata As TDP MMU is becoming main stream than the legacy MMU, the legacy MMU support for TDX isn't implemented. TDX requires KVM mmio caching. Disable TDX support when TDP MMU or mmio caching aren't supported. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/main.c | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 5b48ac4a5fbc..4e9343e759f6 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -103,6 +103,7 @@ module_param_named(flush_on_reuse, force_flush_and_sync= _on_reuse, bool, 0644); * If the hardware supports that we don't need to do shadow paging. */ bool tdp_enabled =3D false; +EXPORT_SYMBOL_GPL(tdp_enabled); =20 static bool __ro_after_init tdp_mmu_allowed; =20 diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index c4cf88987b00..debb48f19cfa 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -58,6 +58,17 @@ static __init int vt_hardware_setup(void) if (enable_ept) kvm_mmu_set_ept_masks(enable_ept_ad_bits, cpu_has_vmx_ept_execute_only()); + /* TDX requires KVM TDP MMU. */ + if (enable_tdx && !tdp_enabled) { + enable_tdx =3D false; + pr_warn_ratelimited("TDX requires TDP MMU. Please enable TDP MMU for TD= X.\n"); + } + + /* TDX requires MMIO caching. */ + if (enable_tdx && !enable_mmio_caching) { + enable_tdx =3D false; + pr_warn_ratelimited("TDX requires mmio caching. Please enable mmio cach= ing for TDX.\n"); + } =20 enable_tdx =3D enable_tdx && !tdx_hardware_setup(&vt_x86_ops); =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05246C001DF for ; Tue, 25 Jul 2023 22:24:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232791AbjGYWWy (ORCPT ); Tue, 25 Jul 2023 18:22:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232539AbjGYWVx (ORCPT ); Tue, 25 Jul 2023 18:21:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A98204C0F; Tue, 25 Jul 2023 15:17:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323451; x=1721859451; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6m95jxADjppkl1Ji0AbiNCY0YbyH0v6Ub7BKNfjdcv0=; b=QT8rYp+v1B+OPc8BO7EVMEmuovj1WSuCr0Un/5+dn3uO20UNlsOpUpMx bjB6ig1oL6u0SpqoSd5ajQdzLsm8fP/uYGbJqSaaUwD5NlfDd3HCf3XJX Ia3hmzzp0l+TzEZ4JyZS9f/Mq/tO32PuMLbOBUWEdS6NPIHDF4UfVK0xE UnBMHzFmccpHHuY2YHTOPLl45wBXGBuZWf7bLr/ADyIjV3+wJNrNIrrwq yqO2BPL29gyS7+R6cn81r026WjiIPLLJdeoaIRJdBZkz9FLgfSYWT7Cgu f0POHaEoOaoIRwzNs5KequV7bdN0JTApR8MtQ8NRjyABafFu+cZDApD4B w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863327" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863327" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938966" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938966" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:42 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 055/115] KVM: TDX: TDP MMU TDX support Date: Tue, 25 Jul 2023 15:14:06 -0700 Message-Id: <5f9b2aeed5710582b06c8e926ade6b75d19d3b13.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Implement hooks of TDP MMU for TDX backend. TLB flush, TLB shootdown, propagating the change private EPT entry to Secure EPT and freeing Secure EPT page. TLB flush handles both shared EPT and private EPT. It flushes shared EPT same as VMX. It also waits for the TDX TLB shootdown. For the hook to free Secure EPT page, unlinks the Secure EPT page from the Secure EPT so that the page can be freed to OS. Propagate the entry change to Secure EPT. The possible entry changes are present -> non-present(zapping) and non-present -> present(population). On population just link the Secure EPT page or the private guest page to the Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent zapping/population, zapping requires synchronous TLB shoot down with the frozen EPT entry. It zaps the secure entry, increments TLB counter, sends IPI to remote vcpus to trigger TLB flush, and then unlinks the private guest page from the Secure EPT. For simplicity, batched zapping with exclude lock is handled as concurrent zapping. Although it's inefficient, it can be optimized in the future. For MMIO SPTE, the spte value changes as follows. initial value (suppress VE bit is set) -> Guest issues MMIO and triggers EPT violation -> KVM updates SPTE value to MMIO value (suppress VE bit is cleared) -> Guest MMIO resumes. It triggers VE exception in guest TD -> Guest VE handler issues TDG.VP.VMCALL -> KVM handles MMIO -> Guest VE handler resumes its execution after MMIO instruction Signed-off-by: Isaku Yamahata --- v14 -> v15: - Implemented tdx_flush_tlb_current() - Removed unnecessary invept in tdx_flush_tlb(). It was carry over from the very old code base. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/mmu/spte.c | 3 +- arch/x86/kvm/vmx/main.c | 71 +++++++- arch/x86/kvm/vmx/tdx.c | 325 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 7 + arch/x86/kvm/vmx/x86_ops.h | 6 + 5 files changed, 407 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index a1f332eb3b59..7c2311425ce4 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -74,7 +74,8 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsign= ed int access) u64 spte =3D generation_mmio_spte_mask(gen); u64 gpa =3D gfn << PAGE_SHIFT; =20 - WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value); + WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value && + !kvm_gfn_shared_mask(vcpu->kvm)); =20 access &=3D shadow_mmio_access_mask; spte |=3D vcpu->kvm->arch.shadow_mmio_value | access; diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index debb48f19cfa..5b499a71701b 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -28,6 +28,7 @@ static int vt_max_vcpus(struct kvm *kvm) =20 return kvm->max_vcpus; } +static int vt_flush_remote_tlbs(struct kvm *kvm); =20 static int vt_hardware_enable(void) { @@ -70,8 +71,22 @@ static __init int vt_hardware_setup(void) pr_warn_ratelimited("TDX requires mmio caching. Please enable mmio cach= ing for TDX.\n"); } =20 + /* + * TDX KVM overrides flush_remote_tlbs method and assumes + * flush_remote_tlbs_range =3D NULL that falls back to + * flush_remote_tlbs. Disable TDX if there are conflicts. + */ + if (vt_x86_ops.flush_remote_tlbs || + vt_x86_ops.flush_remote_tlbs_range) { + enable_tdx =3D false; + pr_warn_ratelimited("TDX requires baremetal. Not Supported on VMM guest.= \n"); + } + enable_tdx =3D enable_tdx && !tdx_hardware_setup(&vt_x86_ops); =20 + if (enable_tdx) + vt_x86_ops.flush_remote_tlbs =3D vt_flush_remote_tlbs; + return 0; } =20 @@ -154,6 +169,54 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool = init_event) vmx_vcpu_reset(vcpu, init_event); } =20 +static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_flush_tlb(vcpu); + return; + } + + vmx_flush_tlb_all(vcpu); +} + +static void vt_flush_tlb_current(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_flush_tlb_current(vcpu); + return; + } + + vmx_flush_tlb_current(vcpu); +} + +static int vt_flush_remote_tlbs(struct kvm *kvm) +{ + if (is_td(kvm)) + return tdx_sept_flush_remote_tlbs(kvm); + + /* + * fallback to KVM_REQ_TLB_FLUSH. + * See kvm_arch_flush_remote_tlb() and kvm_flush_remote_tlbs(). + */ + return -EOPNOTSUPP; +} + +static void vt_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t addr) +{ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return; + + vmx_flush_tlb_gva(vcpu, addr); +} + +static void vt_flush_tlb_guest(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_flush_tlb_guest(vcpu); +} + static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { @@ -244,10 +307,10 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .set_rflags =3D vmx_set_rflags, .get_if_flag =3D vmx_get_if_flag, =20 - .flush_tlb_all =3D vmx_flush_tlb_all, - .flush_tlb_current =3D vmx_flush_tlb_current, - .flush_tlb_gva =3D vmx_flush_tlb_gva, - .flush_tlb_guest =3D vmx_flush_tlb_guest, + .flush_tlb_all =3D vt_flush_tlb_all, + .flush_tlb_current =3D vt_flush_tlb_current, + .flush_tlb_gva =3D vt_flush_tlb_gva, + .flush_tlb_guest =3D vt_flush_tlb_guest, =20 .vcpu_pre_run =3D vmx_vcpu_pre_run, .vcpu_run =3D vmx_vcpu_run, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index f0d138cbe507..d543e78899f0 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -7,6 +7,7 @@ #include "x86_ops.h" #include "mmu.h" #include "tdx.h" +#include "vmx.h" #include "x86.h" =20 #undef pr_fmt @@ -312,6 +313,22 @@ static int tdx_do_tdh_mng_key_config(void *param) =20 int tdx_vm_init(struct kvm *kvm) { + /* + * Because guest TD is protected, VMM can't parse the instruction in TD. + * Instead, guest uses MMIO hypercall. For unmodified device driver, + * #VE needs to be injected for MMIO and #VE handler in TD converts MMIO + * instruction into MMIO hypercall. + * + * SPTE value for MMIO needs to be setup so that #VE is injected into + * TD instead of triggering EPT MISCONFIG. + * - RWX=3D0 so that EPT violation is triggered. + * - suppress #VE bit is cleared to inject #VE. + */ + kvm_mmu_set_mmio_spte_value(kvm, 0); + + /* TODO: Enable 2mb and 1gb large page support. */ + kvm->arch.tdp_max_page_level =3D PG_LEVEL_4K; + /* * This function initializes only KVM software construct. It doesn't * initialize TDX stuff, e.g. TDCS, TDR, TDCX, HKID etc. @@ -406,6 +423,266 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t ro= ot_hpa, int pgd_level) td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); } =20 +static void tdx_unpin(struct kvm *kvm, kvm_pfn_t pfn) +{ + struct page *page =3D pfn_to_page(pfn); + + put_page(page); +} + +static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + hpa_t hpa =3D pfn_to_hpa(pfn); + gpa_t gpa =3D gfn_to_gpa(gfn); + struct tdx_module_output out; + u64 err; + + /* TODO: handle large pages. */ + if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) + return -EINVAL; + + /* + * Because restricted mem doesn't support page migration with + * a_ops->migrate_page (yet), no callback isn't triggered for KVM on + * page migration. Until restricted mem supports page migration, + * prevent page migration. + * TODO: Once restricted mem introduces callback on page migration, + * implement it and remove get_page/put_page(). + */ + get_page(pfn_to_page(pfn)); + + if (likely(is_td_finalized(kvm_tdx))) { + err =3D tdh_mem_page_aug(kvm_tdx->tdr_pa, gpa, hpa, &out); + if (unlikely(err =3D=3D TDX_ERROR_SEPT_BUSY)) { + tdx_unpin(kvm, pfn); + return -EAGAIN; + } + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_PAGE_AUG, err, &out); + tdx_unpin(kvm, pfn); + return -EIO; + } + return 0; + } + + /* TODO: tdh_mem_page_add() comes here for the initial memory. */ + + return 0; +} + +static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) +{ + int tdx_level =3D pg_level_to_tdx_sept_level(level); + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + struct tdx_module_output out; + gpa_t gpa =3D gfn_to_gpa(gfn); + hpa_t hpa =3D pfn_to_hpa(pfn); + hpa_t hpa_with_hkid; + u64 err; + + /* TODO: handle large pages. */ + if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) + return -EINVAL; + + if (unlikely(!is_hkid_assigned(kvm_tdx))) { + /* + * The HKID assigned to this TD was already freed and cache + * was already flushed. We don't have to flush again. + */ + err =3D tdx_reclaim_page(hpa, false, 0); + if (KVM_BUG_ON(err, kvm)) + return -EIO; + tdx_unpin(kvm, pfn); + return 0; + } + + do { + /* + * When zapping private page, write lock is held. So no race + * condition with other vcpu sept operation. Race only with + * TDH.VP.ENTER. + */ + err =3D tdh_mem_page_remove(kvm_tdx->tdr_pa, gpa, tdx_level, &out); + } while (unlikely(err =3D=3D TDX_ERROR_SEPT_BUSY)); + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_PAGE_REMOVE, err, &out); + return -EIO; + } + + hpa_with_hkid =3D set_hkid_to_hpa(hpa, (u16)kvm_tdx->hkid); + do { + /* + * TDX_OPERAND_BUSY can happen on locking PAMT entry. Because + * this page was removed above, other thread shouldn't be + * repeatedly operating on this page. Just retry loop. + */ + err =3D tdh_phymem_page_wbinvd(hpa_with_hkid); + } while (unlikely(err =3D=3D (TDX_OPERAND_BUSY | TDX_OPERAND_ID_RCX))); + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err, NULL); + return -EIO; + } + tdx_clear_page(hpa); + tdx_unpin(kvm, pfn); + return 0; +} + +static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, + enum pg_level level, void *private_spt) +{ + int tdx_level =3D pg_level_to_tdx_sept_level(level); + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + gpa_t gpa =3D gfn_to_gpa(gfn); + hpa_t hpa =3D __pa(private_spt); + struct tdx_module_output out; + u64 err; + + err =3D tdh_mem_sept_add(kvm_tdx->tdr_pa, gpa, tdx_level, hpa, &out); + if (unlikely(err =3D=3D TDX_ERROR_SEPT_BUSY)) + return -EAGAIN; + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_SEPT_ADD, err, &out); + return -EIO; + } + + return 0; +} + +static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level) +{ + int tdx_level =3D pg_level_to_tdx_sept_level(level); + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + gpa_t gpa =3D gfn_to_gpa(gfn) & KVM_HPAGE_MASK(level); + struct tdx_module_output out; + u64 err; + + /* This can be called when destructing guest TD after freeing HKID. */ + if (unlikely(!is_hkid_assigned(kvm_tdx))) + return 0; + + /* For now large page isn't supported yet. */ + WARN_ON_ONCE(level !=3D PG_LEVEL_4K); + err =3D tdh_mem_range_block(kvm_tdx->tdr_pa, gpa, tdx_level, &out); + if (unlikely(err =3D=3D TDX_ERROR_SEPT_BUSY)) + return -EAGAIN; + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_RANGE_BLOCK, err, &out); + return -EIO; + } + return 0; +} + +/* + * TLB shoot down procedure: + * There is a global epoch counter and each vcpu has local epoch counter. + * - TDH.MEM.RANGE.BLOCK(TDR. level, range) on one vcpu + * This blocks the subsequenct creation of TLB translation on that range. + * This corresponds to clear the present bit(all RXW) in EPT entry + * - TDH.MEM.TRACK(TDR): advances the epoch counter which is global. + * - IPI to remote vcpus + * - TDExit and re-entry with TDH.VP.ENTER on remote vcpus + * - On re-entry, TDX module compares the local epoch counter with the glo= bal + * epoch counter. If the local epoch counter is older than the global e= poch + * counter, update the local epoch counter and flushes TLB. + */ +static void tdx_track(struct kvm_tdx *kvm_tdx) +{ + u64 err; + + KVM_BUG_ON(!is_hkid_assigned(kvm_tdx), &kvm_tdx->kvm); + /* If TD isn't finalized, it's before any vcpu running. */ + if (unlikely(!is_td_finalized(kvm_tdx))) + return; + + /* + * tdx_flush_tlb() waits for this function to issue TDH.MEM.TRACK() by + * the counter. The counter is used instead of bool because multiple + * TDH_MEM_TRACK() can be issued concurrently by multiple vcpus. + */ + atomic_inc(&kvm_tdx->tdh_mem_track); + /* + * KVM_REQ_TLB_FLUSH waits for the empty IPI handler, ack_flush(), with + * KVM_REQUEST_WAIT. + */ + kvm_make_all_cpus_request(&kvm_tdx->kvm, KVM_REQ_TLB_FLUSH); + + do { + /* + * kvm_flush_remote_tlbs() doesn't allow to return error and + * retry. + */ + err =3D tdh_mem_track(kvm_tdx->tdr_pa); + } while (unlikely((err & TDX_SEAMCALL_STATUS_MASK) =3D=3D TDX_OPERAND_BUS= Y)); + + /* Release remote vcpu waiting for TDH.MEM.TRACK in tdx_flush_tlb(). */ + atomic_dec(&kvm_tdx->tdh_mem_track); + + if (KVM_BUG_ON(err, &kvm_tdx->kvm)) + pr_tdx_error(TDH_MEM_TRACK, err, NULL); + +} + +static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, + enum pg_level level, void *private_spt) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + + /* + * The HKID assigned to this TD was already freed and cache was + * already flushed. We don't have to flush again. + */ + if (!is_hkid_assigned(kvm_tdx)) + return tdx_reclaim_page(__pa(private_spt), false, 0); + + /* + * free_private_spt() is (obviously) called when a shadow page is being + * zapped. KVM doesn't (yet) zap private SPs while the TD is active. + * Note: This function is for private shadow page. Not for private + * guest page. private guest page can be zapped during TD is active. + * shared <-> private conversion and slot move/deletion. + */ + KVM_BUG_ON(is_hkid_assigned(kvm_tdx), kvm); + return -EINVAL; +} + +int tdx_sept_flush_remote_tlbs(struct kvm *kvm) +{ + struct kvm_tdx *kvm_tdx; + + if (unlikely(!is_td(kvm))) + return -EOPNOTSUPP; + + kvm_tdx =3D to_kvm_tdx(kvm); + if (is_hkid_assigned(kvm_tdx)) + tdx_track(kvm_tdx); + + return 0; +} + +static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) +{ + /* + * TDX requires TLB tracking before dropping private page. Do + * it here, although it is also done later. + * If hkid isn't assigned, the guest is destroying and no vcpu + * runs further. TLB shootdown isn't needed. + * + * TODO: implement with_range version for optimization. + * kvm_flush_remote_tlbs_with_address(kvm, gfn, 1); + * =3D> tdx_sept_flush_remote_tlbs_range(kvm, gfn, + * KVM_PAGES_PER_HPAGE(level)); + */ + if (is_hkid_assigned(to_kvm_tdx(kvm))) + kvm_flush_remote_tlbs(kvm); + + return tdx_sept_drop_private_spte(kvm, gfn, level, pfn); +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; @@ -880,6 +1157,41 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_td= x_cmd *cmd) return ret; } =20 +void tdx_flush_tlb(struct kvm_vcpu *vcpu) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(vcpu->kvm); + + /* + * Don't need to flush shared EPTP: + * "TD VCPU TLB Address Spaced Identifier" in the TDX module spec: + * The TLB entries for TD are tagged with: + * SEAM (1 bit) + * VPID + * Secure EPT root (51:12 bits) with HKID =3D 0 + * PCID + * for *both* Secure-EPT and Shared-EPT. + * TLB flush with Secure-EPT root by tdx_track() results in flushing + * the conversion of both Secure-EPT and Shared-EPT. + */ + + /* + * See tdx_track(). Wait for tlb shootdown initiater to finish + * TDH_MEM_TRACK() so that shared-EPT/secure-EPT TLB is flushed + * on the next TDENTER. + */ + while (atomic_read(&kvm_tdx->tdh_mem_track)) + cpu_relax(); +} + +void tdx_flush_tlb_current(struct kvm_vcpu *vcpu) +{ + /* + * flush_tlb_current() is used only the first time for the vcpu to run. + * As it isn't performance critical, keep this function simple. + */ + tdx_track(to_kvm_tdx(vcpu->kvm)); +} + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { struct kvm_tdx_cmd tdx_cmd; @@ -1145,8 +1457,21 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x8= 6_ops) on_each_cpu(vmx_off, &vmx_tdx.vmx_enabled, true); cpus_read_unlock(); free_cpumask_var(vmx_tdx.vmx_enabled); + if (r) + goto out; + + x86_ops->link_private_spt =3D tdx_sept_link_private_spt; + x86_ops->free_private_spt =3D tdx_sept_free_private_spt; + x86_ops->set_private_spte =3D tdx_sept_set_private_spte; + x86_ops->remove_private_spte =3D tdx_sept_remove_private_spte; + x86_ops->zap_private_spte =3D tdx_sept_zap_private_spte; + + return 0; =20 out: + /* kfree() accepts NULL. */ + kfree(tdx_mng_key_config_lock); + tdx_mng_key_config_lock =3D NULL; return r; } =20 diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index a0faa9942714..6603da8708ad 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -18,6 +18,7 @@ struct kvm_tdx { int hkid; =20 bool finalized; + atomic_t tdh_mem_track; =20 u64 tsc_offset; }; @@ -165,6 +166,12 @@ static __always_inline u64 td_tdcs_exec_read64(struct = kvm_tdx *kvm_tdx, u32 fiel return out.r8; } =20 +static __always_inline int pg_level_to_tdx_sept_level(enum pg_level level) +{ + WARN_ON_ONCE(level =3D=3D PG_LEVEL_NONE); + return level - 1; +} + #else struct kvm_tdx { struct kvm kvm; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 258bafec576a..8c6b7df02df2 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -154,6 +154,9 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_ev= ent); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 +void tdx_flush_tlb(struct kvm_vcpu *vcpu); +void tdx_flush_tlb_current(struct kvm_vcpu *vcpu); +int tdx_sept_flush_remote_tlbs(struct kvm *kvm); void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int root_leve= l); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } @@ -177,6 +180,9 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu= , bool init_event) {} =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 +static inline void tdx_flush_tlb(struct kvm_vcpu *vcpu) {} +static inline void tdx_flush_tlb_current(struct kvm_vcpu *vcpu) {} +static inline int tdx_sept_flush_remote_tlbs(struct kvm *kvm) { return 0; } static inline void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,= int root_level) {} #endif =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2947BC41513 for ; Tue, 25 Jul 2023 22:22:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232768AbjGYWWr (ORCPT ); Tue, 25 Jul 2023 18:22:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232531AbjGYWVw (ORCPT ); Tue, 25 Jul 2023 18:21:52 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 161F44C10; Tue, 25 Jul 2023 15:17:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323452; x=1721859452; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RoRPfiwgJ6rjNJNRpk7JXwHtQZgo3QvdZiTbdTSMJwE=; b=KUGBjQH91al4xUHuPhnbNvcMBC/iBKPsMis7v/isFuSITab7bf+ikCg9 qJSOHyBUhqgEK3Gv4Uk14FPuSr7H9C0RpOj4vQS8LoVlw/lw2DpRoH8Sw 4ymEfS7obMUd3UYUg2Y1E/QN9p7Uozq4jkKnkvTwcLsrfT6QoYVgaHQyz DqOK6TtxyLb7tY1lQOtfg3LyTd6lyPxB94fssde7BplBQmQbZCeBgchqI NMHREByW+0foqhXH3dDEH0NfrRXwE5AdpVkWUjfCVVg9rYcLsbWSEKFRK KoWmhJQGOBlnQ6PI4CBVH5NJQ8EGBVZxcrZUbQHWmjTnpYvI9jvYEpbxc Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863330" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863330" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938969" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938969" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:42 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 056/115] KVM: TDX: MTRR: implement get_mt_mask() for TDX Date: Tue, 25 Jul 2023 15:14:07 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because TDX virtualize cpuid[0x1].EDX[MTRR: bit 12] to fixed 1, guest TD thinks MTRR is supported. Although TDX supports only WB for private GPA, it's desirable to support MTRR for shared GPA. As guest access to MTRR MSRs causes #VE and KVM/x86 tracks the values of MTRR MSRs, the remining part is to implement get_mt_mask method for TDX for shared GPA. Pass around shared bit from kvm fault handler to get_mt_mask method so that it can determine if the gfn is shared or private. Implement get_mt_mask() following vmx case for shared GPA and return WB for private GPA. the existing vmx_get_mt_mask() can't be directly used as CPU state(CR0.CD) is protected. GFN passed to kvm_mtrr_check_gfn_range_consistency() should include shared bit. Suggested-by: Kai Huang Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 10 +++++++++- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 5b499a71701b..2eaed14a9542 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -228,6 +228,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa= _t root_hpa, vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level); } =20 +static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) +{ + if (is_td_vcpu(vcpu)) + return tdx_get_mt_mask(vcpu, gfn, is_mmio); + + return vmx_get_mt_mask(vcpu, gfn, is_mmio); +} + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -346,7 +354,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .set_tss_addr =3D vmx_set_tss_addr, .set_identity_map_addr =3D vmx_set_identity_map_addr, - .get_mt_mask =3D vmx_get_mt_mask, + .get_mt_mask =3D vt_get_mt_mask, =20 .get_exit_info =3D vmx_get_exit_info, =20 diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index d543e78899f0..e367351f8d71 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -344,6 +344,29 @@ int tdx_vm_init(struct kvm *kvm) return 0; } =20 +u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) +{ + if (is_mmio) + return MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT; + + if (!kvm_arch_has_noncoherent_dma(vcpu->kvm)) + return (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; + + /* + * TDX enforces CR0.CD =3D 0 and KVM MTRR emulation enforces writeback. + * TODO: implement MTRR MSR emulation so that + * MTRRCap: SMRR=3D0: SMRR interface unsupported + * WC=3D0: write combining unsupported + * FIX=3D0: Fixed range registers unsupported + * VCNT=3D0: number of variable range regitsers =3D 0 + * MTRRDefType: E=3D1, FE=3D0, type=3Dwriteback only. Don't allow other v= alue. + * E=3D1: enable MTRR + * FE=3D0: disable fixed range MTRRs + * type: default memory type=3Dwriteback + */ + return MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT; +} + int tdx_vcpu_create(struct kvm_vcpu *vcpu) { /* diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 8c6b7df02df2..ed93accd29e6 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -151,6 +151,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); int tdx_vcpu_create(struct kvm_vcpu *vcpu); void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); +u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 @@ -177,6 +178,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __= user *argp) { return -EOP static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTS= UPP; } static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) = {} +static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3904C41513 for ; Tue, 25 Jul 2023 22:24:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232367AbjGYWWu (ORCPT ); Tue, 25 Jul 2023 18:22:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232538AbjGYWVx (ORCPT ); Tue, 25 Jul 2023 18:21:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A7E54C13; Tue, 25 Jul 2023 15:17:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323452; x=1721859452; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TZ+z91imsScQD0p1/VsSa1GqbET/iM4ZKqAoChx+Rzg=; b=lQGYEr3YZjBYd+SF5p4667Qd1kzbKVltuGj6HfYHh8bu8EGXK4w9uVCZ sxgw7snhwzIlPucE7KrhdcnSBj5wNxnHWkP4DTMBhHS05zS1EPUJDddNA 6XNPyZmPrYZVBdkeMv4gI6dZ1wIp6WmjxC2uxxCKgbcIVQXh9sU671bJs zMF4GV4RSBqkqdceDZ4ARruNcEeelf8Iaip8gTrY9WtZGDiQ40MbEqAgk fAzre1yDwWCd73ohRkvYVWswI8xv5YQICOYZKiQ+84xNW1l1rGawIW8uQ nR1so/I/YJO8K8W4OmrVBxDBUWQBiYdIM0HshzXBQRvqhTdde6FS9QxlU g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="357863354" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="357863354" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="1056938979" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="1056938979" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:48 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 057/115] [MARKER] The start of TDX KVM patch series: TD finalization Date: Tue, 25 Jul 2023 15:14:08 -0700 Message-Id: <7823fd77e8fda53cf5af956449222f5a04d96271.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TD finalization. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index c4d67dd9ddf8..46ae049b6b85 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -11,6 +11,7 @@ What qemu can do - TDX VM TYPE is exposed to Qemu. - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. +- Qemu can populate initial guest memory image. =20 Patch Layer status ------------------ @@ -20,8 +21,8 @@ Patch Layer status * TDX architectural definitions: Applied * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied -* TDX EPT violation: Applying -* TD finalization: Not yet +* TDX EPT violation: Applied +* TD finalization: Applying * TD vcpu enter/exit: Not yet * TD vcpu interrupts/exit/hypercall: Not yet =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FF11C04FE0 for ; Tue, 25 Jul 2023 22:18:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231621AbjGYWRl (ORCPT ); Tue, 25 Jul 2023 18:17:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232235AbjGYWQO (ORCPT ); Tue, 25 Jul 2023 18:16:14 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB43430EA; Tue, 25 Jul 2023 15:15:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323351; x=1721859351; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Wvj0sNH9JtePwrau64elH2rCIjd14dw+JkGIDo/IbFM=; b=Qyx0cowZ3xtSX3QO0zhDTKnQotr9aki1CNjId3JEKoUUMArM0RO65jf+ YXI3LG6HarqQDw8YE6P0cBOdPzSGDhyW/XM9whv88l2ydLtNF4e7bHHxi tgWeikVh9QwStZDpWt88uuUMjKKZcfofRBGiEMmVoO+a4OvqbaSyHF78v D2VW0p6QfYbpomBQjU7uZ4Ie6kmHsC0W5eEiYtepse7ID48B0nBC+Tp+P ZpCevIqyt3MDYsf6beMikhHnCl9mCqTtlkbXlN9xTuChK4CoOBpXvQ10/ gbYwWfJxgBQeSPp1z3BmFgvxeItknJ9rrQJoxOuB73SExwPCLKj5eNmQr g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882511" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882511" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001771" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001771" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:49 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 058/115] KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX Date: Tue, 25 Jul 2023 15:14:09 -0700 Message-Id: <6a4c029af70d41b63bcee3d6a1f0c2377f6eb4bd.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Introduce a helper to directly (pun intended) fault-in a TDP page without having to go through the full page fault path. This allows TDX to get the resulting pfn and also allows the RET_PF_* enums to stay in mmu.c where they belong. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- v14 -> v15: - Remove loop in kvm_mmu_map_tdp_page() and return error code based on RET_FP_xxx value to avoid potential infinite loop. The caller should loop on -EAGAIN instead now. --- arch/x86/kvm/mmu.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 58 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 801e3d6b572d..1bca16217da3 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -174,6 +174,9 @@ static inline void kvm_mmu_refresh_passthrough_bits(str= uct kvm_vcpu *vcpu, __kvm_mmu_refresh_passthrough_bits(vcpu, mmu); } =20 +int kvm_mmu_map_tdp_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, + int max_level); + /* * Check if a given access (described through the I/D, W/R and U/S bits of= a * page fault error code pfec) causes a permission fault with the given PTE diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4e9343e759f6..7ef66d8a785b 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4673,6 +4673,64 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct= kvm_page_fault *fault) return direct_page_fault(vcpu, fault); } =20 +int kvm_mmu_map_tdp_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, + int max_level) +{ + int r; + struct kvm_page_fault fault =3D (struct kvm_page_fault) { + .addr =3D gpa, + .error_code =3D error_code, + .exec =3D error_code & PFERR_FETCH_MASK, + .write =3D error_code & PFERR_WRITE_MASK, + .present =3D error_code & PFERR_PRESENT_MASK, + .rsvd =3D error_code & PFERR_RSVD_MASK, + .user =3D error_code & PFERR_USER_MASK, + .prefetch =3D false, + .is_tdp =3D true, + .is_private =3D error_code & PFERR_GUEST_ENC_MASK, + .nx_huge_page_workaround_enabled =3D is_nx_huge_page_enabled(vcpu->kvm), + }; + + WARN_ON_ONCE(!vcpu->arch.mmu->root_role.direct); + fault.gfn =3D gpa_to_gfn(fault.addr) & ~kvm_gfn_shared_mask(vcpu->kvm); + fault.slot =3D kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn); + + r =3D mmu_topup_memory_caches(vcpu, false); + if (r) + return r; + + fault.max_level =3D max_level; + fault.req_level =3D PG_LEVEL_4K; + fault.goal_level =3D PG_LEVEL_4K; + +#ifdef CONFIG_X86_64 + if (tdp_mmu_enabled) + r =3D kvm_tdp_mmu_page_fault(vcpu, &fault); + else +#endif + r =3D direct_page_fault(vcpu, &fault); + + if (is_error_noslot_pfn(fault.pfn) || vcpu->kvm->vm_bugged) + return -EFAULT; + + switch (r) { + case RET_PF_RETRY: + return -EAGAIN; + + case RET_PF_FIXED: + case RET_PF_SPURIOUS: + return 0; + + case RET_PF_CONTINUE: + case RET_PF_EMULATE: + case RET_PF_INVALID: + case RET_PF_USER: + default: + return -EIO; + } +} +EXPORT_SYMBOL_GPL(kvm_mmu_map_tdp_page); + static void nonpaging_init_context(struct kvm_mmu *context) { context->page_fault =3D nonpaging_page_fault; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59B5EC001DF for ; Tue, 25 Jul 2023 22:18:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231646AbjGYWSD (ORCPT ); Tue, 25 Jul 2023 18:18:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231530AbjGYWQW (ORCPT ); Tue, 25 Jul 2023 18:16:22 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85ED4358A; Tue, 25 Jul 2023 15:15:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323353; x=1721859353; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hm+qc6OZKTLiS/cWtmLpuCb6TFeXiIf6J0MfW03XlpY=; b=nYUBWN3d52viTbYAdGpjDKg7qgwLyRhvWR3/3GGVBBGsJfxwTen+Bift 3rzkfJ0mhllgq5ddCmjf589hoJy83Ev6vK8iaQxBmDmxYtPwAQ8vL+3Jq QSvOppd0bEtpCbv0fU7hVm6fH2gv4JDP/QdCgYp+F++dpKIO/hEGYNBU+ pkDnXZ0fxu3m1CJsYiMGSUY13D7dXvmS/3bpEOLoOqAdOosG1eV6AUoHW wnBBlKAGSwdDjKQXVTNmJdqpHIOVsaW1ABTeSR0+ic1HvwcGCu6uwl35Y y8Sbf4pqkOwC9E6NIjpYd7GVXnYeWm1sKHj1CMICcntA7spxcykGkEBHK w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882516" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882516" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001777" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001777" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:49 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 059/115] KVM: TDX: Create initial guest memory Date: Tue, 25 Jul 2023 15:14:10 -0700 Message-Id: <19a589ab40b01c10c3b9addc5c38f3fe64b15ad0.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because the guest memory is protected in TDX, the creation of the initial guest memory requires a dedicated TDX module API, tdh_mem_page_add, instead of directly copying the memory contents into the guest memory in the case of the default VM type. KVM MMU page fault handler callback, private_page_add, handles it. Define new subcommand, KVM_TDX_INIT_MEM_REGION, of VM-scoped KVM_MEMORY_ENCRYPT_OP. It assigns the guest page, copies the initial memory contents into the guest memory, encrypts the guest memory. At the same time, optionally it extends memory measurement of the TDX guest. It calls the KVM MMU page fault(EPT-violation) handler to trigger the callbacks for it. Signed-off-by: Isaku Yamahata Reported-by: gkirkpatrick@google.com --- v14 -> v15: - add a check if TD is finalized or not to tdx_init_mem_region() - return -EAGAIN when partial population Signed-off-by: Isaku Yamahata --- arch/x86/include/uapi/asm/kvm.h | 9 ++ arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/vmx/tdx.c | 158 +++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 2 + tools/arch/x86/include/uapi/asm/kvm.h | 9 ++ 5 files changed, 174 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index 311a7894b712..a1815fcbb0be 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -572,6 +572,7 @@ enum kvm_tdx_cmd_id { KVM_TDX_CAPABILITIES =3D 0, KVM_TDX_INIT_VM, KVM_TDX_INIT_VCPU, + KVM_TDX_INIT_MEM_REGION, =20 KVM_TDX_CMD_NR_MAX, }; @@ -645,4 +646,12 @@ struct kvm_tdx_init_vm { struct kvm_cpuid2 cpuid; }; =20 +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { + __u64 source_addr; + __u64 gpa; + __u64 nr_pages; +}; + #endif /* _ASM_X86_KVM_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 7ef66d8a785b..0d218d930d0a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5704,6 +5704,7 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out: return r; } +EXPORT_SYMBOL(kvm_mmu_load); =20 void kvm_mmu_unload(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index e367351f8d71..32e84c29d35e 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -446,6 +446,21 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t roo= t_hpa, int pgd_level) td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); } =20 +static void tdx_measure_page(struct kvm_tdx *kvm_tdx, hpa_t gpa) +{ + struct tdx_module_output out; + u64 err; + int i; + + for (i =3D 0; i < PAGE_SIZE; i +=3D TDX_EXTENDMR_CHUNKSIZE) { + err =3D tdh_mr_extend(kvm_tdx->tdr_pa, gpa + i, &out); + if (KVM_BUG_ON(err, &kvm_tdx->kvm)) { + pr_tdx_error(TDH_MR_EXTEND, err, &out); + break; + } + } +} + static void tdx_unpin(struct kvm *kvm, kvm_pfn_t pfn) { struct page *page =3D pfn_to_page(pfn); @@ -460,12 +475,10 @@ static int tdx_sept_set_private_spte(struct kvm *kvm,= gfn_t gfn, hpa_t hpa =3D pfn_to_hpa(pfn); gpa_t gpa =3D gfn_to_gpa(gfn); struct tdx_module_output out; + hpa_t source_pa; + bool measure; u64 err; =20 - /* TODO: handle large pages. */ - if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) - return -EINVAL; - /* * Because restricted mem doesn't support page migration with * a_ops->migrate_page (yet), no callback isn't triggered for KVM on @@ -476,7 +489,12 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, = gfn_t gfn, */ get_page(pfn_to_page(pfn)); =20 + /* Build-time faults are induced and handled via TDH_MEM_PAGE_ADD. */ if (likely(is_td_finalized(kvm_tdx))) { + /* TODO: handle large pages. */ + if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) + return -EINVAL; + err =3D tdh_mem_page_aug(kvm_tdx->tdr_pa, gpa, hpa, &out); if (unlikely(err =3D=3D TDX_ERROR_SEPT_BUSY)) { tdx_unpin(kvm, pfn); @@ -490,7 +508,45 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, = gfn_t gfn, return 0; } =20 - /* TODO: tdh_mem_page_add() comes here for the initial memory. */ + /* + * KVM_INIT_MEM_REGION, tdx_init_mem_region(), supports only 4K page + * because tdh_mem_page_add() supports only 4K page. + */ + if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) + return -EINVAL; + + /* + * In case of TDP MMU, fault handler can run concurrently. Note + * 'source_pa' is a TD scope variable, meaning if there are multiple + * threads reaching here with all needing to access 'source_pa', it + * will break. However fortunately this won't happen, because below + * TDH_MEM_PAGE_ADD code path is only used when VM is being created + * before it is running, using KVM_TDX_INIT_MEM_REGION ioctl (which + * always uses vcpu 0's page table and protected by vcpu->mutex). + */ + if (KVM_BUG_ON(kvm_tdx->source_pa =3D=3D INVALID_PAGE, kvm)) { + tdx_unpin(kvm, pfn); + return -EINVAL; + } + + source_pa =3D kvm_tdx->source_pa & ~KVM_TDX_MEASURE_MEMORY_REGION; + measure =3D kvm_tdx->source_pa & KVM_TDX_MEASURE_MEMORY_REGION; + kvm_tdx->source_pa =3D INVALID_PAGE; + + do { + err =3D tdh_mem_page_add(kvm_tdx->tdr_pa, gpa, hpa, source_pa, + &out); + /* + * This path is executed during populating initial guest memory + * image. i.e. before running any vcpu. Race is rare. + */ + } while (unlikely(err =3D=3D TDX_ERROR_SEPT_BUSY)); + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_MEM_PAGE_ADD, err, &out); + tdx_unpin(kvm, pfn); + return -EIO; + } else if (measure) + tdx_measure_page(kvm_tdx, gpa); =20 return 0; } @@ -1215,6 +1271,95 @@ void tdx_flush_tlb_current(struct kvm_vcpu *vcpu) tdx_track(to_kvm_tdx(vcpu->kvm)); } =20 +#define TDX_SEPT_PFERR (PFERR_WRITE_MASK | PFERR_GUEST_ENC_MASK) + +static int tdx_init_mem_region(struct kvm *kvm, struct kvm_tdx_cmd *cmd) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + struct kvm_tdx_init_mem_region region; + struct kvm_vcpu *vcpu; + struct page *page; + int idx, ret =3D 0; + bool added =3D false; + + /* Once TD is finalized, the initial guest memory is fixed. */ + if (is_td_finalized(kvm_tdx)) + return -EINVAL; + + /* The BSP vCPU must be created before initializing memory regions. */ + if (!atomic_read(&kvm->online_vcpus)) + return -EINVAL; + + if (cmd->flags & ~KVM_TDX_MEASURE_MEMORY_REGION) + return -EINVAL; + + if (copy_from_user(®ion, (void __user *)cmd->data, sizeof(region))) + return -EFAULT; + + /* Sanity check */ + if (!IS_ALIGNED(region.source_addr, PAGE_SIZE) || + !IS_ALIGNED(region.gpa, PAGE_SIZE) || + !region.nr_pages || + region.gpa + (region.nr_pages << PAGE_SHIFT) <=3D region.gpa || + !kvm_is_private_gpa(kvm, region.gpa) || + !kvm_is_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHIFT)= )) + return -EINVAL; + + vcpu =3D kvm_get_vcpu(kvm, 0); + if (mutex_lock_killable(&vcpu->mutex)) + return -EINTR; + + vcpu_load(vcpu); + idx =3D srcu_read_lock(&kvm->srcu); + + kvm_mmu_reload(vcpu); + + while (region.nr_pages) { + if (signal_pending(current)) { + ret =3D -ERESTARTSYS; + break; + } + + if (need_resched()) + cond_resched(); + + /* Pin the source page. */ + ret =3D get_user_pages_fast(region.source_addr, 1, 0, &page); + if (ret < 0) + break; + if (ret !=3D 1) { + ret =3D -ENOMEM; + break; + } + + kvm_tdx->source_pa =3D pfn_to_hpa(page_to_pfn(page)) | + (cmd->flags & KVM_TDX_MEASURE_MEMORY_REGION); + + ret =3D kvm_mmu_map_tdp_page(vcpu, region.gpa, TDX_SEPT_PFERR, + PG_LEVEL_4K); + put_page(page); + if (ret) + break; + + region.source_addr +=3D PAGE_SIZE; + region.gpa +=3D PAGE_SIZE; + region.nr_pages--; + added =3D true; + } + + srcu_read_unlock(&kvm->srcu, idx); + vcpu_put(vcpu); + + mutex_unlock(&vcpu->mutex); + + if (added && region.nr_pages > 0) + ret =3D -EAGAIN; + if (copy_to_user((void __user *)cmd->data, ®ion, sizeof(region))) + ret =3D -EFAULT; + + return ret; +} + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { struct kvm_tdx_cmd tdx_cmd; @@ -1234,6 +1379,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) case KVM_TDX_INIT_VM: r =3D tdx_td_init(kvm, &tdx_cmd); break; + case KVM_TDX_INIT_MEM_REGION: + r =3D tdx_init_mem_region(kvm, &tdx_cmd); + break; default: r =3D -EINVAL; goto out; diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 6603da8708ad..24ee0bc3285c 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -17,6 +17,8 @@ struct kvm_tdx { u64 xfam; int hkid; =20 + hpa_t source_pa; + bool finalized; atomic_t tdh_mem_track; =20 diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include= /uapi/asm/kvm.h index 83bd9e3118d1..a3408f6e1124 100644 --- a/tools/arch/x86/include/uapi/asm/kvm.h +++ b/tools/arch/x86/include/uapi/asm/kvm.h @@ -567,6 +567,7 @@ enum kvm_tdx_cmd_id { KVM_TDX_CAPABILITIES =3D 0, KVM_TDX_INIT_VM, KVM_TDX_INIT_VCPU, + KVM_TDX_INIT_MEM_REGION, =20 KVM_TDX_CMD_NR_MAX, }; @@ -648,4 +649,12 @@ struct kvm_tdx_init_vm { }; }; =20 +#define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + +struct kvm_tdx_init_mem_region { + __u64 source_addr; + __u64 gpa; + __u64 nr_pages; +}; + #endif /* _ASM_X86_KVM_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 396D4C001DF for ; Tue, 25 Jul 2023 22:19:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231451AbjGYWTG (ORCPT ); Tue, 25 Jul 2023 18:19:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231826AbjGYWRW (ORCPT ); Tue, 25 Jul 2023 18:17:22 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1BB63C11; Tue, 25 Jul 2023 15:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323363; x=1721859363; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GMV9S+VuISPcx/3i9GUJTDTL+nqi66I0CevPooz+lD4=; b=OfoSPGMJnZlijcCHu7VG6Bdj91BQN5jPshA/T+m5UTp6DDww97r/YBO8 BB3CakXg1Co5Ok10drUXix57qQakUHsIdC75abfKlEIRXQlJq3HKiepAm MwmJVlpW8xxA5S28kb63MHugdZKq7vO9mzjdBHqL+4eV/tmzsJIrRBo2H G6Q4Vq9DN7JNmFzQeUOfVhLnrPYx5wWWX5f92UMzkNcl+jX/05ZRfVtz1 sWtRA2rWPD0u+w9J22bRJMqBAr0x/GqIakPRosPd8d3zQUTk2SctSrFt7 IJI6SI8Fm9r2st5otcusPNOZaOVvA0D8Ro7IbApw/0S0xTmZwm28TXGQm Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882521" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882521" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001780" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001780" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:49 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 060/115] KVM: TDX: Finalize VM initialization Date: Tue, 25 Jul 2023 15:14:11 -0700 Message-Id: <5b164b7f299ad4ed103f52cdb0d603100c9841b3.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata To protect the initial contents of the guest TD, the TDX module measures the guest TD during the build process as SHA-384 measurement. The measurement of the guest TD contents needs to be completed to make the guest TD ready to run. Add a new subcommand, KVM_TDX_FINALIZE_VM, for VM-scoped KVM_MEMORY_ENCRYPT_OP to finalize the measurement and mark the TDX VM ready to run. Signed-off-by: Isaku Yamahata --- v14 -> v15: - removed unconditional tdx_track() by tdx_flush_tlb_current() that does tdx_track(). Signed-off-by: Isaku Yamahata --- arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/tdx.c | 21 +++++++++++++++++++++ tools/arch/x86/include/uapi/asm/kvm.h | 1 + 3 files changed, 23 insertions(+) diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index a1815fcbb0be..1b4134247837 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -573,6 +573,7 @@ enum kvm_tdx_cmd_id { KVM_TDX_INIT_VM, KVM_TDX_INIT_VCPU, KVM_TDX_INIT_MEM_REGION, + KVM_TDX_FINALIZE_VM, =20 KVM_TDX_CMD_NR_MAX, }; diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 32e84c29d35e..63f2b6dc4f27 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1360,6 +1360,24 @@ static int tdx_init_mem_region(struct kvm *kvm, stru= ct kvm_tdx_cmd *cmd) return ret; } =20 +static int tdx_td_finalizemr(struct kvm *kvm) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + u64 err; + + if (!is_hkid_assigned(kvm_tdx) || is_td_finalized(kvm_tdx)) + return -EINVAL; + + err =3D tdh_mr_finalize(kvm_tdx->tdr_pa); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MR_FINALIZE, err, NULL); + return -EIO; + } + + kvm_tdx->finalized =3D true; + return 0; +} + int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { struct kvm_tdx_cmd tdx_cmd; @@ -1382,6 +1400,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) case KVM_TDX_INIT_MEM_REGION: r =3D tdx_init_mem_region(kvm, &tdx_cmd); break; + case KVM_TDX_FINALIZE_VM: + r =3D tdx_td_finalizemr(kvm); + break; default: r =3D -EINVAL; goto out; diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include= /uapi/asm/kvm.h index a3408f6e1124..4753a29a22ec 100644 --- a/tools/arch/x86/include/uapi/asm/kvm.h +++ b/tools/arch/x86/include/uapi/asm/kvm.h @@ -568,6 +568,7 @@ enum kvm_tdx_cmd_id { KVM_TDX_INIT_VM, KVM_TDX_INIT_VCPU, KVM_TDX_INIT_MEM_REGION, + KVM_TDX_FINALIZE_VM, =20 KVM_TDX_CMD_NR_MAX, }; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD801C001E0 for ; Tue, 25 Jul 2023 22:19:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232290AbjGYWTS (ORCPT ); Tue, 25 Jul 2023 18:19:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230010AbjGYWR2 (ORCPT ); Tue, 25 Jul 2023 18:17:28 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D15E11FE2; Tue, 25 Jul 2023 15:16:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323369; x=1721859369; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=R5wOuoM4Onpjff9OlffMGMEw35e7AU+j3x/NDTZE/oY=; b=GlbKC7GH23u/GnaDgknL4no6MhfWYvAdBG7ST7qd7GlOkh0uo68jEpFL q4FoV5Cl5DLQ6QIqP9UQU/bP5/DwxxLSKZ9iTT0MsQ/qtgx0dlLNzyNZk 73N8EitPbfHt/bj995mOAmeztLkjcjaa+JNzfLmDY6nxBOst9xmKZilzR JV0Xz95BiD0EWwpQidmoGXGGZH5Vr2mq3uloJwVIMplDPGSsfGXrafaYU nlb7g2m6suj3ACZ02buMzhH36ZjnTnloblZlJIIRL3RKfmA0Daw1NsWqq 8IJWoRRFJSkTPvFXXg5esyWawbUNIn436aRVqYAetlXszojxsbMIuaAtt g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882525" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882525" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001783" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001783" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:50 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 061/115] [MARKER] The start of TDX KVM patch series: TD vcpu enter/exit Date: Tue, 25 Jul 2023 15:14:12 -0700 Message-Id: <75c132b67aa191c84c2f93511df543bc850e9855.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TD vcpu enter/exit. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index 46ae049b6b85..33e107bcb5cf 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -12,6 +12,7 @@ What qemu can do - Qemu can create/destroy guest of TDX vm type. - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. +- Qemu can finalize guest TD. =20 Patch Layer status ------------------ @@ -22,8 +23,8 @@ Patch Layer status * TD VM creation/destruction: Applied * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied -* TD finalization: Applying -* TD vcpu enter/exit: Not yet +* TD finalization: Applied +* TD vcpu enter/exit: Applying * TD vcpu interrupts/exit/hypercall: Not yet =20 * KVM MMU GPA shared bits: Applied --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 226C9C3DA40 for ; Tue, 25 Jul 2023 22:20:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232315AbjGYWTx (ORCPT ); Tue, 25 Jul 2023 18:19:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232211AbjGYWS1 (ORCPT ); Tue, 25 Jul 2023 18:18:27 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B18F62683; Tue, 25 Jul 2023 15:16:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323372; x=1721859372; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LWbwHQW6jlU/01VYinlidduuoO6CklVPFtQo9gzVTyo=; b=HR/e8fHoFX3RoNcnrCxYjPv+F/gf6ZM9k8OWa8UvEH5gPtNxRrlu6bpV JPcsADwFOn2N/Cs5CgGSm+82KJpHAe0S85J9pki6+m1hqYlONgoSBFdYJ yoCh3aj4F6uW8TJDXamWe/ewH9S0e2x1RVXfdlKsH23Fp85a8nzS6whXH mKdHJfsXJT7A3emYMK/acta6Uqct6FTNVLmrfFEZIgUQuCd4oFf0BImK1 0RGhiPN3eP7WsrUqGwjbKo4xKI8qzAX7gBim9Dp1vxujEVr02BoWqNuBC xtI5ZU+A+OcUxIlg+Q4sFeMv0f5cISNnnMPb5t0mA5jdKM2i+heBtS+DH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882530" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882530" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001787" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001787" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:50 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 062/115] KVM: TDX: Add helper assembly function to TDX vcpu Date: Tue, 25 Jul 2023 15:14:13 -0700 Message-Id: <13bb1e737232a42ddeb9e1586632942872f2c775.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX defines an API to run TDX vcpu with its own ABI. Define an assembly helper function to run TDX vcpu to hide the special ABI so that C code can call it with function call ABI. Signed-off-by: Isaku Yamahata --- v14 -> v15: - use symbolic local label(.Lxxx) instead of numeric local label - optimized Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/tdx.h | 3 +- arch/x86/kvm/vmx/vmenter.S | 164 +++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 97b23325ba5e..75711766159b 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -18,7 +18,8 @@ * Bits 47:40 =3D=3D 0xFF indicate Reserved status code class that never u= sed by * TDX module. */ -#define TDX_ERROR _BITUL(63) +#define TDX_ERROR_BIT 63 +#define TDX_ERROR _BITUL(TDX_ERROR_BIT) #define TDX_SW_ERROR (TDX_ERROR | GENMASK_ULL(47, 40)) #define TDX_SEAMCALL_VMFAILINVALID (TDX_SW_ERROR | _UL(0xFFFF0000)) =20 diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 07e927d4d099..b4f1f6117968 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -6,6 +6,7 @@ #include #include #include +#include #include "kvm-asm-offsets.h" #include "run_flags.h" =20 @@ -31,6 +32,12 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif =20 +#ifdef CONFIG_INTEL_TDX_HOST +#define TDH_VP_ENTER 0 +#define EXIT_REASON_TDCALL 77 +#define seamcall .byte 0x66,0x0f,0x01,0xcf +#endif + .macro VMX_DO_EVENT_IRQOFF call_insn call_target /* * Unconditionally create a stack frame, getting the correct RSP on the @@ -360,3 +367,160 @@ SYM_FUNC_END(vmread_error_trampoline) SYM_FUNC_START(vmx_do_interrupt_irqoff) VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 SYM_FUNC_END(vmx_do_interrupt_irqoff) + +#ifdef CONFIG_INTEL_TDX_HOST + +.pushsection .noinstr.text, "ax" + +/** + * __tdx_vcpu_run - Call SEAMCALL(TDH_VP_ENTER) to run a TD vcpu + * @tdvpr: physical address of TDVPR + * @regs: void * (to registers of TDVCPU) + * @gpr_mask: non-zero if guest registers need to be loaded prior to TDH_V= P_ENTER + * + * Returns: + * TD-Exit Reason + * + * Note: KVM doesn't support using XMM in its hypercalls, it's the HyperV + * code's responsibility to save/restore XMM registers on TDVMCALL. + */ +SYM_FUNC_START(__tdx_vcpu_run) + push %rbp + mov %rsp, %rbp + + push %r15 + push %r14 + push %r13 + push %r12 + push %rbx + + /* Save @regs, which is needed after TDH_VP_ENTER to capture output. */ + push %rsi + + /* Load @tdvpr to RCX */ + mov %rdi, %rcx + + /* No need to load guest GPRs if the last exit wasn't a TDVMCALL. */ + test %dx, %dx + je .Lskip_copy_inputs + + /* Load @regs to RAX, which will be clobbered with $TDH_VP_ENTER anyways.= */ + mov %rsi, %rax + + mov VCPU_RBX(%rax), %rbx + mov VCPU_RDX(%rax), %rdx + mov VCPU_RBP(%rax), %rbp + mov VCPU_RSI(%rax), %rsi + mov VCPU_RDI(%rax), %rdi + + mov VCPU_R8 (%rax), %r8 + mov VCPU_R9 (%rax), %r9 + mov VCPU_R10(%rax), %r10 + mov VCPU_R11(%rax), %r11 + mov VCPU_R12(%rax), %r12 + mov VCPU_R13(%rax), %r13 + mov VCPU_R14(%rax), %r14 + mov VCPU_R15(%rax), %r15 + + /* Load TDH_VP_ENTER to RAX. This kills the @regs pointer! */ +.Lskip_copy_inputs: + mov $TDH_VP_ENTER, %rax + +.Lseamcall: + seamcall + + jc .Lvmfail_invalid + + /* xor-swap (%rsp) and %rax */ + xor (%rsp), %rax + xor %rax, (%rsp) + xor (%rsp), %rax + + /* Skip to the exit path if TDH_VP_ENTER failed. */ + btq $TDX_ERROR_BIT, (%rsp) + jc .Lout_rax + + /* check if TD-exit due to TDVMCALL */ + cmpq $EXIT_REASON_TDCALL, (%rsp) + + /* Jump on non-TDVMCALL */ + jne .Lout_non_tdvmcall + + /* Save all output from SEAMCALL(TDH_VP_ENTER) */ + mov %rbx, VCPU_RBX(%rax) + mov %rbp, VCPU_RBP(%rax) + mov %rsi, VCPU_RSI(%rax) + mov %rdi, VCPU_RDI(%rax) + mov %r10, VCPU_R10(%rax) + mov %r11, VCPU_R11(%rax) + mov %r12, VCPU_R12(%rax) + mov %r13, VCPU_R13(%rax) + mov %r14, VCPU_R14(%rax) + mov %r15, VCPU_R15(%rax) + +.Lout_non_tdvmcall: + mov %rcx, VCPU_RCX(%rax) + mov %rdx, VCPU_RDX(%rax) + mov %r8, VCPU_R8 (%rax) + mov %r9, VCPU_R9 (%rax) + + /* + * Clear all general purpose registers except RSP and RAX to prevent + * speculative use of the guest's values. + */ + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %esi, %esi + xorl %edi, %edi + xorl %ebp, %ebp + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + /* Restore the TD-Exit reason to RAX for return. */ +.Lout_rax: + pop %rax + + /* "POP" @regs. */ +.Lout_regs: + pop %rbx + pop %r12 + pop %r13 + pop %r14 + pop %r15 + + pop %rbp + RET + +.Lvmfail_invalid: + /* + * Use same return value convention to tdxcall.S. + * TDX_SEAMCALL_VMFAILINVALID doesn't conflict with any TDX status code. + */ + mov $TDX_SEAMCALL_VMFAILINVALID, %rax + /* discard pushed %rsi: %rsi is caller-saved. */ + add $8, %rsp + jmp .Lout_regs + +.Lseamcall_faulted: + cmpb $0, kvm_rebooting + je 1f + mov $TDX_SW_ERROR, %r12 + orq %r12, %rax + add $8, %rsp + jmp .Lout_regs +1: ud2 + /* Use FAULT version to know what fault happened. */ + _ASM_EXTABLE_FAULT(.Lseamcall, .Lseamcall_faulted) + +SYM_FUNC_END(__tdx_vcpu_run) + +.popsection + +#endif --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 576EDC07E8A for ; Tue, 25 Jul 2023 22:20:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232496AbjGYWUC (ORCPT ); Tue, 25 Jul 2023 18:20:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232317AbjGYWTA (ORCPT ); Tue, 25 Jul 2023 18:19:00 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A894420E; Tue, 25 Jul 2023 15:16:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323381; x=1721859381; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2uaXAQM0A2D+p5EYwSWS8h/2ppWKo5PyI/SXhZDXqo4=; b=WNDBlX5fitJUXRjUZAz5HsIOd8wt6gCBb6UY6eNn6pK4eAVQKL5wA2vu bLYfkC85WHTWFMCvh5WX6nI6nXoqjNfpif9h6P3vVXUMMc7aJIL4xG7l4 WnIPTsVDz5RkjHrcRH1KfYe/MNpiv69ZK5MTxzqOeW00FHVCdb4/39VMN si4vnJD6unyOJBPTKM5WIzY3nRgtQLTMFa6t4ezaukCP/HMn6/I+lRZDr XHuRzfJON0wU/0KS5/qtzCL1Lu/A+1/btttIFALQqOBgHR4WjT5RJXJug Bpcs5nlCPv2ZDapmuOSLVhCu5plabCBW9hctvzdnh4CguBgd8O2Gj2l5E w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882535" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882535" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001790" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001790" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:51 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 063/115] KVM: TDX: Implement TDX vcpu enter/exit path Date: Tue, 25 Jul 2023 15:14:14 -0700 Message-Id: <12aaaf31492ade3f258fa147d2d115de3e7eadd2.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This patch implements running TDX vcpu. Once vcpu runs on the logical processor (LP), the TDX vcpu is associated with it. When the TDX vcpu moves to another LP, the TDX vcpu needs to flush its status on the LP. When destroying TDX vcpu, it needs to complete flush and flush cpu memory cache. Track which LP the TDX vcpu run and flush it as necessary. Do nothing on sched_in event as TDX doesn't support pause loop. TDX vcpu execution requires restoring PMU debug store after returning back to KVM because the TDX module unconditionally resets the value. To reuse the existing code, export perf_restore_debug_store. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 21 +++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 34 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 1 + 5 files changed, 89 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 2eaed14a9542..be5eb3a6965d 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -169,6 +169,23 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool = init_event) vmx_vcpu_reset(vcpu, init_event); } =20 +static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + /* Unconditionally continue to vcpu_run(). */ + return 1; + + return vmx_vcpu_pre_run(vcpu); +} + +static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return tdx_vcpu_run(vcpu); + + return vmx_vcpu_run(vcpu); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -320,8 +337,8 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .flush_tlb_gva =3D vt_flush_tlb_gva, .flush_tlb_guest =3D vt_flush_tlb_guest, =20 - .vcpu_pre_run =3D vmx_vcpu_pre_run, - .vcpu_run =3D vmx_vcpu_run, + .vcpu_pre_run =3D vt_vcpu_pre_run, + .vcpu_run =3D vt_vcpu_run, .handle_exit =3D vmx_handle_exit, .skip_emulated_instruction =3D vmx_skip_emulated_instruction, .update_emulated_instruction =3D vmx_update_emulated_instruction, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 63f2b6dc4f27..a2c7569ddeb4 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -10,6 +10,9 @@ #include "vmx.h" #include "x86.h" =20 +#include +#include "trace.h" + #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt =20 @@ -441,6 +444,37 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_e= vent) */ } =20 +u64 __tdx_vcpu_run(hpa_t tdvpr, void *regs, u32 regs_mask); + +static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu, + struct vcpu_tdx *tdx) +{ + guest_state_enter_irqoff(); + tdx->exit_reason.full =3D __tdx_vcpu_run(tdx->tdvpr_pa, vcpu->arch.regs, = 0); + guest_state_exit_irqoff(); +} + +fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + if (unlikely(!tdx->initialized)) + return -EINVAL; + if (unlikely(vcpu->kvm->vm_bugged)) { + tdx->exit_reason.full =3D TDX_NON_RECOVERABLE_VCPU; + return EXIT_FASTPATH_NONE; + } + + trace_kvm_entry(vcpu); + + tdx_vcpu_enter_exit(vcpu, tdx); + + vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; + trace_kvm_exit(vcpu, KVM_ISA_VMX); + + return EXIT_FASTPATH_NONE; +} + void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 24ee0bc3285c..f54f27ef006c 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -25,12 +25,45 @@ struct kvm_tdx { u64 tsc_offset; }; =20 +union tdx_exit_reason { + struct { + /* 31:0 mirror the VMX Exit Reason format */ + u64 basic : 16; + u64 reserved16 : 1; + u64 reserved17 : 1; + u64 reserved18 : 1; + u64 reserved19 : 1; + u64 reserved20 : 1; + u64 reserved21 : 1; + u64 reserved22 : 1; + u64 reserved23 : 1; + u64 reserved24 : 1; + u64 reserved25 : 1; + u64 bus_lock_detected : 1; + u64 enclave_mode : 1; + u64 smi_pending_mtf : 1; + u64 smi_from_vmx_root : 1; + u64 reserved30 : 1; + u64 failed_vmentry : 1; + + /* 63:32 are TDX specific */ + u64 details_l1 : 8; + u64 class : 8; + u64 reserved61_48 : 14; + u64 non_recoverable : 1; + u64 error : 1; + }; + u64 full; +}; + struct vcpu_tdx { struct kvm_vcpu vcpu; =20 unsigned long tdvpr_pa; unsigned long *tdvpx_pa; =20 + union tdx_exit_reason exit_reason; + bool initialized; =20 /* diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index ed93accd29e6..fa41d6352d52 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -151,6 +151,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); int tdx_vcpu_create(struct kvm_vcpu *vcpu); void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); +fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); @@ -178,6 +179,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __= user *argp) { return -EOP static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTS= UPP; } static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) = {} +static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT= _FASTPATH_NONE; } static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 2922c4a69a6e..bc8e6531b3f3 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -311,6 +311,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header =3D= { }; =20 u64 __read_mostly host_xcr0; +EXPORT_SYMBOL_GPL(host_xcr0); =20 static struct kmem_cache *x86_emulator_cache; =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DA51C001DF for ; Tue, 25 Jul 2023 22:20:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232355AbjGYWUJ (ORCPT ); Tue, 25 Jul 2023 18:20:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232281AbjGYWTN (ORCPT ); Tue, 25 Jul 2023 18:19:13 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46833421F; Tue, 25 Jul 2023 15:16:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323387; x=1721859387; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e8pBWK3jRYG3MN8kY264VDTwz0W6QJyfEZ/PkE5bflg=; b=jLYRxLf8iHWwc81rXBnjqSoPGdwjOGoSwI47Z2uRpDWxD4mx0g7dBomG J/+IfyhCz+CN/c5dTOgnb6t/b1QeLINqz0+iS71BjKcdJXtgxE1spyIch WAaQql67amtqKG+SpudO30LDtuLPjMO9L8ec7BsPu9jgOjTUH0+27R0LR IpjmtQnuRaofgTXvArMoVdmA/Yh8K/baCkYgVsx/Es+Dy5zLEx9+453XK Vf9CV6cA5DoqdUTAZVeOQDji3FJAV6x6aeJ1DPjAbaVHcHR1T2m0AFnot ubnjIb4TVKswCVFAewU/osyEPqXDN0eJsTeztCo6p+a5y69I6mJQixKfe g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882538" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882538" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001793" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001793" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:51 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 064/115] KVM: TDX: vcpu_run: save/restore host state(host kernel gs) Date: Tue, 25 Jul 2023 15:14:15 -0700 Message-Id: <0a31c0621e61bfa7ebd669a07b11693748b1f1f4.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata On entering/exiting TDX vcpu, Preserved or clobbered CPU state is different from VMX case. Add TDX hooks to save/restore host/guest CPU state. Save/restore kernel GS base MSR. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/main.c | 30 ++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 41 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 4 ++++ arch/x86/kvm/vmx/x86_ops.h | 4 ++++ 4 files changed, 77 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index be5eb3a6965d..d4edb479648e 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -169,6 +169,32 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool = init_event) vmx_vcpu_reset(vcpu, init_event); } =20 +static void vt_prepare_switch_to_guest(struct kvm_vcpu *vcpu) +{ + /* + * All host state is saved/restored across SEAMCALL/SEAMRET, and the + * guest state of a TD is obviously off limits. Deferring MSRs and DRs + * is pointless because the TDX module needs to load *something* so as + * not to expose guest state. + */ + if (is_td_vcpu(vcpu)) { + tdx_prepare_switch_to_guest(vcpu); + return; + } + + vmx_prepare_switch_to_guest(vcpu); +} + +static void vt_vcpu_put(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_vcpu_put(vcpu); + return; + } + + vmx_vcpu_put(vcpu); +} + static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) @@ -304,9 +330,9 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .vcpu_free =3D vt_vcpu_free, .vcpu_reset =3D vt_vcpu_reset, =20 - .prepare_switch_to_guest =3D vmx_prepare_switch_to_guest, + .prepare_switch_to_guest =3D vt_prepare_switch_to_guest, .vcpu_load =3D vmx_vcpu_load, - .vcpu_put =3D vmx_vcpu_put, + .vcpu_put =3D vt_vcpu_put, =20 .update_exception_bitmap =3D vmx_update_exception_bitmap, .get_msr_feature =3D vmx_get_msr_feature, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index a2c7569ddeb4..0a71601c78f9 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include =20 #include =20 @@ -372,6 +373,8 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bo= ol is_mmio) =20 int tdx_vcpu_create(struct kvm_vcpu *vcpu) { + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + /* * On cpu creation, cpuid entry is blank. Forcibly enable * X2APIC feature to allow X2APIC. @@ -396,9 +399,45 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.guest_state_protected =3D !(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG); =20 + tdx->host_state_need_save =3D true; + tdx->host_state_need_restore =3D false; + return 0; } =20 +void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + if (!tdx->host_state_need_save) + return; + + if (likely(is_64bit_mm(current->mm))) + tdx->msr_host_kernel_gs_base =3D current->thread.gsbase; + else + tdx->msr_host_kernel_gs_base =3D read_msr(MSR_KERNEL_GS_BASE); + + tdx->host_state_need_save =3D false; +} + +static void tdx_prepare_switch_to_host(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + tdx->host_state_need_save =3D true; + if (!tdx->host_state_need_restore) + return; + + wrmsrl(MSR_KERNEL_GS_BASE, tdx->msr_host_kernel_gs_base); + tdx->host_state_need_restore =3D false; +} + +void tdx_vcpu_put(struct kvm_vcpu *vcpu) +{ + vmx_vcpu_pi_put(vcpu); + tdx_prepare_switch_to_host(vcpu); +} + void tdx_vcpu_free(struct kvm_vcpu *vcpu) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); @@ -469,6 +508,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) =20 tdx_vcpu_enter_exit(vcpu, tdx); =20 + tdx->host_state_need_restore =3D true; + vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; trace_kvm_exit(vcpu, KVM_ISA_VMX); =20 diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index f54f27ef006c..a3c64a2ec9e0 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -66,6 +66,10 @@ struct vcpu_tdx { =20 bool initialized; =20 + bool host_state_need_save; + bool host_state_need_restore; + u64 msr_host_kernel_gs_base; + /* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work. diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index fa41d6352d52..8fcc5807e594 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -152,6 +152,8 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu); void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu); +void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); +void tdx_vcpu_put(struct kvm_vcpu *vcpu); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); @@ -180,6 +182,8 @@ static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu= ) { return -EOPNOTSUPP; } static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) = {} static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT= _FASTPATH_NONE; } +static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_put(struct kvm_vcpu *vcpu) {} static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61C0EC05051 for ; Tue, 25 Jul 2023 22:20:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232418AbjGYWUV (ORCPT ); Tue, 25 Jul 2023 18:20:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232347AbjGYWTU (ORCPT ); Tue, 25 Jul 2023 18:19:20 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E2434236; Tue, 25 Jul 2023 15:16:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323390; x=1721859390; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hAk+DJFvyWblY2NP5P0nsGacBaB4H36NK+A1TuR8L2k=; b=N9ZGJz+UxkdBba341SV7LgC3BjcuEg8kUIzjEp3Wga+y0DJRTNFOit0s aXleuzhKL4bT2/44ZwrxeLqZoI4WLmqdtW7GkFY3J0iz+PsIGgvWCg9K/ y8kFBdakm2mSaK+l/K66DYlKWCDHU0o5gZxbM2zJ6W87NsNJg5jqDbrPx /lejqEhN53tsZ4nf8SKCiO+MryFkMtI4TuCTrr0zxZ89rEf/QI/Uhyl6D gznW8IW47stJYgJHSPWorRw4CAdRMeHBdd40T/xQnseoOt7Um3GlMWHJu 8+IPnSRjAo52PX/i0MfVh4VaQm85BhajpFmeqQrmVxqw5AFxt0bVe56Hx A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882545" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882545" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001796" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001796" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:51 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 065/115] KVM: TDX: restore host xsave state when exit from the guest TD Date: Tue, 25 Jul 2023 15:14:16 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata On exiting from the guest TD, xsave state is clobbered. Restore xsave state on TD exit. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0a71601c78f9..40b5be05e284 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -2,6 +2,7 @@ #include #include =20 +#include #include =20 #include "capabilities.h" @@ -483,6 +484,22 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_e= vent) */ } =20 +static void tdx_restore_host_xsave_state(struct kvm_vcpu *vcpu) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(vcpu->kvm); + + if (static_cpu_has(X86_FEATURE_XSAVE) && + host_xcr0 !=3D (kvm_tdx->xfam & kvm_caps.supported_xcr0)) + xsetbv(XCR_XFEATURE_ENABLED_MASK, host_xcr0); + if (static_cpu_has(X86_FEATURE_XSAVES) && + /* PT can be exposed to TD guest regardless of KVM's XSS support */ + host_xss !=3D (kvm_tdx->xfam & (kvm_caps.supported_xss | XFEATURE_MAS= K_PT))) + wrmsrl(MSR_IA32_XSS, host_xss); + if (static_cpu_has(X86_FEATURE_PKU) && + (kvm_tdx->xfam & XFEATURE_MASK_PKRU)) + write_pkru(vcpu->arch.host_pkru); +} + u64 __tdx_vcpu_run(hpa_t tdvpr, void *regs, u32 regs_mask); =20 static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu, @@ -508,6 +525,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) =20 tdx_vcpu_enter_exit(vcpu, tdx); =20 + tdx_restore_host_xsave_state(vcpu); tdx->host_state_need_restore =3D true; =20 vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C424C04A94 for ; Tue, 25 Jul 2023 22:20:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232398AbjGYWUR (ORCPT ); Tue, 25 Jul 2023 18:20:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232373AbjGYWTX (ORCPT ); Tue, 25 Jul 2023 18:19:23 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4141F448B; Tue, 25 Jul 2023 15:16:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323393; x=1721859393; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oh5NZENZHav3qekkLXbYrNrfmD/wAIeiCaDNA2VvmoU=; b=VTnMnoUIN++xNLqsVf6yCBgUFRYjG1/G5ceo+t91TUh+uPurjVo8a71q h6PG/0TH6FoEbQpS//aOoob/Irw1/ebLYyKycOhjrFWJQUqgQPYpZLpVP XOpr9oRwNyNEbRYhxs4OqWoAjepkwcNKXcIoR9rReUcsA4p+i29VpkRK1 c1YQextpRHnyDclSVgrqNmwFkeY99gWJrojtgjWbJq05DRkyAfsMOsJXk EufLGTq8l5qXEmiNwXp5n+1CcmUGbINm5t25/zON25Dpgnu/gCr4gQCiA IMFEgz1sy6nf624qoc9kU2ZUzUGJeFn6whxBEpO4Ew1J2tFr6yIeAnTOA A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882552" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882552" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001799" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001799" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:52 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Chao Gao Subject: [PATCH v15 066/115] KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr Date: Tue, 25 Jul 2023 15:14:17 -0700 Message-Id: <6be95f9c821a7d8475926bdc3163e0684181ac11.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Chao Gao Several MSRs are constant and only used in userspace(ring 3). But VMs may have different values. KVM uses kvm_set_user_return_msr() to switch to guest's values and leverages user return notifier to restore them when the kernel is to return to userspace. To eliminate unnecessary wrmsr, KVM also caches the value it wrote to an MSR last time. TDX module unconditionally resets some of these MSRs to architectural INIT state on TD exit. It makes the cached values in kvm_user_return_msrs are inconsistent with values in hardware. This inconsistency needs to be fixed. Otherwise, it may mislead kvm_on_user_return() to skip restoring some MSRs to the host's values. kvm_set_user_return_msr() can help correct this case, but it is not optimal as it always does a wrmsr. So, introduce a variation of kvm_set_user_return_msr() to update cached values and skip that wrmsr. Signed-off-by: Chao Gao Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 9705e9f30068..95c2ed8fdcd6 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2200,6 +2200,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ip= i_bitmap_low, int kvm_add_user_return_msr(u32 msr); int kvm_find_user_return_msr(u32 msr); int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask); +void kvm_user_return_update_cache(unsigned int index, u64 val); =20 static inline bool kvm_is_supported_user_return_msr(u32 msr) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index bc8e6531b3f3..7805987d891d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -436,6 +436,15 @@ static void kvm_user_return_msr_cpu_online(void) } } =20 +static void kvm_user_return_register_notifier(struct kvm_user_return_msrs = *msrs) +{ + if (!msrs->registered) { + msrs->urn.on_user_return =3D kvm_on_user_return; + user_return_notifier_register(&msrs->urn); + msrs->registered =3D true; + } +} + int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask) { unsigned int cpu =3D smp_processor_id(); @@ -450,15 +459,21 @@ int kvm_set_user_return_msr(unsigned slot, u64 value,= u64 mask) return 1; =20 msrs->values[slot].curr =3D value; - if (!msrs->registered) { - msrs->urn.on_user_return =3D kvm_on_user_return; - user_return_notifier_register(&msrs->urn); - msrs->registered =3D true; - } + kvm_user_return_register_notifier(msrs); return 0; } EXPORT_SYMBOL_GPL(kvm_set_user_return_msr); =20 +/* Update the cache, "curr", and register the notifier */ +void kvm_user_return_update_cache(unsigned int slot, u64 value) +{ + struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(user_return_msrs); + + msrs->values[slot].curr =3D value; + kvm_user_return_register_notifier(msrs); +} +EXPORT_SYMBOL_GPL(kvm_user_return_update_cache); + static void drop_user_return_notifiers(void) { unsigned int cpu =3D smp_processor_id(); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC26AEB64DD for ; Tue, 25 Jul 2023 22:20:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232422AbjGYWUu (ORCPT ); Tue, 25 Jul 2023 18:20:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232448AbjGYWTe (ORCPT ); Tue, 25 Jul 2023 18:19:34 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B1832723; Tue, 25 Jul 2023 15:16:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323405; x=1721859405; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zIYfTeFarYuwjevX8T09031zcq9fM/vQrWPxIG21/EM=; b=TUvxMxOXGxyNH2E09wRSkBRCHJcSMl7w2VCUHRe934MQOjmONMldjaMz /VkYyyhYOJ0yZiM6M1AC7aHEtdBeCnWW1rNkWKX8cEdeH8HfV+qOdXGbx fHlEZSkr+VVJAIffRXLE+EqUdWFiX3Degr7Lovyr5fuXL2ym8XVdMjuE5 fZ75GDOVtSjcI/HJbBkdG9LDXm4gsDnv6gX3uQrC2yfqjbbNP82igSD1N brtHlSOEKFz4Nm2qgNUATr+e+RECoexgQUwzf+dRQcMwCpnmDF70jGWzu JMNprfnou/bjUh3fORq/p02hKVQZCIRxLntDe8nbEZURfBtgSAQVMS93F Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882558" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882558" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001802" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001802" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:52 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 067/115] KVM: TDX: restore user ret MSRs Date: Tue, 25 Jul 2023 15:14:18 -0700 Message-Id: <92a5b3170d0f40417f8e2ef20ca853fdf2f01319.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Several user ret MSRs are clobbered on TD exit. Restore those values on TD exit and before returning to ring 3. Because TSX_CTRL requires special treat, this patch doesn't address it. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 40b5be05e284..05657e51406a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -484,6 +484,28 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_e= vent) */ } =20 +struct tdx_uret_msr { + u32 msr; + unsigned int slot; + u64 defval; +}; + +static struct tdx_uret_msr tdx_uret_msrs[] =3D { + {.msr =3D MSR_SYSCALL_MASK, .defval =3D 0x20200 }, + {.msr =3D MSR_STAR,}, + {.msr =3D MSR_LSTAR,}, + {.msr =3D MSR_TSC_AUX,}, +}; + +static void tdx_user_return_update_cache(void) +{ + int i; + + for (i =3D 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) + kvm_user_return_update_cache(tdx_uret_msrs[i].slot, + tdx_uret_msrs[i].defval); +} + static void tdx_restore_host_xsave_state(struct kvm_vcpu *vcpu) { struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(vcpu->kvm); @@ -525,6 +547,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) =20 tdx_vcpu_enter_exit(vcpu, tdx); =20 + tdx_user_return_update_cache(); tdx_restore_host_xsave_state(vcpu); tdx->host_state_need_restore =3D true; =20 @@ -1718,6 +1741,26 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x8= 6_ops) return -EINVAL; } =20 + for (i =3D 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) { + /* + * Here it checks if MSRs (tdx_uret_msrs) can be saved/restored + * before returning to user space. + * + * this_cpu_ptr(user_return_msrs)->registered isn't checked + * because the registration is done at vcpu runtime by + * kvm_set_user_return_msr(). + * Here is setting up cpu feature before running vcpu, + * registered is already false. + */ + tdx_uret_msrs[i].slot =3D kvm_find_user_return_msr(tdx_uret_msrs[i].msr); + if (tdx_uret_msrs[i].slot =3D=3D -1) { + /* If any MSR isn't supported, it is a KVM bug */ + pr_err("MSR %x isn't included by kvm_find_user_return_msr\n", + tdx_uret_msrs[i].msr); + return -EIO; + } + } + max_pkgs =3D topology_max_packages(); tdx_mng_key_config_lock =3D kcalloc(max_pkgs, sizeof(*tdx_mng_key_config_= lock), GFP_KERNEL); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DABB2C001DF for ; Tue, 25 Jul 2023 22:20:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232278AbjGYWUw (ORCPT ); Tue, 25 Jul 2023 18:20:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232452AbjGYWTe (ORCPT ); Tue, 25 Jul 2023 18:19:34 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B9BE44BE; Tue, 25 Jul 2023 15:16:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323406; x=1721859406; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HFH89bGGYO1CEevnA/rPrJnk0mYCrAPQ2IaH6NklX3c=; b=fNtQM2CjUYCHTb9HvQvTDmyz5P1R7y1bDEE0JcBnpVSI6lxyrM2FVWcO nhrVHjUQpKLZGxHSEYJYHsc8Ne47xWHKZxoNUlzvCoPwD7C7VAaFkzmKO 6or47927g/S9xVzimOLLbesRsIDqIb8rTjXmzRB06jOXUCT1BX/SKmLiK JyPu7aNRqasOMaTAnVRKLajszg6MbXHA0nTt3r0JvjRjBH7FXI8DsZnfQ wiFWakhbvq4o23wXuNzZ892fBSVDsqd3LLCuEgqNOHKaVsyTKq/1z598x EzwDGgK1g8X1xsqbEQa4xLTCEy5XZKgFFDOrxIXoUCLr95tPLHwdyFObv Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882562" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882562" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001805" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001805" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:52 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Yang Weijiang Subject: [PATCH v15 068/115] KVM: TDX: Add TSX_CTRL msr into uret_msrs list Date: Tue, 25 Jul 2023 15:14:19 -0700 Message-Id: <6c2d9c76d58cd511e2a271014d6e04eecea45cca.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yang Weijiang TDX module resets the TSX_CTRL MSR to 0 at TD exit if TSX is enabled for TD. Or it preserves the TSX_CTRL MSR if TSX is disabled for TD. VMM can rely on uret_msrs mechanism to defer the reload of host value until exiting to user space. Signed-off-by: Yang Weijiang Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 33 +++++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.h | 8 ++++++++ 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 05657e51406a..1ae72c84d40e 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -496,14 +496,21 @@ static struct tdx_uret_msr tdx_uret_msrs[] =3D { {.msr =3D MSR_LSTAR,}, {.msr =3D MSR_TSC_AUX,}, }; +static unsigned int tdx_uret_tsx_ctrl_slot; =20 -static void tdx_user_return_update_cache(void) +static void tdx_user_return_update_cache(struct kvm_vcpu *vcpu) { int i; =20 for (i =3D 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) kvm_user_return_update_cache(tdx_uret_msrs[i].slot, tdx_uret_msrs[i].defval); + /* + * TSX_CTRL is reset to 0 if guest TSX is supported. Otherwise + * preserved. + */ + if (to_kvm_tdx(vcpu->kvm)->tsx_supported && tdx_uret_tsx_ctrl_slot !=3D -= 1) + kvm_user_return_update_cache(tdx_uret_tsx_ctrl_slot, 0); } =20 static void tdx_restore_host_xsave_state(struct kvm_vcpu *vcpu) @@ -547,7 +554,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) =20 tdx_vcpu_enter_exit(vcpu, tdx); =20 - tdx_user_return_update_cache(); + tdx_user_return_update_cache(vcpu); tdx_restore_host_xsave_state(vcpu); tdx->host_state_need_restore =3D true; =20 @@ -1050,6 +1057,22 @@ static int setup_tdparams_xfam(struct kvm_cpuid2 *cp= uid, struct td_params *td_pa return 0; } =20 +static bool tdparams_tsx_supported(struct kvm_cpuid2 *cpuid) +{ + const struct kvm_cpuid_entry2 *entry; + u64 mask; + u32 ebx; + + entry =3D kvm_find_cpuid_entry2(cpuid->entries, cpuid->nent, 0x7, 0); + if (entry) + ebx =3D entry->ebx; + else + ebx =3D 0; + + mask =3D __feature_bit(X86_FEATURE_HLE) | __feature_bit(X86_FEATURE_RTM); + return ebx & mask; +} + static int setup_tdparams(struct kvm *kvm, struct td_params *td_params, struct kvm_tdx_init_vm *init_vm) { @@ -1095,6 +1118,7 @@ static int setup_tdparams(struct kvm *kvm, struct td_= params *td_params, MEMCPY_SAME_SIZE(td_params->mrowner, init_vm->mrowner); MEMCPY_SAME_SIZE(td_params->mrownerconfig, init_vm->mrownerconfig); =20 + to_kvm_tdx(kvm)->tsx_supported =3D tdparams_tsx_supported(cpuid); return 0; } =20 @@ -1760,6 +1784,11 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x8= 6_ops) return -EIO; } } + tdx_uret_tsx_ctrl_slot =3D kvm_find_user_return_msr(MSR_IA32_TSX_CTRL); + if (tdx_uret_tsx_ctrl_slot =3D=3D -1 && boot_cpu_has(X86_FEATURE_MSR_TSX_= CTRL)) { + pr_err("MSR_IA32_TSX_CTRL isn't included by kvm_find_user_return_msr\n"); + return -EIO; + } =20 max_pkgs =3D topology_max_packages(); tdx_mng_key_config_lock =3D kcalloc(max_pkgs, sizeof(*tdx_mng_key_config_= lock), diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index a3c64a2ec9e0..98c8d07723a1 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -17,6 +17,14 @@ struct kvm_tdx { u64 xfam; int hkid; =20 + /* + * Used on each TD-exit, see tdx_user_return_update_cache(). + * TSX_CTRL value on TD exit + * - set 0 if guest TSX enabled + * - preserved if guest TSX disabled + */ + bool tsx_supported; + hpa_t source_pa; =20 bool finalized; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC130EB64DD for ; Tue, 25 Jul 2023 22:20:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230168AbjGYWUz (ORCPT ); Tue, 25 Jul 2023 18:20:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232244AbjGYWTr (ORCPT ); Tue, 25 Jul 2023 18:19:47 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A5D94693; Tue, 25 Jul 2023 15:16:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323412; x=1721859412; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=J0qMXx4Z1bmL0Bo7ShUuSxkVx6XY1Y4HAlOmBvk4LIo=; b=Kf1s4V/Ob4WYSCRqLjsPH+miYx0fPd6MlsdzstfYum80ajYvherRHfnm Dh+f/KhSeMdZerTyrjVyZR6ijBHof2zX7ocgrgNh6VhiNC0RAFuCumak0 tWrDd63AHqS7YjlTIRaZb33E2Sc4xWHFZv9TdSSkCvsGfTAl45Az1o8fe 96lrxGLL0w+t/IMS8ObLOUbFRgULkBv7yZPV7aJnbjjhVrYa10tYYvVL+ MxR3wRD79HqXIE3nCT4J2P3AnKhq3Ww39OUdPZRSn8Ou40M7ZGBG83HRs m5y5Z/wjz+dpoQZsB2A0Fn+Gy5oMRyVLH1S8c5lUtnrkdIGQwVLolRfsW Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882567" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882567" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001808" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001808" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:53 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 069/115] [MARKER] The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls Date: Tue, 25 Jul 2023 15:14:20 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the start of patch series of TD vcpu exits, interrupts, and hypercalls. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/intel-tdx-layer-status.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst index 33e107bcb5cf..7a16fa284b6f 100644 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ b/Documentation/virt/kvm/intel-tdx-layer-status.rst @@ -13,6 +13,7 @@ What qemu can do - Qemu can create/destroy vcpu of TDX vm type. - Qemu can populate initial guest memory image. - Qemu can finalize guest TD. +- Qemu can start to run vcpu. But vcpu can not make progress yet. =20 Patch Layer status ------------------ @@ -24,7 +25,7 @@ Patch Layer status * TD vcpu creation/destruction: Applied * TDX EPT violation: Applied * TD finalization: Applied -* TD vcpu enter/exit: Applying +* TD vcpu enter/exit: Applied * TD vcpu interrupts/exit/hypercall: Not yet =20 * KVM MMU GPA shared bits: Applied --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F030EB64DD for ; Tue, 25 Jul 2023 22:21:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232554AbjGYWVD (ORCPT ); Tue, 25 Jul 2023 18:21:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232314AbjGYWTw (ORCPT ); Tue, 25 Jul 2023 18:19:52 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21BD346B5; Tue, 25 Jul 2023 15:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323415; x=1721859415; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Fn1ZFUnwK0JunRRUkqWvyzNDXT79+TfdiTiGrjXZKRs=; b=kYFVgjwU4TRiljLUV6OzUv8fRiytBubKP9LUAgqsbA2tPaYqUPq9eZIl RaM8Iv50hS9cBKXz3/VXNj9bBLv0IbYVvawV9FoxfTYME+qUAzMtuYKUF fiD4pZrbl2Hg8eB/cxqqF8F3oGLb5vyB6S/fId0b7W4eV3Z8s6jWtfKG1 0eucRJ4H2QLJ161RjXvFI56hKF2107o6t2cJhOt5cQKVvFmb0kKe3abp4 +Y3NXwypVlPDePvud3Y3U9UMLz3QYu3hOmsi2ih2YGlYivxgTGRWWo7/f AoxZ+NQ4HHuvg2vEy53sul5ZAHhxMCXLkeU+9PXwDr4qyC3CQzRv6VUm+ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882574" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882574" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001811" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001811" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:53 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 070/115] KVM: TDX: complete interrupts after tdexit Date: Tue, 25 Jul 2023 15:14:21 -0700 Message-Id: <0cd95cb8151b8db3c4fb2ce66144beee1c816169.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This corresponds to VMX __vmx_complete_interrupts(). Because TDX virtualize vAPIC, KVM only needs to care NMI injection. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.h | 2 ++ 2 files changed, 12 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 1ae72c84d40e..6e08d4ec132e 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -484,6 +484,14 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_e= vent) */ } =20 +static void tdx_complete_interrupts(struct kvm_vcpu *vcpu) +{ + /* Avoid costly SEAMCALL if no nmi was injected */ + if (vcpu->arch.nmi_injected) + vcpu->arch.nmi_injected =3D td_management_read8(to_tdx(vcpu), + TD_VCPU_PEND_NMI); +} + struct tdx_uret_msr { u32 msr; unsigned int slot; @@ -561,6 +569,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; trace_kvm_exit(vcpu, KVM_ISA_VMX); =20 + tdx_complete_interrupts(vcpu); + return EXIT_FASTPATH_NONE; } =20 diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 98c8d07723a1..2970536e014a 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -200,6 +200,8 @@ TDX_BUILD_TDVPS_ACCESSORS(16, VMCS, vmcs); TDX_BUILD_TDVPS_ACCESSORS(32, VMCS, vmcs); TDX_BUILD_TDVPS_ACCESSORS(64, VMCS, vmcs); =20 +TDX_BUILD_TDVPS_ACCESSORS(8, MANAGEMENT, management); + static __always_inline u64 td_tdcs_exec_read64(struct kvm_tdx *kvm_tdx, u3= 2 field) { struct tdx_module_output out; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEB50C0015E for ; Tue, 25 Jul 2023 22:21:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232471AbjGYWV3 (ORCPT ); Tue, 25 Jul 2023 18:21:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232467AbjGYWT6 (ORCPT ); Tue, 25 Jul 2023 18:19:58 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BD6E49CF; Tue, 25 Jul 2023 15:17:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323420; x=1721859420; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hN9pn4xLjC0uRDj1rBgILXa96lY/y2sl8BLB+qTChQs=; b=ghL23Os5EAbzmZBjRJcaf7lyJ955buYaNCNcvjvTPs++MOVeQysGaj8R TcQ9QGgnX/QaAYmBmlwNMY5kf0SAwQDSxHdWg0b3JWvvVJgctV1CH8Ztq s6u3vT4NO7a7aLQsqa5/v7rsfq/L+NiPVfJMhzecnCl9iDtuFphbT+lXx nqhI+Mgkg9GtLvy1b6YwX8HUGuOyjFu2hXfnjZqCZ8i115Pzby9BzDssx DPM+3H7EfO1TbjbGQX/wayQIW/GLaCeeQPBKkotzXm6kXU2wRFhU2yl2K kGNZL9x8UVGCsWU3sD5POo6Od+kLfyNkLA7dvhWu/m1nzBALl4Qtd7HW+ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882579" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882579" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001814" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001814" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:54 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 071/115] KVM: TDX: restore debug store when TD exit Date: Tue, 25 Jul 2023 15:14:22 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because debug store is clobbered, restore it on TD exit. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/events/intel/ds.c | 1 + arch/x86/kvm/vmx/tdx.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c index df88576d6b2a..71d0b95b80dc 100644 --- a/arch/x86/events/intel/ds.c +++ b/arch/x86/events/intel/ds.c @@ -2423,3 +2423,4 @@ void perf_restore_debug_store(void) =20 wrmsrl(MSR_IA32_DS_AREA, (unsigned long)ds); } +EXPORT_SYMBOL_GPL(perf_restore_debug_store); diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 6e08d4ec132e..b46bd963349c 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -563,6 +563,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) tdx_vcpu_enter_exit(vcpu, tdx); =20 tdx_user_return_update_cache(vcpu); + perf_restore_debug_store(); tdx_restore_host_xsave_state(vcpu); tdx->host_state_need_restore =3D true; =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C307CC0015E for ; Tue, 25 Jul 2023 22:21:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232500AbjGYWVc (ORCPT ); Tue, 25 Jul 2023 18:21:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232480AbjGYWT7 (ORCPT ); Tue, 25 Jul 2023 18:19:59 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC1821738; Tue, 25 Jul 2023 15:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323421; x=1721859421; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oAoRI/MaGc2MZJ2E0YaA8ssWcjEPMxc5IyNgwHBfkA8=; b=ITGoh1aSp1/aWJx5aYhwuUlhkRrmMCuRUJL9527qwImKPq/pW+BA9l4i 2lFOitUYzmz4/eWBwVuGxVHieALRH98hwGDwf/7a9/91wr6lY96Nt3TVL REc6nt1ejwI9mY6goEooYzTDpM4eP63ePRlhQ3Y+fz5pKs6RXJ4693MxG biCrGqlZpHb5Cq8luC4J8pK8XdDBoQ3cl4b0p6zKD3tfjGFtbLluJqnRy cS+9+RxxogQWqZbfIT/g8eyxJh9LMoOLJFDfUjIVIZA13x4YKqISb0Y/I Q1GwavhVyzpY2+7aKmDt4BYaBa0/m58Mnxa/gnvWCEezv6khAqqBag1DG Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882585" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882585" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001817" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001817" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:54 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 072/115] KVM: TDX: handle vcpu migration over logical processor Date: Tue, 25 Jul 2023 15:14:23 -0700 Message-Id: <03580494a8c32eff5cd5accef49fa0910ae2174a.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata For vcpu migration, in the case of VMX, VMCS is flushed on the source pcpu, and load it on the target pcpu. There are corresponding TDX SEAMCALL APIs, call them on vcpu migration. The logic is mostly same as VMX except the TDX SEAMCALLs are used. When shutting down the machine, (VMX or TDX) vcpus needs to be shutdown on each pcpu. Do the similar for TDX with TDX SEAMCALL APIs. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 32 ++++++- arch/x86/kvm/vmx/tdx.c | 165 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 2 + arch/x86/kvm/vmx/x86_ops.h | 4 + 4 files changed, 200 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index d4edb479648e..a0570ff1eae1 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -44,6 +44,14 @@ static int vt_hardware_enable(void) return ret; } =20 +static void vt_hardware_disable(void) +{ + /* Note, TDX *and* VMX need to be disabled if TDX is enabled. */ + if (enable_tdx) + tdx_hardware_disable(); + vmx_hardware_disable(); +} + static __init int vt_hardware_setup(void) { int ret; @@ -212,6 +220,16 @@ static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu) return vmx_vcpu_run(vcpu); } =20 +static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_vcpu_load(vcpu, cpu); + return; + } + + vmx_vcpu_load(vcpu, cpu); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -271,6 +289,14 @@ static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa= _t root_hpa, vmx_load_mmu_pgd(vcpu, root_hpa, pgd_level); } =20 +static void vt_sched_in(struct kvm_vcpu *vcpu, int cpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_sched_in(vcpu, cpu); +} + static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { if (is_td_vcpu(vcpu)) @@ -313,7 +339,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .offline_cpu =3D tdx_offline_cpu, =20 .hardware_enable =3D vt_hardware_enable, - .hardware_disable =3D vmx_hardware_disable, + .hardware_disable =3D vt_hardware_disable, .has_emulated_msr =3D vmx_has_emulated_msr, =20 .is_vm_type_supported =3D vt_is_vm_type_supported, @@ -331,7 +357,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .vcpu_reset =3D vt_vcpu_reset, =20 .prepare_switch_to_guest =3D vt_prepare_switch_to_guest, - .vcpu_load =3D vmx_vcpu_load, + .vcpu_load =3D vt_vcpu_load, .vcpu_put =3D vt_vcpu_put, =20 .update_exception_bitmap =3D vmx_update_exception_bitmap, @@ -417,7 +443,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .request_immediate_exit =3D vmx_request_immediate_exit, =20 - .sched_in =3D vmx_sched_in, + .sched_in =3D vt_sched_in, =20 .cpu_dirty_log_size =3D PML_ENTITY_NUM, .update_cpu_dirty_logging =3D vmx_update_cpu_dirty_logging, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b46bd963349c..259139abb8ba 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -73,6 +73,14 @@ static DEFINE_MUTEX(tdx_lock); static struct mutex *tdx_mng_key_config_lock; static atomic_t nr_configured_hkid; =20 +/* + * A per-CPU list of TD vCPUs associated with a given CPU. Used when a CPU + * is brought down to invoke TDH_VP_FLUSH on the approapriate TD vCPUS. + * Protected by interrupt mask. This list is manipulated in process conte= xt + * of vcpu and IPI callback. See tdx_flush_vp_on_cpu(). + */ +static DEFINE_PER_CPU(struct list_head, associated_tdvcpus); + static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u16 hkid) { return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits); @@ -104,6 +112,35 @@ static inline bool is_td_finalized(struct kvm_tdx *kvm= _tdx) return kvm_tdx->finalized; } =20 +static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu) +{ + list_del(&to_tdx(vcpu)->cpu_list); + + /* + * Ensure tdx->cpu_list is updated is before setting vcpu->cpu to -1, + * otherwise, a different CPU can see vcpu->cpu =3D -1 and add the vCPU + * to its list before its deleted from this CPUs list. + */ + smp_wmb(); + + vcpu->cpu =3D -1; +} + +static void tdx_disassociate_vp_arg(void *vcpu) +{ + tdx_disassociate_vp(vcpu); +} + +static void tdx_disassociate_vp_on_cpu(struct kvm_vcpu *vcpu) +{ + int cpu =3D vcpu->cpu; + + if (unlikely(cpu =3D=3D -1)) + return; + + smp_call_function_single(cpu, tdx_disassociate_vp_arg, vcpu, 1); +} + static void tdx_clear_page(unsigned long page_pa) { const void *zero_page =3D (const void *) __va(page_to_phys(ZERO_PAGE(0))); @@ -186,6 +223,85 @@ static void tdx_reclaim_td_page(unsigned long td_page_= pa) free_page((unsigned long)__va(td_page_pa)); } =20 +struct tdx_flush_vp_arg { + struct kvm_vcpu *vcpu; + u64 err; +}; + +static void tdx_flush_vp(void *arg_) +{ + struct tdx_flush_vp_arg *arg =3D arg_; + struct kvm_vcpu *vcpu =3D arg->vcpu; + u64 err; + + arg->err =3D 0; + lockdep_assert_irqs_disabled(); + + /* Task migration can race with CPU offlining. */ + if (unlikely(vcpu->cpu !=3D raw_smp_processor_id())) + return; + + /* + * No need to do TDH_VP_FLUSH if the vCPU hasn't been initialized. The + * list tracking still needs to be updated so that it's correct if/when + * the vCPU does get initialized. + */ + if (is_td_vcpu_created(to_tdx(vcpu))) { + /* + * No need to retry. TDX Resources needed for TDH.VP.FLUSH are, + * TDVPR as exclusive, TDR as shared, and TDCS as shared. This + * vp flush function is called when destructing vcpu/TD or vcpu + * migration. No other thread uses TDVPR in those cases. + */ + err =3D tdh_vp_flush(to_tdx(vcpu)->tdvpr_pa); + if (unlikely(err && err !=3D TDX_VCPU_NOT_ASSOCIATED)) { + /* + * This function is called in IPI context. Do not use + * printk to avoid console semaphore. + * The caller prints out the error message, instead. + */ + if (err) + arg->err =3D err; + } + } + + tdx_disassociate_vp(vcpu); +} + +static void tdx_flush_vp_on_cpu(struct kvm_vcpu *vcpu) +{ + struct tdx_flush_vp_arg arg =3D { + .vcpu =3D vcpu, + }; + int cpu =3D vcpu->cpu; + + if (unlikely(cpu =3D=3D -1)) + return; + + smp_call_function_single(cpu, tdx_flush_vp, &arg, 1); + if (WARN_ON_ONCE(arg.err)) { + pr_err("cpu: %d ", cpu); + pr_tdx_error(TDH_VP_FLUSH, arg.err, NULL); + } +} + +void tdx_hardware_disable(void) +{ + int cpu =3D raw_smp_processor_id(); + struct list_head *tdvcpus =3D &per_cpu(associated_tdvcpus, cpu); + struct tdx_flush_vp_arg arg; + struct vcpu_tdx *tdx, *tmp; + unsigned long flags; + + local_irq_save(flags); + /* Safe variant needed as tdx_disassociate_vp() deletes the entry. */ + list_for_each_entry_safe(tdx, tmp, tdvcpus, cpu_list) { + arg.vcpu =3D &tdx->vcpu; + tdx_flush_vp(&arg); + } + local_irq_restore(flags); +} + static int tdx_do_tdh_phymem_cache_wb(void *param) { u64 err =3D 0; @@ -210,6 +326,8 @@ void tdx_mmu_release_hkid(struct kvm *kvm) struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); cpumask_var_t packages; bool cpumask_allocated; + struct kvm_vcpu *vcpu; + unsigned long j; u64 err; int ret; int i; @@ -220,6 +338,19 @@ void tdx_mmu_release_hkid(struct kvm *kvm) if (!is_td_created(kvm_tdx)) goto free_hkid; =20 + kvm_for_each_vcpu(j, vcpu, kvm) + tdx_flush_vp_on_cpu(vcpu); + + mutex_lock(&tdx_lock); + err =3D tdh_mng_vpflushdone(kvm_tdx->tdr_pa); + mutex_unlock(&tdx_lock); + if (WARN_ON_ONCE(err)) { + pr_tdx_error(TDH_MNG_VPFLUSHDONE, err, NULL); + pr_err("tdh_mng_vpflushdone failed. HKID %d is leaked.\n", + kvm_tdx->hkid); + return; + } + cpumask_allocated =3D zalloc_cpumask_var(&packages, GFP_KERNEL); cpus_read_lock(); for_each_online_cpu(i) { @@ -406,6 +537,26 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) return 0; } =20 +void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + if (vcpu->cpu =3D=3D cpu) + return; + + tdx_flush_vp_on_cpu(vcpu); + + local_irq_disable(); + /* + * Pairs with the smp_wmb() in tdx_disassociate_vp() to ensure + * vcpu->cpu is read before tdx->cpu_list. + */ + smp_rmb(); + + list_add(&tdx->cpu_list, &per_cpu(associated_tdvcpus, cpu)); + local_irq_enable(); +} + void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); @@ -455,6 +606,16 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu) return; } =20 + /* + * When destroying VM, kvm_unload_vcpu_mmu() calls vcpu_load() for every + * vcpu after they already disassociated from the per cpu list by + * tdx_mmu_release_hkid(). So we need to disassociate them again, + * otherwise the freed vcpu data will be accessed when do + * list_{del,add}() on associated_tdvcpus list later. + */ + tdx_disassociate_vp_on_cpu(vcpu); + WARN_ON_ONCE(vcpu->cpu !=3D -1); + if (tdx->tdvpx_pa) { for (i =3D 0; i < tdx_info.nr_tdvpx_pages; i++) { if (tdx->tdvpx_pa[i]) @@ -1776,6 +1937,10 @@ int __init tdx_hardware_setup(struct kvm_x86_ops *x8= 6_ops) return -EINVAL; } =20 + /* tdx_hardware_disable() uses associated_tdvcpus. */ + for_each_possible_cpu(i) + INIT_LIST_HEAD(&per_cpu(associated_tdvcpus, i)); + for (i =3D 0; i < ARRAY_SIZE(tdx_uret_msrs); i++) { /* * Here it checks if MSRs (tdx_uret_msrs) can be saved/restored diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 2970536e014a..da7e83dc34b8 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -70,6 +70,8 @@ struct vcpu_tdx { unsigned long tdvpr_pa; unsigned long *tdvpx_pa; =20 + struct list_head cpu_list; + union tdx_exit_reason exit_reason; =20 bool initialized; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 8fcc5807e594..231da434a08b 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -138,6 +138,7 @@ void vmx_setup_mce(struct kvm_vcpu *vcpu); #ifdef CONFIG_INTEL_TDX_HOST int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops); void tdx_hardware_unsetup(void); +void tdx_hardware_disable(void); bool tdx_is_vm_type_supported(unsigned long type); int tdx_offline_cpu(void); =20 @@ -154,6 +155,7 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_ev= ent); fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu); void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); void tdx_vcpu_put(struct kvm_vcpu *vcpu); +void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); @@ -165,6 +167,7 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root= _hpa, int root_level); #else static inline int tdx_hardware_setup(struct kvm_x86_ops *x86_ops) { return= -EOPNOTSUPP; } static inline void tdx_hardware_unsetup(void) {} +static inline void tdx_hardware_disable(void) {} static inline bool tdx_is_vm_type_supported(unsigned long type) { return f= alse; } static inline int tdx_offline_cpu(void) { return 0; } =20 @@ -184,6 +187,7 @@ static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu= , bool init_event) {} static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT= _FASTPATH_NONE; } static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_put(struct kvm_vcpu *vcpu) {} +static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {} static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7BD6C001DF for ; Tue, 25 Jul 2023 22:21:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232296AbjGYWVz (ORCPT ); Tue, 25 Jul 2023 18:21:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232537AbjGYWUG (ORCPT ); Tue, 25 Jul 2023 18:20:06 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E70DE49E4; Tue, 25 Jul 2023 15:17:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323432; x=1721859432; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=65Yp9gjr+VOS3oG1MmqhhW7LdPrD4JIpNsaIlvC0Zdc=; b=ZqqY2xzks87Z7Y1eKYiqzgkXLih9Cg0VTEPql9SXeWJKekfPmn0iWaQA zcEuX3SKEzp9hnR6yA0qUDk81mXSN7LOg5Hr0BNSBfl8iQgAi1Bnn8U2n QOAuMKNK7phZLJhKAnn0twNiIeVNKVWig4bVwnYqGKn2Qak+xurycZ886 rxu6uXJ/Nbk4yFWEAdxQHI0yU+saU8M7mXfwNc3IeGfUcRbwFgzYSmkr9 3GFm0/OQNVPejoXEVS6bIIACbGNERNNqzdF6RDttb8XpzAE0MprrKSLmw yE7zQozAcXNvhmMulIN245huOKHGO/Fck65L1KiiW5bfOJy31Br9b00iz Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882590" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882590" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001820" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001820" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:54 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Xiaoyao Li , Sean Christopherson , Chao Gao Subject: [PATCH v15 073/115] KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior Date: Tue, 25 Jul 2023 15:14:24 -0700 Message-Id: <746afca0a586868d0b5074c462d35df28d818775.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Add a flag, KVM_DEBUGREG_AUTO_SWITCHED_GUEST, to skip saving/restoring DRs irrespective of any other flags. TDX-SEAM unconditionally saves and restores guest DRs and reset to architectural INIT state on TD exit. So, KVM needs to save host DRs before TD enter without restoring guest DRs and restore host DRs after TD exit. Opportunistically convert the KVM_DEBUGREG_* definitions to use BIT(). Reported-by: Xiaoyao Li Signed-off-by: Sean Christopherson Co-developed-by: Chao Gao Signed-off-by: Chao Gao Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 10 ++++++++-- arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/x86.c | 11 ++++++++--- 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 95c2ed8fdcd6..42ddf087fe60 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -603,8 +603,14 @@ struct kvm_pmu { struct kvm_pmu_ops; =20 enum { - KVM_DEBUGREG_BP_ENABLED =3D 1, - KVM_DEBUGREG_WONT_EXIT =3D 2, + KVM_DEBUGREG_BP_ENABLED =3D BIT(0), + KVM_DEBUGREG_WONT_EXIT =3D BIT(1), + /* + * Guest debug registers (DR0-3 and DR6) are saved/restored by hardware + * on exit from or enter to guest. KVM needn't switch them. Because DR7 + * is cleared on exit from guest, DR7 need to be saved/restored. + */ + KVM_DEBUGREG_AUTO_SWITCH =3D BIT(2), }; =20 struct kvm_mtrr_range { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 259139abb8ba..7465074a919d 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -523,6 +523,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) =20 vcpu->arch.efer =3D EFER_SCE | EFER_LME | EFER_LMA | EFER_NX; =20 + vcpu->arch.switch_db_regs =3D KVM_DEBUGREG_AUTO_SWITCH; vcpu->arch.cr0_guest_owned_bits =3D -1ul; vcpu->arch.cr4_guest_owned_bits =3D -1ul; =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 7805987d891d..c7d34b04ccdf 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10769,7 +10769,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (vcpu->arch.guest_fpu.xfd_err) wrmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err); =20 - if (unlikely(vcpu->arch.switch_db_regs)) { + if (unlikely(vcpu->arch.switch_db_regs & ~KVM_DEBUGREG_AUTO_SWITCH)) { set_debugreg(0, 7); set_debugreg(vcpu->arch.eff_db[0], 0); set_debugreg(vcpu->arch.eff_db[1], 1); @@ -10815,6 +10815,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) */ if (unlikely(vcpu->arch.switch_db_regs & KVM_DEBUGREG_WONT_EXIT)) { WARN_ON(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP); + WARN_ON(vcpu->arch.switch_db_regs & KVM_DEBUGREG_AUTO_SWITCH); static_call(kvm_x86_sync_dirty_debug_regs)(vcpu); kvm_update_dr0123(vcpu); kvm_update_dr7(vcpu); @@ -10827,8 +10828,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) * care about the messed up debug address registers. But if * we have some of them active, restore the old state. */ - if (hw_breakpoint_active()) - hw_breakpoint_restore(); + if (hw_breakpoint_active()) { + if (!(vcpu->arch.switch_db_regs & KVM_DEBUGREG_AUTO_SWITCH)) + hw_breakpoint_restore(); + else + set_debugreg(__this_cpu_read(cpu_dr7), 7); + } =20 vcpu->arch.last_vmentry_cpu =3D vcpu->cpu; vcpu->arch.last_guest_tsc =3D kvm_read_l1_tsc(vcpu, rdtsc()); --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7D7DC0015E for ; Tue, 25 Jul 2023 22:22:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232640AbjGYWWC (ORCPT ); Tue, 25 Jul 2023 18:22:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232542AbjGYWUG (ORCPT ); Tue, 25 Jul 2023 18:20:06 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 828602D71; Tue, 25 Jul 2023 15:17:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323433; x=1721859433; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fgffzls/qwB5q16Y/ZfsHgdKV+Ez5HD/rK8SyeB0Yw8=; b=gDkSsWso+xbHheaPqijZlvB1xlxoaelUrEWYZpvKwYmgwP54X5yC+bwx NS/W8IIEUSeaoqlKQNQ3LEBXECikAlGCgZzd6wT+ZQIOXk2samOFXA414 dpry24SASNQTGFPn8VIMrtCOr6FDVmdHD8fpgTK6NEgws1UzYGFVlQQ8N PEhFFYKtsvT8f8q0UyBSCgjCDx+RaD7mgo9f0Vp0YpPfR4y6EXRs/CABA Q9G5LPs8Xl7aOtnRxfVRJ91NWKlc5Dc39Egw66NVam9eIw0Afs9TtPJF5 p+i4qV/7mr09E7izO2GLjjcxMhXiZNpAFVHtEN+iFACsv74l2edixLRvc g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882596" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882596" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001826" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001826" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:55 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 074/115] KVM: TDX: Add support for find pending IRQ in a protected local APIC Date: Tue, 25 Jul 2023 15:14:25 -0700 Message-Id: <392163ce8a03a3fe1f086dbda45e85e896302394.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Add flag and hook to KVM's local APIC management to support determining whether or not a TDX guest as a pending IRQ. For TDX vCPUs, the virtual APIC page is owned by the TDX module and cannot be accessed by KVM. As a result, registers that are virtualized by the CPU, e.g. PPR, cannot be read or written by KVM. To deliver interrupts for TDX guests, KVM must send an IRQ to the CPU on the posted interrupt notification vector. And to determine if TDX vCPU has a pending interrupt, KVM must check if there is an outstanding notification. Return "no interrupt" in kvm_apic_has_interrupt() if the guest APIC is protected to short-circuit the various other flows that try to pull an IRQ out of the vAPIC, the only valid operation is querying _if_ an IRQ is pending, KVM can't do anything based on _which_ IRQ is pending. Intentionally omit sanity checks from other flows, e.g. PPR update, so as not to degrade non-TDX guests with unnecessary checks. A well-behaved KVM and userspace will never reach those flows for TDX guests, but reaching them is not fatal if something does go awry. Note, this doesn't handle interrupts that have been delivered to the vCPU but not yet recognized by the core, i.e. interrupts that are sitting in vmcs.GUEST_INTR_STATUS. Querying that state requires a SEAMCALL and will be supported in a future patch. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/irq.c | 3 +++ arch/x86/kvm/lapic.c | 3 +++ arch/x86/kvm/lapic.h | 2 ++ arch/x86/kvm/vmx/main.c | 10 ++++++++++ arch/x86/kvm/vmx/tdx.c | 6 ++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 8 files changed, 28 insertions(+) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index 49f19cfeb11d..663a40418434 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -120,6 +120,7 @@ KVM_X86_OP_OPTIONAL(pi_update_irte) KVM_X86_OP_OPTIONAL(pi_start_assignment) KVM_X86_OP_OPTIONAL(apicv_post_state_restore) KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt) +KVM_X86_OP_OPTIONAL(protected_apic_has_interrupt) KVM_X86_OP_OPTIONAL(set_hv_timer) KVM_X86_OP_OPTIONAL(cancel_hv_timer) KVM_X86_OP(setup_mce) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 42ddf087fe60..8bd2d7df15f9 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1757,6 +1757,7 @@ struct kvm_x86_ops { void (*pi_start_assignment)(struct kvm *kvm); void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu); bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu); + bool (*protected_apic_has_interrupt)(struct kvm_vcpu *vcpu); =20 int (*set_hv_timer)(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, bool *expired); diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c index b2c397dd2bc6..fd6af5530c32 100644 --- a/arch/x86/kvm/irq.c +++ b/arch/x86/kvm/irq.c @@ -100,6 +100,9 @@ int kvm_cpu_has_interrupt(struct kvm_vcpu *v) if (kvm_cpu_has_extint(v)) return 1; =20 + if (lapic_in_kernel(v) && v->arch.apic->guest_apic_protected) + return static_call(kvm_x86_protected_apic_has_interrupt)(v); + return kvm_apic_has_interrupt(v) !=3D -1; /* LAPIC */ } EXPORT_SYMBOL_GPL(kvm_cpu_has_interrupt); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 113ca9661ab2..d74d5eedd262 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -2825,6 +2825,9 @@ int kvm_apic_has_interrupt(struct kvm_vcpu *vcpu) if (!kvm_apic_present(vcpu)) return -1; =20 + if (apic->guest_apic_protected) + return -1; + __apic_update_ppr(apic, &ppr); return apic_has_interrupt_for_ppr(apic, ppr); } diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 0a0ea4b5dd8c..749b7b629c47 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -66,6 +66,8 @@ struct kvm_lapic { bool sw_enabled; bool irr_pending; bool lvt0_in_nmi_mode; + /* Select registers in the vAPIC cannot be read/written. */ + bool guest_apic_protected; /* Number of bits set in ISR. */ s16 isr_count; /* The highest vector set in ISR; if -1 - invalid, must scan ISR. */ diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index a0570ff1eae1..0403fb4621e9 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -94,6 +94,8 @@ static __init int vt_hardware_setup(void) =20 if (enable_tdx) vt_x86_ops.flush_remote_tlbs =3D vt_flush_remote_tlbs; + else + vt_x86_ops.protected_apic_has_interrupt =3D NULL; =20 return 0; } @@ -230,6 +232,13 @@ static void vt_vcpu_load(struct kvm_vcpu *vcpu, int cp= u) vmx_vcpu_load(vcpu, cpu); } =20 +static bool vt_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) +{ + KVM_BUG_ON(!is_td_vcpu(vcpu), vcpu->kvm); + + return tdx_protected_apic_has_interrupt(vcpu); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -420,6 +429,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .sync_pir_to_irr =3D vmx_sync_pir_to_irr, .deliver_interrupt =3D vmx_deliver_interrupt, .dy_apicv_has_pending_interrupt =3D pi_has_pending_interrupt, + .protected_apic_has_interrupt =3D vt_protected_apic_has_interrupt, =20 .set_tss_addr =3D vmx_set_tss_addr, .set_identity_map_addr =3D vmx_set_identity_map_addr, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 7465074a919d..0afffdbf24e0 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -520,6 +520,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) return -EINVAL; =20 fpstate_set_confidential(&vcpu->arch.guest_fpu); + vcpu->arch.apic->guest_apic_protected =3D true; =20 vcpu->arch.efer =3D EFER_SCE | EFER_LME | EFER_LMA | EFER_NX; =20 @@ -558,6 +559,11 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) local_irq_enable(); } =20 +bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) +{ + return pi_has_pending_interrupt(vcpu); +} + void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 231da434a08b..055cc3ad93ff 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -156,6 +156,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu); void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); void tdx_vcpu_put(struct kvm_vcpu *vcpu); void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); +bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); @@ -188,6 +189,7 @@ static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *= vcpu) { return EXIT_FASTP static inline void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_put(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {} +static inline bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)= { return false; } static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3C33C0015E for ; Tue, 25 Jul 2023 22:22:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232301AbjGYWV7 (ORCPT ); Tue, 25 Jul 2023 18:21:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232540AbjGYWUG (ORCPT ); Tue, 25 Jul 2023 18:20:06 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C781749E5; Tue, 25 Jul 2023 15:17:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323433; x=1721859433; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BKB85qgfkyW6H3ZX55etTyd0dqN/2ACXGcD32PGXBas=; b=FeXT+nqw/GnHWRy/UMSfzPYXRfrn+lJf2Bi30mYLTLngUMKFsONGkdcE y71NSYES/QUkOCg1sJKw1b5LiExuXnKTsaEtjlU5rd71HCf6Ya7Ufvqfo 8CicQU+X4fM10CjzZH+u42tuLAPski51mfZAXhTD7tNJDzXZeHVGFxGSG ZeXhL3x/nd6XztTuh88RvYufPm2BxMNFbSa23gi5Kz3Kd0g565mpV0jfc 8sjitonDbnlVhBkVFUVocZFKjNoN3BDHIZd6qq6pqJLFsVI2IodoQdSWy TO0sT8zrQZeh43iTVorfdZrQARpmHvAMYmqX2SfivPK7aySD50+8I+oQm w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882605" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882605" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001830" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001830" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:55 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 075/115] KVM: x86: Assume timer IRQ was injected if APIC state is proteced Date: Tue, 25 Jul 2023 15:14:26 -0700 Message-Id: <74b1fb68ef26552d542c10f3c8eb1ba4f4926afb.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson If APIC state is protected, i.e. the vCPU is a TDX guest, assume a timer IRQ was injected when deciding whether or not to busy wait in the "timer advanced" path. The "real" vIRR is not readable/writable, so trying to query for a pending timer IRQ will return garbage. Note, TDX can scour the PIR if it wants to be more precise and skip the "wait" call entirely. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/lapic.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index d74d5eedd262..d2d1a9531c96 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -1745,8 +1745,17 @@ static void apic_update_lvtt(struct kvm_lapic *apic) static bool lapic_timer_int_injected(struct kvm_vcpu *vcpu) { struct kvm_lapic *apic =3D vcpu->arch.apic; - u32 reg =3D kvm_lapic_get_reg(apic, APIC_LVTT); + u32 reg; =20 + /* + * Assume a timer IRQ was "injected" if the APIC is protected. KVM's + * copy of the vIRR is bogus, it's the responsibility of the caller to + * precisely check whether or not a timer IRQ is pending. + */ + if (apic->guest_apic_protected) + return true; + + reg =3D kvm_lapic_get_reg(apic, APIC_LVTT); if (kvm_apic_hw_enabled(apic)) { int vec =3D reg & APIC_VECTOR_MASK; void *bitmap =3D apic->regs + APIC_ISR; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B863EEB64DD for ; Tue, 25 Jul 2023 22:22:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232653AbjGYWWG (ORCPT ); Tue, 25 Jul 2023 18:22:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231280AbjGYWUg (ORCPT ); Tue, 25 Jul 2023 18:20:36 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C79932D7B; Tue, 25 Jul 2023 15:17:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323436; x=1721859436; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KRXC9I21JsF+xyBURR85AdI/MBUQ0zzUdk4fG5aW4II=; b=BJG5KHrSEgCmh6xOcOcWl3R34j+VebLbUmxNvn1Bjn/nC4/YHdz/XR+v pEALLeUDgjfM646wU+UH6Iwd7M5DT2G2VV2n5biC0IlG+XCHCAky4qNUE 4UlWerafwtGG8FemZuV125mq8Vkxgqr1tcZynbUqqn05hHjnIaS/TyuMd chZt0bgWekdovTmDOqVcSpZ2bdtWNu7Qt3pEbJQ0DNMtve6wmYc+/EWdD oenlAoWwBJeTnL9JP6RgsCxxV/nNSHXwXq0W6ZlDL1A5W7zuMKWR5vyY4 NPJhllBaECPZCfjC+ZeARbmdiKLqbU+E3ydLwTbMIkM0CYl9nMOe8Xwp+ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882608" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882608" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001835" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001835" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:56 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 076/115] KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c Date: Tue, 25 Jul 2023 15:14:27 -0700 Message-Id: <128b752995728d5b39db8c71056d2d346ffb59f4.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata As TDX will use posted_interrupt.c, the use of struct vcpu_vmx is a blocker. Because the members of struct pi_desc pi_desc and struct list_head pi_wakeup_list are only used in posted_interrupt.c, introduce common structure, struct vcpu_pi, make vcpu_vmx and vcpu_tdx has same layout in the top of structure. To minimize the diff size, avoid code conversion like, vmx->pi_desc =3D> vmx->common->pi_desc. Instead add compile time check if the layout is expected. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/posted_intr.c | 41 ++++++++++++++++++++++++++-------- arch/x86/kvm/vmx/posted_intr.h | 11 +++++++++ arch/x86/kvm/vmx/tdx.c | 1 + arch/x86/kvm/vmx/tdx.h | 8 +++++++ arch/x86/kvm/vmx/vmx.h | 14 +++++++----- 5 files changed, 60 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/vmx/posted_intr.c b/arch/x86/kvm/vmx/posted_intr.c index 94c38bea60e7..92de016852ca 100644 --- a/arch/x86/kvm/vmx/posted_intr.c +++ b/arch/x86/kvm/vmx/posted_intr.c @@ -11,6 +11,7 @@ #include "posted_intr.h" #include "trace.h" #include "vmx.h" +#include "tdx.h" =20 /* * Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_han= dler() @@ -31,9 +32,29 @@ static DEFINE_PER_CPU(struct list_head, wakeup_vcpus_on_= cpu); */ static DEFINE_PER_CPU(raw_spinlock_t, wakeup_vcpus_on_cpu_lock); =20 +/* + * The layout of the head of struct vcpu_vmx and struct vcpu_tdx must matc= h with + * struct vcpu_pi. + */ +static_assert(offsetof(struct vcpu_pi, pi_desc) =3D=3D + offsetof(struct vcpu_vmx, pi_desc)); +static_assert(offsetof(struct vcpu_pi, pi_wakeup_list) =3D=3D + offsetof(struct vcpu_vmx, pi_wakeup_list)); +#ifdef CONFIG_INTEL_TDX_HOST +static_assert(offsetof(struct vcpu_pi, pi_desc) =3D=3D + offsetof(struct vcpu_tdx, pi_desc)); +static_assert(offsetof(struct vcpu_pi, pi_wakeup_list) =3D=3D + offsetof(struct vcpu_tdx, pi_wakeup_list)); +#endif + +static inline struct vcpu_pi *vcpu_to_pi(struct kvm_vcpu *vcpu) +{ + return (struct vcpu_pi *)vcpu; +} + static inline struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu) { - return &(to_vmx(vcpu)->pi_desc); + return &vcpu_to_pi(vcpu)->pi_desc; } =20 static int pi_try_set_control(struct pi_desc *pi_desc, u64 *pold, u64 new) @@ -52,8 +73,8 @@ static int pi_try_set_control(struct pi_desc *pi_desc, u6= 4 *pold, u64 new) =20 void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) { - struct pi_desc *pi_desc =3D vcpu_to_pi_desc(vcpu); - struct vcpu_vmx *vmx =3D to_vmx(vcpu); + struct vcpu_pi *vcpu_pi =3D vcpu_to_pi(vcpu); + struct pi_desc *pi_desc =3D &vcpu_pi->pi_desc; struct pi_desc old, new; unsigned long flags; unsigned int dest; @@ -90,7 +111,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) */ if (pi_desc->nv =3D=3D POSTED_INTR_WAKEUP_VECTOR) { raw_spin_lock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu)); - list_del(&vmx->pi_wakeup_list); + list_del(&vcpu_pi->pi_wakeup_list); raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu)); } =20 @@ -145,15 +166,15 @@ static bool vmx_can_use_vtd_pi(struct kvm *kvm) */ static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu) { - struct pi_desc *pi_desc =3D vcpu_to_pi_desc(vcpu); - struct vcpu_vmx *vmx =3D to_vmx(vcpu); + struct vcpu_pi *vcpu_pi =3D vcpu_to_pi(vcpu); + struct pi_desc *pi_desc =3D &vcpu_pi->pi_desc; struct pi_desc old, new; unsigned long flags; =20 local_irq_save(flags); =20 raw_spin_lock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu)); - list_add_tail(&vmx->pi_wakeup_list, + list_add_tail(&vcpu_pi->pi_wakeup_list, &per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu)); raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu)); =20 @@ -190,7 +211,8 @@ static bool vmx_needs_pi_wakeup(struct kvm_vcpu *vcpu) * notification vector is switched to the one that calls * back to the pi_wakeup_handler() function. */ - return vmx_can_use_ipiv(vcpu) || vmx_can_use_vtd_pi(vcpu->kvm); + return (vmx_can_use_ipiv(vcpu) && !is_td_vcpu(vcpu)) || + vmx_can_use_vtd_pi(vcpu->kvm); } =20 void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu) @@ -200,7 +222,8 @@ void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu) if (!vmx_needs_pi_wakeup(vcpu)) return; =20 - if (kvm_vcpu_is_blocking(vcpu) && !vmx_interrupt_blocked(vcpu)) + if (kvm_vcpu_is_blocking(vcpu) && + (is_td_vcpu(vcpu) || !vmx_interrupt_blocked(vcpu))) pi_enable_wakeup_handler(vcpu); =20 /* diff --git a/arch/x86/kvm/vmx/posted_intr.h b/arch/x86/kvm/vmx/posted_intr.h index 26992076552e..2fe8222308b2 100644 --- a/arch/x86/kvm/vmx/posted_intr.h +++ b/arch/x86/kvm/vmx/posted_intr.h @@ -94,6 +94,17 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) (unsigned long *)&pi_desc->control); } =20 +struct vcpu_pi { + struct kvm_vcpu vcpu; + + /* Posted interrupt descriptor */ + struct pi_desc pi_desc; + + /* Used if this vCPU is waiting for PI notification wakeup. */ + struct list_head pi_wakeup_list; + /* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ +}; + void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu); void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu); void pi_wakeup_handler(void); diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0afffdbf24e0..8ffef0dac24e 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -521,6 +521,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) =20 fpstate_set_confidential(&vcpu->arch.guest_fpu); vcpu->arch.apic->guest_apic_protected =3D true; + INIT_LIST_HEAD(&tdx->pi_wakeup_list); =20 vcpu->arch.efer =3D EFER_SCE | EFER_LME | EFER_LMA | EFER_NX; =20 diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index da7e83dc34b8..45c1df7b2e40 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -4,6 +4,7 @@ =20 #ifdef CONFIG_INTEL_TDX_HOST =20 +#include "posted_intr.h" #include "pmu_intel.h" #include "tdx_ops.h" =20 @@ -67,6 +68,13 @@ union tdx_exit_reason { struct vcpu_tdx { struct kvm_vcpu vcpu; =20 + /* Posted interrupt descriptor */ + struct pi_desc pi_desc; + + /* Used if this vCPU is waiting for PI notification wakeup. */ + struct list_head pi_wakeup_list; + /* Until here same layout to struct vcpu_pi. */ + unsigned long tdvpr_pa; unsigned long *tdvpx_pa; =20 diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 0c97328fc3d5..3802b1dcbc1c 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -233,6 +233,14 @@ struct nested_vmx { =20 struct vcpu_vmx { struct kvm_vcpu vcpu; + + /* Posted interrupt descriptor */ + struct pi_desc pi_desc; + + /* Used if this vCPU is waiting for PI notification wakeup. */ + struct list_head pi_wakeup_list; + /* Until here same layout to struct vcpu_pi. */ + u8 fail; u8 x2apic_msr_bitmap_mode; =20 @@ -302,12 +310,6 @@ struct vcpu_vmx { =20 union vmx_exit_reason exit_reason; =20 - /* Posted interrupt descriptor */ - struct pi_desc pi_desc; - - /* Used if this vCPU is waiting for PI notification wakeup. */ - struct list_head pi_wakeup_list; - /* Support for a guest hypervisor (nested VMX) */ struct nested_vmx nested; =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCD95C0015E for ; Tue, 25 Jul 2023 22:22:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232660AbjGYWWI (ORCPT ); Tue, 25 Jul 2023 18:22:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232260AbjGYWUh (ORCPT ); Tue, 25 Jul 2023 18:20:37 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED54B49FE; Tue, 25 Jul 2023 15:17:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323436; x=1721859436; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZfqB1SnxnodMDVWq8T406o97THFJ57sjQgFRu6v2yAQ=; b=fEG+frNETI5P0hQEiWO6aDwoagtkxZv9v4x10aHLDaGIHC5AxC0D+ANF wWawH/Ui87XT2lk9ddKe89JNQKiXhPcH7KNAXjIQqUXQtDuGtgXBg8TEo v/ymaqHDP33QSVz5VfWYzBVcQz7kXqASQ9M1bZeRbvzX3YyhBS+b+9z36 Zo7aFXD7RChg490mZYuFR/BQMuLHqjf9ko20Zd+LAyma9jZakiIMR7rex gbF+fYI6rwWFwPf6CMsZBnFER/6e3y7aDCws2rNk4TjmiyAeH83w50bRV IbSYaVon+edvtmOJRalT3lLMVdkXKHYS1842EeVK3TRzRXUEZJrztITBA w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882614" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882614" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001843" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001843" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:56 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 077/115] KVM: TDX: Implement interrupt injection Date: Tue, 25 Jul 2023 15:14:28 -0700 Message-Id: <82c391f099a7981bcbdf7ab32445d4549d4a0c1c.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX supports interrupt inject into vcpu with posted interrupt. Wire up the corresponding kvm x86 operations to posted interrupt. Move kvm_vcpu_trigger_posted_interrupt() from vmx.c to common.h to share the code. VMX can inject interrupt by setting interrupt information field, VM_ENTRY_INTR_INFO_FIELD, of VMCS. TDX supports interrupt injection only by posted interrupt. Ignore the execution path to access VM_ENTRY_INTR_INFO_FIELD. As cpu state is protected and apicv is enabled for the TDX guest, VMM can inject interrupt by updating posted interrupt descriptor. Treat interrupt can be injected always. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/common.h | 71 ++++++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 93 ++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/posted_intr.c | 2 +- arch/x86/kvm/vmx/posted_intr.h | 2 + arch/x86/kvm/vmx/tdx.c | 25 +++++++++ arch/x86/kvm/vmx/vmx.c | 67 +----------------------- arch/x86/kvm/vmx/x86_ops.h | 7 ++- 7 files changed, 190 insertions(+), 77 deletions(-) diff --git a/arch/x86/kvm/vmx/common.h b/arch/x86/kvm/vmx/common.h index 235908f3e044..747f993cf7de 100644 --- a/arch/x86/kvm/vmx/common.h +++ b/arch/x86/kvm/vmx/common.h @@ -4,6 +4,7 @@ =20 #include =20 +#include "posted_intr.h" #include "mmu.h" =20 static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t = gpa, @@ -30,4 +31,74 @@ static inline int __vmx_handle_ept_violation(struct kvm_= vcpu *vcpu, gpa_t gpa, return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); } =20 +static inline void kvm_vcpu_trigger_posted_interrupt(struct kvm_vcpu *vcpu, + int pi_vec) +{ +#ifdef CONFIG_SMP + if (vcpu->mode =3D=3D IN_GUEST_MODE) { + /* + * The vector of the virtual has already been set in the PIR. + * Send a notification event to deliver the virtual interrupt + * unless the vCPU is the currently running vCPU, i.e. the + * event is being sent from a fastpath VM-Exit handler, in + * which case the PIR will be synced to the vIRR before + * re-entering the guest. + * + * When the target is not the running vCPU, the following + * possibilities emerge: + * + * Case 1: vCPU stays in non-root mode. Sending a notification + * event posts the interrupt to the vCPU. + * + * Case 2: vCPU exits to root mode and is still runnable. The + * PIR will be synced to the vIRR before re-entering the guest. + * Sending a notification event is ok as the host IRQ handler + * will ignore the spurious event. + * + * Case 3: vCPU exits to root mode and is blocked. vcpu_block() + * has already synced PIR to vIRR and never blocks the vCPU if + * the vIRR is not empty. Therefore, a blocked vCPU here does + * not wait for any requested interrupts in PIR, and sending a + * notification event also results in a benign, spurious event. + */ + + if (vcpu !=3D kvm_get_running_vcpu()) + apic->send_IPI_mask(get_cpu_mask(vcpu->cpu), pi_vec); + return; + } +#endif + /* + * The vCPU isn't in the guest; wake the vCPU in case it is blocking, + * otherwise do nothing as KVM will grab the highest priority pending + * IRQ via ->sync_pir_to_irr() in vcpu_enter_guest(). + */ + kvm_vcpu_wake_up(vcpu); +} + +/* + * Send interrupt to vcpu via posted interrupt way. + * 1. If target vcpu is running(non-root mode), send posted interrupt + * notification to vcpu and hardware will sync PIR to vIRR atomically. + * 2. If target vcpu isn't running(root mode), kick it to pick up the + * interrupt from PIR in next vmentry. + */ +static inline void __vmx_deliver_posted_interrupt(struct kvm_vcpu *vcpu, + struct pi_desc *pi_desc, int vector) +{ + if (pi_test_and_set_pir(vector, pi_desc)) + return; + + /* If a previous notification has sent the IPI, nothing to do. */ + if (pi_test_and_set_on(pi_desc)) + return; + + /* + * The implied barrier in pi_test_and_set_on() pairs with the smp_mb_*() + * after setting vcpu->mode in vcpu_enter_guest(), thus the vCPU is + * guaranteed to see PID.ON=3D1 and sync the PIR to IRR if triggering a + * posted interrupt "fails" because vcpu->mode !=3D IN_GUEST_MODE. + */ + kvm_vcpu_trigger_posted_interrupt(vcpu, POSTED_INTR_VECTOR); +} + #endif /* __KVM_X86_VMX_COMMON_H */ diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 0403fb4621e9..ec663813c479 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -239,6 +239,34 @@ static bool vt_protected_apic_has_interrupt(struct kvm= _vcpu *vcpu) return tdx_protected_apic_has_interrupt(vcpu); } =20 +static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu) +{ + struct pi_desc *pi =3D vcpu_to_pi_desc(vcpu); + + pi_clear_on(pi); + memset(pi->pir, 0, sizeof(pi->pir)); +} + +static int vt_sync_pir_to_irr(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return -1; + + return vmx_sync_pir_to_irr(vcpu); +} + +static void vt_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, + int trig_mode, int vector) +{ + if (is_td_vcpu(apic->vcpu)) { + tdx_deliver_interrupt(apic, delivery_mode, trig_mode, + vector); + return; + } + + vmx_deliver_interrupt(apic, delivery_mode, trig_mode, vector); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -306,6 +334,53 @@ static void vt_sched_in(struct kvm_vcpu *vcpu, int cpu) vmx_sched_in(vcpu, cpu); } =20 +static void vt_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask) +{ + if (is_td_vcpu(vcpu)) + return; + vmx_set_interrupt_shadow(vcpu, mask); +} + +static u32 vt_get_interrupt_shadow(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return 0; + + return vmx_get_interrupt_shadow(vcpu); +} + +static void vt_inject_irq(struct kvm_vcpu *vcpu, bool reinjected) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_inject_irq(vcpu, reinjected); +} + +static void vt_cancel_injection(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_cancel_injection(vcpu); +} + +static int vt_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection) +{ + if (is_td_vcpu(vcpu)) + return true; + + return vmx_interrupt_allowed(vcpu, for_injection); +} + +static void vt_enable_irq_window(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_enable_irq_window(vcpu); +} + static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { if (is_td_vcpu(vcpu)) @@ -403,31 +478,31 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .handle_exit =3D vmx_handle_exit, .skip_emulated_instruction =3D vmx_skip_emulated_instruction, .update_emulated_instruction =3D vmx_update_emulated_instruction, - .set_interrupt_shadow =3D vmx_set_interrupt_shadow, - .get_interrupt_shadow =3D vmx_get_interrupt_shadow, + .set_interrupt_shadow =3D vt_set_interrupt_shadow, + .get_interrupt_shadow =3D vt_get_interrupt_shadow, .patch_hypercall =3D vmx_patch_hypercall, - .inject_irq =3D vmx_inject_irq, + .inject_irq =3D vt_inject_irq, .inject_nmi =3D vmx_inject_nmi, .inject_exception =3D vmx_inject_exception, - .cancel_injection =3D vmx_cancel_injection, - .interrupt_allowed =3D vmx_interrupt_allowed, + .cancel_injection =3D vt_cancel_injection, + .interrupt_allowed =3D vt_interrupt_allowed, .nmi_allowed =3D vmx_nmi_allowed, .get_nmi_mask =3D vmx_get_nmi_mask, .set_nmi_mask =3D vmx_set_nmi_mask, .enable_nmi_window =3D vmx_enable_nmi_window, - .enable_irq_window =3D vmx_enable_irq_window, + .enable_irq_window =3D vt_enable_irq_window, .update_cr8_intercept =3D vmx_update_cr8_intercept, .set_virtual_apic_mode =3D vmx_set_virtual_apic_mode, .set_apic_access_page_addr =3D vmx_set_apic_access_page_addr, .refresh_apicv_exec_ctrl =3D vmx_refresh_apicv_exec_ctrl, .load_eoi_exitmap =3D vmx_load_eoi_exitmap, - .apicv_post_state_restore =3D vmx_apicv_post_state_restore, + .apicv_post_state_restore =3D vt_apicv_post_state_restore, .required_apicv_inhibits =3D VMX_REQUIRED_APICV_INHIBITS, .hwapic_irr_update =3D vmx_hwapic_irr_update, .hwapic_isr_update =3D vmx_hwapic_isr_update, .guest_apic_has_interrupt =3D vmx_guest_apic_has_interrupt, - .sync_pir_to_irr =3D vmx_sync_pir_to_irr, - .deliver_interrupt =3D vmx_deliver_interrupt, + .sync_pir_to_irr =3D vt_sync_pir_to_irr, + .deliver_interrupt =3D vt_deliver_interrupt, .dy_apicv_has_pending_interrupt =3D pi_has_pending_interrupt, .protected_apic_has_interrupt =3D vt_protected_apic_has_interrupt, =20 diff --git a/arch/x86/kvm/vmx/posted_intr.c b/arch/x86/kvm/vmx/posted_intr.c index 92de016852ca..2b2da6c18504 100644 --- a/arch/x86/kvm/vmx/posted_intr.c +++ b/arch/x86/kvm/vmx/posted_intr.c @@ -52,7 +52,7 @@ static inline struct vcpu_pi *vcpu_to_pi(struct kvm_vcpu = *vcpu) return (struct vcpu_pi *)vcpu; } =20 -static inline struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu) +struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu) { return &vcpu_to_pi(vcpu)->pi_desc; } diff --git a/arch/x86/kvm/vmx/posted_intr.h b/arch/x86/kvm/vmx/posted_intr.h index 2fe8222308b2..0f9983b6910b 100644 --- a/arch/x86/kvm/vmx/posted_intr.h +++ b/arch/x86/kvm/vmx/posted_intr.h @@ -105,6 +105,8 @@ struct vcpu_pi { /* Until here common layout betwwn vcpu_vmx and vcpu_tdx. */ }; =20 +struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu); + void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu); void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu); void pi_wakeup_handler(void); diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 8ffef0dac24e..a417151dad92 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -7,6 +7,7 @@ =20 #include "capabilities.h" #include "x86_ops.h" +#include "common.h" #include "mmu.h" #include "tdx.h" #include "vmx.h" @@ -534,6 +535,9 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) vcpu->arch.guest_state_protected =3D !(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG); =20 + tdx->pi_desc.nv =3D POSTED_INTR_VECTOR; + tdx->pi_desc.sn =3D 1; + tdx->host_state_need_save =3D true; tdx->host_state_need_restore =3D false; =20 @@ -544,6 +548,7 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); =20 + vmx_vcpu_pi_load(vcpu, cpu); if (vcpu->cpu =3D=3D cpu) return; =20 @@ -729,6 +734,12 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) =20 trace_kvm_entry(vcpu); =20 + if (pi_test_on(&tdx->pi_desc)) { + apic->send_IPI_self(POSTED_INTR_VECTOR); + + kvm_wait_lapic_expire(vcpu); + } + tdx_vcpu_enter_exit(vcpu, tdx); =20 tdx_user_return_update_cache(vcpu); @@ -1065,6 +1076,16 @@ static int tdx_sept_remove_private_spte(struct kvm *= kvm, gfn_t gfn, return tdx_sept_drop_private_spte(kvm, gfn, level, pfn); } =20 +void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, + int trig_mode, int vector) +{ + struct kvm_vcpu *vcpu =3D apic->vcpu; + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + /* TDX supports only posted interrupt. No lapic emulation. */ + __vmx_deliver_posted_interrupt(vcpu, &tdx->pi_desc, vector); +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; @@ -1862,6 +1883,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __use= r *argp) if (ret) return ret; =20 + td_vmcs_write16(tdx, POSTED_INTR_NV, POSTED_INTR_VECTOR); + td_vmcs_write64(tdx, POSTED_INTR_DESC_ADDR, __pa(&tdx->pi_desc)); + td_vmcs_setbit32(tdx, PIN_BASED_VM_EXEC_CONTROL, PIN_BASED_POSTED_INTR); + tdx->initialized =3D true; return 0; } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 26a762df2c23..7a72391d8133 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -4119,50 +4119,6 @@ void vmx_msr_filter_changed(struct kvm_vcpu *vcpu) pt_update_intercept_for_msr(vcpu); } =20 -static inline void kvm_vcpu_trigger_posted_interrupt(struct kvm_vcpu *vcpu, - int pi_vec) -{ -#ifdef CONFIG_SMP - if (vcpu->mode =3D=3D IN_GUEST_MODE) { - /* - * The vector of the virtual has already been set in the PIR. - * Send a notification event to deliver the virtual interrupt - * unless the vCPU is the currently running vCPU, i.e. the - * event is being sent from a fastpath VM-Exit handler, in - * which case the PIR will be synced to the vIRR before - * re-entering the guest. - * - * When the target is not the running vCPU, the following - * possibilities emerge: - * - * Case 1: vCPU stays in non-root mode. Sending a notification - * event posts the interrupt to the vCPU. - * - * Case 2: vCPU exits to root mode and is still runnable. The - * PIR will be synced to the vIRR before re-entering the guest. - * Sending a notification event is ok as the host IRQ handler - * will ignore the spurious event. - * - * Case 3: vCPU exits to root mode and is blocked. vcpu_block() - * has already synced PIR to vIRR and never blocks the vCPU if - * the vIRR is not empty. Therefore, a blocked vCPU here does - * not wait for any requested interrupts in PIR, and sending a - * notification event also results in a benign, spurious event. - */ - - if (vcpu !=3D kvm_get_running_vcpu()) - apic->send_IPI_mask(get_cpu_mask(vcpu->cpu), pi_vec); - return; - } -#endif - /* - * The vCPU isn't in the guest; wake the vCPU in case it is blocking, - * otherwise do nothing as KVM will grab the highest priority pending - * IRQ via ->sync_pir_to_irr() in vcpu_enter_guest(). - */ - kvm_vcpu_wake_up(vcpu); -} - static int vmx_deliver_nested_posted_interrupt(struct kvm_vcpu *vcpu, int vector) { @@ -4215,20 +4171,7 @@ static int vmx_deliver_posted_interrupt(struct kvm_v= cpu *vcpu, int vector) if (!vcpu->arch.apic->apicv_active) return -1; =20 - if (pi_test_and_set_pir(vector, &vmx->pi_desc)) - return 0; - - /* If a previous notification has sent the IPI, nothing to do. */ - if (pi_test_and_set_on(&vmx->pi_desc)) - return 0; - - /* - * The implied barrier in pi_test_and_set_on() pairs with the smp_mb_*() - * after setting vcpu->mode in vcpu_enter_guest(), thus the vCPU is - * guaranteed to see PID.ON=3D1 and sync the PIR to IRR if triggering a - * posted interrupt "fails" because vcpu->mode !=3D IN_GUEST_MODE. - */ - kvm_vcpu_trigger_posted_interrupt(vcpu, POSTED_INTR_VECTOR); + __vmx_deliver_posted_interrupt(vcpu, &vmx->pi_desc, vector); return 0; } =20 @@ -6917,14 +6860,6 @@ void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64= *eoi_exit_bitmap) vmcs_write64(EOI_EXIT_BITMAP3, eoi_exit_bitmap[3]); } =20 -void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) -{ - struct vcpu_vmx *vmx =3D to_vmx(vcpu); - - pi_clear_on(&vmx->pi_desc); - memset(vmx->pi_desc.pir, 0, sizeof(vmx->pi_desc.pir)); -} - void vmx_do_interrupt_irqoff(unsigned long entry); void vmx_do_nmi_irqoff(void); =20 diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 055cc3ad93ff..44d342dd59b9 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -60,7 +60,6 @@ int vmx_check_intercept(struct kvm_vcpu *vcpu, bool vmx_apic_init_signal_blocked(struct kvm_vcpu *vcpu); void vmx_migrate_timers(struct kvm_vcpu *vcpu); void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu); -void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu); bool vmx_check_apicv_inhibit_reasons(enum kvm_apicv_inhibit reason); void vmx_hwapic_irr_update(struct kvm_vcpu *vcpu, int max_irr); void vmx_hwapic_isr_update(int max_isr); @@ -159,6 +158,9 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 +void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, + int trig_mode, int vector); + int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 void tdx_flush_tlb(struct kvm_vcpu *vcpu); @@ -192,6 +194,9 @@ static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu,= int cpu) {} static inline bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)= { return false; } static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 +static inline void tdx_deliver_interrupt(struct kvm_lapic *apic, int deliv= ery_mode, + int trig_mode, int vector) {} + static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 static inline void tdx_flush_tlb(struct kvm_vcpu *vcpu) {} --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01F03C001DF for ; Tue, 25 Jul 2023 22:22:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232478AbjGYWWU (ORCPT ); Tue, 25 Jul 2023 18:22:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbjGYWVZ (ORCPT ); Tue, 25 Jul 2023 18:21:25 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1731D2D43; Tue, 25 Jul 2023 15:17:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323440; x=1721859440; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IiFdHvSfreQxASHx9FGH1Lj9RsEngLdIbtSKEO5CYZ8=; b=NUnMznXtCjMP/EqaTu1MxEfBEn2iSp87d3708qrgPYZQSBFuJRWq+p+a r6BqFTgvILM1RUeLGNKf0OU2AVltw0lhkHPlfWPl+E2U3rfcl4NnF+ccI F5nGpBZEFMtjR2z8epeAVc4vQZlhj944wbdywkPEM6QIMVbBBtDmKeG1h VLB2eSXuUDVzb/DEZJ97qO8PM3xSuLgtg2V3Mzuo1IRqooylBZUSmlTLV joeG092H5Iprq2YQA/VyEsxwugiu/q9T4l/+MIMCOl+hXUseznCBL0h6n Kid3UACzsDArAIKJHvOPMcc+IhumapClxWptqtld2wwWJ9tej+swyMNaJ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882621" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882621" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001849" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001849" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:56 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 078/115] KVM: TDX: Implements vcpu request_immediate_exit Date: Tue, 25 Jul 2023 15:14:29 -0700 Message-Id: <1adaddb9fd96bc5a9783cdcb82bd445e2d0304f2.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Now we are able to inject interrupts into TDX vcpu, it's ready to block TDX vcpu. Wire up kvm x86 methods for blocking/unblocking vcpu for TDX. To unblock on pending events, request immediate exit methods is also needed. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/main.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index ec663813c479..70112388276b 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -381,6 +381,16 @@ static void vt_enable_irq_window(struct kvm_vcpu *vcpu) vmx_enable_irq_window(vcpu); } =20 +static void vt_request_immediate_exit(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + __kvm_request_immediate_exit(vcpu); + return; + } + + vmx_request_immediate_exit(vcpu); +} + static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { if (is_td_vcpu(vcpu)) @@ -526,7 +536,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .check_intercept =3D vmx_check_intercept, .handle_exit_irqoff =3D vmx_handle_exit_irqoff, =20 - .request_immediate_exit =3D vmx_request_immediate_exit, + .request_immediate_exit =3D vt_request_immediate_exit, =20 .sched_in =3D vt_sched_in, =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7172EB64DD for ; Tue, 25 Jul 2023 22:22:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232502AbjGYWWg (ORCPT ); Tue, 25 Jul 2023 18:22:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232492AbjGYWVb (ORCPT ); Tue, 25 Jul 2023 18:21:31 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC4EE30E7; Tue, 25 Jul 2023 15:17:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323444; x=1721859444; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=k9u7yXth60rYaU6FTaPegCXo6uCGVT6h79eiTo2Epvc=; b=QFUG8XhSwP+Pm7Wr2LB31GlxlWKf2X+xDae0Mnzbc7vSzd+RwXwoZoX5 cuETuzngzXWFV3ewnY6T8omWLq43UPyF80Kw4ib9znoeXATBjgzewBPGy UV4kqHKg0N8rTJyN9jChyzbP+0ItvQ7JUl0j3kc3K1pyqHfEp89+e3ev9 q5XZ/63ButzXBy59yajMtI8auMZATmLRfPrcko1Pv+s+GLrm5MyZZCUwf LrETC5eZcOo3Sr0H+9pWzbt/HwxbTXkwlHcZIbQbQrOVCLfJtjBuzMl7Q y3B6rDbG/JVtd0Ci+wAG3MK46Mz9hArRQUOlQ/d19KincmT07LV3WNL// w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882628" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882628" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001852" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001852" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:57 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 079/115] KVM: TDX: Implement methods to inject NMI Date: Tue, 25 Jul 2023 15:14:30 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX vcpu control structure defines one bit for pending NMI for VMM to inject NMI by setting the bit without knowing TDX vcpu NMI states. Because the vcpu state is protected, VMM can't know about NMI states of TDX vcpu. The TDX module handles actual injection and NMI states transition. Add methods for NMI and treat NMI can be injected always. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/main.c | 64 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 5 +++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ 3 files changed, 66 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 70112388276b..64a012110515 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -315,6 +315,60 @@ static void vt_flush_tlb_guest(struct kvm_vcpu *vcpu) vmx_flush_tlb_guest(vcpu); } =20 +static void vt_inject_nmi(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_inject_nmi(vcpu); + return; + } + + vmx_inject_nmi(vcpu); +} + +static int vt_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection) +{ + /* + * The TDX module manages NMI windows and NMI reinjection, and hides NMI + * blocking, all KVM can do is throw an NMI over the wall. + */ + if (is_td_vcpu(vcpu)) + return true; + + return vmx_nmi_allowed(vcpu, for_injection); +} + +static bool vt_get_nmi_mask(struct kvm_vcpu *vcpu) +{ + /* + * Assume NMIs are always unmasked. KVM could query PEND_NMI and treat + * NMIs as masked if a previous NMI is still pending, but SEAMCALLs are + * expensive and the end result is unchanged as the only relevant usage + * of get_nmi_mask() is to limit the number of pending NMIs, i.e. it + * only changes whether KVM or the TDX module drops an NMI. + */ + if (is_td_vcpu(vcpu)) + return false; + + return vmx_get_nmi_mask(vcpu); +} + +static void vt_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_nmi_mask(vcpu, masked); +} + +static void vt_enable_nmi_window(struct kvm_vcpu *vcpu) +{ + /* Refer the comment in vt_get_nmi_mask(). */ + if (is_td_vcpu(vcpu)) + return; + + vmx_enable_nmi_window(vcpu); +} + static void vt_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { @@ -492,14 +546,14 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .get_interrupt_shadow =3D vt_get_interrupt_shadow, .patch_hypercall =3D vmx_patch_hypercall, .inject_irq =3D vt_inject_irq, - .inject_nmi =3D vmx_inject_nmi, + .inject_nmi =3D vt_inject_nmi, .inject_exception =3D vmx_inject_exception, .cancel_injection =3D vt_cancel_injection, .interrupt_allowed =3D vt_interrupt_allowed, - .nmi_allowed =3D vmx_nmi_allowed, - .get_nmi_mask =3D vmx_get_nmi_mask, - .set_nmi_mask =3D vmx_set_nmi_mask, - .enable_nmi_window =3D vmx_enable_nmi_window, + .nmi_allowed =3D vt_nmi_allowed, + .get_nmi_mask =3D vt_get_nmi_mask, + .set_nmi_mask =3D vt_set_nmi_mask, + .enable_nmi_window =3D vt_enable_nmi_window, .enable_irq_window =3D vt_enable_irq_window, .update_cr8_intercept =3D vmx_update_cr8_intercept, .set_virtual_apic_mode =3D vmx_set_virtual_apic_mode, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index a417151dad92..d2a20ac36999 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -755,6 +755,11 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) return EXIT_FASTPATH_NONE; } =20 +void tdx_inject_nmi(struct kvm_vcpu *vcpu) +{ + td_management_write8(to_tdx(vcpu), TD_VCPU_PEND_NMI, 1); +} + void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 44d342dd59b9..ef93b30750ce 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -160,6 +160,7 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bo= ol is_mmio); =20 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, int trig_mode, int vector); +void tdx_inject_nmi(struct kvm_vcpu *vcpu); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 @@ -196,6 +197,7 @@ static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu,= gfn_t gfn, bool is_mmio) =20 static inline void tdx_deliver_interrupt(struct kvm_lapic *apic, int deliv= ery_mode, int trig_mode, int vector) {} +static inline void tdx_inject_nmi(struct kvm_vcpu *vcpu) {} =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE94FEB64DD for ; Tue, 25 Jul 2023 22:22:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232731AbjGYWWc (ORCPT ); Tue, 25 Jul 2023 18:22:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232499AbjGYWVc (ORCPT ); Tue, 25 Jul 2023 18:21:32 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53CF630F2; Tue, 25 Jul 2023 15:17:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323446; x=1721859446; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qKbGFOwuKt5AHiw8B939WZUTvZiMGlmCpoxGrITv5jI=; b=iS77jPu1DciWBuyTXU2vwdk+NY2q/hTioq7UoefsGtzOhnxU4Zk0JSUb nTiic+ryvy0g2agFpJvgrRoVFfYSU7Gm2qgi+F8fppW1DJw9kz1GMNWrP abaX2fAccIhyLRwcbx2fxKPITeIKU3q8XDDslVp0uH/SHrfHcd0e77Ogk kAHMM9soYMztb/aSCFk4u4PJJfVm3JLi9i26dlwhaQvFrrO4JozMMpL1R /RwjTLoLStLw49jWXVpV4aAw/rpNGEpSSa3LNL3yTawwyy0UhSyDn9GIJ eZQzOsJGpM+7+88eeJV9Mg8rmfhVIwCOHOT1E26Yb64Y3nFVYKN12qJdK Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882637" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882637" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001855" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001855" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:57 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 080/115] KVM: VMX: Modify NMI and INTR handlers to take intr_info as function argument Date: Tue, 25 Jul 2023 15:14:31 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson TDX uses different ABI to get information about VM exit. Pass intr_info to the NMI and INTR handlers instead of pulling it from vcpu_vmx in preparation for sharing the bulk of the handlers with TDX. When the guest TD exits to VMM, RAX holds status and exit reason, RCX holds exit qualification etc rather than the VMCS fields because VMM doesn't have access to the VMCS. The eventual code will be VMX: - get exit reason, intr_info, exit_qualification, and etc from VMCS - call NMI/INTR handlers (common code) TDX: - get exit reason, intr_info, exit_qualification, and etc from guest registers - call NMI/INTR handlers (common code) Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/vmx.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 7a72391d8133..11904a720181 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6883,24 +6883,22 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu = *vcpu) rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err); } =20 -static void handle_exception_irqoff(struct vcpu_vmx *vmx) +static void handle_exception_irqoff(struct kvm_vcpu *vcpu, u32 intr_info) { - u32 intr_info =3D vmx_get_intr_info(&vmx->vcpu); - /* if exit due to PF check for async PF */ if (is_page_fault(intr_info)) - vmx->vcpu.arch.apf.host_apf_flags =3D kvm_read_and_reset_apf_flags(); + vcpu->arch.apf.host_apf_flags =3D kvm_read_and_reset_apf_flags(); /* if exit due to NM, handle before interrupts are enabled */ else if (is_nm_fault(intr_info)) - handle_nm_fault_irqoff(&vmx->vcpu); + handle_nm_fault_irqoff(vcpu); /* Handle machine checks before interrupts are enabled */ else if (is_machine_check(intr_info)) kvm_machine_check(); } =20 -static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) +static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, + u32 intr_info) { - u32 intr_info =3D vmx_get_intr_info(vcpu); unsigned int vector =3D intr_info & INTR_INFO_VECTOR_MASK; gate_desc *desc =3D (gate_desc *)host_idt_base + vector; =20 @@ -6923,9 +6921,9 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) return; =20 if (vmx->exit_reason.basic =3D=3D EXIT_REASON_EXTERNAL_INTERRUPT) - handle_external_interrupt_irqoff(vcpu); + handle_external_interrupt_irqoff(vcpu, vmx_get_intr_info(vcpu)); else if (vmx->exit_reason.basic =3D=3D EXIT_REASON_EXCEPTION_NMI) - handle_exception_irqoff(vmx); + handle_exception_irqoff(vcpu, vmx_get_intr_info(vcpu)); } =20 /* --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40073C0015E for ; Tue, 25 Jul 2023 22:24:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232493AbjGYWYS (ORCPT ); Tue, 25 Jul 2023 18:24:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232361AbjGYWV4 (ORCPT ); Tue, 25 Jul 2023 18:21:56 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 884614C1E; Tue, 25 Jul 2023 15:17:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323457; x=1721859457; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bLzrELmsGffT1v1hzJF232c7ECK1zP4p4RkVbHI5aKM=; b=hf5ngk9bnIaYOaC3T+HmYkWxeN3Hlzy1qhkm9HUAZfIaJ1AaZn4v7XJB yPYKGudD4TMsoeeLNe7mPosJFQPs+CO7LvaITmq44X9B+grZF94hvem3F 5EHhQU9krnbo1HRalf0K2xo1i7gnhEes6O+XKTtZ1cvhbFuSnHJZ7dIf8 R3ptm95qPI6F5XDee56tDR53fGkoyUVaevUxsFEXSz0ZHWC/s3aW9NSaX x4WshADYcwYFRNlq2Ev3qNeO9HiIftX7BUsw0lHQchIQ3ttJvbYXBheeL Mfd5AkDjYE0NbRXLhdxh9Ijaq9EbT4g016ij70JpHSIS6dSk8ulgrcr2W A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882644" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882644" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001858" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001858" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:58 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 081/115] KVM: VMX: Move NMI/exception handler to common helper Date: Tue, 25 Jul 2023 15:14:32 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson TDX mostly handles NMI/exception exit mostly the same to VMX case. The difference is how to retrieve exit qualification. To share the code with TDX, move NMI/exception to a common header, common.h. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/common.h | 59 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 68 +++++---------------------------------- 2 files changed, 67 insertions(+), 60 deletions(-) diff --git a/arch/x86/kvm/vmx/common.h b/arch/x86/kvm/vmx/common.h index 747f993cf7de..aaab1d407207 100644 --- a/arch/x86/kvm/vmx/common.h +++ b/arch/x86/kvm/vmx/common.h @@ -4,8 +4,67 @@ =20 #include =20 +#include + #include "posted_intr.h" #include "mmu.h" +#include "vmcs.h" +#include "x86.h" + +extern unsigned long vmx_host_idt_base; +void vmx_do_interrupt_irqoff(unsigned long entry); +void vmx_do_nmi_irqoff(void); + +static inline void vmx_handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) +{ + /* + * Save xfd_err to guest_fpu before interrupt is enabled, so the + * MSR value is not clobbered by the host activity before the guest + * has chance to consume it. + * + * Do not blindly read xfd_err here, since this exception might + * be caused by L1 interception on a platform which doesn't + * support xfd at all. + * + * Do it conditionally upon guest_fpu::xfd. xfd_err matters + * only when xfd contains a non-zero value. + * + * Queuing exception is done in vmx_handle_exit. See comment there. + */ + if (vcpu->arch.guest_fpu.fpstate->xfd) + rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err); +} + +static inline void vmx_handle_exception_irqoff(struct kvm_vcpu *vcpu, + u32 intr_info) +{ + /* if exit due to PF check for async PF */ + if (is_page_fault(intr_info)) + vcpu->arch.apf.host_apf_flags =3D kvm_read_and_reset_apf_flags(); + /* if exit due to NM, handle before interrupts are enabled */ + else if (is_nm_fault(intr_info)) + vmx_handle_nm_fault_irqoff(vcpu); + /* Handle machine checks before interrupts are enabled */ + else if (is_machine_check(intr_info)) + kvm_machine_check(); +} + +static inline void vmx_handle_external_interrupt_irqoff(struct kvm_vcpu *v= cpu, + u32 intr_info) +{ + unsigned int vector =3D intr_info & INTR_INFO_VECTOR_MASK; + gate_desc *desc =3D (gate_desc *)vmx_host_idt_base + vector; + + if (KVM_BUG(!is_external_intr(intr_info), vcpu->kvm, + "unexpected VM-Exit interrupt info: 0x%x", intr_info)) + return; + + kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ); + vmx_do_interrupt_irqoff(gate_offset(desc)); + kvm_after_interrupt(vcpu); + + vcpu->arch.at_instruction_boundary =3D true; +} =20 static inline int __vmx_handle_ept_violation(struct kvm_vcpu *vcpu, gpa_t = gpa, unsigned long exit_qualification) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 11904a720181..cc0234fed7b5 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -527,7 +527,7 @@ static inline void vmx_segment_cache_clear(struct vcpu_= vmx *vmx) vmx->segment_cache.bitmask =3D 0; } =20 -static unsigned long host_idt_base; +unsigned long vmx_host_idt_base; =20 #if IS_ENABLED(CONFIG_HYPERV) static bool __read_mostly enlightened_vmcs =3D true; @@ -4235,7 +4235,7 @@ void vmx_set_constant_host_state(struct vcpu_vmx *vmx) vmcs_write16(HOST_SS_SELECTOR, __KERNEL_DS); /* 22.2.4 */ vmcs_write16(HOST_TR_SELECTOR, GDT_ENTRY_TSS*8); /* 22.2.4 */ =20 - vmcs_writel(HOST_IDTR_BASE, host_idt_base); /* 22.2.4 */ + vmcs_writel(HOST_IDTR_BASE, vmx_host_idt_base); /* 22.2.4 */ =20 vmcs_writel(HOST_RIP, (unsigned long)vmx_vmexit); /* 22.2.5 */ =20 @@ -5132,7 +5132,7 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) intr_info =3D vmx_get_intr_info(vcpu); =20 /* - * Machine checks are handled by handle_exception_irqoff(), or by + * Machine checks are handled by vmx_handle_exception_irqoff(), or by * vmx_vcpu_run() if a #MC occurs on VM-Entry. NMIs are handled by * vmx_vcpu_enter_exit(). */ @@ -5140,7 +5140,7 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu) return 1; =20 /* - * Queue the exception here instead of in handle_nm_fault_irqoff(). + * Queue the exception here instead of in vmx_handle_nm_fault_irqoff(). * This ensures the nested_vmx check is not skipped so vmexit can * be reflected to L1 (when it intercepts #NM) before reaching this * point. @@ -6860,59 +6860,6 @@ void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64= *eoi_exit_bitmap) vmcs_write64(EOI_EXIT_BITMAP3, eoi_exit_bitmap[3]); } =20 -void vmx_do_interrupt_irqoff(unsigned long entry); -void vmx_do_nmi_irqoff(void); - -static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) -{ - /* - * Save xfd_err to guest_fpu before interrupt is enabled, so the - * MSR value is not clobbered by the host activity before the guest - * has chance to consume it. - * - * Do not blindly read xfd_err here, since this exception might - * be caused by L1 interception on a platform which doesn't - * support xfd at all. - * - * Do it conditionally upon guest_fpu::xfd. xfd_err matters - * only when xfd contains a non-zero value. - * - * Queuing exception is done in vmx_handle_exit. See comment there. - */ - if (vcpu->arch.guest_fpu.fpstate->xfd) - rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err); -} - -static void handle_exception_irqoff(struct kvm_vcpu *vcpu, u32 intr_info) -{ - /* if exit due to PF check for async PF */ - if (is_page_fault(intr_info)) - vcpu->arch.apf.host_apf_flags =3D kvm_read_and_reset_apf_flags(); - /* if exit due to NM, handle before interrupts are enabled */ - else if (is_nm_fault(intr_info)) - handle_nm_fault_irqoff(vcpu); - /* Handle machine checks before interrupts are enabled */ - else if (is_machine_check(intr_info)) - kvm_machine_check(); -} - -static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, - u32 intr_info) -{ - unsigned int vector =3D intr_info & INTR_INFO_VECTOR_MASK; - gate_desc *desc =3D (gate_desc *)host_idt_base + vector; - - if (KVM_BUG(!is_external_intr(intr_info), vcpu->kvm, - "unexpected VM-Exit interrupt info: 0x%x", intr_info)) - return; - - kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ); - vmx_do_interrupt_irqoff(gate_offset(desc)); - kvm_after_interrupt(vcpu); - - vcpu->arch.at_instruction_boundary =3D true; -} - void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -6921,9 +6868,10 @@ void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu) return; =20 if (vmx->exit_reason.basic =3D=3D EXIT_REASON_EXTERNAL_INTERRUPT) - handle_external_interrupt_irqoff(vcpu, vmx_get_intr_info(vcpu)); + vmx_handle_external_interrupt_irqoff(vcpu, + vmx_get_intr_info(vcpu)); else if (vmx->exit_reason.basic =3D=3D EXIT_REASON_EXCEPTION_NMI) - handle_exception_irqoff(vcpu, vmx_get_intr_info(vcpu)); + vmx_handle_exception_irqoff(vcpu, vmx_get_intr_info(vcpu)); } =20 /* @@ -8205,7 +8153,7 @@ __init int vmx_hardware_setup(void) for_each_possible_cpu(cpu) INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu)); store_idt(&dt); - host_idt_base =3D dt.address; + vmx_host_idt_base =3D dt.address; =20 vmx_setup_user_return_msrs(); =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD669C001DF for ; Tue, 25 Jul 2023 22:24:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232575AbjGYWYX (ORCPT ); Tue, 25 Jul 2023 18:24:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232372AbjGYWWA (ORCPT ); Tue, 25 Jul 2023 18:22:00 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1080C4C20; Tue, 25 Jul 2023 15:17:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323458; x=1721859458; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yffpgCm1vzGyx4bV+WRMVxatCK4JGCBdo5kYG+oAprc=; b=YYhgVvGLzSnmO43G9wlNQ4CI/4BxYlMTHyGDOre781JHaUOygeaSTp/K xiU0B8pzhwuB5p13pGHi0rracTxXg0uphpQjzRZDJ+6AeSVF7jrsMKZ4S zlqF9Fvx5nuNUej0rmR+L9LcK6FnFO0JbBoXuvueMvKRPazMxeb2m1ZJW 2cCMB4WJbxCwdPedbovs+fJM/vB9KdCdQT4m/KakEEXMkwIkRZBgldzFs Ek4bUqMukjfT1fofSWm6kifLvEa4kt4wQ4TEoVGPaeDr41sLwmXYE7xG0 KqE5hU9meOdhYpBWHB8q2+utpbQWKuPYr/sEFj/rhnX1CsE42W2MI7aeN g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882650" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882650" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001861" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001861" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:58 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 082/115] KVM: x86: Split core of hypercall emulation to helper function Date: Tue, 25 Jul 2023 15:14:33 -0700 Message-Id: <74e951ee3015699c76c66210d6152ebdbb0ad48c.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson By necessity, TDX will use a different register ABI for hypercalls. Break out the core functionality so that it may be reused for TDX. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 4 +++ arch/x86/kvm/x86.c | 54 ++++++++++++++++++++------------- 2 files changed, 37 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 8bd2d7df15f9..c58ceded3437 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2118,6 +2118,10 @@ static inline void kvm_clear_apicv_inhibit(struct kv= m *kvm, kvm_set_or_clear_apicv_inhibit(kvm, reason, false); } =20 +unsigned long __kvm_emulate_hypercall(struct kvm_vcpu *vcpu, unsigned long= nr, + unsigned long a0, unsigned long a1, + unsigned long a2, unsigned long a3, + int op_64_bit); int kvm_emulate_hypercall(struct kvm_vcpu *vcpu); =20 int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_= code, diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c7d34b04ccdf..747cc86c60dc 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9838,26 +9838,15 @@ static int complete_hypercall_exit(struct kvm_vcpu = *vcpu) return kvm_skip_emulated_instruction(vcpu); } =20 -int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) +unsigned long __kvm_emulate_hypercall(struct kvm_vcpu *vcpu, unsigned long= nr, + unsigned long a0, unsigned long a1, + unsigned long a2, unsigned long a3, + int op_64_bit) { - unsigned long nr, a0, a1, a2, a3, ret; - int op_64_bit; - - if (kvm_xen_hypercall_enabled(vcpu->kvm)) - return kvm_xen_hypercall(vcpu); - - if (kvm_hv_hypercall_enabled(vcpu)) - return kvm_hv_hypercall(vcpu); - - nr =3D kvm_rax_read(vcpu); - a0 =3D kvm_rbx_read(vcpu); - a1 =3D kvm_rcx_read(vcpu); - a2 =3D kvm_rdx_read(vcpu); - a3 =3D kvm_rsi_read(vcpu); + unsigned long ret; =20 trace_kvm_hypercall(nr, a0, a1, a2, a3); =20 - op_64_bit =3D is_64_bit_hypercall(vcpu); if (!op_64_bit) { nr &=3D 0xFFFFFFFF; a0 &=3D 0xFFFFFFFF; @@ -9866,11 +9855,6 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) a3 &=3D 0xFFFFFFFF; } =20 - if (static_call(kvm_x86_get_cpl)(vcpu) !=3D 0) { - ret =3D -KVM_EPERM; - goto out; - } - ret =3D -KVM_ENOSYS; =20 switch (nr) { @@ -9933,6 +9917,34 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) ret =3D -KVM_ENOSYS; break; } + return ret; +} +EXPORT_SYMBOL_GPL(__kvm_emulate_hypercall); + +int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) +{ + unsigned long nr, a0, a1, a2, a3, ret; + int op_64_bit; + + if (kvm_xen_hypercall_enabled(vcpu->kvm)) + return kvm_xen_hypercall(vcpu); + + if (kvm_hv_hypercall_enabled(vcpu)) + return kvm_hv_hypercall(vcpu); + + nr =3D kvm_rax_read(vcpu); + a0 =3D kvm_rbx_read(vcpu); + a1 =3D kvm_rcx_read(vcpu); + a2 =3D kvm_rdx_read(vcpu); + a3 =3D kvm_rsi_read(vcpu); + op_64_bit =3D is_64_bit_hypercall(vcpu); + + if (static_call(kvm_x86_get_cpl)(vcpu) !=3D 0) { + ret =3D -KVM_EPERM; + goto out; + } + + ret =3D __kvm_emulate_hypercall(vcpu, nr, a0, a1, a2, a3, op_64_bit); out: if (!op_64_bit) ret =3D (u32)ret; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A7CAC0015E for ; Tue, 25 Jul 2023 22:24:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232664AbjGYWYZ (ORCPT ); Tue, 25 Jul 2023 18:24:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232251AbjGYWWP (ORCPT ); Tue, 25 Jul 2023 18:22:15 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EAD04C22; Tue, 25 Jul 2023 15:17:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323459; x=1721859459; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yQwnM+45nN/r8YGU3sVxuAc/7M8Zki+0SnMQzAsAoeM=; b=HpvtVyhMoeRIc1YOuyqBEUS7gSJCqjU8OQ5tumpJjwojjhpvtrzHmZLY W0TVcMeXgiVzzuLmf1/7IuYvWCU6xnL4l1BMmkWvW3Ervpmm+G1p9tCm+ fk0Gkc6EhEK9SxQTE8xx9uiOO0xEc1LDVzTJrnN6pU/pMk/qj/1Zru6qm uv53O8+ffrIsjF8DrV8ELbJSvhE29rwuvDaeCttod9sl2r3OuGpKoX1VQ f7aiZjxMogB7xfKT0G7M1wc2rehDVXJ/ZnAWxV/3JapkyMepSqHFxI7Hk Qc64Q2JbO9Y4SytuFO0NMA5sQplY6tcGhNkk4wjhttT8/msvzzub6IJEV A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882656" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882656" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001864" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001864" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:59 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 083/115] KVM: TDX: Add a place holder to handle TDX VM exit Date: Tue, 25 Jul 2023 15:14:34 -0700 Message-Id: <22dc517603a186964874ee6f75c5f45b6d5c8708.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Wire up handle_exit and handle_exit_irqoff methods and add a place holder to handle VM exit. Add helper functions to get exit info, exit qualification, etc. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/main.c | 37 ++++++++++++- arch/x86/kvm/vmx/tdx.c | 110 +++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 10 ++++ 3 files changed, 154 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 64a012110515..2774533128af 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -239,6 +239,25 @@ static bool vt_protected_apic_has_interrupt(struct kvm= _vcpu *vcpu) return tdx_protected_apic_has_interrupt(vcpu); } =20 +static int vt_handle_exit(struct kvm_vcpu *vcpu, + enum exit_fastpath_completion fastpath) +{ + if (is_td_vcpu(vcpu)) + return tdx_handle_exit(vcpu, fastpath); + + return vmx_handle_exit(vcpu, fastpath); +} + +static void vt_handle_exit_irqoff(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_handle_exit_irqoff(vcpu); + return; + } + + vmx_handle_exit_irqoff(vcpu); +} + static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu) { struct pi_desc *pi =3D vcpu_to_pi_desc(vcpu); @@ -445,6 +464,18 @@ static void vt_request_immediate_exit(struct kvm_vcpu = *vcpu) vmx_request_immediate_exit(vcpu); } =20 +static void vt_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, + u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) +{ + if (is_td_vcpu(vcpu)) { + tdx_get_exit_info(vcpu, reason, info1, info2, intr_info, + error_code); + return; + } + + vmx_get_exit_info(vcpu, reason, info1, info2, intr_info, error_code); +} + static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { if (is_td_vcpu(vcpu)) @@ -539,7 +570,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .vcpu_pre_run =3D vt_vcpu_pre_run, .vcpu_run =3D vt_vcpu_run, - .handle_exit =3D vmx_handle_exit, + .handle_exit =3D vt_handle_exit, .skip_emulated_instruction =3D vmx_skip_emulated_instruction, .update_emulated_instruction =3D vmx_update_emulated_instruction, .set_interrupt_shadow =3D vt_set_interrupt_shadow, @@ -574,7 +605,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .set_identity_map_addr =3D vmx_set_identity_map_addr, .get_mt_mask =3D vt_get_mt_mask, =20 - .get_exit_info =3D vmx_get_exit_info, + .get_exit_info =3D vt_get_exit_info, =20 .vcpu_after_set_cpuid =3D vmx_vcpu_after_set_cpuid, =20 @@ -588,7 +619,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .load_mmu_pgd =3D vt_load_mmu_pgd, =20 .check_intercept =3D vmx_check_intercept, - .handle_exit_irqoff =3D vmx_handle_exit_irqoff, + .handle_exit_irqoff =3D vt_handle_exit_irqoff, =20 .request_immediate_exit =3D vt_request_immediate_exit, =20 diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index d2a20ac36999..865d7ae30813 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -87,6 +87,26 @@ static __always_inline hpa_t set_hkid_to_hpa(hpa_t pa, u= 16 hkid) return pa | ((hpa_t)hkid << boot_cpu_data.x86_phys_bits); } =20 +static __always_inline unsigned long tdexit_exit_qual(struct kvm_vcpu *vcp= u) +{ + return kvm_rcx_read(vcpu); +} + +static __always_inline unsigned long tdexit_ext_exit_qual(struct kvm_vcpu = *vcpu) +{ + return kvm_rdx_read(vcpu); +} + +static __always_inline unsigned long tdexit_gpa(struct kvm_vcpu *vcpu) +{ + return kvm_r8_read(vcpu); +} + +static __always_inline unsigned long tdexit_intr_info(struct kvm_vcpu *vcp= u) +{ + return kvm_r9_read(vcpu); +} + static inline bool is_td_vcpu_created(struct vcpu_tdx *tdx) { return tdx->tdvpr_pa; @@ -718,6 +738,12 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcp= u *vcpu, { guest_state_enter_irqoff(); tdx->exit_reason.full =3D __tdx_vcpu_run(tdx->tdvpr_pa, vcpu->arch.regs, = 0); + if ((u16)tdx->exit_reason.basic =3D=3D EXIT_REASON_EXCEPTION_NMI && + is_nmi(tdexit_intr_info(vcpu))) { + kvm_before_interrupt(vcpu, KVM_HANDLING_NMI); + vmx_do_nmi_irqoff(); + kvm_after_interrupt(vcpu); + } guest_state_exit_irqoff(); } =20 @@ -760,6 +786,25 @@ void tdx_inject_nmi(struct kvm_vcpu *vcpu) td_management_write8(to_tdx(vcpu), TD_VCPU_PEND_NMI, 1); } =20 +void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + u16 exit_reason =3D tdx->exit_reason.basic; + + if (exit_reason =3D=3D EXIT_REASON_EXTERNAL_INTERRUPT) + vmx_handle_external_interrupt_irqoff(vcpu, + tdexit_intr_info(vcpu)); + else if (exit_reason =3D=3D EXIT_REASON_EXCEPTION_NMI) + vmx_handle_exception_irqoff(vcpu, tdexit_intr_info(vcpu)); +} + +static int tdx_handle_triple_fault(struct kvm_vcpu *vcpu) +{ + vcpu->run->exit_reason =3D KVM_EXIT_SHUTDOWN; + vcpu->mmio_needed =3D 0; + return 0; +} + void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); @@ -1091,6 +1136,71 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, i= nt delivery_mode, __vmx_deliver_posted_interrupt(vcpu, &tdx->pi_desc, vector); } =20 +int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) +{ + union tdx_exit_reason exit_reason =3D to_tdx(vcpu)->exit_reason; + + /* See the comment of tdh_sept_seamcall(). */ + if (unlikely(exit_reason.full =3D=3D (TDX_OPERAND_BUSY | TDX_OPERAND_ID_S= EPT))) + return 1; + + /* + * TDH.VP.ENTRY checks TD EPOCH which contend with TDH.MEM.TRACK and + * vcpu TDH.VP.ENTER. + */ + if (unlikely(exit_reason.full =3D=3D (TDX_OPERAND_BUSY | TDX_OPERAND_ID_T= D_EPOCH))) + return 1; + + if (unlikely(exit_reason.full =3D=3D TDX_SEAMCALL_UD)) { + kvm_spurious_fault(); + /* + * In the case of reboot or kexec, loop with TDH.VP.ENTER and + * TDX_SEAMCALL_UD to avoid unnecessarily activity. + */ + return 1; + } + + if (unlikely(exit_reason.non_recoverable || exit_reason.error)) { + if (unlikely(exit_reason.basic =3D=3D EXIT_REASON_TRIPLE_FAULT)) + return tdx_handle_triple_fault(vcpu); + + kvm_pr_unimpl("TD exit 0x%llx, %d hkid 0x%x hkid pa 0x%llx\n", + exit_reason.full, exit_reason.basic, + to_kvm_tdx(vcpu->kvm)->hkid, + set_hkid_to_hpa(0, to_kvm_tdx(vcpu->kvm)->hkid)); + goto unhandled_exit; + } + + WARN_ON_ONCE(fastpath !=3D EXIT_FASTPATH_NONE); + + switch (exit_reason.basic) { + default: + break; + } + +unhandled_exit: + vcpu->run->exit_reason =3D KVM_EXIT_INTERNAL_ERROR; + vcpu->run->internal.suberror =3D KVM_INTERNAL_ERROR_UNEXPECTED_EXIT_REASO= N; + vcpu->run->internal.ndata =3D 2; + vcpu->run->internal.data[0] =3D exit_reason.full; + vcpu->run->internal.data[1] =3D vcpu->arch.last_vmentry_cpu; + return 0; +} + +void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, + u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + *reason =3D tdx->exit_reason.full; + + *info1 =3D tdexit_exit_qual(vcpu); + *info2 =3D tdexit_ext_exit_qual(vcpu); + + *intr_info =3D tdexit_intr_info(vcpu); + *error_code =3D 0; +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index ef93b30750ce..850844cdeadf 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -156,11 +156,16 @@ void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcp= u); void tdx_vcpu_put(struct kvm_vcpu *vcpu); void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu); +void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu); +int tdx_handle_exit(struct kvm_vcpu *vcpu, + enum exit_fastpath_completion fastpath); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); =20 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, int trig_mode, int vector); void tdx_inject_nmi(struct kvm_vcpu *vcpu); +void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, + u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 @@ -193,11 +198,16 @@ static inline void tdx_prepare_switch_to_guest(struct= kvm_vcpu *vcpu) {} static inline void tdx_vcpu_put(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) {} static inline bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)= { return false; } +static inline void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) {} +static inline int tdx_handle_exit(struct kvm_vcpu *vcpu, + enum exit_fastpath_completion fastpath) { return 0; } static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is= _mmio) { return 0; } =20 static inline void tdx_deliver_interrupt(struct kvm_lapic *apic, int deliv= ery_mode, int trig_mode, int vector) {} static inline void tdx_inject_nmi(struct kvm_vcpu *vcpu) {} +static inline void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u= 64 *info1, + u64 *info2, u32 *intr_info, u32 *error_code) {} =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7F58C41513 for ; Tue, 25 Jul 2023 22:24:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232753AbjGYWYc (ORCPT ); Tue, 25 Jul 2023 18:24:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230407AbjGYWWT (ORCPT ); Tue, 25 Jul 2023 18:22:19 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61E304C3A; Tue, 25 Jul 2023 15:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323465; x=1721859465; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OBXugoeG43vdHjZJrGLYTbh1iSwz4x9BVpSGJUSumzA=; b=YlBRuxUWchhJ3vQRm49EZySMhox0lJ+MpGbRgRjDW0AT7U3nRlQAZebd 8+xndJfEB3s45Cp/5/2vwV+0bGZOnOUeCv0HKDwqFmx6HAbHSjMWnUH6T l5Asy/iX1ZcC6l4x6/hmL2XNTbZnpEf8evruWWxK60rO2Vb1+94q+DTM2 V89vOSvvR7CT8hOlQQZ+pL4UV9R8AS+oBS5UJLwVoa12qwrJvXdHo91Bo +jv4FAOWtqagUFwl0jri34E+qBXzUpLB1ulmWibCO9myps4aKsCHwmZky H9e4lrphMs2U2m9uHxaR0Lhu9LqRGXz67wqjn8gRNGHBeczZ+74yD7CVO w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882661" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882661" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001868" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001868" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:59 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Yao Yuan Subject: [PATCH v15 084/115] KVM: TDX: Handle vmentry failure for INTEL TD guest Date: Tue, 25 Jul 2023 15:14:35 -0700 Message-Id: <4e1b23fd3a7d4ccb3543d5420b15741d0bd63499.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yao Yuan TDX module passes control back to VMM if it failed to vmentry for a TD, use same exit reason to notify user space, align with VMX. If VMM corrupted TD VMCS, machine check during entry can happens. vm exit reason will be EXIT_REASON_MCE_DURING_VMENTRY. If VMM corrupted TD VMCS with debug TD by TDH.VP.WR, the exit reason would be EXIT_REASON_INVALID_STATE or EXIT_REASON_MSR_LOAD_FAIL. Signed-off-by: Yao Yuan Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 865d7ae30813..0dfd6ea07aa0 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1171,6 +1171,28 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t fastpath) goto unhandled_exit; } =20 + /* + * When TDX module saw VMEXIT_REASON_FAILED_VMENTER_MC etc, TDH.VP.ENTER + * returns with TDX_SUCCESS | exit_reason with failed_vmentry =3D 1. + * Because TDX module maintains TD VMCS correctness, usually vmentry + * failure shouldn't happen. In some corner cases it can happen. For + * example + * - machine check during entry: EXIT_REASON_MCE_DURING_VMENTRY + * - TDH.VP.WR with debug TD. VMM can corrupt TD VMCS + * - EXIT_REASON_INVALID_STATE + * - EXIT_REASON_MSR_LOAD_FAIL + */ + if (unlikely(exit_reason.failed_vmentry)) { + pr_err("TDExit: exit_reason 0x%016llx qualification=3D%016lx ext_qualifi= cation=3D%016lx\n", + exit_reason.full, tdexit_exit_qual(vcpu), tdexit_ext_exit_qual(vc= pu)); + vcpu->run->exit_reason =3D KVM_EXIT_FAIL_ENTRY; + vcpu->run->fail_entry.hardware_entry_failure_reason + =3D exit_reason.full; + vcpu->run->fail_entry.cpu =3D vcpu->arch.last_vmentry_cpu; + + return 0; + } + WARN_ON_ONCE(fastpath !=3D EXIT_FASTPATH_NONE); =20 switch (exit_reason.basic) { --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91231C001DF for ; Tue, 25 Jul 2023 22:24:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232693AbjGYWY3 (ORCPT ); Tue, 25 Jul 2023 18:24:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232555AbjGYWWU (ORCPT ); Tue, 25 Jul 2023 18:22:20 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AAF74C3F; Tue, 25 Jul 2023 15:17:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323466; x=1721859466; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WcaDgfkogRdzhJ1xqHvkpBoLwuvf368y385r/77EQew=; b=HV/mvXgFtYiGXq2ZUaE5JVKv+q+07RyT26xO7kn8oitB5/A0XROPlWO+ h4DJEdpUV5SGi2f5uo3Up871EjVII00CGnkpVXUuvIEco9WQTZw3Cw/cG fUdhZMzQ2kEmizDOmge7rat/bwfIdii53yhGgKw2zEVaKWu/cevq7U7a1 36L3F2DwIMHcy7+mTeJ629WxyjuaxzDR3mPXbkVQEUy/b870XJEDRMjwe opIa4YZF7LfnSi6SVlgqE6hXx403CUQuFI9obhGAJwtqfOFvz8D0vavR0 5n6vySLaFnqIdx9Ishvreo0NXOPGQ4GnXuSgk1uyEW1xabWXntNPWwwQn w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882665" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882665" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001872" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001872" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:15:59 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 085/115] KVM: TDX: handle EXIT_REASON_OTHER_SMI Date: Tue, 25 Jul 2023 15:14:36 -0700 Message-Id: <68d4f34501980bb0d352e97a339571684b222a2f.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata If the control reaches EXIT_REASON_OTHER_SMI, #SMI is delivered and handled right after returning from the TDX module to KVM nothing needs to be done in KVM. Continue TDX vcpu execution. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/include/uapi/asm/vmx.h | 1 + arch/x86/kvm/vmx/tdx.c | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/arch/x86/include/uapi/asm/vmx.h b/arch/x86/include/uapi/asm/vm= x.h index a5faf6d88f1b..b3a30ef3efdd 100644 --- a/arch/x86/include/uapi/asm/vmx.h +++ b/arch/x86/include/uapi/asm/vmx.h @@ -34,6 +34,7 @@ #define EXIT_REASON_TRIPLE_FAULT 2 #define EXIT_REASON_INIT_SIGNAL 3 #define EXIT_REASON_SIPI_SIGNAL 4 +#define EXIT_REASON_OTHER_SMI 6 =20 #define EXIT_REASON_INTERRUPT_WINDOW 7 #define EXIT_REASON_NMI_WINDOW 8 diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0dfd6ea07aa0..0d92253ea40e 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1196,6 +1196,13 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t fastpath) WARN_ON_ONCE(fastpath !=3D EXIT_FASTPATH_NONE); =20 switch (exit_reason.basic) { + case EXIT_REASON_OTHER_SMI: + /* + * If reach here, it's not a Machine Check System Management + * Interrupt(MSMI). #SMI is delivered and handled right after + * SEAMRET, nothing needs to be done in KVM. + */ + return 1; default: break; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60A4CEB64DD for ; Tue, 25 Jul 2023 22:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232808AbjGYWYh (ORCPT ); Tue, 25 Jul 2023 18:24:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232697AbjGYWWV (ORCPT ); Tue, 25 Jul 2023 18:22:21 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7924F4EC5; Tue, 25 Jul 2023 15:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323471; x=1721859471; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7lKJ6bOIOJBVCdWzKOJhecQyu0qSSTJAZTUYFEWCbkQ=; b=l5w4VWLpwj9Bs4aZIcmRFTodHtJl8UPAP5oHF34FmxwzBRyDaAZNP2dr uDf423V+FgKTXcl7Ru/nA75jSf6K0xJHjnFUyWjrBUMsXKfK9V5xsbKok tm6wdzs1UHo6hzT5CPS4iKyKZ4hSwZKqXDFVo0+JFfwRMYWHOkyw3J/j+ n+RkSNUzQRV9vGGAPFYZwCRpZuaypcaF9WUNzWiaLFPx6SiCw6l7Rk1tN 9jzk9WSo/+b2Txj4znjPWLTGl8EsJmiX/zsTUGS2HWaEE2vPfTKvlf2bP qv/LmWdtbibDULT346bBEfLjax+4HSTN6DbuIEIvzSc4laNQo8/TgzmMw Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882669" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882669" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001876" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001876" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:00 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 086/115] KVM: TDX: handle ept violation/misconfig exit Date: Tue, 25 Jul 2023 15:14:37 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata On EPT violation, call a common function, __vmx_handle_ept_violation() to trigger x86 MMU code. On EPT misconfiguration, exit to ring 3 with KVM_EXIT_UNKNOWN. because EPT misconfiguration can't happen as MMIO is trigged by TDG.VP.VMCALL. No point to set a misconfiguration value for the fast path. Signed-off-by: Isaku Yamahata --- v14 -> v15: - use PFERR_GUEST_ENC_MASK to tell the fault is private Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/common.h | 3 +++ arch/x86/kvm/vmx/tdx.c | 46 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/arch/x86/kvm/vmx/common.h b/arch/x86/kvm/vmx/common.h index aaab1d407207..e4fec792a3ae 100644 --- a/arch/x86/kvm/vmx/common.h +++ b/arch/x86/kvm/vmx/common.h @@ -87,6 +87,9 @@ static inline int __vmx_handle_ept_violation(struct kvm_v= cpu *vcpu, gpa_t gpa, error_code |=3D (exit_qualification & EPT_VIOLATION_GVA_TRANSLATED) !=3D = 0 ? PFERR_GUEST_FINAL_MASK : PFERR_GUEST_PAGE_MASK; =20 + if (kvm_is_private_gpa(vcpu->kvm, gpa)) + error_code |=3D PFERR_GUEST_ENC_MASK; + return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0); } =20 diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0d92253ea40e..1a41e12f0942 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1136,6 +1136,48 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, i= nt delivery_mode, __vmx_deliver_posted_interrupt(vcpu, &tdx->pi_desc, vector); } =20 +static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) +{ + unsigned long exit_qual; + + if (kvm_is_private_gpa(vcpu->kvm, tdexit_gpa(vcpu))) { + /* + * Always treat SEPT violations as write faults. Ignore the + * EXIT_QUALIFICATION reported by TDX-SEAM for SEPT violations. + * TD private pages are always RWX in the SEPT tables, + * i.e. they're always mapped writable. Just as importantly, + * treating SEPT violations as write faults is necessary to + * avoid COW allocations, which will cause TDAUGPAGE failures + * due to aliasing a single HPA to multiple GPAs. + */ +#define TDX_SEPT_VIOLATION_EXIT_QUAL EPT_VIOLATION_ACC_WRITE + exit_qual =3D TDX_SEPT_VIOLATION_EXIT_QUAL; + } else { + exit_qual =3D tdexit_exit_qual(vcpu); + if (exit_qual & EPT_VIOLATION_ACC_INSTR) { + pr_warn("kvm: TDX instr fetch to shared GPA =3D 0x%lx @ RIP =3D 0x%lx\n= ", + tdexit_gpa(vcpu), kvm_rip_read(vcpu)); + vcpu->run->exit_reason =3D KVM_EXIT_EXCEPTION; + vcpu->run->ex.exception =3D PF_VECTOR; + vcpu->run->ex.error_code =3D exit_qual; + return 0; + } + } + + trace_kvm_page_fault(vcpu, tdexit_gpa(vcpu), exit_qual); + return __vmx_handle_ept_violation(vcpu, tdexit_gpa(vcpu), exit_qual); +} + +static int tdx_handle_ept_misconfig(struct kvm_vcpu *vcpu) +{ + WARN_ON_ONCE(1); + + vcpu->run->exit_reason =3D KVM_EXIT_UNKNOWN; + vcpu->run->hw.hardware_exit_reason =3D EXIT_REASON_EPT_MISCONFIG; + + return 0; +} + int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) { union tdx_exit_reason exit_reason =3D to_tdx(vcpu)->exit_reason; @@ -1196,6 +1238,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t fastpath) WARN_ON_ONCE(fastpath !=3D EXIT_FASTPATH_NONE); =20 switch (exit_reason.basic) { + case EXIT_REASON_EPT_VIOLATION: + return tdx_handle_ept_violation(vcpu); + case EXIT_REASON_EPT_MISCONFIG: + return tdx_handle_ept_misconfig(vcpu); case EXIT_REASON_OTHER_SMI: /* * If reach here, it's not a Machine Check System Management --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 299A2C04A6A for ; Tue, 25 Jul 2023 22:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232889AbjGYWYp (ORCPT ); Tue, 25 Jul 2023 18:24:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232741AbjGYWWf (ORCPT ); Tue, 25 Jul 2023 18:22:35 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 949D14ED8; Tue, 25 Jul 2023 15:17:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323475; x=1721859475; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TyobtdtuH/YAqwgdKL24pQLPaZbJ8c3URiYLS/2JpXE=; b=TcR8N+XFJNBs/EU6I2QyvKNg3FyLVVSlwdzHYxkevqeeUXPDoCYoCDtK jAoUZfBw/H4pBH/uOMZ3Ucl7H5t8nA2GcxZpcjhDEF90tjqtXoobLJOpQ 5X1RZV2K+qUXFWbOdunxH2DXmiWZA+Teh4Fm4jNBFZisqT4v58S3jWpBd ztWJbtAD8qNRnX3hyO8mNL0svRu7g7QxMOp/LxAU/WQsUC6Kn0RCPGbXq QwhSX7bX2i91idmkLgQgbcS9qQyS9gLYKNrp2XzuKao+WQlBG5gBTxb6Q boTM2ivlP6aJ1olHlaUwHrQrPga1qFhKBeAg2U33DzlbhqdoNwGGWzWJD Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882675" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882675" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001879" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001879" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:00 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 087/115] KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT Date: Tue, 25 Jul 2023 15:14:38 -0700 Message-Id: <65e113025130e082a409d12b58f4282c2801d46b.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because guest TD state is protected, exceptions in guest TDs can't be intercepted. TDX VMM doesn't need to handle exceptions. tdx_handle_exit_irqoff() handles NMI and machine check. Ignore NMI and machine check and continue guest TD execution. For external interrupt, increment stats same to the VMX case. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 1a41e12f0942..45b521e05ceb 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -798,6 +798,25 @@ void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) vmx_handle_exception_irqoff(vcpu, tdexit_intr_info(vcpu)); } =20 +static int tdx_handle_exception(struct kvm_vcpu *vcpu) +{ + u32 intr_info =3D tdexit_intr_info(vcpu); + + if (is_nmi(intr_info) || is_machine_check(intr_info)) + return 1; + + kvm_pr_unimpl("unexpected exception 0x%x(exit_reason 0x%llx qual 0x%lx)\n= ", + intr_info, + to_tdx(vcpu)->exit_reason.full, tdexit_exit_qual(vcpu)); + return -EFAULT; +} + +static int tdx_handle_external_interrupt(struct kvm_vcpu *vcpu) +{ + ++vcpu->stat.irq_exits; + return 1; +} + static int tdx_handle_triple_fault(struct kvm_vcpu *vcpu) { vcpu->run->exit_reason =3D KVM_EXIT_SHUTDOWN; @@ -1238,6 +1257,10 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t fastpath) WARN_ON_ONCE(fastpath !=3D EXIT_FASTPATH_NONE); =20 switch (exit_reason.basic) { + case EXIT_REASON_EXCEPTION_NMI: + return tdx_handle_exception(vcpu); + case EXIT_REASON_EXTERNAL_INTERRUPT: + return tdx_handle_external_interrupt(vcpu); case EXIT_REASON_EPT_VIOLATION: return tdx_handle_ept_violation(vcpu); case EXIT_REASON_EPT_MISCONFIG: --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1298C0015E for ; Tue, 25 Jul 2023 22:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232869AbjGYWYm (ORCPT ); Tue, 25 Jul 2023 18:24:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232747AbjGYWWp (ORCPT ); Tue, 25 Jul 2023 18:22:45 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8747A4EDD; Tue, 25 Jul 2023 15:17:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323477; x=1721859477; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TDllGKwd/erLe7EXvszdxaN+tTcjl339wwIEld81mrM=; b=L7RkXrJwjsqlzbSUoTR1qlaXN+ehIgifRVWAVO4+JUeUq7PedP8KC2mr kZFGCZMeer7vx3hWKdQ0YxSMTX08AGnaDi/EWIdvl4aOYGe4Oxfy/Ytf4 PCJ7zrJ/2luU75WQYQOnAcf6O7GuJrnoB+KCMlFXmbUzpaxSCKVrfRWl4 2Pa3gBpsH8X7fhefwS7QrqQmRsNii7s1xN8QhQelZKtrgY3Z7tpjHuuME lOD6bJe6iwQm/wG7nEKMHU9iosTq9RbEjilMia2YwA7DKUkrzCxGP7kde epEubidz0AktvhkKsKK/H+RfNj+BfqDWR3W15MD+pCxkvnKstvN31W8jU g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882681" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882681" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001883" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001883" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:01 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 088/115] KVM: TDX: Handle EXIT_REASON_OTHER_SMI with MSMI Date: Tue, 25 Jul 2023 15:14:39 -0700 Message-Id: <5658a07b6e119070190886d056a35a8f0e660539.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata When BIOS eMCA MCE-SMI morphing is enabled, the #MC is morphed to MSMI (Machine Check System Management Interrupt). Then the SMI causes TD exit with the read reason of EXIT_REASON_OTHER_SMI with MSMI bit set in the exit qualification to KVM instead of EXIT_REASON_EXCEPTION_NMI with MC exception. Handle EXIT_REASON_OTHER_SMI with MSMI bit set in the exit qualification as MCE(Machine Check Exception) happened during TD guest running. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 40 ++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx_arch.h | 2 ++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 45b521e05ceb..e56eeb8d0ec7 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -796,6 +796,30 @@ void tdx_handle_exit_irqoff(struct kvm_vcpu *vcpu) tdexit_intr_info(vcpu)); else if (exit_reason =3D=3D EXIT_REASON_EXCEPTION_NMI) vmx_handle_exception_irqoff(vcpu, tdexit_intr_info(vcpu)); + else if (unlikely(tdx->exit_reason.non_recoverable || + tdx->exit_reason.error)) { + /* + * The only reason it gets EXIT_REASON_OTHER_SMI is there is an + * #MSMI(Machine Check System Management Interrupt) with + * exit_qualification bit 0 set in TD guest. + * The #MSMI is delivered right after SEAMCALL returns, + * and an #MC is delivered to host kernel after SMI handler + * returns. + * + * The #MC right after SEAMCALL is fixed up and skipped in #MC + * handler because it's an #MC happens in TD guest we cannot + * handle it with host's context. + * + * Call KVM's machine check handler explicitly here. + */ + if (tdx->exit_reason.basic =3D=3D EXIT_REASON_OTHER_SMI) { + unsigned long exit_qual; + + exit_qual =3D tdexit_exit_qual(vcpu); + if (exit_qual & TD_EXIT_OTHER_SMI_IS_MSMI) + kvm_machine_check(); + } + } } =20 static int tdx_handle_exception(struct kvm_vcpu *vcpu) @@ -1229,6 +1253,11 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t fastpath) exit_reason.full, exit_reason.basic, to_kvm_tdx(vcpu->kvm)->hkid, set_hkid_to_hpa(0, to_kvm_tdx(vcpu->kvm)->hkid)); + + /* + * tdx_handle_exit_irqoff() handled EXIT_REASON_OTHER_SMI. It + * must be handled before enabling preemption because it's #MC. + */ goto unhandled_exit; } =20 @@ -1267,9 +1296,14 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_= t fastpath) return tdx_handle_ept_misconfig(vcpu); case EXIT_REASON_OTHER_SMI: /* - * If reach here, it's not a Machine Check System Management - * Interrupt(MSMI). #SMI is delivered and handled right after - * SEAMRET, nothing needs to be done in KVM. + * Unlike VMX, all the SMI in SEAM non-root mode (i.e. when + * TD guest vcpu is running) will cause TD exit to TDX module, + * then SEAMRET to KVM. Once it exits to KVM, SMI is delivered + * and handled right away. + * + * - If it's an Machine Check System Management Interrupt + * (MSMI), it's handled above due to non_recoverable bit set. + * - If it's not an MSMI, don't need to do anything here. */ return 1; default: diff --git a/arch/x86/kvm/vmx/tdx_arch.h b/arch/x86/kvm/vmx/tdx_arch.h index 942a0e561a7b..8860c7571b1f 100644 --- a/arch/x86/kvm/vmx/tdx_arch.h +++ b/arch/x86/kvm/vmx/tdx_arch.h @@ -46,6 +46,8 @@ #define TDG_VP_VMCALL_REPORT_FATAL_ERROR 0x10003 #define TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT 0x10004 =20 +#define TD_EXIT_OTHER_SMI_IS_MSMI BIT(1) + /* TDX control structure (TDR/TDCS/TDVPS) field access codes */ #define TDX_NON_ARCH BIT_ULL(63) #define TDX_CLASS_SHIFT 56 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11B92EB64DD for ; Tue, 25 Jul 2023 22:24:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232957AbjGYWYy (ORCPT ); Tue, 25 Jul 2023 18:24:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232844AbjGYWXD (ORCPT ); Tue, 25 Jul 2023 18:23:03 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E0445244; Tue, 25 Jul 2023 15:18:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323494; x=1721859494; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6xccofk7PGEqoOCeiBTJlus7HLCsTUvyZb5exAlIw4A=; b=mcxnoOO5YiEKLaE0gkjA9uSgdIa2RJ8yJMsKLiELXx5a+E8TeAOI8qmF IfGdxKLRYIPnlHlfPYS6cCIsl3GU79VrsZgUB2mWq7DX/ELfI3DZDpzOK TprGJw2C+HJNyoil+An24PluP2WxQs2JKLmP8IiNl81CFi/nFJOOkJ/vU hvFxNn/qy1nhQVUdqdftfzwPpW/GGmIJDVDWiLBSwCs9Nzt9B1CgclKbq 1qJZ9Ia7gWluvw0PQ40OsrLqCC35TYXk3VjWhXlKPYjjSxGsDcbg1tE76 x6NCYPlsDHCHgcyrsDCPw8Fum2qza3H22z5D7VsGAu8Adk6AF4idAsYWZ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882685" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882685" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001886" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001886" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:01 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Xiaoyao Li , Sean Christopherson Subject: [PATCH v15 089/115] KVM: TDX: Add a place holder for handler of TDX hypercalls (TDG.VP.VMCALL) Date: Tue, 25 Jul 2023 15:14:40 -0700 Message-Id: <4b6737290264b1938f16f9de4ce7613a98de7454.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata The TDX module specification defines TDG.VP.VMCALL API (TDVMCALL for short) for the guest TD to call hypercall to VMM. When the guest TD issues TDG.VP.VMCALL, the guest TD exits to VMM with a new exit reason of TDVMCALL. The arguments from the guest TD and returned values from the VMM are passed in the guest registers. The guest RCX registers indicates which registers are used. Define helper functions to access those registers as ABI. Define the TDVMCALL exit reason, which is carved out from the VMX exit reason namespace as the TDVMCALL exit from TDX guest to TDX-SEAM is really just a VM-Exit. Add a place holder to handle TDVMCALL exit. Co-developed-by: Xiaoyao Li Signed-off-by: Xiaoyao Li Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata --- arch/x86/include/uapi/asm/vmx.h | 4 ++- arch/x86/kvm/vmx/tdx.c | 56 ++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 13 ++++++++ 3 files changed, 71 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/uapi/asm/vmx.h b/arch/x86/include/uapi/asm/vm= x.h index b3a30ef3efdd..f0f4a4cf84a7 100644 --- a/arch/x86/include/uapi/asm/vmx.h +++ b/arch/x86/include/uapi/asm/vmx.h @@ -93,6 +93,7 @@ #define EXIT_REASON_TPAUSE 68 #define EXIT_REASON_BUS_LOCK 74 #define EXIT_REASON_NOTIFY 75 +#define EXIT_REASON_TDCALL 77 =20 #define VMX_EXIT_REASONS \ { EXIT_REASON_EXCEPTION_NMI, "EXCEPTION_NMI" }, \ @@ -156,7 +157,8 @@ { EXIT_REASON_UMWAIT, "UMWAIT" }, \ { EXIT_REASON_TPAUSE, "TPAUSE" }, \ { EXIT_REASON_BUS_LOCK, "BUS_LOCK" }, \ - { EXIT_REASON_NOTIFY, "NOTIFY" } + { EXIT_REASON_NOTIFY, "NOTIFY" }, \ + { EXIT_REASON_TDCALL, "TDCALL" } =20 #define VMX_EXIT_REASON_FLAGS \ { VMX_EXIT_REASONS_FAILED_VMENTRY, "FAILED_VMENTRY" } diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index e56eeb8d0ec7..0a1ccd16d17f 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -107,6 +107,41 @@ static __always_inline unsigned long tdexit_intr_info(= struct kvm_vcpu *vcpu) return kvm_r9_read(vcpu); } =20 +#define BUILD_TDVMCALL_ACCESSORS(param, gpr) \ +static __always_inline \ +unsigned long tdvmcall_##param##_read(struct kvm_vcpu *vcpu) \ +{ \ + return kvm_##gpr##_read(vcpu); \ +} \ +static __always_inline void tdvmcall_##param##_write(struct kvm_vcpu *vcpu= , \ + unsigned long val) \ +{ \ + kvm_##gpr##_write(vcpu, val); \ +} +BUILD_TDVMCALL_ACCESSORS(a0, r12); +BUILD_TDVMCALL_ACCESSORS(a1, r13); +BUILD_TDVMCALL_ACCESSORS(a2, r14); +BUILD_TDVMCALL_ACCESSORS(a3, r15); + +static __always_inline unsigned long tdvmcall_exit_type(struct kvm_vcpu *v= cpu) +{ + return kvm_r10_read(vcpu); +} +static __always_inline unsigned long tdvmcall_leaf(struct kvm_vcpu *vcpu) +{ + return kvm_r11_read(vcpu); +} +static __always_inline void tdvmcall_set_return_code(struct kvm_vcpu *vcpu, + long val) +{ + kvm_r10_write(vcpu, val); +} +static __always_inline void tdvmcall_set_return_val(struct kvm_vcpu *vcpu, + unsigned long val) +{ + kvm_r11_write(vcpu, val); +} + static inline bool is_td_vcpu_created(struct vcpu_tdx *tdx) { return tdx->tdvpr_pa; @@ -737,7 +772,8 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu= *vcpu, struct vcpu_tdx *tdx) { guest_state_enter_irqoff(); - tdx->exit_reason.full =3D __tdx_vcpu_run(tdx->tdvpr_pa, vcpu->arch.regs, = 0); + tdx->exit_reason.full =3D __tdx_vcpu_run(tdx->tdvpr_pa, vcpu->arch.regs, + tdx->tdvmcall.regs_mask); if ((u16)tdx->exit_reason.basic =3D=3D EXIT_REASON_EXCEPTION_NMI && is_nmi(tdexit_intr_info(vcpu))) { kvm_before_interrupt(vcpu, KVM_HANDLING_NMI); @@ -778,6 +814,11 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) =20 tdx_complete_interrupts(vcpu); =20 + if (tdx->exit_reason.basic =3D=3D EXIT_REASON_TDCALL) + tdx->tdvmcall.rcx =3D vcpu->arch.regs[VCPU_REGS_RCX]; + else + tdx->tdvmcall.rcx =3D 0; + return EXIT_FASTPATH_NONE; } =20 @@ -848,6 +889,17 @@ static int tdx_handle_triple_fault(struct kvm_vcpu *vc= pu) return 0; } =20 +static int handle_tdvmcall(struct kvm_vcpu *vcpu) +{ + switch (tdvmcall_leaf(vcpu)) { + default: + break; + } + + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); + return 1; +} + void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); @@ -1290,6 +1342,8 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t= fastpath) return tdx_handle_exception(vcpu); case EXIT_REASON_EXTERNAL_INTERRUPT: return tdx_handle_external_interrupt(vcpu); + case EXIT_REASON_TDCALL: + return handle_tdvmcall(vcpu); case EXIT_REASON_EPT_VIOLATION: return tdx_handle_ept_violation(vcpu); case EXIT_REASON_EPT_MISCONFIG: diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 45c1df7b2e40..e03f7192dfab 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -80,6 +80,19 @@ struct vcpu_tdx { =20 struct list_head cpu_list; =20 + union { + struct { + union { + struct { + u16 gpr_mask; + u16 xmm_mask; + }; + u32 regs_mask; + }; + u32 reserved; + }; + u64 rcx; + } tdvmcall; union tdx_exit_reason exit_reason; =20 bool initialized; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D64CFC001DF for ; Tue, 25 Jul 2023 22:24:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232838AbjGYWYu (ORCPT ); Tue, 25 Jul 2023 18:24:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232842AbjGYWXD (ORCPT ); Tue, 25 Jul 2023 18:23:03 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AB5A5249; Tue, 25 Jul 2023 15:18:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323494; x=1721859494; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0201nZOQNDeXU1m+PIlO4jk2v5Jd3RpUfZElkXE2XPI=; b=eC1rgBXWX3B0CH1TbmzoWaRoZdZDDeF36mQC69KwnAIponCJySHC7OH0 E41Sc9kledoJGT/KOz9HOOBjFcgkHkzpiuSLrIeA2GFaisz4KwhwXcmzp v5zGUhRMkiiHRLAlxfgNGacFE0GPxHP9Y2BUDcOlDoSvnY2sVi+B6+Fp+ KY+QOlpdtCRZ4RXoXV/JZiZkqHnQ6Zq3lU6eMx6Fy6WIXbbem9AO1Pooc OWfE6wuaqjO7sEx8WcmpcJsWPpSgDwUhu+mH/610hN1nlOjni01ryO7Qd uAd+TFHhEVsYmXWKAajBmWJY3OdhcigjHeBILx+SISKObhna7hAdBWoxz w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882690" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882690" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001890" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001890" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:02 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 090/115] KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL Date: Tue, 25 Jul 2023 15:14:41 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata The TDX Guest-Host communication interface (GHCI) specification defines the ABI for the guest TD to issue hypercall. It reserves vendor specific arguments for VMM specific use. Use it as KVM hypercall and handle it. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0a1ccd16d17f..19500a05f7b5 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -889,8 +889,39 @@ static int tdx_handle_triple_fault(struct kvm_vcpu *vc= pu) return 0; } =20 +static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu) +{ + unsigned long nr, a0, a1, a2, a3, ret; + + /* + * ABI for KVM tdvmcall argument: + * In Guest-Hypervisor Communication Interface(GHCI) specification, + * Non-zero leaf number (R10 !=3D 0) is defined to indicate + * vendor-specific. KVM uses this for KVM hypercall. NOTE: KVM + * hypercall number starts from one. Zero isn't used for KVM hypercall + * number. + * + * R10: KVM hypercall number + * arguments: R11, R12, R13, R14. + */ + nr =3D kvm_r10_read(vcpu); + a0 =3D kvm_r11_read(vcpu); + a1 =3D kvm_r12_read(vcpu); + a2 =3D kvm_r13_read(vcpu); + a3 =3D kvm_r14_read(vcpu); + + ret =3D __kvm_emulate_hypercall(vcpu, nr, a0, a1, a2, a3, true); + + tdvmcall_set_return_code(vcpu, ret); + + return 1; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { + if (tdvmcall_exit_type(vcpu)) + return tdx_emulate_vmcall(vcpu); + switch (tdvmcall_leaf(vcpu)) { default: break; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F492EB64DD for ; Tue, 25 Jul 2023 22:25:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232986AbjGYWY7 (ORCPT ); Tue, 25 Jul 2023 18:24:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232864AbjGYWXF (ORCPT ); Tue, 25 Jul 2023 18:23:05 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A18D8524E; Tue, 25 Jul 2023 15:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323495; x=1721859495; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hSGK2LlFv0TvHSe7F1KxzRM+dpHoN60qKGZl44FONDw=; b=P7yrNAqgSE+eetptqDO0fXb91Kl5jUBp54Jj1fycMt1F4MULH0QTvkij F1rgiutXLIy6iBdDybyrorAU/XSGFoXC5TFw8ZSefV3EqHtQfJBOJJtcZ PneIJckz6evwl42XIpkNCv6yjrwgFV9F9or7eNOBZSEC3Am6B7WbCyVR2 eQ8129+DEf93FxYjyUCHIgttBdEnTcF0oJVuhpQYHxZSd7obIpOi7WDH7 v28NcLLaNYt/p7dUQMJznJ9+JcE701q+vWCGtiKvWe6+7792zBemSDIfe VNThqiPHgrhv1uwsWEO3bNsSBvZXQwCjW7QicF2ZGTHCUZ4X/mCCJRRQg w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882696" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882696" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001893" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001893" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:02 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 091/115] KVM: TDX: Add KVM Exit for TDX TDG.VP.VMCALL Date: Tue, 25 Jul 2023 15:14:42 -0700 Message-Id: <9754513226ad834533c8e97039b50e5c0470caa4.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Some of TDG.VP.VMCALL require device model, for example, qemu, to handle them on behalf of kvm kernel module. TDG_VP_VMCALL_REPORT_FATAL_ERROR, TDG_VP_VMCALL_MAP_GPA, TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT, and TDG_VP_VMCALL_GET_QUOTE requires user space VMM handling. Introduce new kvm exit, KVM_EXIT_TDX, and functions to setup it. TDG_VP_VMCALL_INVALID_OPERAND is set as default return value to avoid random value. Device model should update R10 if necessary. Signed-off-by: Isaku Yamahata --- v14 -> v15: - updated struct kvm_tdx_exit with union - export constants for reg bitmask Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 84 +++++++++++++++++++++++++++++++++++++- include/uapi/linux/kvm.h | 87 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 19500a05f7b5..0e95e5c79337 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -917,6 +917,78 @@ static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu) return 1; } =20 +static int tdx_complete_vp_vmcall(struct kvm_vcpu *vcpu) +{ + struct kvm_tdx_vmcall *tdx_vmcall =3D &vcpu->run->tdx.u.vmcall; + __u64 reg_mask =3D kvm_rcx_read(vcpu); + +#define COPY_REG(MASK, REG) \ + do { \ + if (reg_mask & TDX_VMCALL_REG_MASK_ ## MASK) \ + kvm_## REG ## _write(vcpu, tdx_vmcall->out_ ## REG); \ + } while (0) + + + COPY_REG(R10, r10); + COPY_REG(R11, r11); + COPY_REG(R12, r12); + COPY_REG(R13, r13); + COPY_REG(R14, r14); + COPY_REG(R15, r15); + COPY_REG(RBX, rbx); + COPY_REG(RDI, rdi); + COPY_REG(RSI, rsi); + COPY_REG(R8, r8); + COPY_REG(R9, r9); + COPY_REG(RDX, rdx); + +#undef COPY_REG + + return 1; +} + +static int tdx_vp_vmcall_to_user(struct kvm_vcpu *vcpu) +{ + struct kvm_tdx_vmcall *tdx_vmcall =3D &vcpu->run->tdx.u.vmcall; + __u64 reg_mask; + + vcpu->arch.complete_userspace_io =3D tdx_complete_vp_vmcall; + memset(tdx_vmcall, 0, sizeof(*tdx_vmcall)); + + vcpu->run->exit_reason =3D KVM_EXIT_TDX; + vcpu->run->tdx.type =3D KVM_EXIT_TDX_VMCALL; + + reg_mask =3D kvm_rcx_read(vcpu); + tdx_vmcall->reg_mask =3D reg_mask; + +#define COPY_REG(MASK, REG) \ + do { \ + if (reg_mask & TDX_VMCALL_REG_MASK_ ## MASK) { \ + tdx_vmcall->in_ ## REG =3D kvm_ ## REG ## _read(vcpu); \ + tdx_vmcall->out_ ## REG =3D tdx_vmcall->in_ ## REG; \ + } \ + } while (0) + + + COPY_REG(R10, r10); + COPY_REG(R11, r11); + COPY_REG(R12, r12); + COPY_REG(R13, r13); + COPY_REG(R14, r14); + COPY_REG(R15, r15); + COPY_REG(RBX, rbx); + COPY_REG(RDI, rdi); + COPY_REG(RSI, rsi); + COPY_REG(R8, r8); + COPY_REG(R9, r9); + COPY_REG(RDX, rdx); + +#undef COPY_REG + + /* notify userspace to handle the request */ + return 0; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) @@ -927,8 +999,16 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu) break; } =20 - tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); - return 1; + /* + * Unknown VMCALL. Toss the request to the user space VMM, e.g. qemu, + * as it may know how to handle. + * + * Those VMCALLs require user space VMM: + * TDG_VP_VMCALL_REPORT_FATAL_ERROR, TDG_VP_VMCALL_MAP_GPA, + * TDG_VP_VMCALL_SETUP_EVENT_NOTIFY_INTERRUPT, and + * TDG_VP_VMCALL_GET_QUOTE. + */ + return tdx_vp_vmcall_to_user(vcpu); } =20 void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index eb900344a054..c46321c48c0f 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -237,6 +237,90 @@ struct kvm_xen_exit { } u; }; =20 +struct kvm_tdx_exit { +#define KVM_EXIT_TDX_VMCALL 1 + __u32 type; + __u32 pad; + + union { + struct kvm_tdx_vmcall { + /* + * RAX(bit 0), RCX(bit 1) and RSP(bit 4) are reserved. + * RAX(bit 0): TDG.VP.VMCALL status code. + * RCX(bit 1): bitmap for used registers. + * RSP(bit 4): the caller stack. + */ +#define TDX_VMCALL_REG_MASK_RBX BIT_ULL(2) +#define TDX_VMCALL_REG_MASK_RDX BIT_ULL(3) +#define TDX_VMCALL_REG_MASK_RSI BIT_ULL(6) +#define TDX_VMCALL_REG_MASK_RDI BIT_ULL(7) +#define TDX_VMCALL_REG_MASK_R8 BIT_ULL(8) +#define TDX_VMCALL_REG_MASK_R9 BIT_ULL(9) +#define TDX_VMCALL_REG_MASK_R10 BIT_ULL(10) +#define TDX_VMCALL_REG_MASK_R11 BIT_ULL(11) +#define TDX_VMCALL_REG_MASK_R12 BIT_ULL(12) +#define TDX_VMCALL_REG_MASK_R13 BIT_ULL(13) +#define TDX_VMCALL_REG_MASK_R14 BIT_ULL(14) +#define TDX_VMCALL_REG_MASK_R15 BIT_ULL(15) + union { + __u64 in_rcx; + __u64 reg_mask; + }; + + /* + * Guest-Host-Communication Interface for TDX spec + * defines the ABI for TDG.VP.VMCALL. + */ + /* Input parameters: guest -> VMM */ + union { + __u64 in_r10; + __u64 type; + }; + union { + __u64 in_r11; + __u64 subfunction; + }; + /* + * Subfunction specific. + * Registers are used in this order to pass input + * arguments. r12=3Darg0, r13=3Darg1, etc. + */ + __u64 in_r12; + __u64 in_r13; + __u64 in_r14; + __u64 in_r15; + __u64 in_rbx; + __u64 in_rdi; + __u64 in_rsi; + __u64 in_r8; + __u64 in_r9; + __u64 in_rdx; + + /* Output parameters: VMM -> guest */ + union { + __u64 out_r10; + __u64 status_code; + }; + /* + * Subfunction specific. + * Registers are used in this order to output return + * values. r11=3Dret0, r12=3Dret1, etc. + */ + __u64 out_r11; + __u64 out_r12; + __u64 out_r13; + __u64 out_r14; + __u64 out_r15; + __u64 out_rbx; + __u64 out_rdi; + __u64 out_rsi; + __u64 out_r8; + __u64 out_r9; + __u64 out_rdx; + } vmcall; + } u; +}; + #define KVM_S390_GET_SKEYS_NONE 1 #define KVM_S390_SKEYS_MAX 1048576 =20 @@ -279,6 +363,7 @@ struct kvm_xen_exit { #define KVM_EXIT_RISCV_CSR 36 #define KVM_EXIT_NOTIFY 37 #define KVM_EXIT_MEMORY_FAULT 38 +#define KVM_EXIT_TDX 39 =20 /* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */ @@ -532,6 +617,8 @@ struct kvm_run { __u64 gpa; __u64 size; } memory; + /* KVM_EXIT_TDX_VMCALL */ + struct kvm_tdx_exit tdx; /* Fix the size of the union. */ char padding[256]; }; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57F7CEB64DD for ; Tue, 25 Jul 2023 22:25:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230317AbjGYWZD (ORCPT ); Tue, 25 Jul 2023 18:25:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232893AbjGYWXI (ORCPT ); Tue, 25 Jul 2023 18:23:08 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19278525A; Tue, 25 Jul 2023 15:18:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323500; x=1721859500; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6MnXbxUuZVLlZM6cT7ntdzX4oNXlyMD//p82EXhTiNI=; b=Mgh28IKKYFB5rKOsxhjkLpDTJPFcmeCO7e74u7u+m2D382KItvnKxLza FDfY37jmwlCQuOZg40C18r3k4qWI7kd4ConChIZkCkpx8oIs2htUN9Qt5 0zw54Qw2K9jyy4tai0PumvwfFc3RVEvGJeC1mLWsHOHnwFcU88dRJ19sN mFnEu3eWwPCB9RW7WWbaZNyfHeDE9LKQFGw/PhEQm7hg3pPLP23z6z5fv OjpbB9MGdK62jpFC0n4/Xd2i/6Fr+Cpuf87Qaswio7gTDCJ4oMlUwmBbZ 9DRhLt5XgfU3pVR8vQa0uUv2KgHnRZXCUskWpQhxR4oY57dwNPQCn9jDt Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882701" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882701" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001896" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001896" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:02 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 092/115] KVM: TDX: Handle TDX PV CPUID hypercall Date: Tue, 25 Jul 2023 15:14:43 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Wire up TDX PV CPUID hypercall to the KVM backend function. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 0e95e5c79337..9760d592bc68 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -989,12 +989,34 @@ static int tdx_vp_vmcall_to_user(struct kvm_vcpu *vcp= u) return 0; } =20 +static int tdx_emulate_cpuid(struct kvm_vcpu *vcpu) +{ + u32 eax, ebx, ecx, edx; + + /* EAX and ECX for cpuid is stored in R12 and R13. */ + eax =3D tdvmcall_a0_read(vcpu); + ecx =3D tdvmcall_a1_read(vcpu); + + kvm_cpuid(vcpu, &eax, &ebx, &ecx, &edx, false); + + tdvmcall_a0_write(vcpu, eax); + tdvmcall_a1_write(vcpu, ebx); + tdvmcall_a2_write(vcpu, ecx); + tdvmcall_a3_write(vcpu, edx); + + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + + return 1; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) return tdx_emulate_vmcall(vcpu); =20 switch (tdvmcall_leaf(vcpu)) { + case EXIT_REASON_CPUID: + return tdx_emulate_cpuid(vcpu); default: break; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13DFDC001E0 for ; Tue, 25 Jul 2023 22:25:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233018AbjGYWZG (ORCPT ); Tue, 25 Jul 2023 18:25:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232903AbjGYWXI (ORCPT ); Tue, 25 Jul 2023 18:23:08 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33D62525F; Tue, 25 Jul 2023 15:18:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323502; x=1721859502; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zAvts/lhrxcaq+89MhHSSlnL5ZiM8bhS++C5LhP2iVo=; b=AgC59eOS5/vPeYHRv2V5CC9bxLaFdF1NQOJPm+5jgmQDxLuXU76IeCzc yaVs4/RMfkbXP5tZXMreZiJEIZG8cmeFdqXvp0LzLNBvOeKSwp+CF0ZgR WPmhdrJ0Zvk5uUkgo4IlCPe6o1PWK47Es1dSxiPHrJFD1z4Vgcd29UXQS HiJtQPk6Kd3+HluN/tg76m6gfCOpesx89/nBCHJ6DE0dg7/10+e3OryZZ HWFCyXulPb43RKcy+XsY/0Phkfx5q3G3irEQRCiCZ67HXEycox5//hxUD EVjRzf34Njqft9voSFDulzVFTb0MOLEgOqKd+58SqLP/F3eDgUWcWHCs8 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882705" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882705" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001899" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001899" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:03 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 093/115] KVM: TDX: Handle TDX PV HLT hypercall Date: Tue, 25 Jul 2023 15:14:44 -0700 Message-Id: <5b1c274d0a8d672a439e0c380e9d8965e711488f.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Wire up TDX PV HLT hypercall to the KVM backend function. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 42 +++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.h | 3 +++ 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 9760d592bc68..d3ea368db5c6 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -622,7 +622,32 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) =20 bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) { - return pi_has_pending_interrupt(vcpu); + bool ret =3D pi_has_pending_interrupt(vcpu); + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + if (ret || vcpu->arch.mp_state !=3D KVM_MP_STATE_HALTED) + return true; + + if (tdx->interrupt_disabled_hlt) + return false; + + /* + * This is for the case where the virtual interrupt is recognized, + * i.e. set in vmcs.RVI, between the STI and "HLT". KVM doesn't have + * access to RVI and the interrupt is no longer in the PID (because it + * was "recognized". It doesn't get delivered in the guest because the + * TDCALL completes before interrupts are enabled. + * + * TDX modules sets RVI while in an STI interrupt shadow. + * - TDExit(typically TDG.VP.VMCALL) from the guest to TDX module. + * The interrupt shadow at this point is gone. + * - It knows that there is an interrupt that can be delivered + * (RVI > PPR && EFLAGS.IF=3D1, the other conditions of 29.2.2 don't + * matter) + * - It forwards the TDExit nevertheless, to a clueless hypervisor that + * has no way to glean either RVI or PPR. + */ + return !!xchg(&tdx->buggy_hlt_workaround, 0); } =20 void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) @@ -1009,6 +1034,17 @@ static int tdx_emulate_cpuid(struct kvm_vcpu *vcpu) return 1; } =20 +static int tdx_emulate_hlt(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx =3D to_tdx(vcpu); + + /* See tdx_protected_apic_has_interrupt() to avoid heavy seamcall */ + tdx->interrupt_disabled_hlt =3D tdvmcall_a0_read(vcpu); + + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + return kvm_emulate_halt_noskip(vcpu); +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) @@ -1017,6 +1053,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu) switch (tdvmcall_leaf(vcpu)) { case EXIT_REASON_CPUID: return tdx_emulate_cpuid(vcpu); + case EXIT_REASON_HLT: + return tdx_emulate_hlt(vcpu); default: break; } @@ -1360,6 +1398,8 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, in= t delivery_mode, struct kvm_vcpu *vcpu =3D apic->vcpu; struct vcpu_tdx *tdx =3D to_tdx(vcpu); =20 + /* See comment in tdx_protected_apic_has_interrupt(). */ + tdx->buggy_hlt_workaround =3D 1; /* TDX supports only posted interrupt. No lapic emulation. */ __vmx_deliver_posted_interrupt(vcpu, &tdx->pi_desc, vector); } diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index e03f7192dfab..c0cc09cb77ba 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -101,6 +101,9 @@ struct vcpu_tdx { bool host_state_need_restore; u64 msr_host_kernel_gs_base; =20 + bool interrupt_disabled_hlt; + unsigned int buggy_hlt_workaround; + /* * Dummy to make pmu_intel not corrupt memory. * TODO: Support PMU for TDX. Future work. --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B84AC001E0 for ; Tue, 25 Jul 2023 22:25:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233041AbjGYWZK (ORCPT ); Tue, 25 Jul 2023 18:25:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232910AbjGYWXJ (ORCPT ); Tue, 25 Jul 2023 18:23:09 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C68945264; Tue, 25 Jul 2023 15:18:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323504; x=1721859504; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8V2SRp2uSRLLY3IXZcPybdE4foVsDrQlY95Wj3aquPE=; b=HBegaGZ3J3VOv0VfitOpKR1yDRgBRJlggP+cBVw5/mBkIsR+aSximBk1 M5UyGIvt7VdSiyCbQKs92w5d7jxMFdtzcJDRw7k46328soRrJoLY87xJS E+GQXZa/i7FPCGTpYMzUI9c4dNIf7lBp9ASr47k7H2xUvSwaeKpCgS53a wgpThEQ81rPJj0nCko0N9Sd42+29o9a7iji+/n7qDu9LSPESaCKiXtSOQ tVfgDlYzJVn2Is6jA0eKe+X7rVGM9oihtEi7I7vuCnWaRiDSAO0SlBizk 478MeDC/4hygJr+yoey0g6KqVNsKyW4jODei5GdbrKIq8wuFp0JEvfk0U g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882709" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882709" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001903" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001903" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:03 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 094/115] KVM: TDX: Handle TDX PV port io hypercall Date: Tue, 25 Jul 2023 15:14:45 -0700 Message-Id: <1e1f52c90dede77d0feadb19b1dc16bb66d85a35.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Wire up TDX PV port IO hypercall to the KVM backend function. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index d3ea368db5c6..965c5fecea6c 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1045,6 +1045,61 @@ static int tdx_emulate_hlt(struct kvm_vcpu *vcpu) return kvm_emulate_halt_noskip(vcpu); } =20 +static int tdx_complete_pio_in(struct kvm_vcpu *vcpu) +{ + struct x86_emulate_ctxt *ctxt =3D vcpu->arch.emulate_ctxt; + unsigned long val =3D 0; + int ret; + + WARN_ON_ONCE(vcpu->arch.pio.count !=3D 1); + + ret =3D ctxt->ops->pio_in_emulated(ctxt, vcpu->arch.pio.size, + vcpu->arch.pio.port, &val, 1); + WARN_ON_ONCE(!ret); + + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + tdvmcall_set_return_val(vcpu, val); + + return 1; +} + +static int tdx_emulate_io(struct kvm_vcpu *vcpu) +{ + struct x86_emulate_ctxt *ctxt =3D vcpu->arch.emulate_ctxt; + unsigned long val =3D 0; + unsigned int port; + int size, ret; + bool write; + + ++vcpu->stat.io_exits; + + size =3D tdvmcall_a0_read(vcpu); + write =3D tdvmcall_a1_read(vcpu); + port =3D tdvmcall_a2_read(vcpu); + + if (size !=3D 1 && size !=3D 2 && size !=3D 4) { + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); + return 1; + } + + if (write) { + val =3D tdvmcall_a3_read(vcpu); + ret =3D ctxt->ops->pio_out_emulated(ctxt, size, port, &val, 1); + + /* No need for a complete_userspace_io callback. */ + vcpu->arch.pio.count =3D 0; + } else { + ret =3D ctxt->ops->pio_in_emulated(ctxt, size, port, &val, 1); + if (!ret) + vcpu->arch.complete_userspace_io =3D tdx_complete_pio_in; + else + tdvmcall_set_return_val(vcpu, val); + } + if (ret) + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + return ret; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) @@ -1055,6 +1110,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu) return tdx_emulate_cpuid(vcpu); case EXIT_REASON_HLT: return tdx_emulate_hlt(vcpu); + case EXIT_REASON_IO_INSTRUCTION: + return tdx_emulate_io(vcpu); default: break; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7977C001DF for ; Tue, 25 Jul 2023 22:25:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232845AbjGYWZN (ORCPT ); Tue, 25 Jul 2023 18:25:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232942AbjGYWXM (ORCPT ); Tue, 25 Jul 2023 18:23:12 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E1402D78; Tue, 25 Jul 2023 15:18:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323508; x=1721859508; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eiGUrVA5quiO+3ftMhbMiiBQjxsLAuQ6jgYPFkSsacM=; b=Z/aAQ0nazeungQv7QJajgZ3DaTRB94ozQn0YBBu2qXJ3V8dXTUAU3juK dtEcSjEqTz5E80DEO0tUpqM55jhrVJOT7kg+NtVITcm2TWjutzoN1lIuc 2i7XbhULoXdKcOPY0yuAAiybPiAEgbE+x6SMR7X3lOB8IujWWr1A3E3mt MrByEXbhJM/WrtuGtMjNJ5alP0SmgExiBTJR2WTHf85fH1oAFFYQx5Ioo Qm/oemFBb6xqsAUsiZq31s3Q3McmJiMr3V1gRAhASHPJX6CJbHGqBAm+2 LFZNomwxNt8AT/BnNRGGHvU5JCSiGOKXkxZm3iQqBuWgOd5PEWDjROOX8 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882713" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882713" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001912" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001912" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:03 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com, Sean Christopherson Subject: [PATCH v15 095/115] KVM: TDX: Handle TDX PV MMIO hypercall Date: Tue, 25 Jul 2023 15:14:46 -0700 Message-Id: <294b19ab4b5fe6cc4293e2c9e27045538dad3609.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Export kvm_io_bus_read and kvm_mmio tracepoint and wire up TDX PV MMIO hypercall to the KVM backend functions. kvm_io_bus_read/write() searches KVM device emulated in kernel of the given MMIO address and emulates the MMIO. As TDX PV MMIO also needs it, export kvm_io_bus_read(). kvm_io_bus_write() is already exported. TDX PV MMIO emulates some of MMIO itself. To add trace point consistently with x86 kvm, export kvm_mmio tracepoint. Signed-off-by: Sean Christopherson Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx.c | 114 +++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 1 + virt/kvm/kvm_main.c | 2 + 3 files changed, 117 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 965c5fecea6c..efdfda11931a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1100,6 +1100,118 @@ static int tdx_emulate_io(struct kvm_vcpu *vcpu) return ret; } =20 +static int tdx_complete_mmio(struct kvm_vcpu *vcpu) +{ + unsigned long val =3D 0; + gpa_t gpa; + int size; + + KVM_BUG_ON(vcpu->mmio_needed !=3D 1, vcpu->kvm); + vcpu->mmio_needed =3D 0; + + if (!vcpu->mmio_is_write) { + gpa =3D vcpu->mmio_fragments[0].gpa; + size =3D vcpu->mmio_fragments[0].len; + + memcpy(&val, vcpu->run->mmio.data, size); + tdvmcall_set_return_val(vcpu, val); + trace_kvm_mmio(KVM_TRACE_MMIO_READ, size, gpa, &val); + } + return 1; +} + +static inline int tdx_mmio_write(struct kvm_vcpu *vcpu, gpa_t gpa, int siz= e, + unsigned long val) +{ + if (kvm_iodevice_write(vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && + kvm_io_bus_write(vcpu, KVM_MMIO_BUS, gpa, size, &val)) + return -EOPNOTSUPP; + + trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, size, gpa, &val); + return 0; +} + +static inline int tdx_mmio_read(struct kvm_vcpu *vcpu, gpa_t gpa, int size) +{ + unsigned long val; + + if (kvm_iodevice_read(vcpu, &vcpu->arch.apic->dev, gpa, size, &val) && + kvm_io_bus_read(vcpu, KVM_MMIO_BUS, gpa, size, &val)) + return -EOPNOTSUPP; + + tdvmcall_set_return_val(vcpu, val); + trace_kvm_mmio(KVM_TRACE_MMIO_READ, size, gpa, &val); + return 0; +} + +static int tdx_emulate_mmio(struct kvm_vcpu *vcpu) +{ + struct kvm_memory_slot *slot; + int size, write, r; + unsigned long val; + gpa_t gpa; + + KVM_BUG_ON(vcpu->mmio_needed, vcpu->kvm); + + size =3D tdvmcall_a0_read(vcpu); + write =3D tdvmcall_a1_read(vcpu); + gpa =3D tdvmcall_a2_read(vcpu); + val =3D write ? tdvmcall_a3_read(vcpu) : 0; + + if (size !=3D 1 && size !=3D 2 && size !=3D 4 && size !=3D 8) + goto error; + if (write !=3D 0 && write !=3D 1) + goto error; + + /* Strip the shared bit, allow MMIO with and without it set. */ + gpa =3D gpa & ~gfn_to_gpa(kvm_gfn_shared_mask(vcpu->kvm)); + + if (size > 8u || ((gpa + size - 1) ^ gpa) & PAGE_MASK) + goto error; + + slot =3D kvm_vcpu_gfn_to_memslot(vcpu, gpa_to_gfn(gpa)); + if (slot && !(slot->flags & KVM_MEMSLOT_INVALID)) + goto error; + + if (!kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) { + trace_kvm_fast_mmio(gpa); + return 1; + } + + if (write) + r =3D tdx_mmio_write(vcpu, gpa, size, val); + else + r =3D tdx_mmio_read(vcpu, gpa, size); + if (!r) { + /* Kernel completed device emulation. */ + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + return 1; + } + + /* Request the device emulation to userspace device model. */ + vcpu->mmio_needed =3D 1; + vcpu->mmio_is_write =3D write; + vcpu->arch.complete_userspace_io =3D tdx_complete_mmio; + + vcpu->run->mmio.phys_addr =3D gpa; + vcpu->run->mmio.len =3D size; + vcpu->run->mmio.is_write =3D write; + vcpu->run->exit_reason =3D KVM_EXIT_MMIO; + + if (write) { + memcpy(vcpu->run->mmio.data, &val, size); + } else { + vcpu->mmio_fragments[0].gpa =3D gpa; + vcpu->mmio_fragments[0].len =3D size; + trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, size, gpa, NULL); + } + return 0; + +error: + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); + return 1; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) @@ -1112,6 +1224,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu) return tdx_emulate_hlt(vcpu); case EXIT_REASON_IO_INSTRUCTION: return tdx_emulate_io(vcpu); + case EXIT_REASON_EPT_VIOLATION: + return tdx_emulate_mmio(vcpu); default: break; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 747cc86c60dc..f2d07cbaa12d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -13698,6 +13698,7 @@ EXPORT_SYMBOL_GPL(kvm_sev_es_string_io); =20 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_entry); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_exit); +EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_mmio); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_fast_mmio); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_inj_virq); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_page_fault); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 0c277e1f5f12..e56cfb22df89 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2587,6 +2587,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struc= t kvm_vcpu *vcpu, gfn_t gfn =20 return NULL; } +EXPORT_SYMBOL_GPL(kvm_vcpu_gfn_to_memslot); =20 bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn) { @@ -5780,6 +5781,7 @@ int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_b= us bus_idx, gpa_t addr, r =3D __kvm_io_bus_read(vcpu, bus, &range, val); return r < 0 ? r : 0; } +EXPORT_SYMBOL_GPL(kvm_io_bus_read); =20 /* Caller must hold slots_lock. */ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t a= ddr, --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE32FC0015E for ; Tue, 25 Jul 2023 22:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233065AbjGYWZT (ORCPT ); Tue, 25 Jul 2023 18:25:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232955AbjGYWXO (ORCPT ); Tue, 25 Jul 2023 18:23:14 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B91DB3A97; Tue, 25 Jul 2023 15:18:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323509; x=1721859509; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PSQNj6d59PHEYQQpzGT6gzHz2d6z1MMi0SssdsccrCA=; b=MYT3q3CQqke7PxrYQwp3hPi4huEMGYDZOthjhU1CIAOJU4byLWNaSI+N Mpn14PUfY0iHwvoPhJB1TUWUhOPZ0XExf5tINB2rofcUtp+jIe4lqtkrd 3bZD/bGzQgt1hy7qEUyAFayNMLK1+gdwAfk5geuYQU/BBb3lYTFb3m1he Ccld0FEvoJjf7onFD3PkbYYU5pD5dQYtuIOlKAC3FDPioQF0wCA/dEest Z5hpwcT8whbHFg1wWeS41L/7oVwBPkR/A799Z5GE9YSP8YmF0AChUWeRE cCoUuug+EuS9Fswt0sTP+l3BRGO+UUE5z4qwLG6W9jKKA2bLHutHcKAnw w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882717" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882717" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001917" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001917" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:04 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 096/115] KVM: TDX: Implement callbacks for MSR operations for TDX Date: Tue, 25 Jul 2023 15:14:47 -0700 Message-Id: <94c66e909469e80ebe4def37ff765381576265cf.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Implements set_msr/get_msr/has_emulated_msr methods for TDX to handle hypercall from guest TD for paravirtualized rdmsr and wrmsr. The TDX module virtualizes MSRs. For some MSRs, it injects #VE to the guest TD upon RDMSR or WRMSR. The exact list of such MSRs are defined in the spec. Upon #VE, the guest TD may execute hypercalls, TDG.VP.VMCALL and TDG.VP.VMCALL, which are defined in GHCI (Guest-Host Communication Interface) so that the host VMM (e.g. KVM) can virtualize the MSRs. There are three classes of MSRs virtualization. - non-configurable: TDX module directly virtualizes it. VMM can't configure. the value set by KVM_SET_MSR_INDEX_LIST is ignored. - configurable: TDX module directly virtualizes it. VMM can configure at the VM creation time. The value set by KVM_SET_MSR_INDEX_LIST is used. - #VE case Guest TD would issue TDG.VP.VMCALL and VMM handles the MSR hypercall. The value set by KVM_SET_MSR_INDEX_LIST is used. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/main.c | 44 +++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.c | 70 ++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 6 ++++ arch/x86/kvm/x86.c | 1 - arch/x86/kvm/x86.h | 2 ++ 5 files changed, 118 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 2774533128af..a05640c6916b 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -258,6 +258,42 @@ static void vt_handle_exit_irqoff(struct kvm_vcpu *vcp= u) vmx_handle_exit_irqoff(vcpu); } =20 +static int vt_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + if (unlikely(is_td_vcpu(vcpu))) + return tdx_set_msr(vcpu, msr_info); + + return vmx_set_msr(vcpu, msr_info); +} + +/* + * The kvm parameter can be NULL (module initialization, or invocation bef= ore + * VM creation). Be sure to check the kvm parameter before using it. + */ +static bool vt_has_emulated_msr(struct kvm *kvm, u32 index) +{ + if (kvm && is_td(kvm)) + return tdx_has_emulated_msr(index, true); + + return vmx_has_emulated_msr(kvm, index); +} + +static int vt_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + if (unlikely(is_td_vcpu(vcpu))) + return tdx_get_msr(vcpu, msr_info); + + return vmx_get_msr(vcpu, msr_info); +} + +static void vt_msr_filter_changed(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_msr_filter_changed(vcpu); +} + static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu) { struct pi_desc *pi =3D vcpu_to_pi_desc(vcpu); @@ -519,7 +555,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .hardware_enable =3D vt_hardware_enable, .hardware_disable =3D vt_hardware_disable, - .has_emulated_msr =3D vmx_has_emulated_msr, + .has_emulated_msr =3D vt_has_emulated_msr, =20 .is_vm_type_supported =3D vt_is_vm_type_supported, .max_vcpus =3D vt_max_vcpus, @@ -541,8 +577,8 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .update_exception_bitmap =3D vmx_update_exception_bitmap, .get_msr_feature =3D vmx_get_msr_feature, - .get_msr =3D vmx_get_msr, - .set_msr =3D vmx_set_msr, + .get_msr =3D vt_get_msr, + .set_msr =3D vt_set_msr, .get_segment_base =3D vmx_get_segment_base, .get_segment =3D vmx_get_segment, .set_segment =3D vmx_set_segment, @@ -651,7 +687,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .apic_init_signal_blocked =3D vmx_apic_init_signal_blocked, .migrate_timers =3D vmx_migrate_timers, =20 - .msr_filter_changed =3D vmx_msr_filter_changed, + .msr_filter_changed =3D vt_msr_filter_changed, .complete_emulated_msr =3D kvm_complete_insn_gp, =20 .vcpu_deliver_sipi_vector =3D kvm_vcpu_deliver_sipi_vector, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index efdfda11931a..dc31b052f6a7 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1731,6 +1731,76 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *r= eason, *error_code =3D 0; } =20 +static bool tdx_is_emulated_kvm_msr(u32 index, bool write) +{ + switch (index) { + case MSR_KVM_POLL_CONTROL: + return true; + default: + return false; + } +} + +bool tdx_has_emulated_msr(u32 index, bool write) +{ + switch (index) { + case MSR_IA32_UCODE_REV: + case MSR_IA32_ARCH_CAPABILITIES: + case MSR_IA32_POWER_CTL: + case MSR_IA32_CR_PAT: + case MSR_IA32_TSC_DEADLINE: + case MSR_IA32_MISC_ENABLE: + case MSR_PLATFORM_INFO: + case MSR_MISC_FEATURES_ENABLES: + case MSR_IA32_MCG_CAP: + case MSR_IA32_MCG_STATUS: + case MSR_IA32_MCG_CTL: + case MSR_IA32_MCG_EXT_CTL: + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + /* MSR_IA32_MCx_{CTL, STATUS, ADDR, MISC, CTL2} */ + return true; + case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: + /* + * x2APIC registers that are virtualized by the CPU can't be + * emulated, KVM doesn't have access to the virtual APIC page. + */ + switch (index) { + case X2APIC_MSR(APIC_TASKPRI): + case X2APIC_MSR(APIC_PROCPRI): + case X2APIC_MSR(APIC_EOI): + case X2APIC_MSR(APIC_ISR) ... X2APIC_MSR(APIC_ISR + APIC_ISR_NR): + case X2APIC_MSR(APIC_TMR) ... X2APIC_MSR(APIC_TMR + APIC_ISR_NR): + case X2APIC_MSR(APIC_IRR) ... X2APIC_MSR(APIC_IRR + APIC_ISR_NR): + return false; + default: + return true; + } + case MSR_IA32_APICBASE: + case MSR_EFER: + return !write; + case 0x4b564d00 ... 0x4b564dff: + /* KVM custom MSRs */ + return tdx_is_emulated_kvm_msr(index, write); + default: + return false; + } +} + +int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) +{ + if (tdx_has_emulated_msr(msr->index, false)) + return kvm_get_msr_common(vcpu, msr); + return 1; +} + +int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) +{ + if (tdx_has_emulated_msr(msr->index, true)) + return kvm_set_msr_common(vcpu, msr); + return 1; +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 850844cdeadf..ca070cb3348e 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -166,6 +166,9 @@ void tdx_deliver_interrupt(struct kvm_lapic *apic, int = delivery_mode, void tdx_inject_nmi(struct kvm_vcpu *vcpu); void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code); +bool tdx_has_emulated_msr(u32 index, bool write); +int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); +int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 @@ -208,6 +211,9 @@ static inline void tdx_deliver_interrupt(struct kvm_lap= ic *apic, int delivery_mo static inline void tdx_inject_nmi(struct kvm_vcpu *vcpu) {} static inline void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u= 64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) {} +static inline bool tdx_has_emulated_msr(u32 index, bool write) { return fa= lse; } +static inline int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)= { return 1; } +static inline int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)= { return 1; } =20 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f2d07cbaa12d..d6ec1ee6d8e1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -89,7 +89,6 @@ #include "trace.h" =20 #define MAX_IO_MSRS 256 -#define KVM_MAX_MCE_BANKS 32 =20 struct kvm_caps kvm_caps __read_mostly =3D { .supported_mce_cap =3D MCG_CTL_P | MCG_SER_P, diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 7de3a45f655a..5795ca0e75e5 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -9,6 +9,8 @@ #include "kvm_cache_regs.h" #include "kvm_emulate.h" =20 +#define KVM_MAX_MCE_BANKS 32 + bool __kvm_is_vm_type_supported(unsigned long type); =20 struct kvm_caps { --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E56FAEB64DD for ; Tue, 25 Jul 2023 22:25:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233097AbjGYWZb (ORCPT ); Tue, 25 Jul 2023 18:25:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233027AbjGYWXU (ORCPT ); Tue, 25 Jul 2023 18:23:20 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61BA155AE; Tue, 25 Jul 2023 15:18:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323519; x=1721859519; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ccMcBcc23AgnL9dY1KKhHHjOC35FCd6UT6pYEdXqS/4=; b=VrqdoaG/c9SSRnC1fXnESLE5GSATuKVV3Ia0CW1Cq1h7WxK2brY7FPp6 BLuoiA8cBYqIef5eSJv29mQjqcR09V9ZHvgfcspM/OGXbvfWYdMIQ3Jk+ Uc85EXoMAiEE+c5ZMyJW1JozsxYnQ66RwJhr9mwLtYGs9/+trWlQAMjvy GENHSCVLPjEl4ZISXniLFtJTVbp5xjyLfc8hMfPDshYkK/rzfEtbOHY4J asePxwZ0/EM+QouvyJLMrzEwpHiXEraQJVviKyN36c+xw2j574gI9g/ZP oJSKjxHC45qYidBfuPjbDeQ/ovVBlMmRWFqYcTht+lMMHSBIrx6bcQUD8 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882723" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882723" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001920" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001920" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:04 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 097/115] KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall Date: Tue, 25 Jul 2023 15:14:48 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Wire up TDX PV rdmsr/wrmsr hypercall to the KVM backend function. Signed-off-by: Isaku Yamahata Reviewed-by: Paolo Bonzini --- arch/x86/kvm/vmx/tdx.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index dc31b052f6a7..98bdcfc06283 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1212,6 +1212,41 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu) return 1; } =20 +static int tdx_emulate_rdmsr(struct kvm_vcpu *vcpu) +{ + u32 index =3D tdvmcall_a0_read(vcpu); + u64 data; + + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ) || + kvm_get_msr(vcpu, index, &data)) { + trace_kvm_msr_read_ex(index); + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); + return 1; + } + trace_kvm_msr_read(index, data); + + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + tdvmcall_set_return_val(vcpu, data); + return 1; +} + +static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu) +{ + u32 index =3D tdvmcall_a0_read(vcpu); + u64 data =3D tdvmcall_a1_read(vcpu); + + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_WRITE) || + kvm_set_msr(vcpu, index, data)) { + trace_kvm_msr_write_ex(index, data); + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); + return 1; + } + + trace_kvm_msr_write(index, data); + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + return 1; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) @@ -1226,6 +1261,10 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu) return tdx_emulate_io(vcpu); case EXIT_REASON_EPT_VIOLATION: return tdx_emulate_mmio(vcpu); + case EXIT_REASON_MSR_READ: + return tdx_emulate_rdmsr(vcpu); + case EXIT_REASON_MSR_WRITE: + return tdx_emulate_wrmsr(vcpu); default: break; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 247F5EB64DD for ; Tue, 25 Jul 2023 22:25:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231666AbjGYWZ1 (ORCPT ); Tue, 25 Jul 2023 18:25:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233032AbjGYWXU (ORCPT ); Tue, 25 Jul 2023 18:23:20 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 661D455AF; Tue, 25 Jul 2023 15:18:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323519; x=1721859519; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RpcCPxBgvrrF2LrPG1804injK1mZCQv/0KkU30+BTQU=; b=D8aLsDUI+CF5OUHRmelLNC9kgu5ww4Okvi9JDVMlLJZEKDe2PLzLlAu7 WTee/a2VoB/kgAmR7QrCJaMKnmfF/CpFGIkNhUdeY6IqJW413KLsBE7OX Lg1gs1lIIPA2h2WjeHZEYjvdAoeAKiRnNCghVfDE4ZFmLlC5bI1tEaoAo 6a5foJrZK5q2UVzUinkBSqpnY5rEb7OJ6HE4uaOl2sOQ/nHlSw+C61ED6 SiD6yMAylEIsIhnPEg34aB/9q65w1JNLJ/2F/Cn5EOwqE0yYznTlpMhqQ Me/7ZjPvjDINUjp8jugELdkw0HQ8TqjJH4kkAysBFemC3eG9+H8J3pFAb g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882727" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882727" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001923" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001923" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:05 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 098/115] KVM: TDX: Handle MSR MTRRCap and MTRRDefType access Date: Tue, 25 Jul 2023 15:14:49 -0700 Message-Id: <08fc4e86a10adeac0379b8dd364d2ee96b467dce.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Handle MTRRCap RO MSR to return all features are unsupported and handle MTRRDefType MSR to accept only E=3D1,FE=3D0,type=3Dwriteback. enable MTRR, disable Fixed range MTRRs, default memory type=3Dwriteback TDX virtualizes that cpuid to report MTRR to guest TD and TDX enforces guest CR0.CD=3D0. If guest tries to set CR0.CD=3D1, it results in #GP. Whi= le updating MTRR requires to set CR0.CD=3D1 (and other cache flushing operations). It means guest TD can't update MTRR. Virtualize MTRR as all features disabled and default memory type as writeback. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 99 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 82 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 98bdcfc06283..3775db455f29 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -544,18 +544,7 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, b= ool is_mmio) if (!kvm_arch_has_noncoherent_dma(vcpu->kvm)) return (MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; =20 - /* - * TDX enforces CR0.CD =3D 0 and KVM MTRR emulation enforces writeback. - * TODO: implement MTRR MSR emulation so that - * MTRRCap: SMRR=3D0: SMRR interface unsupported - * WC=3D0: write combining unsupported - * FIX=3D0: Fixed range registers unsupported - * VCNT=3D0: number of variable range regitsers =3D 0 - * MTRRDefType: E=3D1, FE=3D0, type=3Dwriteback only. Don't allow other v= alue. - * E=3D1: enable MTRR - * FE=3D0: disable fixed range MTRRs - * type: default memory type=3Dwriteback - */ + /* TDX enforces CR0.CD =3D 0 and KVM MTRR emulation enforces writeback. */ return MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT; } =20 @@ -1786,7 +1775,9 @@ bool tdx_has_emulated_msr(u32 index, bool write) case MSR_IA32_UCODE_REV: case MSR_IA32_ARCH_CAPABILITIES: case MSR_IA32_POWER_CTL: + case MSR_MTRRcap: case MSR_IA32_CR_PAT: + case MSR_MTRRdefType: case MSR_IA32_TSC_DEADLINE: case MSR_IA32_MISC_ENABLE: case MSR_PLATFORM_INFO: @@ -1828,16 +1819,47 @@ bool tdx_has_emulated_msr(u32 index, bool write) =20 int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) { - if (tdx_has_emulated_msr(msr->index, false)) - return kvm_get_msr_common(vcpu, msr); - return 1; + switch (msr->index) { + case MSR_MTRRcap: + /* + * Override kvm_mtrr_get_msr() which hardcodes the value. + * Report SMRR =3D 0, WC =3D 0, FIX =3D 0 VCNT =3D 0 to disable MTRR + * effectively. + */ + msr->data =3D 0; + return 0; + default: + if (tdx_has_emulated_msr(msr->index, false)) + return kvm_get_msr_common(vcpu, msr); + return 1; + } } =20 int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) { - if (tdx_has_emulated_msr(msr->index, true)) + switch (msr->index) { + case MSR_MTRRdefType: + /* + * Allow writeback only for all memory. + * Because it's reported that fixed range MTRR isn't supported + * and VCNT=3D0, enforce MTRRDefType.FE =3D 0 and don't care + * variable range MTRRs. Only default memory type matters. + * + * bit 11 E: MTRR enable/disable + * bit 12 FE: Fixed-range MTRRs enable/disable + * (E, FE) =3D (1, 1): enable MTRR and Fixed range MTRR + * (E, FE) =3D (1, 0): enable MTRR, disable Fixed range MTRR + * (E, FE) =3D (0, *): disable all MTRRs. all physical memory + * is UC + */ + if (msr->data !=3D ((1 << 11) | MTRR_TYPE_WRBACK)) + return 1; return kvm_set_msr_common(vcpu, msr); - return 1; + default: + if (tdx_has_emulated_msr(msr->index, true)) + return kvm_set_msr_common(vcpu, msr); + return 1; + } } =20 static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) @@ -2596,6 +2618,45 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u= 64 vcpu_rcx) return ret; } =20 +static int tdx_vcpu_init_mtrr(struct kvm_vcpu *vcpu) +{ + struct msr_data msr; + int ret; + int i; + + /* + * To avoid confusion with reporting VNCT =3D 0, explicitly disable + * vaiale-range reisters. + */ + for (i =3D 0; i < KVM_NR_VAR_MTRR; i++) { + /* phymask */ + msr =3D (struct msr_data) { + .host_initiated =3D true, + .index =3D 0x200 + 2 * i + 1, + .data =3D 0, /* valid =3D 0 to disable. */ + }; + ret =3D kvm_set_msr_common(vcpu, &msr); + if (ret) + return -EINVAL; + } + + /* Set MTRR to use writeback on reset. */ + msr =3D (struct msr_data) { + .host_initiated =3D true, + .index =3D MSR_MTRRdefType, + /* + * Set E(enable MTRR)=3D1, FE(enable fixed range MTRR)=3D0, default + * type=3Dwriteback on reset to avoid UC. Note E=3D0 means all + * memory is UC. + */ + .data =3D (1 << 11) | MTRR_TYPE_WRBACK, + }; + ret =3D kvm_set_msr_common(vcpu, &msr); + if (ret) + return -EINVAL; + return 0; +} + int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { struct msr_data apic_base_msr; @@ -2633,6 +2694,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __use= r *argp) if (kvm_set_apic_base(vcpu, &apic_base_msr)) return -EINVAL; =20 + ret =3D tdx_vcpu_init_mtrr(vcpu); + if (ret) + return ret; + ret =3D tdx_td_vcpu_init(vcpu, (u64)cmd.data); if (ret) return ret; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6519EC001DF for ; Tue, 25 Jul 2023 22:25:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232926AbjGYWZX (ORCPT ); Tue, 25 Jul 2023 18:25:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233036AbjGYWXV (ORCPT ); Tue, 25 Jul 2023 18:23:21 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4044A55B8; Tue, 25 Jul 2023 15:18:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323521; x=1721859521; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/ls1BzqS9FKY4HkJaupcxI2ZIkdzdAwcHI/XSkpQbg0=; b=SyqGjzF7TLfl/BvUJXcAGQCeogsNq9YkdRa/iry2rRk+9vODAQCaTS+O 82xDaYQTiXccEj4wDDAJKdZOteTiBr6BPX+50M0MQjk/CkQj6E/pRRUBP bJCA8WYYGoIX2z4mhWSIPM4EvqW+n5R4fBgAiVhObHZY33aMT+KHj+REZ 3JpcNE5ZC9uK3dxz6BePx4abE/Gzmj7RSCngY3n1kFUTAo5UI2T04YXUS gEKNQW0HdZyvbjKss8V75BTDweNczw6KQnr/PyvZxPBCDhzo5n1e2DJ7f aptoB+B9mWUWXec+aqhpJRcGyBzUoznmOO1Ayc+f8KQS9GMeY1pEpfGih A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882732" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882732" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001927" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001927" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:05 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 099/115] KVM: TDX: Handle MSR IA32_FEAT_CTL MSR and IA32_MCG_EXT_CTL Date: Tue, 25 Jul 2023 15:14:50 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata MCE and MCA is advertised via cpuid based on the TDX module spec. Guest kernel can access IA32_FEAT_CTL for checking if LMCE is enabled by platform and IA32_MCG_EXT_CTL to enable LMCE. Make TDX KVM handle them. Otherwise guest MSR access to them with TDG.VP.VMCALL on VE results in GP in guest. Because LMCE is disabled with qemu by default, "-cpu lmce=3Don" to qemu command line is needed to reproduce it. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 3775db455f29..77052f49481a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1806,6 +1806,7 @@ bool tdx_has_emulated_msr(u32 index, bool write) default: return true; } + case MSR_IA32_FEAT_CTL: case MSR_IA32_APICBASE: case MSR_EFER: return !write; @@ -1820,6 +1821,20 @@ bool tdx_has_emulated_msr(u32 index, bool write) int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) { switch (msr->index) { + case MSR_IA32_FEAT_CTL: + /* + * MCE and MCA are advertised via cpuid. guest kernel could + * check if LMCE is enabled or not. + */ + msr->data =3D FEAT_CTL_LOCKED; + if (vcpu->arch.mcg_cap & MCG_LMCE_P) + msr->data |=3D FEAT_CTL_LMCE_ENABLED; + return 0; + case MSR_IA32_MCG_EXT_CTL: + if (!msr->host_initiated && !(vcpu->arch.mcg_cap & MCG_LMCE_P)) + return 1; + msr->data =3D vcpu->arch.mcg_ext_ctl; + return 0; case MSR_MTRRcap: /* * Override kvm_mtrr_get_msr() which hardcodes the value. @@ -1838,6 +1853,11 @@ int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_da= ta *msr) int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) { switch (msr->index) { + case MSR_IA32_MCG_EXT_CTL: + if (!msr->host_initiated && !(vcpu->arch.mcg_cap & MCG_LMCE_P)) + return 1; + vcpu->arch.mcg_ext_ctl =3D msr->data; + return 0; case MSR_MTRRdefType: /* * Allow writeback only for all memory. --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 570FBEB64DD for ; Tue, 25 Jul 2023 22:25:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232934AbjGYWZg (ORCPT ); Tue, 25 Jul 2023 18:25:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233072AbjGYWXY (ORCPT ); Tue, 25 Jul 2023 18:23:24 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F2F93C01; Tue, 25 Jul 2023 15:18:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323525; x=1721859525; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=18s/5jF2F7qugI8wqlhLGvFV9SRmY3sKou5s8wvtIz8=; b=hVfo5+3KnyiRDALOsfshWuh/tGRYWwxCOPEpTVCM6rmuTsIgI8Vj/ud0 xe9SVWb5IjX254YjhQEUN+yOrE+4+/bnfjfU+lO9ooZbud6YinHUkDDfa EV2uUqT667V+IHFuw/FDf8zXo1nwfkPJQUlfzIdVlzq5jqoJAKbI/akv+ cezn4EDu9TGbprWMOFlolnnosQUkDNWB9AwsbUC+IYvaoR/z3klQLQOQ7 yCddefdR4s6+G3mgbWtouuLWOrg3mRj98uG6XpYtVT0MMlU2cwjlHKP7R 5es4TM7cQgBGFytLnHnGY10QG7KRn84c1r54fSlpE8lpOCLuXPsnRymTY Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882736" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882736" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001932" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001932" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:06 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 100/115] KVM: TDX: Handle TDG.VP.VMCALL hypercall Date: Tue, 25 Jul 2023 15:14:51 -0700 Message-Id: <590ced75e0bf1b003b755adfeac8622653d7e321.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Implement TDG.VP.VMCALL hypercall. If the input value is zero, return success code and zero in output registers. TDG.VP.VMCALL hypercall is a subleaf of TDG.VP.VMCALL to enumerate which TDG.VP.VMCALL sub leaves are supported. This hypercall is for future enhancement of the Guest-Host-Communication Interface (GHCI) specification. The GHCI version of 344426-001US defines it to require input R12 to be zero and to return zero in output registers, R11, R12, R13, and R14 so that guest TD enumerates no enhancement. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/tdx.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 77052f49481a..639fab4fc2cb 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1236,6 +1236,20 @@ static int tdx_emulate_wrmsr(struct kvm_vcpu *vcpu) return 1; } =20 +static int tdx_get_td_vm_call_info(struct kvm_vcpu *vcpu) +{ + if (tdvmcall_a0_read(vcpu)) + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_INVALID_OPERAND); + else { + tdvmcall_set_return_code(vcpu, TDG_VP_VMCALL_SUCCESS); + kvm_r11_write(vcpu, 0); + tdvmcall_a0_write(vcpu, 0); + tdvmcall_a1_write(vcpu, 0); + tdvmcall_a2_write(vcpu, 0); + } + return 1; +} + static int handle_tdvmcall(struct kvm_vcpu *vcpu) { if (tdvmcall_exit_type(vcpu)) @@ -1254,6 +1268,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu) return tdx_emulate_rdmsr(vcpu); case EXIT_REASON_MSR_WRITE: return tdx_emulate_wrmsr(vcpu); + case TDG_VP_VMCALL_GET_TD_VM_CALL_INFO: + return tdx_get_td_vm_call_info(vcpu); default: break; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB9BAEB64DD for ; Tue, 25 Jul 2023 22:25:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233003AbjGYWZl (ORCPT ); Tue, 25 Jul 2023 18:25:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233081AbjGYWXZ (ORCPT ); Tue, 25 Jul 2023 18:23:25 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBBBE59C8; Tue, 25 Jul 2023 15:18:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323526; x=1721859526; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bN4RJChk86YfL7+QzEZwomRb6o6QyzquH+PVEzfTX/M=; b=BcjO7bWvYLMQqdc4T6AmAYd47rfU6mpNG4d4DEPyyD1m0gH5T7Ur1mub v13a1PXkcKI5Q+e3FqO5Mgs6Pzuvdn1rAkqDO/ToCC6RkfbS//jE3EH6/ 64EBfWN6IbzfmDsuDtbsglDv3wV7ueIG7pKV5DjPh18fETGi93oLYxfZg dBk5Rj0pLlPI/o5Px4KVJ4QO1vYpf4oC15TIGnpKKxagf1PuKqG1GmVNY k6u1SH4uXU0dn5h7EeirboA4pXK9pzb0GZGpT2SrF06XvqNe1ybMJJkgJ +hHuWgtGfZMtRsfzlOtYap6p235sw7e3wB2k2kUzXEbjGT+qSLz5oEwk+ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882741" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882741" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001935" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001935" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:06 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 101/115] KVM: TDX: Silently discard SMI request Date: Tue, 25 Jul 2023 15:14:52 -0700 Message-Id: <92f65dfaf9e2430a42d629f75482f0f0a8993ca4.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX doesn't support system-management mode (SMM) and system-management interrupt (SMI) in guest TDs. Because guest state (vcpu state, memory state) is protected, it must go through the TDX module APIs to change guest state, injecting SMI and changing vcpu mode into SMM. The TDX module doesn't provide a way for VMM to inject SMI into guest TD and a way for VMM to switch guest vcpu mode into SMM. We have two options in KVM when handling SMM or SMI in the guest TD or the device model (e.g. QEMU): 1) silently ignore the request or 2) return a meaningful error. For simplicity, we implemented the option 1). Signed-off-by: Isaku Yamahata --- arch/x86/kvm/smm.h | 7 +++++- arch/x86/kvm/vmx/main.c | 45 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 29 ++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 12 ++++++++++ 4 files changed, 88 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/smm.h b/arch/x86/kvm/smm.h index a1cf2ac5bd78..bc77902f5c18 100644 --- a/arch/x86/kvm/smm.h +++ b/arch/x86/kvm/smm.h @@ -142,7 +142,12 @@ union kvm_smram { =20 static inline int kvm_inject_smi(struct kvm_vcpu *vcpu) { - kvm_make_request(KVM_REQ_SMI, vcpu); + /* + * If SMM isn't supported (e.g. TDX), silently discard SMI request. + * Assume that SMM supported =3D MSR_IA32_SMBASE supported. + */ + if (static_call(kvm_x86_has_emulated_msr)(vcpu->kvm, MSR_IA32_SMBASE)) + kvm_make_request(KVM_REQ_SMI, vcpu); return 0; } =20 diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index a05640c6916b..d7e64093461e 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -294,6 +294,43 @@ static void vt_msr_filter_changed(struct kvm_vcpu *vcp= u) vmx_msr_filter_changed(vcpu); } =20 +#ifdef CONFIG_KVM_SMM +static int vt_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection) +{ + if (is_td_vcpu(vcpu)) + return tdx_smi_allowed(vcpu, for_injection); + + return vmx_smi_allowed(vcpu, for_injection); +} + +static int vt_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) +{ + if (unlikely(is_td_vcpu(vcpu))) + return tdx_enter_smm(vcpu, smram); + + return vmx_enter_smm(vcpu, smram); +} + +static int vt_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smra= m) +{ + if (unlikely(is_td_vcpu(vcpu))) + return tdx_leave_smm(vcpu, smram); + + return vmx_leave_smm(vcpu, smram); +} + +static void vt_enable_smi_window(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + tdx_enable_smi_window(vcpu); + return; + } + + /* RSM will cause a vmexit anyway. */ + vmx_enable_smi_window(vcpu); +} +#endif + static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu) { struct pi_desc *pi =3D vcpu_to_pi_desc(vcpu); @@ -677,10 +714,10 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .setup_mce =3D vmx_setup_mce, =20 #ifdef CONFIG_KVM_SMM - .smi_allowed =3D vmx_smi_allowed, - .enter_smm =3D vmx_enter_smm, - .leave_smm =3D vmx_leave_smm, - .enable_smi_window =3D vmx_enable_smi_window, + .smi_allowed =3D vt_smi_allowed, + .enter_smm =3D vt_enter_smm, + .leave_smm =3D vt_leave_smm, + .enable_smi_window =3D vt_enable_smi_window, #endif =20 .can_emulate_instruction =3D vmx_can_emulate_instruction, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 639fab4fc2cb..14b05e51d10a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1898,6 +1898,35 @@ int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_da= ta *msr) } } =20 +#ifdef CONFIG_KVM_SMM +int tdx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection) +{ + /* SMI isn't supported for TDX. */ + WARN_ON_ONCE(1); + return false; +} + +int tdx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) +{ + /* smi_allowed() is always false for TDX as above. */ + WARN_ON_ONCE(1); + return 0; +} + +int tdx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) +{ + WARN_ON_ONCE(1); + return 0; +} + +void tdx_enable_smi_window(struct kvm_vcpu *vcpu) +{ + /* SMI isn't supported for TDX. Silently discard SMI request. */ + WARN_ON_ONCE(1); + vcpu->arch.smi_pending =3D false; +} +#endif + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index ca070cb3348e..91b5f91a8f66 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -223,4 +223,16 @@ static inline int tdx_sept_flush_remote_tlbs(struct kv= m *kvm) { return 0; } static inline void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa,= int root_level) {} #endif =20 +#if defined(CONFIG_INTEL_TDX_HOST) && defined(CONFIG_KVM_SMM) +int tdx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection); +int tdx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram); +int tdx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram); +void tdx_enable_smi_window(struct kvm_vcpu *vcpu); +#else +static inline int tdx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injectio= n) { return false; } +static inline int tdx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *sm= ram) { return 0; } +static inline int tdx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smr= am *smram) { return 0; } +static inline void tdx_enable_smi_window(struct kvm_vcpu *vcpu) {} +#endif + #endif /* __KVM_X86_VMX_X86_OPS_H */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A315EB64DD for ; Tue, 25 Jul 2023 22:25:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233015AbjGYWZq (ORCPT ); Tue, 25 Jul 2023 18:25:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233100AbjGYWX1 (ORCPT ); Tue, 25 Jul 2023 18:23:27 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2918259D1; Tue, 25 Jul 2023 15:18:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323528; x=1721859528; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HGfxQOqsSsXje5TLQgCTu8XBtPumaiFEv5x79lM4dV8=; b=Oj5jSTfdlUGshp+tx6HHyw4lxCxhTK4p4lcb2zcjiTuXILGypb9W+2nR OXu1LiqxwQpBEj9W6bdACiNKU3S9J7ACFrPJvHkpX54N5/M67sGMgHaSx YP80MlrkW+tNCYOSqXSIbpjnfnwqfYRQc4ALVkEkGIySRl9zm0js7qcix o9PXhpP02+TlfqT/7Jgk50FWKvOfmiTBHcghOfV6St386MSZ/F7nz2l/y VFxy+RoBfcJ3cHxShamik4cbvUzGTXR0Fke5kGTTMHQMZuOpoRJHGZrbB 6TQY9y/9guLbi2yuhNYlBGys5MTdDAca0LuPJvVhI3eY+xXFboAG8+aBp w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882744" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882744" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001940" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001940" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:07 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 102/115] KVM: TDX: Silently ignore INIT/SIPI Date: Tue, 25 Jul 2023 15:14:53 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata The TDX module API doesn't provide API for VMM to inject INIT IPI and SIPI. Instead it defines the different protocols to boot application processors. Ignore INIT and SIPI events for the TDX guest. There are two options. 1) (silently) ignore INIT/SIPI request or 2) return error to guest TDs somehow. Given that TDX guest is paravirtualized to boot AP, the option 1 is chosen for simplicity. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/lapic.c | 19 +++++++++++------- arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/vmx/main.c | 32 ++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 4 ++-- 6 files changed, 48 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index 663a40418434..ba9cc4ac9093 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -145,6 +145,7 @@ KVM_X86_OP_OPTIONAL(migrate_timers) KVM_X86_OP(msr_filter_changed) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) +KVM_X86_OP(vcpu_deliver_init) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) KVM_X86_OP_OPTIONAL(gmem_invalidate) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index c58ceded3437..291d36a668e5 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1793,6 +1793,7 @@ struct kvm_x86_ops { int (*complete_emulated_msr)(struct kvm_vcpu *vcpu, int err); =20 void (*vcpu_deliver_sipi_vector)(struct kvm_vcpu *vcpu, u8 vector); + void (*vcpu_deliver_init)(struct kvm_vcpu *vcpu); =20 /* * Returns vCPU specific APICv inhibit reasons @@ -2033,6 +2034,7 @@ void kvm_get_segment(struct kvm_vcpu *vcpu, struct kv= m_segment *var, int seg); void kvm_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg); int kvm_load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector, int s= eg); void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); +void kvm_vcpu_deliver_init(struct kvm_vcpu *vcpu); =20 int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, int reason, bool has_error_code, u32 error_code); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index d2d1a9531c96..3cf8284c56c5 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -3231,6 +3231,16 @@ int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 = data, unsigned long len) return 0; } =20 +void kvm_vcpu_deliver_init(struct kvm_vcpu *vcpu) +{ + kvm_vcpu_reset(vcpu, true); + if (kvm_vcpu_is_bsp(vcpu)) + vcpu->arch.mp_state =3D KVM_MP_STATE_RUNNABLE; + else + vcpu->arch.mp_state =3D KVM_MP_STATE_INIT_RECEIVED; +} +EXPORT_SYMBOL_GPL(kvm_vcpu_deliver_init); + int kvm_apic_accept_events(struct kvm_vcpu *vcpu) { struct kvm_lapic *apic =3D vcpu->arch.apic; @@ -3262,13 +3272,8 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu) return 0; } =20 - if (test_and_clear_bit(KVM_APIC_INIT, &apic->pending_events)) { - kvm_vcpu_reset(vcpu, true); - if (kvm_vcpu_is_bsp(apic->vcpu)) - vcpu->arch.mp_state =3D KVM_MP_STATE_RUNNABLE; - else - vcpu->arch.mp_state =3D KVM_MP_STATE_INIT_RECEIVED; - } + if (test_and_clear_bit(KVM_APIC_INIT, &apic->pending_events)) + static_call(kvm_x86_vcpu_deliver_init)(vcpu); if (test_and_clear_bit(KVM_APIC_SIPI, &apic->pending_events)) { if (vcpu->arch.mp_state =3D=3D KVM_MP_STATE_INIT_RECEIVED) { /* evaluate pending_events before reading the vector */ diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index d681dd7ad397..3560927145b5 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4918,6 +4918,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata =3D { .complete_emulated_msr =3D svm_complete_emulated_msr, =20 .vcpu_deliver_sipi_vector =3D svm_vcpu_deliver_sipi_vector, + .vcpu_deliver_init =3D kvm_vcpu_deliver_init, .vcpu_get_apicv_inhibit_reasons =3D avic_vcpu_get_apicv_inhibit_reasons, }; =20 diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index d7e64093461e..7e9e6adcbf49 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -331,6 +331,14 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu) } #endif =20 +static bool vt_apic_init_signal_blocked(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return true; + + return vmx_apic_init_signal_blocked(vcpu); +} + static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu) { struct pi_desc *pi =3D vcpu_to_pi_desc(vcpu); @@ -359,6 +367,25 @@ static void vt_deliver_interrupt(struct kvm_lapic *api= c, int delivery_mode, vmx_deliver_interrupt(apic, delivery_mode, trig_mode, vector); } =20 +static void vt_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) +{ + if (is_td_vcpu(vcpu)) + return; + + kvm_vcpu_deliver_sipi_vector(vcpu, vector); +} + +static void vt_vcpu_deliver_init(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + /* TDX doesn't support INIT. Ignore INIT event */ + vcpu->arch.mp_state =3D KVM_MP_STATE_RUNNABLE; + return; + } + + kvm_vcpu_deliver_init(vcpu); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -721,13 +748,14 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { #endif =20 .can_emulate_instruction =3D vmx_can_emulate_instruction, - .apic_init_signal_blocked =3D vmx_apic_init_signal_blocked, + .apic_init_signal_blocked =3D vt_apic_init_signal_blocked, .migrate_timers =3D vmx_migrate_timers, =20 .msr_filter_changed =3D vt_msr_filter_changed, .complete_emulated_msr =3D kvm_complete_insn_gp, =20 - .vcpu_deliver_sipi_vector =3D kvm_vcpu_deliver_sipi_vector, + .vcpu_deliver_sipi_vector =3D vt_vcpu_deliver_sipi_vector, + .vcpu_deliver_init =3D vt_vcpu_deliver_init, =20 .mem_enc_ioctl =3D vt_mem_enc_ioctl, .vcpu_mem_enc_ioctl =3D vt_vcpu_mem_enc_ioctl, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 14b05e51d10a..34fb3146f702 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -715,8 +715,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu) void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) { =20 - /* Ignore INIT silently because TDX doesn't support INIT event. */ - if (init_event) + /* vcpu_deliver_init method silently discards INIT event. */ + if (KVM_BUG_ON(init_event, vcpu->kvm)) return; if (KVM_BUG_ON(is_td_vcpu_created(to_tdx(vcpu)), vcpu->kvm)) return; --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5351C001DF for ; Tue, 25 Jul 2023 22:25:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233100AbjGYWZt (ORCPT ); Tue, 25 Jul 2023 18:25:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233132AbjGYWXa (ORCPT ); Tue, 25 Jul 2023 18:23:30 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0784659DF; Tue, 25 Jul 2023 15:18:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323532; x=1721859532; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XFGxJx+wSq7gHClpnyZgMPPz7lQiiBja8GZBZUPOkw0=; b=KcHtt+DY7IdsDyrspVR9imj+cHAUttBOC8dbP0NXUUfMHSFXol/Z33Y9 M1/vOuo9P6GJR5NxqQ+KjEG3wJDv43uj0Ipuj+bEtCFL3D6ux9JgMSJcL /UB4IjjTbJUbEKgL12R2shWxtbOeH2RxJZ701Oo8XpSIRoyGd+0bdofT2 wzCHobcRSSXRZXjHkM3AIobXCy1dBJi+iKQo6XaPQyCN8l+7ZBtoe0jUp 6mUwKe/A8apU1GRPLAGdlvERRB+y/XUafBtZ5pf/r/c1h6D2VqoETWoQm UIUUaPvPXP7mNcTspTPlwRCQt3e1OkPY4YNyDdFtEdOspoLzP1F+gWCOx A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882749" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882749" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001943" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001943" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:07 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 103/115] KVM: TDX: Add methods to ignore accesses to CPU state Date: Tue, 25 Jul 2023 15:14:54 -0700 Message-Id: <089bedecf9c618177745eede18d06542ba4e6938.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX protects TDX guest state from VMM. Implement access methods for TDX guest state to ignore them or return zero. Because those methods can be called by kvm ioctls to set/get cpu registers, they don't have KVM_BUG_ON except one method. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 269 +++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 49 ++++++- arch/x86/kvm/vmx/x86_ops.h | 13 ++ 3 files changed, 304 insertions(+), 27 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 7e9e6adcbf49..0164c9dd1bfa 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -386,6 +386,184 @@ static void vt_vcpu_deliver_init(struct kvm_vcpu *vcp= u) kvm_vcpu_deliver_init(vcpu); } =20 +static void vt_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_vcpu_after_set_cpuid(vcpu); +} + +static void vt_update_exception_bitmap(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_update_exception_bitmap(vcpu); +} + +static u64 vt_get_segment_base(struct kvm_vcpu *vcpu, int seg) +{ + if (is_td_vcpu(vcpu)) + return tdx_get_segment_base(vcpu, seg); + + return vmx_get_segment_base(vcpu, seg); +} + +static void vt_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, + int seg) +{ + if (is_td_vcpu(vcpu)) { + tdx_get_segment(vcpu, var, seg); + return; + } + + vmx_get_segment(vcpu, var, seg); +} + +static void vt_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, + int seg) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_segment(vcpu, var, seg); +} + +static int vt_get_cpl(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return tdx_get_cpl(vcpu); + + return vmx_get_cpl(vcpu); +} + +static void vt_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l) +{ + if (is_td_vcpu(vcpu)) { + *db =3D 0; + *l =3D 0; + return; + } + + vmx_get_cs_db_l_bits(vcpu, db, l); +} + +static void vt_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_cr0(vcpu, cr0); +} + +static void vt_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_cr4(vcpu, cr4); +} + +static int vt_set_efer(struct kvm_vcpu *vcpu, u64 efer) +{ + if (is_td_vcpu(vcpu)) + return 0; + + return vmx_set_efer(vcpu, efer); +} + +static void vt_get_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +{ + if (is_td_vcpu(vcpu)) { + memset(dt, 0, sizeof(*dt)); + return; + } + + vmx_get_idt(vcpu, dt); +} + +static void vt_set_idt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_idt(vcpu, dt); +} + +static void vt_get_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +{ + if (is_td_vcpu(vcpu)) { + memset(dt, 0, sizeof(*dt)); + return; + } + + vmx_get_gdt(vcpu, dt); +} + +static void vt_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_gdt(vcpu, dt); +} + +static void vt_set_dr7(struct kvm_vcpu *vcpu, unsigned long val) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_dr7(vcpu, val); +} + +static void vt_sync_dirty_debug_regs(struct kvm_vcpu *vcpu) +{ + /* + * MOV-DR exiting is always cleared for TD guest, even in debug mode. + * Thus KVM_DEBUGREG_WONT_EXIT can never be set and it should never + * reach here for TD vcpu. + */ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return; + + vmx_sync_dirty_debug_regs(vcpu); +} + +static void vt_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) +{ + if (is_td_vcpu(vcpu)) { + tdx_cache_reg(vcpu, reg); + return; + } + + vmx_cache_reg(vcpu, reg); +} + +static unsigned long vt_get_rflags(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return tdx_get_rflags(vcpu); + + return vmx_get_rflags(vcpu); +} + +static void vt_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_rflags(vcpu, rflags); +} + +static bool vt_get_if_flag(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return false; + + return vmx_get_if_flag(vcpu); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -530,6 +708,14 @@ static void vt_inject_irq(struct kvm_vcpu *vcpu, bool = reinjected) vmx_inject_irq(vcpu, reinjected); } =20 +static void vt_inject_exception(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_inject_exception(vcpu); +} + static void vt_cancel_injection(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) @@ -576,6 +762,39 @@ static void vt_get_exit_info(struct kvm_vcpu *vcpu, u3= 2 *reason, vmx_get_exit_info(vcpu, reason, info1, info2, intr_info, error_code); } =20 + +static void vt_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int ir= r) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_update_cr8_intercept(vcpu, tpr, irr); +} + +static void vt_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitma= p) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_load_eoi_exitmap(vcpu, eoi_exit_bitmap); +} + +static int vt_set_tss_addr(struct kvm *kvm, unsigned int addr) +{ + if (is_td(kvm)) + return 0; + + return vmx_set_tss_addr(kvm, addr); +} + +static int vt_set_identity_map_addr(struct kvm *kvm, u64 ident_addr) +{ + if (is_td(kvm)) + return 0; + + return vmx_set_identity_map_addr(kvm, ident_addr); +} + static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { if (is_td_vcpu(vcpu)) @@ -639,29 +858,29 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .vcpu_load =3D vt_vcpu_load, .vcpu_put =3D vt_vcpu_put, =20 - .update_exception_bitmap =3D vmx_update_exception_bitmap, + .update_exception_bitmap =3D vt_update_exception_bitmap, .get_msr_feature =3D vmx_get_msr_feature, .get_msr =3D vt_get_msr, .set_msr =3D vt_set_msr, - .get_segment_base =3D vmx_get_segment_base, - .get_segment =3D vmx_get_segment, - .set_segment =3D vmx_set_segment, - .get_cpl =3D vmx_get_cpl, - .get_cs_db_l_bits =3D vmx_get_cs_db_l_bits, - .set_cr0 =3D vmx_set_cr0, + .get_segment_base =3D vt_get_segment_base, + .get_segment =3D vt_get_segment, + .set_segment =3D vt_set_segment, + .get_cpl =3D vt_get_cpl, + .get_cs_db_l_bits =3D vt_get_cs_db_l_bits, + .set_cr0 =3D vt_set_cr0, .is_valid_cr4 =3D vmx_is_valid_cr4, - .set_cr4 =3D vmx_set_cr4, - .set_efer =3D vmx_set_efer, - .get_idt =3D vmx_get_idt, - .set_idt =3D vmx_set_idt, - .get_gdt =3D vmx_get_gdt, - .set_gdt =3D vmx_set_gdt, - .set_dr7 =3D vmx_set_dr7, - .sync_dirty_debug_regs =3D vmx_sync_dirty_debug_regs, - .cache_reg =3D vmx_cache_reg, - .get_rflags =3D vmx_get_rflags, - .set_rflags =3D vmx_set_rflags, - .get_if_flag =3D vmx_get_if_flag, + .set_cr4 =3D vt_set_cr4, + .set_efer =3D vt_set_efer, + .get_idt =3D vt_get_idt, + .set_idt =3D vt_set_idt, + .get_gdt =3D vt_get_gdt, + .set_gdt =3D vt_set_gdt, + .set_dr7 =3D vt_set_dr7, + .sync_dirty_debug_regs =3D vt_sync_dirty_debug_regs, + .cache_reg =3D vt_cache_reg, + .get_rflags =3D vt_get_rflags, + .set_rflags =3D vt_set_rflags, + .get_if_flag =3D vt_get_if_flag, =20 .flush_tlb_all =3D vt_flush_tlb_all, .flush_tlb_current =3D vt_flush_tlb_current, @@ -678,7 +897,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .patch_hypercall =3D vmx_patch_hypercall, .inject_irq =3D vt_inject_irq, .inject_nmi =3D vt_inject_nmi, - .inject_exception =3D vmx_inject_exception, + .inject_exception =3D vt_inject_exception, .cancel_injection =3D vt_cancel_injection, .interrupt_allowed =3D vt_interrupt_allowed, .nmi_allowed =3D vt_nmi_allowed, @@ -686,11 +905,11 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .set_nmi_mask =3D vt_set_nmi_mask, .enable_nmi_window =3D vt_enable_nmi_window, .enable_irq_window =3D vt_enable_irq_window, - .update_cr8_intercept =3D vmx_update_cr8_intercept, + .update_cr8_intercept =3D vt_update_cr8_intercept, .set_virtual_apic_mode =3D vmx_set_virtual_apic_mode, .set_apic_access_page_addr =3D vmx_set_apic_access_page_addr, .refresh_apicv_exec_ctrl =3D vmx_refresh_apicv_exec_ctrl, - .load_eoi_exitmap =3D vmx_load_eoi_exitmap, + .load_eoi_exitmap =3D vt_load_eoi_exitmap, .apicv_post_state_restore =3D vt_apicv_post_state_restore, .required_apicv_inhibits =3D VMX_REQUIRED_APICV_INHIBITS, .hwapic_irr_update =3D vmx_hwapic_irr_update, @@ -701,13 +920,13 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .dy_apicv_has_pending_interrupt =3D pi_has_pending_interrupt, .protected_apic_has_interrupt =3D vt_protected_apic_has_interrupt, =20 - .set_tss_addr =3D vmx_set_tss_addr, - .set_identity_map_addr =3D vmx_set_identity_map_addr, + .set_tss_addr =3D vt_set_tss_addr, + .set_identity_map_addr =3D vt_set_identity_map_addr, .get_mt_mask =3D vt_get_mt_mask, =20 .get_exit_info =3D vt_get_exit_info, =20 - .vcpu_after_set_cpuid =3D vmx_vcpu_after_set_cpuid, + .vcpu_after_set_cpuid =3D vt_vcpu_after_set_cpuid, =20 .has_wbinvd_exit =3D cpu_has_vmx_wbinvd_exit, =20 diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 34fb3146f702..c9151ea25793 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -3,6 +3,7 @@ #include =20 #include +#include #include =20 #include "capabilities.h" @@ -576,8 +577,15 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) =20 vcpu->arch.tsc_offset =3D to_kvm_tdx(vcpu->kvm)->tsc_offset; vcpu->arch.l1_tsc_offset =3D vcpu->arch.tsc_offset; - vcpu->arch.guest_state_protected =3D - !(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG); + /* + * TODO: support off-TD debug. If TD DEBUG is enabled, guest state + * can be accessed. guest_state_protected =3D false. and kvm ioctl to + * access CPU states should be usable for user space VMM (e.g. qemu). + * + * vcpu->arch.guest_state_protected =3D + * !(to_kvm_tdx(vcpu->kvm)->attributes & TDX_TD_ATTRIBUTE_DEBUG); + */ + vcpu->arch.guest_state_protected =3D true; =20 tdx->pi_desc.nv =3D POSTED_INTR_VECTOR; tdx->pi_desc.sn =3D 1; @@ -1927,6 +1935,43 @@ void tdx_enable_smi_window(struct kvm_vcpu *vcpu) } #endif =20 +int tdx_get_cpl(struct kvm_vcpu *vcpu) +{ + return 0; +} + +void tdx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) +{ + kvm_register_mark_available(vcpu, reg); + switch (reg) { + case VCPU_REGS_RSP: + case VCPU_REGS_RIP: + case VCPU_EXREG_PDPTR: + case VCPU_EXREG_CR0: + case VCPU_EXREG_CR3: + case VCPU_EXREG_CR4: + break; + default: + KVM_BUG_ON(1, vcpu->kvm); + break; + } +} + +unsigned long tdx_get_rflags(struct kvm_vcpu *vcpu) +{ + return 0; +} + +u64 tdx_get_segment_base(struct kvm_vcpu *vcpu, int seg) +{ + return 0; +} + +void tdx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg) +{ + memset(var, 0, sizeof(*var)); +} + static int tdx_get_capabilities(struct kvm_tdx_cmd *cmd) { struct kvm_tdx_capabilities __user *user_caps; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 91b5f91a8f66..0acbf5d34bff 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -170,6 +170,12 @@ bool tdx_has_emulated_msr(u32 index, bool write); int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); =20 +int tdx_get_cpl(struct kvm_vcpu *vcpu); +void tdx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg); +unsigned long tdx_get_rflags(struct kvm_vcpu *vcpu); +u64 tdx_get_segment_base(struct kvm_vcpu *vcpu, int seg); +void tdx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg); + int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); =20 void tdx_flush_tlb(struct kvm_vcpu *vcpu); @@ -215,6 +221,13 @@ static inline bool tdx_has_emulated_msr(u32 index, boo= l write) { return false; } static inline int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)= { return 1; } static inline int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)= { return 1; } =20 +static inline int tdx_get_cpl(struct kvm_vcpu *vcpu) { return 0; } +static inline void tdx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) = {} +static inline unsigned long tdx_get_rflags(struct kvm_vcpu *vcpu) { return= 0; } +static inline u64 tdx_get_segment_base(struct kvm_vcpu *vcpu, int seg) { r= eturn 0; } +static inline void tdx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segme= nt *var, + int seg) {} + static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)= { return -EOPNOTSUPP; } =20 static inline void tdx_flush_tlb(struct kvm_vcpu *vcpu) {} --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2B96EB64DD for ; Tue, 25 Jul 2023 22:25:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233102AbjGYWZy (ORCPT ); Tue, 25 Jul 2023 18:25:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233156AbjGYWXc (ORCPT ); Tue, 25 Jul 2023 18:23:32 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E89B759EA; Tue, 25 Jul 2023 15:18:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323534; x=1721859534; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2jaicnteAX7AqViZAIAdnp7uZ9I/T+xHcvc1Erw16UE=; b=I6L19VO3zePfQcK/VaCEWJa+xzLVeT0pSMXhOOoCBiHC1JnPoyssknjS MTSOY4m/2Wel4LB8CardJkGNnXY9Nrd3X9CmKTxY3uqiOAQzVAGmpHXRt zCa5qGPqfY9repRoc0Y1/NiZ8JNNyWOkxdkTb9kKNbUU+tT3NOSRlk+Dc EGgqLo2gW0Rsp7YXsfYREYI/RmPOBoFyQqzjQokFE/0Hc+VhdtKrjqkDD zQFgW8CPQ0QhpVADqmMJ+uuiHN2WGFPFzy6UNOLiCDfiBtjAlr09DXmS7 ivCfQHDqu1ZRo3NEJe1ADwamPWVRKkreJF/QlnxKom6apMNC1qY/8XZpl w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882753" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882753" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001946" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001946" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:07 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 104/115] KVM: TDX: Add methods to ignore guest instruction emulation Date: Tue, 25 Jul 2023 15:14:55 -0700 Message-Id: <50866e9bb3d9f18b5359fabdb5d469811c8b1c58.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because TDX protects TDX guest state from VMM, instructions in guest memory cannot be emulated. Implement methods to ignore guest instruction emulator. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 0164c9dd1bfa..fc443afbdbc7 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -331,6 +331,30 @@ static void vt_enable_smi_window(struct kvm_vcpu *vcpu) } #endif =20 +static bool vt_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_typ= e, + void *insn, int insn_len) +{ + if (is_td_vcpu(vcpu)) + return false; + + return vmx_can_emulate_instruction(vcpu, emul_type, insn, insn_len); +} + +static int vt_check_intercept(struct kvm_vcpu *vcpu, + struct x86_instruction_info *info, + enum x86_intercept_stage stage, + struct x86_exception *exception) +{ + /* + * This call back is triggered by the x86 instruction emulator. TDX + * doesn't allow guest memory inspection. + */ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return X86EMUL_UNHANDLEABLE; + + return vmx_check_intercept(vcpu, info, stage, exception); +} + static bool vt_apic_init_signal_blocked(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) @@ -937,7 +961,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .load_mmu_pgd =3D vt_load_mmu_pgd, =20 - .check_intercept =3D vmx_check_intercept, + .check_intercept =3D vt_check_intercept, .handle_exit_irqoff =3D vt_handle_exit_irqoff, =20 .request_immediate_exit =3D vt_request_immediate_exit, @@ -966,7 +990,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .enable_smi_window =3D vt_enable_smi_window, #endif =20 - .can_emulate_instruction =3D vmx_can_emulate_instruction, + .can_emulate_instruction =3D vt_can_emulate_instruction, .apic_init_signal_blocked =3D vt_apic_init_signal_blocked, .migrate_timers =3D vmx_migrate_timers, =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB6C7EB64DD for ; Tue, 25 Jul 2023 22:25:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232740AbjGYWZ5 (ORCPT ); Tue, 25 Jul 2023 18:25:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233224AbjGYWXj (ORCPT ); Tue, 25 Jul 2023 18:23:39 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E80041989; Tue, 25 Jul 2023 15:19:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323543; x=1721859543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XyzpfNDy09U+StlZT/CFQG86sC3VE7VJAC/TT4mf+pA=; b=EDgHvTFJA+x97pyEmw/oxGrcLpk02rch/PT1MGVlVfn08tbZN/qXDc+d WZN4ejK0JEmE7ZUbPGLKW0O/0l2e4TRlBxBZgpKO3VWXOFAvC4F7QpGA+ F/md6MLUSd43c183JaH2LfsgBhbzxbpI7ngOXolQnMkd/I0w8xROkFEbo 8JgyoLKWwIr81Ef+ltbDIy8YcQfsKecK/PQgcp0RhnMMIu5+DMmBSVIDk C7ySFxR7lwOenUnmgppoCJnwuO8KHEkm3jZgpTJ1blkf5NXe1Gb/TwPhk Vga+k8uIg8R9VxDs3AwKCcC/cCi0J2jrqAQeLjVWjAMW4vHpX9rKy00ou g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882758" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882758" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001949" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001949" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:08 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 105/115] KVM: TDX: Add a method to ignore dirty logging Date: Tue, 25 Jul 2023 15:14:56 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Currently TDX KVM doesn't support tracking dirty pages (yet). Implement a method to ignore it. Because the flag for kvm memory slot to enable dirty logging isn't accepted for TDX, warn on the method is called for TDX. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index fc443afbdbc7..38a782c28b72 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -827,6 +827,14 @@ static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t = gfn, bool is_mmio) return vmx_get_mt_mask(vcpu, gfn, is_mmio); } =20 +static void vt_update_cpu_dirty_logging(struct kvm_vcpu *vcpu) +{ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return; + + vmx_update_cpu_dirty_logging(vcpu); +} + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -969,7 +977,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .sched_in =3D vt_sched_in, =20 .cpu_dirty_log_size =3D PML_ENTITY_NUM, - .update_cpu_dirty_logging =3D vmx_update_cpu_dirty_logging, + .update_cpu_dirty_logging =3D vt_update_cpu_dirty_logging, =20 .nested_ops =3D &vmx_nested_ops, =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFB76EB64DD for ; Tue, 25 Jul 2023 22:26:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232815AbjGYW0C (ORCPT ); Tue, 25 Jul 2023 18:26:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233234AbjGYWXk (ORCPT ); Tue, 25 Jul 2023 18:23:40 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C00E51FE2; Tue, 25 Jul 2023 15:19:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323544; x=1721859544; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nmGQUIRYutZ8GYkQpZch+sNrtHiOxg6NOGiFVVLF1qk=; b=UoU/UNBF4o4K9ahP7sLMz6SA3wk4IM/yKXArdpCPLhfYx6vkwXSPZvBF /OKl2GWiIb7QCaq4UuJwQcc1r8RBIfht3YD8VWxlF045lncOkVb0vfIWl k6bWQnjvr5Ed994LqvILinoeyBqyiAULpXXGs8bUFOwCJ2gnHoq7IjhnU 41JVJUM2EMzGXFq/TX+7Jy9/NqpeoCWMKKbvRRrsgmdki7LMR7uNVSwf6 M1NNF4qr6fRPGSFC+VKGpmuGGtWsWA3v1Z3tKH3XrZnsqY/ZW87wY04j9 DMTQCshr6Ey8+xZkaWlixIyEWSiILGr87H/zz+EGFkB5wpZMPv8dxy+Bm A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882764" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882764" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001952" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001952" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:08 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 106/115] KVM: TDX: Add methods to ignore VMX preemption timer Date: Tue, 25 Jul 2023 15:14:57 -0700 Message-Id: <08069b5e67f5ad24a3c4dd3c5cbf7cca6850b469.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX doesn't support VMX preemption timer. Implement access methods for VMM to ignore VMX preemption timer. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 38a782c28b72..c2ad9c734376 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -835,6 +835,27 @@ static void vt_update_cpu_dirty_logging(struct kvm_vcp= u *vcpu) vmx_update_cpu_dirty_logging(vcpu); } =20 +#ifdef CONFIG_X86_64 +static int vt_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, + bool *expired) +{ + /* VMX-preemption timer isn't available for TDX. */ + if (is_td_vcpu(vcpu)) + return -EINVAL; + + return vmx_set_hv_timer(vcpu, guest_deadline_tsc, expired); +} + +static void vt_cancel_hv_timer(struct kvm_vcpu *vcpu) +{ + /* VMX-preemption timer can't be set. See vt_set_hv_timer(). */ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return; + + vmx_cancel_hv_timer(vcpu); +} +#endif + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -985,8 +1006,8 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .pi_start_assignment =3D vmx_pi_start_assignment, =20 #ifdef CONFIG_X86_64 - .set_hv_timer =3D vmx_set_hv_timer, - .cancel_hv_timer =3D vmx_cancel_hv_timer, + .set_hv_timer =3D vt_set_hv_timer, + .cancel_hv_timer =3D vt_cancel_hv_timer, #endif =20 .setup_mce =3D vmx_setup_mce, --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7DDBEB64DD for ; Tue, 25 Jul 2023 22:26:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232892AbjGYW0c (ORCPT ); Tue, 25 Jul 2023 18:26:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233244AbjGYWXk (ORCPT ); Tue, 25 Jul 2023 18:23:40 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A3C33A9E; Tue, 25 Jul 2023 15:19:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323545; x=1721859545; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cKuLNRzp3wBlzce5fueNwYUzg7I0dNiO3o6pLEg3h/k=; b=Yrmuos0Z1W2EJesqfM/x6E4MovYMAXcYlsjJvLT7doRcNQ7ro2o1lZdQ tc47Sk6JJ+41aRM7ndjeymJnmrG88CKFAjTUcCvKNeP/zqmU4lOTcKOUT Qwwzq5Ed4aZBCYEwyBqZZveJ5G+PUH8Q+e14Pf6q55Jzomm3L+s75DOS/ 0aaZPmacDehNYBJrrhE2Fy3YaeX/0+Cy9zzWv9OHgg83dBFLDfySGwqjl bCeNRQrqDYRtczNelf4rdeZH5cHJebtbDLrDm4yG2OtnrqsPviIr9ZOyn rCemAtstJIdER78OWE+FpQy5rzyFgeuhzsaRZmSAlV5D3Z3eSjdOhOZqH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882769" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882769" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001957" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001957" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:09 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 107/115] KVM: TDX: Add methods to ignore accesses to TSC Date: Tue, 25 Jul 2023 15:14:58 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX protects TDX guest TSC state from VMM. Implement access methods to ignore guest TSC. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 44 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index c2ad9c734376..ad74900bbc56 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -827,6 +827,42 @@ static u8 vt_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t = gfn, bool is_mmio) return vmx_get_mt_mask(vcpu, gfn, is_mmio); } =20 +static u64 vt_get_l2_tsc_offset(struct kvm_vcpu *vcpu) +{ + /* TDX doesn't support L2 guest at the moment. */ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return 0; + + return vmx_get_l2_tsc_offset(vcpu); +} + +static u64 vt_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu) +{ + /* TDX doesn't support L2 guest at the moment. */ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return 0; + + return vmx_get_l2_tsc_multiplier(vcpu); +} + +static void vt_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset) +{ + /* In TDX, tsc offset can't be changed. */ + if (is_td_vcpu(vcpu)) + return; + + vmx_write_tsc_offset(vcpu, offset); +} + +static void vt_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 multiplier) +{ + /* In TDX, tsc multiplier can't be changed. */ + if (is_td_vcpu(vcpu)) + return; + + vmx_write_tsc_multiplier(vcpu, multiplier); +} + static void vt_update_cpu_dirty_logging(struct kvm_vcpu *vcpu) { if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) @@ -983,10 +1019,10 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .has_wbinvd_exit =3D cpu_has_vmx_wbinvd_exit, =20 - .get_l2_tsc_offset =3D vmx_get_l2_tsc_offset, - .get_l2_tsc_multiplier =3D vmx_get_l2_tsc_multiplier, - .write_tsc_offset =3D vmx_write_tsc_offset, - .write_tsc_multiplier =3D vmx_write_tsc_multiplier, + .get_l2_tsc_offset =3D vt_get_l2_tsc_offset, + .get_l2_tsc_multiplier =3D vt_get_l2_tsc_multiplier, + .write_tsc_offset =3D vt_write_tsc_offset, + .write_tsc_multiplier =3D vt_write_tsc_multiplier, =20 .load_mmu_pgd =3D vt_load_mmu_pgd, =20 --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95230EB64DD for ; Tue, 25 Jul 2023 22:26:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232938AbjGYW0F (ORCPT ); Tue, 25 Jul 2023 18:26:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233263AbjGYWXm (ORCPT ); Tue, 25 Jul 2023 18:23:42 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C648E47; Tue, 25 Jul 2023 15:19:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323549; x=1721859549; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SygMEPperMXke5HjXl3/l5VxH7fusAvDX6P2uJF8SEo=; b=DPWPZyU5xKH64/kfl300+Ka+JOwEM0hKtbZ0sej6zbNVWlVDkio6yvby CMe1aLCrMCeihnpCRgKEIbfm2KU8IPVlCVm9cMGvyLJP43Bio4fylv216 e+s54QcRXt3z5VfrvENzVfuTgrLx8V/N2q3qfKd+wWJhfmaDCBw8i6Laq 73Yv8HOPXx1FAOdAy7HOnu8PB7E7Mj16TmbhwweqDgfUVGEDGG1xlQ1kE iZ1R3z9UI4IKaWtSGgUuJvbdgCr1OLh3ic72QkNn6VZw+syzLa/FfbHvU DzCpLAKpucL1wme/HH2Wdz/gUPNGlmgd0/3ti1LvGusQKx8kdjQU7VPa4 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882775" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882775" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001960" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001960" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:09 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 108/115] KVM: TDX: Ignore setting up mce Date: Tue, 25 Jul 2023 15:14:59 -0700 Message-Id: <8816f291483bf3d1ff96cd52d7fc389c49948ec4.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because vmx_set_mce function is VMX specific and it cannot be used for TDX. Add vt stub to ignore setting up mce for TDX. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index ad74900bbc56..d235268b2a76 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -892,6 +892,14 @@ static void vt_cancel_hv_timer(struct kvm_vcpu *vcpu) } #endif =20 +static void vt_setup_mce(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_setup_mce(vcpu); +} + static int vt_mem_enc_ioctl(struct kvm *kvm, void __user *argp) { if (!is_td(kvm)) @@ -1046,7 +1054,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .cancel_hv_timer =3D vt_cancel_hv_timer, #endif =20 - .setup_mce =3D vmx_setup_mce, + .setup_mce =3D vt_setup_mce, =20 #ifdef CONFIG_KVM_SMM .smi_allowed =3D vt_smi_allowed, --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C79E7EB64DD for ; Tue, 25 Jul 2023 22:26:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233034AbjGYW0J (ORCPT ); Tue, 25 Jul 2023 18:26:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233280AbjGYWXn (ORCPT ); Tue, 25 Jul 2023 18:23:43 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3843E63; Tue, 25 Jul 2023 15:19:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323550; x=1721859550; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F1na9iHUMKw4mKC0owR/NCG+6golikTlvHP4q5QoXuk=; b=NPD3OHvpVHpTaTT3xFB8OrKFgJjM6od9MNkoWxAGb4AIvjztIvPnrvqR MpXDPRfxbyml9mlnxvDHrwlqDb4a653SEjgnKArhQw2HBBNiLtvV79qk9 tCm5kokrYDPTo2cknI31OJgAKO4PyJ1bOw/uVjFyLbNW4K61v4RlPtOYU bQSODe30trSXa3Tv7cmNS9N3SApG5AOkFyCX7cBBx3c8/62SllaOz9t3x A3OmRy5Ja+tqLSgRgmDgVksUqBN46GwzDEyjBxVcrmAgotmuP9WVMZXCi AlTM4IZdih6zMBuEbyXuFkVHh52R4CKBlS+rRZHR+URWE0xZIP471i0yU Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882783" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882783" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001963" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001963" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:10 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 109/115] KVM: TDX: Add a method to ignore for TDX to ignore hypercall patch Date: Tue, 25 Jul 2023 15:15:00 -0700 Message-Id: <943309c5d446e59e056e29809a9130e1f2687016.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Because guest TD memory is protected, VMM patching guest binary for hypercall instruction isn't possible. Add a method to ignore hypercall patching with a warning. Note: guest TD kernel needs to be modified to use TDG.VP.VMCALL for hypercall. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index d235268b2a76..563b11679d00 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -724,6 +724,19 @@ static u32 vt_get_interrupt_shadow(struct kvm_vcpu *vc= pu) return vmx_get_interrupt_shadow(vcpu); } =20 +static void vt_patch_hypercall(struct kvm_vcpu *vcpu, + unsigned char *hypercall) +{ + /* + * Because guest memory is protected, guest can't be patched. TD kernel + * is modified to use TDG.VP.VMCAL for hypercall. + */ + if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) + return; + + vmx_patch_hypercall(vcpu, hypercall); +} + static void vt_inject_irq(struct kvm_vcpu *vcpu, bool reinjected) { if (is_td_vcpu(vcpu)) @@ -991,7 +1004,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .update_emulated_instruction =3D vmx_update_emulated_instruction, .set_interrupt_shadow =3D vt_set_interrupt_shadow, .get_interrupt_shadow =3D vt_get_interrupt_shadow, - .patch_hypercall =3D vmx_patch_hypercall, + .patch_hypercall =3D vt_patch_hypercall, .inject_irq =3D vt_inject_irq, .inject_nmi =3D vt_inject_nmi, .inject_exception =3D vt_inject_exception, --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E6FCEB64DD for ; Tue, 25 Jul 2023 22:26:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232853AbjGYW0N (ORCPT ); Tue, 25 Jul 2023 18:26:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233297AbjGYWXo (ORCPT ); Tue, 25 Jul 2023 18:23:44 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F06BD5B82; Tue, 25 Jul 2023 15:19:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323552; x=1721859552; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YO46SYnqcP+9S78JVxrKn3UlX9WKoCDB5qDkbdh1FC4=; b=N6ISkotFyO+yFYTmB7BC68BZCQYYTjgVmWdI5vAPK647AUSBxwldnzWH OYinZfCtACAqhUeI8binWlE5YnUg70RkNFS5KYWY3s//UB+Nh52r/Zq5A TNp1JpWAHhWFfQ8dqGdxVfpHbjzusczb/5z5VFNZxI/S1mT6hDJ9C2qXD fbcXLrSfNhC1IYOdY2R9tRENRr71Np3aPCaSKoNDqN18Mka3aJXcIynFS 2wzeulliwQPZru+V/4oNYPSGnvq4qBFfMFvd426bVvUeCbUFtbEppB+Kt Lp5/yAGLHt5bAKybBozq856XUHepsexhNe63DLdur8kVWZa4jmrRLNuGf Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882787" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882787" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001966" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001966" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:10 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 110/115] KVM: TDX: Add methods to ignore virtual apic related operation Date: Tue, 25 Jul 2023 15:15:01 -0700 Message-Id: <94560142e7d3c309dd2af42e9f2faec95a40188e.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata TDX protects TDX guest APIC state from VMM. Implement access methods of TDX guest vAPIC state to ignore them or return zero. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 61 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.c | 6 ++++ arch/x86/kvm/vmx/x86_ops.h | 3 ++ 3 files changed, 64 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 563b11679d00..e148d871b0a6 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -363,6 +363,14 @@ static bool vt_apic_init_signal_blocked(struct kvm_vcp= u *vcpu) return vmx_apic_init_signal_blocked(vcpu); } =20 +static void vt_set_virtual_apic_mode(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return tdx_set_virtual_apic_mode(vcpu); + + return vmx_set_virtual_apic_mode(vcpu); +} + static void vt_apicv_post_state_restore(struct kvm_vcpu *vcpu) { struct pi_desc *pi =3D vcpu_to_pi_desc(vcpu); @@ -371,6 +379,31 @@ static void vt_apicv_post_state_restore(struct kvm_vcp= u *vcpu) memset(pi->pir, 0, sizeof(pi->pir)); } =20 +static void vt_hwapic_irr_update(struct kvm_vcpu *vcpu, int max_irr) +{ + if (is_td_vcpu(vcpu)) + return; + + return vmx_hwapic_irr_update(vcpu, max_irr); +} + +static void vt_hwapic_isr_update(int max_isr) +{ + if (is_td_vcpu(kvm_get_running_vcpu())) + return; + + return vmx_hwapic_isr_update(max_isr); +} + +static bool vt_guest_apic_has_interrupt(struct kvm_vcpu *vcpu) +{ + /* TDX doesn't support L2 at the moment. */ + if (WARN_ON_ONCE(is_td_vcpu(vcpu))) + return false; + + return vmx_guest_apic_has_interrupt(vcpu); +} + static int vt_sync_pir_to_irr(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) @@ -808,6 +841,22 @@ static void vt_update_cr8_intercept(struct kvm_vcpu *v= cpu, int tpr, int irr) vmx_update_cr8_intercept(vcpu, tpr, irr); } =20 +static void vt_set_apic_access_page_addr(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return; + + vmx_set_apic_access_page_addr(vcpu); +} + +static void vt_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu) +{ + if (WARN_ON_ONCE(is_td_vcpu(vcpu))) + return; + + vmx_refresh_apicv_exec_ctrl(vcpu); +} + static void vt_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitma= p) { if (is_td_vcpu(vcpu)) @@ -1016,15 +1065,15 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { .enable_nmi_window =3D vt_enable_nmi_window, .enable_irq_window =3D vt_enable_irq_window, .update_cr8_intercept =3D vt_update_cr8_intercept, - .set_virtual_apic_mode =3D vmx_set_virtual_apic_mode, - .set_apic_access_page_addr =3D vmx_set_apic_access_page_addr, - .refresh_apicv_exec_ctrl =3D vmx_refresh_apicv_exec_ctrl, + .set_virtual_apic_mode =3D vt_set_virtual_apic_mode, + .set_apic_access_page_addr =3D vt_set_apic_access_page_addr, + .refresh_apicv_exec_ctrl =3D vt_refresh_apicv_exec_ctrl, .load_eoi_exitmap =3D vt_load_eoi_exitmap, .apicv_post_state_restore =3D vt_apicv_post_state_restore, .required_apicv_inhibits =3D VMX_REQUIRED_APICV_INHIBITS, - .hwapic_irr_update =3D vmx_hwapic_irr_update, - .hwapic_isr_update =3D vmx_hwapic_isr_update, - .guest_apic_has_interrupt =3D vmx_guest_apic_has_interrupt, + .hwapic_irr_update =3D vt_hwapic_irr_update, + .hwapic_isr_update =3D vt_hwapic_isr_update, + .guest_apic_has_interrupt =3D vt_guest_apic_has_interrupt, .sync_pir_to_irr =3D vt_sync_pir_to_irr, .deliver_interrupt =3D vt_deliver_interrupt, .dy_apicv_has_pending_interrupt =3D pi_has_pending_interrupt, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index c9151ea25793..7eeddc15d14f 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1935,6 +1935,12 @@ void tdx_enable_smi_window(struct kvm_vcpu *vcpu) } #endif =20 +void tdx_set_virtual_apic_mode(struct kvm_vcpu *vcpu) +{ + /* Only x2APIC mode is supported for TD. */ + WARN_ON_ONCE(kvm_get_apic_mode(vcpu) !=3D LAPIC_MODE_X2APIC); +} + int tdx_get_cpl(struct kvm_vcpu *vcpu) { return 0; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 0acbf5d34bff..07eb0e7a5696 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -169,6 +169,7 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reas= on, bool tdx_has_emulated_msr(u32 index, bool write); int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); +void tdx_set_virtual_apic_mode(struct kvm_vcpu *vcpu); =20 int tdx_get_cpl(struct kvm_vcpu *vcpu); void tdx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg); @@ -221,6 +222,8 @@ static inline bool tdx_has_emulated_msr(u32 index, bool= write) { return false; } static inline int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)= { return 1; } static inline int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)= { return 1; } =20 +static inline void tdx_set_virtual_apic_mode(struct kvm_vcpu *vcpu) {} + static inline int tdx_get_cpl(struct kvm_vcpu *vcpu) { return 0; } static inline void tdx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) = {} static inline unsigned long tdx_get_rflags(struct kvm_vcpu *vcpu) { return= 0; } --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC87AC0015E for ; Tue, 25 Jul 2023 22:26:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233143AbjGYW0U (ORCPT ); Tue, 25 Jul 2023 18:26:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233328AbjGYWXr (ORCPT ); Tue, 25 Jul 2023 18:23:47 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 550325B91; Tue, 25 Jul 2023 15:19:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323557; x=1721859557; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=B8rHbW51qfJ+KxKr7O+BRLSEIy30uwyo0BYKLpNksFg=; b=UCrynRQ9uqY4LV5/l6de1rjrGMpR0gTeiVWW8lqLcsVsTnmlGf+s4/vA swJGNe2p5IBCbd5e/a4+tbAMLu+enEHodvZGjtc0Ui2T8b008lEbIuBwz KbGnwY7LiviPqjrxt11wAS6JNOz4w67nI+UKRb2Pk/TraUum/PK0V8jAu fYa6p0GseZayj39t+PR3trWWQqyoPPkflAL3fIPUjt84E8UAVpeiGc8cN pUdf1RzGmAyYD2CDOubRUBvHCBja68cQmx8aY2c9fWLbmBvqjgn+iNOYt Ssr1IBzdOlwT233i912oBegefRyl1QqVmKTUScc1xH2T6DfwdtRMMn93n A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882791" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882791" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001971" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001971" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:11 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 111/115] Documentation/virt/kvm: Document on Trust Domain Extensions(TDX) Date: Tue, 25 Jul 2023 15:15:02 -0700 Message-Id: <50cc7dad735ecfd67d019002a1aa851f2df06038.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Add documentation to Intel Trusted Domain Extensions(TDX) support. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/api.rst | 9 +- Documentation/virt/kvm/x86/index.rst | 1 + Documentation/virt/kvm/x86/intel-tdx.rst | 362 +++++++++++++++++++++++ 3 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 Documentation/virt/kvm/x86/intel-tdx.rst diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 9f7b95327c2a..2df931611c11 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -1373,6 +1373,9 @@ the memory region are automatically reflected into th= e guest. For example, an mmap() that affects the region will be made visible immediately. Another example is madvise(MADV_DROP). =20 +For TDX guest, deleting/moving memory region loses guest memory contents. +Read only region isn't supported. Only as-id 0 is supported. + Note: On arm64, a write generated by the page-table walker (to update the Access and Dirty flags, for example) never results in a KVM_EXIT_MMIO exit when the slot has the KVM_MEM_READONLY flag. This @@ -4690,7 +4693,7 @@ H_GET_CPU_CHARACTERISTICS hypercall. =20 :Capability: basic :Architectures: x86 -:Type: vm +:Type: vm ioctl, vcpu ioctl :Parameters: an opaque platform specific structure (in/out) :Returns: 0 on success; -1 on error =20 @@ -4702,6 +4705,10 @@ Currently, this ioctl is used for issuing Secure Enc= rypted Virtualization (SEV) commands on AMD Processors. The SEV commands are defined in Documentation/virt/kvm/x86/amd-memory-encryption.rst. =20 +Currently, this ioctl is used for issuing Trusted Domain Extensions +(TDX) commands on Intel Processors. The TDX commands are defined in +Documentation/virt/kvm/intel-tdx.rst. + 4.111 KVM_MEMORY_ENCRYPT_REG_REGION ----------------------------------- =20 diff --git a/Documentation/virt/kvm/x86/index.rst b/Documentation/virt/kvm/= x86/index.rst index 9ece6b8dc817..851e99174762 100644 --- a/Documentation/virt/kvm/x86/index.rst +++ b/Documentation/virt/kvm/x86/index.rst @@ -11,6 +11,7 @@ KVM for x86 systems cpuid errata hypercalls + intel-tdx mmu msr nested-vmx diff --git a/Documentation/virt/kvm/x86/intel-tdx.rst b/Documentation/virt/= kvm/x86/intel-tdx.rst new file mode 100644 index 000000000000..a1b10e99c1ff --- /dev/null +++ b/Documentation/virt/kvm/x86/intel-tdx.rst @@ -0,0 +1,362 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Intel Trust Domain Extensions (TDX) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Overview +=3D=3D=3D=3D=3D=3D=3D=3D +TDX stands for Trust Domain Extensions which isolates VMs from +the virtual-machine manager (VMM)/hypervisor and any other software on +the platform. For details, see the specifications [1]_, whitepaper [2]_, +architectural extensions specification [3]_, module documentation [4]_, +loader interface specification [5]_, guest-hypervisor communication +interface [6]_, virtual firmware design guide [7]_, and other resources +([8]_, [9]_, [10]_, [11]_, and [12]_). + + +API description +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +KVM_MEMORY_ENCRYPT_OP +--------------------- +:Type: vm ioctl, vcpu ioctl + +For TDX operations, KVM_MEMORY_ENCRYPT_OP is re-purposed to be generic +ioctl with TDX specific sub ioctl command. + +:: + + /* Trust Domain eXtension sub-ioctl() commands. */ + enum kvm_tdx_cmd_id { + KVM_TDX_CAPABILITIES =3D 0, + KVM_TDX_INIT_VM, + KVM_TDX_INIT_VCPU, + KVM_TDX_INIT_MEM_REGION, + KVM_TDX_FINALIZE_VM, + + KVM_TDX_CMD_NR_MAX, + }; + + struct kvm_tdx_cmd { + /* enum kvm_tdx_cmd_id */ + __u32 id; + /* flags for sub-commend. If sub-command doesn't use this, set zer= o. */ + __u32 flags; + /* + * data for each sub-command. An immediate or a pointer to the act= ual + * data in process virtual address. If sub-command doesn't use it, + * set zero. + */ + __u64 data; + /* + * Auxiliary error code. The sub-command may return TDX SEAMCALL + * status code in addition to -Exxx. + * Defined for consistency with struct kvm_sev_cmd. + */ + __u64 error; + /* Reserved: Defined for consistency with struct kvm_sev_cmd. */ + __u64 unused; + }; + +KVM_TDX_CAPABILITIES +-------------------- +:Type: vm ioctl + +Subset of TDSYSINFO_STRCUCT retrieved by TDH.SYS.INFO TDX SEAM call will be +returned. Which describes about Intel TDX module. + +- id: KVM_TDX_CAPABILITIES +- flags: must be 0 +- data: pointer to struct kvm_tdx_capabilities +- error: must be 0 +- unused: must be 0 + +:: + + struct kvm_tdx_cpuid_config { + __u32 leaf; + __u32 sub_leaf; + __u32 eax; + __u32 ebx; + __u32 ecx; + __u32 edx; + }; + + struct kvm_tdx_capabilities { + __u64 attrs_fixed0; + __u64 attrs_fixed1; + __u64 xfam_fixed0; + __u64 xfam_fixed1; + #define TDX_CAP_GPAW_48 (1 << 0) + #define TDX_CAP_GPAW_52 (1 << 1) + __u32 supported_gpaw; + __u32 padding; + __u64 reserved[251]; + + __u32 nr_cpuid_configs; + struct kvm_tdx_cpuid_config cpuid_configs[]; + }; + + +KVM_TDX_INIT_VM +--------------- +:Type: vm ioctl + +Does additional VM initialization specific to TDX which corresponds to +TDH.MNG.INIT TDX SEAM call. + +- id: KVM_TDX_INIT_VM +- flags: must be 0 +- data: pointer to struct kvm_tdx_init_vm +- error: must be 0 +- unused: must be 0 + +:: + + struct kvm_tdx_init_vm { + __u64 attributes; + __u64 mrconfigid[6]; /* sha384 digest */ + __u64 mrowner[6]; /* sha384 digest */ + __u64 mrownerconfig[6]; /* sha348 digest */ + __u64 reserved[1004]; /* must be zero for future extensi= bility */ + + struct kvm_cpuid2 cpuid; + }; + + +KVM_TDX_INIT_VCPU +----------------- +:Type: vcpu ioctl + +Does additional VCPU initialization specific to TDX which corresponds to +TDH.VP.INIT TDX SEAM call. + +- id: KVM_TDX_INIT_VCPU +- flags: must be 0 +- data: initial value of the guest TD VCPU RCX +- error: must be 0 +- unused: must be 0 + +KVM_TDX_INIT_MEM_REGION +----------------------- +:Type: vm ioctl + +Encrypt a memory continuous region which corresponding to TDH.MEM.PAGE.ADD +TDX SEAM call. +If KVM_TDX_MEASURE_MEMORY_REGION flag is specified, it also extends measur= ement +which corresponds to TDH.MR.EXTEND TDX SEAM call. + +- id: KVM_TDX_INIT_VCPU +- flags: flags + currently only KVM_TDX_MEASURE_MEMORY_REGION is defined +- data: pointer to struct kvm_tdx_init_mem_region +- error: must be 0 +- unused: must be 0 + +:: + + #define KVM_TDX_MEASURE_MEMORY_REGION (1UL << 0) + + struct kvm_tdx_init_mem_region { + __u64 source_addr; + __u64 gpa; + __u64 nr_pages; + }; + + +KVM_TDX_FINALIZE_VM +------------------- +:Type: vm ioctl + +Complete measurement of the initial TD contents and mark it ready to run +which corresponds to TDH.MR.FINALIZE + +- id: KVM_TDX_FINALIZE_VM +- flags: must be 0 +- data: must be 0 +- error: must be 0 +- unused: must be 0 + +KVM TDX creation flow +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +In addition to KVM normal flow, new TDX ioctls need to be called. The con= trol flow +looks like as follows. + +#. system wide capability check + + * KVM_CAP_VM_TYPES: check if VM type is supported and if KVM_X86_TDX_VM + is supported. + +#. creating VM + + * KVM_CREATE_VM + * KVM_TDX_CAPABILITIES: query if TDX is supported on the platform. + * KVM_ENABLE_CAP_VM(KVM_CAP_MAX_VCPUS): set max_vcpus. KVM_MAX_VCPUS by + default. KVM_MAX_VCPUS is not a part of ABI, but kernel internal con= stant + that is subject to change. Because max vcpus is a part of attestatio= n, max + vcpus should be explicitly set. + * KVM_SET_TSC_KHZ for vm. optional + * KVM_TDX_INIT_VM: pass TDX specific VM parameters. + +#. creating VCPU + + * KVM_CREATE_VCPU + * KVM_TDX_INIT_VCPU: pass TDX specific VCPU parameters. + * KVM_SET_CPUID2: Enable CPUID[0x1].ECX.X2APIC(bit 21)=3D1 so that the = following + setting of MSR_IA32_APIC_BASE success. Without this, + KVM_SET_MSRS(MSR_IA32_APIC_BASE) fails. + * KVM_SET_MSRS: Set the initial reset value of MSR_IA32_APIC_BASE to + APIC_DEFAULT_ADDRESS(0xfee00000) | XAPIC_ENABLE(bit 10) | + X2APIC_ENABLE(bit 11) [| MSR_IA32_APICBASE_BSP(bit 8) optional] + +#. initializing guest memory + + * allocate guest memory and initialize page same to normal KVM case + In TDX case, parse and load TDVF into guest memory in addition. + * KVM_TDX_INIT_MEM_REGION to add and measure guest pages. + If the pages has contents above, those pages need to be added. + Otherwise the contents will be lost and guest sees zero pages. + * KVM_TDX_FINALIAZE_VM: Finalize VM and measurement + This must be after KVM_TDX_INIT_MEM_REGION. + +#. run vcpu + +Design discussion +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Coexistence of normal(VMX) VM and TD VM +--------------------------------------- +It's required to allow both legacy(normal VMX) VMs and new TD VMs to +coexist. Otherwise the benefits of VM flexibility would be eliminated. +The main issue for it is that the logic of kvm_x86_ops callbacks for +TDX is different from VMX. On the other hand, the variable, +kvm_x86_ops, is global single variable. Not per-VM, not per-vcpu. + +Several points to be considered: + + * No or minimal overhead when TDX is disabled(CONFIG_INTEL_TDX_HOST=3Dn). + * Avoid overhead of indirect call via function pointers. + * Contain the changes under arch/x86/kvm/vmx directory and share logic + with VMX for maintenance. + Even though the ways to operation on VM (VMX instruction vs TDX + SEAM call) are different, the basic idea remains the same. So, many + logic can be shared. + * Future maintenance + The huge change of kvm_x86_ops in (near) future isn't expected. + a centralized file is acceptable. + +- Wrapping kvm x86_ops: The current choice + + Introduce dedicated file for arch/x86/kvm/vmx/main.c (the name, + main.c, is just chosen to show main entry points for callbacks.) and + wrapper functions around all the callbacks with + "if (is-tdx) tdx-callback() else vmx-callback()". + + Pros: + + - No major change in common x86 KVM code. The change is (mostly) + contained under arch/x86/kvm/vmx/. + - When TDX is disabled(CONFIG_INTEL_TDX_HOST=3Dn), the overhead is + optimized out. + - Micro optimization by avoiding function pointer. + + Cons: + + - Many boiler plates in arch/x86/kvm/vmx/main.c. + +KVM MMU Changes +--------------- +KVM MMU needs to be enhanced to handle Secure/Shared-EPT. The +high-level execution flow is mostly same to normal EPT case. +EPT violation/misconfiguration -> invoke TDP fault handler -> +resolve TDP fault -> resume execution. (or emulate MMIO) +The difference is, that S-EPT is operated(read/write) via TDX SEAM +call which is expensive instead of direct read/write EPT entry. +One bit of GPA (51 or 47 bit) is repurposed so that it means shared +with host(if set to 1) or private to TD(if cleared to 0). + +- The current implementation + + * Reuse the existing MMU code with minimal update. Because the + execution flow is mostly same. But additional operation, TDX call + for S-EPT, is needed. So add hooks for it to kvm_x86_ops. + * For performance, minimize TDX SEAM call to operate on S-EPT. When + getting corresponding S-EPT pages/entry from faulting GPA, don't + use TDX SEAM call to read S-EPT entry. Instead create shadow copy + in host memory. + Repurpose the existing kvm_mmu_page as shadow copy of S-EPT and + associate S-EPT to it. + * Treats share bit as attributes. mask/unmask the bit where + necessary to keep the existing traversing code works. + Introduce kvm.arch.gfn_shared_mask and use "if (gfn_share_mask)" + for special case. + + * 0 : for non-TDX case + * 51 or 47 bit set for TDX case. + + Pros: + + - Large code reuse with minimal new hooks. + - Execution path is same. + + Cons: + + - Complicates the existing code. + - Repurpose kvm_mmu_page as shadow of Secure-EPT can be confusing. + +New KVM API, ioctl (sub)command, to manage TD VMs +------------------------------------------------- +Additional KVM APIs are needed to control TD VMs. The operations on TD +VMs are specific to TDX. + +- Piggyback and repurpose KVM_MEMORY_ENCRYPT_OP + + Although operations for TD VMs aren't necessarily related to memory + encryption, define sub operations of KVM_MEMORY_ENCRYPT_OP for TDX speci= fic + ioctls. + + Pros: + + - No major change in common x86 KVM code. + - Follows the SEV case. + + Cons: + + - The sub operations of KVM_MEMORY_ENCRYPT_OP aren't necessarily memory + encryption, but operations on TD VMs. + +References +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. [1] TDX specification + https://software.intel.com/content/www/us/en/develop/articles/intel-tru= st-domain-extensions.html +.. [2] Intel Trust Domain Extensions (Intel TDX) + https://software.intel.com/content/dam/develop/external/us/en/documents= /tdx-whitepaper-final9-17.pdf +.. [3] Intel CPU Architectural Extensions Specification + https://software.intel.com/content/dam/develop/external/us/en/documents= /intel-tdx-cpu-architectural-specification.pdf +.. [4] Intel TDX Module 1.0 EAS + https://software.intel.com/content/dam/develop/external/us/en/documents= /intel-tdx-module-1eas.pdf +.. [5] Intel TDX Loader Interface Specification + https://software.intel.com/content/dam/develop/external/us/en/documents= /intel-tdx-seamldr-interface-specification.pdf +.. [6] Intel TDX Guest-Hypervisor Communication Interface + https://software.intel.com/content/dam/develop/external/us/en/documents= /intel-tdx-guest-hypervisor-communication-interface.pdf +.. [7] Intel TDX Virtual Firmware Design Guide + https://software.intel.com/content/dam/develop/external/us/en/documents= /tdx-virtual-firmware-design-guide-rev-1. +.. [8] intel public github + + * kvm TDX branch: https://github.com/intel/tdx/tree/kvm + * TDX guest branch: https://github.com/intel/tdx/tree/guest + +.. [9] tdvf + https://github.com/tianocore/edk2-staging/tree/TDVF +.. [10] KVM forum 2020: Intel Virtualization Technology Extensions to + Enable Hardware Isolated VMs + https://osseu2020.sched.com/event/eDzm/intel-virtualization-technolog= y-extensions-to-enable-hardware-isolated-vms-sean-christopherson-intel +.. [11] Linux Security Summit EU 2020: + Architectural Extensions for Hardware Virtual Machine Isolation + to Advance Confidential Computing in Public Clouds - Ravi Sahita + & Jun Nakajima, Intel Corporation + https://osseu2020.sched.com/event/eDOx/architectural-extensions-for-h= ardware-virtual-machine-isolation-to-advance-confidential-computing-in-publ= ic-clouds-ravi-sahita-jun-nakajima-intel-corporation +.. [12] [RFCv2,00/16] KVM protected memory extension + https://lore.kernel.org/all/20201020061859.18385-1-kirill.shutemov@li= nux.intel.com/ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACEBDEB64DD for ; Tue, 25 Jul 2023 22:26:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233131AbjGYW0R (ORCPT ); Tue, 25 Jul 2023 18:26:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233338AbjGYWXs (ORCPT ); Tue, 25 Jul 2023 18:23:48 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13AD35B98; Tue, 25 Jul 2023 15:19:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323559; x=1721859559; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gDn2qlctsRhiSfk4B8miWiyBiv/7pxkEkr9nT08qBkA=; b=VMGFzLP9e0eojrafTnWibvpuRZTXPhgZtd/BCw8SP1UKncmMSXPMolK2 GxWnyqA1LD280x51Ww3eo1/gSNU4Urt944Ol+dJv2LnjVbxTFtXW9GZBx RbLMnfwhBHcBCvBD/0060MGkUQZJCD0g8AIybc54Af4Zc9MMpNuwcjjXk J6o6G9Wgwmgd051mXJ3awdYeclg+i8Ogc5lhC5vs1/3fcqEQJDoXQKI22 XkX+n79rrOFaFMT1Wzh/OLCd48Jx8/k6uXUIcJPolF0R0rIO0iDZF8iTX BYYeh/0oitTd5M9jGy/xgoW4vG7OaTFqO07vruUNj7+NLQIxmKGDJUzSR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882795" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882795" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001974" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001974" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:11 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 112/115] KVM: x86: design documentation on TDX support of x86 KVM TDP MMU Date: Tue, 25 Jul 2023 15:15:03 -0700 Message-Id: <45841e343e180b72dade7114a7f18030fcae784c.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Add a high level design document on TDX changes to TDP MMU. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/x86/index.rst | 1 + Documentation/virt/kvm/x86/tdx-tdp-mmu.rst | 443 +++++++++++++++++++++ 2 files changed, 444 insertions(+) create mode 100644 Documentation/virt/kvm/x86/tdx-tdp-mmu.rst diff --git a/Documentation/virt/kvm/x86/index.rst b/Documentation/virt/kvm/= x86/index.rst index 851e99174762..63a78bd41b16 100644 --- a/Documentation/virt/kvm/x86/index.rst +++ b/Documentation/virt/kvm/x86/index.rst @@ -16,4 +16,5 @@ KVM for x86 systems msr nested-vmx running-nested-guests + tdx-tdp-mmu timekeeping diff --git a/Documentation/virt/kvm/x86/tdx-tdp-mmu.rst b/Documentation/vir= t/kvm/x86/tdx-tdp-mmu.rst new file mode 100644 index 000000000000..49d103720272 --- /dev/null +++ b/Documentation/virt/kvm/x86/tdx-tdp-mmu.rst @@ -0,0 +1,443 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Design of TDP MMU for TDX support +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D +This document describes a (high level) design for TDX support of KVM TDP M= MU of +x86 KVM. + +In this document, we use "TD" or "guest TD" to differentiate it from the c= urrent +"VM" (Virtual Machine), which is supported by KVM today. + + +Background of TDX +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +TD private memory is designed to hold TD private content, encrypted by the= CPU +using the TD ephemeral key. An encryption engine holds a table of encrypt= ion +keys, and an encryption key is selected for each memory transaction based = on a +Host Key Identifier (HKID). By design, the host VMM does not have access = to the +encryption keys. + +In the first generation of MKTME, HKID is "stolen" from the physical addre= ss by +allocating a configurable number of bits from the top of the physical addr= ess. +The HKID space is partitioned into shared HKIDs for legacy MKTME accesses = and +private HKIDs for SEAM-mode-only accesses. We use 0 for the shared HKID o= n the +host so that MKTME can be opaque or bypassed on the host. + +During TDX non-root operation (i.e. guest TD), memory accesses can be qual= ified +as either shared or private, based on the value of a new SHARED bit in the= Guest +Physical Address (GPA). The CPU translates shared GPAs using the usual VM= X EPT +(Extended Page Table) or "Shared EPT" (in this document), which resides in= the +host VMM memory. The Shared EPT is directly managed by the host VMM - the= same +as with the current VMX. Since guest TDs usually require I/O, and the data +exchange needs to be done via shared memory, thus KVM needs to use the cur= rent +EPT functionality even for TDs. + +The CPU translates private GPAs using a separate Secure EPT. The Secure E= PT +pages are encrypted and integrity-protected with the TD's ephemeral privat= e key. +Secure EPT can be managed _indirectly_ by the host VMM, using the TDX inte= rface +functions (SEAMCALLs), and thus conceptually Secure EPT is a subset of EPT +because not all functionalities are available. + +Since the execution of such interface functions takes much longer time than +accessing memory directly, in KVM we use the existing TDP code to mirror t= he +Secure EPT for the TD. And we think there are at least two options today in +terms of the timing for executing such SEAMCALLs: + +1. synchronous, i.e. while walking the TDP page tables, or +2. post-walk, i.e. record what needs to be done to the real Secure EPT dur= ing + the walk, and execute SEAMCALLs later. + +The option 1 seems to be more intuitive and simpler, but the Secure EPT +concurrency rules are different from the ones of the TDP or EPT. For examp= le, +MEM.SEPT.RD acquire shared access to the whole Secure EPT tree of the targ= et + +Secure EPT(SEPT) operations +--------------------------- +Secure EPT is an Extended Page Table for GPA-to-HPA translation of TD priv= ate +HPA. A Secure EPT is designed to be encrypted with the TD's ephemeral pri= vate +key. SEPT pages are allocated by the host VMM via Intel TDX functions, but= their +content is intended to be hidden and is not architectural. + +Unlike the conventional EPT, the CPU can't directly read/write its entry. +Instead, TDX SEAMCALL API is used. Several SEAMCALLs correspond to operat= ion on +the EPT entry. + +* TDH.MEM.SEPT.ADD(): + + Add a secure EPT page from the secure EPT tree. This corresponds to upd= ating + the non-leaf EPT entry with present bit set + +* TDH.MEM.SEPT.REMOVE(): + + Remove the secure page from the secure EPT tree. There is no correspond= ing + to the EPT operation. + +* TDH.MEM.SEPT.RD(): + + Read the secure EPT entry. This corresponds to reading the EPT entry as + memory. Please note that this is much slower than direct memory reading. + +* TDH.MEM.PAGE.ADD() and TDH.MEM.PAGE.AUG(): + + Add a private page to the secure EPT tree. This corresponds to updating= the + leaf EPT entry with present bit set. + +* THD.MEM.PAGE.REMOVE(): + + Remove a private page from the secure EPT tree. There is no correspondi= ng + to the EPT operation. + +* TDH.MEM.RANGE.BLOCK(): + + This (mostly) corresponds to clearing the present bit of the leaf EPT en= try. + Note that the private page is still linked in the secure EPT. To remove= it + from the secure EPT, TDH.MEM.SEPT.REMOVE() and TDH.MEM.PAGE.REMOVE() nee= ds to + be called. + +* TDH.MEM.TRACK(): + + Increment the TLB epoch counter. This (mostly) corresponds to EPT TLB fl= ush. + Note that the private page is still linked in the secure EPT. To remove= it + from the secure EPT, tdh_mem_page_remove() needs to be called. + + +Adding private page +------------------- +The procedure of populating the private page looks as follows. + +1. TDH.MEM.SEPT.ADD(512G level) +2. TDH.MEM.SEPT.ADD(1G level) +3. TDH.MEM.SEPT.ADD(2M level) +4. TDH.MEM.PAGE.AUG(4K level) + +Those operations correspond to updating the EPT entries. + +Dropping private page and TLB shootdown +--------------------------------------- +The procedure of dropping the private page looks as follows. + +1. TDH.MEM.RANGE.BLOCK(4K level) + + This mostly corresponds to clear the present bit in the EPT entry. This + prevents (or blocks) TLB entry from creating in the future. Note that = the + private page is still linked in the secure EPT tree and the existing ca= che + entry in the TLB isn't flushed. + +2. TDH.MEM.TRACK(range) and TLB shootdown + + This mostly corresponds to the EPT TLB shootdown. Because all vcpus sh= are + the same Secure EPT, all vcpus need to flush TLB. + + * TDH.MEM.TRACK(range) by one vcpu. It increments the global internal = TLB + epoch counter. + + * send IPI to remote vcpus + * Other vcpu exits to VMM from guest TD and then re-enter. TDH.VP.ENTER= (). + * TDH.VP.ENTER() checks the TLB epoch counter and If its TLB is old, fl= ush + TLB. + + Note that only single vcpu issues tdh_mem_track(). + + Note that the private page is still linked in the secure EPT tree, unli= ke the + conventional EPT. + +3. TDH.MEM.PAGE.PROMOTE, TDH.MEM.PAGEDEMOTE(), TDH.MEM.PAGE.RELOCATE(), or + TDH.MEM.PAGE.REMOVE() + + There is no corresponding operation to the conventional EPT. + + * When changing page size (e.g. 4K <-> 2M) TDH.MEM.PAGE.PROMOTE() or + TDH.MEM.PAGE.DEMOTE() is used. During those operation, the guest pag= e is + kept referenced in the Secure EPT. + + * When migrating page, TDH.MEM.PAGE.RELOCATE(). This requires both sou= rce + page and destination page. + * when destroying TD, TDH.MEM.PAGE.REMOVE() removes the private page fr= om the + secure EPT tree. In this case TLB shootdown is not needed because vc= pus + don't run any more. + +The basic idea for TDX support +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D +Because shared EPT is the same as the existing EPT, use the existing logic= for +shared EPT. On the other hand, secure EPT requires additional operations +instead of directly reading/writing of the EPT entry. + +On EPT violation, The KVM mmu walks down the EPT tree from the root, deter= mines +the EPT entry to operate, and updates the entry. If necessary, a TLB shoot= down +is done. Because it's very slow to directly walk secure EPT by TDX SEAMCA= LL, +TDH.MEM.SEPT.RD(), the mirror of secure EPT is created and maintained. Add +hooks to KVM MMU to reuse the existing code. + +EPT violation on shared GPA +--------------------------- +(1) EPT violation on shared GPA or zapping shared GPA + :: + + walk down shared EPT tree (the existing code) + | + | + V + shared EPT tree (CPU refers.) + +(2) update the EPT entry. (the existing code) + + TLB shootdown in the case of zapping. + + +EPT violation on private GPA +---------------------------- +(1) EPT violation on private GPA or zapping private GPA + :: + + walk down the mirror of secure EPT tree (mostly same as the existi= ng code) + | + | + V + mirror of secure EPT tree (KVM MMU software only. reuse of the exi= sting code) + +(2) update the (mirrored) EPT entry. (mostly same as the existing code) + +(3) call the hooks with what EPT entry is changed + :: + + | + NEW: hooks in KVM MMU + | + V + secure EPT root(CPU refers) + +(4) the TDX backend calls necessary TDX SEAMCALLs to update real secure EP= T. + +The major modification is to add hooks for the TDX backend for additional +operations and to pass down which EPT, shared EPT, or private EPT is used,= and +twist the behavior if we're operating on private EPT. + +The following depicts the relationship. +:: + + KVM | TDX module + | | | + -------------+---------- | | + | | | | + V V | | + shared GPA private GPA | V + CPU shared EPT pointer KVM private EPT pointer | CPU secure EPT poin= ter + | | | | + | | | | + V V | V + shared EPT private EPT<-------mirror----->Secure EPT + | | | | + | \--------------------+------\ | + | | | | + V | V V + shared guest page | private guest page + | + | + non-encrypted memory | encrypted memory + | + +shared EPT: CPU and KVM walk with shared GPA + Maintained by the existing code +private EPT: KVM walks with private GPA + Maintained by the twisted existing code +secure EPT: CPU walks with private GPA. + Maintained by TDX module with TDX SEAMCALLs via hooks + + +Tracking private EPT page +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +Shared EPT pages are managed by struct kvm_mmu_page. They are linked in a= list +structure. When necessary, the list is traversed to operate on. Private = EPT +pages have different characteristics. For example, private pages can't be +swapped out. When shrinking memory, we'd like to traverse only shared EPT= pages +and skip private EPT pages. Likewise, page migration isn't supported for +private pages (yet). Introduce an additional list to track shared EPT pag= es and +track private EPT pages independently. + +At the beginning of EPT violation, the fault handler knows fault GPA, thus= it +knows which EPT to operate on, private or shared. If it's private EPT, +an additional task is done. Something like "if (private) { callback a hoo= k }". +Since the fault handler has deep function calls, it's cumbersome to hold t= he +information of which EPT is operating. Options to mitigate it are + +1. Pass the information as an argument for the function call. +2. Record the information in struct kvm_mmu_page somehow. +3. Record the information in vcpu structure. + +Option 2 was chosen. Because option 1 requires modifying all the function= s. It +would affect badly to the normal case. Option 3 doesn't work well because= in +some cases, we need to walk both private and shared EPT. + +The role of the EPT page can be utilized and one bit can be curved out from +unused bits in struct kvm_mmu_page_role. When allocating the EPT page, +initialize the information. Mostly struct kvm_mmu_page is available because +we're operating on EPT pages. + + +The conversion of private GPA and shared GPA +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +A page of a given GPA can be assigned to only private GPA xor shared GPA a= t one +time. (This is the restriction by KVM implementation to avoid doubling gu= est +memory usage. Not by TDX architecture.) The GPA can't be accessed +simultaneously via both private GPA and shared GPA. On guest startup, all= the +GPAs are assigned as private. Guest converts the range of GPA to shared (= or +private) from private (or shared) by MapGPA hypercall. MapGPA hypercall t= akes +the start GPA and the size of the region. If the given start GPA is shared +(shared bit set), VMM converts the region into shared (if it's already sha= red, +nop). + +If the guest TD triggers an EPT violation on the already converted region, +i.e. EPT violation on private(or shared) GPA when page is shared(or privat= e), +the access won't be allowed. KVM_EXIT_MEMORY_FAULT is triggered. The user +space VMM will decide how to handle it. + +If the guest access private (or shared) GPA after the conversion to shared= (or +private), the following sequence will be observed + +1. MapGPA(shared GPA: shared bit set) hypercall +2. KVM cause KVM_TDX_EXIT with hypercall to the user space VMM. +3. The user space VMM converts the GPA with KVM_SET_MEMORY_ATTRIBUTES(shar= ed). +4. The user space VMM resumes vcpu execution with KVM_VCPU_RUN +5. Guest TD accesses private GPA (shared bit cleared) +6. KVM gets EPT violation on private GPA (shared bit cleared) +7. KVM finds the GPA was set to be shared in the xarray while the faulting= GPA + is private (shared bit cleared) +8. KVM_EXIT_MEMORY_FAULT. User space VMM, e.g. qemu, decide what to do. + Typically requests KVM conversion of GPA without MapGPA hypercall. +9. KVM converts GPA from shared to private with + KVM_SET_MEMORY_ATTRIBUTES(private) +10. Resume vcpu execution + +At step 9, user space VMM may think such memory access is due to race, let= vcpu +resume without conversion with the expectation that other vcpu issues MapG= PA. +Or user space VMM may think such memory access is doubtful and the guest is +trying to attack VMM. It may throttle vcpu execution as mitigation or fin= ally +kill such a guest. Or user space VMM may think it's a bug of the guest TD= , kill +the guest TD. + +This sequence is not efficient. Guest TD shouldn't access private (or sha= red) +GPA after converting GPA to shared (or private). Although KVM can handle = it, +it's sub-optimal and won't be optimized. + +The original TDP MMU and race condition +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Because vcpus share the EPT, once the EPT entry is zapped, we need to shoo= tdown +TLB. Send IPI to remote vcpus. Remote vcpus flush their down TLBs. Unti= l TLB +shootdown is done, vcpus may reference the zapped guest page. + +TDP MMU uses read lock of mmu_lock to mitigate vcpu contention. When read= lock +is obtained, it depends on the atomic update of the EPT entry. (On the ot= her +hand legacy MMU uses write lock.) When vcpu is populating/zapping the EPT= entry +with a read lock held, other vcpu may be populating or zapping the same EPT +entry at the same time. + +To avoid the race condition, the entry is frozen. It means the EPT entry = is set +to the special value, REMOVED_SPTE which clears the present bit. And then= after +TLB shootdown, update the EPT entry to the final value. + +Concurrent zapping +------------------ +1. read lock +2. freeze the EPT entry (atomically set the value to REMOVED_SPTE) + If other vcpu froze the entry, restart page fault. +3. TLB shootdown + + * send IPI to remote vcpus + * TLB flush (local and remote) + + For each entry update, TLB shootdown is needed because of the + concurrency. +4. atomically set the EPT entry to the final value +5. read unlock + +Concurrent populating +--------------------- +In the case of populating the non-present EPT entry, atomically update the= EPT +entry. + +1. read lock + +2. atomically update the EPT entry + If other vcpu frozen the entry or updated the entry, restart page fault. + +3. read unlock + +In the case of updating the present EPT entry (e.g. page migration), the +operation is split into two. Zapping the entry and populating the entry. + +1. read lock +2. zap the EPT entry. follow the concurrent zapping case. +3. populate the non-present EPT entry. +4. read unlock + +Non-concurrent batched zapping +------------------------------ +In some cases, zapping the ranges is done exclusively with a write lock he= ld. +In this case, the TLB shootdown is batched into one. + +1. write lock +2. zap the EPT entries by traversing them +3. TLB shootdown +4. write unlock + +For Secure EPT, TDX SEAMCALLs are needed in addition to updating the mirro= red +EPT entry. + +TDX concurrent zapping +---------------------- +Add a hook for TDX SEAMCALLs at the step of the TLB shootdown. + +1. read lock +2. freeze the EPT entry(set the value to REMOVED_SPTE) +3. TLB shootdown via a hook + + * TLB.MEM.RANGE.BLOCK() + * TLB.MEM.TRACK() + * send IPI to remote vcpus + +4. set the EPT entry to the final value +5. read unlock + +TDX concurrent populating +------------------------- +TDX SEAMCALLs are required in addition to operating the mirrored EPT entry= . The +frozen entry is utilized by following the zapping case to avoid the race +condition. A hook can be added. + +1. read lock +2. freeze the EPT entry +3. hook + + * TDH_MEM_SEPT_ADD() for non-leaf or TDH_MEM_PAGE_AUG() for leaf. + +4. set the EPT entry to the final value +5. read unlock + +Without freezing the entry, the following race can happen. Suppose two vc= pus +are faulting on the same GPA and the 2M and 4K level entries aren't popula= ted +yet. + +* vcpu 1: update 2M level EPT entry +* vcpu 2: update 4K level EPT entry +* vcpu 2: TDX SEAMCALL to update 4K secure EPT entry =3D> error +* vcpu 1: TDX SEAMCALL to update 2M secure EPT entry + + +TDX non-concurrent batched zapping +---------------------------------- +For simplicity, the procedure of concurrent populating is utilized. The +procedure can be optimized later. + + +Co-existing with unmapping guest private memory +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +TODO. This needs to be addressed. + + +Restrictions or future work +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +The following features aren't supported yet at the moment. + +* optimizing non-concurrent zap +* Large page +* Page migration --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCD60EB64DD for ; Tue, 25 Jul 2023 22:26:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233148AbjGYW0W (ORCPT ); Tue, 25 Jul 2023 18:26:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233380AbjGYWXw (ORCPT ); Tue, 25 Jul 2023 18:23:52 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35F1C26B6; Tue, 25 Jul 2023 15:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323569; x=1721859569; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oacKdQeMh8QuirRSr8ipRHm85rmLeDu5QPcTK7AXcqY=; b=PqQffRyldbnRumqqjYLFc4M1uBnuIfa5rDEBDQN2uQO06gmo0CMTFlqY B0V0ULRV3IjsxZ8OWtpbb3rr0m7T9Qf9bJzS3avWM6mfaaGB/9gpZ64nZ aFjUMp31P+7qHg1XPQ+TovhvERSxKcVpGN6WekmFFZsDxBlIHr38h+xXj BXGRsxbRmbag6l3AcnFdj2a6EGEejXZO0dIK601po+mgxoLQWn5SansEf YZk8nHFe2jUFcXaYYZNCw8szkHUwAr3gutMrDwQ4POu8KmroCH1Zvb9dq KwwnBfy70bA7emS69CARyekmn5s+lMB0z+qHyOedv+F+od4vHJ5YTKbFv Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882800" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882800" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001978" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001978" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:12 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 113/115] RFC: KVM: x86: Add x86 callback to check cpuid Date: Tue, 25 Jul 2023 15:15:04 -0700 Message-Id: <8787693c245ceeeada515fcca5ef78da3a1a7343.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata The x86 backend should check the consistency of KVM_SET_CPUID2 because it has its constraint. Add a callback for it. The backend code will come as another patch. Suggested-by: Sean Christopherson Link: https://lore.kernel.org/lkml/ZDiGpCkXOcCm074O@google.com/ Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm-x86-ops.h | 2 ++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/cpuid.c | 6 +++++- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index ba9cc4ac9093..aaa7db45d809 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -20,6 +20,8 @@ KVM_X86_OP(hardware_disable) KVM_X86_OP(hardware_unsetup) KVM_X86_OP_OPTIONAL_RET0(offline_cpu) KVM_X86_OP(has_emulated_msr) +/* TODO: Once all backend implemented this op, remove _OPTIONAL_RET0. */ +KVM_X86_OP_OPTIONAL_RET0(vcpu_check_cpuid) KVM_X86_OP(vcpu_after_set_cpuid) KVM_X86_OP(is_vm_type_supported) KVM_X86_OP_OPTIONAL(max_vcpus); diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 291d36a668e5..304c01945115 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1590,6 +1590,7 @@ struct kvm_x86_ops { void (*hardware_unsetup)(void); int (*offline_cpu)(void); bool (*has_emulated_msr)(struct kvm *kvm, u32 index); + int (*vcpu_check_cpuid)(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e= 2, int nent); void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu); =20 bool (*is_vm_type_supported)(unsigned long vm_type); diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 09b83f7c228d..de10a2de1dd5 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -123,6 +123,7 @@ static int kvm_check_cpuid(struct kvm_vcpu *vcpu, { struct kvm_cpuid_entry2 *best; u64 xfeatures; + int r; =20 /* * The existing code assumes virtual address is 48-bit or 57-bit in the @@ -150,7 +151,10 @@ static int kvm_check_cpuid(struct kvm_vcpu *vcpu, if (!xfeatures) return 0; =20 - return fpu_enable_guest_xfd_features(&vcpu->arch.guest_fpu, xfeatures); + r =3D fpu_enable_guest_xfd_features(&vcpu->arch.guest_fpu, xfeatures); + if (r) + return r; + return static_call(kvm_x86_vcpu_check_cpuid)(vcpu, entries, nent); } =20 /* Check whether the supplied CPUID data is equal to what is already set f= or the vCPU. */ --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 151ADEB64DD for ; Tue, 25 Jul 2023 22:26:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233155AbjGYW0Z (ORCPT ); Tue, 25 Jul 2023 18:26:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233387AbjGYWXx (ORCPT ); Tue, 25 Jul 2023 18:23:53 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12A342707; Tue, 25 Jul 2023 15:19:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323570; x=1721859570; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6xxNYR8lmUaKCKOtRckHc9gTmBpZfe4Z0/GM6xw9K5A=; b=gxG0rh1Wvoeo56W7+ElPqxGwi+/qrtU6RCN9LSvu7QWgVjYgLzEPXRdo P9dL2koQatP760h8jLW4mRNIkLa4Ia+bsBKfv0jkIMjRn80Oulv9Vo9qb PK9cI9kg71VLm3JEqFPMuMLcf9fl0HIiJ7WtGVB+W8UDmNeg0ozm/QG8c sUjedD9u8xuAXAIc6lmCb3kh5MxvIm3jYXEJX6+8+vHkF8IaAW+a/oyT0 GiYDZf8ZhYTqMMS53W9JGgUgDztpvy+/fODBD7/UVWAQSHO09YJQex2cp psIHgVgMzWd+jHYMBGyRWDoA3fLPvjzzbwlta/Ys1H9GvmcsqVV+IR91G g==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882806" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882806" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001981" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001981" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:12 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 114/115] RFC: KVM: x86, TDX: Add check for KVM_SET_CPUID2 Date: Tue, 25 Jul 2023 15:15:05 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata Implement a hook of KVM_SET_CPUID2 for additional consistency check. Intel TDX or AMD SEV has a restriction on the value of cpuid. For example, some values must be the same between all vcpus. Check if the new values are consistent with the old values. The check is light because the cpuid consistency is very model specific and complicated. The user space VMM should set cpuid and MSRs consistently. Suggested-by: Sean Christopherson Link: https://lore.kernel.org/lkml/ZDiGpCkXOcCm074O@google.com/ Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 10 ++++++ arch/x86/kvm/vmx/tdx.c | 69 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/vmx/tdx.h | 7 ++++ arch/x86/kvm/vmx/x86_ops.h | 4 +++ 4 files changed, 86 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index e148d871b0a6..96823f018e60 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -443,6 +443,15 @@ static void vt_vcpu_deliver_init(struct kvm_vcpu *vcpu) kvm_vcpu_deliver_init(vcpu); } =20 +static int vt_vcpu_check_cpuid(struct kvm_vcpu *vcpu, + struct kvm_cpuid_entry2 *e2, int nent) +{ + if (is_td_vcpu(vcpu)) + return tdx_vcpu_check_cpuid(vcpu, e2, nent); + + return 0; +} + static void vt_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) @@ -1085,6 +1094,7 @@ struct kvm_x86_ops vt_x86_ops __initdata =3D { =20 .get_exit_info =3D vt_get_exit_info, =20 + .vcpu_check_cpuid =3D vt_vcpu_check_cpuid, .vcpu_after_set_cpuid =3D vt_vcpu_after_set_cpuid, =20 .has_wbinvd_exit =3D cpu_has_vmx_wbinvd_exit, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 7eeddc15d14f..1a8a3fa92303 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -479,6 +479,9 @@ void tdx_vm_free(struct kvm *kvm) =20 free_page((unsigned long)__va(kvm_tdx->tdr_pa)); kvm_tdx->tdr_pa =3D 0; + + kfree(kvm_tdx->cpuid); + kvm_tdx->cpuid =3D NULL; } =20 static int tdx_do_tdh_mng_key_config(void *param) @@ -596,6 +599,44 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) return 0; } =20 +int tdx_vcpu_check_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e= 2, int nent) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(vcpu->kvm); + const struct tdsysinfo_struct *tdsysinfo; + int i; + + tdsysinfo =3D tdx_get_sysinfo(); + if (!tdsysinfo) + return -EOPNOTSUPP; + + /* + * Simple check that new cpuid is consistent with created one. + * For simplicity, only trivial check. Don't try comprehensive checks + * with the cpuid virtualization table in the TDX module spec. + */ + for (i =3D 0; i < tdsysinfo->num_cpuid_config; i++) { + const struct tdx_cpuid_config *config =3D &tdsysinfo->cpuid_configs[i]; + u32 index =3D config->sub_leaf =3D=3D TDX_CPUID_NO_SUBLEAF ? 0 : config-= >sub_leaf; + const struct kvm_cpuid_entry2 *old =3D + kvm_find_cpuid_entry2(kvm_tdx->cpuid, kvm_tdx->cpuid_nent, + config->leaf, index); + const struct kvm_cpuid_entry2 *new =3D kvm_find_cpuid_entry2(e2, nent, + config->leaf, index); + + if (!!old !=3D !!new) + return -EINVAL; + if (!old && !new) + continue; + + if ((old->eax ^ new->eax) & config->eax || + (old->ebx ^ new->ebx) & config->ebx || + (old->ecx ^ new->ecx) & config->ecx || + (old->edx ^ new->edx) & config->edx) + return -EINVAL; + } + return 0; +} + void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); @@ -2068,10 +2109,12 @@ static int setup_tdparams_eptp_controls(struct kvm_= cpuid2 *cpuid, return 0; } =20 -static void setup_tdparams_cpuids(const struct tdsysinfo_struct *tdsysinfo, +static void setup_tdparams_cpuids(struct kvm *kvm, + const struct tdsysinfo_struct *tdsysinfo, struct kvm_cpuid2 *cpuid, struct td_params *td_params) { + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); int i; =20 /* @@ -2079,6 +2122,7 @@ static void setup_tdparams_cpuids(const struct tdsysi= nfo_struct *tdsysinfo, * be same to the one of struct tdsysinfo.{num_cpuid_config, cpuid_config= s} * It's assumed that td_params was zeroed. */ + kvm_tdx->cpuid_nent =3D 0; for (i =3D 0; i < tdsysinfo->num_cpuid_config; i++) { const struct tdx_cpuid_config *config =3D &tdsysinfo->cpuid_configs[i]; /* TDX_CPUID_NO_SUBLEAF in TDX CPUID_CONFIG means index =3D 0. */ @@ -2101,6 +2145,10 @@ static void setup_tdparams_cpuids(const struct tdsys= info_struct *tdsysinfo, value->ebx =3D entry->ebx & config->ebx; value->ecx =3D entry->ecx & config->ecx; value->edx =3D entry->edx & config->edx; + + /* Remember the setting to check for KVM_SET_CPUID2. */ + kvm_tdx->cpuid[kvm_tdx->cpuid_nent] =3D *entry; + kvm_tdx->cpuid_nent++; } } =20 @@ -2196,7 +2244,7 @@ static int setup_tdparams(struct kvm *kvm, struct td_= params *td_params, ret =3D setup_tdparams_eptp_controls(cpuid, td_params); if (ret) return ret; - setup_tdparams_cpuids(tdsysinfo, cpuid, td_params); + setup_tdparams_cpuids(kvm, tdsysinfo, cpuid, td_params); ret =3D setup_tdparams_xfam(cpuid, td_params); if (ret) return ret; @@ -2410,11 +2458,19 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_= tdx_cmd *cmd) if (cmd->flags) return -EINVAL; =20 + WARN_ON_ONCE(kvm_tdx->cpuid); + kvm_tdx->cpuid =3D kzalloc(sizeof(init_vm->cpuid.entries[0]) * KVM_MAX_CP= UID_ENTRIES, + GFP_KERNEL); + if (!kvm_tdx->cpuid) + return -ENOMEM; + init_vm =3D kzalloc(sizeof(*init_vm) + sizeof(init_vm->cpuid.entries[0]) * KVM_MAX_CPUID_ENTRIES, GFP_KERNEL); - if (!init_vm) - return -ENOMEM; + if (!init_vm) { + ret =3D -ENOMEM; + goto out; + } if (copy_from_user(init_vm, (void __user *)cmd->data, sizeof(*init_vm))) { ret =3D -EFAULT; goto out; @@ -2464,6 +2520,11 @@ static int tdx_td_init(struct kvm *kvm, struct kvm_t= dx_cmd *cmd) =20 out: /* kfree() accepts NULL. */ + if (ret) { + kfree(kvm_tdx->cpuid); + kvm_tdx->cpuid =3D NULL; + kvm_tdx->cpuid_nent =3D 0; + } kfree(init_vm); kfree(td_params); return ret; diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index c0cc09cb77ba..aff740a775bd 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -32,6 +32,13 @@ struct kvm_tdx { atomic_t tdh_mem_track; =20 u64 tsc_offset; + + /* + * For KVM_SET_CPUID to check consistency. Remember the one passed to + * TDH.MNG_INIT + */ + int cpuid_nent; + struct kvm_cpuid_entry2 *cpuid; }; =20 union tdx_exit_reason { diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 07eb0e7a5696..89660dd6cc5b 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -163,6 +163,8 @@ u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bo= ol is_mmio); =20 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, int trig_mode, int vector); +int tdx_vcpu_check_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e= 2, + int nent); void tdx_inject_nmi(struct kvm_vcpu *vcpu); void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code); @@ -215,6 +217,8 @@ static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu,= gfn_t gfn, bool is_mmio) =20 static inline void tdx_deliver_interrupt(struct kvm_lapic *apic, int deliv= ery_mode, int trig_mode, int vector) {} +static inline int tdx_vcpu_check_cpuid(struct kvm_vcpu *vcpu, struct kvm_c= puid_entry2 *e2, + int nent) { return -EOPNOTSUPP; } static inline void tdx_inject_nmi(struct kvm_vcpu *vcpu) {} static inline void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u= 64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) {} --=20 2.25.1 From nobody Sat Feb 7 20:47:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66648EB64DD for ; Tue, 25 Jul 2023 22:26:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233166AbjGYW03 (ORCPT ); Tue, 25 Jul 2023 18:26:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233399AbjGYWXy (ORCPT ); Tue, 25 Jul 2023 18:23:54 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76EAF3C3C; Tue, 25 Jul 2023 15:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690323571; x=1721859571; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kTm2N9RfSmKUFkaBFlwPDYytjDkSW5CSsRnqKsK/nWI=; b=eHOvJlQAMRsxUqHYQK6SIhD6YXdW/jZHPBYr1d1S4vzJRtnJkSzMuQxm qzg7LJMT1WzrwoHg75GUyrFvZ8KJFautQTNpxYbBh75DmqXCvsPNY7LL1 856Q0L5iPmtsHcaMlHdfi6tAskvqkd2pGRS5a8Q/oT1TJ+Bj9SwdcNLZN XB2805IjTFHEfoEBZW7gud+gi6ZKvlFDr3piKbV8YJJ4ltEAN9qRO960t BbvRGAG2Ya4e/BgpELjUmBQXO0SY+JtGQtEYMChqx8fkkn7XpVSB2zoFS VsbK9vKXhvdVtimq8IZgs2xlMz0qiSgO7DxeqrEi6KZ915boPEIUHDYFq Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="367882814" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="367882814" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10782"; a="840001986" X-IronPort-AV: E=Sophos;i="6.01,231,1684825200"; d="scan'208";a="840001986" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jul 2023 15:16:13 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com, hang.yuan@intel.com, tina.zhang@intel.com Subject: [PATCH v15 115/115] [MARKER] the end of (the first phase of) TDX KVM patch series Date: Tue, 25 Jul 2023 15:15:06 -0700 Message-Id: <64bf19be16f1c5652af1b9692b08b4a479cdcf57.1690322424.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Isaku Yamahata This empty commit is to mark the end of (the first phase of) patch series of TDX KVM support. Signed-off-by: Isaku Yamahata --- Documentation/virt/kvm/index.rst | 1 - .../virt/kvm/intel-tdx-layer-status.rst | 33 ------------------- 2 files changed, 34 deletions(-) delete mode 100644 Documentation/virt/kvm/intel-tdx-layer-status.rst diff --git a/Documentation/virt/kvm/index.rst b/Documentation/virt/kvm/inde= x.rst index ccff56dca2b1..5e78a8fc2fbd 100644 --- a/Documentation/virt/kvm/index.rst +++ b/Documentation/virt/kvm/index.rst @@ -20,4 +20,3 @@ KVM halt-polling review-checklist =20 - intel-tdx-layer-status diff --git a/Documentation/virt/kvm/intel-tdx-layer-status.rst b/Documentat= ion/virt/kvm/intel-tdx-layer-status.rst deleted file mode 100644 index 7a16fa284b6f..000000000000 --- a/Documentation/virt/kvm/intel-tdx-layer-status.rst +++ /dev/null @@ -1,33 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -Intel Trust Dodmain Extensions(TDX) -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Layer status -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -What qemu can do ----------------- -- TDX VM TYPE is exposed to Qemu. -- Qemu can create/destroy guest of TDX vm type. -- Qemu can create/destroy vcpu of TDX vm type. -- Qemu can populate initial guest memory image. -- Qemu can finalize guest TD. -- Qemu can start to run vcpu. But vcpu can not make progress yet. - -Patch Layer status ------------------- - Patch layer Status - -* TDX, VMX coexistence: Applied -* TDX architectural definitions: Applied -* TD VM creation/destruction: Applied -* TD vcpu creation/destruction: Applied -* TDX EPT violation: Applied -* TD finalization: Applied -* TD vcpu enter/exit: Applied -* TD vcpu interrupts/exit/hypercall: Not yet - -* KVM MMU GPA shared bits: Applied -* KVM TDP refactoring for TDX: Applied -* KVM TDP MMU hooks: Applied --=20 2.25.1