From nobody Tue Jun 23 22:31:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B06E1C433EF for ; Thu, 24 Feb 2022 19:19:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233628AbiBXTUD (ORCPT ); Thu, 24 Feb 2022 14:20:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233639AbiBXTTx (ORCPT ); Thu, 24 Feb 2022 14:19:53 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B0BD1B84C9 for ; Thu, 24 Feb 2022 11:19:23 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id f128-20020a623886000000b004e152a2c149so1762384pfa.5 for ; Thu, 24 Feb 2022 11:19:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=yieelq7XzoMm9U+sAB44XjR1x4+Di8aVqI+Ht06XBSY=; b=BP3ft1zkTKcGkK0f2+HsK7YjABI81wEEkbltSB6pfEqw5GleoGN3Rbmaw4xhmLahBg zFJE25ew/HqvKM3hS1EI5nqz+Tbo9kI7x0uwTu49dyjupTeta0I+LRaEFXPcD5kRcLa+ rf1Plaby0PUZoHqizdWUa7xQJh45NumpLi5vhFOxL86X4waXeMwbzYVznNaiA+5KYLTl /mZ9UEzhi87gfwzjaPWkWLB6WAAMdVUrFZmqHIVcHoyY10+IEQSh24u6wIUtk2Pqrsej GzRs96vkd9G8p/T4JAA+pER+i+hTWykK9JqkHFkN9LXRpr8Z+bGwgA/yS4R92HaasltI EyrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=yieelq7XzoMm9U+sAB44XjR1x4+Di8aVqI+Ht06XBSY=; b=SxJhW9HuPpGcqjOvygKzIZtqGP7iLmNhCtp/J2CABH2jcJDLmFXqkx3RaFj2laZbDu gAtc5HIjP1WJJRLyMiQu/6pFxnvT9MHt4wD/wLhTlMxagu3XWoNfGrWK9r9dSF/kd0tz 9VxBrmSCX96WLtFdqMKijejF8iBB9lJsEsM2oYTSBIsWwqCXTvTu/KzlxNiXutxfLb0s m374028aPl4InJK8NF5pioWQT9akqKTnY2xEvS3GC4hziUTH/xLvDgDMs63/Sedy2ZsA z9BPiAaLYAV9ADGF1zxvBsJyPwFLYm8G9ynncI8adyMHCVtZAiUdVSKbDq97LqcHPgVH aLMg== X-Gm-Message-State: AOAM531donw9iYqIqmwfscnynMjjftJwRTYiJoGKM1jr4RvIuykjDK0e s/mgTEIPjbcDysKtQD2BQ6SyvskgqJ8= X-Google-Smtp-Source: ABdhPJxEOKP7SjXFgqmje7/U4JD92c0LMC57eWxrJRQqbSQt6GELqxMbBC9x4ksWOzaksc5zc6zN7bKMSPg= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:a17:902:654b:b0:14d:964d:7578 with SMTP id d11-20020a170902654b00b0014d964d7578mr4168544pln.166.1645730362575; Thu, 24 Feb 2022 11:19:22 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 24 Feb 2022 19:19:16 +0000 In-Reply-To: <20220224191917.3508476-1-seanjc@google.com> Message-Id: <20220224191917.3508476-2-seanjc@google.com> Mime-Version: 1.0 References: <20220224191917.3508476-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH 1/2] Revert "KVM: VMX: Save HOST_CR3 in vmx_set_host_fs_gs()" From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Wanpeng Li , Lai Jiangshan Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Undo a nested VMX fix as a step toward reverting the commit it fixed, 15ad9762d69f ("KVM: VMX: Save HOST_CR3 in vmx_prepare_switch_to_guest()"), as the underlying premise that "host CR3 in the vcpu thread can only be changed when scheduling" is wrong. This reverts commit a9f2705ec84449e3b8d70c804766f8e97e23080d. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/nested.c | 3 +-- arch/x86/kvm/vmx/vmx.c | 20 +++++++++++--------- arch/x86/kvm/vmx/vmx.h | 5 ++--- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index ba34e94049c7..c12f95004a72 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -246,8 +246,7 @@ static void vmx_sync_vmcs_host_state(struct vcpu_vmx *v= mx, src =3D &prev->host_state; dest =3D &vmx->loaded_vmcs->host_state; =20 - vmx_set_vmcs_host_state(dest, src->cr3, src->fs_sel, src->gs_sel, - src->fs_base, src->gs_base); + vmx_set_host_fs_gs(dest, src->fs_sel, src->gs_sel, src->fs_base, src->gs_= base); dest->ldt_sel =3D src->ldt_sel; #ifdef CONFIG_X86_64 dest->ds_sel =3D src->ds_sel; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index efda5e4d6247..beb68cd28aca 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1080,14 +1080,9 @@ static void pt_guest_exit(struct vcpu_vmx *vmx) wrmsrl(MSR_IA32_RTIT_CTL, vmx->pt_desc.host.ctl); } =20 -void vmx_set_vmcs_host_state(struct vmcs_host_state *host, unsigned long c= r3, - u16 fs_sel, u16 gs_sel, - unsigned long fs_base, unsigned long gs_base) +void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_s= el, + unsigned long fs_base, unsigned long gs_base) { - if (unlikely(cr3 !=3D host->cr3)) { - vmcs_writel(HOST_CR3, cr3); - host->cr3 =3D cr3; - } if (unlikely(fs_sel !=3D host->fs_sel)) { if (!(fs_sel & 7)) vmcs_write16(HOST_FS_SELECTOR, fs_sel); @@ -1119,6 +1114,7 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcp= u) #ifdef CONFIG_X86_64 int cpu =3D raw_smp_processor_id(); #endif + unsigned long cr3; unsigned long fs_base, gs_base; u16 fs_sel, gs_sel; int i; @@ -1182,8 +1178,14 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vc= pu) gs_base =3D segment_base(gs_sel); #endif =20 - vmx_set_vmcs_host_state(host_state, __get_current_cr3_fast(), - fs_sel, gs_sel, fs_base, gs_base); + vmx_set_host_fs_gs(host_state, fs_sel, gs_sel, fs_base, gs_base); + + /* Host CR3 including its PCID is stable when guest state is loaded. */ + cr3 =3D __get_current_cr3_fast(); + if (unlikely(cr3 !=3D host_state->cr3)) { + vmcs_writel(HOST_CR3, cr3); + host_state->cr3 =3D cr3; + } =20 vmx->guest_state_loaded =3D true; } diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 7f2c82e7f38f..9c6bfcd84008 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -374,9 +374,8 @@ int allocate_vpid(void); void free_vpid(int vpid); void vmx_set_constant_host_state(struct vcpu_vmx *vmx); void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); -void vmx_set_vmcs_host_state(struct vmcs_host_state *host, unsigned long c= r3, - u16 fs_sel, u16 gs_sel, - unsigned long fs_base, unsigned long gs_base); +void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_s= el, + unsigned long fs_base, unsigned long gs_base); int vmx_get_cpl(struct kvm_vcpu *vcpu); bool vmx_emulation_required(struct kvm_vcpu *vcpu); unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu); --=20 2.35.1.574.g5d30c73bfb-goog From nobody Tue Jun 23 22:31:51 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C1BBC433EF for ; Thu, 24 Feb 2022 19:19:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233607AbiBXTUH (ORCPT ); Thu, 24 Feb 2022 14:20:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233660AbiBXTTz (ORCPT ); Thu, 24 Feb 2022 14:19:55 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A05AD1B84C9 for ; Thu, 24 Feb 2022 11:19:24 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id 62-20020a621541000000b004f110fdb1aeso1749490pfv.13 for ; Thu, 24 Feb 2022 11:19:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=77vvkSuzcDk7zhAWEb8twucG4UQRTj6P/P2a2+pyH2Q=; b=FiiEBYW65gPsOiKThbC8fvL5RuX2I3jgc28a+qA3vXDZjOq6Gw+PoEKzredP7K3ID0 CfU/gq3AZ9OvMjiOECa8n/dIyFGaufwgpWSfrn0HF2OLE2WmOOwmh4VjsHSNQqarF4ad 29/0LAg5KcmP1v+S0fR7GipN/sLtuPiUeZs1XU4Fgc2n3ZqdUJtpFg3kBU67VWG9LsEl 8zjjYl8RmhQ20JwgzeMKVN0ANQ/D9j+KucOGSIqPAOhZRZRYUhRa1E96kJwnYGDtvQln kAJHT1F62k0S5cDraR7KfeUMJ7UxAvwRIcWcPEAuJQsx/zhkln78xNi2+twyV+Jc8Q4U rcWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=77vvkSuzcDk7zhAWEb8twucG4UQRTj6P/P2a2+pyH2Q=; b=JB5mA4DzgOD7+2IgpSy1g9XDSWu3KsAeeUld8rjNx8Rn6s9PWl7Db72dbSHXhFMmPo eKyiRHYyE0euB+VGOd4Ba75Wi8lqZO5gKQLDrlVi/fZYZUgXXdVEMe+YUuiL7pK8qH7S LIj8Go/2oW8/PW/XhTgrnX1YUCD+x2cNIHSEjHnbKBJWTldcrAjdYTo7iCd4adIxaiAb rvXrHsTIkw9qsPFLyhFhPIYsv48/XG07wAjhOkL7IkDcveHxBK4LXtCS+6TFMter3tN1 tmHH0h5n0Y5cL3ZNKSQxjE3o5WQMDf2hFAL4ynDbfwOkfz9DG6uAROkEVLUDPFLxNG1i PTSQ== X-Gm-Message-State: AOAM531qcaoCCA8up6MCE2YJw4z4zVE0xyZiwVJPnAVObBmAHu2ghKfy hSReAi+IlA/NIBMZLGTceQLO+WEKEHg= X-Google-Smtp-Source: ABdhPJxLW6lOsAb3h8pc0hPSWq/uCMIdF/OvVb4876vbICb9U7jEj1oHgwhGbvsZ2qHOsJSTgSGtCbgKk8Y= X-Received: from seanjc.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e5]) (user=seanjc job=sendgmr) by 2002:aa7:8495:0:b0:4e1:6419:3d3c with SMTP id u21-20020aa78495000000b004e164193d3cmr4003515pfn.57.1645730364112; Thu, 24 Feb 2022 11:19:24 -0800 (PST) Reply-To: Sean Christopherson Date: Thu, 24 Feb 2022 19:19:17 +0000 In-Reply-To: <20220224191917.3508476-1-seanjc@google.com> Message-Id: <20220224191917.3508476-3-seanjc@google.com> Mime-Version: 1.0 References: <20220224191917.3508476-1-seanjc@google.com> X-Mailer: git-send-email 2.35.1.574.g5d30c73bfb-goog Subject: [PATCH 2/2] Revert "KVM: VMX: Save HOST_CR3 in vmx_prepare_switch_to_guest()" From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Wanpeng Li , Lai Jiangshan Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Revert back to refreshing vmcs.HOST_CR3 immediately prior to VM-Enter. The PCID (ASID) part of CR3 can be bumped without KVM being scheduled out, as the kernel will switch CR3 during __text_poke(), e.g. in response to a static key toggling. If switch_mm_irqs_off() chooses a new ASID for the mm associate with KVM, KVM will do VM-Enter =3D> VM-Exit with a stale vmcs.HOST_CR3. Add a comment to explain why KVM must wait until VM-Enter is imminent to refresh vmcs.HOST_CR3. The following splat was captured by stashing vmcs.HOST_CR3 in kvm_vcpu and adding a WARN in load_new_mm_cr3() to fire if a new ASID is being loaded for the KVM-associated mm while KVM has a "running" vCPU: static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) { struct kvm_vcpu *vcpu =3D kvm_get_running_vcpu(); ... WARN(vcpu && (vcpu->cr3 & GENMASK(11, 0)) !=3D (new_mm_cr3 & GENMASK(11, 0= )) && (vcpu->cr3 & PHYSICAL_PAGE_MASK) =3D=3D (new_mm_cr3 & PHYSICAL_PAGE_M= ASK), "KVM is hosed, loading CR3 =3D %lx, vmcs.HOST_CR3 =3D %lx", new_mm_cr= 3, vcpu->cr3); } ------------[ cut here ]------------ KVM is hosed, loading CR3 =3D 8000000105393004, vmcs.HOST_CR3 =3D 1053930= 03 WARNING: CPU: 4 PID: 20717 at arch/x86/mm/tlb.c:291 load_new_mm_cr3+0x82/= 0xe0 Modules linked in: vhost_net vhost vhost_iotlb tap kvm_intel CPU: 4 PID: 20717 Comm: stable Tainted: G W 5.17.0-rc3+ #7= 47 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:load_new_mm_cr3+0x82/0xe0 RSP: 0018:ffffc9000489fa98 EFLAGS: 00010082 RAX: 0000000000000000 RBX: 8000000105393004 RCX: 0000000000000027 RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff888277d1b788 RBP: 0000000000000004 R08: ffff888277d1b780 R09: ffffc9000489f8b8 R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000 R13: ffff88810678a800 R14: 0000000000000004 R15: 0000000000000c33 FS: 00007fa9f0e72700(0000) GS:ffff888277d00000(0000) knlGS:0000000000000= 000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001001b5003 CR4: 0000000000172ea0 Call Trace: switch_mm_irqs_off+0x1cb/0x460 __text_poke+0x308/0x3e0 text_poke_bp_batch+0x168/0x220 text_poke_finish+0x1b/0x30 arch_jump_label_transform_apply+0x18/0x30 static_key_slow_inc_cpuslocked+0x7c/0x90 static_key_slow_inc+0x16/0x20 kvm_lapic_set_base+0x116/0x190 kvm_set_apic_base+0xa5/0xe0 kvm_set_msr_common+0x2f4/0xf60 vmx_set_msr+0x355/0xe70 [kvm_intel] kvm_set_msr_ignored_check+0x91/0x230 kvm_emulate_wrmsr+0x36/0x120 vmx_handle_exit+0x609/0x6c0 [kvm_intel] kvm_arch_vcpu_ioctl_run+0x146f/0x1b80 kvm_vcpu_ioctl+0x279/0x690 __x64_sys_ioctl+0x83/0xb0 do_syscall_64+0x3b/0xc0 entry_SYSCALL_64_after_hwframe+0x44/0xae ---[ end trace 0000000000000000 ]--- This reverts commit 15ad9762d69fd8e40a4a51828c1d6b0c1b8fbea0. Fixes: 15ad9762d69f ("KVM: VMX: Save HOST_CR3 in vmx_prepare_switch_to_gues= t()") Reported-by: Wanpeng Li Cc: Lai Jiangshan Signed-off-by: Sean Christopherson Acked-by: Lai Jiangshan --- arch/x86/kvm/vmx/nested.c | 8 +++++++- arch/x86/kvm/vmx/vmx.c | 24 ++++++++++++++---------- 2 files changed, 21 insertions(+), 11 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index c12f95004a72..dc822a1d403d 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3055,7 +3055,7 @@ static int nested_vmx_check_guest_state(struct kvm_vc= pu *vcpu, static int nested_vmx_check_vmentry_hw(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); - unsigned long cr4; + unsigned long cr3, cr4; bool vm_fail; =20 if (!nested_early_check) @@ -3078,6 +3078,12 @@ static int nested_vmx_check_vmentry_hw(struct kvm_vc= pu *vcpu) */ vmcs_writel(GUEST_RFLAGS, 0); =20 + cr3 =3D __get_current_cr3_fast(); + if (unlikely(cr3 !=3D vmx->loaded_vmcs->host_state.cr3)) { + vmcs_writel(HOST_CR3, cr3); + vmx->loaded_vmcs->host_state.cr3 =3D cr3; + } + cr4 =3D cr4_read_shadow(); if (unlikely(cr4 !=3D vmx->loaded_vmcs->host_state.cr4)) { vmcs_writel(HOST_CR4, cr4); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index beb68cd28aca..b730d799c26e 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1114,7 +1114,6 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcp= u) #ifdef CONFIG_X86_64 int cpu =3D raw_smp_processor_id(); #endif - unsigned long cr3; unsigned long fs_base, gs_base; u16 fs_sel, gs_sel; int i; @@ -1179,14 +1178,6 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vc= pu) #endif =20 vmx_set_host_fs_gs(host_state, fs_sel, gs_sel, fs_base, gs_base); - - /* Host CR3 including its PCID is stable when guest state is loaded. */ - cr3 =3D __get_current_cr3_fast(); - if (unlikely(cr3 !=3D host_state->cr3)) { - vmcs_writel(HOST_CR3, cr3); - host_state->cr3 =3D cr3; - } - vmx->guest_state_loaded =3D true; } =20 @@ -6793,7 +6784,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); - unsigned long cr4; + unsigned long cr3, cr4; =20 /* Record the guest's net vcpu time for enforced NMI injections. */ if (unlikely(!enable_vnmi && @@ -6836,6 +6827,19 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]); vcpu->arch.regs_dirty =3D 0; =20 + /* + * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately + * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time + * it switches back to the current->mm, which can occur in KVM context + * when switching to a temporary mm to patch kernel code, e.g. if KVM + * toggles a static key while handling a VM-Exit. + */ + cr3 =3D __get_current_cr3_fast(); + if (unlikely(cr3 !=3D vmx->loaded_vmcs->host_state.cr3)) { + vmcs_writel(HOST_CR3, cr3); + vmx->loaded_vmcs->host_state.cr3 =3D cr3; + } + cr4 =3D cr4_read_shadow(); if (unlikely(cr4 !=3D vmx->loaded_vmcs->host_state.cr4)) { vmcs_writel(HOST_CR4, cr4); --=20 2.35.1.574.g5d30c73bfb-goog