From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3484C4332F for ; Thu, 17 Feb 2022 21:03:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244125AbiBQVEH (ORCPT ); Thu, 17 Feb 2022 16:04:07 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243866AbiBQVED (ORCPT ); Thu, 17 Feb 2022 16:04:03 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B59D411109A for ; Thu, 17 Feb 2022 13:03:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G/CPDOAT8bzl2lNB0wAlIxIvLWdbJ2Gig5UCTwUtGwg=; b=bjBEYqrN8rWMqIyI2kXyxQbeqtpz8VkdP2VIvQ/f73tw3knsm+6bb4U6+BbxNo9mFMewYp vMzgk6grvFnvEByxuJtw8gVcmMaXe+Bo3++O2zMgyT5q186poZVI76aEGhl2w8tJ12LhUB OeCffyc9LUbxbEZ/Cz9paJqTrfACxR0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-18-f527QjhrOGCw6lnX8liIuA-1; Thu, 17 Feb 2022 16:03:43 -0500 X-MC-Unique: f527QjhrOGCw6lnX8liIuA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0A0672F4A; Thu, 17 Feb 2022 21:03:42 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id A148546982; Thu, 17 Feb 2022 21:03:41 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com, stable@vger.kernel.org Subject: [PATCH v2 01/18] KVM: x86: host-initiated EFER.LME write affects the MMU Date: Thu, 17 Feb 2022 16:03:23 -0500 Message-Id: <20220217210340.312449-2-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" While the guest runs, EFER.LME cannot change unless CR0.PG is clear, and th= erefore EFER.NX is the only bit that can affect the MMU role. However, set_efer ac= cepts a host-initiated change to EFER.LME even with CR0.PG=3D1. In that case, the MMU has to be reset. Fixes: 11988499e62b ("KVM: x86: Skip EFER vs. guest CPUID checks for host-i= nitiated writes") Cc: stable@vger.kernel.org Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/x86.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 51faa2c76ca5..a5a50cfeffff 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -48,6 +48,7 @@ X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_PKE) =20 #define KVM_MMU_CR0_ROLE_BITS (X86_CR0_PG | X86_CR0_WP) +#define KVM_MMU_EFER_ROLE_BITS (EFER_LME | EFER_NX) =20 static __always_inline u64 rsvd_bits(int s, int e) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index d3da64106685..99a58c25f5c2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1647,7 +1647,7 @@ static int set_efer(struct kvm_vcpu *vcpu, struct msr= _data *msr_info) } =20 /* Update reserved bits */ - if ((efer ^ old_efer) & EFER_NX) + if ((efer ^ old_efer) & KVM_MMU_EFER_ROLE_BITS) kvm_mmu_reset_context(vcpu); =20 return 0; --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3733C433F5 for ; Thu, 17 Feb 2022 21:04:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244443AbiBQVER (ORCPT ); Thu, 17 Feb 2022 16:04:17 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243881AbiBQVEE (ORCPT ); Thu, 17 Feb 2022 16:04:04 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id BF58A12E173 for ; Thu, 17 Feb 2022 13:03:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=siCrqXZl1+wKhm4ojkQILAxFN3Tkl6EhHzMI4WXy5Ao=; b=VjtL44gc8qqOJDaujl89m+Q55vDbIfkRAO8Amw5UtErGRVbInfmaYUFMrZaXmqV6ikE3eU w9Us7q82lNoAGxOS8qoSZc8A2APrHJFjim2TN6kLZSgo06fQ7kdRSTCQrUBa8RLo7O6Viu yKvJLIspXFkGFwN9sY1q5P0j0sCQqsU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-621-acOMUbJOPP-_E1kmjwNxvw-1; Thu, 17 Feb 2022 16:03:43 -0500 X-MC-Unique: acOMUbJOPP-_E1kmjwNxvw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7204B2F47; Thu, 17 Feb 2022 21:03:42 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 23B1546982; Thu, 17 Feb 2022 21:03:42 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 02/18] KVM: x86: do not deliver asynchronous page faults if CR0.PG=0 Date: Thu, 17 Feb 2022 16:03:24 -0500 Message-Id: <20220217210340.312449-3-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Enabling async page faults is nonsensical if paging is disabled, but it is allowed because CR0.PG=3D0 does not clear the async page fault MSR. Just ignore them and only use the artificial halt state, similar to what happens in guest mode if async #PF vmexits are disabled. Given the increasingly complex logic, and the nicer code if the new "if" is placed last, opportunistically change the "||" into a chain of "if (...) return false" statements. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/kvm/x86.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 99a58c25f5c2..b912eef5dc1a 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -12270,14 +12270,28 @@ static inline bool apf_pageready_slot_free(struct= kvm_vcpu *vcpu) =20 static bool kvm_can_deliver_async_pf(struct kvm_vcpu *vcpu) { - if (!vcpu->arch.apf.delivery_as_pf_vmexit && is_guest_mode(vcpu)) + + if (!kvm_pv_async_pf_enabled(vcpu)) return false; =20 - if (!kvm_pv_async_pf_enabled(vcpu) || - (vcpu->arch.apf.send_user_only && static_call(kvm_x86_get_cpl)(vcpu) = =3D=3D 0)) + if (vcpu->arch.apf.send_user_only && + static_call(kvm_x86_get_cpl)(vcpu) =3D=3D 0) return false; =20 - return true; + if (is_guest_mode(vcpu)) { + /* + * L1 needs to opt into the special #PF vmexits that are + * used to deliver async page faults. + */ + return vcpu->arch.apf.delivery_as_pf_vmexit; + } else { + /* + * Play it safe in case the guest does a quick real mode + * foray. The real mode IDT is unlikely to have a #PF + * exception setup. + */ + return is_paging(vcpu); + } } =20 bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu) --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7CD4C433F5 for ; Thu, 17 Feb 2022 21:04:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244587AbiBQVEV (ORCPT ); Thu, 17 Feb 2022 16:04:21 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234111AbiBQVEE (ORCPT ); Thu, 17 Feb 2022 16:04:04 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7533615F62F for ; Thu, 17 Feb 2022 13:03:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vLJOjKLZ39eFv5+IT8kSwjIWjbjOHHgriRQmWUrdZO4=; b=gpdMXMhHeCEYQGvACxXYeF259FJyNjJ0LRvBqbymHOhsbFwlNBttJ1ZVhiMUSXlOWrDbHo Pweo/46pL5KpbY7y11QJ0ibFBc6d957vaG+Z7C9a9HbLIrWY//qfC8lVSapsyLL/285n6O 4jz+zhhJYhro7NF88eIJDo2Su0h7XLY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-639-NlK-EqKMMwa3o5ycz1cgAw-1; Thu, 17 Feb 2022 16:03:44 -0500 X-MC-Unique: NlK-EqKMMwa3o5ycz1cgAw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D9CDF1006AA0; Thu, 17 Feb 2022 21:03:42 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BEB146982; Thu, 17 Feb 2022 21:03:42 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 03/18] KVM: x86/mmu: WARN if PAE roots linger after kvm_mmu_unload Date: Thu, 17 Feb 2022 16:03:25 -0500 Message-Id: <20220217210340.312449-4-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Signed-off-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 296f8723f9ae..a67071ac80f3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5086,12 +5086,21 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) return r; } =20 +static void __kvm_mmu_unload(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) +{ + int i; + kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOTS_ALL); + WARN_ON(VALID_PAGE(mmu->root_hpa)); + if (mmu->pae_root) { + for (i =3D 0; i < 4; ++i) + WARN_ON(IS_VALID_PAE_ROOT(mmu->pae_root[i])); + } +} + void kvm_mmu_unload(struct kvm_vcpu *vcpu) { - kvm_mmu_free_roots(vcpu, &vcpu->arch.root_mmu, KVM_MMU_ROOTS_ALL); - WARN_ON(VALID_PAGE(vcpu->arch.root_mmu.root_hpa)); - kvm_mmu_free_roots(vcpu, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); - WARN_ON(VALID_PAGE(vcpu->arch.guest_mmu.root_hpa)); + __kvm_mmu_unload(vcpu, &vcpu->arch.root_mmu); + __kvm_mmu_unload(vcpu, &vcpu->arch.guest_mmu); } =20 static bool need_remote_flush(u64 old, u64 new) --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BD34C433EF for ; Thu, 17 Feb 2022 21:04:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239578AbiBQVE3 (ORCPT ); Thu, 17 Feb 2022 16:04:29 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243968AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 8151A606C0 for ; Thu, 17 Feb 2022 13:03:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OHp7eCJ3K/70kXWnZkbsnuPJwutGvkoHsNxiUQwN8KY=; b=R72TgHWC2A2oXt87/+kgM4i/roSciXgN82bG7gPrB/ByF4ZrJ/rUvp1/4KO+2hVIe6OMgG uDg4bYHyCSMgYmLQ+dAB04Szw9rerM+QyXHX1lLzcKbResN2UwtTQpN1OaK2TWkzKrV3/D to+Ai+M7q/+/8484gZrmcNtE+RKYcF4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-653-dgYWI2tIOnm4JoAoRWi8lg-1; Thu, 17 Feb 2022 16:03:46 -0500 X-MC-Unique: dgYWI2tIOnm4JoAoRWi8lg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4DC7D1091DA1; Thu, 17 Feb 2022 21:03:43 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3F6E46985; Thu, 17 Feb 2022 21:03:42 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 04/18] KVM: x86/mmu: avoid NULL-pointer dereference on page freeing bugs Date: Thu, 17 Feb 2022 16:03:26 -0500 Message-Id: <20220217210340.312449-5-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" WARN and bail if KVM attempts to free a root that isn't backed by a shadow page. KVM allocates a bare page for "special" roots, e.g. when using PAE paging or shadowing 2/3/4-level page tables with 4/5-level, and so root_hpa will be valid but won't be backed by a shadow page. It's all too easy to blindly call mmu_free_root_page() on root_hpa, be nice and WARN instead of crashing KVM and possibly the kernel. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/kvm/mmu/mmu.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a67071ac80f3..6ea423b00824 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3222,6 +3222,8 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t= *root_hpa, return; =20 sp =3D to_shadow_page(*root_hpa & PT64_BASE_ADDR_MASK); + if (WARN_ON(!sp)) + return; =20 if (is_tdp_mmu_page(sp)) kvm_tdp_mmu_put_root(kvm, sp, false); --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F51DC433F5 for ; Thu, 17 Feb 2022 21:04:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245566AbiBQVFJ (ORCPT ); Thu, 17 Feb 2022 16:05:09 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243962AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 47CF3163D7A for ; Thu, 17 Feb 2022 13:03:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=chHThbB+KzWgmw4HoFBqJADpSy+ftlKTRk5i/1W7I4o=; b=hMfLiDgw17IjWQCXKRDb7CzsG2+weWvYYs+6kEF2BmgfdyAst22e/5ovIQ+t88iiEIaY0B Fxb84pRzPIXvp9z5XqhSeS7PbdVICNgYuepGwV4xRcmsXCuBl6vDDw4Bi2WF+b5r2oVctk k6zvtq5zOhaAOdCWh5U1m+lTGtBq1Hc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-562-BMktKajQPiayVxLofW7GIg-1; Thu, 17 Feb 2022 16:03:44 -0500 X-MC-Unique: BMktKajQPiayVxLofW7GIg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B66101006AA3; Thu, 17 Feb 2022 21:03:43 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 68F7046982; Thu, 17 Feb 2022 21:03:43 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 05/18] KVM: x86/mmu: use struct kvm_mmu_root_info for mmu->root Date: Thu, 17 Feb 2022 16:03:27 -0500 Message-Id: <20220217210340.312449-6-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The root_hpa and root_pgd fields form essentially a struct kvm_mmu_root_inf= o. Use the struct to have more consistency between mmu->root and mmu->prev_roots. The patch is entirely search and replace except for cached_root_available, which does not need a temporary struct kvm_mmu_root_info anymore. Reviewed-by: Sean Christopherson Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/include/asm/kvm_host.h | 3 +- arch/x86/kvm/mmu.h | 4 +- arch/x86/kvm/mmu/mmu.c | 69 +++++++++++++++------------------ arch/x86/kvm/mmu/mmu_audit.c | 4 +- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 2 +- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/x86.c | 2 +- 10 files changed, 42 insertions(+), 50 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 8e512f25a930..6442facfd5c0 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -432,8 +432,7 @@ struct kvm_mmu { int (*sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp); void (*invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa); - hpa_t root_hpa; - gpa_t root_pgd; + struct kvm_mmu_root_info root; union kvm_mmu_role mmu_role; u8 root_level; u8 shadow_root_level; diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index a5a50cfeffff..1d0c1904d69a 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -85,7 +85,7 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); =20 static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { - if (likely(vcpu->arch.mmu->root_hpa !=3D INVALID_PAGE)) + if (likely(vcpu->arch.mmu->root.hpa !=3D INVALID_PAGE)) return 0; =20 return kvm_mmu_load(vcpu); @@ -107,7 +107,7 @@ static inline unsigned long kvm_get_active_pcid(struct = kvm_vcpu *vcpu) =20 static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) { - u64 root_hpa =3D vcpu->arch.mmu->root_hpa; + u64 root_hpa =3D vcpu->arch.mmu->root.hpa; =20 if (!VALID_PAGE(root_hpa)) return; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 6ea423b00824..a478667d7561 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2162,7 +2162,7 @@ static void shadow_walk_init_using_root(struct kvm_sh= adow_walk_iterator *iterato * prev_root is currently only used for 64-bit hosts. So only * the active root_hpa is valid here. */ - BUG_ON(root !=3D vcpu->arch.mmu->root_hpa); + BUG_ON(root !=3D vcpu->arch.mmu->root.hpa); =20 iterator->shadow_addr =3D vcpu->arch.mmu->pae_root[(addr >> 30) & 3]; @@ -2176,7 +2176,7 @@ static void shadow_walk_init_using_root(struct kvm_sh= adow_walk_iterator *iterato static void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator, struct kvm_vcpu *vcpu, u64 addr) { - shadow_walk_init_using_root(iterator, vcpu, vcpu->arch.mmu->root_hpa, + shadow_walk_init_using_root(iterator, vcpu, vcpu->arch.mmu->root.hpa, addr); } =20 @@ -3245,7 +3245,7 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct= kvm_mmu *mmu, BUILD_BUG_ON(KVM_MMU_NUM_PREV_ROOTS >=3D BITS_PER_LONG); =20 /* Before acquiring the MMU lock, see if we need to do any real work. */ - if (!(free_active_root && VALID_PAGE(mmu->root_hpa))) { + if (!(free_active_root && VALID_PAGE(mmu->root.hpa))) { for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) if ((roots_to_free & KVM_MMU_ROOT_PREVIOUS(i)) && VALID_PAGE(mmu->prev_roots[i].hpa)) @@ -3265,7 +3265,7 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct= kvm_mmu *mmu, if (free_active_root) { if (mmu->shadow_root_level >=3D PT64_ROOT_4LEVEL && (mmu->root_level >=3D PT64_ROOT_4LEVEL || mmu->direct_map)) { - mmu_free_root_page(kvm, &mmu->root_hpa, &invalid_list); + mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i =3D 0; i < 4; ++i) { if (!IS_VALID_PAE_ROOT(mmu->pae_root[i])) @@ -3276,8 +3276,8 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct= kvm_mmu *mmu, mmu->pae_root[i] =3D INVALID_PAE_ROOT; } } - mmu->root_hpa =3D INVALID_PAGE; - mmu->root_pgd =3D 0; + mmu->root.hpa =3D INVALID_PAGE; + mmu->root.pgd =3D 0; } =20 kvm_mmu_commit_zap_page(kvm, &invalid_list); @@ -3350,10 +3350,10 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *= vcpu) =20 if (is_tdp_mmu_enabled(vcpu->kvm)) { root =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu); - mmu->root_hpa =3D root; + mmu->root.hpa =3D root; } else if (shadow_root_level >=3D PT64_ROOT_4LEVEL) { root =3D mmu_alloc_root(vcpu, 0, 0, shadow_root_level, true); - mmu->root_hpa =3D root; + mmu->root.hpa =3D root; } else if (shadow_root_level =3D=3D PT32E_ROOT_LEVEL) { if (WARN_ON_ONCE(!mmu->pae_root)) { r =3D -EIO; @@ -3368,15 +3368,15 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *= vcpu) mmu->pae_root[i] =3D root | PT_PRESENT_MASK | shadow_me_mask; } - mmu->root_hpa =3D __pa(mmu->pae_root); + mmu->root.hpa =3D __pa(mmu->pae_root); } else { WARN_ONCE(1, "Bad TDP root level =3D %d\n", shadow_root_level); r =3D -EIO; goto out_unlock; } =20 - /* root_pgd is ignored for direct MMUs. */ - mmu->root_pgd =3D 0; + /* root.pgd is ignored for direct MMUs. */ + mmu->root.pgd =3D 0; out_unlock: write_unlock(&vcpu->kvm->mmu_lock); return r; @@ -3489,7 +3489,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vc= pu) if (mmu->root_level >=3D PT64_ROOT_4LEVEL) { root =3D mmu_alloc_root(vcpu, root_gfn, 0, mmu->shadow_root_level, false); - mmu->root_hpa =3D root; + mmu->root.hpa =3D root; goto set_root_pgd; } =20 @@ -3539,14 +3539,14 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *= vcpu) } =20 if (mmu->shadow_root_level =3D=3D PT64_ROOT_5LEVEL) - mmu->root_hpa =3D __pa(mmu->pml5_root); + mmu->root.hpa =3D __pa(mmu->pml5_root); else if (mmu->shadow_root_level =3D=3D PT64_ROOT_4LEVEL) - mmu->root_hpa =3D __pa(mmu->pml4_root); + mmu->root.hpa =3D __pa(mmu->pml4_root); else - mmu->root_hpa =3D __pa(mmu->pae_root); + mmu->root.hpa =3D __pa(mmu->pae_root); =20 set_root_pgd: - mmu->root_pgd =3D root_pgd; + mmu->root.pgd =3D root_pgd; out_unlock: write_unlock(&vcpu->kvm->mmu_lock); =20 @@ -3659,13 +3659,13 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) if (vcpu->arch.mmu->direct_map) return; =20 - if (!VALID_PAGE(vcpu->arch.mmu->root_hpa)) + if (!VALID_PAGE(vcpu->arch.mmu->root.hpa)) return; =20 vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY); =20 if (vcpu->arch.mmu->root_level >=3D PT64_ROOT_4LEVEL) { - hpa_t root =3D vcpu->arch.mmu->root_hpa; + hpa_t root =3D vcpu->arch.mmu->root.hpa; sp =3D to_shadow_page(root); =20 if (!is_unsync_root(root)) @@ -3956,7 +3956,7 @@ static bool kvm_faultin_pfn(struct kvm_vcpu *vcpu, st= ruct kvm_page_fault *fault, static bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, int mmu_seq) { - struct kvm_mmu_page *sp =3D to_shadow_page(vcpu->arch.mmu->root_hpa); + struct kvm_mmu_page *sp =3D to_shadow_page(vcpu->arch.mmu->root.hpa); =20 /* Special roots, e.g. pae_root, are not backed by shadow pages. */ if (sp && is_obsolete_sp(vcpu->kvm, sp)) @@ -4113,34 +4113,27 @@ static inline bool is_root_usable(struct kvm_mmu_ro= ot_info *root, gpa_t pgd, /* * Find out if a previously cached root matching the new pgd/role is avail= able. * The current root is also inserted into the cache. - * If a matching root was found, it is assigned to kvm_mmu->root_hpa and t= rue is + * If a matching root was found, it is assigned to kvm_mmu->root.hpa and t= rue is * returned. - * Otherwise, the LRU root from the cache is assigned to kvm_mmu->root_hpa= and + * Otherwise, the LRU root from the cache is assigned to kvm_mmu->root.hpa= and * false is returned. This root should now be freed by the caller. */ static bool cached_root_available(struct kvm_vcpu *vcpu, gpa_t new_pgd, union kvm_mmu_page_role new_role) { uint i; - struct kvm_mmu_root_info root; struct kvm_mmu *mmu =3D vcpu->arch.mmu; =20 - root.pgd =3D mmu->root_pgd; - root.hpa =3D mmu->root_hpa; - - if (is_root_usable(&root, new_pgd, new_role)) + if (is_root_usable(&mmu->root, new_pgd, new_role)) return true; =20 for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) { - swap(root, mmu->prev_roots[i]); + swap(mmu->root, mmu->prev_roots[i]); =20 - if (is_root_usable(&root, new_pgd, new_role)) + if (is_root_usable(&mmu->root, new_pgd, new_role)) break; } =20 - mmu->root_hpa =3D root.hpa; - mmu->root_pgd =3D root.pgd; - return i < KVM_MMU_NUM_PREV_ROOTS; } =20 @@ -4196,7 +4189,7 @@ static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, = gpa_t new_pgd, */ if (!new_role.direct) __clear_sp_write_flooding_count( - to_shadow_page(vcpu->arch.mmu->root_hpa)); + to_shadow_page(vcpu->arch.mmu->root.hpa)); } =20 void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) @@ -5092,7 +5085,7 @@ static void __kvm_mmu_unload(struct kvm_vcpu *vcpu, s= truct kvm_mmu *mmu) { int i; kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOTS_ALL); - WARN_ON(VALID_PAGE(mmu->root_hpa)); + WARN_ON(VALID_PAGE(mmu->root.hpa)); if (mmu->pae_root) { for (i =3D 0; i < 4; ++i) WARN_ON(IS_VALID_PAE_ROOT(mmu->pae_root[i])); @@ -5287,7 +5280,7 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t c= r2_or_gpa, u64 error_code, int r, emulation_type =3D EMULTYPE_PF; bool direct =3D vcpu->arch.mmu->direct_map; =20 - if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root_hpa))) + if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) return RET_PF_RETRY; =20 r =3D RET_PF_INVALID; @@ -5359,7 +5352,7 @@ void kvm_mmu_invalidate_gva(struct kvm_vcpu *vcpu, st= ruct kvm_mmu *mmu, return; =20 if (root_hpa =3D=3D INVALID_PAGE) { - mmu->invlpg(vcpu, gva, mmu->root_hpa); + mmu->invlpg(vcpu, gva, mmu->root.hpa); =20 /* * INVLPG is required to invalidate any global mappings for the VA, @@ -5395,7 +5388,7 @@ void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t= gva, unsigned long pcid) uint i; =20 if (pcid =3D=3D kvm_get_active_pcid(vcpu)) { - mmu->invlpg(vcpu, gva, mmu->root_hpa); + mmu->invlpg(vcpu, gva, mmu->root.hpa); tlb_flush =3D true; } =20 @@ -5508,8 +5501,8 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, st= ruct kvm_mmu *mmu) struct page *page; int i; =20 - mmu->root_hpa =3D INVALID_PAGE; - mmu->root_pgd =3D 0; + mmu->root.hpa =3D INVALID_PAGE; + mmu->root.pgd =3D 0; for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) mmu->prev_roots[i] =3D KVM_MMU_ROOT_INFO_INVALID; =20 diff --git a/arch/x86/kvm/mmu/mmu_audit.c b/arch/x86/kvm/mmu/mmu_audit.c index f31fdb874f1f..3e5d62a25350 100644 --- a/arch/x86/kvm/mmu/mmu_audit.c +++ b/arch/x86/kvm/mmu/mmu_audit.c @@ -56,11 +56,11 @@ static void mmu_spte_walk(struct kvm_vcpu *vcpu, inspec= t_spte_fn fn) int i; struct kvm_mmu_page *sp; =20 - if (!VALID_PAGE(vcpu->arch.mmu->root_hpa)) + if (!VALID_PAGE(vcpu->arch.mmu->root.hpa)) return; =20 if (vcpu->arch.mmu->root_level >=3D PT64_ROOT_4LEVEL) { - hpa_t root =3D vcpu->arch.mmu->root_hpa; + hpa_t root =3D vcpu->arch.mmu->root.hpa; =20 sp =3D to_shadow_page(root); __mmu_spte_walk(vcpu, sp, fn, vcpu->arch.mmu->root_level); diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 5b5bdac97c7b..346f3bad3cb9 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -668,7 +668,7 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct k= vm_page_fault *fault, if (FNAME(gpte_changed)(vcpu, gw, top_level)) goto out_gpte_changed; =20 - if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root_hpa))) + if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) goto out_gpte_changed; =20 for (shadow_walk_init(&it, vcpu, fault->addr); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 8def8f810cb0..debf08212f12 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -657,7 +657,7 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct= kvm *kvm, else =20 #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, to_shadow_page(_mmu->root_hpa), _start, _end) + for_each_tdp_pte(_iter, to_shadow_page(_mmu->root.hpa), _start, _end) =20 /* * Yield if the MMU lock is contended or this thread needs to return contr= ol diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3f987785702a..57c73d8f76ce 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -95,7 +95,7 @@ static inline bool is_tdp_mmu_page(struct kvm_mmu_page *s= p) { return sp->tdp_mmu static inline bool is_tdp_mmu(struct kvm_mmu *mmu) { struct kvm_mmu_page *sp; - hpa_t hpa =3D mmu->root_hpa; + hpa_t hpa =3D mmu->root.hpa; =20 if (WARN_ON(!VALID_PAGE(hpa))) return false; diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index c73e4d938ddc..29289ecca223 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -5466,7 +5466,7 @@ static int handle_invept(struct kvm_vcpu *vcpu) VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); =20 roots_to_free =3D 0; - if (nested_ept_root_matches(mmu->root_hpa, mmu->root_pgd, + if (nested_ept_root_matches(mmu->root.hpa, mmu->root.pgd, operand.eptp)) roots_to_free |=3D KVM_MMU_ROOT_CURRENT; =20 diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d8547144d3b7..b183dfc41d74 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -2952,7 +2952,7 @@ static inline int vmx_get_current_vpid(struct kvm_vcp= u *vcpu) static void vmx_flush_tlb_current(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; - u64 root_hpa =3D mmu->root_hpa; + u64 root_hpa =3D mmu->root.hpa; =20 /* No flush required if the current context is invalid. */ if (!VALID_PAGE(root_hpa)) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b912eef5dc1a..c0d7256e3a78 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -762,7 +762,7 @@ bool kvm_inject_emulated_page_fault(struct kvm_vcpu *vc= pu, if ((fault->error_code & PFERR_PRESENT_MASK) && !(fault->error_code & PFERR_RSVD_MASK)) kvm_mmu_invalidate_gva(vcpu, fault_mmu, fault->address, - fault_mmu->root_hpa); + fault_mmu->root.hpa); =20 fault_mmu->inject_page_fault(vcpu, fault); return fault->nested_page_fault; --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3A8CC433EF for ; Thu, 17 Feb 2022 21:04:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244303AbiBQVEN (ORCPT ); Thu, 17 Feb 2022 16:04:13 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243853AbiBQVEC (ORCPT ); Thu, 17 Feb 2022 16:04:02 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 76EDAD1080 for ; Thu, 17 Feb 2022 13:03:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9EeiJmrNVxBypIUwOEl5IfkA+kYU2Kz2k4uxn3ETf0w=; b=YqGKDyE8iXtUsZ9BEeUDOWvWdvQmLI3dbGuRWlY1EzPlPMJ3A52Z92ksGgSyTjZLVO3Wlx YvJMHoTEyIrEnk59Rq3qxZFRIRNSqLFfvDM2qvWAjvKk8JJVQ8++T3BZeDN98y8asci4jC sSZHB3M5IpeOuXGoi4hmH92sxcR8KBM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-302-B0NOLb-tO5uPcypCRKhefw-1; Thu, 17 Feb 2022 16:03:45 -0500 X-MC-Unique: B0NOLb-tO5uPcypCRKhefw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 29F671006AA4; Thu, 17 Feb 2022 21:03:44 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id D0D2346982; Thu, 17 Feb 2022 21:03:43 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 06/18] KVM: x86/mmu: do not consult levels when freeing roots Date: Thu, 17 Feb 2022 16:03:28 -0500 Message-Id: <20220217210340.312449-7-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Right now, PGD caching requires a complicated dance of first computing the MMU role and passing it to __kvm_mmu_new_pgd(), and then separately cal= ling kvm_init_mmu(). Part of this is due to kvm_mmu_free_roots using mmu->root_level and mmu->shadow_root_level to distinguish whether the page table uses a single root or 4 PAE roots. Because kvm_init_mmu() can overwrite mmu->root_level, kvm_mmu_free_roots() must be called before kvm_init_mmu(). However, even after kvm_init_mmu() there is a way to detect whether the page table may hold PAE roots, as root.hpa isn't backed by a shadow when it points at PAE roots. Using this method results in simpler code, and is one less obstacle in moving all calls to __kvm_mmu_new_pgd() after the MMU has been initialized. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/kvm/mmu/mmu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a478667d7561..e1578f71feae 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3240,12 +3240,15 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, stru= ct kvm_mmu *mmu, struct kvm *kvm =3D vcpu->kvm; int i; LIST_HEAD(invalid_list); - bool free_active_root =3D roots_to_free & KVM_MMU_ROOT_CURRENT; + bool free_active_root; =20 BUILD_BUG_ON(KVM_MMU_NUM_PREV_ROOTS >=3D BITS_PER_LONG); =20 /* Before acquiring the MMU lock, see if we need to do any real work. */ - if (!(free_active_root && VALID_PAGE(mmu->root.hpa))) { + free_active_root =3D (roots_to_free & KVM_MMU_ROOT_CURRENT) + && VALID_PAGE(mmu->root.hpa); + + if (!free_active_root) { for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) if ((roots_to_free & KVM_MMU_ROOT_PREVIOUS(i)) && VALID_PAGE(mmu->prev_roots[i].hpa)) @@ -3263,8 +3266,7 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct= kvm_mmu *mmu, &invalid_list); =20 if (free_active_root) { - if (mmu->shadow_root_level >=3D PT64_ROOT_4LEVEL && - (mmu->root_level >=3D PT64_ROOT_4LEVEL || mmu->direct_map)) { + if (to_shadow_page(mmu->root.hpa)) { mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i =3D 0; i < 4; ++i) { --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65652C4332F for ; Thu, 17 Feb 2022 21:04:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244749AbiBQVE0 (ORCPT ); Thu, 17 Feb 2022 16:04:26 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243948AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EAE00164D31 for ; Thu, 17 Feb 2022 13:03:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=greicK0A3/Bhl1WwndRbJf57VmhSiHQc1qeb/4OzOEo=; b=bhCxWLi63FT3GZXzxMWKG5yIMVHGs+X6xVnRJDHTWjqG4FIi8cSr1tjJkyiwGhgwYDEw3+ 9si1CD4X75033beG40cErXfqc5glVcL3dXA7SPI871Ps8wb7X2wiVukBftglKNLBuoRp2T pYoiQpMWsRmRFn5Xb8YpByeEl6Pipzo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-568-pbi92wF3N26i6kL4ofPfZQ-1; Thu, 17 Feb 2022 16:03:45 -0500 X-MC-Unique: pbi92wF3N26i6kL4ofPfZQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AEBFC1800D50; Thu, 17 Feb 2022 21:03:44 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4485D46985; Thu, 17 Feb 2022 21:03:44 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com, Lai Jiangshan Subject: [PATCH v2 07/18] KVM: x86/mmu: Do not use guest root level in audit Date: Thu, 17 Feb 2022 16:03:29 -0500 Message-Id: <20220217210340.312449-8-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Walking from the root page of the shadow page table should start with the level of the shadow page table: shadow_root_level; do not consult the level in order to check whether the root has a single root or uses pae_root, either, and use to_shadow_page instead. Also tweak audit_mappings(), where the current walking level is more valuable to print. Signed-off-by: Lai Jiangshan Signed-off-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu_audit.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu_audit.c b/arch/x86/kvm/mmu/mmu_audit.c index 3e5d62a25350..d1c59aed0465 100644 --- a/arch/x86/kvm/mmu/mmu_audit.c +++ b/arch/x86/kvm/mmu/mmu_audit.c @@ -53,17 +53,16 @@ static void __mmu_spte_walk(struct kvm_vcpu *vcpu, stru= ct kvm_mmu_page *sp, =20 static void mmu_spte_walk(struct kvm_vcpu *vcpu, inspect_spte_fn fn) { - int i; + hpa_t root =3D vcpu->arch.mmu->root.hpa; struct kvm_mmu_page *sp; + int i; =20 - if (!VALID_PAGE(vcpu->arch.mmu->root.hpa)) + if (!VALID_PAGE(root)) return; =20 - if (vcpu->arch.mmu->root_level >=3D PT64_ROOT_4LEVEL) { - hpa_t root =3D vcpu->arch.mmu->root.hpa; - - sp =3D to_shadow_page(root); - __mmu_spte_walk(vcpu, sp, fn, vcpu->arch.mmu->root_level); + sp =3D to_shadow_page(root); + if (sp) { + __mmu_spte_walk(vcpu, sp, fn, vcpu->arch.mmu->shadow_root_level); return; } =20 @@ -119,8 +118,7 @@ static void audit_mappings(struct kvm_vcpu *vcpu, u64 *= sptep, int level) hpa =3D pfn << PAGE_SHIFT; if ((*sptep & PT64_BASE_ADDR_MASK) !=3D hpa) audit_printk(vcpu->kvm, "levels %d pfn %llx hpa %llx " - "ent %llxn", vcpu->arch.mmu->root_level, pfn, - hpa, *sptep); + "ent %llxn", level, pfn, hpa, *sptep); } =20 static void inspect_spte_has_rmap(struct kvm *kvm, u64 *sptep) --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 441AAC433F5 for ; Thu, 17 Feb 2022 21:05:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245093AbiBQVFP (ORCPT ); Thu, 17 Feb 2022 16:05:15 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243934AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E2FD6164D07 for ; Thu, 17 Feb 2022 13:03:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cowP525RDo3JvSn3YZ8N95snotZ2rMBxu2rrsrRfihc=; b=SHjf0SJ9gcm6XfYKPnHuYTXV+bOkzFiSyyeqvFbRM0K/+2zjRXNNdvYYFwBPQExmoFanNC dlZ9fVNMkadbBOW0lAT+0uBrpJCA7EG2tw7t9LGdehjKhaJiPcule0WxYK7fQm3QfhX7ri 1lRcEmgVWAXDpbsR6prpIAlPSK6+OTU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-fuAT82pXN1aZLabKrQq08w-1; Thu, 17 Feb 2022 16:03:46 -0500 X-MC-Unique: fuAT82pXN1aZLabKrQq08w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4DCCA801AC5; Thu, 17 Feb 2022 21:03:45 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id C9A7B46982; Thu, 17 Feb 2022 21:03:44 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 08/18] KVM: x86/mmu: do not pass vcpu to root freeing functions Date: Thu, 17 Feb 2022 16:03:30 -0500 Message-Id: <20220217210340.312449-9-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" These functions only operate on a given MMU, of which there are two in a vC= PU. They also need a struct kvm in order to lock the mmu_lock, but they do not needed anything else in the struct kvm_vcpu. So, pass the vcpu->kvm direct= ly to them. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/mmu/mmu.c | 21 +++++++++++---------- arch/x86/kvm/vmx/nested.c | 8 ++++---- arch/x86/kvm/x86.c | 4 ++-- 4 files changed, 19 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 6442facfd5c0..79f37ccc8726 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1780,9 +1780,9 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu); void kvm_update_dr7(struct kvm_vcpu *vcpu); =20 int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn); -void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, +void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, ulong roots_to_free); -void kvm_mmu_free_guest_mode_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *= mmu); +void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu); gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, struct x86_exception *exception); gpa_t kvm_mmu_gva_to_gpa_fetch(struct kvm_vcpu *vcpu, gva_t gva, diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index e1578f71feae..0f2de811e871 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3234,10 +3234,9 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_= t *root_hpa, } =20 /* roots_to_free must be some combination of the KVM_MMU_ROOT_* flags */ -void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, +void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, ulong roots_to_free) { - struct kvm *kvm =3D vcpu->kvm; int i; LIST_HEAD(invalid_list); bool free_active_root; @@ -3287,7 +3286,7 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct= kvm_mmu *mmu, } EXPORT_SYMBOL_GPL(kvm_mmu_free_roots); =20 -void kvm_mmu_free_guest_mode_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *= mmu) +void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) { unsigned long roots_to_free =3D 0; hpa_t root_hpa; @@ -3309,7 +3308,7 @@ void kvm_mmu_free_guest_mode_roots(struct kvm_vcpu *v= cpu, struct kvm_mmu *mmu) roots_to_free |=3D KVM_MMU_ROOT_PREVIOUS(i); } =20 - kvm_mmu_free_roots(vcpu, mmu, roots_to_free); + kvm_mmu_free_roots(kvm, mmu, roots_to_free); } EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots); =20 @@ -3710,7 +3709,7 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu) roots_to_free |=3D KVM_MMU_ROOT_PREVIOUS(i); =20 /* sync prev_roots by simply freeing them */ - kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, roots_to_free); + kvm_mmu_free_roots(vcpu->kvm, vcpu->arch.mmu, roots_to_free); } =20 static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *m= mu, @@ -4159,8 +4158,10 @@ static bool fast_pgd_switch(struct kvm_vcpu *vcpu, g= pa_t new_pgd, static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd, union kvm_mmu_page_role new_role) { + struct kvm_mmu *mmu =3D vcpu->arch.mmu; + if (!fast_pgd_switch(vcpu, new_pgd, new_role)) { - kvm_mmu_free_roots(vcpu, vcpu->arch.mmu, KVM_MMU_ROOT_CURRENT); + kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); return; } =20 @@ -5083,10 +5084,10 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) return r; } =20 -static void __kvm_mmu_unload(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) +static void __kvm_mmu_unload(struct kvm *kvm, struct kvm_mmu *mmu) { int i; - kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOTS_ALL); + kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOTS_ALL); WARN_ON(VALID_PAGE(mmu->root.hpa)); if (mmu->pae_root) { for (i =3D 0; i < 4; ++i) @@ -5096,8 +5097,8 @@ static void __kvm_mmu_unload(struct kvm_vcpu *vcpu, s= truct kvm_mmu *mmu) =20 void kvm_mmu_unload(struct kvm_vcpu *vcpu) { - __kvm_mmu_unload(vcpu, &vcpu->arch.root_mmu); - __kvm_mmu_unload(vcpu, &vcpu->arch.guest_mmu); + __kvm_mmu_unload(vcpu->kvm, &vcpu->arch.root_mmu); + __kvm_mmu_unload(vcpu->kvm, &vcpu->arch.guest_mmu); } =20 static bool need_remote_flush(u64 old, u64 new) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 29289ecca223..b7bc634d35e2 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -321,7 +321,7 @@ static void free_nested(struct kvm_vcpu *vcpu) kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map, true); vmx->nested.pi_desc =3D NULL; =20 - kvm_mmu_free_roots(vcpu, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); + kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); =20 nested_release_evmcs(vcpu); =20 @@ -5007,7 +5007,7 @@ static inline void nested_release_vmcs12(struct kvm_v= cpu *vcpu) vmx->nested.current_vmptr >> PAGE_SHIFT, vmx->nested.cached_vmcs12, 0, VMCS12_SIZE); =20 - kvm_mmu_free_roots(vcpu, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); + kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); =20 vmx->nested.current_vmptr =3D INVALID_GPA; } @@ -5486,7 +5486,7 @@ static int handle_invept(struct kvm_vcpu *vcpu) } =20 if (roots_to_free) - kvm_mmu_free_roots(vcpu, mmu, roots_to_free); + kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free); =20 return nested_vmx_succeed(vcpu); } @@ -5575,7 +5575,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) * TODO: sync only the affected SPTEs for INVDIVIDUAL_ADDR. */ if (!enable_ept) - kvm_mmu_free_guest_mode_roots(vcpu, &vcpu->arch.root_mmu); + kvm_mmu_free_guest_mode_roots(vcpu->kvm, &vcpu->arch.root_mmu); =20 return nested_vmx_succeed(vcpu); } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c0d7256e3a78..6aefd7ac7039 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -855,7 +855,7 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr= 3) * Shadow page roots need to be reconstructed instead. */ if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) - kvm_mmu_free_roots(vcpu, mmu, KVM_MMU_ROOT_CURRENT); + kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); =20 memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); @@ -1156,7 +1156,7 @@ static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu= , unsigned long pcid) if (kvm_get_pcid(vcpu, mmu->prev_roots[i].pgd) =3D=3D pcid) roots_to_free |=3D KVM_MMU_ROOT_PREVIOUS(i); =20 - kvm_mmu_free_roots(vcpu, mmu, roots_to_free); + kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free); } =20 int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4515CC4332F for ; Thu, 17 Feb 2022 21:04:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244651AbiBQVEX (ORCPT ); Thu, 17 Feb 2022 16:04:23 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243907AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2E1F8163D79 for ; Thu, 17 Feb 2022 13:03:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQkaRkpDm0Ym0G1tYvl1LrDOa9a/hWfn0PzBdx+ovns=; b=etHbskm5Lbl4V9ONRGzzI/TKz+fA7f/SKnSnVvxmpiyOi0tFSAJTC0IfPJC5v9MX3DiL9F TUWMXByImwdmESm19+VvzKbncMwb70wbK1mU7PQ7Ro7NaK8p8IHoxb9Kzc8MVzBA81pF/g wlAk7YlvRMLjrGiYaqgWelOw+h3aQrg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-588-3OA7P5LIP7OT8lIVmsVEvg-1; Thu, 17 Feb 2022 16:03:46 -0500 X-MC-Unique: 3OA7P5LIP7OT8lIVmsVEvg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8C2081091DA0; Thu, 17 Feb 2022 21:03:45 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E26A46983; Thu, 17 Feb 2022 21:03:45 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 09/18] KVM: x86/mmu: look for a cached PGD when going from 32-bit to 64-bit Date: Thu, 17 Feb 2022 16:03:31 -0500 Message-Id: <20220217210340.312449-10-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Right now, PGD caching avoids placing a PAE root in the cache by using the old value of mmu->root_level and mmu->shadow_root_level; it does not look for a cached PGD if the old root is a PAE one, and then frees it using kvm_mmu_free_roots. Change the logic instead to free the uncacheable root early. This way, __kvm_new_mmu_pgd is able to look up the cache when going from 32-bit to 64-bit (if there is a hit, the invalid root becomes the least recently used). An example of this is nested virtualization with shadow paging, when a 64-bit L1 runs a 32-bit L2. As a side effect (which is actually the reason why this patch was written), PGD caching does not use the old value of mmu->root_level and mmu->shadow_root_level anymore. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/kvm/mmu/mmu.c | 82 ++++++++++++++++++++++++++++++------------ 1 file changed, 59 insertions(+), 23 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 0f2de811e871..da324a317000 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4107,52 +4107,88 @@ static inline bool is_root_usable(struct kvm_mmu_ro= ot_info *root, gpa_t pgd, union kvm_mmu_page_role role) { return (role.direct || pgd =3D=3D root->pgd) && - VALID_PAGE(root->hpa) && to_shadow_page(root->hpa) && + VALID_PAGE(root->hpa) && role.word =3D=3D to_shadow_page(root->hpa)->role.word; } =20 /* - * Find out if a previously cached root matching the new pgd/role is avail= able. - * The current root is also inserted into the cache. - * If a matching root was found, it is assigned to kvm_mmu->root.hpa and t= rue is - * returned. - * Otherwise, the LRU root from the cache is assigned to kvm_mmu->root.hpa= and - * false is returned. This root should now be freed by the caller. + * Find out if a previously cached root matching the new pgd/role is avail= able, + * and insert the current root as the MRU in the cache. + * If a matching root is found, it is assigned to kvm_mmu->root and + * true is returned. + * If no match is found, kvm_mmu->root is left invalid, the LRU root is + * evicted to make room for the current root, and false is returned. */ -static bool cached_root_available(struct kvm_vcpu *vcpu, gpa_t new_pgd, - union kvm_mmu_page_role new_role) +static bool cached_root_find_and_keep_current(struct kvm *kvm, struct kvm_= mmu *mmu, + gpa_t new_pgd, + union kvm_mmu_page_role new_role) { uint i; - struct kvm_mmu *mmu =3D vcpu->arch.mmu; =20 if (is_root_usable(&mmu->root, new_pgd, new_role)) return true; =20 for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) { + /* + * The swaps end up rotating the cache like this: + * C 0 1 2 3 (on entry to the function) + * 0 C 1 2 3 + * 1 C 0 2 3 + * 2 C 0 1 3 + * 3 C 0 1 2 (on exit from the loop) + */ swap(mmu->root, mmu->prev_roots[i]); - if (is_root_usable(&mmu->root, new_pgd, new_role)) - break; + return true; } =20 - return i < KVM_MMU_NUM_PREV_ROOTS; + kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); + return false; } =20 -static bool fast_pgd_switch(struct kvm_vcpu *vcpu, gpa_t new_pgd, - union kvm_mmu_page_role new_role) +/* + * Find out if a previously cached root matching the new pgd/role is avail= able. + * On entry, mmu->root is invalid. + * If a matching root is found, it is assigned to kvm_mmu->root, the LRU e= ntry + * of the cache becomes invalid, and true is returned. + * If no match is found, kvm_mmu->root is left invalid and false is return= ed. + */ +static bool cached_root_find_without_current(struct kvm *kvm, struct kvm_m= mu *mmu, + gpa_t new_pgd, + union kvm_mmu_page_role new_role) { - struct kvm_mmu *mmu =3D vcpu->arch.mmu; + uint i; + + for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) + if (is_root_usable(&mmu->prev_roots[i], new_pgd, new_role)) + goto hit; =20 + return false; + +hit: + swap(mmu->root, mmu->prev_roots[i]); + /* Bubble up the remaining roots. */ + for (; i < KVM_MMU_NUM_PREV_ROOTS - 1; i++) + mmu->prev_roots[i] =3D mmu->prev_roots[i + 1]; + mmu->prev_roots[i].hpa =3D INVALID_PAGE; + return true; +} + +static bool fast_pgd_switch(struct kvm *kvm, struct kvm_mmu *mmu, + gpa_t new_pgd, union kvm_mmu_page_role new_role) +{ /* - * For now, limit the fast switch to 64-bit hosts+VMs in order to avoid + * For now, limit the caching to 64-bit hosts+VMs in order to avoid * having to deal with PDPTEs. We may add support for 32-bit hosts/VMs * later if necessary. */ - if (mmu->shadow_root_level >=3D PT64_ROOT_4LEVEL && - mmu->root_level >=3D PT64_ROOT_4LEVEL) - return cached_root_available(vcpu, new_pgd, new_role); + if (VALID_PAGE(mmu->root.hpa) && !to_shadow_page(mmu->root.hpa)) + kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); =20 - return false; + if (VALID_PAGE(mmu->root.hpa)) + return cached_root_find_and_keep_current(kvm, mmu, new_pgd, new_role); + else + return cached_root_find_without_current(kvm, mmu, new_pgd, new_role); } =20 static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd, @@ -4160,8 +4196,8 @@ static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, = gpa_t new_pgd, { struct kvm_mmu *mmu =3D vcpu->arch.mmu; =20 - if (!fast_pgd_switch(vcpu, new_pgd, new_role)) { - kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); + if (!fast_pgd_switch(vcpu->kvm, mmu, new_pgd, new_role)) { + /* kvm_mmu_ensure_valid_pgd will set up a new root. */ return; } =20 --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E531C433EF for ; Thu, 17 Feb 2022 21:04:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231642AbiBQVEz (ORCPT ); Thu, 17 Feb 2022 16:04:55 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:38024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244111AbiBQVEH (ORCPT ); Thu, 17 Feb 2022 16:04:07 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 13E69606C0 for ; Thu, 17 Feb 2022 13:03:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=azf3rzszlIQOlGktX/+DfnkIwBQlpFBhEAOhLBzgXWQ=; b=X4d6+MXhUGLf3r15GKldvBj/YxAun/6nistxtQ8op6gID9IUoWyQppPSL69ZF19WnvfZO2 m5PC6R92Kdq73fgS1OuCDd7H5y0XhT26IA04Jn8maBqwpvF+EvSwmBJPXgaeY3x+SbNNO5 R3uKBXt0RunK+ohD7ck9lRdaJR15HTk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-634-ddX2YMV7MyGwBM1sgv02bw-1; Thu, 17 Feb 2022 16:03:47 -0500 X-MC-Unique: ddX2YMV7MyGwBM1sgv02bw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9F15B1091DA2; Thu, 17 Feb 2022 21:03:46 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2446C6AB90; Thu, 17 Feb 2022 21:03:46 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 10/18] KVM: x86/mmu: load new PGD after the shadow MMU is initialized Date: Thu, 17 Feb 2022 16:03:32 -0500 Message-Id: <20220217210340.312449-11-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now that __kvm_mmu_new_pgd does not look at the MMU's root_level and shadow_root_level anymore, pull the PGD load after the initialization of the shadow MMUs. Besides being more intuitive, this enables future simplifications and optimizations because it's not necessary anymore to compute the role outside kvm_init_mmu. In particular, kvm_mmu_reset_context was not attempting to use a cached PGD to avoid having to figure out the new role. It will soon be able to follow what nested_{vmx,svm}_load_cr3 are doing, and avoid unloading all the cached roots. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 37 +++++++++++++++++-------------------- arch/x86/kvm/svm/nested.c | 6 +++--- arch/x86/kvm/vmx/nested.c | 6 +++--- 3 files changed, 23 insertions(+), 26 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index da324a317000..906a9244ad28 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4903,9 +4903,8 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, u= nsigned long cr0, =20 new_role =3D kvm_calc_shadow_npt_root_page_role(vcpu, ®s); =20 - __kvm_mmu_new_pgd(vcpu, nested_cr3, new_role.base); - shadow_mmu_init_context(vcpu, context, ®s, new_role); + __kvm_mmu_new_pgd(vcpu, nested_cr3, new_role.base); } EXPORT_SYMBOL_GPL(kvm_init_shadow_npt_mmu); =20 @@ -4943,27 +4942,25 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu,= bool execonly, kvm_calc_shadow_ept_root_page_role(vcpu, accessed_dirty, execonly, level); =20 - __kvm_mmu_new_pgd(vcpu, new_eptp, new_role.base); - - if (new_role.as_u64 =3D=3D context->mmu_role.as_u64) - return; - - context->mmu_role.as_u64 =3D new_role.as_u64; + if (new_role.as_u64 !=3D context->mmu_role.as_u64) { + context->mmu_role.as_u64 =3D new_role.as_u64; =20 - context->shadow_root_level =3D level; + context->shadow_root_level =3D level; =20 - context->ept_ad =3D accessed_dirty; - context->page_fault =3D ept_page_fault; - context->gva_to_gpa =3D ept_gva_to_gpa; - context->sync_page =3D ept_sync_page; - context->invlpg =3D ept_invlpg; - context->root_level =3D level; - context->direct_map =3D false; + context->ept_ad =3D accessed_dirty; + context->page_fault =3D ept_page_fault; + context->gva_to_gpa =3D ept_gva_to_gpa; + context->sync_page =3D ept_sync_page; + context->invlpg =3D ept_invlpg; + context->root_level =3D level; + context->direct_map =3D false; + update_permission_bitmask(context, true); + context->pkru_mask =3D 0; + reset_rsvds_bits_mask_ept(vcpu, context, execonly, huge_page_level); + reset_ept_shadow_zero_bits_mask(context, execonly); + } =20 - update_permission_bitmask(context, true); - context->pkru_mask =3D 0; - reset_rsvds_bits_mask_ept(vcpu, context, execonly, huge_page_level); - reset_ept_shadow_zero_bits_mask(context, execonly); + __kvm_mmu_new_pgd(vcpu, new_eptp, new_role.base); } EXPORT_SYMBOL_GPL(kvm_init_shadow_ept_mmu); =20 diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index f284e61451c8..96bab464967f 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -492,14 +492,14 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu,= unsigned long cr3, CC(!load_pdptrs(vcpu, cr3))) return -EINVAL; =20 - if (!nested_npt) - kvm_mmu_new_pgd(vcpu, cr3); - vcpu->arch.cr3 =3D cr3; =20 /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ kvm_init_mmu(vcpu); =20 + if (!nested_npt) + kvm_mmu_new_pgd(vcpu, cr3); + return 0; } =20 diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index b7bc634d35e2..1dfe23963a9e 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1126,15 +1126,15 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcp= u, unsigned long cr3, return -EINVAL; } =20 - if (!nested_ept) - kvm_mmu_new_pgd(vcpu, cr3); - vcpu->arch.cr3 =3D cr3; kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); =20 /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ kvm_init_mmu(vcpu); =20 + if (!nested_ept) + kvm_mmu_new_pgd(vcpu, cr3); + return 0; } =20 --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0634C433F5 for ; Thu, 17 Feb 2022 21:04:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245060AbiBQVEs (ORCPT ); Thu, 17 Feb 2022 16:04:48 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243956AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 59BA085643 for ; Thu, 17 Feb 2022 13:03:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bb07ybGCMBRl5oIy12eV8DM3Y/j7y0FXE0a9BHr0sbA=; b=SXT7+sVt+kzL9tftzYpQU8JKzzYV22hhBQ4uUNc/0hcLVqU0i1aJrM17noG4e6TYcTSktH ULunN+AeSJGe15uKG+us+m+3o+tVTChNbH0isaF2EHz06kUn+8FfxXKXKvZGa9gPgjs6pk 3wZIJ9RyIuundRlb/tabF6gZ4uR1v+Q= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-660-h7MQuTrJOqClz9PJVAEJXA-1; Thu, 17 Feb 2022 16:03:47 -0500 X-MC-Unique: h7MQuTrJOqClz9PJVAEJXA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E8AB31800D50; Thu, 17 Feb 2022 21:03:46 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F0B36AB95; Thu, 17 Feb 2022 21:03:46 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 11/18] KVM: x86/mmu: Always use current mmu's role when loading new PGD Date: Thu, 17 Feb 2022 16:03:33 -0500 Message-Id: <20220217210340.312449-12-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Since the guest PGD is now loaded after the MMU has been set up completely, the desired role for a cache hit is simply the current mmu_role. There is no need to compute it again, so __kvm_mmu_new_pgd can be folded in kvm_mmu_new_pgd. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 29 ++++------------------------- 1 file changed, 4 insertions(+), 25 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 906a9244ad28..b01160716c6a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -190,8 +190,6 @@ struct kmem_cache *mmu_page_header_cache; static struct percpu_counter kvm_total_used_mmu_pages; =20 static void mmu_spte_set(u64 *sptep, u64 spte); -static union kvm_mmu_page_role -kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu); =20 struct kvm_mmu_role_regs { const unsigned long cr0; @@ -4191,10 +4189,10 @@ static bool fast_pgd_switch(struct kvm *kvm, struct= kvm_mmu *mmu, return cached_root_find_without_current(kvm, mmu, new_pgd, new_role); } =20 -static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd, - union kvm_mmu_page_role new_role) +void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; + union kvm_mmu_page_role new_role =3D mmu->mmu_role.base; =20 if (!fast_pgd_switch(vcpu->kvm, mmu, new_pgd, new_role)) { /* kvm_mmu_ensure_valid_pgd will set up a new root. */ @@ -4230,11 +4228,6 @@ static void __kvm_mmu_new_pgd(struct kvm_vcpu *vcpu,= gpa_t new_pgd, __clear_sp_write_flooding_count( to_shadow_page(vcpu->arch.mmu->root.hpa)); } - -void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) -{ - __kvm_mmu_new_pgd(vcpu, new_pgd, kvm_mmu_calc_root_page_role(vcpu)); -} EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); =20 static unsigned long get_cr3(struct kvm_vcpu *vcpu) @@ -4904,7 +4897,7 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, u= nsigned long cr0, new_role =3D kvm_calc_shadow_npt_root_page_role(vcpu, ®s); =20 shadow_mmu_init_context(vcpu, context, ®s, new_role); - __kvm_mmu_new_pgd(vcpu, nested_cr3, new_role.base); + kvm_mmu_new_pgd(vcpu, nested_cr3); } EXPORT_SYMBOL_GPL(kvm_init_shadow_npt_mmu); =20 @@ -4960,7 +4953,7 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, b= ool execonly, reset_ept_shadow_zero_bits_mask(context, execonly); } =20 - __kvm_mmu_new_pgd(vcpu, new_eptp, new_role.base); + kvm_mmu_new_pgd(vcpu, new_eptp); } EXPORT_SYMBOL_GPL(kvm_init_shadow_ept_mmu); =20 @@ -5045,20 +5038,6 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_GPL(kvm_init_mmu); =20 -static union kvm_mmu_page_role -kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu) -{ - struct kvm_mmu_role_regs regs =3D vcpu_to_role_regs(vcpu); - union kvm_mmu_role role; - - if (tdp_enabled) - role =3D kvm_calc_tdp_mmu_root_page_role(vcpu, ®s, true); - else - role =3D kvm_calc_shadow_mmu_root_page_role(vcpu, ®s, true); - - return role.base; -} - void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) { /* --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77CA0C433EF for ; Thu, 17 Feb 2022 21:04:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244973AbiBQVEd (ORCPT ); Thu, 17 Feb 2022 16:04:33 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243990AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C963F1017EF for ; Thu, 17 Feb 2022 13:03:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kbg+OqZri34BoKCLL1H4Wwzp/4I4YJYM0faw8mk1Jng=; b=icxNAzZRATbf/2YWGXsk3hwd/+HSKt1Bev+d2zDtQupZ34h2UWe7qlita67HfgZ/JClnd5 b8uf96FUYwD6zwK5bLQjo4wNkdYkDPJKoigltHFXcyPCQ78s4QQ02/l+/Pkr1irzwgFW/p bBh0+dW2sgeo42lPtbwhwX03ODyL/sM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-642-qvdlAysIMd-OoGmt1adXEg-1; Thu, 17 Feb 2022 16:03:48 -0500 X-MC-Unique: qvdlAysIMd-OoGmt1adXEg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5400C814245; Thu, 17 Feb 2022 21:03:47 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0492C6AB95; Thu, 17 Feb 2022 21:03:46 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 12/18] KVM: x86/mmu: clear MMIO cache when unloading the MMU Date: Thu, 17 Feb 2022 16:03:34 -0500 Message-Id: <20220217210340.312449-13-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For cleanliness, do not leave a stale GVA in the cache after all the roots = are cleared. In practice, kvm_mmu_load will go through kvm_mmu_sync_roots if paging is on, and will not use vcpu_match_mmio_gva at all if paging is off. However, leaving data in the cache might cause bugs in the future. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b01160716c6a..4e8e3e9530ca 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5111,6 +5111,7 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu) { __kvm_mmu_unload(vcpu->kvm, &vcpu->arch.root_mmu); __kvm_mmu_unload(vcpu->kvm, &vcpu->arch.guest_mmu); + vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY); } =20 static bool need_remote_flush(u64 old, u64 new) --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7429AC433EF for ; Thu, 17 Feb 2022 21:04:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244710AbiBQVEl (ORCPT ); Thu, 17 Feb 2022 16:04:41 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244007AbiBQVEF (ORCPT ); Thu, 17 Feb 2022 16:04:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id DF42411109A for ; Thu, 17 Feb 2022 13:03:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=31iQ22wsRKZ1WzBCnkFcq5Zu4wTNEFOLwesoT4L8SYI=; b=RQOT881FLxB5rvHuL5E58Z+WkRPVULiM7oV8VQU17wBEwD6LZzAqBQ+DQorA6HZjv5592n sWCDPE26ASXffy0BXnB5KXaSIe8TW1WFn0dpwckfpoZJLokKrW+McqI0shAmIbKfI9hSiu Ep0Zuf/RHxBGM/mWAdhPK8a60iBp0Ts= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-128-cb2CIOjkPdSQLWPCg5vjOw-1; Thu, 17 Feb 2022 16:03:48 -0500 X-MC-Unique: cb2CIOjkPdSQLWPCg5vjOw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BD001801AC5; Thu, 17 Feb 2022 21:03:47 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6E3936AB9D; Thu, 17 Feb 2022 21:03:47 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 13/18] KVM: x86: reset and reinitialize the MMU in __set_sregs_common Date: Thu, 17 Feb 2022 16:03:35 -0500 Message-Id: <20220217210340.312449-14-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Do a full unload of the MMU in KVM_SET_SREGS and KVM_SEST_REGS2, in preparation for not doing so in kvm_mmu_reset_context. There is no need to delay the reset until after the return, so do it directly in the __set_sregs_common function and remove the mmu_reset_needed output parameter. Signed-off-by: Paolo Bonzini Reviewed-by: Sean Christopherson --- arch/x86/kvm/x86.c | 32 +++++++++++++------------------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 6aefd7ac7039..f10878aa5b20 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10730,7 +10730,7 @@ static bool kvm_is_valid_sregs(struct kvm_vcpu *vcp= u, struct kvm_sregs *sregs) } =20 static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sre= gs, - int *mmu_reset_needed, bool update_pdptrs) + int update_pdptrs) { struct msr_data apic_base_msr; int idx; @@ -10755,29 +10755,31 @@ static int __set_sregs_common(struct kvm_vcpu *vc= pu, struct kvm_sregs *sregs, static_call(kvm_x86_set_gdt)(vcpu, &dt); =20 vcpu->arch.cr2 =3D sregs->cr2; - *mmu_reset_needed |=3D kvm_read_cr3(vcpu) !=3D sregs->cr3; + + if (vcpu->arch.efer !=3D sregs->efer || + kvm_read_cr0(vcpu) !=3D sregs->cr0 || + vcpu->arch.cr3 !=3D sregs->cr3 || !update_pdptrs || + kvm_read_cr4(vcpu) !=3D sregs->cr4) + kvm_mmu_unload(vcpu); + vcpu->arch.cr3 =3D sregs->cr3; kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); static_call_cond(kvm_x86_post_set_cr3)(vcpu, sregs->cr3); =20 kvm_set_cr8(vcpu, sregs->cr8); =20 - *mmu_reset_needed |=3D vcpu->arch.efer !=3D sregs->efer; static_call(kvm_x86_set_efer)(vcpu, sregs->efer); =20 - *mmu_reset_needed |=3D kvm_read_cr0(vcpu) !=3D sregs->cr0; static_call(kvm_x86_set_cr0)(vcpu, sregs->cr0); vcpu->arch.cr0 =3D sregs->cr0; =20 - *mmu_reset_needed |=3D kvm_read_cr4(vcpu) !=3D sregs->cr4; static_call(kvm_x86_set_cr4)(vcpu, sregs->cr4); =20 + kvm_init_mmu(vcpu); if (update_pdptrs) { idx =3D srcu_read_lock(&vcpu->kvm->srcu); - if (is_pae_paging(vcpu)) { + if (is_pae_paging(vcpu)) load_pdptrs(vcpu, kvm_read_cr3(vcpu)); - *mmu_reset_needed =3D 1; - } srcu_read_unlock(&vcpu->kvm->srcu, idx); } =20 @@ -10805,15 +10807,11 @@ static int __set_sregs_common(struct kvm_vcpu *vc= pu, struct kvm_sregs *sregs, static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) { int pending_vec, max_bits; - int mmu_reset_needed =3D 0; - int ret =3D __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); + int ret =3D __set_sregs_common(vcpu, sregs, true); =20 if (ret) return ret; =20 - if (mmu_reset_needed) - kvm_mmu_reset_context(vcpu); - max_bits =3D KVM_NR_INTERRUPTS; pending_vec =3D find_first_bit( (const unsigned long *)sregs->interrupt_bitmap, max_bits); @@ -10828,7 +10826,6 @@ static int __set_sregs(struct kvm_vcpu *vcpu, struc= t kvm_sregs *sregs) =20 static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) { - int mmu_reset_needed =3D 0; bool valid_pdptrs =3D sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; bool pae =3D (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && !(sregs2->efer & EFER_LMA); @@ -10840,8 +10837,7 @@ static int __set_sregs2(struct kvm_vcpu *vcpu, stru= ct kvm_sregs2 *sregs2) if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) return -EINVAL; =20 - ret =3D __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, - &mmu_reset_needed, !valid_pdptrs); + ret =3D __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, !valid_pdptr= s); if (ret) return ret; =20 @@ -10850,11 +10846,9 @@ static int __set_sregs2(struct kvm_vcpu *vcpu, str= uct kvm_sregs2 *sregs2) kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); =20 kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); - mmu_reset_needed =3D 1; vcpu->arch.pdptrs_from_userspace =3D true; + /* kvm_mmu_reload will be called on the next entry. */ } - if (mmu_reset_needed) - kvm_mmu_reset_context(vcpu); return 0; } =20 --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1F71C433EF for ; Thu, 17 Feb 2022 21:05:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240781AbiBQVFT (ORCPT ); Thu, 17 Feb 2022 16:05:19 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:38182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244181AbiBQVEI (ORCPT ); Thu, 17 Feb 2022 16:04:08 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id A89A5606C0 for ; Thu, 17 Feb 2022 13:03:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZgAtpSiEllHqyj81dALZNFNhaF5xNC1SM5IGKvMVmio=; b=Wq//u8JsbhX+I9EkFKQGt8xjXOmWZ5XxVOaXPTsInXTk/LMJQhJm1aJU3rpPWAvxawKwCM jwMAmhasLMw9npVVqlJSrrMuC2SU3eVw0hsNzcJfOaAEs0E/Tt1+Vj6TNvRVovvWtSq1x2 t7qX8zqMJd8ihTJTs7tmEBXNkn9rMG0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-121-9h1SdD80Nu6TLhpGroVYYg-1; Thu, 17 Feb 2022 16:03:49 -0500 X-MC-Unique: 9h1SdD80Nu6TLhpGroVYYg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 329692F4A; Thu, 17 Feb 2022 21:03:48 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id D7EAF6AB90; Thu, 17 Feb 2022 21:03:47 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 14/18] KVM: x86/mmu: avoid indirect call for get_cr3 Date: Thu, 17 Feb 2022 16:03:36 -0500 Message-Id: <20220217210340.312449-15-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Most of the time, calls to get_guest_pgd result in calling kvm_read_cr3 (the exception is only nested TDP). Hardcode the default instead of using the get_cr3 function, avoiding a retpoline if they are enabled. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky --- arch/x86/kvm/mmu.h | 13 +++++++++++++ arch/x86/kvm/mmu/mmu.c | 15 +++++---------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/x86.c | 2 +- 4 files changed, 20 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 1d0c1904d69a..1808d6814ddb 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -116,6 +116,19 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *v= cpu) vcpu->arch.mmu->shadow_root_level); } =20 +static inline gpa_t __kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu, struct = kvm_mmu *mmu) +{ + if (!mmu->get_guest_pgd) + return kvm_read_cr3(vcpu); + else + return mmu->get_guest_pgd(vcpu); +} + +static inline gpa_t kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu) +{ + return __kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu); +} + struct kvm_page_fault { /* arguments to kvm_mmu_do_page_fault. */ const gpa_t addr; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4e8e3e9530ca..d422d0d2adf8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3451,7 +3451,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vc= pu) unsigned i; int r; =20 - root_pgd =3D mmu->get_guest_pgd(vcpu); + root_pgd =3D kvm_mmu_get_guest_pgd(vcpu); root_gfn =3D root_pgd >> PAGE_SHIFT; =20 if (mmu_check_root(vcpu, root_gfn)) @@ -3881,7 +3881,7 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *= vcpu, gpa_t cr2_or_gpa, arch.token =3D (vcpu->arch.apf.id++ << 12) | vcpu->vcpu_id; arch.gfn =3D gfn; arch.direct_map =3D vcpu->arch.mmu->direct_map; - arch.cr3 =3D vcpu->arch.mmu->get_guest_pgd(vcpu); + arch.cr3 =3D kvm_mmu_get_guest_pgd(vcpu); =20 return kvm_setup_async_pf(vcpu, cr2_or_gpa, kvm_vcpu_gfn_to_hva(vcpu, gfn), &arch); @@ -4230,11 +4230,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t ne= w_pgd) } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); =20 -static unsigned long get_cr3(struct kvm_vcpu *vcpu) -{ - return kvm_read_cr3(vcpu); -} - static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) { @@ -4789,7 +4784,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) context->invlpg =3D NULL; context->shadow_root_level =3D kvm_mmu_get_tdp_level(vcpu); context->direct_map =3D true; - context->get_guest_pgd =3D get_cr3; + context->get_guest_pgd =3D NULL; /* use kvm_read_cr3 */ context->get_pdptr =3D kvm_pdptr_read; context->inject_page_fault =3D kvm_inject_page_fault; context->root_level =3D role_regs_to_root_level(®s); @@ -4964,7 +4959,7 @@ static void init_kvm_softmmu(struct kvm_vcpu *vcpu) =20 kvm_init_shadow_mmu(vcpu, ®s); =20 - context->get_guest_pgd =3D get_cr3; + context->get_guest_pgd =3D NULL; /* use kvm_read_cr3 */ context->get_pdptr =3D kvm_pdptr_read; context->inject_page_fault =3D kvm_inject_page_fault; } @@ -4996,7 +4991,7 @@ static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu) return; =20 g_context->mmu_role.as_u64 =3D new_role.as_u64; - g_context->get_guest_pgd =3D get_cr3; + g_context->get_guest_pgd =3D NULL; /* use kvm_read_cr3 */ g_context->get_pdptr =3D kvm_pdptr_read; g_context->inject_page_fault =3D kvm_inject_page_fault; g_context->root_level =3D new_role.base.level; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 346f3bad3cb9..1a85aba837b2 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -362,7 +362,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker= *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: walker->level =3D mmu->root_level; - pte =3D mmu->get_guest_pgd(vcpu); + pte =3D __kvm_mmu_get_guest_pgd(vcpu, mmu); have_ad =3D PT_HAVE_ACCESSED_DIRTY(mmu); =20 #if PTTYPE =3D=3D 64 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f10878aa5b20..adcee7c305ca 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -12161,7 +12161,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcp= u, struct kvm_async_pf *work) return; =20 if (!vcpu->arch.mmu->direct_map && - work->arch.cr3 !=3D vcpu->arch.mmu->get_guest_pgd(vcpu)) + work->arch.cr3 !=3D kvm_mmu_get_guest_pgd(vcpu)) return; =20 kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true); --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D727C433EF for ; Thu, 17 Feb 2022 21:04:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245243AbiBQVE5 (ORCPT ); Thu, 17 Feb 2022 16:04:57 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244074AbiBQVEG (ORCPT ); Thu, 17 Feb 2022 16:04:06 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id BD71C16041B for ; Thu, 17 Feb 2022 13:03:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R9Q0pLb8P4BlDqpXLzpYKuId39pvX37TO7tEfEhTs28=; b=Mt6hptAWDxEDt3K8nps3wBbYsfcKv58cHJrfuBrWq/cs9drOuKFU+HvbzHsSVSrRA9NMEJ 8XuIkIVtf5WEjzzRT+dTaI+TnEfDgXF6kO1XyIFWeK0F5ay7GvKzmMxk/i/tjkVvDkHwdu LGx/GfPnlTmMdYGTNmkILckBTCVSZjw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-WKyzD1ubNqOHwUg7aB2yxQ-1; Thu, 17 Feb 2022 16:03:49 -0500 X-MC-Unique: WKyzD1ubNqOHwUg7aB2yxQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9C18F2F4D; Thu, 17 Feb 2022 21:03:48 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D2E36AB90; Thu, 17 Feb 2022 21:03:48 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 15/18] KVM: x86/mmu: rename kvm_mmu_new_pgd, introduce variant that calls get_guest_pgd Date: Thu, 17 Feb 2022 16:03:37 -0500 Message-Id: <20220217210340.312449-16-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In the common case, the argument to kvm_mmu_new_pgd is already in vcpu->arch.cr3, but that does not work when the guest_mmu is in use. In that case, the root for L1 TDP tables needs to be retrieved via vendor code. Besides, kvm_mmu_new_pgd is a bad name: it can be used also when the role bits change, not just when the PGD changes. Kill two birds with one stone by renaming the old kvm_mmu_new_pgd to __kvm_mmu_update_root. The non-__ version, kvm_mmu_update_root, covers the common case, including nested TDP, by calling the get_guest_pgd callback to retrieve the desired PGD pointer. Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++---- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/x86.c | 2 +- 5 files changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 79f37ccc8726..319ac0918aa2 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1808,7 +1808,7 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva); void kvm_mmu_invalidate_gva(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gva_t gva, hpa_t root_hpa); void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long p= cid); -void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd); +void kvm_mmu_update_root(struct kvm_vcpu *vcpu); =20 void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, int tdp_max_root_level, int tdp_huge_page_level); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index d422d0d2adf8..c44b5114f947 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4189,7 +4189,7 @@ static bool fast_pgd_switch(struct kvm *kvm, struct k= vm_mmu *mmu, return cached_root_find_without_current(kvm, mmu, new_pgd, new_role); } =20 -void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) +static void __kvm_mmu_update_root(struct kvm_vcpu *vcpu, gpa_t new_pgd) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; union kvm_mmu_page_role new_role =3D mmu->mmu_role.base; @@ -4228,7 +4228,14 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t ne= w_pgd) __clear_sp_write_flooding_count( to_shadow_page(vcpu->arch.mmu->root.hpa)); } -EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); + +void kvm_mmu_update_root(struct kvm_vcpu *vcpu) +{ + gpa_t new_pgd =3D kvm_mmu_get_guest_pgd(vcpu); + + __kvm_mmu_update_root(vcpu, new_pgd); +} +EXPORT_SYMBOL_GPL(kvm_mmu_update_root); =20 static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) @@ -4892,7 +4899,7 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, u= nsigned long cr0, new_role =3D kvm_calc_shadow_npt_root_page_role(vcpu, ®s); =20 shadow_mmu_init_context(vcpu, context, ®s, new_role); - kvm_mmu_new_pgd(vcpu, nested_cr3); + __kvm_mmu_update_root(vcpu, nested_cr3); } EXPORT_SYMBOL_GPL(kvm_init_shadow_npt_mmu); =20 @@ -4948,7 +4955,7 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, b= ool execonly, reset_ept_shadow_zero_bits_mask(context, execonly); } =20 - kvm_mmu_new_pgd(vcpu, new_eptp); + __kvm_mmu_update_root(vcpu, new_eptp); } EXPORT_SYMBOL_GPL(kvm_init_shadow_ept_mmu); =20 diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 96bab464967f..2386fadae9ed 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -498,7 +498,7 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, u= nsigned long cr3, kvm_init_mmu(vcpu); =20 if (!nested_npt) - kvm_mmu_new_pgd(vcpu, cr3); + kvm_mmu_update_root(vcpu); =20 return 0; } diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 1dfe23963a9e..2dbd7a9ada84 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1133,7 +1133,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu,= unsigned long cr3, kvm_init_mmu(vcpu); =20 if (!nested_ept) - kvm_mmu_new_pgd(vcpu, cr3); + kvm_mmu_update_root(vcpu); =20 return 0; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index adcee7c305ca..9800c8883a48 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1189,7 +1189,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long = cr3) return 1; =20 if (cr3 !=3D kvm_read_cr3(vcpu)) - kvm_mmu_new_pgd(vcpu, cr3); + kvm_mmu_update_root(vcpu); =20 vcpu->arch.cr3 =3D cr3; kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F881C433F5 for ; Thu, 17 Feb 2022 21:04:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245304AbiBQVFA (ORCPT ); Thu, 17 Feb 2022 16:05:00 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244127AbiBQVEH (ORCPT ); Thu, 17 Feb 2022 16:04:07 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6305485643 for ; Thu, 17 Feb 2022 13:03:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=81CUj4DjODMnVGrRe0STdd54o6R4Y7aXMKzbSnkZtJE=; b=YSXJw+RvhOVyfm4o4u4Auq/vH1JmKATlBmAm+kZoZU3iOWZySidLHGYVfPN7gDmXf6PksF wzMHX7z1Ng5YzkG+mG2vdFH7mVHB8+Zm2/k8jvWf/ICXJIGiui34jpX2/LluvfvF3mnX3p cgPqtAzOh49Cauj+mrtuIBrR/u6d7xU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-56-XlrGCRQfPPKL-v2OHu4MxA-1; Thu, 17 Feb 2022 16:03:50 -0500 X-MC-Unique: XlrGCRQfPPKL-v2OHu4MxA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 12240801ADB; Thu, 17 Feb 2022 21:03:49 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6F4C6ABA4; Thu, 17 Feb 2022 21:03:48 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 16/18] KVM: x86: introduce KVM_REQ_MMU_UPDATE_ROOT Date: Thu, 17 Feb 2022 16:03:38 -0500 Message-Id: <20220217210340.312449-17-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Whenever KVM knows the page role flags have changed, it needs to drop the current MMU root and possibly load one from the prev_roots cache. Currently it is papering over some overly simplistic code by just dropping _all_ roots, so that the root will be reloaded by kvm_mmu_reload, but this has bad performance for the TDP MMU (which drops the whole of the page tables when freeing a root, without the performance safety net of a hash table). To do this, KVM needs to do a more kvm_mmu_update_root call from kvm_mmu_reset_context. Introduce a new request bit so that the call can be delayed until after a possible KVM_REQ_MMU_RELOAD, which would kill all hopes of finding a cached PGD. Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/x86.c | 13 +++++++++++-- 4 files changed, 14 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 319ac0918aa2..532cda546eb9 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -102,6 +102,7 @@ #define KVM_REQ_MSR_FILTER_CHANGED KVM_ARCH_REQ(29) #define KVM_REQ_UPDATE_CPU_DIRTY_LOGGING \ KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_MMU_UPDATE_ROOT KVM_ARCH_REQ(31) =20 #define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \ diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 2386fadae9ed..8e6e62d8df36 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -498,7 +498,7 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, u= nsigned long cr3, kvm_init_mmu(vcpu); =20 if (!nested_npt) - kvm_mmu_update_root(vcpu); + kvm_make_request(KVM_REQ_MMU_UPDATE_ROOT, vcpu); =20 return 0; } diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 2dbd7a9ada84..c3595bc0a02d 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1133,7 +1133,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu,= unsigned long cr3, kvm_init_mmu(vcpu); =20 if (!nested_ept) - kvm_mmu_update_root(vcpu); + kvm_make_request(KVM_REQ_MMU_UPDATE_ROOT, vcpu); =20 return 0; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9800c8883a48..9043548e6baf 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1189,7 +1189,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long = cr3) return 1; =20 if (cr3 !=3D kvm_read_cr3(vcpu)) - kvm_mmu_update_root(vcpu); + kvm_make_request(KVM_REQ_MMU_UPDATE_ROOT, vcpu); =20 vcpu->arch.cr3 =3D cr3; kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); @@ -9835,8 +9835,15 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } - if (kvm_check_request(KVM_REQ_MMU_RELOAD, vcpu)) + if (kvm_check_request(KVM_REQ_MMU_RELOAD, vcpu)) { kvm_mmu_unload(vcpu); + + /* + * Dropping all roots leaves no hope for loading a cached + * one. Let kvm_mmu_reload build a new one. + */ + kvm_clear_request(KVM_REQ_MMU_UPDATE_ROOT, vcpu); + } if (kvm_check_request(KVM_REQ_MIGRATE_TIMER, vcpu)) __kvm_migrate_timers(vcpu); if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu)) @@ -9848,6 +9855,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (unlikely(r)) goto out; } + if (kvm_check_request(KVM_REQ_MMU_UPDATE_ROOT, vcpu)) + kvm_mmu_update_root(vcpu); if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu)) kvm_mmu_sync_roots(vcpu); if (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu)) --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3656BC433EF for ; Thu, 17 Feb 2022 21:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245490AbiBQVFV (ORCPT ); Thu, 17 Feb 2022 16:05:21 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:38384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244255AbiBQVEL (ORCPT ); Thu, 17 Feb 2022 16:04:11 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id BF7A51017EF for ; Thu, 17 Feb 2022 13:03:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NX/p+3ovzAsh1RMwMjnRGHxJsp1FvXTx8d19l00fGLI=; b=WkHleUe/TbcMKTnGo0WPjIFPZBc7Peqd3z2aBcYxY1dCWJdmQFmi6VAwHqMfTBxqxysX8E PTgNbGICrmjt+954CWAwvGuBU6uOZnRwRta/ESRsAUWKDoJVKdB8Y3FER9HjiEYnWZWDGJ O+/mJL+cC8xo3VRm/X6VoiAa13kLqkk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-569-at2ZsQzxN1-8gbUmI3ucJw-1; Thu, 17 Feb 2022 16:03:50 -0500 X-MC-Unique: at2ZsQzxN1-8gbUmI3ucJw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7B661814245; Thu, 17 Feb 2022 21:03:49 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2C82F6AB90; Thu, 17 Feb 2022 21:03:49 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 17/18] KVM: x86: flush TLB separately from MMU reset Date: Thu, 17 Feb 2022 16:03:39 -0500 Message-Id: <20220217210340.312449-18-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For both CR0 and CR4, disassociate the TLB flush logic from the MMU role logic. Instead of relying on kvm_mmu_reset_context() being a superset of various TLB flushes (which is not necessarily going to be the case in the future), always call it if the role changes but also set the various TLB flush requests according to what is in the manual. Signed-off-by: Paolo Bonzini Reviewed-by: Maxim Levitsky Reviewed-by: Sean Christopherson --- arch/x86/kvm/x86.c | 58 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 18 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9043548e6baf..2b4663dfcd8d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -871,6 +871,13 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned = long old_cr0, unsigned lon if ((cr0 ^ old_cr0) & X86_CR0_PG) { kvm_clear_async_pf_completion_queue(vcpu); kvm_async_pf_hash_reset(vcpu); + + /* + * Clearing CR0.PG is defined to flush the TLB from the guest's + * perspective. + */ + if (!(cr0 & X86_CR0_PG)) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); } =20 if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS) @@ -1057,28 +1064,41 @@ EXPORT_SYMBOL_GPL(kvm_is_valid_cr4); =20 void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsign= ed long cr4) { + if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) + kvm_mmu_reset_context(vcpu); + /* - * If any role bit is changed, the MMU needs to be reset. - * - * If CR4.PCIDE is changed 1 -> 0, the guest TLB must be flushed. * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB * according to the SDM; however, stale prev_roots could be reused * incorrectly in the future after a MOV to CR3 with NOFLUSH=3D1, so we - * free them all. KVM_REQ_MMU_RELOAD is fit for the both cases; it - * is slow, but changing CR4.PCIDE is a rare case. - * - * If CR4.PGE is changed, the guest TLB must be flushed. - * - * Note: resetting MMU is a superset of KVM_REQ_MMU_RELOAD and - * KVM_REQ_MMU_RELOAD is a superset of KVM_REQ_TLB_FLUSH_GUEST, hence - * the usage of "else if". + * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST + * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, + * so fall through. */ - if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) - kvm_mmu_reset_context(vcpu); - else if ((cr4 ^ old_cr4) & X86_CR4_PCIDE) + if (!tdp_enabled && + (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) kvm_make_request(KVM_REQ_MMU_RELOAD, vcpu); - else if ((cr4 ^ old_cr4) & X86_CR4_PGE) - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + + /* + * The TLB has to be flushed for all PCIDs on: + * - CR4.PCIDE changed from 1 to 0 + * - any change to CR4.PGE + * + * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. + */ + if (((cr4 ^ old_cr4) & X86_CR4_PGE) || + (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + + /* + * The TLB has to be flushed for the current PCID on: + * - CR4.SMEP changed from 0 to 1 + * - any change to CR4.PAE + */ + else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || + ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) + kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); + } EXPORT_SYMBOL_GPL(kvm_post_set_cr4); =20 @@ -11323,15 +11343,17 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool i= nit_event) static_call(kvm_x86_update_exception_bitmap)(vcpu); =20 /* - * Reset the MMU context if paging was enabled prior to INIT (which is + * A TLB flush is needed if paging was enabled prior to INIT (which is * implied if CR0.PG=3D1 as CR0 will be '0' prior to RESET). Unlike the * standard CR0/CR4/EFER modification paths, only CR0.PG needs to be * checked because it is unconditionally cleared on INIT and all other * paging related bits are ignored if paging is disabled, i.e. CR0.WP, * CR4, and EFER changes are all irrelevant if CR0.PG was '0'. */ - if (old_cr0 & X86_CR0_PG) + if (old_cr0 & X86_CR0_PG) { + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); kvm_mmu_reset_context(vcpu); + } =20 /* * Intel's SDM states that all TLB entries are flushed on INIT. AMD's --=20 2.31.1 From nobody Sun Jun 28 00:10:22 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FC5DC433EF for ; Thu, 17 Feb 2022 21:04:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242205AbiBQVFD (ORCPT ); Thu, 17 Feb 2022 16:05:03 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:37894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244150AbiBQVEI (ORCPT ); Thu, 17 Feb 2022 16:04:08 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 31A9511109A for ; Thu, 17 Feb 2022 13:03:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645131832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mUS9+xPfstqadLkqJ3LrklUPZUjyiG4wtcGNmDoEiYw=; b=eX8kValO+Vn8GdzD8zYvYtbf3z/mWMWT35EjvrG92qY68fEgXUMIldxtw9HB9NT4HvoZlw aQxyJilmyi0k8BvGWmDYrp92xdLDrY7k6MF0NHBUhPYMryIRIpz/57a0UIE0cRMqWDxFLk twwy/4POj0yTYAxl+05z1pdeHsAo3AQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-311-gP4HMt8lMwyQIBWXz0nXDg-1; Thu, 17 Feb 2022 16:03:51 -0500 X-MC-Unique: gP4HMt8lMwyQIBWXz0nXDg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E48C01091DA0; Thu, 17 Feb 2022 21:03:49 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 95C486AB90; Thu, 17 Feb 2022 21:03:49 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com Subject: [PATCH v2 18/18] KVM: x86: do not unload MMU roots on all role changes Date: Thu, 17 Feb 2022 16:03:40 -0500 Message-Id: <20220217210340.312449-19-pbonzini@redhat.com> In-Reply-To: <20220217210340.312449-1-pbonzini@redhat.com> References: <20220217210340.312449-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" kvm_mmu_reset_context is called on all role changes and right now it calls kvm_mmu_unload. With the legacy MMU this is a relatively cheap operation; the previous PGDs remains in the hash table and is picked up immediately on the next page fault. With the TDP MMU, however, the roots are thrown away for good and a full rebuild of the page tables is necessary, which is many times more expensive. Fortunately, throwing away the roots is not necessary except when the manual says a TLB flush is required: - changing CR0.PG from 1 to 0 (because it flushes the TLB according to the x86 architecture specification) - changing CPUID (which changes the interpretation of page tables in ways not reflected by the role). - changing CR4.SMEP from 0 to 1 (not doing so actually breaks access.c) Except for these cases, once the MMU has updated the CPU/MMU roles and metadata it is enough to force-reload the current value of CR3. KVM will look up the cached roots for an entry with the right role and PGD, and only if the cache misses a new root will be created. Measuring with vmexit.flat from kvm-unit-tests shows the following improvement: TDP legacy shadow before 46754 5096 5150 after 4879 4875 5006 which is for very small page tables. The impact is however much larger when running as an L1 hypervisor, because the new page tables cause extra work for L0 to shadow them. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c44b5114f947..913cc7229bf4 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5043,8 +5043,8 @@ EXPORT_SYMBOL_GPL(kvm_init_mmu); void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) { /* - * Invalidate all MMU roles to force them to reinitialize as CPUID - * information is factored into reserved bit calculations. + * Invalidate all MMU roles and roots to force them to reinitialize, + * as CPUID information is factored into reserved bit calculations. * * Correctly handling multiple vCPU models with respect to paging and * physical address properties) in a single VM would require tracking @@ -5057,6 +5057,7 @@ void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) vcpu->arch.root_mmu.mmu_role.ext.valid =3D 0; vcpu->arch.guest_mmu.mmu_role.ext.valid =3D 0; vcpu->arch.nested_mmu.mmu_role.ext.valid =3D 0; + kvm_mmu_unload(vcpu); kvm_mmu_reset_context(vcpu); =20 /* @@ -5068,8 +5069,8 @@ void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) =20 void kvm_mmu_reset_context(struct kvm_vcpu *vcpu) { - kvm_mmu_unload(vcpu); kvm_init_mmu(vcpu); + kvm_make_request(KVM_REQ_MMU_UPDATE_ROOT, vcpu); } EXPORT_SYMBOL_GPL(kvm_mmu_reset_context); =20 --=20 2.31.1