From nobody Fri Jun 19 20:12:24 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D359DC433EF for ; Wed, 30 Mar 2022 13:21:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345908AbiC3NXQ (ORCPT ); Wed, 30 Mar 2022 09:23:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345943AbiC3NXA (ORCPT ); Wed, 30 Mar 2022 09:23:00 -0400 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29761488AB; Wed, 30 Mar 2022 06:21:14 -0700 (PDT) Received: by mail-pf1-x434.google.com with SMTP id z16so18682698pfh.3; Wed, 30 Mar 2022 06:21:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tD3lBLMBPhXyTx2fpA17/l0uIt12wjZgqNVRbnBMosg=; b=ANjPMfRYL5pCiGqbGnRfShBvM6miknLRiWivXIBBlgRMQs8eqpq7q0/qE1mYsRbj+4 f3AA4HjKTLVxD319VqhbA37UA0rQl9e7JYMOQS+0xAD9NH57qaSltvMCiZN2DQFMWj0U 6ST7YfM2hxMN/E77q3EXWoRUV+yRVfGgwDtycXXhtIxeEyDtWa0cIfn0DKTpVw7kz5Z9 6M+/DVE+iMj6MloHxoo1tSp0f1mSkYjgfwOf3NazDEIGR9+cdWB4xEhiPd+xAMVFsevu VU1UQRNkztNlxXJsqyljSJqFSUL7P6KCmLSnWzx+QYji1GaSUrkOOYmT3DWtRVKpv0On jhaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tD3lBLMBPhXyTx2fpA17/l0uIt12wjZgqNVRbnBMosg=; b=COCzK88AgUWxvRkdMEEbpofv0DsNpW67nSmQaLF7KlfdzChwUH33lX2PRYosvYkfyc hbXB8rbl+PTp6YHRkKQ1lUTVBV5uH5EArNLe6dzG2zpJOieQOguAVvSLZC61PhzemYdl /rA+t5x27hfVZRZuP35beYMP/ZpyTaFkDraxtn17rjEfJHVWfl6jeG3VYdXWXHYppRwc qj+qGhF8uwa75pnyEwLw9MpENZk8v/PaKqvoRovDAb40TA2ambCuggXmn/sIQAUbf35m ipaNeqJ5VEhIaIhkmL1ZY443E9VadZcuzt8f9xWj9sHvbG8RfkOPLGTso2nCmeQvs+St UbYA== X-Gm-Message-State: AOAM531HrcuWzi1kVkjbYboUtajDRH4HOlnCTKTsA05ZNNlr11J3VxKE x2D4PdaIaJQ6kkv0J8WTlL5EwZo36C0= X-Google-Smtp-Source: ABdhPJx3NfPW7EjuGG+gAR5EnXmejzeLt6pF1BUn7snRg9HaaoNwo+hjKLSdfaF6r7571Ymc4BeJQw== X-Received: by 2002:a63:2b8f:0:b0:398:a43d:dfd9 with SMTP id r137-20020a632b8f000000b00398a43ddfd9mr3329766pgr.478.1648646473251; Wed, 30 Mar 2022 06:21:13 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id j70-20020a638b49000000b003985b5ddaa1sm8685955pge.49.2022.03.30.06.21.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Mar 2022 06:21:12 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Paolo Bonzini , Sean Christopherson Cc: Lai Jiangshan , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" Subject: [RFC PATCH V3 1/4] KVM: X86: Add arguement gfn and role to kvm_mmu_alloc_page() Date: Wed, 30 Mar 2022 21:21:49 +0800 Message-Id: <20220330132152.4568-2-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220330132152.4568-1-jiangshanlai@gmail.com> References: <20220330132152.4568-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan kvm_mmu_alloc_page() will access to more bits of the role. Signed-off-by: Lai Jiangshan --- arch/x86/kvm/mmu/mmu.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1361eb4599b4..02eae110cbe1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1706,13 +1706,14 @@ static void drop_parent_pte(struct kvm_mmu_page *sp, mmu_spte_clear_no_track(parent_pte); } =20 -static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, int = direct) +static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, gfn_= t gfn, + union kvm_mmu_page_role role) { struct kvm_mmu_page *sp; =20 sp =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache); sp->spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); - if (!direct) + if (!role.direct) sp->gfns =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_gfn_array_cache); set_page_private(virt_to_page(sp->spt), (unsigned long)sp); =20 @@ -1724,6 +1725,8 @@ static struct kvm_mmu_page *kvm_mmu_alloc_page(struct= kvm_vcpu *vcpu, int direct sp->mmu_valid_gen =3D vcpu->kvm->arch.mmu_valid_gen; list_add(&sp->link, &vcpu->kvm->arch.active_mmu_pages); kvm_mod_used_mmu_pages(vcpu->kvm, +1); + sp->gfn =3D gfn; + sp->role =3D role; return sp; } =20 @@ -2107,10 +2110,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct = kvm_vcpu *vcpu, =20 ++vcpu->kvm->stat.mmu_cache_miss; =20 - sp =3D kvm_mmu_alloc_page(vcpu, direct); - - sp->gfn =3D gfn; - sp->role =3D role; + sp =3D kvm_mmu_alloc_page(vcpu, gfn, role); hlist_add_head(&sp->hash_link, sp_list); if (!direct) { account_shadowed(vcpu->kvm, sp); --=20 2.19.1.6.gb485710b From nobody Fri Jun 19 20:12:24 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C92CC433F5 for ; Wed, 30 Mar 2022 13:21:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345965AbiC3NX1 (ORCPT ); Wed, 30 Mar 2022 09:23:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345969AbiC3NXG (ORCPT ); Wed, 30 Mar 2022 09:23:06 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65ED1488AB; Wed, 30 Mar 2022 06:21:20 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id g9-20020a17090ace8900b001c7cce3c0aeso2086679pju.2; Wed, 30 Mar 2022 06:21:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=c+K9A9T1MT5Pu4zwSaAayliqn5gty12AfuascxX8vj4=; b=nEtLG65GgDBD2ThZgHCDsZSsWRfepUZEr5TeRQW+HtYt93+fIBsLwgJEoJ+SrOVbsx I3DpuEUxZPrVmRiHyTmTZ3k2yh+BkJ57O3gLmdc0LtMWpE9Q+uUco6nio+wdSFHbMBHt D4/hR/62epIpkTU7rrMrS1WhkTGD8MYRXv0IMGIsd53JvKVUXkF4fYMMGcWEevaH7M/8 k306OdR8OwTQs9qkiHnjwTXmYDuJenGzIvsOxBfB+f+o+b4moEVER02SmAwgPju/EP1c NcKiUJ24zRt6c/SZMteODK5UJZKGBwpbApoi1ZhFEf3FMAYSA5bpdNr1DOuB1ZSTy9aX GjmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=c+K9A9T1MT5Pu4zwSaAayliqn5gty12AfuascxX8vj4=; b=pG6gjLl4bhH01L6PqU6C16WXi9ivhyXBrAuDkBfv7EluGg4F1Mey+/SQB0kfodow56 cXohSU3tsUF7rawrIRydAHtKy9SWMDqn7wLYQiud5Qb6LcldQqCcs62q9Ky4c9NWZvMQ hD/Qkk45qAFxjFuy2Q/QjI0yPHGITt18/IRvwNX9Js6xpaZpd6sKTGyIfgSpnC80rb4m XH8OAHMwAi9nPtnaGIgEEefZU3y5evfS7Jf5cDMVHHgO8pk+CMt6ekW+y+/UdbO8S5Rz GHxw3afy+G5pCSFeuW59fK8uR9c7IWJgQw4szjo61sXsQpfpuBJgOSQ5cv6pYi/NsONS I/VQ== X-Gm-Message-State: AOAM532lUfXpoB+UJc9n0bkOrqk0bLkuoDigiCD2fMW7k6U/9OXC6K0A 7mwkFeboGqFNmw8rg/4cU70uSNfwlEc= X-Google-Smtp-Source: ABdhPJyH4MR2bW42j1R3tqij52APbZ6yL1b9+zWFJQj/5rP4QGk3QtQkM5+iX50/sGAz6Uus6QIn4Q== X-Received: by 2002:a17:90a:1548:b0:1c9:8181:9e70 with SMTP id y8-20020a17090a154800b001c981819e70mr5089434pja.78.1648646479563; Wed, 30 Mar 2022 06:21:19 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id x29-20020aa79a5d000000b004f0ef1822d3sm23525250pfj.128.2022.03.30.06.21.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Mar 2022 06:21:19 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Paolo Bonzini , Sean Christopherson Cc: Lai Jiangshan , Jonathan Corbet , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-doc@vger.kernel.org Subject: [RFC PATCH V3 2/4] KVM: X86: Introduce role.glevel for level expanded pagetable Date: Wed, 30 Mar 2022 21:21:50 +0800 Message-Id: <20220330132152.4568-3-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220330132152.4568-1-jiangshanlai@gmail.com> References: <20220330132152.4568-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Level expansion occurs when mmu->shadow_root_level > mmu->root_level. There are several cases that can cuase level expansion: shadow mmu (shadow paging for 32 bit guest): case1: gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D0 shadow nested NPT (for 32bit L1 hypervisor): case2: gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D0,hEFER_LMA=3D0 case3: gCR0_PG=3D1,gEFER_LMA=3D0,hEFER_LMA=3D1 shadow nested NPT (for 64bit L1 hypervisor): case4: gEFER_LMA=3D1,gCR4_LA57=3D0,hEFER_LMA=3D1,hCR4_LA57=3D1 When level expansion occurs (32bit guest, case1-3), special roots are often used. But case4 is not using special roots. It uses shadow page without fully aware of the specialty. It might work accidentally: 1) The root_page (root_sp->spt) is allocated with level =3D 5, and root_sp->spt[0] is allocated with the same gfn and the same role except role.level =3D 4. Luckly that they are different shadow pages. 2) FNAME(walk_addr_generic) sets walker->table_gfn[4] and walker->pt_access[4], which are normally unused when mmu->shadow_root_level =3D=3D mmu->root_level =3D=3D 4, so that FNAME(fetch) can use them to allocate shadow page for root_sp->spt[0] and link them when shadow_root_level =3D=3D 5. But it has problems. If the guest switches from gCR4_LA57=3D0 to gCR4_LA57=3D1 (or vice verse) and uses the same gfn as the root for the nNPT before and after switching gCR4_LA57. The host (hCR4_LA57=3D1) wold use the same root_sp for guest even guest switches gCR4_LA57. The guest will see unexpected page mapped and L2 can hurts L1. It is lucky the the problem can't hurt L0. The root_sp should be like role.direct=3D1 sometimes: its contents are not backed by gptes, root_sp->gfns is meaningless. For a normal high level sp, sp->gfns is often unused and kept zero, but it could be relevant and meaningful when sp->gfns is used because they are back by concret gptes. For expanded root_sp described before, root_sp is just a portal to contribute root_sp->spt[0], and root_sp should not have root_sp->gfns and root_sp->spt[0] should not be dropped if gpte[0] of the root gfn is changed. This patch adds role.glevel to address the two problems. With the new role.glevel, passthrough sp can be created for expanded shadow pagetable: 0 < role.glevel < role.level. An alternative way to fix the problem of case4 is that: also using the special root pml5_root for it. But it would required to change many other places because it is assumption that special roots is only used for 32bit guests. This patch also paves the way to use passthrough shadow page for case1-3, but that requires the special handling or PAE paging, so the extensive usage of it is in later patches. Signed-off-by: Lai Jiangshan --- Documentation/virt/kvm/mmu.rst | 7 +++++++ arch/x86/include/asm/kvm_host.h | 5 +++-- arch/x86/kvm/mmu/mmu.c | 21 +++++++++++++++++---- arch/x86/kvm/mmu/paging_tmpl.h | 1 + 4 files changed, 28 insertions(+), 6 deletions(-) diff --git a/Documentation/virt/kvm/mmu.rst b/Documentation/virt/kvm/mmu.rst index 5b1ebad24c77..dee0e96d694a 100644 --- a/Documentation/virt/kvm/mmu.rst +++ b/Documentation/virt/kvm/mmu.rst @@ -202,6 +202,13 @@ Shadow pages contain the following information: Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D bits before Haswell; shadow EPT page tables also cannot use A/D bits if the L1 hypervisor does not enable them. + role.glevel: + The level in guest pagetable if the sp is indirect. Is 0 if the sp + is direct without corresponding guest pagetable, like TDP or !CR0.PG. + When role.level > guest paging level, indirect sp is created on the + top with role.glevel =3D guest paging level and acks as passthrough sp + and its contents are specially installed rather than the translations + of the corresponding guest pagetable. gfn: Either the guest page table containing the translations shadowed by th= is page, or the base page frame for linear translations. See role.direct. diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 9694dd5e6ccc..67e1bccaf472 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -314,7 +314,7 @@ struct kvm_kernel_irq_routing_entry; * cr0_wp=3D0, therefore these three bits only give rise to 5 possibil= ities. * * Therefore, the maximum number of possible upper-level shadow pages for a - * single gfn is a bit less than 2^13. + * single gfn is a bit less than 2^15. */ union kvm_mmu_page_role { u32 word; @@ -331,7 +331,8 @@ union kvm_mmu_page_role { unsigned smap_andnot_wp:1; unsigned ad_disabled:1; unsigned guest_mode:1; - unsigned :6; + unsigned glevel:4; + unsigned :2; =20 /* * This is left at the top of the word so that diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 02eae110cbe1..d53037df8177 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -737,8 +737,12 @@ static void mmu_free_pte_list_desc(struct pte_list_des= c *pte_list_desc) =20 static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index) { - if (!sp->role.direct) + if (!sp->role.direct) { + if (unlikely(sp->role.glevel < sp->role.level)) + return sp->gfn; + return sp->gfns[index]; + } =20 return sp->gfn + (index << ((sp->role.level - 1) * PT64_LEVEL_BITS)); } @@ -746,6 +750,11 @@ static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page = *sp, int index) static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t= gfn) { if (!sp->role.direct) { + if (unlikely(sp->role.glevel < sp->role.level)) { + WARN_ON_ONCE(gfn !=3D sp->gfn); + return; + } + sp->gfns[index] =3D gfn; return; } @@ -1674,8 +1683,7 @@ static void kvm_mmu_free_page(struct kvm_mmu_page *sp) hlist_del(&sp->hash_link); list_del(&sp->link); free_page((unsigned long)sp->spt); - if (!sp->role.direct) - free_page((unsigned long)sp->gfns); + free_page((unsigned long)sp->gfns); kmem_cache_free(mmu_page_header_cache, sp); } =20 @@ -1713,7 +1721,7 @@ static struct kvm_mmu_page *kvm_mmu_alloc_page(struct= kvm_vcpu *vcpu, gfn_t gfn, =20 sp =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache); sp->spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); - if (!role.direct) + if (role.glevel =3D=3D role.level) sp->gfns =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_gfn_array_cache); set_page_private(virt_to_page(sp->spt), (unsigned long)sp); =20 @@ -2054,6 +2062,8 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct k= vm_vcpu *vcpu, quadrant &=3D (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1; role.quadrant =3D quadrant; } + if (level < role.glevel) + role.glevel =3D level; =20 sp_list =3D &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)]; for_each_valid_sp(vcpu->kvm, sp, sp_list) { @@ -4817,6 +4827,7 @@ kvm_calc_shadow_root_page_role_common(struct kvm_vcpu= *vcpu, role.base.smep_andnot_wp =3D role.ext.cr4_smep && !____is_cr0_wp(regs); role.base.smap_andnot_wp =3D role.ext.cr4_smap && !____is_cr0_wp(regs); role.base.has_4_byte_gpte =3D ____is_cr0_pg(regs) && !____is_cr4_pae(regs= ); + role.base.glevel =3D role_regs_to_root_level(regs); =20 return role; } @@ -5312,6 +5323,8 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, = gpa_t gpa, ++vcpu->kvm->stat.mmu_pte_write; =20 for_each_gfn_indirect_valid_sp(vcpu->kvm, sp, gfn) { + if (sp->role.glevel < sp->role.level) + continue; if (detect_write_misaligned(sp, gpa, bytes) || detect_write_flooding(sp)) { kvm_mmu_prepare_zap_page(vcpu->kvm, sp, &invalid_list); diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 8621188b46df..67489a060eba 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -1042,6 +1042,7 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, st= ruct kvm_mmu_page *sp) .level =3D 0xf, .access =3D 0x7, .quadrant =3D 0x3, + .glevel =3D 0xf, }; =20 /* --=20 2.19.1.6.gb485710b From nobody Fri Jun 19 20:12:24 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56896C433F5 for ; Wed, 30 Mar 2022 13:21:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345986AbiC3NXf (ORCPT ); Wed, 30 Mar 2022 09:23:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345873AbiC3NXM (ORCPT ); Wed, 30 Mar 2022 09:23:12 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E28848E6C; Wed, 30 Mar 2022 06:21:26 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id g9-20020a17090ace8900b001c7cce3c0aeso2087025pju.2; Wed, 30 Mar 2022 06:21:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pDMDEFNEn7HVY/Qsj+Lf/FLfZqskxaYIp4y/ijMrqEY=; b=QbEH5L1tWKkSjfeM1+BIBle22GbgK8duSpZvdLvHwmAXU5lip66cJ07vXIftjO9K1Y 8NeM/0egdKbmJtYC3Nh9JDBs8M33Fv+T3DTLZGPw9G4ZrvS884JEgvZ34K8Lh4Cv4Y84 QRITD3LlZ5XW2O11yG3/YhDaesW8vLVqUZpu6NmlfobMlBzXzfk5ZGwlKSlQHUsKWNpR Vhyh6eT8CWg9jo1MbHx87k4QAX1Kmb4V3idXTTu601nVOzIV/Ns5VUBB+pDrzo5SnDpA FZALzFurRyFVYeVGmSQXYxMzspLKNjBGqccFsOHQ4JuZiIyx0WmZWR1x3NJmSyF33IMN 67vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pDMDEFNEn7HVY/Qsj+Lf/FLfZqskxaYIp4y/ijMrqEY=; b=llsfgFgj+r3zswQRnFNz6snOahSFnrA4DJc90aTrer02D5SGpl/RluIRJiSJL2az21 dvS8G4kRogv5qpG4H/r5U1LeJHjMzn48tuLWnVIJEjcVYb9gSv8cnFcCPTUFPssXMVNT 60R/8I2GCNWuD30YKQCoM9QHgz+9HBV/SjOq7I5x5gsY9TEMcQuTMG9UUoSIiaaS7drq d7lEAwjKopF+rT6B+PcIfXbCJZNXBTz3Jvlpxg/8lAWZ637QFoTZMg1kfTUN2h/GhxAV /Z4egCB8KWeYlyr6mg4jpe6O9aIvk0UYWuXzC4syrK+J4dIbxCtwiiYZkkjm/Y75fnug 3oyg== X-Gm-Message-State: AOAM530AfKs5ls3tQ2ji3MDHHoJCdMosLgdXZ++ClwmQqvgCQP9vMPu2 VJlBC+fOGj7PQq/0eqkk3uc8ZaDjc28= X-Google-Smtp-Source: ABdhPJxQjrVBYb/kYTyNsNGKulkpYWM/GX76kZE9EShtlSwMtwmnuWW+HtAiBX4Apdri2/J1W5uqMg== X-Received: by 2002:a17:902:c745:b0:153:b0e:8586 with SMTP id q5-20020a170902c74500b001530b0e8586mr34828057plq.9.1648646485418; Wed, 30 Mar 2022 06:21:25 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id l6-20020a17090a660600b001c985b0cb53sm6406221pjj.26.2022.03.30.06.21.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Mar 2022 06:21:25 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Paolo Bonzini , Sean Christopherson Cc: Lai Jiangshan , Jonathan Corbet , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-doc@vger.kernel.org Subject: [RFC PATCH V3 3/4] KVM: X86: Alloc role.pae_root shadow page Date: Wed, 30 Mar 2022 21:21:51 +0800 Message-Id: <20220330132152.4568-4-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220330132152.4568-1-jiangshanlai@gmail.com> References: <20220330132152.4568-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Currently pae_root is special root page, this patch adds facility to allow using kvm_mmu_get_page() to allocate pae_root shadow page. When kvm_mmu_get_page() is called for role.level =3D=3D PT32E_ROOT_LEVEL and vcpu->arch.mmu->shadow_root_level =3D=3D PT32E_ROOT_LEVEL, it will get a PAE root pagetable and set role.pae_root=3D1 for freeing. The role.pae_root bit is needed in the page role because: o PAE roots must be allocated below 4gb (for kvm_mmu_get_page()) o PAE roots can not be encrypted (for kvm_mmu_get_page()) o Must be re-encrypted when freeing (for kvm_mmu_free_page()) o PAE root's PDPTE is special (for link_shadow_page()) o Not share the decrypted low-address pagetable with non-PAE-root ones or vice verse. (for kvm_mmu_get_page(), the crucial reason) Both role.pae_root in link_shadow_page() and in kvm_mmu_get_page() can be possible changed to use shadow_root_level and role.level instead. But in kvm_mmu_free_page(), it can't use vcpu->arch.mmu->shadow_root_level. PAE roots must be allocated below 4gb (CR3 has only 32 bits). So a cache is introduced (mmu_pae_root_cache). No functionality changed since this code is not activated because when vcpu->arch.mmu->shadow_root_level =3D=3D PT32E_ROOT_LEVEL, kvm_mmu_get_page= () is only called for level =3D=3D 1 or 2 now. Signed-off-by: Lai Jiangshan --- Documentation/virt/kvm/mmu.rst | 2 + arch/x86/include/asm/kvm_host.h | 9 +++- arch/x86/kvm/mmu/mmu.c | 78 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/paging_tmpl.h | 1 + 4 files changed, 86 insertions(+), 4 deletions(-) diff --git a/Documentation/virt/kvm/mmu.rst b/Documentation/virt/kvm/mmu.rst index dee0e96d694a..800f1eba55b3 100644 --- a/Documentation/virt/kvm/mmu.rst +++ b/Documentation/virt/kvm/mmu.rst @@ -209,6 +209,8 @@ Shadow pages contain the following information: top with role.glevel =3D guest paging level and acks as passthrough sp and its contents are specially installed rather than the translations of the corresponding guest pagetable. + role.pae_root: + Is 1 if it is a PAE root. gfn: Either the guest page table containing the translations shadowed by th= is page, or the base page frame for linear translations. See role.direct. diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 67e1bccaf472..658c493e7617 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -313,6 +313,11 @@ struct kvm_kernel_irq_routing_entry; * - on top of this, smep_andnot_wp and smap_andnot_wp are only set if * cr0_wp=3D0, therefore these three bits only give rise to 5 possibil= ities. * + * - pae_root can only be set when level=3D3, so combinations for level = and + * pae_root can be seen as 2/3/3-page_root/4/5, a.k.a 5 possibilities. + * Combined with cr0_wp, smep_andnot_wp and smap_andnot_wp, it will be + * 5X5 =3D 25 < 2^5. + * * Therefore, the maximum number of possible upper-level shadow pages for a * single gfn is a bit less than 2^15. */ @@ -332,7 +337,8 @@ union kvm_mmu_page_role { unsigned ad_disabled:1; unsigned guest_mode:1; unsigned glevel:4; - unsigned :2; + unsigned pae_root:1; + unsigned :1; =20 /* * This is left at the top of the word so that @@ -699,6 +705,7 @@ struct kvm_vcpu_arch { struct kvm_mmu_memory_cache mmu_shadow_page_cache; struct kvm_mmu_memory_cache mmu_gfn_array_cache; struct kvm_mmu_memory_cache mmu_page_header_cache; + void *mmu_pae_root_cache; =20 /* * QEMU userspace and the guest each have their own FPU state. diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index d53037df8177..81ccaa7c1165 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -694,6 +694,35 @@ static void walk_shadow_page_lockless_end(struct kvm_v= cpu *vcpu) } } =20 +static int mmu_topup_pae_root_cache(struct kvm_vcpu *vcpu) +{ + struct page *page; + + if (vcpu->arch.mmu->shadow_root_level !=3D PT32E_ROOT_LEVEL) + return 0; + if (vcpu->arch.mmu_pae_root_cache) + return 0; + + page =3D alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_DMA32); + if (!page) + return -ENOMEM; + vcpu->arch.mmu_pae_root_cache =3D page_address(page); + + /* + * CR3 is only 32 bits when PAE paging is used, thus it's impossible to + * get the CPU to treat the PDPTEs as encrypted. Decrypt the page so + * that KVM's writes and the CPU's reads get along. Note, this is + * only necessary when using shadow paging, as 64-bit NPT can get at + * the C-bit even when shadowing 32-bit NPT, and SME isn't supported + * by 32-bit kernels (when KVM itself uses 32-bit NPT). + */ + if (!tdp_enabled) + set_memory_decrypted((unsigned long)vcpu->arch.mmu_pae_root_cache, 1); + else + WARN_ON_ONCE(shadow_me_mask); + return 0; +} + static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indir= ect) { int r; @@ -705,6 +734,9 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcp= u, bool maybe_indirect) return r; r =3D kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache, PT64_ROOT_MAX_LEVEL); + if (r) + return r; + r =3D mmu_topup_pae_root_cache(vcpu); if (r) return r; if (maybe_indirect) { @@ -717,12 +749,23 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *v= cpu, bool maybe_indirect) PT64_ROOT_MAX_LEVEL); } =20 +static void mmu_free_pae_root(void *root_pt) +{ + if (!tdp_enabled) + set_memory_encrypted((unsigned long)root_pt, 1); + free_page((unsigned long)root_pt); +} + static void mmu_free_memory_caches(struct kvm_vcpu *vcpu) { kvm_mmu_free_memory_cache(&vcpu->arch.mmu_pte_list_desc_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadow_page_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_gfn_array_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache); + if (vcpu->arch.mmu_pae_root_cache) { + mmu_free_pae_root(vcpu->arch.mmu_pae_root_cache); + vcpu->arch.mmu_pae_root_cache =3D NULL; + } } =20 static struct pte_list_desc *mmu_alloc_pte_list_desc(struct kvm_vcpu *vcpu) @@ -1682,7 +1725,10 @@ static void kvm_mmu_free_page(struct kvm_mmu_page *s= p) MMU_WARN_ON(!is_empty_shadow_page(sp->spt)); hlist_del(&sp->hash_link); list_del(&sp->link); - free_page((unsigned long)sp->spt); + if (sp->role.pae_root) + mmu_free_pae_root(sp->spt); + else + free_page((unsigned long)sp->spt); free_page((unsigned long)sp->gfns); kmem_cache_free(mmu_page_header_cache, sp); } @@ -1720,7 +1766,12 @@ static struct kvm_mmu_page *kvm_mmu_alloc_page(struc= t kvm_vcpu *vcpu, gfn_t gfn, struct kvm_mmu_page *sp; =20 sp =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache); - sp->spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); + if (!role.pae_root) { + sp->spt =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache= ); + } else { + sp->spt =3D vcpu->arch.mmu_pae_root_cache; + vcpu->arch.mmu_pae_root_cache =3D NULL; + } if (role.glevel =3D=3D role.level) sp->gfns =3D kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_gfn_array_cache); set_page_private(virt_to_page(sp->spt), (unsigned long)sp); @@ -2064,6 +2115,8 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct k= vm_vcpu *vcpu, } if (level < role.glevel) role.glevel =3D level; + if (level !=3D PT32E_ROOT_LEVEL) + role.pae_root =3D 0; =20 sp_list =3D &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)]; for_each_valid_sp(vcpu->kvm, sp, sp_list) { @@ -2199,14 +2252,26 @@ static void shadow_walk_next(struct kvm_shadow_walk= _iterator *iterator) __shadow_walk_next(iterator, *iterator->sptep); } =20 +static u64 make_pae_pdpte(u64 *child_pt) +{ + /* The only ignore bits in PDPTE are 11:9. */ + BUILD_BUG_ON(!(GENMASK(11,9) & SPTE_MMU_PRESENT_MASK)); + return __pa(child_pt) | PT_PRESENT_MASK | SPTE_MMU_PRESENT_MASK | + shadow_me_mask; +} + static void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, struct kvm_mmu_page *sp) { + struct kvm_mmu_page *parent_sp =3D sptep_to_sp(sptep); u64 spte; =20 BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK !=3D PT_WRITABLE_MASK); =20 - spte =3D make_nonleaf_spte(sp->spt, sp_ad_disabled(sp)); + if (!parent_sp->role.pae_root) + spte =3D make_nonleaf_spte(sp->spt, sp_ad_disabled(sp)); + else + spte =3D make_pae_pdpte(sp->spt); =20 mmu_spte_set(sptep, spte); =20 @@ -4782,6 +4847,8 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, role.base.level =3D kvm_mmu_get_tdp_level(vcpu); role.base.direct =3D true; role.base.has_4_byte_gpte =3D false; + if (role.base.level =3D=3D PT32E_ROOT_LEVEL) + role.base.pae_root =3D 1; =20 return role; } @@ -4848,6 +4915,9 @@ kvm_calc_shadow_mmu_root_page_role(struct kvm_vcpu *v= cpu, else role.base.level =3D PT64_ROOT_4LEVEL; =20 + if (role.base.level =3D=3D PT32E_ROOT_LEVEL) + role.base.pae_root =3D 1; + return role; } =20 @@ -4893,6 +4963,8 @@ kvm_calc_shadow_npt_root_page_role(struct kvm_vcpu *v= cpu, =20 role.base.direct =3D false; role.base.level =3D kvm_mmu_get_tdp_level(vcpu); + if (role.base.level =3D=3D PT32E_ROOT_LEVEL) + role.base.pae_root =3D 1; =20 return role; } diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 67489a060eba..1015f33e0758 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -1043,6 +1043,7 @@ static int FNAME(sync_page)(struct kvm_vcpu *vcpu, st= ruct kvm_mmu_page *sp) .access =3D 0x7, .quadrant =3D 0x3, .glevel =3D 0xf, + .pae_root =3D 0x1, }; =20 /* --=20 2.19.1.6.gb485710b From nobody Fri Jun 19 20:12:24 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E76BAC433EF for ; Wed, 30 Mar 2022 13:22:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346011AbiC3NXs (ORCPT ); Wed, 30 Mar 2022 09:23:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345945AbiC3NXX (ORCPT ); Wed, 30 Mar 2022 09:23:23 -0400 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6033849258; Wed, 30 Mar 2022 06:21:32 -0700 (PDT) Received: by mail-pf1-x42a.google.com with SMTP id t2so18666674pfj.10; Wed, 30 Mar 2022 06:21:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8nw22u+xHMe9fEPa90YlblMBN0qhe5TLiCzKqkUTaPk=; b=DGTDb2piuMNSX1ocTvi3AgZ/zh/nZgXbI1NmaBMNFkyNFHFe87oL8m3+25F8Ikpox+ rSuMgxRb1BqcIavmW45SICWb0Y1by0mxVPKMi0VAvI6KSc6ta3UNl0GnDHICTMVrYKTv OREwfevpS0zD9+v3JFN9mmRRldzMXToModiCtR7/W7/Uq4ykBL5dPwFecdRbjMnhl07T BTH1qqJvc/E9ZJ2RA3i/ewSADD+hTaSccvnVAJTQZtzaLx5sAodP2YNMWbwYm8iIY0IH AXolyffPAd4jxSFZKP5ph9350Z6akdlSZ/JmdVaCrlekOQkffAfeomkCg3zz9/FfSfrf sryw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8nw22u+xHMe9fEPa90YlblMBN0qhe5TLiCzKqkUTaPk=; b=WUjXAMcJydBgBd5fmVMVBUovw+cc3IC4tkmjs/mPZxsQA5J9DJGjnUFlUX/fyNjxDt adIEM6byGi3TGY38eZFe1aPS5C6BI4K73uFC1+JyPFwCWDAwWvQATvrAcfvstPC1WUOx OVktmaF7Q2VT4nZKuOzI6U1iG79twJaUVQvTnrcU8We8ej2OJP2JiTNwC/9jthtxL3mF seYdi7FU1A96Vj5SZstT45AestRg8AryorSAvxIPtJNXhzUHH8zZhzidGyCGGbEvkqs9 mR7dwjDS4NtokAi/z3K+x7STi9oMwSj3fOQ4IAzXuDKSRGCuotMZg4F5TM0l2X22l7hE gjOg== X-Gm-Message-State: AOAM533GfB/eI0TvWRJcJQqA7lRTzDza6PQHwD6yCHRjNavL1tcES7hW NrbeWI1D+KHB/e7mFbzLCNoe6IV7UUo= X-Google-Smtp-Source: ABdhPJz7oQNtfn+VtdgpdE1iMMIo/rzxa/gB6z+S92MZdUmP63OJ/qYU2hRbag2b+u4RaQm68ZOUgg== X-Received: by 2002:a05:6a00:2887:b0:4fa:e10c:7ca with SMTP id ch7-20020a056a00288700b004fae10c07camr32963153pfb.9.1648646491590; Wed, 30 Mar 2022 06:21:31 -0700 (PDT) Received: from localhost ([198.11.178.15]) by smtp.gmail.com with ESMTPSA id h6-20020a056a00218600b004f65315bb37sm25254776pfi.13.2022.03.30.06.21.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 30 Mar 2022 06:21:31 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Paolo Bonzini , Sean Christopherson Cc: Lai Jiangshan , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" Subject: [RFC PATCH V3 4/4] KVM: X86: Use passthrough and pae_root shadow page for 32bit guests Date: Wed, 30 Mar 2022 21:21:52 +0800 Message-Id: <20220330132152.4568-5-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220330132152.4568-1-jiangshanlai@gmail.com> References: <20220330132152.4568-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Use role.pae_root =3D 1 for shadow_root_level =3D=3D 3 no matter if it is shadow MMU or not. When it is shadow MMU, level expansion might occur and use passthrough sp (0 < role.glevel < role.level) for expanded shadow pagetable. And remove the unneeded special roots. Now all the root pages and pagetable pointed by a present spte in kvm_mmu are backed by struct kvm_mmu_page, and to_shadow_page() is guaranteed to be not NULL. shadow_walk() and the intialization of shadow page are much simplified since there is not special roots. Affect cases: direct mmu (nonpaping for 32 bit guest): gCR0_PG=3D0 (pae_root=3D1) shadow mmu (shadow paping for 32 bit guest): gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D0 (pae_root=3D1,passthrough) gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D1 (pae_root=3D1,no passthrough) direct mmu (NPT for 32bit host): hEFER_LMA=3D0 (pae_root=3D1) shadow nested NPT (for 32bit L1 hypervisor): gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D0,hEFER_LMA=3D0 (pae_root=3D1,passthrough) gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D1,hEFER_LMA=3D0 (pae_root=3D1,no passthrough) gCR0_PG=3D1,gEFER_LMA=3D0,gCR4_PSE=3D{0|1},hEFER_LMA=3D1,hCR4_LA57=3D{0|1} (pae_root=3D0,passthrough) (default_pae_pdpte is not used even guest is using PAE paging) Shadow nested NPT for 64bit L1 hypervisor has been already handled: gEFER_LMA=3D1,gCR4_LA57=3D0,hEFER_LMA=3D1,hCR4_LA57=3D1 (pae_root=3D0,passthrough) FNAME(walk_addr_generic) adds initialization code for shadow nested NPT for 32bit L1 hypervisor when the level increment might be more than one, for example, 2->4, 2->5, 3->5. After this patch, the PAE Page-Directory-Pointer-Table is also write protected (including NPT's). Signed-off-by: Lai Jiangshan --- arch/x86/include/asm/kvm_host.h | 4 - arch/x86/kvm/mmu/mmu.c | 290 +------------------------------- arch/x86/kvm/mmu/paging_tmpl.h | 13 +- 3 files changed, 20 insertions(+), 287 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 658c493e7617..26aab4418844 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -467,10 +467,6 @@ struct kvm_mmu { */ u32 pkru_mask; =20 - u64 *pae_root; - u64 *pml4_root; - u64 *pml5_root; - /* * check zero bits on shadow page table entries, these * bits include not only hardware reserved bits but also diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 81ccaa7c1165..27498caa3990 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2196,26 +2196,6 @@ static void shadow_walk_init_using_root(struct kvm_s= hadow_walk_iterator *iterato iterator->addr =3D addr; iterator->shadow_addr =3D root; iterator->level =3D vcpu->arch.mmu->shadow_root_level; - - if (iterator->level >=3D PT64_ROOT_4LEVEL && - vcpu->arch.mmu->root_level < PT64_ROOT_4LEVEL && - !vcpu->arch.mmu->direct_map) - iterator->level =3D PT32E_ROOT_LEVEL; - - if (iterator->level =3D=3D PT32E_ROOT_LEVEL) { - /* - * prev_root is currently only used for 64-bit hosts. So only - * the active root_hpa is valid here. - */ - BUG_ON(root !=3D vcpu->arch.mmu->root.hpa); - - iterator->shadow_addr - =3D vcpu->arch.mmu->pae_root[(addr >> 30) & 3]; - iterator->shadow_addr &=3D PT64_BASE_ADDR_MASK; - --iterator->level; - if (!iterator->shadow_addr) - iterator->level =3D 0; - } } =20 static void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator, @@ -3328,18 +3308,7 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_= mmu *mmu, &invalid_list); =20 if (free_active_root) { - if (to_shadow_page(mmu->root.hpa)) { - mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); - } else if (mmu->pae_root) { - for (i =3D 0; i < 4; ++i) { - if (!IS_VALID_PAE_ROOT(mmu->pae_root[i])) - continue; - - mmu_free_root_page(kvm, &mmu->pae_root[i], - &invalid_list); - mmu->pae_root[i] =3D INVALID_PAE_ROOT; - } - } + mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); mmu->root.hpa =3D INVALID_PAGE; mmu->root.pgd =3D 0; } @@ -3404,7 +3373,6 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vc= pu) struct kvm_mmu *mmu =3D vcpu->arch.mmu; u8 shadow_root_level =3D mmu->shadow_root_level; hpa_t root; - unsigned i; int r; =20 write_lock(&vcpu->kvm->mmu_lock); @@ -3415,24 +3383,9 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *v= cpu) if (is_tdp_mmu_enabled(vcpu->kvm)) { root =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu); mmu->root.hpa =3D root; - } else if (shadow_root_level >=3D PT64_ROOT_4LEVEL) { + } else if (shadow_root_level >=3D PT32E_ROOT_LEVEL) { root =3D mmu_alloc_root(vcpu, 0, 0, shadow_root_level, true); mmu->root.hpa =3D root; - } else if (shadow_root_level =3D=3D PT32E_ROOT_LEVEL) { - if (WARN_ON_ONCE(!mmu->pae_root)) { - r =3D -EIO; - goto out_unlock; - } - - for (i =3D 0; i < 4; ++i) { - WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); - - root =3D mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), - i << 30, PT32_ROOT_LEVEL, true); - mmu->pae_root[i] =3D root | PT_PRESENT_MASK | - shadow_me_mask; - } - mmu->root.hpa =3D __pa(mmu->pae_root); } else { WARN_ONCE(1, "Bad TDP root level =3D %d\n", shadow_root_level); r =3D -EIO; @@ -3510,10 +3463,8 @@ static int mmu_first_shadow_root_alloc(struct kvm *k= vm) static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; - u64 pdptrs[4], pm_mask; gfn_t root_gfn, root_pgd; hpa_t root; - unsigned i; int r; =20 root_pgd =3D mmu->get_guest_pgd(vcpu); @@ -3522,21 +3473,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *v= cpu) if (mmu_check_root(vcpu, root_gfn)) return 1; =20 - /* - * On SVM, reading PDPTRs might access guest memory, which might fault - * and thus might sleep. Grab the PDPTRs before acquiring mmu_lock. - */ - if (mmu->root_level =3D=3D PT32E_ROOT_LEVEL) { - for (i =3D 0; i < 4; ++i) { - pdptrs[i] =3D mmu->get_pdptr(vcpu, i); - if (!(pdptrs[i] & PT_PRESENT_MASK)) - continue; - - if (mmu_check_root(vcpu, pdptrs[i] >> PAGE_SHIFT)) - return 1; - } - } - r =3D mmu_first_shadow_root_alloc(vcpu->kvm); if (r) return r; @@ -3546,70 +3482,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *v= cpu) if (r < 0) goto out_unlock; =20 - /* - * Do we shadow a long mode page table? If so we need to - * write-protect the guests page table root. - */ - if (mmu->root_level >=3D PT64_ROOT_4LEVEL) { - root =3D mmu_alloc_root(vcpu, root_gfn, 0, - mmu->shadow_root_level, false); - mmu->root.hpa =3D root; - goto set_root_pgd; - } - - if (WARN_ON_ONCE(!mmu->pae_root)) { - r =3D -EIO; - goto out_unlock; - } - - /* - * We shadow a 32 bit page table. This may be a legacy 2-level - * or a PAE 3-level page table. In either case we need to be aware that - * the shadow page table may be a PAE or a long mode page table. - */ - pm_mask =3D PT_PRESENT_MASK | shadow_me_mask; - if (mmu->shadow_root_level >=3D PT64_ROOT_4LEVEL) { - pm_mask |=3D PT_ACCESSED_MASK | PT_WRITABLE_MASK | PT_USER_MASK; - - if (WARN_ON_ONCE(!mmu->pml4_root)) { - r =3D -EIO; - goto out_unlock; - } - mmu->pml4_root[0] =3D __pa(mmu->pae_root) | pm_mask; - - if (mmu->shadow_root_level =3D=3D PT64_ROOT_5LEVEL) { - if (WARN_ON_ONCE(!mmu->pml5_root)) { - r =3D -EIO; - goto out_unlock; - } - mmu->pml5_root[0] =3D __pa(mmu->pml4_root) | pm_mask; - } - } - - for (i =3D 0; i < 4; ++i) { - WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); - - if (mmu->root_level =3D=3D PT32E_ROOT_LEVEL) { - if (!(pdptrs[i] & PT_PRESENT_MASK)) { - mmu->pae_root[i] =3D INVALID_PAE_ROOT; - continue; - } - root_gfn =3D pdptrs[i] >> PAGE_SHIFT; - } - - root =3D mmu_alloc_root(vcpu, root_gfn, i << 30, - PT32_ROOT_LEVEL, false); - mmu->pae_root[i] =3D root | pm_mask; - } - - if (mmu->shadow_root_level =3D=3D PT64_ROOT_5LEVEL) - mmu->root.hpa =3D __pa(mmu->pml5_root); - else if (mmu->shadow_root_level =3D=3D PT64_ROOT_4LEVEL) - mmu->root.hpa =3D __pa(mmu->pml4_root); - else - mmu->root.hpa =3D __pa(mmu->pae_root); - -set_root_pgd: + root =3D mmu_alloc_root(vcpu, root_gfn, 0, + mmu->shadow_root_level, false); + mmu->root.hpa =3D root; mmu->root.pgd =3D root_pgd; out_unlock: write_unlock(&vcpu->kvm->mmu_lock); @@ -3617,77 +3492,6 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *v= cpu) return r; } =20 -static int mmu_alloc_special_roots(struct kvm_vcpu *vcpu) -{ - struct kvm_mmu *mmu =3D vcpu->arch.mmu; - bool need_pml5 =3D mmu->shadow_root_level > PT64_ROOT_4LEVEL; - u64 *pml5_root =3D NULL; - u64 *pml4_root =3D NULL; - u64 *pae_root; - - /* - * When shadowing 32-bit or PAE NPT with 64-bit NPT, the PML4 and PDP - * tables are allocated and initialized at root creation as there is no - * equivalent level in the guest's NPT to shadow. Allocate the tables - * on demand, as running a 32-bit L1 VMM on 64-bit KVM is very rare. - */ - if (mmu->direct_map || mmu->root_level >=3D PT64_ROOT_4LEVEL || - mmu->shadow_root_level < PT64_ROOT_4LEVEL) - return 0; - - /* - * NPT, the only paging mode that uses this horror, uses a fixed number - * of levels for the shadow page tables, e.g. all MMUs are 4-level or - * all MMus are 5-level. Thus, this can safely require that pml5_root - * is allocated if the other roots are valid and pml5 is needed, as any - * prior MMU would also have required pml5. - */ - if (mmu->pae_root && mmu->pml4_root && (!need_pml5 || mmu->pml5_root)) - return 0; - - /* - * The special roots should always be allocated in concert. Yell and - * bail if KVM ends up in a state where only one of the roots is valid. - */ - if (WARN_ON_ONCE(!tdp_enabled || mmu->pae_root || mmu->pml4_root || - (need_pml5 && mmu->pml5_root))) - return -EIO; - - /* - * Unlike 32-bit NPT, the PDP table doesn't need to be in low mem, and - * doesn't need to be decrypted. - */ - pae_root =3D (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (!pae_root) - return -ENOMEM; - -#ifdef CONFIG_X86_64 - pml4_root =3D (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (!pml4_root) - goto err_pml4; - - if (need_pml5) { - pml5_root =3D (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (!pml5_root) - goto err_pml5; - } -#endif - - mmu->pae_root =3D pae_root; - mmu->pml4_root =3D pml4_root; - mmu->pml5_root =3D pml5_root; - - return 0; - -#ifdef CONFIG_X86_64 -err_pml5: - free_page((unsigned long)pml4_root); -err_pml4: - free_page((unsigned long)pae_root); - return -ENOMEM; -#endif -} - static bool is_unsync_root(hpa_t root) { struct kvm_mmu_page *sp; @@ -3725,8 +3529,7 @@ static bool is_unsync_root(hpa_t root) =20 void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) { - int i; - struct kvm_mmu_page *sp; + hpa_t root =3D vcpu->arch.mmu->root.hpa; =20 if (vcpu->arch.mmu->direct_map) return; @@ -3736,31 +3539,11 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) =20 vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY); =20 - if (vcpu->arch.mmu->root_level >=3D PT64_ROOT_4LEVEL) { - hpa_t root =3D vcpu->arch.mmu->root.hpa; - sp =3D to_shadow_page(root); - - if (!is_unsync_root(root)) - return; - - write_lock(&vcpu->kvm->mmu_lock); - mmu_sync_children(vcpu, sp, true); - write_unlock(&vcpu->kvm->mmu_lock); + if (!is_unsync_root(root)) return; - } =20 write_lock(&vcpu->kvm->mmu_lock); - - for (i =3D 0; i < 4; ++i) { - hpa_t root =3D vcpu->arch.mmu->pae_root[i]; - - if (IS_VALID_PAE_ROOT(root)) { - root &=3D PT64_BASE_ADDR_MASK; - sp =3D to_shadow_page(root); - mmu_sync_children(vcpu, sp, true); - } - } - + mmu_sync_children(vcpu, to_shadow_page(root), true); write_unlock(&vcpu->kvm->mmu_lock); } =20 @@ -5162,9 +4945,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) int r; =20 r =3D mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->direct_map); - if (r) - goto out; - r =3D mmu_alloc_special_roots(vcpu); if (r) goto out; if (vcpu->arch.mmu->direct_map) @@ -5635,65 +5415,14 @@ slot_handle_level_4k(struct kvm *kvm, const struct = kvm_memory_slot *memslot, PG_LEVEL_4K, flush_on_yield); } =20 -static void free_mmu_pages(struct kvm_mmu *mmu) -{ - if (!tdp_enabled && mmu->pae_root) - set_memory_encrypted((unsigned long)mmu->pae_root, 1); - free_page((unsigned long)mmu->pae_root); - free_page((unsigned long)mmu->pml4_root); - free_page((unsigned long)mmu->pml5_root); -} - static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) { - struct page *page; int i; =20 mmu->root.hpa =3D INVALID_PAGE; mmu->root.pgd =3D 0; for (i =3D 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) mmu->prev_roots[i] =3D KVM_MMU_ROOT_INFO_INVALID; - - /* vcpu->arch.guest_mmu isn't used when !tdp_enabled. */ - if (!tdp_enabled && mmu =3D=3D &vcpu->arch.guest_mmu) - return 0; - - /* - * When using PAE paging, the four PDPTEs are treated as 'root' pages, - * while the PDP table is a per-vCPU construct that's allocated at MMU - * creation. When emulating 32-bit mode, cr3 is only 32 bits even on - * x86_64. Therefore we need to allocate the PDP table in the first - * 4GB of memory, which happens to fit the DMA32 zone. TDP paging - * generally doesn't use PAE paging and can skip allocating the PDP - * table. The main exception, handled here, is SVM's 32-bit NPT. The - * other exception is for shadowing L1's 32-bit or PAE NPT on 64-bit - * KVM; that horror is handled on-demand by mmu_alloc_special_roots(). - */ - if (tdp_enabled && kvm_mmu_get_tdp_level(vcpu) > PT32E_ROOT_LEVEL) - return 0; - - page =3D alloc_page(GFP_KERNEL_ACCOUNT | __GFP_DMA32); - if (!page) - return -ENOMEM; - - mmu->pae_root =3D page_address(page); - - /* - * CR3 is only 32 bits when PAE paging is used, thus it's impossible to - * get the CPU to treat the PDPTEs as encrypted. Decrypt the page so - * that KVM's writes and the CPU's reads get along. Note, this is - * only necessary when using shadow paging, as 64-bit NPT can get at - * the C-bit even when shadowing 32-bit NPT, and SME isn't supported - * by 32-bit kernels (when KVM itself uses 32-bit NPT). - */ - if (!tdp_enabled) - set_memory_decrypted((unsigned long)mmu->pae_root, 1); - else - WARN_ON_ONCE(shadow_me_mask); - - for (i =3D 0; i < 4; ++i) - mmu->pae_root[i] =3D INVALID_PAE_ROOT; - return 0; } =20 @@ -5722,7 +5451,6 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu) =20 return ret; fail_allocate_root: - free_mmu_pages(&vcpu->arch.guest_mmu); return ret; } =20 @@ -6363,8 +6091,6 @@ int kvm_mmu_module_init(void) void kvm_mmu_destroy(struct kvm_vcpu *vcpu) { kvm_mmu_unload(vcpu); - free_mmu_pages(&vcpu->arch.root_mmu); - free_mmu_pages(&vcpu->arch.guest_mmu); mmu_free_memory_caches(vcpu); } =20 diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 1015f33e0758..62a762bbcf87 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -365,6 +365,16 @@ static int FNAME(walk_addr_generic)(struct guest_walke= r *walker, pte =3D mmu->get_guest_pgd(vcpu); have_ad =3D PT_HAVE_ACCESSED_DIRTY(mmu); =20 + /* kvm_mmu_get_page() might use this values for allocating passthrough + * shadow page. + */ + walker->table_gfn[4] =3D gpte_to_gfn(pte); + walker->pt_access[4] =3D ACC_ALL; + walker->table_gfn[3] =3D gpte_to_gfn(pte); + walker->pt_access[3] =3D ACC_ALL; + walker->table_gfn[2] =3D gpte_to_gfn(pte); + walker->pt_access[2] =3D ACC_ALL; + #if PTTYPE =3D=3D 64 walk_nx_mask =3D 1ULL << PT64_NX_SHIFT; if (walker->level =3D=3D PT32E_ROOT_LEVEL) { @@ -710,7 +720,8 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct k= vm_page_fault *fault, * Verify that the gpte in the page we've just write * protected is still there. */ - if (FNAME(gpte_changed)(vcpu, gw, it.level - 1)) + if (it.level - 1 < top_level && + FNAME(gpte_changed)(vcpu, gw, it.level - 1)) goto out_gpte_changed; =20 if (sp) --=20 2.19.1.6.gb485710b