From nobody Wed Feb 11 15:24:47 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 937BAC76196 for ; Fri, 7 Apr 2023 20:20:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231183AbjDGUUa (ORCPT ); Fri, 7 Apr 2023 16:20:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231231AbjDGUUS (ORCPT ); Fri, 7 Apr 2023 16:20:18 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D93FBC65F for ; Fri, 7 Apr 2023 13:20:07 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-54bfc4e0330so79098737b3.3 for ; Fri, 07 Apr 2023 13:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680898807; x=1683490807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=reXZKVgJAcUhKBFqAhAfXZIMQul2LNsezKlEEoHtiLFTJvynT88LRPnZVM8H43my2O 0AdWQfw3AEOAoOIF7B7AWJxVFSFD/w3Q2pEfhk9TLSFsHjsMCNwDjt1nToC3Erq9lXvB g1LbxQEVzYJTdlQ5MYz5hDc9P8gRoRV3iaOFlg0OdPEkY2HKDJ8CNZ5ywg2KOO/iF5t6 lVplsBeH6uvx9ev0WzAiyQG3vF+n90o7T5CR+YbfiXxyMGTfkTZG4CFGpeaqYlO+IJif dO3yYbv3bKl2jMrlV2SrXMEB8JHo/vfYywtDCcW7wAKIKF7yN3CbU+qRkgehxk1sT0AX eQgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680898807; x=1683490807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gDOIz19mFLXVjh2KEN4BbSPwap6txMB8kY9CE7752rg=; b=7JnEeDrobtxT7XUksu4yoXsn8s+3qK9fTEQymgEIjbh3vbTWZvWX2Q9uihCi6oCBB8 oj5m7e9rU+aWgvxf2ThTUWE9Vwlda1XfO98XPUiBz5X489kG7lcsw7KYepgbWm1lOxW1 8AxMNnrxWf4Fdt9ljYWkvE2F143n8xHw/MnuSJRVEjK4aFhDIrhJfQozgFxt8gpr9zAP S7qa5KRkLAlDB+nGhIJLgKsfJ44E+zivoG6/zRVImGI9Nl6hiyADXqxox2xf0B9eGwqs DgnFmZ2wkHTRRzrdsyrkh7sNRLzYm592GlYoSZtPmBAbotBK789krz9u/kMGTBtMcBNS amEQ== X-Gm-Message-State: AAQBX9eaNlf3CYb4IAKUd3DnLvKeJdWBF8btMQxmnBD61pydccvdmeVf VoAxVi5czxKowqb+Lx6I1/OFjEGMkA== X-Google-Smtp-Source: AKy350bhhE80rWXZROApfO8xLPaZj9ts5o8abPpaSzK+xmA5azXLcrO5jVnu2rk1aXyBf4QNsfLxthLxYg== X-Received: from sagi.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:241b]) (user=sagis job=sendgmr) by 2002:a05:6902:909:b0:a27:3ecc:ffe7 with SMTP id bu9-20020a056902090900b00a273eccffe7mr5158014ybb.3.1680898807168; Fri, 07 Apr 2023 13:20:07 -0700 (PDT) Date: Fri, 7 Apr 2023 20:19:20 +0000 In-Reply-To: <20230407201921.2703758-1-sagis@google.com> Mime-Version: 1.0 References: <20230407201921.2703758-1-sagis@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230407201921.2703758-5-sagis@google.com> Subject: [RFC PATCH 4/5] KVM: TDX: Implement moving private pages between 2 TDs From: Sagi Shahar To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org Cc: Sean Christopherson , Paolo Bonzini , Isaku Yamahata , Erdem Aktas , David Matlack , Kai Huang , Zhi Wang , Chao Peng , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Sagi Shahar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added functionality for moving the private EPT table from one TD to a new one. This function moves the root of the private EPT table and overwrite the root of the destination. Signed-off-by: Sagi Shahar --- arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 77 +++++++++++++++++++++++++++++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 3 ++ 4 files changed, 137 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index d10b08eeaefee..09bae7fe18a12 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -120,6 +120,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu); void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); +int kvm_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu); =20 static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a35f2e7f9bc70..1acc9338323da 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3789,6 +3789,66 @@ static int mmu_first_shadow_root_alloc(struct kvm *k= vm) return r; } =20 +int kvm_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu) +{ + struct kvm_mmu *mmu =3D vcpu->arch.mmu; + struct kvm_mmu *src_mmu =3D src_vcpu->arch.mmu; + gfn_t gfn_shared =3D kvm_gfn_shared_mask(vcpu->kvm); + hpa_t private_root_hpa, shared_root_hpa; + int r =3D -EINVAL; + + // Hold locks for both src and dst. Always take the src lock first. + write_lock(&src_vcpu->kvm->mmu_lock); + write_lock(&vcpu->kvm->mmu_lock); + + if (!gfn_shared) + goto out_unlock; + + WARN_ON_ONCE(!is_tdp_mmu_active(vcpu)); + WARN_ON_ONCE(!is_tdp_mmu_active(src_vcpu)); + + r =3D mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->root_role.direct); + if (r) + goto out_unlock; + + /* + * The private root is moved from the src to the dst and is marked as + * invalid in the src. + */ + private_root_hpa =3D kvm_tdp_mmu_move_private_pages_from(vcpu, src_vcpu); + if (private_root_hpa =3D=3D INVALID_PAGE) { + /* + * This likely means that the private root was already moved by + * another vCPU. + */ + private_root_hpa =3D kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(vcpu, true); + if (private_root_hpa =3D=3D INVALID_PAGE) { + r =3D -EINVAL; + goto out_unlock; + } + } + + mmu->private_root_hpa =3D private_root_hpa; + src_mmu->private_root_hpa =3D INVALID_PAGE; + + /* + * The shared root is allocated normally and is not moved from the src. + */ + shared_root_hpa =3D kvm_tdp_mmu_get_vcpu_root_hpa(vcpu, false); + mmu->root.hpa =3D shared_root_hpa; + + kvm_mmu_load_pgd(vcpu); + static_call(kvm_x86_flush_tlb_current)(vcpu); + +out_unlock: + write_unlock(&vcpu->kvm->mmu_lock); + write_unlock(&src_vcpu->kvm->mmu_lock); + + return r; +} +EXPORT_SYMBOL(kvm_mmu_move_private_pages_from); + static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 327dee4f6170e..685528fdc0ad6 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -296,6 +296,23 @@ static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, t= dp_ptep_t sptep, trace_kvm_mmu_get_page(sp, true); } =20 +static struct kvm_mmu_page * +kvm_tdp_mmu_get_vcpu_root_no_alloc(struct kvm_vcpu *vcpu, union kvm_mmu_pa= ge_role role) +{ + struct kvm *kvm =3D vcpu->kvm; + struct kvm_mmu_page *root; + + lockdep_assert_held_read(&kvm->mmu_lock); + + for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { + if (root->role.word =3D=3D role.word && + kvm_tdp_mmu_get_root(root)) + return root; + } + + return NULL; +} + static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(struct kvm_vcpu *vcp= u, bool private) { @@ -311,11 +328,9 @@ static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(= struct kvm_vcpu *vcpu, */ if (private) kvm_mmu_page_role_set_private(&role); - for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { - if (root->role.word =3D=3D role.word && - kvm_tdp_mmu_get_root(root)) - goto out; - } + root =3D kvm_tdp_mmu_get_vcpu_root_no_alloc(vcpu, role); + if (!!root) + goto out; =20 root =3D tdp_mmu_alloc_sp(vcpu, role); tdp_mmu_init_sp(root, NULL, 0); @@ -330,6 +345,58 @@ static struct kvm_mmu_page *kvm_tdp_mmu_get_vcpu_root(= struct kvm_vcpu *vcpu, return root; } =20 +hpa_t kvm_tdp_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu) +{ + union kvm_mmu_page_role role =3D vcpu->arch.mmu->root_role; + struct kvm *kvm =3D vcpu->kvm; + struct kvm *src_kvm =3D src_vcpu->kvm; + struct kvm_mmu_page *private_root =3D NULL; + struct kvm_mmu_page *root; + s64 num_private_pages, old; + + lockdep_assert_held_write(&vcpu->kvm->mmu_lock); + lockdep_assert_held_write(&src_vcpu->kvm->mmu_lock); + + /* Find the private root of the source. */ + kvm_mmu_page_role_set_private(&role); + for_each_tdp_mmu_root(src_kvm, root, kvm_mmu_role_as_id(role)) { + if (root->role.word =3D=3D role.word) { + private_root =3D root; + break; + } + } + if (!private_root) + return INVALID_PAGE; + + /* Remove the private root from the src kvm and add it to dst kvm. */ + list_del_rcu(&private_root->link); + list_add_rcu(&private_root->link, &kvm->arch.tdp_mmu_roots); + + num_private_pages =3D atomic64_read(&src_kvm->arch.tdp_private_mmu_pages); + old =3D atomic64_cmpxchg(&kvm->arch.tdp_private_mmu_pages, 0, + num_private_pages); + /* The destination VM should have no private pages at this point. */ + WARN_ON(old); + atomic64_set(&src_kvm->arch.tdp_private_mmu_pages, 0); + + return __pa(private_root->spt); +} + +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(struct kvm_vcpu *vcpu, bool p= rivate) +{ + struct kvm_mmu_page *root; + union kvm_mmu_page_role role =3D vcpu->arch.mmu->root_role; + + if (private) + kvm_mmu_page_role_set_private(&role); + root =3D kvm_tdp_mmu_get_vcpu_root_no_alloc(vcpu, role); + if (!root) + return INVALID_PAGE; + + return __pa(root->spt); +} + hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private) { return __pa(kvm_tdp_mmu_get_vcpu_root(vcpu, private)->spt); diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3ae3c3b8642ac..0e9d38432673d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -11,6 +11,9 @@ int kvm_mmu_init_tdp_mmu(struct kvm *kvm); void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm); =20 hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu, bool private); +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa_no_alloc(struct kvm_vcpu *vcpu, bool p= rivate); +hpa_t kvm_tdp_mmu_move_private_pages_from(struct kvm_vcpu *vcpu, + struct kvm_vcpu *src_vcpu); =20 __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm_mmu_page *= root) { --=20 2.40.0.348.gf938b09366-goog