From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46DEA4C9F for ; Fri, 29 Aug 2025 00:06:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425985; cv=none; b=FdxPA/nfHAPqbmIXCtsMwJbKwAxjaGouFLi3PSGBlSmaFbZdDmj3mOK/DqqWwsi2CpXfURDR9vYIea2u4eUrHmrlHwHrn9FOGVVHKe96/QvH6HBxMkKDk4grxKRKgPxs06kvKvupjMG+fu9Am9X8UM9f4sYq2TR8wgv73HYbVoY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425985; c=relaxed/simple; bh=wOBgvXCRAiu/hoH9lp/TtNCi0CROexOvvzcS9prFsj0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cm9LvAG6C8nPChb3PwgVALLg0rGn9wTcBS30XprNrhMpH9rqoEHcZ3IIt9mXBG7IzPdoSSvCd4nC9NLaLo81/+1EiEtwyKlGj/RROXZd38YTWc3DptI7lgErlr8gfhBa3L6RMDlzadnuCvZzMaxnKQY5ktczkBJBhnvQ6cFMKIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=K1vp0yqU; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="K1vp0yqU" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b47174c8fd2so2145700a12.2 for ; Thu, 28 Aug 2025 17:06:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425983; x=1757030783; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=kN41JsVo/+Hu/VK2aKospmiW+UbLhaUfceiD5wkfXWs=; b=K1vp0yqUUiN/qr4DCsm5pByzKTi+rEbuhoPIr7t+74MbpeIYEZDF5FkhoCP8p70VXw 2evsGbgOw+dtQXKM6d4cvPdNony8kgPnzcJcbEbzrXehJGSGXH8ehvJgHJU65CMRgHdS Wzy8e38BzoVJLsSNpS09Z9zjIJoi+PQhVY2TR8M/QGdgbDvs/RrJnL0A4+MxV109LSV6 aj3KdWGV9ZtXv/VzshqLZbyvHEuU5X63YApev96A+G77WKL28iUs/QM5Ta6hJOaRPeiK uDH+05HzDV/onIqx2UvAu/5PB+d25dVuIJD+0Y6X7WAExE5uZz5QO/VgmwEQS2HNJpF1 jnHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425983; x=1757030783; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kN41JsVo/+Hu/VK2aKospmiW+UbLhaUfceiD5wkfXWs=; b=XuWRlORkIjtr8sbx/iKC41hGP4d55S/MMKOIUOk8B89jrDIDkYHUXO+Ej/q7puy4xk 8Pek4XdCoMERQPiuE/BwU6ekSi6K06jje9xflvi2efSyJoKqwkNk/smFKLbxtXrG2+XG fSLK7SEEMDM5Vr7dhvrqs7OL9kgoN1Hhbf/Iq9E8z+Gr3X+cNf7j+1gU0n23+W0DJoY2 mO9p9BwwLi9hbPwQhDTcBh1XIWXWFYdAblDJwh0tICHaDloCoks4ML2HcV9A2lH68zfN BoE9FbwybNIRiAIC0kFR7JmwX7MiS0zd+sjWEEmuiVwmLwezmBaaQgAXyTDTHnHuDznz RCKA== X-Forwarded-Encrypted: i=1; AJvYcCV+DrPV/bz2dg1D6YWLDg3N//HboHfASfUnnWBbSF3lGzOhlnYdcr98Xxtmp/8jLiZwpPtSYhXUVI+aQP0=@vger.kernel.org X-Gm-Message-State: AOJu0YwUqoc/GbgtdV0ve25bql+IlwrX8p4R0FOC3fmwWHW5EpW9lbBZ YEZ8k3edqUBAgq058a2Hme+ejX8Axgalo6lnlepTRoHDxb2GCWk0Ad1Sx75CIHUhpe4iJljusxP 7JJr9jg== X-Google-Smtp-Source: AGHT+IFvRGkDVD9A1RgamItKrcAi4g0yGHO3XYJeIxd/1I7Yw9nU5hV2wzRGP2NL4PULhLR+r74hxCnLI84= X-Received: from pjbpm6.prod.google.com ([2002:a17:90b:3c46:b0:325:238b:5dc6]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:1591:b0:243:a21c:3730 with SMTP id adf61e73a8af0-243a21c3b63mr10436147637.31.1756425983578; Thu, 28 Aug 2025 17:06:23 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:01 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-2-seanjc@google.com> Subject: [RFC PATCH v2 01/18] KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop TDX's sanity check that an S-EPT mapping isn't zapped between creating said mapping and doing TDH.MEM.PAGE.ADD, as the check is simultaneously superfluous and incomplete. Per commit 2608f1057601 ("KVM: x86/tdp_mmu: Add a helper function to walk down the TDP MMU"), the justification for introducing kvm_tdp_mmu_gpa_is_mapped() was to check that the target gfn was pre-populated, with a link that points to this snippet: : > One small question: : > : > What if the memory region passed to KVM_TDX_INIT_MEM_REGION hasn't bee= n pre- : > populated? If we want to make KVM_TDX_INIT_MEM_REGION work with these= regions, : > then we still need to do the real map. Or we can make KVM_TDX_INIT_ME= M_REGION : > return error when it finds the region hasn't been pre-populated? : : Return an error. I don't love the idea of bleeding so many TDX details = into : userspace, but I'm pretty sure that ship sailed a long, long time ago. But that justification makes little sense for the final code, as simply doing TDH.MEM.PAGE.ADD without a paranoid sanity check will return an error if the S-EPT mapping is invalid (as evidenced by the code being guarded with CONFIG_KVM_PROVE_MMU=3Dy). The sanity check is also incomplete in the sense that mmu_lock is dropped between the check and TDH.MEM.PAGE.ADD, i.e. will only detect KVM bugs that zap SPTEs in a very specific window. Removing the sanity check will allow removing kvm_tdp_mmu_gpa_is_mapped(), which has no business being exposed to vendor code. Reviewed-by: Ira Weiny Reviewed-by: Kai Huang Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/tdx.c | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 6784aaaced87..71da245d160f 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -3175,20 +3175,6 @@ static int tdx_gmem_post_populate(struct kvm *kvm, g= fn_t gfn, kvm_pfn_t pfn, if (ret < 0) goto out; =20 - /* - * The private mem cannot be zapped after kvm_tdp_map_page() - * because all paths are covered by slots_lock and the - * filemap invalidate lock. Check that they are indeed enough. - */ - if (IS_ENABLED(CONFIG_KVM_PROVE_MMU)) { - scoped_guard(read_lock, &kvm->mmu_lock) { - if (KVM_BUG_ON(!kvm_tdp_mmu_gpa_is_mapped(vcpu, gpa), kvm)) { - ret =3D -EIO; - goto out; - } - } - } - ret =3D 0; err =3D tdh_mem_page_add(&kvm_tdx->td, gpa, pfn_to_page(pfn), src_page, &entry, &level_state); --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5ECF1DFF7 for ; Fri, 29 Aug 2025 00:06:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425987; cv=none; b=bKD7y5PaD/tnqdbdeT+qiN2F0W+Lb+hWXYMEWD2uRUu/0f2icfgNm6OUiZjaXKz9sXqJM3EmFLbrAu9vWC/HnsLwFShj5hgTx01SP7tPY62KR4GfbN8GXiuhFp9RqVTAGeG1MvNYZxCtqkMpH2vOc11lOkjOJR1cDqqTx88ocvM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425987; c=relaxed/simple; bh=KCT7Nt9F54BAobdhz2zR/stlMdrvovd8onIkINoAfd8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hbNV4zNmSiTlEsJMtN3QdXY01WhjLDQ+OlER0gjwDiwOnTwAnZ4cy6lAaDod4L3uvXjjmWf3fkdv43aX6YLTY0AMjnPMJwprm6a4wwz+En5YHE+zQ9KUs/fyt4WgM0QsfOS0Q7OyNhEGoQdc9CYcMKCayabyHWaqD6SQp7Lg/BI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4FKfHUlh; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4FKfHUlh" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-3251961140bso1467371a91.2 for ; Thu, 28 Aug 2025 17:06:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425985; x=1757030785; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=QsIXvnuSh6UUFD5YwAdWYJa2UJyM2Kr57W7DFFX+dvk=; b=4FKfHUlh22KoahnDmFNzByidSTXPl8NsRcAQxmNBO1mfxUXDwVqJBJmNMhK7tEHwWt 3+Y/u9KyV2IPkHOg3AuDmqqjkhDOOqmK1bQJlZIzgSCi5u1CBTXpUDP3CAYMUMRe2iuz uH8xX9asHGXlb1VcTChRHwu0Q89Zsxsyc9FJ3JOKcBxcnv1s3OJexkFPLbKqZj5dKFHo Nr/zSgjnL9RDEtPOyPvYt5PGR/i8ig6SJVUd2h3eZnmX9hRkXu8hk6YVIrhvY1ELfDEX sLJbkuVBc47KLstym/aTUkVJmU3AT8HhpAYPnFo9fdp6evgMF+zdvS32Vif5/22czUhD AOzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425985; x=1757030785; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QsIXvnuSh6UUFD5YwAdWYJa2UJyM2Kr57W7DFFX+dvk=; b=O1043N7Lx2JnBoztAbwPnqDXp40CQFnaaagMZibdlVWY6LaMOLArcMsr+EU0bmYHVO jANKiJtHaOlcgzkr8dD49QEOmrqLkFavTx3VhGjekzeeTOB3QAx2TGtVae8n3veHzOTH KwGkAbx/Jb5bHgIMwZDrEwfQ3Odfu4Xj+XFrTCNVm/b/5RkUdkqLalJhwpirj/3AXdud 5xDbNDEhP91ox1i/h9S84nwQisXx0K/0lKCPlsP6POGPK0+AkHUXLMcxTUD0T/aQAA0x M2MU5GJA4wQFHRp4Qqxh6e1YLQn9FhQ4fn6eoRbj2V6C1rPI0uzTn3al6PZh6uMdfUwo gqCg== X-Forwarded-Encrypted: i=1; AJvYcCWbZQWu0W3pzMonhxgg/nRFWc3ZDYVfIVtfBJP8z4+nSzcZe494dNd1lEuXsFcGK+cTD2FvXS9e2WD03E8=@vger.kernel.org X-Gm-Message-State: AOJu0YyGFtEqZNHq1i4E73Q4s0U5+LaL3KNGPYTf0PdEv129clNF9XWS KTfRV0/KdiUGqbWilyXhhlM73hov+IKTW7NxAAkAr7c/KHsd9xYIarF4Dprz3ModIUq7TinzNrG MMohP7w== X-Google-Smtp-Source: AGHT+IGQe12O8dbqInnyVob7Pm608CuTv+jQ1L1sCiFT2qw6Fyu996exnKxTQRc1JDH1tdMc9QZ8SiBK2nQ= X-Received: from pjbsc16.prod.google.com ([2002:a17:90b:5110:b0:327:d636:2d1e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e890:b0:246:e8cc:8cef with SMTP id d9443c01a7336-246e8cc919bmr202029185ad.3.1756425985105; Thu, 28 Aug 2025 17:06:25 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:02 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-3-seanjc@google.com> Subject: [RFC PATCH v2 02/18] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add and use a new API for mapping a private pfn from guest_memfd into the TDP MMU from TDX's post-populate hook instead of partially open-coding the functionality into the TDX code. Sharing code with the pre-fault path sounded good on paper, but it's fatally flawed as simulating a fault loses the pfn, and calling back into gmem to re-retrieve the pfn creates locking problems, e.g. kvm_gmem_populate() already holds the gmem invalidation lock. Providing a dedicated API will also removing several MMU exports that ideally would not be exposed outside of the MMU, let alone to vendor code. On that topic, opportunistically drop the kvm_mmu_load() export. Leave kvm_tdp_mmu_gpa_is_mapped() alone for now; the entire commit that added kvm_tdp_mmu_gpa_is_mapped() will be removed in the near future. Cc: Michael Roth Cc: Yan Zhao Cc: Ira Weiny Cc: Vishal Annapurve Cc: Rick Edgecombe Link: https://lore.kernel.org/all/20250709232103.zwmufocd3l7sqk7y@amd.com Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.c | 10 +++---- 3 files changed, 63 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index b4b6860ab971..697b90a97f43 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -259,6 +259,7 @@ extern bool tdp_mmu_enabled; =20 bool kvm_tdp_mmu_gpa_is_mapped(struct kvm_vcpu *vcpu, u64 gpa); int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 = *level); +int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_= t pfn); =20 static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 92ff15969a36..65300e43d6a1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4994,6 +4994,65 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu = *vcpu, return min(range->size, end - range->gpa); } =20 +int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_= t pfn) +{ + struct kvm_page_fault fault =3D { + .addr =3D gfn_to_gpa(gfn), + .error_code =3D PFERR_GUEST_FINAL_MASK | PFERR_PRIVATE_ACCESS, + .prefetch =3D true, + .is_tdp =3D true, + .nx_huge_page_workaround_enabled =3D is_nx_huge_page_enabled(vcpu->kvm), + + .max_level =3D PG_LEVEL_4K, + .req_level =3D PG_LEVEL_4K, + .goal_level =3D PG_LEVEL_4K, + .is_private =3D true, + + .gfn =3D gfn, + .slot =3D kvm_vcpu_gfn_to_memslot(vcpu, gfn), + .pfn =3D pfn, + .map_writable =3D true, + }; + struct kvm *kvm =3D vcpu->kvm; + int r; + + lockdep_assert_held(&kvm->slots_lock); + + if (KVM_BUG_ON(!tdp_mmu_enabled, kvm)) + return -EIO; + + if (kvm_gfn_is_write_tracked(kvm, fault.slot, fault.gfn)) + return -EPERM; + + r =3D kvm_mmu_reload(vcpu); + if (r) + return r; + + r =3D mmu_topup_memory_caches(vcpu, false); + if (r) + return r; + + do { + if (signal_pending(current)) + return -EINTR; + + if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) + return -EIO; + + cond_resched(); + + guard(read_lock)(&kvm->mmu_lock); + + r =3D kvm_tdp_mmu_map(vcpu, &fault); + } while (r =3D=3D RET_PF_RETRY); + + if (r !=3D RET_PF_FIXED) + return -EIO; + + return 0; +} +EXPORT_SYMBOL_GPL(kvm_tdp_mmu_map_private_pfn); + static void nonpaging_init_context(struct kvm_mmu *context) { context->page_fault =3D nonpaging_page_fault; @@ -5977,7 +6036,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out: return r; } -EXPORT_SYMBOL_GPL(kvm_mmu_load); =20 void kvm_mmu_unload(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 71da245d160f..c83e1ff02827 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -3151,15 +3151,12 @@ struct tdx_gmem_post_populate_arg { static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pf= n, void __user *src, int order, void *_arg) { - u64 error_code =3D PFERR_GUEST_FINAL_MASK | PFERR_PRIVATE_ACCESS; - struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); struct tdx_gmem_post_populate_arg *arg =3D _arg; - struct kvm_vcpu *vcpu =3D arg->vcpu; + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + u64 err, entry, level_state; gpa_t gpa =3D gfn_to_gpa(gfn); - u8 level =3D PG_LEVEL_4K; struct page *src_page; int ret, i; - u64 err, entry, level_state; =20 /* * Get the source page if it has been faulted in. Return failure if the @@ -3171,7 +3168,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gf= n_t gfn, kvm_pfn_t pfn, if (ret !=3D 1) return -ENOMEM; =20 - ret =3D kvm_tdp_map_page(vcpu, gpa, error_code, &level); + ret =3D kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn); if (ret < 0) goto out; =20 @@ -3234,7 +3231,6 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *= vcpu, struct kvm_tdx_cmd *c !vt_is_tdx_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHI= FT) - 1)) return -EINVAL; =20 - kvm_mmu_reload(vcpu); ret =3D 0; while (region.nr_pages) { if (signal_pending(current)) { --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A55787404E for ; Fri, 29 Aug 2025 00:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425989; cv=none; b=jB/4O68513qDQGfSU4IDjG53fftAC3H4q3sovCyRR+3B1GBc6KEJvm+ZfFFi49O1KnlyKejoZmKdBb0SVFElzTr3CWYOBqrrpbQxFXCmQTHBhYUmwY3DfNbLTw/4JZFVKmR+oWtTl6wOLd/BuEdbNAkWR98OwdE6ljAtpGLR+3E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425989; c=relaxed/simple; bh=+SN/7jwzsBSGX3ZaBL+TtK4mD4SxZwJY0/a1k70+r9I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nBv5zqoxcQyKjZYtvD8Z4Y2cWd4YCGB83v9bKD7d3x8fii8nphclUCxSlYT6BmqpEnGqhIwwhPa/8m5QxZelLdkohkF5FwCI3+8YHW7yDS+gt8VJSV6KaWUcL/ytY8v0ewllvR6kkwcwm7tXryzUhsTyNrZ2HmsxCU/wFy7xVL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lH9rkEbs; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lH9rkEbs" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b49de53348aso936638a12.1 for ; Thu, 28 Aug 2025 17:06:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425987; x=1757030787; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=IfvFc9viaP2S1pT15SAkIYCawAjZc30p7N/9YScLJsM=; b=lH9rkEbsFPOJjPknjc4SXPjeCmsVlbafN6X3/n6Zoc+w0d1ZhuXEHxmvlf8esEfLuM GoREXU3DszMGLQ9w7KUkBRAKrtujbYg3LYuqU7c+qlYZSGCcUbSURmfrPUWajgk2tBZ3 0ZMtBmDDk6YSNYGIt0hT3R5B5TGdk+ivy6iNtKRytDkq7NYaKR9EVcO0Z4chxytXD0Ma soQt2cY0z6KOutwLAX9LZjB40+xWCQxtEup95hXGwza1q+AEa097pt81BdnGaGBub6FV W2zjPu0sUitLwjELpMzBs30AcyOoWzBzhuu/Jh2iVpLFyec6dHheKC4n2/2YxgA+hw2Z o5Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425987; x=1757030787; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IfvFc9viaP2S1pT15SAkIYCawAjZc30p7N/9YScLJsM=; b=YWfmboWUrJMeEzu9Km+FxJW6H8hd3mTlga//jqp33xoM76EfNFXP1HkKnQWCloehZP WuWft0RwZ96SpGOhT5KhvfGZShZdPAdi0sROMBSEdk+acJ7hmgoSX5M1xd9kETzSk0eP Fyfa4fVR3Mlz57cTcBGhAzjt0cBIwjDnAX7RM2Or5FRU40qbdXZ4Jj7EKmtuL0ptNcnc PGSxhCuRlsDk0j0lQ0apo8PoVuEJ1e2Ucl/DaJqs0cR/s+qVkrpOSrDFiavOI5muXyza NSkuE2Q7tCX0XnyuktTA8FBDaS7Kj3QNRs2vVHIEBA4B+JjnD5k8wfJP2pzKWuyWbiq0 WHrg== X-Forwarded-Encrypted: i=1; AJvYcCUy/ThrI30n1XBG1s5jEd4JM0fKcHsc61SXjkCeUg2L3pERNiyU6p2t5JSIuNbChXmwU0qMkuBwcImwjRY=@vger.kernel.org X-Gm-Message-State: AOJu0YwHrWU3D9/PlPdwcMEjAdxqYx2s/dZEj833u4V+2hiAiiENfDUH fH3LHxoDOwvnI1yFpHs+pSiHPVnIzYfnnCsHVi30+dXFpfVvWTxCcCMgwto8PVBPPrt8VqjWJnO cLcnJfQ== X-Google-Smtp-Source: AGHT+IFQQpXTh6rHGsE3ofmt/dZ3Md25hDKkJ4b99R0cEkcoLYtIyYzC64YgEZMJ6vkZUCO2UTc7vcpF+CE= X-Received: from pjvf13.prod.google.com ([2002:a17:90a:da8d:b0:31c:4a51:8b75]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:7f96:b0:240:792:e87a with SMTP id adf61e73a8af0-2438facc65cmr15373768637.3.1756425986990; Thu, 28 Aug 2025 17:06:26 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:03 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-4-seanjc@google.com> Subject: [RFC PATCH v2 03/18] Revert "KVM: x86/tdp_mmu: Add a helper function to walk down the TDP MMU" From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the helper and exports that were added to allow TDX code to reuse kvm_tdp_map_page() for its gmem post-populate flow now that a dedicated TDP MMU API is provided to install a mapping given a gfn+pfn pair. This reverts commit 2608f105760115e94a03efd9f12f8fbfd1f9af4b. Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/mmu.h | 2 -- arch/x86/kvm/mmu/mmu.c | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++-------------------------------- 3 files changed, 7 insertions(+), 36 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 697b90a97f43..dc6b965cea4f 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -257,8 +257,6 @@ extern bool tdp_mmu_enabled; #define tdp_mmu_enabled false #endif =20 -bool kvm_tdp_mmu_gpa_is_mapped(struct kvm_vcpu *vcpu, u64 gpa); -int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 = *level); int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_= t pfn); =20 static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 65300e43d6a1..f808c437d738 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4904,7 +4904,8 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct = kvm_page_fault *fault) return direct_page_fault(vcpu, fault); } =20 -int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 = *level) +static int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_co= de, + u8 *level) { int r; =20 @@ -4946,7 +4947,6 @@ int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa= , u64 error_code, u8 *level return -EIO; } } -EXPORT_SYMBOL_GPL(kvm_tdp_map_page); =20 long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, struct kvm_pre_fault_memory *range) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 31d921705dee..3ea2dd64ce72 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1939,13 +1939,16 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, * * Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}. */ -static int __kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sp= tes, - struct kvm_mmu_page *root) +int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, + int *root_level) { + struct kvm_mmu_page *root =3D root_to_sp(vcpu->arch.mmu->root.hpa); struct tdp_iter iter; gfn_t gfn =3D addr >> PAGE_SHIFT; int leaf =3D -1; =20 + *root_level =3D vcpu->arch.mmu->root_role.level; + for_each_tdp_pte(iter, vcpu->kvm, root, gfn, gfn + 1) { leaf =3D iter.level; sptes[leaf] =3D iter.old_spte; @@ -1954,36 +1957,6 @@ static int __kvm_tdp_mmu_get_walk(struct kvm_vcpu *v= cpu, u64 addr, u64 *sptes, return leaf; } =20 -int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, - int *root_level) -{ - struct kvm_mmu_page *root =3D root_to_sp(vcpu->arch.mmu->root.hpa); - *root_level =3D vcpu->arch.mmu->root_role.level; - - return __kvm_tdp_mmu_get_walk(vcpu, addr, sptes, root); -} - -bool kvm_tdp_mmu_gpa_is_mapped(struct kvm_vcpu *vcpu, u64 gpa) -{ - struct kvm *kvm =3D vcpu->kvm; - bool is_direct =3D kvm_is_addr_direct(kvm, gpa); - hpa_t root =3D is_direct ? vcpu->arch.mmu->root.hpa : - vcpu->arch.mmu->mirror_root_hpa; - u64 sptes[PT64_ROOT_MAX_LEVEL + 1], spte; - int leaf; - - lockdep_assert_held(&kvm->mmu_lock); - rcu_read_lock(); - leaf =3D __kvm_tdp_mmu_get_walk(vcpu, gpa, sptes, root_to_sp(root)); - rcu_read_unlock(); - if (leaf < 0) - return false; - - spte =3D sptes[leaf]; - return is_shadow_present_pte(spte) && is_last_spte(spte, leaf); -} -EXPORT_SYMBOL_GPL(kvm_tdp_mmu_gpa_is_mapped); - /* * Returns the last level spte pointer of the shadow page walk for the giv= en * gpa, and sets *spte to the spte value. This spte may be non-preset. If = no --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59BCA4C9F for ; Fri, 29 Aug 2025 00:06:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425990; cv=none; b=GNve3umKOYqWxQl5WR36v+WhYX0rsu9M5a3PUQasemJPYHrfXuVZDipe1MuuOweYjp9dds1M4ggQEbBaACEmiJFQVT2vUpCYvr80k6tNj0UPg0ZUul4Q21vZi9b3xui3NajdeEoLvzv4ijVxSogfDkiKKRk0TiUPwdKpNdpZB0o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425990; c=relaxed/simple; bh=6yIUkZeN6bMoRkgKDRajINf2nZ43Ns9yxSaUcclg4QY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GmLQCrAS0cqW4De4m5iv3Wf7jEYFgIADebI6/DAw5KgSd94MAknlkUSHuaTqkbbdHk//iIBBxSCIkQ11c2c/OgTA+ortxynHls7AqWdSQaoVoZ8Vw+isdNTboJ6rSgkGmsJgfbsW1msNW5ndd1/DGlVz61WvzaAW24rpSO+BGVA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bR/AYtRB; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bR/AYtRB" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-b47174b3427so1259734a12.2 for ; Thu, 28 Aug 2025 17:06:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425989; x=1757030789; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=fiTedu7sqH0jEzzfnRQK+Pd8x3O94MW62vCUbcDlgmI=; b=bR/AYtRBNaTUgG9Rb+GCSjUaQHXFlOZIDhOkorCmu7zUUGVV7UqzGssDvbZ1TT/3hM mvpqg3fyU1jRADzH4YE8fOsmlDFrAsQvthGnoSN4FnbYgLu586cECp9WZfNSuWny0yqI 5wAQZykKocuHxF4X+rGYmqvDXGohYgIgoW8PzPwu2MLk7v1LOSthHxkCnB94R6wzu5y8 UNvvHH0mZAIpAPfRsr1pNgjiwhppHlXRT2epG5FSC+Tt8VjSQS5WOe0ejLAioaNd5tRY as5lXkN2f4bZfxgsHPybXyWnjonZXdl/ePPobMbebevI/d0CPihdgoyIwGbLr99QcEXx dapQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425989; x=1757030789; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fiTedu7sqH0jEzzfnRQK+Pd8x3O94MW62vCUbcDlgmI=; b=T3wDgr5+uHT+0rI4C/W1YPNan6TA7QSLi2ZoUnAcZqo9xkIpA6Znqbl99Oi41yWz// v3R0zK9VGX/9t77vwrRm8R8PI18NaXVxT7wTqK3RKKrKnj5yMOeBO1ZpY7wENEKAsH4W 3aP/F89nMhWZXvmNM4FnL/d9aGkMH/LhVKCGVPL0nvFfEKmGYpwhLggkmBgCwGam+31U MKJfXFmCrRMz3lOyJIMJ9JV+2LzyyysYHCfzIe3HqVJLK6WZWXJvLig22DdvzcBQX/NZ eQUSPX98sEl3cw7s6We4OAUuc510Sla9GtML5O5EVY5VU4vX1T/aJNYSkrNKfMVOL7zH Jv+Q== X-Forwarded-Encrypted: i=1; AJvYcCW6zAxsT14r2Pahj8FHtJIDqffOwyNA5w81WlwDq9pOKabvTLBRhdDkXYAj7NvRdmvPqLDT15s+QuVpTpE=@vger.kernel.org X-Gm-Message-State: AOJu0YzYcOvXFFAxglDtISQpY+oldfzAkvEwUUY1YEF654QWMCeHMD0b NelmDryBaZTC5RUsE0b/MNTEmsIN3LtoJD4SZPv0lxixaromhh5oZpaICHhF6DDXmMjWOLtvrNu mFBZsIw== X-Google-Smtp-Source: AGHT+IFZNlUvYoCp4G8CHMrlHBqcMouxg+12z/QGwQt6VYAqJdmYsFGUXf61w8vimDXzgykBxzSIvpAX7rQ= X-Received: from pjbsv14.prod.google.com ([2002:a17:90b:538e:b0:321:c23e:5e41]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:3546:b0:246:d703:cf83 with SMTP id d9443c01a7336-246d703e19fmr236188325ad.17.1756425988741; Thu, 28 Aug 2025 17:06:28 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:04 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-5-seanjc@google.com> Subject: [RFC PATCH v2 04/18] KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_page_prefault() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename kvm_tdp_map_page() to kvm_tdp_page_prefault() now that it's used only by kvm_arch_vcpu_pre_fault_memory(). No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/mmu/mmu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f808c437d738..dddeda7f05eb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4904,8 +4904,8 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct = kvm_page_fault *fault) return direct_page_fault(vcpu, fault); } =20 -static int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_co= de, - u8 *level) +static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa, + u64 error_code, u8 *level) { int r; =20 @@ -4982,7 +4982,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *= vcpu, * Shadow paging uses GVA for kvm page fault, so restrict to * two-dimensional paging. */ - r =3D kvm_tdp_map_page(vcpu, range->gpa | direct_bits, error_code, &level= ); + r =3D kvm_tdp_page_prefault(vcpu, range->gpa | direct_bits, error_code, &= level); if (r < 0) return r; =20 --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F28451C28E for ; Fri, 29 Aug 2025 00:06:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425992; cv=none; b=PmQQOG/OgXzBYx00JuR/T4N0xlvphrWcedxu4gPUoF+VU+EtsvUmLYnhuqOWrmB9BGVmybyL/4OpWoBrgbnMsTQe3GJEzkgI3IrbHPyXPzZcNorlQDuk4M18BX4AoIjInXKzmSfkRQ7CtPPIegqvGX3fHF+hGh8sceOl5/XgO7c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425992; c=relaxed/simple; bh=mJfwIk8zt4v2vwPMXySyeVbwDMhDnveaRkJ0ykfhjFE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j2hsQciehAJWD5P+gzfPNJMkvpPUfNnysZ+CE8vKQ2FnKxMbd1mb584comhUarJbkd6D2V8SXT5Ht26HPl/HDCDUQT4qAN4tjNn2M8uC0tweYc+Gn4NJezR+q4NMNlpvc9Rr6K6TtJbDmsIs7P0NNbN5hhg+xBcxc2SAY9J2vqA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=K78A8uzY; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="K78A8uzY" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-32769224506so1463290a91.2 for ; Thu, 28 Aug 2025 17:06:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425990; x=1757030790; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=wV1Wcluuc6pxP1WLIJ1v7G0uRRxI7XfMddwIt6Dax6U=; b=K78A8uzYtqvFpR8f47ABNQ2qnhkr0v3hzt8C7x731dpBd6vCDRxlzzNNuYyhA2k6xf S+qAxGH1SSArqNhF+uFvtSjPOALkP5eeF0wubW4B9eDbx3YoLeBSGrQxxaBd3ndr9SNv ESWxGmDqsdJI/socji+OgUfplCgftzi5ch7ZC4OjkqzOBsXOKOIzdgo+/4jUSMTY7ok7 fNj4u2XwVAJetY/z/J6teWVRnwITMJDgwEbhzuGN5egMEgum8U9JeCAU8N7iU7YtoGUQ r3/625ud/xjB1KJuZXNmWL8xqcoc+nQEQIPigqoxVfsiv9FMO6xL9SkBiQrQkarXf0u0 47Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425990; x=1757030790; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wV1Wcluuc6pxP1WLIJ1v7G0uRRxI7XfMddwIt6Dax6U=; b=XK9pRpFUtv2m6VmcahD87KK6f94QKwBD0mS8j/x9/ZBS7lPtSNMm3HI0FlSTyXVIT1 hzNMj3SqjyhJcXn8Vm+6oyfXuKDpUgA6E+L9g2YNhLt2fM3sWPID+jbW9LJn/lJ/T2x2 NbEJxILlgi5kebmF9dSg34TrNMXTu0Ph32J9QwNXG8HX5/Fdj/BESUpapHbnpwZdPU/Z 90c8bycdGracRQL80DH7CXap7Z8eqcXqLMdODSrZ1fT62nBdpHZ3BxOtWk+YJV/1qW/y 01Ax+/W2WSau/feo4PK/RFCNfingk6+Uq/bt4U8uNp4W5q7T7i8ZQpcN5aed6y04lyJ4 Xk7g== X-Forwarded-Encrypted: i=1; AJvYcCXIKvz9wsxlu0VuAL4WM5uGp69cW8lNMpFNSgTMDhTa96Ct+IQhoKY9fWqp1VYuj+Tg7+n2Pjjb4r2t7Cw=@vger.kernel.org X-Gm-Message-State: AOJu0YzkwGzoeAzzyw/+vm/MPvTKG+WJcwcceMnzCWw+GPWMQq42K5Yr pCIkyABlDGaFtVJDhG46bIx1XUH6bz4AOy5MXFn07t4Q0TUVBisKZd9eBb3f2tEKKUSh5PTK884 GXVN1Kw== X-Google-Smtp-Source: AGHT+IGfNSHJC1tKpnIi9nf7ERhW+CnNYjAM/RgFZvKWfrFvpVxfSo19+e7Hp391VLd1R2FN0hILyDkjQJU= X-Received: from pjyp15.prod.google.com ([2002:a17:90a:e70f:b0:312:e266:f849]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:384c:b0:31e:d9f0:9b96 with SMTP id 98e67ed59e1d1-32515ef8acbmr32399421a91.14.1756425990471; Thu, 28 Aug 2025 17:06:30 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:05 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-6-seanjc@google.com> Subject: [RFC PATCH v2 05/18] KVM: TDX: Drop superfluous page pinning in S-EPT management From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yan Zhao Don't explicitly pin pages when mapping pages into the S-EPT, guest_memfd doesn't support page migration in any capacity, i.e. there are no migrate callbacks because guest_memfd pages *can't* be migrated. See the WARN in kvm_gmem_migrate_folio(). Eliminating TDX's explicit pinning will also enable guest_memfd to support in-place conversion between shared and private memory[1][2]. Because KVM cannot distinguish between speculative/transient refcounts and the intentional refcount for TDX on private pages[3], failing to release private page refcount in TDX could cause guest_memfd to indefinitely wait on decreasing the refcount for the splitting. Under normal conditions, not holding an extra page refcount in TDX is safe because guest_memfd ensures pages are retained until its invalidation notification to KVM MMU is completed. However, if there're bugs in KVM/TDX module, not holding an extra refcount when a page is mapped in S-EPT could result in a page being released from guest_memfd while still mapped in the S-EPT. But, doing work to make a fatal error slightly less fatal is a net negative when that extra work adds complexity and confusion. Several approaches were considered to address the refcount issue, including - Attempting to modify the KVM unmap operation to return a failure, which was deemed too complex and potentially incorrect[4]. - Increasing the folio reference count only upon S-EPT zapping failure[5]. - Use page flags or page_ext to indicate a page is still used by TDX[6], which does not work for HVO (HugeTLB Vmemmap Optimization). - Setting HWPOISON bit or leveraging folio_set_hugetlb_hwpoison()[7]. Due to the complexity or inappropriateness of these approaches, and the fact that S-EPT zapping failure is currently only possible when there are bugs in the KVM or TDX module, which is very rare in a production kernel, a straightforward approach of simply not holding the page reference count in TDX was chosen[8]. When S-EPT zapping errors occur, KVM_BUG_ON() is invoked to kick off all vCPUs and mark the VM as dead. Although there is a potential window that a private page mapped in the S-EPT could be reallocated and used outside the VM, the loud warning from KVM_BUG_ON() should provide sufficient debug information. To be robust against bugs, the user can enable panic_on_warn as normal. Link: https://lore.kernel.org/all/cover.1747264138.git.ackerleytng@google.c= om [1] Link: https://youtu.be/UnBKahkAon4 [2] Link: https://lore.kernel.org/all/CAGtprH_ypohFy9TOJ8Emm_roT4XbQUtLKZNFcM6F= r+fhTFkE0Q@mail.gmail.com [3] Link: https://lore.kernel.org/all/aEEEJbTzlncbRaRA@yzhao56-desk.sh.intel.co= m [4] Link: https://lore.kernel.org/all/aE%2Fq9VKkmaCcuwpU@yzhao56-desk.sh.intel.= com [5] Link: https://lore.kernel.org/all/aFkeBtuNBN1RrDAJ@yzhao56-desk.sh.intel.co= m [6] Link: https://lore.kernel.org/all/diqzy0tikran.fsf@ackerleytng-ctop.c.googl= ers.com [7] Link: https://lore.kernel.org/all/53ea5239f8ef9d8df9af593647243c10435fd219.= camel@intel.com [8] Suggested-by: Vishal Annapurve Suggested-by: Ackerley Tng Suggested-by: Rick Edgecombe Signed-off-by: Yan Zhao Reviewed-by: Ira Weiny Reviewed-by: Kai Huang [sean: extract out of hugepage series, massage changelog accordingly] Signed-off-by: Sean Christopherson Reviewed-by: Binbin Wu --- arch/x86/kvm/vmx/tdx.c | 28 ++++------------------------ 1 file changed, 4 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index c83e1ff02827..f24f8635b433 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1586,29 +1586,22 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t = root_hpa, int pgd_level) td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa); } =20 -static void tdx_unpin(struct kvm *kvm, struct page *page) -{ - put_page(page); -} - static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, - enum pg_level level, struct page *page) + enum pg_level level, kvm_pfn_t pfn) { int tdx_level =3D pg_level_to_tdx_sept_level(level); struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + struct page *page =3D pfn_to_page(pfn); gpa_t gpa =3D gfn_to_gpa(gfn); u64 entry, level_state; u64 err; =20 err =3D tdh_mem_page_aug(&kvm_tdx->td, gpa, tdx_level, page, &entry, &lev= el_state); - if (unlikely(tdx_operand_busy(err))) { - tdx_unpin(kvm, page); + if (unlikely(tdx_operand_busy(err))) return -EBUSY; - } =20 if (KVM_BUG_ON(err, kvm)) { pr_tdx_error_2(TDH_MEM_PAGE_AUG, err, entry, level_state); - tdx_unpin(kvm, page); return -EIO; } =20 @@ -1642,29 +1635,18 @@ static int tdx_sept_set_private_spte(struct kvm *kv= m, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) { struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - struct page *page =3D pfn_to_page(pfn); =20 /* TODO: handle large pages. */ if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) return -EINVAL; =20 - /* - * Because guest_memfd doesn't support page migration with - * a_ops->migrate_folio (yet), no callback is triggered for KVM on page - * migration. Until guest_memfd supports page migration, prevent page - * migration. - * TODO: Once guest_memfd introduces callback on page migration, - * implement it and remove get_page/put_page(). - */ - get_page(page); - /* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching * barrier in tdx_td_finalize(). */ smp_rmb(); if (likely(kvm_tdx->state =3D=3D TD_STATE_RUNNABLE)) - return tdx_mem_page_aug(kvm, gfn, level, page); + return tdx_mem_page_aug(kvm, gfn, level, pfn); =20 return tdx_mem_page_record_premap_cnt(kvm, gfn, level, pfn); } @@ -1715,7 +1697,6 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm= , gfn_t gfn, return -EIO; } tdx_clear_page(page); - tdx_unpin(kvm, page); return 0; } =20 @@ -1795,7 +1776,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm,= gfn_t gfn, if (tdx_is_sept_zap_err_due_to_premap(kvm_tdx, err, entry, level) && !KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) { atomic64_dec(&kvm_tdx->nr_premapped); - tdx_unpin(kvm, page); return 0; } =20 --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCB131A9F80 for ; Fri, 29 Aug 2025 00:06:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425994; cv=none; b=SRPcp7P8vGYUoPZXipjfwibCIGRtDHXs5v9opA7Qr5v8GSDXYC+Ub5t41kzFHksLA5/q5OxG9YnF6B5qYC0vtwp/ZVUKAOREqYLJFY9Gnx6S6bHoQiDRT2cjADv2XJcX56jrmSjTir3Uq0QxQHGKZu3QoEr+wZtQ+UIIuW5PK5M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425994; c=relaxed/simple; bh=guMBlrm/rcFiyVpqt0hr4LlDYx3tQGzdjshtf5uFvKE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ay67DMK8nDldLWXha4jHh7NVuqMXfm4Gx82ajQemCaSWcU4bB5ia0xFJmKgCD6wcRPXuLzgLTw9eIP9YE/x6QBRaLGthkM4t1C7hUGUtNVRhGHlvh13DegqXDVq+87oWRHCkkwDikllEJkZwAV7fHymCsRgm4LmM7bHQf60h3Po= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yCd7pOGw; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yCd7pOGw" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-771e4a8b533so1434123b3a.3 for ; Thu, 28 Aug 2025 17:06:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425992; x=1757030792; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=pc+C5r9UxosSdMvNvv24ZE703L5ZdmGhUK9UzcuK6dE=; b=yCd7pOGwWQVHc3T8jDXWjaOQEvGvz2haegZuBWqdzMNHUqizvtOYi9EmzkRDU4U0lH bvzfB7EwvnCSzREPtP4YRB8DHa20BeDiQBtkhbAzJem1hMh9xAGpqgWAJJ+a3XyjNhpr Abc6IPZAKoJOQdiJkdiNPsNz0i09/ArKyYPT3UP1mBAIvzJyXJKbxP0pFVDPE2ZSghIP IZj0hZ1jpafRXfBBHtSfPHz0WORJBs5i1RTs051wIFQ22mWQRUy69sTP7YLM3Zak/nP6 WlWdiC1gBCOCmPHodNFHqfn6bRfLmA19SvKX0v7u8zz9yNZvwceIcyTW5j67fWeBxdHE i9Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425992; x=1757030792; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pc+C5r9UxosSdMvNvv24ZE703L5ZdmGhUK9UzcuK6dE=; b=CgWjWgnNaXcsSpMKFyRDG9U0EmF4huFElZmZXt+0FOB2/DSJCsm2njMYio//Df3Ajp J/PgJUWKUC58qcxfZ5ATJbsK0c7t+0KpddzAHrtt+7aXsWz4d55JBHrdPq6BHhCjgqxT ZuUAmra9yQpPzAlpj8uAQ+0cD2mN7Vs+5ZIVaI7JjZnlMrpSiVQ5R/QxpWYXLwOSodFP SQkJ0U5R5+BNR8FNn1NLl66CSyn05dl0A5byTjvSAguEHg1tClmBoEbH3htQtSq+kV3e 8wvG5uh+AQsyKbUm32BG+zVOQwzzU5g+p+IPSwzHoGh1PvQea3ow/WwLXwiB6F+lOqW4 oaRA== X-Forwarded-Encrypted: i=1; AJvYcCWGZ2Ac/QY6KULl3g8loJigQbsfYjn7vxjZoUsJF09+VYVgeHoPgi25mVADU7Y9QEnDtXu6kpKr4F0aXWk=@vger.kernel.org X-Gm-Message-State: AOJu0YyzUEF/21W2WCogzNuEmU0Zd7lONCEKMMwqKPa6p+8gLqa2JbC/ ibYd3fq4ZnZ0NSItoU8FMs00E0SZlZ24jF+o1ui3z6M9X0kagX3MkfKF2EbNKxwxUX/bnySsVLO 4/y+ZCg== X-Google-Smtp-Source: AGHT+IGsQEGpqIkG4palZicWIZvUBh20hOQat16whhSvJD3x6nNIGewZDiu5LE2a2PmQKQicnX6o2AzYyqk= X-Received: from pfbca20.prod.google.com ([2002:a05:6a00:4194:b0:76c:4298:869f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:88b:b0:749:bc7:1577 with SMTP id d2e1a72fcca58-7702fa4f67emr31500813b3a.9.1756425992004; Thu, 28 Aug 2025 17:06:32 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:06 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-7-seanjc@google.com> Subject: [RFC PATCH v2 06/18] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Return -EIO when a KVM_BUG_ON() is tripped, as KVM's ABI is to return -EIO when a VM has been killed due to a KVM bug, not -EINVAL. Note, many (all?) of the affected paths never propagate the error code to userspace, i.e. this is about internal consistency more than anything else. Signed-off-by: Sean Christopherson Reviewed-by: Binbin Wu Reviewed-by: Ira Weiny Reviewed-by: Rick Edgecombe --- arch/x86/kvm/vmx/tdx.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index f24f8635b433..50a9d81dad53 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1624,7 +1624,7 @@ static int tdx_mem_page_record_premap_cnt(struct kvm = *kvm, gfn_t gfn, struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); =20 if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm)) - return -EINVAL; + return -EIO; =20 /* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc(&kvm_tdx->nr_premapped); @@ -1638,7 +1638,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm,= gfn_t gfn, =20 /* TODO: handle large pages. */ if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) - return -EINVAL; + return -EIO; =20 /* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1661,10 +1661,10 @@ static int tdx_sept_drop_private_spte(struct kvm *k= vm, gfn_t gfn, =20 /* TODO: handle large pages. */ if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) - return -EINVAL; + return -EIO; =20 if (KVM_BUG_ON(!is_hkid_assigned(kvm_tdx), kvm)) - return -EINVAL; + return -EIO; =20 /* * When zapping private page, write lock is held. So no race condition @@ -1849,7 +1849,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm,= gfn_t gfn, * and slot move/deletion. */ if (KVM_BUG_ON(is_hkid_assigned(kvm_tdx), kvm)) - return -EINVAL; + return -EIO; =20 /* * The HKID assigned to this TD was already freed and cache was @@ -1870,7 +1870,7 @@ static int tdx_sept_remove_private_spte(struct kvm *k= vm, gfn_t gfn, * there can't be anything populated in the private EPT. */ if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm)) - return -EINVAL; + return -EIO; =20 ret =3D tdx_sept_zap_private_spte(kvm, gfn, level, page); if (ret <=3D 0) --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A65971D61BB for ; Fri, 29 Aug 2025 00:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425996; cv=none; b=hjovnYuhA4d3DGWYYmd1KiAAce0M1xGFQhT29+KBCFyaFY1xWEvTxx4NjducwcoGfSmzF90z9/s2Cy/LgZtpEhUrK0pGvV2A8GkuafoR/ajtLayAGm1JEP5hlWPiMOw/J/qYgiVmpNuGA3/wOluXXLSTteJaxV4MbJqSUDKGITE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425996; c=relaxed/simple; bh=r/3zHVEQRVjjS59stLqVm327G9mTH4dmq7IdLAwoFjA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aFDSv7vnfPvcZwEgawPSoX1sZtlVEvJ+BcJVXjgMq01xiUqYqQlcNE6mKVYdFEEWLWNrisGjoUq1HK3gFcBPtfq/mkF/INM7viL8R8VFxBw7KilxS9ljWwKBT7EEX92oZTWjRK5KVgOQMqLRlUlUAq70IHZO6mNrD4vpQafFZ4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rfJcURXE; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rfJcURXE" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-b47173a00e8so1188296a12.1 for ; Thu, 28 Aug 2025 17:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425994; x=1757030794; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=zLX9mH2pJ6CieCfe/KZk+xGWiI2ikAnsCO1rb0udaH8=; b=rfJcURXEEL7zRWgs4vsrAWmUFw721T3oB7HrjzCmFXDOl1Bwod4vXNvsoPiFnvmu5y XF6hyzhOlAte71+iSwcjdLbglpGyrb5dmrwPZpvLJavO0oGJd1pNutM44SOUVKlWjG8v Usqt3pa2Hpq6MNaRuiCIV4nLHw26bGJfa2uhs4zXJXD2q11rSO9HfyogtHbcz65FlRyq 74tJLRny0J5hIG1mh2wpdU1m22LU4tLL+nwbX6gR80/X9xZa9mxl9EKZJxii8qcE0kP1 5Hwk5W/I89v9bXJVBkarLzk+EAV/iwkRGBb9nc/eKaAJaWq4GWwV/Ybgf/ztY0zQf2ZF 8JQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425994; x=1757030794; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zLX9mH2pJ6CieCfe/KZk+xGWiI2ikAnsCO1rb0udaH8=; b=mmXswPhvWrTcv5hDt9MdBctIZTnRv1uQNoO5pyC767x1dupltovaJZl3wbdONEZyD2 Ow0/aUjiF06pyvouUt5g3dsQroZTGZUJHEuPVbIswGZHGbRfTNVhCtRVRcvKHJNMXKlB 49Gzj8gXrSkSpJiN3bsNwt9oZ1KkIuij+7VZRqAPXqXm8hNCnMddtWrDQ0ZxYt8N/zTh GQvmS5S3TW/v/JsKVXbZbMlYCX9670yse5ZbzbOJ0cKx/6g3EfWPNY10edAqgE7q57bE 0Y04JKEIGVkpzLwyMTR+nObqpPTmxWySNB+kcXLj9EeRq9abp/KAm1OifYvf1Ep1GLyN 48NA== X-Forwarded-Encrypted: i=1; AJvYcCX3VcLAoFIDvwJuDghUqZwGNxepaYH/DQcFfLEBZn6onjv237VfOX/GLmzTMgEIw5ThyQRJkQ1hHYfNEC4=@vger.kernel.org X-Gm-Message-State: AOJu0YwZSC6jjcxVB/Fet73eRdcpekboxsFvcBgLkTey/oRoCyRTHh6j QzKlvGrLc9L81F6Gws645dL+y+hhfyBcAGW0et1aojB3MSbgauuAqU5/h8+SX2Y+5gbirPvOjli KI3o4yw== X-Google-Smtp-Source: AGHT+IFYNfjkckW49CdVPG0lAwBRb8biFN/AaTS3cgVRThu+320Td61aGs0d05hkOzNiLg/kzzdXrDa63Mc= X-Received: from plbmg12.prod.google.com ([2002:a17:903:348c:b0:248:d665:58c7]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2286:b0:248:75da:f791 with SMTP id d9443c01a7336-24875dafd03mr143314385ad.47.1756425993908; Thu, 28 Aug 2025 17:06:33 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:07 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-8-seanjc@google.com> Subject: [RFC PATCH v2 07/18] KVM: TDX: Fold tdx_sept_drop_private_spte() into tdx_sept_remove_private_spte() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold tdx_sept_drop_private_spte() into tdx_sept_remove_private_spte() to avoid having to differnatiate between "zap", "drop", and "remove", and to eliminate dead code due to redundant checks, e.g. on an HKID being assigned. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Binbin Wu --- arch/x86/kvm/vmx/tdx.c | 90 +++++++++++++++++++----------------------- 1 file changed, 40 insertions(+), 50 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 50a9d81dad53..8cb6a2627eb2 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1651,55 +1651,6 @@ static int tdx_sept_set_private_spte(struct kvm *kvm= , gfn_t gfn, return tdx_mem_page_record_premap_cnt(kvm, gfn, level, pfn); } =20 -static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, - enum pg_level level, struct page *page) -{ - int tdx_level =3D pg_level_to_tdx_sept_level(level); - struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - gpa_t gpa =3D gfn_to_gpa(gfn); - u64 err, entry, level_state; - - /* TODO: handle large pages. */ - if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) - return -EIO; - - if (KVM_BUG_ON(!is_hkid_assigned(kvm_tdx), kvm)) - return -EIO; - - /* - * When zapping private page, write lock is held. So no race condition - * with other vcpu sept operation. - * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs. - */ - err =3D tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry, - &level_state); - - if (unlikely(tdx_operand_busy(err))) { - /* - * The second retry is expected to succeed after kicking off all - * other vCPUs and prevent them from invoking TDH.VP.ENTER. - */ - tdx_no_vcpus_enter_start(kvm); - err =3D tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry, - &level_state); - tdx_no_vcpus_enter_stop(kvm); - } - - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_PAGE_REMOVE, err, entry, level_state); - return -EIO; - } - - err =3D tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page); - - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err); - return -EIO; - } - tdx_clear_page(page); - return 0; -} - static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level, void *private_spt) { @@ -1861,7 +1812,11 @@ static int tdx_sept_free_private_spt(struct kvm *kvm= , gfn_t gfn, static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) { + int tdx_level =3D pg_level_to_tdx_sept_level(level); + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); struct page *page =3D pfn_to_page(pfn); + gpa_t gpa =3D gfn_to_gpa(gfn); + u64 err, entry, level_state; int ret; =20 /* @@ -1872,6 +1827,10 @@ static int tdx_sept_remove_private_spte(struct kvm *= kvm, gfn_t gfn, if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm)) return -EIO; =20 + /* TODO: handle large pages. */ + if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) + return -EIO; + ret =3D tdx_sept_zap_private_spte(kvm, gfn, level, page); if (ret <=3D 0) return ret; @@ -1882,7 +1841,38 @@ static int tdx_sept_remove_private_spte(struct kvm *= kvm, gfn_t gfn, */ tdx_track(kvm); =20 - return tdx_sept_drop_private_spte(kvm, gfn, level, page); + /* + * When zapping private page, write lock is held. So no race condition + * with other vcpu sept operation. + * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs. + */ + err =3D tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry, + &level_state); + + if (unlikely(tdx_operand_busy(err))) { + /* + * The second retry is expected to succeed after kicking off all + * other vCPUs and prevent them from invoking TDH.VP.ENTER. + */ + tdx_no_vcpus_enter_start(kvm); + err =3D tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry, + &level_state); + tdx_no_vcpus_enter_stop(kvm); + } + + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error_2(TDH_MEM_PAGE_REMOVE, err, entry, level_state); + return -EIO; + } + + err =3D tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page); + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err); + return -EIO; + } + + tdx_clear_page(page); + return 0; } =20 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 149561E3769 for ; Fri, 29 Aug 2025 00:06:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425997; cv=none; b=nDzmF4Q07WP8RaBTuYlE1YNXkG73q4TumkPWOznv5wiWZMdvFqh4l+A1gooIn9iTc0+AHYV/YZuOjBrTZiw6CH2gKZJYGUTpAAUpIe/Pwo3xD/AqeFTKu+oQ4sx9kP2YCkNs8up6tdPNCn6YrdX4+ZyZwXZXaB9GDDqA2Xe8Esk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425997; c=relaxed/simple; bh=JfzT5/FL1aVHecSHpgPG+2w3eYSbVpsJ5usNCfH68Rc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=L98IOcZszNYouOakC7i8e+gZms+ali32CfX7Ama9O0i3sznAAAXeN6AjQnFVYyGI6IoVV8tWCahGjBZLGZvwxhgZTVrJHYdxOl6H/DnH6ucJ7rUbbtx2+WKI8XCnwmGPEj6f4KK2uWkNx/pnsGK7H3GqyMuULKBPZ5MefoSbm+w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=trB5XW/a; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="trB5XW/a" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-24458274406so28398835ad.3 for ; Thu, 28 Aug 2025 17:06:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425995; x=1757030795; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=azRRbHHODPAI7Q8DbQGHZhxFKo0H66S6km5uv5YyGbc=; b=trB5XW/aoEgvXi+VZVcQJ8r9KYecwgqEaYhAOfF96l3arqKDyuS7QA8ADvnvR3XKW/ +UJ0VGk/GCXJXFxWEpkqLQbLkFb5Z3SiRxQMeA3tMlzlFXyf7U6HcTOWhmcMIxWuZwVQ b3hY6I0X8rOcP6vNSYfRU3FH4jgJyDkyiuGmHfCCT1ui934kNkSPAfCtAWHkWtnIxmgQ adcsNyEgYWGUTzVrOdDD5HruJbfz+EnLc7fofT1JvtwTmrmD+OvOby9PqqDu/clJiEo6 tKJIuetsOXVFtJgmcBPlfr+dTvR95YAGaC9IP1Avm/HQ6WOshmnJnbEjAq5anl9idhhz murg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425995; x=1757030795; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=azRRbHHODPAI7Q8DbQGHZhxFKo0H66S6km5uv5YyGbc=; b=tf0pRIPrnqLkM+iIMoc5Oe441WkAtyEIVGu5DmerhT+UgZMPCkyqOOSVXWIBhuG2D4 jHB6Cw3opsgItuHar9SgAi4rYnvlHEHc6J4z4CtXIgMrsQtsXWXGpg1w6HaqwpY0cbqT 7Wy9Zlzc6Y5LpNMW1utHvdIzqqbMvg6HqGjLDGAXEg+tHGiXogBicZ2OHoiZGGGflN9Z SBOwBxn2iA2VURmLYZEsNM/JqrqoknRbF3SWpWGHS2QeWFSUOWd/pAXae+B/EzZ5JmDT G/wODMGWI/BnnrGg2/oE1Wg//OcfL7gB6rzlIPiV2Fq0hSKShehCaLKOCq/Fxw37LCZ9 3CXA== X-Forwarded-Encrypted: i=1; AJvYcCWywS1FZI5/cmtXcVjk9a3Y6dq1kcgdtqey7hHqWGFDmSE1Ok9kvzbZpuqp2oar7eXtcrTjmL3eae9FzgE=@vger.kernel.org X-Gm-Message-State: AOJu0YwLs/SnYH5fiOmAmVSniQJoP7dxXkfbRqXATft7FrTjDgfxI6gy cmgNKj0ol/iivSBvk4SGx8/cY1JiG6/COlFggz754NuSYKtaAvckIheYPXSaoyPpBVs7FHF1MPX ed8YhsQ== X-Google-Smtp-Source: AGHT+IFwuejFD8NgObdi1RH1kHcvd/8tHC7uqcH6Kw+5g86hzjv1+yfj38VQpRu3YeTnyRiyW+LZbxmigXQ= X-Received: from pjj6.prod.google.com ([2002:a17:90b:5546:b0:31f:6ddd:ef5]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:f543:b0:246:7a11:3775 with SMTP id d9443c01a7336-2467a113b1amr262897495ad.48.1756425995581; Thu, 28 Aug 2025 17:06:35 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:08 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-9-seanjc@google.com> Subject: [RFC PATCH v2 08/18] KVM: x86/mmu: Drop the return code from kvm_x86_ops.remove_external_spte() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop the return code from kvm_x86_ops.remove_external_spte(), a.k.a. tdx_sept_remove_private_spte(), as KVM simply does a KVM_BUG_ON() failure, and that KVM_BUG_ON() is redundant since all error paths in TDX also do a KVM_BUG_ON(). Opportunistically pass the spte instead of the pfn, as the API is clearly about removing an spte. Suggested-by: Rick Edgecombe Signed-off-by: Sean Christopherson Reviewed-by: Binbin Wu --- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 8 ++------ arch/x86/kvm/vmx/tdx.c | 17 ++++++++--------- 3 files changed, 12 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 0d3cc0fc27af..d0a8404a6b8f 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1852,8 +1852,8 @@ struct kvm_x86_ops { void *external_spt); =20 /* Update external page table from spte getting removed, and flush TLB. */ - int (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level lev= el, - kvm_pfn_t pfn_for_gfn); + void (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level le= vel, + u64 spte); =20 bool (*has_wbinvd_exit)(void); =20 diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 3ea2dd64ce72..78ee085f7cbc 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -362,9 +362,6 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct k= vm_mmu_page *sp) static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, int level) { - kvm_pfn_t old_pfn =3D spte_to_pfn(old_spte); - int ret; - /* * External (TDX) SPTEs are limited to PG_LEVEL_4K, and external * PTs are removed in a special order, involving free_external_spt(). @@ -377,9 +374,8 @@ static void remove_external_spte(struct kvm *kvm, gfn_t= gfn, u64 old_spte, =20 /* Zapping leaf spte is allowed only when write lock is held. */ lockdep_assert_held_write(&kvm->mmu_lock); - /* Because write lock is held, operation should success. */ - ret =3D kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_pfn); - KVM_BUG_ON(ret, kvm); + + kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_spte); } =20 /** diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 8cb6a2627eb2..07f9ad1fbfb6 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1809,12 +1809,12 @@ static int tdx_sept_free_private_spt(struct kvm *kv= m, gfn_t gfn, return tdx_reclaim_page(virt_to_page(private_spt)); } =20 -static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, - enum pg_level level, kvm_pfn_t pfn) +static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, u64 spte) { + struct page *page =3D pfn_to_page(spte_to_pfn(spte)); int tdx_level =3D pg_level_to_tdx_sept_level(level); struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - struct page *page =3D pfn_to_page(pfn); gpa_t gpa =3D gfn_to_gpa(gfn); u64 err, entry, level_state; int ret; @@ -1825,15 +1825,15 @@ static int tdx_sept_remove_private_spte(struct kvm = *kvm, gfn_t gfn, * there can't be anything populated in the private EPT. */ if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm)) - return -EIO; + return; =20 /* TODO: handle large pages. */ if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) - return -EIO; + return; =20 ret =3D tdx_sept_zap_private_spte(kvm, gfn, level, page); if (ret <=3D 0) - return ret; + return; =20 /* * TDX requires TLB tracking before dropping private page. Do @@ -1862,17 +1862,16 @@ static int tdx_sept_remove_private_spte(struct kvm = *kvm, gfn_t gfn, =20 if (KVM_BUG_ON(err, kvm)) { pr_tdx_error_2(TDH_MEM_PAGE_REMOVE, err, entry, level_state); - return -EIO; + return; } =20 err =3D tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page); if (KVM_BUG_ON(err, kvm)) { pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err); - return -EIO; + return; } =20 tdx_clear_page(page); - return 0; } =20 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A67B81FBEB1 for ; Fri, 29 Aug 2025 00:06:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425999; cv=none; b=R4C9pHr6Skbb1+QW3KxRy47tOuyUunvk5BwUG0TvlXPAZD/f0QRSaUcrse3VfzV2zUW6shDogInqT1LZzirFWIONWVPpRk1JjgIpqivWQ0l9/x5WAd2BWhSzGk4NHsDN331avyOkmO2jZ+bCaOI3jpsnyNVvFh9pE7gxK1A9zko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756425999; c=relaxed/simple; bh=f3WKeCxN8el5k5LrAboMphhH9rXooGmaPRmdkWif71U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=o9z3/CXvkg9f+ymisSOMbDiFWxzBuMPDv8ZWDyf4ZvrE74JpzcZH8uof4eXQVQHlW3W5ZCQ6So1GOPboXdkeXfZu0aVuswjNQ8MR/N1h6mmNR7Melu67W531fVK8w5SHv74TL/SwYEojGv71ywdrsXO2fEhXDqZzxxTPHu9LKeU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fDvguf3x; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fDvguf3x" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b4c746c020cso531948a12.1 for ; Thu, 28 Aug 2025 17:06:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425997; x=1757030797; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=4vFObST+7ZlqgxAzKSnwUtFLpnriAXTWXVO/mc38gG8=; b=fDvguf3xTaXUj/7eMva20CV20zom1ivQbtzX4XSrEczWsbaOvTIu748zZGfQz02Mrv 3om5ov8zdIokmJUghjpJdxQEt+jcq5KZLtcs9ykQC2shCO22vCYIoYAq2ABDVq60lROW ghtBz+BcvWMNZuIRXeaHJxJondQJSRsKqe2/zNs8Nsr3zXtC8/7KoZSW8modSmNzYUFw H3qToRpGuqLV74vHee1ttymDZR1vEw3m5hpt/V5w6j1RXGnkTUOIU+eoNG5XBei5nfUJ 2/I351mtQ9xWBV70129km/2NCtQ38KDt94ieCrF+Uc9yn9czyJ5H1j5XYbgivXFUhies If9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425997; x=1757030797; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4vFObST+7ZlqgxAzKSnwUtFLpnriAXTWXVO/mc38gG8=; b=sP/NO5oMm67JzL5qB3y6JCQOf+AAVw0Y+ozIZW8Ira+8ljN5pvDCK2Asn0LwWVQ0l6 c2tMjux08AaDAV560ntwv9zUdrNPGJxfwm1Mji6HiX58AiCYv+ZmX663aeej5iaLWjLu UGpPSVx9JHOlyyhra2u9caK7pdPZZljyHNgY1wb3yRrDX4hIL0Jz6OUtPXvMu0wz/may qM/Sv+TJUIbnp2bd3Cpu3q85xzEXaDEQS642HwmxoQh5UYgXyUSFxGsxwvCT12vXiKpe cUakl31Gd/gugw8Dic/+DsaEVy0Pm74Qs2Y6wwZ+rmbkbbFp0xKHrxzDOVnJoULIMaaN I+iQ== X-Forwarded-Encrypted: i=1; AJvYcCXUR73qGDgC69oN5ljGZPiQ3iVOObCoiGfTwKsfBs6IpKJdsr4s2iGlmQACAYexgxz8V2FDaurAeukru7c=@vger.kernel.org X-Gm-Message-State: AOJu0YyRU+KC0t7QBnbicFlnyx7sKuGCmCpwmuXJQ9rBTQysDEaY3i5R l6lHw9jeTVtyhcOFGg8tQm+VJPZNXAHhOX43trNkFxLzDI1dkhoftZYocAI2k+DlC2ftVT+MDhX FwF0JdA== X-Google-Smtp-Source: AGHT+IEG9WztooY4bnn6abzlnsjj0dOg/1Kl7H1cw1QcQlPgDxUccZQ1mG0Culs2cO/x0pGpmGMyNJwe/+c= X-Received: from pgww7.prod.google.com ([2002:a05:6a02:2c87:b0:b47:8e6:9c33]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:7488:b0:243:9b05:ac7d with SMTP id adf61e73a8af0-2439b05b008mr14000596637.36.1756425997067; Thu, 28 Aug 2025 17:06:37 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:09 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-10-seanjc@google.com> Subject: [RFC PATCH v2 09/18] KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Return -EIO immediately from tdx_sept_zap_private_spte() if the number of to-be-added pages underflows, so that the following "KVM_BUG_ON(err, kvm)" isn't also triggered. Isolating the check from the "is premap error" if-statement will also allow adding a lockdep assertion that premap errors are encountered if and only if slots_lock is held. Reviewed-by: Rick Edgecombe Signed-off-by: Sean Christopherson Reviewed-by: Binbin Wu --- arch/x86/kvm/vmx/tdx.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 07f9ad1fbfb6..cafd618ca43c 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1724,8 +1724,10 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm= , gfn_t gfn, err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level= _state); tdx_no_vcpus_enter_stop(kvm); } - if (tdx_is_sept_zap_err_due_to_premap(kvm_tdx, err, entry, level) && - !KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) { + if (tdx_is_sept_zap_err_due_to_premap(kvm_tdx, err, entry, level)) { + if (KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) + return -EIO; + atomic64_dec(&kvm_tdx->nr_premapped); return 0; } --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A367217716 for ; Fri, 29 Aug 2025 00:06:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426002; cv=none; b=ZjXzBqsjb861zPNwx/bgcshnnOj83AvfShBwTrPUsdkFyAZjK6gEuL9HEGW+AMfoQyGJhGM2erH7eUBUTQiQkloasr4RuDXQVQDGLJTWuGVP+rtSj+gUXjbGcKaNSRgMjQcrf08agGbZBN3e3+FQc7DWdPy2+Aaf5JZtcQ25ag0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426002; c=relaxed/simple; bh=gyZpzdfigGs09+ImBbJAsRRHU5xktCFn0Yw4iaqicyc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rAD5gstW920EieLdFGG9aCWauDH+TLd0SEtDgd0CKmLauFBjJeLDDcruymAIYifP3ajSl0vCRA5u30sCVQbKqTp/YI6mfA+W7t8XUbA0UiT97TRQggkW3cF9tNl9CHquim0GxmosR9YeGtgGOFNA/uGvbc6vF+XqCwRq9wl9FA4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=iWNrXZTC; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="iWNrXZTC" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-3276575ae5eso1598871a91.1 for ; Thu, 28 Aug 2025 17:06:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756425999; x=1757030799; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ZwIMlaifmPTtVhzroBYxRLA8wwFTYLN4Sjt92uj7DWY=; b=iWNrXZTC+uVaaFU5eSNGiMo0IzTNPcS0L0iY8hZaoiECryVm2Fb1XWas14NZKUsBEz vcmHqQJ3MegEnZaP318JkRdsKWGoSLVIxvu7DJaEpTirImAZXgdX+RRpQJ+QHgFcCdwb 7Xr/BUnezzapGgp/cwDCZ+oJ8ZFGffg+bKRh0UH4Y/whrNgi5H2q7NlI1LHNmsZApVhl VO4VbsnycnuqMgJ3TrgGkCn5vit9DAqQC4ImI5BoVxHbfg6guWV3bK9592HtOdnXz1YA GSjP3IT8g4vsbvn5asAFPvYJqbfWbDxM+aJJDZqo1XISUKLl6QXcUBb8xaY5JNqC9t11 jBBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756425999; x=1757030799; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZwIMlaifmPTtVhzroBYxRLA8wwFTYLN4Sjt92uj7DWY=; b=k7DHZqY1jUHjX2b4jgER2GY5XrN5R7QqOKKd/XZsz9j+fk4S06dCprTOGDyaiPgYLB Oys/Y9J5wfmxAKXmSsrIlJX6v/ML9iB6uO/506MdB5Yo8itnFObhkj1aQEJiGK0d+AOZ LsfvbsKb3nHRnSnhq7hW/Nb9ArZP0zrRSBsCeTkvNHc4e8tSAQMK80VuaQdtPrnzpETN oIherFcIPVEW41TFihb+XSiYAhxgrXTOBZP7hiZSIcyjgXCorOVsmS8dRkUmPbphhk9B ZmkUsCpO/UOTJxeqwMLpm52eYCZLwbluimdTnHS1Vefep1H2XWHUNYNd1B1dRzNjIXNV g8Ug== X-Forwarded-Encrypted: i=1; AJvYcCXTFmlj7lITXbWPXuKqPQuTIqNoJaZPl7eIGEILiQoZHevyfLSk6umEO1rq2TsH2h/lHronzOKxTnkc5sU=@vger.kernel.org X-Gm-Message-State: AOJu0Yz3jtZ8NbYsf3G86RBFx5skoL7+kQmW5hSK2JCM4bjWoyjNt2B/ DRkeYA1ESZSkJdd9SzNYfpcFRl+hYPeO0EvcfjPODtl0GuX3Sghw7Dz5CmdxHrMwra7ztA7we5E 9Pq2wtg== X-Google-Smtp-Source: AGHT+IF5M1lm57K1T4cESxSZmN/2EWRqfoLHHHsMaumxyGGp3Yd/18q0BTU+oBYQUA0CJrPoD1MkMyVS590= X-Received: from pjbcz5.prod.google.com ([2002:a17:90a:d445:b0:327:50fa:eff9]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1c06:b0:327:e59d:2cc2 with SMTP id 98e67ed59e1d1-327e59d2f16mr639721a91.10.1756425998989; Thu, 28 Aug 2025 17:06:38 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:10 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-11-seanjc@google.com> Subject: [RFC PATCH v2 10/18] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use atomic64_dec_return() when decrementing the number of "pre-mapped" S-EPT pages to ensure that the count can't go negative without KVM noticing. In theory, checking for '0' and then decrementing in a separate operation could miss a 0=3D>-1 transition. In practice, such a condition is impossible because nr_premapped is protected by slots_lock, i.e. doesn't actually need to be an atomic (that wart will be addressed shortly). Don't bother trying to keep the count non-negative, as the KVM_BUG_ON() ensures the VM is dead, i.e. there's no point in trying to limp along. Reviewed-by: Rick Edgecombe Reviewed-by: Ira Weiny Signed-off-by: Sean Christopherson Reviewed-by: Binbin Wu --- arch/x86/kvm/vmx/tdx.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index cafd618ca43c..fe0815d542e3 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1725,10 +1725,9 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm= , gfn_t gfn, tdx_no_vcpus_enter_stop(kvm); } if (tdx_is_sept_zap_err_due_to_premap(kvm_tdx, err, entry, level)) { - if (KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) + if (KVM_BUG_ON(atomic64_dec_return(&kvm_tdx->nr_premapped) < 0, kvm)) return -EIO; =20 - atomic64_dec(&kvm_tdx->nr_premapped); return 0; } =20 @@ -3151,8 +3150,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gf= n_t gfn, kvm_pfn_t pfn, goto out; } =20 - if (!KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) - atomic64_dec(&kvm_tdx->nr_premapped); + KVM_BUG_ON(atomic64_dec_return(&kvm_tdx->nr_premapped) < 0, kvm); =20 if (arg->flags & KVM_TDX_MEASURE_MEMORY_REGION) { for (i =3D 0; i < PAGE_SIZE; i +=3D TDX_EXTENDMR_CHUNKSIZE) { --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 616F3220F38 for ; Fri, 29 Aug 2025 00:06:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426002; cv=none; b=uatOARwlK8a0Y3lOe88ANXAVDkKp0Vtnsso/Lrf+RXlHMffax55Kn+Vae4C0q41c/LaE4hdIYhg6YSWpk84qND1D+4GpdtF29/hF5MiHz+xR9V9xtgggfYYYqBprze+J4nsBp1KTcGwo+ysVXwVD1cyGFZ5L1n3zrvFJD5fTzNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426002; c=relaxed/simple; bh=+ii6r4N6rkBoNCWhPTC+aQwPcbsLf6WTE0C1KJpqLtM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RE3AUP1+MxPKOFKocr50uC8neLST/clHPAVJD7CZ6deakfRJdsWAOlEz58B4TLX1eDUpK3FTncKw8Er4NRH1vMkNrvC4kyH8Kn2IJjaZBOQYVvy93ubi614lH2NsgFXOdy9POO6XRbuJIzj/y2S1JLJap2O2v27Xqp401U1rD2c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qPyb8w9o; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qPyb8w9o" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-327594fd627so1489863a91.3 for ; Thu, 28 Aug 2025 17:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426001; x=1757030801; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=1YmgvG2RcaVdULnIb8Wec5M7oDs3P0onCjyYEMew5Ns=; b=qPyb8w9oWmCXYr15dtSP4y5Ym3U+tKxk9ExRv/uNuve/ywT4vsZkY30nQjUUv37JX0 i3BJ2fdtVLXy/Lq6vgmKuqwkVzDyWqCozmlgGrB7AgQ+VvY2vLiIYmymBU95Iw7GO54t a5XmwxX2qSftIXRJI9XECd4CHaZlbZWZYQplK00ZpWAly+gap8W7+FSO+YfKiWAsdoK3 uWpOZd8ZQgyi71uilknbSFVeS6K0wc75sb+G9Hg0KdfQstZixQIcyGSOXO49rpfMQJF/ l+ODr18EQaUDmUMw8BR6KWBUU+33Jpp/szgisUGg13dnfqr/o2bNU14z+jjdF9wowTZh j8iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426001; x=1757030801; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1YmgvG2RcaVdULnIb8Wec5M7oDs3P0onCjyYEMew5Ns=; b=si/9wc4ajwEDXUwQf6v2BEBxwdjVtZkqh6XZ6oTedw0Q+GzlbliPRnxRv2QEqn7V2X A+7M8kdkVbQbEI5OqWMZZLsRqEvg8CZPsP5iMdnpHV3Mw7/OU0/hWAgUSKmy4MizQxbD Zg5tK5+vQUMJBmJvu1rb1WaugetwBy2txx9MfukQ8NJO4ErrNm2jJt3shzBpV2f9xSb/ 5arSxa1BjmfBjFJS+VKoMCmOMkboC6CEZwJy34AivkY90cYZahEOpsZBr1QgjXdRuaH7 RJP76IZ9c7FBBfC2/rbivYpa5JMUxpE+T1M/E/dtFUsqEWQOhLjSX7D6WqwYNQ6H9PQb RB0w== X-Forwarded-Encrypted: i=1; AJvYcCX0iQR0KecUQtO8+TIpzjz3r6BI9AmPYJs24c6OwWBppaBzeLWmORtWsT2aroiqcTh3zbyFLmC0B1BcLrM=@vger.kernel.org X-Gm-Message-State: AOJu0YyjybTj9qYEUsOgUw1a5S3NRa0oXf0lpNI1NxzjuOG7cdge9l30 QhPrukzU5Vf5Xlr/efWxSjh//brqk2/P1o+Nb5U9HNfLikDu+3AvuLr58L2lIpwqSFsol3gys74 8tFinzA== X-Google-Smtp-Source: AGHT+IEf2CEnlJGHZtsmzrHqeOd1ndTjoACnl8CdMuKToTrbMjiAb+CrTNoHK0d+D+dNU6wwE2PLQDR3nAM= X-Received: from pjbsk5.prod.google.com ([2002:a17:90b:2dc5:b0:325:9cb3:419e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3c45:b0:327:b824:2257 with SMTP id 98e67ed59e1d1-327b8242bbemr4417101a91.32.1756426000789; Thu, 28 Aug 2025 17:06:40 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:11 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-12-seanjc@google.com> Subject: [RFC PATCH v2 11/18] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold tdx_mem_page_record_premap_cnt() into tdx_sept_set_private_spte() as providing a one-off helper for effectively three lines of code is at best a wash, and splitting the code makes the comment for smp_rmb() _extremely_ confusing as the comment talks about reading kvm->arch.pre_fault_allowed before kvm_tdx->state, but the immediately visible code does the exact opposite. Opportunistically rewrite the comments to more explicitly explain who is checking what, as well as _why_ the ordering matters. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/vmx/tdx.c | 49 ++++++++++++++++++------------------------ 1 file changed, 21 insertions(+), 28 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index fe0815d542e3..06dd2861eba7 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1608,29 +1608,6 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t g= fn, return 0; } =20 -/* - * KVM_TDX_INIT_MEM_REGION calls kvm_gmem_populate() to map guest pages; t= he - * callback tdx_gmem_post_populate() then maps pages into private memory. - * through the a seamcall TDH.MEM.PAGE.ADD(). The SEAMCALL also requires = the - * private EPT structures for the page to have been built before, which is - * done via kvm_tdp_map_page(). nr_premapped counts the number of pages th= at - * were added to the EPT structures but not added with TDH.MEM.PAGE.ADD(). - * The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there - * are no half-initialized shared EPT pages. - */ -static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn, - enum pg_level level, kvm_pfn_t pfn) -{ - struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - - if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm)) - return -EIO; - - /* nr_premapped will be decreased when tdh_mem_page_add() is called. */ - atomic64_inc(&kvm_tdx->nr_premapped); - return 0; -} - static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) { @@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kv= m, gfn_t gfn, return -EIO; =20 /* - * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching - * barrier in tdx_td_finalize(). + * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() + * before kvm_tdx->state. Userspace must not be allowed to pre-fault + * arbitrary memory until the initial memory image is finalized. Pairs + * with the smp_wmb() in tdx_td_finalize(). */ smp_rmb(); - if (likely(kvm_tdx->state =3D=3D TD_STATE_RUNNABLE)) - return tdx_mem_page_aug(kvm, gfn, level, pfn); =20 - return tdx_mem_page_record_premap_cnt(kvm, gfn, level, pfn); + /* + * If the TD isn't finalized/runnable, then userspace is initializing + * the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of + * pages that need to be mapped and initialized via TDH.MEM.PAGE.ADD. + * KVM_TDX_FINALIZE_VM checks the counter to ensure all mapped pages + * have been added to the image, to prevent running the TD with a + * valid mapping in the mirror EPT, but not in the S-EPT. + */ + if (unlikely(kvm_tdx->state !=3D TD_STATE_RUNNABLE)) { + if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm)) + return -EIO; + + atomic64_inc(&kvm_tdx->nr_premapped); + return 0; + } + + return tdx_mem_page_aug(kvm, gfn, level, pfn); } =20 static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F8FA223311 for ; Fri, 29 Aug 2025 00:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426004; cv=none; b=rj0cbdED42PvrNT+zeMpPgDlEADTNosea8vly7iTQ0qdqvX5VxfZ0LvcW78vtu3qrLdrIO6/QbTGP/K2UmIqVJfEWApNP1oZCk8oazKc2gUPlAz4XmhduddvqjIaYgbAHRnCr7bcejcHi5VnESyivrwMwvYF4SW4kG1X3ixsLg4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426004; c=relaxed/simple; bh=902UVUZmCSPpi8X+dDv5xOlcB4XyEnFTmnxe/BF6V3o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qo3nReOA4AICu+d0Wzhhjn9WA0Ime7aaSxXMpsCr0isGXQN33HfGPVCJytpdyM8pf2xXQpZVDsCvscF9uTImvqMyOK6qmYwfNqrNctuXS/WzwQjXl29h9wjGzd4oRirHy7ec6Sau2C3kFyXGSf51JRXL1ZOq0Sc+ERExQriEL7g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=osvxEVSf; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="osvxEVSf" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-76e2e60221fso2378488b3a.0 for ; Thu, 28 Aug 2025 17:06:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426002; x=1757030802; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=I8qRi06qGVaXga/Xlpn8ZvFudxvmpq5UoDX0cKOu9ig=; b=osvxEVSf/PKfE62dS8D7rWg+JomvMX7Tm8vM17OIhVXLOCegJ3FO0HxNK1kqgyBtoP JwN10rBUrHhWs1jtXMkzpLMwE/3xCrytbv5JGcNhggUOGkONzmvCbJScMIudkBq3QNfG gwozla2eWgr18PmeZfCxGG/yTlcZ44l6EHD4qPvhKVIGcaD7Ln1kOZUXkAt0FWtoI30Y Gx334suTFe9lcHSscY+06VjYKnvwqemXpvhBSI8Fm+UGnNQfHyGFaffEZdSn8pNfXuq6 SQW30xbvud0OjH25urGpS6b+Dsz+MJ1+W5KKaBZ/w5dWoyu7Xbh9NIcvHkFqgPdlcymL Lfzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426002; x=1757030802; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=I8qRi06qGVaXga/Xlpn8ZvFudxvmpq5UoDX0cKOu9ig=; b=r4L9dETnT4wKfaMCjviH3InQl2FaSK0DtTm4ItlAPmYRe9IQ951IoZEe75wxTVZ4MV YmlKgjNooooCET+T7+jK+CvFGJm2IRFiSmbRYWrWR7EJaA6G6+Y++exQ2V4weqTKeHsV 0ayHK1hsGtCDZCx7ycyFrlviP3K+aHTdAFIgkMtRMzeRpcG39RKrbOOqtiTlxUdSAGhY JgO7eCeyTMm70dyoKY5+EVsa3T9vT83+0U5ydRH7d5x4JVqNXZ4H3eRxrNj0pTwCbWlo ainfQ3bGWb8BLA73nUj+giXYTmW9bPgJAZkKXjdYXzYXJHrA1KsephfmwJuJGFJZ+mVx O2nA== X-Forwarded-Encrypted: i=1; AJvYcCVgluziJ2N+BnNEvprvGIYXUMTAg2C/xqjpV722mfeKsb7zOlL2e6ZVwJxBqyvUywg/xLn5SkYPm8Dx0Po=@vger.kernel.org X-Gm-Message-State: AOJu0YzmPHpOMVrWJ6YFEyqBtUGVLcaG6/TyFpsa/eJayQMm2RTpzyQJ Q6SYzUpoSPH+s07RktaCXBjvRNNj62j/H43cRj6OcDo2Cv6RJfSq28YXnV6uo5jckS2Q69EAnQU 2fm3Gcw== X-Google-Smtp-Source: AGHT+IFg+0Lyt/ONHma24rkTg8rYgWVOQSWd8hCU5VIhmhLogULJMsRMy1cHwlhM00wbNXuZW4pEW5RvOgc= X-Received: from pfbhg14.prod.google.com ([2002:a05:6a00:860e:b0:771:e396:a7fa]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:1a94:b0:771:e8be:8390 with SMTP id d2e1a72fcca58-771e8be87b2mr21526977b3a.14.1756426002390; Thu, 28 Aug 2025 17:06:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:12 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-13-seanjc@google.com> Subject: [RFC PATCH v2 12/18] KVM: TDX: Bug the VM if extended the initial measurement fails From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" WARN and terminate the VM if TDH_MR_EXTEND fails, as extending the measurement should fail if and only if there is a KVM bug, or if the S-EPT mapping is invalid, and it should be impossibe for the S-EPT mappings to be removed between kvm_tdp_mmu_map_private_pfn() and tdh_mr_extend(). Holding slots_lock prevents zaps due to memslot updates, filemap_invalidate_lock() prevents zaps due to guest_memfd PUNCH_HOLE, and all usage of kvm_zap_gfn_range() is mutually exclusive with S-EPT entries that can be used for the initial image. The call from sev.c is obviously mutually exclusive, TDX disallows KVM_X86_QUIRK_IGNORE_GUEST_PAT so same goes for kvm_noncoherent_dma_assignment_start_or_stop, and while __kvm_set_or_clear_apicv_inhibit() can likely be tripped while building the image, the APIC page has its own non-guest_memfd memslot and so can't be used for the initial image, which means that too is mutually exclusive. Opportunistically switch to "goto" to jump around the measurement code, partly to make it clear that KVM needs to bail entirely if extending the measurement fails, partly in anticipation of reworking how and when TDH_MEM_PAGE_ADD is done. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/tdx.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 06dd2861eba7..bc92e87a1dbb 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -3145,14 +3145,22 @@ static int tdx_gmem_post_populate(struct kvm *kvm, = gfn_t gfn, kvm_pfn_t pfn, =20 KVM_BUG_ON(atomic64_dec_return(&kvm_tdx->nr_premapped) < 0, kvm); =20 - if (arg->flags & KVM_TDX_MEASURE_MEMORY_REGION) { - for (i =3D 0; i < PAGE_SIZE; i +=3D TDX_EXTENDMR_CHUNKSIZE) { - err =3D tdh_mr_extend(&kvm_tdx->td, gpa + i, &entry, - &level_state); - if (err) { - ret =3D -EIO; - break; - } + if (!(arg->flags & KVM_TDX_MEASURE_MEMORY_REGION)) + goto out; + + /* + * Note, MR.EXTEND can fail if the S-EPT mapping is somehow removed + * between mapping the pfn and now, but slots_lock prevents memslot + * updates, filemap_invalidate_lock() prevents guest_memfd updates, + * mmu_notifier events can't reach S-EPT entries, and KVM's internal + * zapping flows are mutually exclusive with S-EPT mappings. + */ + for (i =3D 0; i < PAGE_SIZE; i +=3D TDX_EXTENDMR_CHUNKSIZE) { + err =3D tdh_mr_extend(&kvm_tdx->td, gpa + i, &entry, &level_state); + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error_2(TDH_MR_EXTEND, err, entry, level_state); + ret =3D -EIO; + goto out; } } =20 --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6876622B5A3 for ; Fri, 29 Aug 2025 00:06:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426006; cv=none; b=YB0ytiIGJcj39kyLMTIEL4AHGA76RUjUl/fQxeR4CzObpTzb5oksxts79MqZHraK2Mg2a9UEedTU+fO3XVcOpbLxAxnacTfTuRGfQNuae49OqQvnI3de2ee33O7zoefaEyjQDMZgCDxzv9/CCY0dMKiPRarfJjYQfH7JAQfGC8k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426006; c=relaxed/simple; bh=HyfJiBpVHCuZlDqW7ohHtriaFUQ49KsER+ZnpREL/g0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pjOf1aeIZkwLIY7C3bu58mQbSzDMm9fPXQCeV3/2PPCxEZUzpZB6QrVzFgcBxHX3Njr0xCX/y9XRcHKvK3DWcaW5SRHuXJ9Fxx6eVkuIwbZNz/HY+5qEHjITOOVJ4OrRibT9bEUdCDkR2JRyjS8F86yGszfGscTYPmR76/LlYVI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mu27k08e; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mu27k08e" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b4c409a09a8so2095040a12.2 for ; Thu, 28 Aug 2025 17:06:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426004; x=1757030804; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=VbpGnyEHWXwz/TRJVnLJd8uLlBlZrJ7BSh7u4XEu1X0=; b=mu27k08eJO5NS6o6o/EpfW0m+Ocm4lhTPqS9Xs1UPeXKhsYJoNIJE3DrwMsQV/sj7T xQpNLKm4FWzo5X6ljrp5I6yPMtGbtHuNSvqSCV1iOcRkhvTIs+gPMXLa5P9UmnIKzAPN TNL1Rni1QyyGSf6SFPdW1QL4kkBX/iRml+DSJCg0idfFPvE4W5BL8tLCw9gjrBpo9xgV uTuHYzIca2sJHqe7ZTDCH00c4kKqdCsDPzHeujXln4ky0hH1lT7vOpMYTbQq7qyKUsAl jH0W+LUT19ZKlDJR0w3oayPPnZzgepBAlGKPlE8t/Kzn04Rd5wmt+ELNtAvl8D12Ig7A 7P+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426004; x=1757030804; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VbpGnyEHWXwz/TRJVnLJd8uLlBlZrJ7BSh7u4XEu1X0=; b=YhOCo/Tp/PH3PNyN24f0yoiGXdn5OESaYQBCM8k9C/lD55iMXvAhn4vsoCT6d76RB8 lBQGyv/VlkgaDWosS4CMRu9EiOuBVaeK8YQSQezq6HvatT0t3R1484zZ130OeN97mLvW 5HE6ruOzWDb7OzING3Py2ViaiLF9XQDoy7MCsJa1p4Giu+Br4UB7IQ9msz1tdRGZUZ0J N+wKWPDek6JXkLW63zq+wlRet9UcLigQwn22DUJf0E9Ou+XVgAO0QoFeGw5traRaDkmC wozswlBmeRQo6RJ3JwADCteE2KYrLCDFazstgollTH7tNUyJ4V2K1vblu3dn11qTXIB1 d4rw== X-Forwarded-Encrypted: i=1; AJvYcCWTZF3XEoB4w8J432vUvm1vfR4nEPdQ8fxLgvkpUjlFBMLXGZs2v85cr7wkdTrFyrzWgbZ4rkGdpAMteOU=@vger.kernel.org X-Gm-Message-State: AOJu0YxlwzlDuyS3HVvJbk68NU8CYoe3JDbPVa8hn8oEqMKRaM6//2Nf qzCrAKq/k2N2JDwy5ReDn1NI6ot/V4W4+DAdFIeJVsJOxdVpugllUlw78a0g95jJdp3G5t1T0cP ETHV85Q== X-Google-Smtp-Source: AGHT+IHNpwYw1dAX53OCyvKMsMk+1BB8l5v84bcH2U+gB7FSG3HLavCtoR3umBBmr4HKgqJiAUiYcNGZOBQ= X-Received: from pghi20.prod.google.com ([2002:a63:e914:0:b0:b49:dd13:ba03]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:2449:b0:243:b018:f8a5 with SMTP id adf61e73a8af0-243b018fbc2mr5965084637.6.1756426003918; Thu, 28 Aug 2025 17:06:43 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:13 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-14-seanjc@google.com> Subject: [RFC PATCH v2 13/18] KVM: TDX: ADD pages to the TD image while populating mirror EPT entries From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When populating the initial memory image for a TDX guest, ADD pages to the TD as part of establishing the mappings in the mirror EPT, as opposed to creating the mappings and then doing ADD after the fact. Doing ADD in the S-EPT callbacks eliminates the need to track "premapped" pages, as the mirror EPT (M-EPT) and S-EPT are always synchronized, e.g. if ADD fails, KVM reverts to the previous M-EPT entry (guaranteed to be !PRESENT). Eliminating the hole where the M-EPT can have a mapping that doesn't exist in the S-EPT in turn obviates the need to handle errors that are unique to encountering a missing S-EPT entry (see tdx_is_sept_zap_err_due_to_premap()= ). Keeping the M-EPT and S-EPT synchronized also eliminates the need to check for unconsumed "premap" entries during tdx_td_finalize(), as there simply can't be any such entries. Dropping that check in particular reduces the overall cognitive load, as the managemented of nr_premapped with respect to removal of S-EPT is _very_ subtle. E.g. successful removal of an S-EPT entry after it completed ADD doesn't adjust nr_premapped, but it's not clear why that's "ok" but having half-baked entries is not (it's not truly "ok" in that removing pages from the image will likely prevent the guest from booting, but from KVM's perspective it's "ok"). Doing ADD in the S-EPT path requires passing an argument via a scratch field, but the current approach of tracking the number of "premapped" pages effectively does the same. And the "premapped" counter is much more dangerous, as it doesn't have a singular lock to protect its usage, since nr_premapped can be modified as soon as mmu_lock is dropped, at least in theory. I.e. nr_premapped is guarded by slots_lock, but only for "happy" paths. Note, this approach was used/tried at various points in TDX development, but was ultimately discarded due to a desire to avoid stashing temporary state in kvm_tdx. But as above, KVM ended up with such state anyways, and fully committing to using temporary state provides better access rules (100% guarded by slots_lock), and makes several edge cases flat out impossible. Note #2, continue to extend the measurement outside of mmu_lock, as it's a slow operation (typically 16 SEAMCALLs per page whose data is included in the measurement), and doesn't *need* to be done under mmu_lock, e.g. for consistency purposes. However, MR.EXTEND isn't _that_ slow, e.g. ~1ms latency to measure a full page, so if it needs to be done under mmu_lock in the future, e.g. because KVM gains a flow that can remove S-EPT entries uring KVM_TDX_INIT_MEM_REGION, then extending the measurement can also be moved into the S-EPT mapping path (again, only if absolutely necessary). P.S. _If_ MR.EXTEND is moved into the S-EPT path, take care not to return an error up the stack if TDH_MR_EXTEND fails, as removing the M-EPT entry but not the S-EPT entry would result in inconsistent state! Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/vmx/tdx.c | 116 ++++++++++++++--------------------------- arch/x86/kvm/vmx/tdx.h | 8 ++- 2 files changed, 46 insertions(+), 78 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index bc92e87a1dbb..00c3dc376690 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1586,6 +1586,32 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t r= oot_hpa, int pgd_level) td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa); } =20 +static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level leve= l, + kvm_pfn_t pfn) +{ + struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); + u64 err, entry, level_state; + gpa_t gpa =3D gfn_to_gpa(gfn); + + lockdep_assert_held(&kvm->slots_lock); + + if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm) || + KVM_BUG_ON(!kvm_tdx->page_add_src, kvm)) + return -EIO; + + err =3D tdh_mem_page_add(&kvm_tdx->td, gpa, pfn_to_page(pfn), + kvm_tdx->page_add_src, &entry, &level_state); + if (unlikely(tdx_operand_busy(err))) + return -EBUSY; + + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error_2(TDH_MEM_PAGE_ADD, err, entry, level_state); + return -EIO; + } + + return 0; +} + static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) { @@ -1627,19 +1653,10 @@ static int tdx_sept_set_private_spte(struct kvm *kv= m, gfn_t gfn, =20 /* * If the TD isn't finalized/runnable, then userspace is initializing - * the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of - * pages that need to be mapped and initialized via TDH.MEM.PAGE.ADD. - * KVM_TDX_FINALIZE_VM checks the counter to ensure all mapped pages - * have been added to the image, to prevent running the TD with a - * valid mapping in the mirror EPT, but not in the S-EPT. + * the VM image via KVM_TDX_INIT_MEM_REGION; ADD the page to the TD. */ - if (unlikely(kvm_tdx->state !=3D TD_STATE_RUNNABLE)) { - if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm)) - return -EIO; - - atomic64_inc(&kvm_tdx->nr_premapped); - return 0; - } + if (unlikely(kvm_tdx->state !=3D TD_STATE_RUNNABLE)) + return tdx_mem_page_add(kvm, gfn, level, pfn); =20 return tdx_mem_page_aug(kvm, gfn, level, pfn); } @@ -1665,39 +1682,6 @@ static int tdx_sept_link_private_spt(struct kvm *kvm= , gfn_t gfn, return 0; } =20 -/* - * Check if the error returned from a SEPT zap SEAMCALL is due to that a p= age is - * mapped by KVM_TDX_INIT_MEM_REGION without tdh_mem_page_add() being call= ed - * successfully. - * - * Since tdh_mem_sept_add() must have been invoked successfully before a - * non-leaf entry present in the mirrored page table, the SEPT ZAP related - * SEAMCALLs should not encounter err TDX_EPT_WALK_FAILED. They should ins= tead - * find TDX_EPT_ENTRY_STATE_INCORRECT due to an empty leaf entry found in = the - * SEPT. - * - * Further check if the returned entry from SEPT walking is with RWX permi= ssions - * to filter out anything unexpected. - * - * Note: @level is pg_level, not the tdx_level. The tdx_level extracted fr= om - * level_state returned from a SEAMCALL error is the same as that passed i= nto - * the SEAMCALL. - */ -static int tdx_is_sept_zap_err_due_to_premap(struct kvm_tdx *kvm_tdx, u64 = err, - u64 entry, int level) -{ - if (!err || kvm_tdx->state =3D=3D TD_STATE_RUNNABLE) - return false; - - if (err !=3D (TDX_EPT_ENTRY_STATE_INCORRECT | TDX_OPERAND_ID_RCX)) - return false; - - if ((is_last_spte(entry, level) && (entry & VMX_EPT_RWX_MASK))) - return false; - - return true; -} - static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, struct page *page) { @@ -1717,12 +1701,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm= , gfn_t gfn, err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level= _state); tdx_no_vcpus_enter_stop(kvm); } - if (tdx_is_sept_zap_err_due_to_premap(kvm_tdx, err, entry, level)) { - if (KVM_BUG_ON(atomic64_dec_return(&kvm_tdx->nr_premapped) < 0, kvm)) - return -EIO; - - return 0; - } =20 if (KVM_BUG_ON(err, kvm)) { pr_tdx_error_2(TDH_MEM_RANGE_BLOCK, err, entry, level_state); @@ -2827,12 +2805,6 @@ static int tdx_td_finalize(struct kvm *kvm, struct k= vm_tdx_cmd *cmd) =20 if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state =3D=3D TD_STATE_RUNNABLE) return -EINVAL; - /* - * Pages are pending for KVM_TDX_INIT_MEM_REGION to issue - * TDH.MEM.PAGE.ADD(). - */ - if (atomic64_read(&kvm_tdx->nr_premapped)) - return -EINVAL; =20 cmd->hw_error =3D tdh_mr_finalize(&kvm_tdx->td); if (tdx_operand_busy(cmd->hw_error)) @@ -3116,11 +3088,14 @@ static int tdx_gmem_post_populate(struct kvm *kvm, = gfn_t gfn, kvm_pfn_t pfn, { struct tdx_gmem_post_populate_arg *arg =3D _arg; struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - u64 err, entry, level_state; gpa_t gpa =3D gfn_to_gpa(gfn); + u64 err, entry, level_state; struct page *src_page; int ret, i; =20 + if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm)) + return -EIO; + /* * Get the source page if it has been faulted in. Return failure if the * source page has been swapped out or unmapped in primary memory. @@ -3131,22 +3106,14 @@ static int tdx_gmem_post_populate(struct kvm *kvm, = gfn_t gfn, kvm_pfn_t pfn, if (ret !=3D 1) return -ENOMEM; =20 + kvm_tdx->page_add_src =3D src_page; ret =3D kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn); - if (ret < 0) - goto out; + kvm_tdx->page_add_src =3D NULL; =20 - ret =3D 0; - err =3D tdh_mem_page_add(&kvm_tdx->td, gpa, pfn_to_page(pfn), - src_page, &entry, &level_state); - if (err) { - ret =3D unlikely(tdx_operand_busy(err)) ? -EBUSY : -EIO; - goto out; - } + put_page(src_page); =20 - KVM_BUG_ON(atomic64_dec_return(&kvm_tdx->nr_premapped) < 0, kvm); - - if (!(arg->flags & KVM_TDX_MEASURE_MEMORY_REGION)) - goto out; + if (ret || !(arg->flags & KVM_TDX_MEASURE_MEMORY_REGION)) + return ret; =20 /* * Note, MR.EXTEND can fail if the S-EPT mapping is somehow removed @@ -3159,14 +3126,11 @@ static int tdx_gmem_post_populate(struct kvm *kvm, = gfn_t gfn, kvm_pfn_t pfn, err =3D tdh_mr_extend(&kvm_tdx->td, gpa + i, &entry, &level_state); if (KVM_BUG_ON(err, kvm)) { pr_tdx_error_2(TDH_MR_EXTEND, err, entry, level_state); - ret =3D -EIO; - goto out; + return -EIO; } } =20 -out: - put_page(src_page); - return ret; + return 0; } =20 static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_= cmd *cmd) diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index ca39a9391db1..1b00adbbaf77 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -36,8 +36,12 @@ struct kvm_tdx { =20 struct tdx_td td; =20 - /* For KVM_TDX_INIT_MEM_REGION. */ - atomic64_t nr_premapped; + /* + * Scratch pointer used to pass the source page to tdx_mem_page_add. + * Protected by slots_lock, and non-NULL only when mapping a private + * pfn via tdx_gmem_post_populate(). + */ + struct page *page_add_src; =20 /* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 354C4239E97 for ; Fri, 29 Aug 2025 00:06:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426007; cv=none; b=PVQJlVWXaNZGyjDB07bNbbENscdDTPIix6h8m4k+xzBYHk59P21GArLGsiMjLZox1sEJ1UC8aNCO1zNBwwKtpvl1Jp6nMg6eLE6bJ+7tvi5WgdZfwCS5jnMKf/fj+K/C9eKwFlBekSCnmtBCeNFMu4AQ+ASbaebj7YQufbb3+ys= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426007; c=relaxed/simple; bh=Jokid0DO/rV7NnsZCR0z+fHP5XwwBXdL/NPXkdhHIK0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TimQB0WMwlINzMLLf57nuI4cQ4fsiKILk3uchY7BFJ27kyPKfWBUTaqAQo9EYiGNNs42aUbgsVDL+eKifZO6h6DJTVWzIkxpw0Y8TB2QKJ6umz7V7k8GQdV+iaOflD4TJNrsVCwf05rkweO9OtWlNHSM1u4NbK08S3nAQpeBk4o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tJJcdevi; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tJJcdevi" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b4c73924056so512167a12.3 for ; Thu, 28 Aug 2025 17:06:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426005; x=1757030805; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=U7cQHpMeOyPtobGJP9EpNG+eqgAlak3ByWO0DfJC83o=; b=tJJcdevizseaGw4GX+6mb2XBsSb/79QNcKkntmuMIZRTuv29Zm8cflhsaG6jjzafhC 8MtXhq5eSmGbY33bKTqsV5Fv0aqHkl1d2DS3C2+lbfKd+GBLsggt87aVa+XrKW1UmJYK e/Ij/fdaxQ12lC+pLcyBosYI8EP7CySAohBykfUDk95k/IMJvwz+CE9P3d6CU2nypxSJ QbBkUupx7BSRbdYIt0xRfL5xlw+yyiju+D8KK0yr2RidiqEXHG0dY8xImMsjFVzeLmTr yD3jS5y2UTTtsQsttPxguly7lWPOLGXaRePV989bjfQWnDLbXg4OOQ01YanbicuSM2IK lMgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426005; x=1757030805; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=U7cQHpMeOyPtobGJP9EpNG+eqgAlak3ByWO0DfJC83o=; b=EjgRuMb+6Bn8/L0Y34GJJVg8ngZW2jPmqJaNPiTff8QUgPEEwyJ80emVTzUiBZboMm q0lTRbBSqbYbiLepCCWud71yUKLRi9Cfg5y8EvOlELszP0OjDD6TkeMGJQil/raGngaD B2Jvi79F2GD/SmkbM1gHbYA2VLVTSd6KsP+1GMDCJeTWuLcMjh4/zboPBZ6SdJanM3Fe Zewp6G6lvKgqL+NOrvf/kYf/L8I0BrLxDjm5cJFEfCMp4Kh9FU2pB4AQpcx2tbJA49K5 vJ0VJOEqOgIF9/EuU0hcqp7af2H1H7IEBpsEU3teyz0krz0+zloq+sS9LCTjC3M9iigC TmLw== X-Forwarded-Encrypted: i=1; AJvYcCUIC2xhXLXIFLXL4TahIa/Vw4XXE4gAlH5FUVSjwkMTV5YmtjhlJdyidgGoh/qOlIiMcJJAcZTLdXD81zE=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1inGouiGgzeh1BlyWR773iwwTkGfOqnNlds6t4SgCF1AUM3z6 pqF6FzYeCaLstiO70ZSRQGLXnVJJBlkE3hya9NLrOwkPWXDA14HVOJAI6dyI8NkwcNPuTN/qrG0 i+KlZ0A== X-Google-Smtp-Source: AGHT+IFUjUY1nfQcAvMaxHbMyGGpEn+Y2J7C9qldGHayAPHPs3JLhhD2fMtARg3auKDuKGakE4vLVFSBP0g= X-Received: from pfbjo20.prod.google.com ([2002:a05:6a00:9094:b0:771:e6da:3861]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:7347:b0:243:b190:d139 with SMTP id adf61e73a8af0-243b190d746mr5814517637.39.1756426005459; Thu, 28 Aug 2025 17:06:45 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:14 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-15-seanjc@google.com> Subject: [RFC PATCH v2 14/18] KVM: TDX: Fold tdx_sept_zap_private_spte() into tdx_sept_remove_private_spte() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do TDH_MEM_RANGE_BLOCK directly in tdx_sept_remove_private_spte() instead of using a one-off helper now that the nr_premapped tracking is gone. Opportunistically drop the WARN on hugepages, which was dead code (see the KVM_BUG_ON() in tdx_sept_remove_private_spte()). No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/vmx/tdx.c | 41 +++++++++++------------------------------ 1 file changed, 11 insertions(+), 30 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 00c3dc376690..aa6d88629dae 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1682,33 +1682,6 @@ static int tdx_sept_link_private_spt(struct kvm *kvm= , gfn_t gfn, return 0; } =20 -static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, - enum pg_level level, struct page *page) -{ - int tdx_level =3D pg_level_to_tdx_sept_level(level); - struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - gpa_t gpa =3D gfn_to_gpa(gfn) & KVM_HPAGE_MASK(level); - u64 err, entry, level_state; - - /* For now large page isn't supported yet. */ - WARN_ON_ONCE(level !=3D PG_LEVEL_4K); - - err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level_= state); - - if (unlikely(tdx_operand_busy(err))) { - /* After no vCPUs enter, the second retry is expected to succeed */ - tdx_no_vcpus_enter_start(kvm); - err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level= _state); - tdx_no_vcpus_enter_stop(kvm); - } - - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_RANGE_BLOCK, err, entry, level_state); - return -EIO; - } - return 1; -} - /* * Ensure shared and private EPTs to be flushed on all vCPUs. * tdh_mem_track() is the only caller that increases TD epoch. An increase= in @@ -1789,7 +1762,6 @@ static void tdx_sept_remove_private_spte(struct kvm *= kvm, gfn_t gfn, struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); gpa_t gpa =3D gfn_to_gpa(gfn); u64 err, entry, level_state; - int ret; =20 /* * HKID is released after all private pages have been removed, and set @@ -1803,9 +1775,18 @@ static void tdx_sept_remove_private_spte(struct kvm = *kvm, gfn_t gfn, if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) return; =20 - ret =3D tdx_sept_zap_private_spte(kvm, gfn, level, page); - if (ret <=3D 0) + err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level_= state); + if (unlikely(tdx_operand_busy(err))) { + /* After no vCPUs enter, the second retry is expected to succeed */ + tdx_no_vcpus_enter_start(kvm); + err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level= _state); + tdx_no_vcpus_enter_stop(kvm); + } + + if (KVM_BUG_ON(err, kvm)) { + pr_tdx_error_2(TDH_MEM_RANGE_BLOCK, err, entry, level_state); return; + } =20 /* * TDX requires TLB tracking before dropping private page. Do --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E489D246762 for ; Fri, 29 Aug 2025 00:06:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426010; cv=none; b=cvKg7XNEns3XsgZ2YAmjyZJhwy/azgxFEYlWxpe6J7COT6j9VCg4G7+k8MqhdFfGxxjLPAJK4GXH85cVX/dqGUDQ1me2UzA/iE4xg9bSo04vDTYeO/EpcEosylH5Q/VZDifQUeV8m3wSYVhTWvKiYgQzcNVVRpbeD8PaTbL/kcA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426010; c=relaxed/simple; bh=qytjR+kel2w3eORLh2zH54skNQJE5v9ARQq1wlkBUKc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sW6BsyZDGbsxD7+NbFz0j/AB3TlXV8xa+RdFKbi7Eeg2O9LQZ+QwmHKJ3rtnvvbFCAx8dasvqizsXKq2F8c/OnooNxoesQVTkxCgnOLW6bpMI1BMkNyminXeZFNonXl2RemSOXfYu/ehax27PH7v89pOF4RtBU2ueDpA+9KhLvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=wGiBEFMb; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="wGiBEFMb" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7720f59c0f6so1368268b3a.3 for ; Thu, 28 Aug 2025 17:06:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426007; x=1757030807; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=+Dy3rkmYpzV2fykEfNOl1VwjG9yNy+xSxuP+2h5ZcwU=; b=wGiBEFMbCvES6/D3tnlT9pDZdZXfy7Xv4oLLZ5/LzrAgXdmXQF/HN7ejwIK3aERcSC FOWEimKc/l40HqHvCmjg+KGMThrtOgrtiZUmfpcEYrrWvZq/Lp/a/mpEn52S84M5vyDh +NKTkCEIPz2XPPOb2b0ltSzbTToUQpExiRnqItoXQCitRi6HDOHp7Xx9rAVt7n2QpcH2 kQv28XPqOPUj+0XMbe3NscGg94kZNtl12dLngcbkwhAl7nQ3yWYR5/bdQoBi9yeIJUti wmPHplEL4b7nEwM80uJq9I3DIXajCV+t4o5pkkxVOMEW/2+D/R96PG8+FdEZRXKWR9cP bYqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426007; x=1757030807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+Dy3rkmYpzV2fykEfNOl1VwjG9yNy+xSxuP+2h5ZcwU=; b=A3AwEblI70iqLXEn2X66ppBi0Ct+kTslSVnzAvXkGiyBLJLKKATVY50P+siDO1geW5 CaB5Mh1R0jc4TJC4jMOeIzdYdFNlNJCq7iBgOqECcYCl1+pUCuwZ+2NNoPYytkkZIaeV Jpky/NrJ/A+lTzsNOwt+/ocSwIohJEvmtydQA6ggyIONn5A19q/9I9P1yZCDDlB/HRE1 KO7C9Mlz8jPiOvOlMYz8yX6pvGeFY96NBo6PIIvs2IJbjzMClUoeDNOGJHAcsPppFVT+ pyA2cfnX9K/K/IgYAouUvKjZtwMN29e//9v2JpkrBAAQOAw7d905Cs16NdvWaNyKTl1I ptjA== X-Forwarded-Encrypted: i=1; AJvYcCXofX37gWhw8x0UjI07qZXIjv++VOKV2d0wU4ey7+JtdYlTBBVLDiyDfgkmOP0STZwqKnFdtHSplDECAyc=@vger.kernel.org X-Gm-Message-State: AOJu0Yy6cNl8oMHf4ORb+Av85a4t9JzxwzPVJTCUXCquab2XJEhuIgjM 72toNUIgRcG1Yig8LUYQJnmm2q9RA744LunGgj1xJdUA1LI//uc/B5i+6pLK6nAbfZinNzrTd7i wcpB6Hw== X-Google-Smtp-Source: AGHT+IGfPCg7l4hBn7D2jwXdjrEUEovsveWlHE4vLEm6cr6KCYbMUVFnNMwsEgdOByingguf4XTMW3yZ568= X-Received: from pgax22.prod.google.com ([2002:a05:6a02:2e56:b0:b42:3628:5c78]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:1585:b0:232:7c7b:1c7b with SMTP id adf61e73a8af0-24340c429e2mr40822536637.14.1756426007254; Thu, 28 Aug 2025 17:06:47 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:15 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-16-seanjc@google.com> Subject: [RFC PATCH v2 15/18] KVM: TDX: Combine KVM_BUG_ON + pr_tdx_error() into TDX_BUG_ON() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add TDX_BUG_ON() macros (with varying numbers of arguments) to deduplicate the myriad flows that do KVM_BUG_ON()/WARN_ON_ONCE() followed by a call to pr_tdx_error(). In addition to reducing boilerplate copy+paste code, this also helps ensure that KVM provides consistent handling of SEAMCALL errors. Opportunistically convert a handful of bare WARN_ON_ONCE() paths to the equivalent of KVM_BUG_ON(), i.e. have them terminate the VM. If a SEAMCALL error is fatal enough to WARN on, it's fatal enough to terminate the TD. Signed-off-by: Sean Christopherson Reviewed-by: Rick Edgecombe --- arch/x86/kvm/vmx/tdx.c | 114 +++++++++++++++++------------------------ 1 file changed, 47 insertions(+), 67 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index aa6d88629dae..df9b4496cd01 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -24,20 +24,32 @@ #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt =20 -#define pr_tdx_error(__fn, __err) \ - pr_err_ratelimited("SEAMCALL %s failed: 0x%llx\n", #__fn, __err) +#define __TDX_BUG_ON(__err, __f, __kvm, __fmt, __args...) \ +({ \ + struct kvm *_kvm =3D (__kvm); \ + bool __ret =3D !!(__err); \ + \ + if (WARN_ON_ONCE(__ret && (!_kvm || !_kvm->vm_bugged))) { \ + if (_kvm) \ + kvm_vm_bugged(_kvm); \ + pr_err_ratelimited("SEAMCALL " __f " failed: 0x%llx" __fmt "\n",\ + __err, __args); \ + } \ + unlikely(__ret); \ +}) =20 -#define __pr_tdx_error_N(__fn_str, __err, __fmt, ...) \ - pr_err_ratelimited("SEAMCALL " __fn_str " failed: 0x%llx, " __fmt, __err= , __VA_ARGS__) +#define TDX_BUG_ON(__err, __fn, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, "%s", "") =20 -#define pr_tdx_error_1(__fn, __err, __rcx) \ - __pr_tdx_error_N(#__fn, __err, "rcx 0x%llx\n", __rcx) +#define TDX_BUG_ON_1(__err, __fn, __rcx, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx", __rcx) =20 -#define pr_tdx_error_2(__fn, __err, __rcx, __rdx) \ - __pr_tdx_error_N(#__fn, __err, "rcx 0x%llx, rdx 0x%llx\n", __rcx, __rdx) +#define TDX_BUG_ON_2(__err, __fn, __rcx, __rdx, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx", __rcx, __rd= x) + +#define TDX_BUG_ON_3(__err, __fn, __rcx, __rdx, __r8, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx, r8 0x%llx", = __rcx, __rdx, __r8) =20 -#define pr_tdx_error_3(__fn, __err, __rcx, __rdx, __r8) \ - __pr_tdx_error_N(#__fn, __err, "rcx 0x%llx, rdx 0x%llx, r8 0x%llx\n", __r= cx, __rdx, __r8) =20 bool enable_tdx __ro_after_init; module_param_named(tdx, enable_tdx, bool, 0444); @@ -332,10 +344,9 @@ static int __tdx_reclaim_page(struct page *page) * before the HKID is released and control pages have also been * released at this point, so there is no possibility of contention. */ - if (WARN_ON_ONCE(err)) { - pr_tdx_error_3(TDH_PHYMEM_PAGE_RECLAIM, err, rcx, rdx, r8); + if (TDX_BUG_ON_3(err, TDH_PHYMEM_PAGE_RECLAIM, rcx, rdx, r8, NULL)) return -EIO; - } + return 0; } =20 @@ -423,8 +434,8 @@ static void tdx_flush_vp_on_cpu(struct kvm_vcpu *vcpu) return; =20 smp_call_function_single(cpu, tdx_flush_vp, &arg, 1); - if (KVM_BUG_ON(arg.err, vcpu->kvm)) - pr_tdx_error(TDH_VP_FLUSH, arg.err); + + TDX_BUG_ON(arg.err, TDH_VP_FLUSH, vcpu->kvm); } =20 void tdx_disable_virtualization_cpu(void) @@ -473,8 +484,7 @@ static void smp_func_do_phymem_cache_wb(void *unused) } =20 out: - if (WARN_ON_ONCE(err)) - pr_tdx_error(TDH_PHYMEM_CACHE_WB, err); + TDX_BUG_ON(err, TDH_PHYMEM_CACHE_WB, NULL); } =20 void tdx_mmu_release_hkid(struct kvm *kvm) @@ -513,8 +523,7 @@ void tdx_mmu_release_hkid(struct kvm *kvm) err =3D tdh_mng_vpflushdone(&kvm_tdx->td); if (err =3D=3D TDX_FLUSHVP_NOT_DONE) goto out; - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error(TDH_MNG_VPFLUSHDONE, err); + if (TDX_BUG_ON(err, TDH_MNG_VPFLUSHDONE, kvm)) { pr_err("tdh_mng_vpflushdone() failed. HKID %d is leaked.\n", kvm_tdx->hkid); goto out; @@ -537,8 +546,7 @@ void tdx_mmu_release_hkid(struct kvm *kvm) * tdh_mng_key_freeid() will fail. */ err =3D tdh_mng_key_freeid(&kvm_tdx->td); - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error(TDH_MNG_KEY_FREEID, err); + if (TDX_BUG_ON(err, TDH_MNG_KEY_FREEID, kvm)) { pr_err("tdh_mng_key_freeid() failed. HKID %d is leaked.\n", kvm_tdx->hkid); } else { @@ -589,10 +597,9 @@ static void tdx_reclaim_td_control_pages(struct kvm *k= vm) * when it is reclaiming TDCS). */ err =3D tdh_phymem_page_wbinvd_tdr(&kvm_tdx->td); - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err); + if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm)) return; - } + tdx_clear_page(kvm_tdx->td.tdr_page); =20 __free_page(kvm_tdx->td.tdr_page); @@ -615,11 +622,8 @@ static int tdx_do_tdh_mng_key_config(void *param) =20 /* TDX_RND_NO_ENTROPY related retries are handled by sc_retry() */ err =3D tdh_mng_key_config(&kvm_tdx->td); - - if (KVM_BUG_ON(err, &kvm_tdx->kvm)) { - pr_tdx_error(TDH_MNG_KEY_CONFIG, err); + if (TDX_BUG_ON(err, TDH_MNG_KEY_CONFIG, &kvm_tdx->kvm)) return -EIO; - } =20 return 0; } @@ -1604,10 +1608,8 @@ static int tdx_mem_page_add(struct kvm *kvm, gfn_t g= fn, enum pg_level level, if (unlikely(tdx_operand_busy(err))) return -EBUSY; =20 - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_PAGE_ADD, err, entry, level_state); + if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_ADD, entry, level_state, kvm)) return -EIO; - } =20 return 0; } @@ -1626,10 +1628,8 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t g= fn, if (unlikely(tdx_operand_busy(err))) return -EBUSY; =20 - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_PAGE_AUG, err, entry, level_state); + if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_AUG, entry, level_state, kvm)) return -EIO; - } =20 return 0; } @@ -1674,10 +1674,8 @@ static int tdx_sept_link_private_spt(struct kvm *kvm= , gfn_t gfn, if (unlikely(tdx_operand_busy(err))) return -EBUSY; =20 - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_SEPT_ADD, err, entry, level_state); + if (TDX_BUG_ON_2(err, TDH_MEM_SEPT_ADD, entry, level_state, kvm)) return -EIO; - } =20 return 0; } @@ -1725,8 +1723,7 @@ static void tdx_track(struct kvm *kvm) tdx_no_vcpus_enter_stop(kvm); } =20 - if (KVM_BUG_ON(err, kvm)) - pr_tdx_error(TDH_MEM_TRACK, err); + TDX_BUG_ON(err, TDH_MEM_TRACK, kvm); =20 kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); } @@ -1783,10 +1780,8 @@ static void tdx_sept_remove_private_spte(struct kvm = *kvm, gfn_t gfn, tdx_no_vcpus_enter_stop(kvm); } =20 - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_RANGE_BLOCK, err, entry, level_state); + if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm)) return; - } =20 /* * TDX requires TLB tracking before dropping private page. Do @@ -1813,16 +1808,12 @@ static void tdx_sept_remove_private_spte(struct kvm= *kvm, gfn_t gfn, tdx_no_vcpus_enter_stop(kvm); } =20 - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MEM_PAGE_REMOVE, err, entry, level_state); + if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm)) return; - } =20 err =3D tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page); - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err); + if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm)) return; - } =20 tdx_clear_page(page); } @@ -2451,8 +2442,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_p= arams *td_params, goto free_packages; } =20 - if (WARN_ON_ONCE(err)) { - pr_tdx_error(TDH_MNG_CREATE, err); + if (TDX_BUG_ON(err, TDH_MNG_CREATE, kvm)) { ret =3D -EIO; goto free_packages; } @@ -2493,8 +2483,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_p= arams *td_params, ret =3D -EAGAIN; goto teardown; } - if (WARN_ON_ONCE(err)) { - pr_tdx_error(TDH_MNG_ADDCX, err); + if (TDX_BUG_ON(err, TDH_MNG_ADDCX, kvm)) { ret =3D -EIO; goto teardown; } @@ -2511,8 +2500,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_p= arams *td_params, *seamcall_err =3D err; ret =3D -EINVAL; goto teardown; - } else if (WARN_ON_ONCE(err)) { - pr_tdx_error_1(TDH_MNG_INIT, err, rcx); + } else if (TDX_BUG_ON_1(err, TDH_MNG_INIT, rcx, kvm)) { ret =3D -EIO; goto teardown; } @@ -2790,10 +2778,8 @@ static int tdx_td_finalize(struct kvm *kvm, struct k= vm_tdx_cmd *cmd) cmd->hw_error =3D tdh_mr_finalize(&kvm_tdx->td); if (tdx_operand_busy(cmd->hw_error)) return -EBUSY; - if (KVM_BUG_ON(cmd->hw_error, kvm)) { - pr_tdx_error(TDH_MR_FINALIZE, cmd->hw_error); + if (TDX_BUG_ON(cmd->hw_error, TDH_MR_FINALIZE, kvm)) return -EIO; - } =20 kvm_tdx->state =3D TD_STATE_RUNNABLE; /* TD_STATE_RUNNABLE must be set before 'pre_fault_allowed' */ @@ -2873,16 +2859,14 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, = u64 vcpu_rcx) } =20 err =3D tdh_vp_create(&kvm_tdx->td, &tdx->vp); - if (KVM_BUG_ON(err, vcpu->kvm)) { + if (TDX_BUG_ON(err, TDH_VP_CREATE, vcpu->kvm)) { ret =3D -EIO; - pr_tdx_error(TDH_VP_CREATE, err); goto free_tdcx; } =20 for (i =3D 0; i < kvm_tdx->td.tdcx_nr_pages; i++) { err =3D tdh_vp_addcx(&tdx->vp, tdx->vp.tdcx_pages[i]); - if (KVM_BUG_ON(err, vcpu->kvm)) { - pr_tdx_error(TDH_VP_ADDCX, err); + if (TDX_BUG_ON(err, TDH_VP_ADDCX, vcpu->kvm)) { /* * Pages already added are reclaimed by the vcpu_free * method, but the rest are freed here. @@ -2896,10 +2880,8 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u= 64 vcpu_rcx) } =20 err =3D tdh_vp_init(&tdx->vp, vcpu_rcx, vcpu->vcpu_id); - if (KVM_BUG_ON(err, vcpu->kvm)) { - pr_tdx_error(TDH_VP_INIT, err); + if (TDX_BUG_ON(err, TDH_VP_INIT, vcpu->kvm)) return -EIO; - } =20 vcpu->arch.mp_state =3D KVM_MP_STATE_RUNNABLE; =20 @@ -3105,10 +3087,8 @@ static int tdx_gmem_post_populate(struct kvm *kvm, g= fn_t gfn, kvm_pfn_t pfn, */ for (i =3D 0; i < PAGE_SIZE; i +=3D TDX_EXTENDMR_CHUNKSIZE) { err =3D tdh_mr_extend(&kvm_tdx->td, gpa + i, &entry, &level_state); - if (KVM_BUG_ON(err, kvm)) { - pr_tdx_error_2(TDH_MR_EXTEND, err, entry, level_state); + if (TDX_BUG_ON_2(err, TDH_MR_EXTEND, entry, level_state, kvm)) return -EIO; - } } =20 return 0; --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84CF7246BA5 for ; Fri, 29 Aug 2025 00:06:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426010; cv=none; b=PZVZQPAcKq16qQ9Zi+vV9olg34Asvbi03Zn9YZb5WaJkEOplj7Il6YP5fpLRW97WjI89A+VsN3crWMuoLNxW86EH21nx++5njdKYxAoSknJQeCI2N+P/ZUIAPH3L67QVZxqFXBzBnLpmCRlhRSNsldTqExsjpKkdXi9xegEp9G8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426010; c=relaxed/simple; bh=Tp7vZw/4x7cT5P0MVANvtyH0Aos8EMa3Yb5qw1+gtSY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OlKSP2K/XqZX9tNhm//Xh7fSJcGjkKqKNalsxBWmBReiaDlc3x6XqSb9UB/pX6pdhvPqKkEN5vpIy8a6pcvDolxC+ZyuVWxSGWAVkTH5rEdrQcjDOXcyAtSA3FBVXZb3YL1Aa9NZ4cZDqLTQLbuBgOTK2ICa+b0T+9F/3T8EQk0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qctSZMWs; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qctSZMWs" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-772178f07f8so1238478b3a.1 for ; Thu, 28 Aug 2025 17:06:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426009; x=1757030809; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=lGptiAZO7CmFsu6Op41QCgd3wduY0PZh1ZBQnGfhN3g=; b=qctSZMWszcdwOcWJ2u9i6YGxcZLzgIN/zYdVV6uBZvst+q8RCAyf7DvefMwwRTLNQo D8t8qMn/17YrT0xVoH6dVpbEQbXitGrvEhmyCN6dYY+kMHQkt+oR8BTzUE7+g5EG17no EmsOq9euS07LahNd1uLJRw+J2ZTO3Dodl7n1wOp3znSFiiysvA8ho97+jCVMf4d+ngkz 3QWwJW0mxtwPkldtwQU9JTsbKGpgW4mvo3lWun51zZLXY+RhKzq4oWk24f98LLz8pM5F qAzNiiaH4PpLw/z0pqeEe+HAdn8V2uAUe6aiGVaFhObyRkKEeJ4ZF6wzK598ByJ2NipD uPvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426009; x=1757030809; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lGptiAZO7CmFsu6Op41QCgd3wduY0PZh1ZBQnGfhN3g=; b=b5mY4Vi39m/kEBX/L7+WVfYX8U4JtXlMtzOcOuJ563gY1lRukUaLoxnujtVDJGZ4Y7 CZ1FEjQoelfxfR5oTEoIsdzGgE69i9rfxQQX+SXbl/SEGN5zSem6bWp0rcSfY5dZLbr6 XBS/ZNSeE1aPghGV3j76D2aXFpLqoPn9sQT9ilTXCepU1UyqkmgKQZEsFGWyrYzdQrYZ eXkOkMNmQwolWNc0C/uyvaLK8c2yboQohSQ6i4QNgHK+Mxun3l2Mh8gL+t8uE2DTZmMG y48DYs8TbDkVXFl+QchRxSLBGgkZ4EQ/qcPLPw8MqSDg0igh9mWO++Fofb12NQKUx2e7 Ce4g== X-Forwarded-Encrypted: i=1; AJvYcCVrPcEHvO+k2tx63/GBkJ59pBSBadhNJOx5kDS/oqFKfwDk6Hc+JWAd/g7g0hvMZM4NraWJN6PM6zEEM+8=@vger.kernel.org X-Gm-Message-State: AOJu0Yzzxn0nUi5W6ZSQE6QZSSCd2ZLX5rFAtUVaWeS565leCnHAUbPW xxuyVc49T+ZlNNyFhTaJ+1hkVq8vK6Qh2QVvQGj2lAi/M3YMModP+uovUerhhlRA0YZNSIu/YYW 1QImegQ== X-Google-Smtp-Source: AGHT+IGPnFjJJcAxcfWGqx+FCTkwrZqqa6fjukRalr0DOQARC9pSx1FQL9VEBdTmVjF2hDXjOKUOxbz2eFY= X-Received: from pgbdp2.prod.google.com ([2002:a05:6a02:f02:b0:b42:da4:ef4]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:a109:b0:243:c315:5122 with SMTP id adf61e73a8af0-243c3155335mr548500637.10.1756426008991; Thu, 28 Aug 2025 17:06:48 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:16 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-17-seanjc@google.com> Subject: [RFC PATCH v2 16/18] KVM: TDX: Derive error argument names from the local variable names From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When printing SEAMCALL errors, use the name of the variable holding an error parameter instead of the register from whence it came, so that flows which use descriptive variable names will similarly print descriptive error messages. Suggested-by: Rick Edgecombe Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/tdx.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index df9b4496cd01..b73f260a55fd 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -41,14 +41,15 @@ #define TDX_BUG_ON(__err, __fn, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, "%s", "") =20 -#define TDX_BUG_ON_1(__err, __fn, __rcx, __kvm) \ - __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx", __rcx) +#define TDX_BUG_ON_1(__err, __fn, a1, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx", a1) =20 -#define TDX_BUG_ON_2(__err, __fn, __rcx, __rdx, __kvm) \ - __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx", __rcx, __rd= x) +#define TDX_BUG_ON_2(__err, __fn, a1, a2, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 " 0x%llx", a1,= a2) =20 -#define TDX_BUG_ON_3(__err, __fn, __rcx, __rdx, __r8, __kvm) \ - __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx, r8 0x%llx", = __rcx, __rdx, __r8) +#define TDX_BUG_ON_3(__err, __fn, a1, a2, a3, __kvm) \ + __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #= a3 " 0x%llx", \ + a1, a2, a3) =20 =20 bool enable_tdx __ro_after_init; --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CAF82522B4 for ; Fri, 29 Aug 2025 00:06:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426012; cv=none; b=g41D4nSePFzB0yPy5oBuCYp7cRkiJ/r5aTRcAlHVFrDbyg1qZK5vm5f3ahTWwcfTq7yr6g4u/sHw0gw2WBVve9/lvwg/q6yia7XLc2Ki8H0QTyi0Pb0Wfu1p5b3XfZSfgZJdUXfp7QlUZ712O8vZ7UYe+BgFkMSyfWGvB1P4HZE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426012; c=relaxed/simple; bh=H65S3rJ44uz1vGqMB8iQWHr3yMc9skrgmgdydv2RDBA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=G1aFboYPv4wgTqDKce+QIjKH88SkZI/HC7LkKv6bVaNSXJ2YukS5X99ob8gy+1LQw7sazrMdaLBROP2Ip1ckkDjwnbVh1ohO/E1MDQ9dT3xp3tRZfo2qD71IUFDHS/1hlhJiP1YDsy1R07v8cP30B9Zey0W3xP5wD9tnuhLHg3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=h7aEbgmf; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="h7aEbgmf" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3234811cab3so1748036a91.3 for ; Thu, 28 Aug 2025 17:06:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426011; x=1757030811; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=RxLpwttBWSyhDMcXeP9tOwiMvx8JT1XOIrXDB90LYs0=; b=h7aEbgmfH+cGre/Y2hSqw0HDXlCJOU5E8gTFMEVIQY2NBWKPF/f3Aof7R/4NYyTOEK XFbvZ2EtoskhR2KUItwuHL1lDI6xI/n6oyIIwxLxfAn/pZ29t1OGDjf+JEHG6omlDpyz fC8wDMl6OM9sRglQLx7ogsBJMlQnObo18oL3OYk+ciYqXzdMuCy7GAmlPd+ncncRSNbn tcwXiymIvsFRQkwqba89MhM6bjt/+hyb0A6djJf5t23thqLtGBQh0rbtszDaAiHCz4k9 gIH8Mx4k6qtXS+Tzqfegzgg91JYPUtr+VueaJWQ5Q5l4Qov0HK6fRBUaNlA6oZ8SwiuS F6MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426011; x=1757030811; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RxLpwttBWSyhDMcXeP9tOwiMvx8JT1XOIrXDB90LYs0=; b=EpuyadTBcKfaVsTFkYJLLCb+c0REETox+cf/raGj86ICHSG351iAV0NAkLNgO3uQRd ouvkOOfUPemZBlHPGTKXVY+0G7LcTkPmZdx+tMAThToFVGKgqj4sKEBEk4YaPiID6Nx7 Q+mjnhUkyY1wHy8reHNNyexZLbDU/kwIy031gD90aNaN1vzihFGytR6CK1sOTay/fG2D 6fg6q6bz+KQ0A61t2TTXq1fbH/u4Nigs28V71ke15EE471vUx/+0qMLjaZQIvl3yJDdF tT8IhpFjyIiMX8pI+4a6cYKjoZs7SWiVnu5DaoW5DSjRGreRatRG2FNRxcnSJzN98rIh bTXg== X-Forwarded-Encrypted: i=1; AJvYcCWbpY/mTHJJ6CButsbzraE3ONJ98ahTirG5q4gVP3VuOmjxPWF2ukVoQhs3QW4Tjd7zRzlbMQ2ef7/BIeI=@vger.kernel.org X-Gm-Message-State: AOJu0YwivUz0YYOXV0SeWaC0VPHXvWxv8Ao9ZJtNvcl1+K68AM87xEJ5 KXjXeva4yLvx6dNgST4G6u2UzDEfLuSO4CcXZpAuyTTAI/4Srfd/btZ1r3EIwaBGwTy17LOpMVN PHihv9g== X-Google-Smtp-Source: AGHT+IGLzPaUhXZ+u9d9SfG1XeBM9vvj2fmTjF1JBq9az/V7jTBWQ9o6GVfcj9yTochgKtIXUfKJVyXd6sU= X-Received: from pjbnb7.prod.google.com ([2002:a17:90b:35c7:b0:327:41f6:db15]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4c02:b0:324:ece9:6afb with SMTP id 98e67ed59e1d1-32515eadfb6mr29318208a91.3.1756426010777; Thu, 28 Aug 2025 17:06:50 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:17 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-18-seanjc@google.com> Subject: [RFC PATCH v2 17/18] KVM: TDX: Assert that mmu_lock is held for write when removing S-EPT entries From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unconditionally assert that mmu_lock is held for write when removing S-EPT entries, not just when removing S-EPT entries triggers certain conditions, e.g. needs to do TDH_MEM_TRACK or kick vCPUs out of the guest. Conditionally asserting implies that it's safe to hold mmu_lock for read when those paths aren't hit, which is simply not true, as KVM doesn't support removing S-EPT entries under read-lock. Only two paths lead to remove_external_spte(), and both paths asserts that mmu_lock is held for write (tdp_mmu_set_spte() via lockdep, and handle_removed_pt() via KVM_BUG_ON()). Deliberately leave lockdep assertions in the "no vCPUs" helpers to document that wait_for_sept_zap is guarded by holding mmu_lock for write. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/tdx.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b73f260a55fd..aa740eeb1c2a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1714,8 +1714,6 @@ static void tdx_track(struct kvm *kvm) if (unlikely(kvm_tdx->state !=3D TD_STATE_RUNNABLE)) return; =20 - lockdep_assert_held_write(&kvm->mmu_lock); - err =3D tdh_mem_track(&kvm_tdx->td); if (unlikely(tdx_operand_busy(err))) { /* After no vCPUs enter, the second retry is expected to succeed */ @@ -1761,6 +1759,8 @@ static void tdx_sept_remove_private_spte(struct kvm *= kvm, gfn_t gfn, gpa_t gpa =3D gfn_to_gpa(gfn); u64 err, entry, level_state; =20 + lockdep_assert_held_write(&kvm->mmu_lock); + /* * HKID is released after all private pages have been removed, and set * before any might be populated. Warn if zapping is attempted when --=20 2.51.0.318.gd7df087d1a-goog From nobody Fri Oct 3 14:29:35 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49B2025C6FF for ; Fri, 29 Aug 2025 00:06:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426015; cv=none; b=P3Vw6HC3uPA9+L83AWvnJbZxlvSDQ29al/KS8ke7ylSEY4KzVcnMvXS3niZEgaC74DPDtglC7bdWVEBv9C97f8ka8rOkAY8fhfLSM/N6oGR7z/EK6ji2E2yzxRsQYW2G5U8nqkxsQWa+BvhdCeLqaJhZog13eedFDBVWPVwq2XA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756426015; c=relaxed/simple; bh=f7lHMZ8qJpaqmwv29QAX2RMeTqQVMVNV4EYMHS+5iLo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LqBck1p7KBy+vkzUAR73PYRnWNnsqEx8H5w6KHrT7nkp0R0zCXsPRuM8qhsVOcIs5nhKPfgnHLLSJqxQMsbeXv2U29bb1wgNMEDlN3cRJqCwxmpUBmFyhCm4ui6hnFiVxnsUs76q+0Y+OT5R7tWuClkmnbLSc3l0loPiZMpmQkg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=iN8DUsPd; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="iN8DUsPd" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-76e2ea9366aso1372795b3a.2 for ; Thu, 28 Aug 2025 17:06:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756426012; x=1757030812; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=UOslP8m3lu4+oiJ697IMjpcaNHz3/4VIfEbOPXSKr7o=; b=iN8DUsPdbZ7mO3yl/EoO210QIlp3eCEDZVo+jYSUjCDvzYiR3vQ+K/L3qmOyCfekQc EeUXISMgucH7kaP17ZFLcPX79k5MU9v6WOqwEPtSadJrqiX4fA3gW661pH9VVCBWw7HC PdNMf2A0uRP1F5Dyg7bgUQXaPygeOuqmxV9PvRkO5psPw849MzQV/iP2MAy6lx/9uBOV sF0FNIkWtxPWdapOdSEZiwXuv/fmGDAP15tvgvhFbph4hqTllrcF9LH+aqfBBcOBvPR+ gqF5/ekHYQU1+CLbSK0H9H/jlubfFhy0nd2/taG792kyPauNQMDXyknqDOmuMIybZ/9q T3Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756426012; x=1757030812; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UOslP8m3lu4+oiJ697IMjpcaNHz3/4VIfEbOPXSKr7o=; b=m3YuDHlC3psvwrIN393qBVCju3yPjBLXCCVy6ztFW3HxXx+H6OuYnDJOUvX7kxY5uf gGLafKp1bjIYMrsezcQDi64434heM5IQuThrrkao5goZ3mkNTwqP+RWxim7xCFwwRLZb 15Tk0xOzSbrSa2hCck3dDj+3tknHx+5z80QimX57jcfqsM+WnPOGhXAAM38vTM2jRvrQ O712s8yr6tPv+W+mektuRA38bSaYq5q/ANLtY0SE0nUbcJhjlruYjFR+h/DdWV93FiO8 uxgb+ArPaW0JWZeJCuTR7wg/O5D/cUIdL5JYj1NZns3PZewfqrbZbxUMpp20rxooIOiW jkZA== X-Forwarded-Encrypted: i=1; AJvYcCWxLRhLhYURu6zbRgIahpa16Bx3VWUTSNQbfsFbdWz5B2dD3Y0xHuXThLY27Z66nueNJkVKLDyUoVik5vg=@vger.kernel.org X-Gm-Message-State: AOJu0Yz1JZCiMkrt8MBiDzrYuVpEGmZofOF2nJFx6TdStaOLkvXzn6wc Y+fSpe65ipHgWfDgr+zznx1QEP+NsF4tvaCSe1mutBupM+D02E1aiSkiujqrHhMD01a2r/7w+Nn 4CYXDaQ== X-Google-Smtp-Source: AGHT+IGzyu2K+Q3S0JyMe3DS6644rOW4vURK+W47x0/xR9XONWf8AM78cfPpCJTbVijPrA+3JwtyO448ezo= X-Received: from pfbhg23.prod.google.com ([2002:a05:6a00:8617:b0:76e:313a:6f90]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:181e:b0:772:921:e32b with SMTP id d2e1a72fcca58-77209220d75mr10908984b3a.25.1756426012474; Thu, 28 Aug 2025 17:06:52 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 28 Aug 2025 17:06:18 -0700 In-Reply-To: <20250829000618.351013-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250829000618.351013-1-seanjc@google.com> X-Mailer: git-send-email 2.51.0.318.gd7df087d1a-goog Message-ID: <20250829000618.351013-19-seanjc@google.com> Subject: [RFC PATCH v2 18/18] KVM: TDX: Add macro to retry SEAMCALLs when forcing vCPUs out of guest From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ira Weiny , Kai Huang , Michael Roth , Yan Zhao , Vishal Annapurve , Rick Edgecombe , Ackerley Tng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a macro to handle kicking vCPUs out of the guest and retrying SEAMCALLs on -EBUSY instead of providing small helpers to be used by each SEAMCALL. Wrapping the SEAMCALLs in a macro makes it a little harder to tease out which SEAMCALL is being made, but significantly reduces the amount of copy+paste code and makes it all but impossible to leave an elevated wait_for_sept_zap. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/tdx.c | 72 ++++++++++++++---------------------------- 1 file changed, 23 insertions(+), 49 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index aa740eeb1c2a..d6c9defad9cd 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -313,25 +313,24 @@ static void tdx_clear_page(struct page *page) __mb(); } =20 -static void tdx_no_vcpus_enter_start(struct kvm *kvm) -{ - struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - - lockdep_assert_held_write(&kvm->mmu_lock); - - WRITE_ONCE(kvm_tdx->wait_for_sept_zap, true); - - kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); -} - -static void tdx_no_vcpus_enter_stop(struct kvm *kvm) -{ - struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); - - lockdep_assert_held_write(&kvm->mmu_lock); - - WRITE_ONCE(kvm_tdx->wait_for_sept_zap, false); -} +#define tdh_do_no_vcpus(tdh_func, kvm, args...) \ +({ \ + struct kvm_tdx *__kvm_tdx =3D to_kvm_tdx(kvm); \ + u64 __err; \ + \ + lockdep_assert_held_write(&kvm->mmu_lock); \ + \ + __err =3D tdh_func(args); \ + if (unlikely(tdx_operand_busy(__err))) { \ + WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, true); \ + kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); \ + \ + __err =3D tdh_func(args); \ + \ + WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, false); \ + } \ + __err; \ +}) =20 /* TDH.PHYMEM.PAGE.RECLAIM is allowed only when destroying the TD. */ static int __tdx_reclaim_page(struct page *page) @@ -1714,14 +1713,7 @@ static void tdx_track(struct kvm *kvm) if (unlikely(kvm_tdx->state !=3D TD_STATE_RUNNABLE)) return; =20 - err =3D tdh_mem_track(&kvm_tdx->td); - if (unlikely(tdx_operand_busy(err))) { - /* After no vCPUs enter, the second retry is expected to succeed */ - tdx_no_vcpus_enter_start(kvm); - err =3D tdh_mem_track(&kvm_tdx->td); - tdx_no_vcpus_enter_stop(kvm); - } - + err =3D tdh_do_no_vcpus(tdh_mem_track, kvm, &kvm_tdx->td); TDX_BUG_ON(err, TDH_MEM_TRACK, kvm); =20 kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); @@ -1773,14 +1765,8 @@ static void tdx_sept_remove_private_spte(struct kvm = *kvm, gfn_t gfn, if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) return; =20 - err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level_= state); - if (unlikely(tdx_operand_busy(err))) { - /* After no vCPUs enter, the second retry is expected to succeed */ - tdx_no_vcpus_enter_start(kvm); - err =3D tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level= _state); - tdx_no_vcpus_enter_stop(kvm); - } - + err =3D tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa, + tdx_level, &entry, &level_state); if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm)) return; =20 @@ -1795,20 +1781,8 @@ static void tdx_sept_remove_private_spte(struct kvm = *kvm, gfn_t gfn, * with other vcpu sept operation. * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs. */ - err =3D tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry, - &level_state); - - if (unlikely(tdx_operand_busy(err))) { - /* - * The second retry is expected to succeed after kicking off all - * other vCPUs and prevent them from invoking TDH.VP.ENTER. - */ - tdx_no_vcpus_enter_start(kvm); - err =3D tdh_mem_page_remove(&kvm_tdx->td, gpa, tdx_level, &entry, - &level_state); - tdx_no_vcpus_enter_stop(kvm); - } - + err =3D tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa, + tdx_level, &entry, &level_state); if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm)) return; =20 --=20 2.51.0.318.gd7df087d1a-goog