From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5BF1D3191CE; Thu, 3 Sep 2026 01:51:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400289; cv=none; b=l/N4dSQYafQH6jm5kzw9f4iOSZWFB3I2+zBSD5H/AXlVv9sgr+NcaNzyV4s9WeLfvOtanxmNuYKaas0wBMJEOJBqO4xDZFqMeiATK8XBd1AeiOAdCRR3LsfS0C23LDJpyCVklTOwB77IyCFs0onr/+9DMwBX2Wt4yFaPvxp48GA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400289; c=relaxed/simple; bh=maHjz0J4NTePFy90Iwp4ujESHtfZEnZOTgCms4Pdg3w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DL3BUrHVe9rvIlJTaKNDwuloqbnUa95i/+m5wEL8Gk/fSEEmosmzgy4HQ/i4Wqm/PpT3e2GyMEpvZGWyDSDTiwJKpDZqSkoH7C23Ne13rqZIpP1zXnRV8Z+H8tJYiJTKVQpUUczUOfKlofNeXt8GRHbv9H4wgAgsgWxdikYkOyE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=lTSuK02m; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="lTSuK02m" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400286; x=1819936286; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=maHjz0J4NTePFy90Iwp4ujESHtfZEnZOTgCms4Pdg3w=; b=lTSuK02mcQos2VwXBYHTh9HTY5kmQgEJnwx4Fic8s639lKQeKNk08WiI o2rhcJ6HtB6MRxRqXGRZXujLWDtnn1H46jCStkFnfn5immae6810I/z0V YIrMeOoBJhGqwbbgBePS96ffOFKyeUHojVgdINuFBsGJQYc8cEUTcpxcH ame4m5TOlpodyYMslMt4q+ixpYeSdGKOQ6oPthwfG5WePfGTlEngo1r3j U216rhiLO6IPif7rpPzpZWAgMtQn5/tSWhV5HgIA4SJdQbqwnWsAO/a8e /5osNzP0wEMLRASJT9JMNdXnhLqEQcK8HceMMLKPZ5Gff/D4q9AJu1XoJ w==; X-CSE-ConnectionGUID: bStfSnUFQ0O+KPF2Pp50cw== X-CSE-MsgGUID: Id90/GtnSOeKrdrjYKDneA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469111" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469111" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 X-CSE-ConnectionGUID: ybkpX6fYTDmp2CnEJSQ3aw== X-CSE-MsgGUID: pyklPRSaRxqqMAqiyxUdOA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770212" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:21 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu , Dave Hansen Subject: [PATCH v10 01/11] x86/virt/tdx: Simplify PAMT layout calculation Date: Wed, 2 Sep 2026 18:51:03 -0700 Message-ID: <20260903015113.93343-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For each memory region that the TDX module might use (called TDMR), three separate traditional PAMT allocations are needed. There is one for each supported page size (1GB, 2MB, 4KB). These store information on each page in the TDMR. In Linux, they are allocated out of one physically contiguous block, in order to more efficiently use some internal TDX module bookkeeping resources. So some simple math is needed to break the single large allocation into three smaller allocations for each page size. There are some commonalities in the math needed to calculate the base and size for each smaller allocation, and so an effort was made to share logic across the three. Unfortunately doing this turned out unnaturally tortured, with a loop iterating over the three page sizes, only to call into a function with case statements for each page size. In the future Dynamic PAMT will add more logic that is special to the 4KB page size, making the benefit of the math sharing even more questionable. Three is not a very high number, so get rid of the loop and just duplicate the small calculation three times. In doing so, setup for future Dynamic PAMT changes. Since the loop that iterates over it is gone, further simplify the code by dropping the array of intermediate size and base storage. Just store the values to their final locations. AI was used under supervision to collect/apply feedback, review code and workshop logs. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: Chao Gao Reviewed-by: Yan Zhao Reviewed-by: Tony Lindgren Reviewed-by: Dave Hansen Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta --- v10: - Fix whitespace change in tdmrs_set_up_pamt_all() (AI nit checker) v8: - Fix PAMT capitalization in comment (Sohil) - Delete stale info in log, that Chao mentioned last time (Binbin) v7: - Remove accidental whitespace changes (Kiryl) - Drop stale sentence in log (Chao) - Better patch subject (Yan) - Drop Assisted-by tag and cover AI use in log (Dave) --- arch/x86/virt/vmx/tdx/tdx.c | 90 ++++++++++++------------------------- 1 file changed, 28 insertions(+), 62 deletions(-) diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 1b9ff749dd8e0..51b191bf12258 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -514,31 +514,21 @@ static __init int fill_out_tdmrs(struct list_head *tm= b_list, * Calculate PAMT size given a TDMR and a page size. The returned * PAMT size is always aligned up to 4K page boundary. */ -static __init unsigned long tdmr_get_pamt_sz(struct tdmr_info *tdmr, int p= gsz, - u16 pamt_entry_size) +static __init unsigned long tdmr_get_pamt_sz(struct tdmr_info *tdmr, int p= gsz) { unsigned long pamt_sz, nr_pamt_entries; + const int tdx_pg_size_shift[TDX_PS_NR] =3D { PAGE_SHIFT, PMD_SHIFT, PUD_S= HIFT }; + const u16 pamt_entry_size[TDX_PS_NR] =3D { + tdx_sysinfo.tdmr.pamt_4k_entry_size, + tdx_sysinfo.tdmr.pamt_2m_entry_size, + tdx_sysinfo.tdmr.pamt_1g_entry_size, + }; =20 - switch (pgsz) { - case TDX_PS_4K: - nr_pamt_entries =3D tdmr->size >> PAGE_SHIFT; - break; - case TDX_PS_2M: - nr_pamt_entries =3D tdmr->size >> PMD_SHIFT; - break; - case TDX_PS_1G: - nr_pamt_entries =3D tdmr->size >> PUD_SHIFT; - break; - default: - WARN_ON_ONCE(1); - return 0; - } + nr_pamt_entries =3D tdmr->size >> tdx_pg_size_shift[pgsz]; + pamt_sz =3D nr_pamt_entries * pamt_entry_size[pgsz]; =20 - pamt_sz =3D nr_pamt_entries * pamt_entry_size; /* TDX requires PAMT size must be 4K aligned */ - pamt_sz =3D ALIGN(pamt_sz, PAGE_SIZE); - - return pamt_sz; + return PAGE_ALIGN(pamt_sz); } =20 /* @@ -576,15 +566,11 @@ static __init int tdmr_get_nid(struct tdmr_info *tdmr= , struct list_head *tmb_lis * within @tdmr, and set up PAMTs for @tdmr. */ static __init int tdmr_set_up_pamt(struct tdmr_info *tdmr, - struct list_head *tmb_list, - u16 pamt_entry_size[]) + struct list_head *tmb_list) { - unsigned long pamt_base[TDX_PS_NR]; - unsigned long pamt_size[TDX_PS_NR]; - unsigned long tdmr_pamt_base; unsigned long tdmr_pamt_size; struct page *pamt; - int pgsz, nid; + int nid; =20 nid =3D tdmr_get_nid(tdmr, tmb_list); =20 @@ -592,12 +578,10 @@ static __init int tdmr_set_up_pamt(struct tdmr_info *= tdmr, * Calculate the PAMT size for each TDX supported page size * and the total PAMT size. */ - tdmr_pamt_size =3D 0; - for (pgsz =3D TDX_PS_4K; pgsz < TDX_PS_NR; pgsz++) { - pamt_size[pgsz] =3D tdmr_get_pamt_sz(tdmr, pgsz, - pamt_entry_size[pgsz]); - tdmr_pamt_size +=3D pamt_size[pgsz]; - } + tdmr->pamt_4k_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_4K); + tdmr->pamt_2m_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_2M); + tdmr->pamt_1g_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_1G); + tdmr_pamt_size =3D tdmr->pamt_4k_size + tdmr->pamt_2m_size + tdmr->pamt_1= g_size; =20 /* * Allocate one chunk of physically contiguous memory for all @@ -606,25 +590,17 @@ static __init int tdmr_set_up_pamt(struct tdmr_info *= tdmr, */ pamt =3D alloc_contig_pages(tdmr_pamt_size >> PAGE_SHIFT, GFP_KERNEL, nid, &node_online_map); + + /* + * tdmr->pamt_4k_base is still zero so the error + * path of the caller will skip freeing the PAMT. + */ if (!pamt) return -ENOMEM; =20 - /* - * Break the contiguous allocation back up into the - * individual PAMTs for each page size. - */ - tdmr_pamt_base =3D page_to_pfn(pamt) << PAGE_SHIFT; - for (pgsz =3D TDX_PS_4K; pgsz < TDX_PS_NR; pgsz++) { - pamt_base[pgsz] =3D tdmr_pamt_base; - tdmr_pamt_base +=3D pamt_size[pgsz]; - } - - tdmr->pamt_4k_base =3D pamt_base[TDX_PS_4K]; - tdmr->pamt_4k_size =3D pamt_size[TDX_PS_4K]; - tdmr->pamt_2m_base =3D pamt_base[TDX_PS_2M]; - tdmr->pamt_2m_size =3D pamt_size[TDX_PS_2M]; - tdmr->pamt_1g_base =3D pamt_base[TDX_PS_1G]; - tdmr->pamt_1g_size =3D pamt_size[TDX_PS_1G]; + tdmr->pamt_4k_base =3D page_to_phys(pamt); + tdmr->pamt_2m_base =3D tdmr->pamt_4k_base + tdmr->pamt_4k_size; + tdmr->pamt_1g_base =3D tdmr->pamt_2m_base + tdmr->pamt_2m_size; =20 return 0; } @@ -655,10 +631,7 @@ static __init void tdmr_do_pamt_func(struct tdmr_info = *tdmr, tdmr_get_pamt(tdmr, &pamt_base, &pamt_size); =20 /* Do nothing if PAMT hasn't been allocated for this TDMR */ - if (!pamt_size) - return; - - if (WARN_ON_ONCE(!pamt_base)) + if (!pamt_base) return; =20 pamt_func(pamt_base, pamt_size); @@ -684,14 +657,12 @@ static __init void tdmrs_free_pamt_all(struct tdmr_in= fo_list *tdmr_list) =20 /* Allocate and set up PAMTs for all TDMRs */ static __init int tdmrs_set_up_pamt_all(struct tdmr_info_list *tdmr_list, - struct list_head *tmb_list, - u16 pamt_entry_size[]) + struct list_head *tmb_list) { int i, ret =3D 0; =20 for (i =3D 0; i < tdmr_list->nr_consumed_tdmrs; i++) { - ret =3D tdmr_set_up_pamt(tdmr_entry(tdmr_list, i), tmb_list, - pamt_entry_size); + ret =3D tdmr_set_up_pamt(tdmr_entry(tdmr_list, i), tmb_list); if (ret) goto err; } @@ -968,18 +939,13 @@ static __init int construct_tdmrs(struct list_head *t= mb_list, struct tdmr_info_list *tdmr_list, struct tdx_sys_info_tdmr *sysinfo_tdmr) { - u16 pamt_entry_size[TDX_PS_NR] =3D { - sysinfo_tdmr->pamt_4k_entry_size, - sysinfo_tdmr->pamt_2m_entry_size, - sysinfo_tdmr->pamt_1g_entry_size, - }; int ret; =20 ret =3D fill_out_tdmrs(tmb_list, tdmr_list); if (ret) return ret; =20 - ret =3D tdmrs_set_up_pamt_all(tdmr_list, tmb_list, pamt_entry_size); + ret =3D tdmrs_set_up_pamt_all(tdmr_list, tmb_list); if (ret) return ret; =20 --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C25B93009E2; Thu, 3 Sep 2026 01:51:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400289; cv=none; b=L57ntcKBO23Hd3iKAVaFIuIUBjv8Ho+YpTZeWcTL6AX84oRO1fARCLaKuPe3P6mUu0uqUTMikVU1fBmw65MseL/z1s8JZcPvD5g0UVj1grCi/ZkTBan1tMTtvw45+guoIFxidvCQ/qiagBVZC0jNSRAua3Xl4KK+IMGrFE/5/8M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400289; c=relaxed/simple; bh=hvztLrJ4PSwRHtsh0fvfEfYYCEyJ1k5t2jSDdBWb7G0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XFEZEI33twGZrxic91I1LnZ30m2VmmwtMd+TOnSQiUh3ru9G8DX96SgKgNerzIjYWgI6AH4olNbkE8LduIZYFMqY8m60blSTN+aDQ62rRWXWyWff7KLh2br7SR9YcTLaDbCOvViM9RHN3izvq2p4vB20Scebzr4Of16u5TX0CI4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AHYU2Wx1; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AHYU2Wx1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400286; x=1819936286; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hvztLrJ4PSwRHtsh0fvfEfYYCEyJ1k5t2jSDdBWb7G0=; b=AHYU2Wx1JRGVWjQt3q9PPiDvXedTN5iYc60zimJxAnu+obtCtzDbZ0p9 SmHN3Hv8FAy5Tp1LFpxdMSR6R6DGyBgbbcddxqTC09DGh2HiQ+5FqCrDf UR9HALCo9QOLLA02zjfWA3SmMuELxwD/jNuBm9NnIzmKsKgdR4I3aAGmr n8yQ9248GriI1L0SLDcygEB5MgW9Jg8sgmdvnR52ek8kWNKpHqY3tp+1/ VYIImTYwyyey+fch/m3j3NmeH7/s5775MM0AEDCkkaScX+btXnciOA95o DfDE+G/vkSZspSJijO4iGtL+FwQ3mpcOcLEj5w9p2hUXbALSomEaw4cSd A==; X-CSE-ConnectionGUID: BvpQxDKjTayw+jDwODo3Sw== X-CSE-MsgGUID: 1PrMfPcUR3C1fq8sBAwofQ== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469124" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469124" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 X-CSE-ConnectionGUID: s2BkgkKHTbq5g/7rHSmwaw== X-CSE-MsgGUID: Xs/X7ACUTl25VeX6OmP6Kw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770215" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:21 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu , Dave Hansen Subject: [PATCH v10 02/11] x86/virt/tdx: Allocate page bitmap for Dynamic PAMT Date: Wed, 2 Sep 2026 18:51:04 -0700 Message-ID: <20260903015113.93343-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The TDX Physical Address Metadata Table (PAMT) holds data about the physical memory used by TDX, and must be allocated by the kernel during TDX module initialization. The exact size of the required PAMT memory is determined by the TDX module and may vary between TDX module versions. Currently it is approximately 0.4% of the system memory. This is a significant commitment, especially if it is not known upfront whether the machine will run any TDX guests. Each memory region that the TDX module might use needs three separate PAMT allocations. One for each supported page size (1GB, 2MB, 4KB). The TDX module supports a new feature designed to reduce PAMT overhead called Dynamic PAMT (DPAMT). Under DPAMT the 4KB level is allocated dynamically during runtime, while the 1GB and 2MB levels remain allocated on TDX module initialization. However, in the details, DPAMT still needs some smaller per 4KB page scoped data (currently it is 1 bit per page). The TDX module exposes the number of bits as a separate piece of metadata than the 4KB static allocation for normal PAMT. Although the size is enumerated differently, it is handed to the TDX module in the same way the 4KB page size PAMT allocation is for normal PAMT. Begin to implement DPAMT in the kernel by reading the bits-per-page needed for DPAMT. Calculate the size needed for the bitmap, and use it instead of the 4KB size determined for normal PAMT, in the case of DPAMT. The existing metadata reading code was generated by a script, but the current plan is to stop generating this code, as the script has continued to need adjustments. So add manually written code and adjust the comment about it being autogenerated to be more generic. Start to adopt a more normal kernel code style without the ternary statements and if conditionals assignments that the auto generated code has. AI was used under supervision to collect/apply feedback, review code and workshop logs. Based on a patch originally by Kiryl Shutsemau. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Tony Lindgren Reviewed-by: Yan Zhao Reviewed-by: Dave Hansen Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta --- v10: - Remove other "autogenerated" references. (AI nit checker) - Change "Dynamic PAMT" to "DPAMT" in comments, and in the second reference in the logs. (Dave) v9: - Switch to byte size hint in metadata field id (Sashiko) v7: - Re-order pamt size calculations for greater readability (Kiryl) - Move comment to its own line in tdx_supports_dynamic_pamt() (Yan, Sohil) - Log tweak (Sohil) - Make comment in metadata reading more appropriate (Sohil) - Drop Assisted-by tag and cover AI use in log (Dave) - Move tdx_supports_dynamic_pamt() to not static inline to reduce churn in later changes v6: - Improve comment (Binbin) - Log tweaks - Mark tdmr_get_pamt_bitmap_sz() __init in response to upstream changes - Switch to more normal kernel code style, even though it differs from the existing auto generated code. --- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/include/asm/tdx_global_metadata.h | 9 ++++--- arch/x86/virt/vmx/tdx/tdx.c | 29 +++++++++++++++++++-- arch/x86/virt/vmx/tdx/tdx_global_metadata.c | 23 +++++++++++++++- 4 files changed, 57 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 89e97d5761d89..d414064436221 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -118,6 +118,8 @@ static inline bool tdx_supports_runtime_update(const st= ruct tdx_sys_info *sysinf return sysinfo->features.tdx_features0 & TDX_FEATURES0_TD_PRESERVING; } =20 +bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo); + int tdx_guest_keyid_alloc(void); u32 tdx_get_nr_guest_keyids(void); void tdx_guest_keyid_free(unsigned int keyid); diff --git a/arch/x86/include/asm/tdx_global_metadata.h b/arch/x86/include/= asm/tdx_global_metadata.h index 41150d546589c..8a3cc1a2a41e2 100644 --- a/arch/x86/include/asm/tdx_global_metadata.h +++ b/arch/x86/include/asm/tdx_global_metadata.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* Automatically generated TDX global metadata structures. */ -#ifndef _X86_VIRT_TDX_AUTO_GENERATED_TDX_GLOBAL_METADATA_H -#define _X86_VIRT_TDX_AUTO_GENERATED_TDX_GLOBAL_METADATA_H +/* TDX global metadata structures. */ +#ifndef _X86_VIRT_TDX_TDX_GLOBAL_METADATA_H +#define _X86_VIRT_TDX_TDX_GLOBAL_METADATA_H =20 #include =20 @@ -21,6 +21,9 @@ struct tdx_sys_info_tdmr { u16 pamt_4k_entry_size; u16 pamt_2m_entry_size; u16 pamt_1g_entry_size; + + /* Optional metadata, if DPAMT is supported */ + u8 pamt_page_bitmap_entry_bits; }; =20 struct tdx_sys_info_td_ctrl { diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 51b191bf12258..54074ddcce858 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -510,6 +510,18 @@ static __init int fill_out_tdmrs(struct list_head *tmb= _list, return 0; } =20 +static __init unsigned long tdmr_get_pamt_bitmap_sz(struct tdmr_info *tdmr) +{ + unsigned long pamt_sz, nr_pamt_entries; + int bits_per_entry; + + bits_per_entry =3D tdx_sysinfo.tdmr.pamt_page_bitmap_entry_bits; + nr_pamt_entries =3D tdmr->size >> PAGE_SHIFT; + pamt_sz =3D DIV_ROUND_UP(nr_pamt_entries * bits_per_entry, BITS_PER_BYTE); + + return PAGE_ALIGN(pamt_sz); +} + /* * Calculate PAMT size given a TDMR and a page size. The returned * PAMT size is always aligned up to 4K page boundary. @@ -578,9 +590,16 @@ static __init int tdmr_set_up_pamt(struct tdmr_info *t= dmr, * Calculate the PAMT size for each TDX supported page size * and the total PAMT size. */ - tdmr->pamt_4k_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_4K); - tdmr->pamt_2m_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_2M); tdmr->pamt_1g_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_1G); + tdmr->pamt_2m_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_2M); + + if (tdx_supports_dynamic_pamt(&tdx_sysinfo)) { + /* With DPAMT, PAMT_4K is replaced with a bitmap */ + tdmr->pamt_4k_size =3D tdmr_get_pamt_bitmap_sz(tdmr); + } else { + tdmr->pamt_4k_size =3D tdmr_get_pamt_sz(tdmr, TDX_PS_4K); + } + tdmr_pamt_size =3D tdmr->pamt_4k_size + tdmr->pamt_2m_size + tdmr->pamt_1= g_size; =20 /* @@ -1971,6 +1990,12 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, kvm_pfn_t = pfn) } EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid); =20 +bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo) +{ + /* To be enabled when kernel is ready. */ + return false; +} + void tdx_sys_disable(void) { struct tdx_module_args args =3D {}; diff --git a/arch/x86/virt/vmx/tdx/tdx_global_metadata.c b/arch/x86/virt/vm= x/tdx/tdx_global_metadata.c index e49c300f23d43..98ebf17aab1ce 100644 --- a/arch/x86/virt/vmx/tdx/tdx_global_metadata.c +++ b/arch/x86/virt/vmx/tdx/tdx_global_metadata.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * Automatically generated functions to read TDX global metadata. + * Functions to read TDX global metadata. * * This file doesn't compile on its own as it lacks of inclusion * of SEAMCALL wrapper primitive which reads global metadata. @@ -33,6 +33,18 @@ static __init int get_tdx_sys_info_features(struct tdx_s= ys_info_features *sysinf return ret; } =20 +static __init int get_tdx_sys_info_tdmr_dpamt(struct tdx_sys_info_tdmr *sy= sinfo_tdmr) +{ + int ret; + u64 val; + + ret =3D read_sys_metadata_field(0x9100000000000013, &val); + if (!ret) + sysinfo_tdmr->pamt_page_bitmap_entry_bits =3D val; + + return ret; +} + static __init int get_tdx_sys_info_tdmr(struct tdx_sys_info_tdmr *sysinfo_= tdmr) { int ret =3D 0; @@ -129,5 +141,14 @@ static __init int get_tdx_sys_info(struct tdx_sys_info= *sysinfo) ret =3D ret ?: get_tdx_sys_info_td_ctrl(&sysinfo->td_ctrl); ret =3D ret ?: get_tdx_sys_info_td_conf(&sysinfo->td_conf); =20 + /* + * The kernel supports using TDX without DPAMT, so + * avoid reporting failure if it's not supported. Don't + * try to support buggy TDX modules that advertise + * DPAMT but don't expose the metadata. + */ + if (!ret && tdx_supports_dynamic_pamt(sysinfo)) + ret =3D get_tdx_sys_info_tdmr_dpamt(&sysinfo->tdmr); + return ret; } --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5372232142B; Thu, 3 Sep 2026 01:51:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400290; cv=none; b=eeS+1CHUk80+d/J9Gr2FX9JiSWOPKtav9eyCDx3gHgsNew+EBXTwWcW8Qj11URib56DcGXpP3wk0nUY/A8n5lEaKoz/9vf2DcHFxbPCt4UfbiVYnP+qZTPnKz5/1nQhed6gpsi2ucsZBmPJyhhvX4gRi/9vtMs04Sf6WKCvNefI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400290; c=relaxed/simple; bh=TY8J5Ch3wZ2pp/ME9tYfTBbVh4Ftdl+nu8+TiOsTr4Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LwilrMKT348dlQOXuBQqE4hel0wbmQur6AY34bcW3Odt3ADHUcjT927s4H5gCw2Lze03THQEb3qF4nMAGp1jAVIlMMtSZd33rrAeq5FMGo+L9KxvWOYCWd0kpqDxQJpfiGI2Ms/4JrN0Nxzl+Rsv1S0bZnobWtq/+xZQDY59ShQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=iNxMmgaS; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="iNxMmgaS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400287; x=1819936287; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TY8J5Ch3wZ2pp/ME9tYfTBbVh4Ftdl+nu8+TiOsTr4Q=; b=iNxMmgaSMXa0jV4sL89aKQtE17KMgmcmIFG1LRP/u4d2LPysJhpBRG4t uELXmE6zOOHgSOJJU6E1KkRAsLgGcfk0mXdtgnjHJbdWkTkQN0q5IECny anq0BX/y2G48WRxToqj9i9auecHfzq8bWozfYEbMoUTp5HzTh9FQB7ePy NijvK0ancV34vEP7ZVGx9KNf150BWdTmax1H1A7wFNfYr17LFO58RM9UO ClRsLFwbefCA7f8wP/IbcUDGwblcVLRzTaw44Z6JeKN2CIjvPmzPTIbTd krIFXdaHIsQC600HsOK/6cc0bFN77Txss0hfhU+NZNvd7lzOmF2B/cYy+ w==; X-CSE-ConnectionGUID: ASQWjAnCSAaYOYmIsH8mgg== X-CSE-MsgGUID: 3e+T2aBxSHKUPgphw02sfA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469134" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469134" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 X-CSE-ConnectionGUID: no+LsAO8R3imMWpUY1Z10A== X-CSE-MsgGUID: Lpc8+a84RE2jCUNJacou3w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770219" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:21 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu Subject: [PATCH v10 03/11] x86/virt/tdx: Add __tdx_pamt_get/put() helpers Date: Wed, 2 Sep 2026 18:51:05 -0700 Message-ID: <20260903015113.93343-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add helpers to use when allocating or preparing pages that are handed to the TDX module for use as control/S-EPT pages, and thus need Dynamic PAMT (DPAMT) adjustments. The TDX module tracks some state for each page of physical memory that it might use. It calls this state the PAMT. It includes separate state for each page size a physical page could be utilized at within the TDX module (1GB, 2MB, 4KB). In DPAMT, only the 4KB page size state is allocated dynamically. KVM will need to hand pages to the TDX module that it will use at 4KB granularity. So these pages will need DPAMT backing added before they are used by the TDX module, and removed afterwards. Create __tdx_pamt_get/put() to handle installing DPAMT 4KB backing for pages that are already allocated (such as KVM's use of S-EPT page tables or guest private memory). Have them take a pfn instead of a struct page, as future changes will want to use these helpers for guest pages which are tracked by PFN. Also add __tdx_alloc_control_page() and __tdx_free_control_page() to handle both page allocation and DPAMT installation. Make them behave like normal alloc/free functions where allocation can fail in the case of no memory, but free (with any necessary DPAMT release) always succeeds. Do this so they can eventually support the existing TDX flows that require teardowns to succeed. Don't CLFLUSH the DPAMT pages handed to the TDX module, as is done for some other SEAMCALLs, as the TDX docs specify that this is only needed on "TD private memory or TD control structure page". Since these allocations will be easily user triggerable, account the memory. Only one pair of DPAMT pages is required for each 2MB-aligned physical region, so multiple callers could trip over each other if they try to manage the shared backing for two separate 4KB pages contained in one. To build the logic up iteratively, don't do anything to handle pages from the same 2MB region yet. Functionality to handle this will be added before DPAMT can be enabled. AI was used under supervision to collect/apply feedback, split patches, review code and workshop logs. Based on a patch originally by Kiryl Shutsemau. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Chao Gao Reviewed-by: Binbin Wu Reviewed-by: Tony Lindgren Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta Reviewed-by: Dave Hansen --- v10: - Clarify in comments about the limitations of the helpers added. (Dave) - Add comments in pamt_2mb_arg() about what is going on (Dave) - Prepend helpers with __ (Dave) - Align tdx_module_args args (Dave) - Standardize "TDX Module" (Dave) - Remove exports for newly private __ functions. - Move TDX_DPAMT_ENTRY_PAGE_CNT out of header to sole file that uses it. (AI nit checker) - Standardize for loop iterators declaration. (Nit checker) - Change "Dynamic PAMT" to "DPAMT" in comments, and in the second reference in the logs. (Dave) v9: - Address about how the log doesn't fully cover the limitations at this stage. (Sashiko) v7: - Comment improvements (Chao) - Drop unneeded addition of mm.h header include (Binbin) - Log clarity, code style nits (Sohil) - Drop Assisted-by tag and cover AI use in log (Dave) --- arch/x86/virt/vmx/tdx/tdx.c | 197 ++++++++++++++++++++++++++++++++++++ arch/x86/virt/vmx/tdx/tdx.h | 2 + 2 files changed, 199 insertions(+) diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 54074ddcce858..9caaa5de28817 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -46,6 +46,9 @@ #include "seamcall_internal.h" #include "tdx.h" =20 +/* Number of DPAMT pages to be provided to TDX module per 2MB region of PA= */ +#define TDX_DPAMT_ENTRY_PAGE_CNT 2 + struct tdx_module_state { bool initialized; bool sysinit_done; @@ -1996,6 +1999,200 @@ bool tdx_supports_dynamic_pamt(const struct tdx_sys= _info *sysinfo) return false; } =20 +static int alloc_pamt_array(struct page **pamt_pages) +{ + int i, j; + + for (i =3D 0; i < TDX_DPAMT_ENTRY_PAGE_CNT; i++) { + pamt_pages[i] =3D alloc_page(GFP_KERNEL_ACCOUNT); + if (!pamt_pages[i]) + goto err; + } + + return 0; + +err: + for (j =3D 0; j < i; j++) + __free_page(pamt_pages[j]); + + return -ENOMEM; +} + +static void free_pamt_array(struct page **pamt_pages) +{ + int i; + + for (i =3D 0; i < TDX_DPAMT_ENTRY_PAGE_CNT; i++) { + /* + * Reset pages unconditionally to cover cases + * where they were passed to the TDX module. + */ + tdx_quirk_reset_paddr(page_to_phys(pamt_pages[i]), PAGE_SIZE); + + __free_page(pamt_pages[i]); + } +} + +/* Helper for building DPAMT seamcall() arguments. */ +static u64 pamt_2mb_arg(kvm_pfn_t pfn) +{ + /* Find the 2MB-wide DPAMT region for 'pfn': */ + unsigned long hpa_2mb =3D ALIGN_DOWN(pfn << PAGE_SHIFT, PMD_SIZE); + + /* + * TDX ABI requires specifying the page level the installed DPAMT + * backing will cover, even though today only 2MB is supported. + */ + return hpa_2mb | TDX_PS_2M; +} + +/* Add PAMT backing for the 2MB region surrounding the given pfn. */ +static u64 tdh_phymem_pamt_add(kvm_pfn_t pfn, struct page **pamt_pages) +{ + struct tdx_module_args args =3D { + .rcx =3D pamt_2mb_arg(pfn), + .rdx =3D page_to_phys(pamt_pages[0]), + .r8 =3D page_to_phys(pamt_pages[1]), + }; + + return seamcall(TDH_PHYMEM_PAMT_ADD, &args); +} + +/* Remove PAMT backing for the 2MB region surrounding the given pfn. */ +static u64 tdh_phymem_pamt_remove(kvm_pfn_t pfn, struct page **pamt_pages) +{ + struct tdx_module_args args =3D { + .rcx =3D pamt_2mb_arg(pfn), + }; + u64 ret; + + ret =3D seamcall_ret(TDH_PHYMEM_PAMT_REMOVE, &args); + if (ret) + return ret; + + /* Copy PAMT pages out of the struct per the TDX ABI */ + pamt_pages[0] =3D phys_to_page(args.rdx); + pamt_pages[1] =3D phys_to_page(args.r8); + + return 0; +} + +/* + * Allocate DPAMT memory for the 2MB aligned region surrounding + * the given page. + * + * Only call this when the pfn is known not to already have Dynamic + * PAMT pages in the TDX module for it. + * + * Effectively it is not (yet) like a get, and more like a manual + * manipulation of the DPAMT backing for the 2MB aligned range + * covered by the pfn. + */ +static int __tdx_pamt_get(kvm_pfn_t pfn) +{ + struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT]; + u64 tdx_status; + int ret; + + if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) + return 0; + + ret =3D alloc_pamt_array(pamt_pages); + if (ret) + return ret; + + tdx_status =3D tdh_phymem_pamt_add(pfn, pamt_pages); + if (tdx_status !=3D TDX_SUCCESS) { + ret =3D -EIO; + goto out_free; + } + + return 0; + +out_free: + free_pamt_array(pamt_pages); + + return ret; +} + +/* + * Free DPAMT memory for the 2MB aligned region surrounding the + * given page. Only call this when the pfn is known to already + * have DPAMT pages in the TDX module for it, and no other pfns + * in the aligned 2MB physical region still need it. + * + * Don't make multiple calls concurrently of __tdx_pamt_get/put(), + * as there is no protections from races. + * + * Effectively it is not (yet) like a refcounted put, and more like a + * manual manipulation of the DPAMT backing for the 2MB aligned + * range covered by the pfn. + */ +static void __tdx_pamt_put(kvm_pfn_t pfn) +{ + struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT] =3D {}; + u64 tdx_status; + + if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) + return; + + tdx_status =3D tdh_phymem_pamt_remove(pfn, pamt_pages); + + /* + * Don't free pamt_pages as it could hold garbage when + * tdh_phymem_pamt_remove() fails. Don't panic/BUG_ON(), as + * there is no risk of data corruption, but do yell loudly as + * failure indicates a kernel bug, memory is being leaked, and + * the dangling PAMT entry may cause future operations to fail. + */ + if (WARN_ON_ONCE(tdx_status !=3D TDX_SUCCESS)) + return; + + free_pamt_array(pamt_pages); +} + +/* + * Return a page that can be gifted to the TDX module for use as a "contro= l" + * page, i.e. pages that are used for control structures for a given TDX + * guest, and thus obtain TDX protections, including PAMT tracking. + * + * This function is currently only safe to call once. And not safe to call + * if __tdx_pamt_get() is called before or after. + */ +static struct page * __maybe_unused __tdx_alloc_control_page(void) +{ + struct page *page; + + page =3D alloc_page(GFP_KERNEL_ACCOUNT); + if (!page) + return NULL; + + if (__tdx_pamt_get(page_to_pfn(page))) { + __free_page(page); + return NULL; + } + + return page; +} + +/* + * Free a page that was gifted to the TDX module for use as a control + * page. After this, the page is no longer protected by TDX. + * + * Like __tdx_pamt_put(), this is currently only safe to call this when + * a page is already known to have DPAMT pages in the TDX module for + * it, and no other pages in the aligned 2MB physical region will + * still need the backing. + */ +static void __maybe_unused __tdx_free_control_page(struct page *page) +{ + if (!page) + return; + + __tdx_pamt_put(page_to_pfn(page)); + __free_page(page); +} + void tdx_sys_disable(void) { struct tdx_module_args args =3D {}; diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h index bdfd0e1e337ac..a886c54decaad 100644 --- a/arch/x86/virt/vmx/tdx/tdx.h +++ b/arch/x86/virt/vmx/tdx/tdx.h @@ -48,6 +48,8 @@ #define TDH_SYS_CONFIG 45 #define TDH_SYS_SHUTDOWN 52 #define TDH_SYS_UPDATE 53 +#define TDH_PHYMEM_PAMT_ADD 58 +#define TDH_PHYMEM_PAMT_REMOVE 59 #define TDH_SYS_DISABLE 69 =20 /* --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AAF632AADE; Thu, 3 Sep 2026 01:51:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400292; cv=none; b=PKdpDyvOcDpGSZFXjjv+JuofCyjSAp7LdxQoXfgRCzaGv/LsV1rrM9t0tGZa2Z3IW59/qZ0I03hLJ1/mi1vyfcd9AVKIAWwBcBzquZeKrQBCO3yecvqAdUqf8QdVM5exysSPgAt9ZarEXGNZjtaT8ykN7vBJPaRPj6ajRjdaS9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400292; c=relaxed/simple; bh=LFKsaTmvbX23e6XFggv+Jg60B/YJjMdQfJhm7MlXoH8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=icyG145fheOEIJeq32UMiuwkIFXkjWkBQDFGJHjH+Vj+6KPXQ/5/fVuDXkVM+y6bzahDenmDHy4S8q78wviHds4tHUKBfpcv/Qkph2q10PY6tRWUlJPi3U95feo08MbOrKkXH4M7G6RoaKTrVbF89iPrlcdgFeRJoHD1rrIUDsw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ll5Sg71E; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ll5Sg71E" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400290; x=1819936290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LFKsaTmvbX23e6XFggv+Jg60B/YJjMdQfJhm7MlXoH8=; b=ll5Sg71E96d1fYLkM4Zm92/rtOVLS+IU0Qfk+hwXUts8k9AZdmB0TKLw PXnwy9Tfzf1bLDaONhVMt4R9jv0Z713DSRSErHF5WKPXuejTD+xGbL5Nq oN/6+biwKS0kN7q/aJ5EGmGhOao8WRKxE3c0CeMkmFQObAbUnEku8zYWT 5ONnyF6CZOhLlZAWAYvr/xi7crPY6B19mDB3/Zy4GsSJ/KsNTEAi8PFlv X9Gjnoc16sSZH/83fv8hBMEsEpRXsvvQ83alzCiqU62Q5wMOPj59pFJut UGqcnqrWmhHvkrxhbhb6NoMM0TlumXTpfw6PFAFGgLv9kbgLqI/Ciaa/G g==; X-CSE-ConnectionGUID: Go5ZLVt/Qjq6SVAY0A8gTg== X-CSE-MsgGUID: nDtDIloFR/61SwB+EYG01Q== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469146" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469146" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:23 -0700 X-CSE-ConnectionGUID: yPaKBYjNQr6k2eJGCU+VZQ== X-CSE-MsgGUID: BQkGR8I4R4mq8mxOadprKQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770222" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu Subject: [PATCH v10 04/11] x86/virt/tdx: Allocate refcounts for Dynamic PAMT memory Date: Wed, 2 Sep 2026 18:51:06 -0700 Message-ID: <20260903015113.93343-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The PAMT memory holds metadata for all possible TDX protected memory. Each physical address range is covered by PAMT entries at three levels (1GB, 2MB, 4KB). With Dynamic PAMT (DPAMT), the 4KB level of PAMT is allocated on demand. The kernel supplies the TDX module with page pairs to store the 4KB level entries, which cover 2MB of host physical memory. The kernel must provide this page pair before using pages from the range for TDX. If this is not done, SEAMCALLs that give the pages to be protected by the TDX module will fail. Allocate reference counters for every 2MB range to track TDX memory usage. This can be used to handle concurrent get/put callers, in order to accurately determine when the dynamic 4KB level of DPAMT needs to be allocated and when it can be freed. This allocation will currently consume 2MB for every 1TB of address space from 0 to max_pfn. The allocation size will depend on how the RAM is physically laid out. In a worst case scenario where the entire 52 bit address space is covered this would be 8GB. Then the DPAMT refcount allocations could hypothetically cause the savings from DPAMT to go negative on exotic platforms with sparse, small amounts of memory. Future changes could reduce this refcount overhead to be only allocating refcounts for physical ranges that contain memory that TDX can use. However, this is left for future work. AI was used under supervision to collect/apply feedback, review code and workshop logs. Based on a patch originally by Kiryl Shutsemau. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Chao Gao Reviewed-by: Yan Zhao Reviewed-by: Tony Lindgren Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta --- v10: - Change "Dynamic PAMT" to "DPAMT" in comments, and in the second reference in the logs. (Dave) - Rename some other pamt gunk to dpamt (Dave) v7: - Annotate functions __init. (Chao) - Log tweaks. (Yan) - Standardize on memory units in the text. (Sohil) - Delete unneeded comment. (Sohil) - Use vzalloc(). (Sohil) - Drop Assisted-by tag and cover AI use in log. (Dave) v6: - Remove confusing reference to allocating PAMT memory in pamt_refcounts comment. (Yan) - Rename "metadata" function names that really deal with refcounts, as metadata already has a different meaning in TDX. - Move tdx_find_pamt_refcount() to this patch to aid in reviewability. --- arch/x86/virt/vmx/tdx/tdx.c | 57 +++++++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 9caaa5de28817..305289bd673be 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -66,6 +67,14 @@ static DEFINE_PER_CPU(bool, tdx_lp_initialized); =20 static struct tdmr_info_list tdx_tdmr_list; =20 +/* + * On a machine with DPAMT, the kernel maintains a reference counter + * for every 2MB range. The counter indicates how many users there are for + * the DPAMT at the 2MB range. The kernel allocates DPAMT refcounts at + * initialization. + */ +static atomic_t *dpamt_refcounts; + /* All TDX-usable memory regions. Protected by mem_hotplug_lock. */ static LIST_HEAD(tdx_memlist); =20 @@ -255,6 +264,42 @@ static struct syscore tdx_syscore =3D { .ops =3D &tdx_syscore_ops, }; =20 +/* + * Allocate DPAMT reference counters for all physical memory. + * + * It consumes 2MB for every 1TB of physical memory. + */ +static __init int init_dpamt_refcounts(void) +{ + size_t size =3D DIV_ROUND_UP(max_pfn, PTRS_PER_PTE) * sizeof(*dpamt_refco= unts); + + if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) + return 0; + + dpamt_refcounts =3D vzalloc(size); + if (!dpamt_refcounts) + return -ENOMEM; + + return 0; +} + +static __init void free_dpamt_refcounts(void) +{ + if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) + return; + + vfree(dpamt_refcounts); + dpamt_refcounts =3D NULL; +} + +static __maybe_unused atomic_t *tdx_find_dpamt_refcount(unsigned long pfn) +{ + /* Find which PMD a PFN is in. */ + unsigned long index =3D pfn >> (PMD_SHIFT - PAGE_SHIFT); + + return &dpamt_refcounts[index]; +} + /* * Add a memory region as a TDX memory block. The caller must make sure * all memory regions are added in address ascending order and don't @@ -1155,10 +1200,14 @@ static __init int init_tdx_module(void) */ get_online_mems(); =20 - ret =3D build_tdx_memlist(&tdx_memlist); + ret =3D init_dpamt_refcounts(); if (ret) goto out_put_tdxmem; =20 + ret =3D build_tdx_memlist(&tdx_memlist); + if (ret) + goto err_free_dpamt_refcounts; + /* Allocate enough space for constructing TDMRs */ ret =3D alloc_tdmr_list(&tdx_tdmr_list, &tdx_sysinfo.tdmr); if (ret) @@ -1208,6 +1257,8 @@ static __init int init_tdx_module(void) free_tdmr_list(&tdx_tdmr_list); err_free_tdxmem: free_tdx_memlist(&tdx_memlist); +err_free_dpamt_refcounts: + free_dpamt_refcounts(); goto out_put_tdxmem; } =20 @@ -2159,7 +2210,7 @@ static void __tdx_pamt_put(kvm_pfn_t pfn) * This function is currently only safe to call once. And not safe to call * if __tdx_pamt_get() is called before or after. */ -static struct page * __maybe_unused __tdx_alloc_control_page(void) +static __maybe_unused struct page *__tdx_alloc_control_page(void) { struct page *page; =20 @@ -2184,7 +2235,7 @@ static struct page * __maybe_unused __tdx_alloc_contr= ol_page(void) * it, and no other pages in the aligned 2MB physical region will * still need the backing. */ -static void __maybe_unused __tdx_free_control_page(struct page *page) +static __maybe_unused void __tdx_free_control_page(struct page *page) { if (!page) return; --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 797D03264D5; Thu, 3 Sep 2026 01:51:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400294; cv=none; b=S+H9GWeWUapiYDY3Hr91bfjcMHw+nVb26hvmbECwPzN78+zTtP4IAL4OJCndWX1pdyw0AsJcu2jSPGhxwlnvWC03O/5LNmMXF5pQfRrA6GCg73z31ACJ1E7DyTIP6XRpPo5NgfGPrm6RYnyjgkajzhYzvxOfAq9tUA0VjMJcWZg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400294; c=relaxed/simple; bh=0ATsA5a4UXrVPY7m3QHhBQ8gRb0cejBmI6g7gF/r/xY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nP/n5WaqsADLGVSa4rxNQdI9hYAyAmi9mmE22zNifySLaFXzWxwrUTVXXDZ+vGJFVfW3nHc6uOctZdZLujf9W8A/D7PRxBLw7gg1iyXIklwcBAEcb9Megmi0Mq0iMznaTcU/39js2DPWj/Rvlcl8vL8yq8TsRL8M157ax7NXqa0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=NXzxTgsO; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="NXzxTgsO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400291; x=1819936291; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0ATsA5a4UXrVPY7m3QHhBQ8gRb0cejBmI6g7gF/r/xY=; b=NXzxTgsOc1Vgaxyee3varqn05WZ42Cv2J3yloywKgi7D6efbPVZiMEbM zTOx4qbmubb+tXZ2R7jsCH6PVpRj9w7DFCOkTqzX+cG9PtDG1Y9Hy/yZX ArlgzqPuls64zo62zhzKA6O0RH1Aa+2R5/wbnCfCXoLFLorCetQwZVpHg oI1Ub3elITxXvageubEUik38W6YkXINPmTV/4kMLunxSiY7vdmxL/GnIQ UweSlaEeexxIOuFY3GU1tBaVOUN63TPXMrfTGXnC8CkXsE4quCW95ZO8J Cb1HZWMZwuVToY+YzaNPxlRRZJqL0f37yugpk8nKsz8fB0p7023kEnsvu A==; X-CSE-ConnectionGUID: 9xxsW+VlR6OsRAc0hJ3kHQ== X-CSE-MsgGUID: IkUiffTnRXOh27GENfPxsA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469161" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469161" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:23 -0700 X-CSE-ConnectionGUID: wgbZty8sS7WcBud055nXVQ== X-CSE-MsgGUID: 8KLNpbu9TYiT5opgk4abCQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770226" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu , Dave Hansen Subject: [PATCH v10 05/11] x86/virt/tdx: Handle multiple callers in tdx_pamt_get/put() Date: Wed, 2 Sep 2026 18:51:07 -0700 Message-ID: <20260903015113.93343-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __tdx_pamt_get()/__tdx_pamt_put() unconditionally add or remove Dynamic PAMT (DPAMT) backing for the 2MB region covering the passed page. However, multiple callers can add or remove 4KB pages that fall within the same 2MB region and in that scenario only a single PAMT entry is required. Make the helpers handle only adding/removing DPAMT backing when required, by refcounting each 2MB range. Gate the actual DPAMT add and remove on refcount transitions (0->1 and 1->0). Serialize the refcount check and SEAMCALL with a global spinlock so the read-decide-act sequence is atomic. This also avoids TDX module BUSY errors, as the DPAMT add and remove SEAMCALLs take internal TDX module locks for the 2MB ranges of the specified PFN and the PAMT page pair PFNs. So simultaneous attempts on the same 2MB ranges of the PFNs would otherwise encounter an error, which would not be handleable in the put case. The lock is global and heavyweight. Use simple conditional logic to keep correctness obvious. This will be optimized in a later change. The dpamt_refcounts[] are atomic_t's. They do not strictly need to be because all access is protected by pamt_lock. The overhead of an atomic_t in this situation is minuscule compared to the global lock. Leave the atomic_t in place to enable future optimization with minimal churn. Since the DPAMT helpers are broadly functional now, drop the "__" to rename them tdx_pamt_get/put() and tdx_alloc/free_control_page(). Export them for use in KVM in subsequent changes. AI was used under supervision to collect/apply feedback, split patches, review code and workshop logs. Based on a patch originally by Kiryl Shutsemau. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Chao Gao Reviewed-by: Yan Zhao Reviewed-by: Tony Lindgren Reviewed-by: Nikolay Borisov Reviewed-by: Dave Hansen Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta --- v10: - Rename the helper functions in this patch after changes to patch "x86/virt/tdx: Add tdx_alloc/free_control_page() helpers". - Remove the comments around helper limitations, leaving to be as they were in v9, except for the "TDX module" wording fix. - Export tdx_alloc/free_control_page() in this patch as a result of changes in "x86/virt/tdx: Add tdx_alloc/free_control_page() helpers" to make them private functions initially. Since they now become more broadly functional here, do the export in this patch. - For consistency, also move the tdx_pamt_get/put() export here too. - Change "Dynamic PAMT" to "DPAMT" in comments, and in the second reference in the logs. (Dave) - Rename "pamt" gunk to "dpamt". (Dave) v8: - Fix PAMT capitalization in comment (Sohil) v7: - Convert scoped_guard() blocks to use normal spin_un/lock() for the sake of making next patches diff cleaner - Drop __maybe_unused from tdx_find_pamt_refcount() (Binbin) - Switch to atomic_inc_not_zero() (Dave) - Justify use of atomic_t in log (Sohil) - Log/comments (Yan) - Drop Assisted-by tag and cover AI use in log (Dave) --- arch/x86/include/asm/tdx.h | 6 +++ arch/x86/virt/vmx/tdx/tdx.c | 102 ++++++++++++++++++++---------------- 2 files changed, 63 insertions(+), 45 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index d414064436221..f7442ad20e46d 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -120,12 +120,18 @@ static inline bool tdx_supports_runtime_update(const = struct tdx_sys_info *sysinf =20 bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo); =20 +int tdx_pamt_get(kvm_pfn_t pfn); +void tdx_pamt_put(kvm_pfn_t pfn); + int tdx_guest_keyid_alloc(void); u32 tdx_get_nr_guest_keyids(void); void tdx_guest_keyid_free(unsigned int keyid); =20 void tdx_quirk_reset_paddr(unsigned long base, unsigned long size); =20 +struct page *tdx_alloc_control_page(void); +void tdx_free_control_page(struct page *page); + struct tdx_td { /* TD root structure: */ struct page *tdr_page; diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 305289bd673be..c347600a0aabb 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -292,7 +292,7 @@ static __init void free_dpamt_refcounts(void) dpamt_refcounts =3D NULL; } =20 -static __maybe_unused atomic_t *tdx_find_dpamt_refcount(unsigned long pfn) +static atomic_t *tdx_find_dpamt_refcount(unsigned long pfn) { /* Find which PMD a PFN is in. */ unsigned long index =3D pfn >> (PMD_SHIFT - PAGE_SHIFT); @@ -2097,7 +2097,7 @@ static u64 pamt_2mb_arg(kvm_pfn_t pfn) return hpa_2mb | TDX_PS_2M; } =20 -/* Add PAMT backing for the 2MB region surrounding the given pfn. */ +/* Add DPAMT backing for the 2MB region surrounding the given pfn. */ static u64 tdh_phymem_pamt_add(kvm_pfn_t pfn, struct page **pamt_pages) { struct tdx_module_args args =3D { @@ -2109,7 +2109,7 @@ static u64 tdh_phymem_pamt_add(kvm_pfn_t pfn, struct = page **pamt_pages) return seamcall(TDH_PHYMEM_PAMT_ADD, &args); } =20 -/* Remove PAMT backing for the 2MB region surrounding the given pfn. */ +/* Remove DPAMT backing for the 2MB region surrounding the given pfn. */ static u64 tdh_phymem_pamt_remove(kvm_pfn_t pfn, struct page **pamt_pages) { struct tdx_module_args args =3D { @@ -2128,20 +2128,14 @@ static u64 tdh_phymem_pamt_remove(kvm_pfn_t pfn, st= ruct page **pamt_pages) return 0; } =20 -/* - * Allocate DPAMT memory for the 2MB aligned region surrounding - * the given page. - * - * Only call this when the pfn is known not to already have Dynamic - * PAMT pages in the TDX module for it. - * - * Effectively it is not (yet) like a get, and more like a manual - * manipulation of the DPAMT backing for the 2MB aligned range - * covered by the pfn. - */ -static int __tdx_pamt_get(kvm_pfn_t pfn) +/* Serializes adding/removing DPAMT memory */ +static DEFINE_SPINLOCK(dpamt_lock); + +/* Bump DPAMT refcount for the given pfn and allocate DPAMT backing if nee= ded. */ +int tdx_pamt_get(kvm_pfn_t pfn) { struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT]; + atomic_t *dpamt_refcount; u64 tdx_status; int ret; =20 @@ -2152,41 +2146,59 @@ static int __tdx_pamt_get(kvm_pfn_t pfn) if (ret) return ret; =20 + dpamt_refcount =3D tdx_find_dpamt_refcount(pfn); + + spin_lock(&dpamt_lock); + + /* + * If the DPAMT entry is already added (i.e. refcount >=3D 1), + * then just increment the refcount. + */ + if (atomic_inc_not_zero(dpamt_refcount)) + goto out_free; + + /* Try to add the PAMT page and take the refcount 0->1. */ tdx_status =3D tdh_phymem_pamt_add(pfn, pamt_pages); - if (tdx_status !=3D TDX_SUCCESS) { + if (WARN_ON_ONCE(tdx_status !=3D TDX_SUCCESS)) { ret =3D -EIO; goto out_free; } =20 + atomic_set(dpamt_refcount, 1); + spin_unlock(&dpamt_lock); return 0; =20 out_free: + spin_unlock(&dpamt_lock); free_pamt_array(pamt_pages); =20 return ret; } +EXPORT_SYMBOL_FOR_KVM(tdx_pamt_get); =20 -/* - * Free DPAMT memory for the 2MB aligned region surrounding the - * given page. Only call this when the pfn is known to already - * have DPAMT pages in the TDX module for it, and no other pfns - * in the aligned 2MB physical region still need it. - * - * Don't make multiple calls concurrently of __tdx_pamt_get/put(), - * as there is no protections from races. - * - * Effectively it is not (yet) like a refcounted put, and more like a - * manual manipulation of the DPAMT backing for the 2MB aligned - * range covered by the pfn. - */ -static void __tdx_pamt_put(kvm_pfn_t pfn) +/* Drop DPAMT refcount for the given pfn and free DPAMT backing if needed.= */ +void tdx_pamt_put(kvm_pfn_t pfn) { struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT] =3D {}; + atomic_t *dpamt_refcount; u64 tdx_status; =20 if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) return; =20 + dpamt_refcount =3D tdx_find_dpamt_refcount(pfn); + + spin_lock(&dpamt_lock); + /* + * If there is more than 1 reference on the DPAMT entry, don't + * remove it yet. Just decrement the refcount. + */ + if (atomic_read(dpamt_refcount) > 1) { + atomic_dec(dpamt_refcount); + goto out_unlock; + } + + /* Try to remove the pamt page and take the refcount 1->0. */ tdx_status =3D tdh_phymem_pamt_remove(pfn, pamt_pages); =20 /* @@ -2194,23 +2206,26 @@ static void __tdx_pamt_put(kvm_pfn_t pfn) * tdh_phymem_pamt_remove() fails. Don't panic/BUG_ON(), as * there is no risk of data corruption, but do yell loudly as * failure indicates a kernel bug, memory is being leaked, and - * the dangling PAMT entry may cause future operations to fail. + * the dangling DPAMT entry may cause future operations to fail. */ if (WARN_ON_ONCE(tdx_status !=3D TDX_SUCCESS)) - return; + goto out_unlock; =20 + atomic_set(dpamt_refcount, 0); + spin_unlock(&dpamt_lock); free_pamt_array(pamt_pages); + return; +out_unlock: + spin_unlock(&dpamt_lock); } +EXPORT_SYMBOL_FOR_KVM(tdx_pamt_put); =20 /* * Return a page that can be gifted to the TDX module for use as a "contro= l" * page, i.e. pages that are used for control structures for a given TDX - * guest, and thus obtain TDX protections, including PAMT tracking. - * - * This function is currently only safe to call once. And not safe to call - * if __tdx_pamt_get() is called before or after. + * guest, and thus obtain TDX protections, including DPAMT tracking. */ -static __maybe_unused struct page *__tdx_alloc_control_page(void) +struct page *tdx_alloc_control_page(void) { struct page *page; =20 @@ -2218,31 +2233,28 @@ static __maybe_unused struct page *__tdx_alloc_cont= rol_page(void) if (!page) return NULL; =20 - if (__tdx_pamt_get(page_to_pfn(page))) { + if (tdx_pamt_get(page_to_pfn(page))) { __free_page(page); return NULL; } =20 return page; } +EXPORT_SYMBOL_FOR_KVM(tdx_alloc_control_page); =20 /* * Free a page that was gifted to the TDX module for use as a control * page. After this, the page is no longer protected by TDX. - * - * Like __tdx_pamt_put(), this is currently only safe to call this when - * a page is already known to have DPAMT pages in the TDX module for - * it, and no other pages in the aligned 2MB physical region will - * still need the backing. */ -static __maybe_unused void __tdx_free_control_page(struct page *page) +void tdx_free_control_page(struct page *page) { if (!page) return; =20 - __tdx_pamt_put(page_to_pfn(page)); + tdx_pamt_put(page_to_pfn(page)); __free_page(page); } +EXPORT_SYMBOL_FOR_KVM(tdx_free_control_page); =20 void tdx_sys_disable(void) { --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED80C305680; Thu, 3 Sep 2026 01:51:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400294; cv=none; b=NcKEDyWtsHvPS3llVKjEJAwQTR6K5WCoXKJJltC0BJI1hHoXTTJ1FzUllsYdu0bzlagRwKpyrUI6BeMLUtaKpTOvQ2P8T5d/GaQPLIqd1J18d6mokIm1O3W38DnJeEjojDLcouwbrhebyhQ0FGSBjXC+Pax3yUewe5Osxdi8gJg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400294; c=relaxed/simple; bh=La5TSnve7LEhFz6cfbRU27H/X/gt9pcVEVuJ+3YN1X4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j6yEcJFizkondx7mmbvHHHmidJ6MAdjbJsh1s3pkMEUYH1ffiyH4ZpFkm/7ehZabs/1iaVQcufdlLJs0u1H7uOd//qzne4nYj55CESNCnpDYuVWfSswqTyscG5KzCvtWwenHAdsMUm+dksODVA+Jb6A5lXHrGgp1u/qb7xR2Noo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=F4Gl76hD; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="F4Gl76hD" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400290; x=1819936290; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=La5TSnve7LEhFz6cfbRU27H/X/gt9pcVEVuJ+3YN1X4=; b=F4Gl76hDYjRXSRs3wkg5jkrcvTOXpluChd/xrRUyLXx4eRh5+N+4h8ZM ICpxmWWnDc8kGUFxNIvS4wGcw74Cfd8V0akjCnuE+t2abNfmj0LK3MnTo 0j3EeBMVqxSHSX5MLG35Go9GwhjcejRriYit1z3VfzS6hsgFnOA3QmTLB 1C3JheTpX0xvzEIuE7AfvcK4JydLCeiXoNQHtS8+2IIBvyBppuHpHkhr0 2UHUFtqUC+VDeLkVmTuDPuek5B5ktm7ENwGBKfOX1TIQtDozUqK3tM+S+ g01e7J1Wp6mQ4CpUv/GDqIZQxL7VKA8GNpCxCcW9YZ7hefWhRolYTrK7W g==; X-CSE-ConnectionGUID: O7Gy742iRB2vEMjluZ0JbA== X-CSE-MsgGUID: hGam1aYpQ+evBiqJsn0RPQ== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469165" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469165" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:23 -0700 X-CSE-ConnectionGUID: 4Ku2yq3FRcKvvw9mp6XvMQ== X-CSE-MsgGUID: n0XP+eJ0RU6qnuYqoQr5vQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770229" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu , Dave Hansen Subject: [PATCH v10 06/11] KVM: TDX: Allocate PAMT memory for TD and vCPU control structures Date: Wed, 2 Sep 2026 18:51:08 -0700 Message-ID: <20260903015113.93343-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kirill A. Shutemov" Use control page helpers for allocating and freeing TD control structures, such that these operations can work for Dynamic PAMT. The TDX module tracks some state for each page of physical memory that it might use. It calls this state the PAMT. It includes separate state for each page size a physical page could be utilized at within the TDX module (1GB, 2MB, 4KB). In Dynamic PAMT, only the 4KB page size state is allocated dynamically. So the kernel must ensure PAMT backing is installed for any 4KB page being gifted to the TDX module, and must tear down the backing when all associated gifted pages are reclaimed. TD scoped control pages (TDR, TDCS) and vCPU scoped control pages (TDVPR, TDCX) are all handed to the TDX module at 4KB page size and are therefore subject to this requirement. Replace the raw alloc_page()/__free_page() calls for these pages with tdx_alloc/free_control_page(). Switching between special Dynamic PAMT operations or normal page alloc/free operations is handled internally in tdx_alloc/free_control_page(). So don't check for Dynamic PAMT around these calls. Just call them unconditionally. Similarly, drop the NULL checks before freeing, as tdx_free_control_page() handles NULL internally. No functional change intended when DPAMT is not in use. Signed-off-by: Kirill A. Shutemov [sean: handle alloc+free+reclaim in one patch] Signed-off-by: Sean Christopherson [rick: enhance log, reviewing, rebase, with help from AI tooling] Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Chao Gao Reviewed-by: Yan Zhao Reviewed-by: Tony Lindgren Reviewed-by: Nikolay Borisov Reviewed-by: Dave Hansen Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta Acked-by: Sean Christopherson --- v7: - Fixup tags (Sean) - Missing word in log (Binbin) - Log smoothness (Yan) --- arch/x86/kvm/vmx/tdx.c | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b272c20586a74..3592596b5649a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -362,7 +362,7 @@ static void tdx_reclaim_control_page(struct page *ctrl_= page) if (tdx_reclaim_page(ctrl_page)) return; =20 - __free_page(ctrl_page); + tdx_free_control_page(ctrl_page); } =20 struct tdx_flush_vp_arg { @@ -589,7 +589,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kv= m) =20 tdx_quirk_reset_paddr(page_to_phys(kvm_tdx->td.tdr_page), PAGE_SIZE); =20 - __free_page(kvm_tdx->td.tdr_page); + tdx_free_control_page(kvm_tdx->td.tdr_page); kvm_tdx->td.tdr_page =3D NULL; } =20 @@ -2456,7 +2456,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_p= arams *td_params, =20 ret =3D -ENOMEM; =20 - tdr_page =3D alloc_page(GFP_KERNEL_ACCOUNT); + tdr_page =3D tdx_alloc_control_page(); if (!tdr_page) goto free_hkid; =20 @@ -2469,7 +2469,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_p= arams *td_params, goto free_tdr; =20 for (i =3D 0; i < kvm_tdx->td.tdcs_nr_pages; i++) { - tdcs_pages[i] =3D alloc_page(GFP_KERNEL_ACCOUNT); + tdcs_pages[i] =3D tdx_alloc_control_page(); if (!tdcs_pages[i]) goto free_tdcs; } @@ -2587,10 +2587,8 @@ static int __tdx_td_init(struct kvm *kvm, struct td_= params *td_params, teardown: /* Only free pages not yet added, so start at 'i' */ for (; i < kvm_tdx->td.tdcs_nr_pages; i++) { - if (tdcs_pages[i]) { - __free_page(tdcs_pages[i]); - tdcs_pages[i] =3D NULL; - } + tdx_free_control_page(tdcs_pages[i]); + tdcs_pages[i] =3D NULL; } if (!kvm_tdx->td.tdcs_pages) kfree(tdcs_pages); @@ -2605,16 +2603,13 @@ static int __tdx_td_init(struct kvm *kvm, struct td= _params *td_params, free_cpumask_var(packages); =20 free_tdcs: - for (i =3D 0; i < kvm_tdx->td.tdcs_nr_pages; i++) { - if (tdcs_pages[i]) - __free_page(tdcs_pages[i]); - } + for (i =3D 0; i < kvm_tdx->td.tdcs_nr_pages; i++) + tdx_free_control_page(tdcs_pages[i]); kfree(tdcs_pages); kvm_tdx->td.tdcs_pages =3D NULL; =20 free_tdr: - if (tdr_page) - __free_page(tdr_page); + tdx_free_control_page(tdr_page); kvm_tdx->td.tdr_page =3D NULL; =20 free_hkid: @@ -2948,7 +2943,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u6= 4 vcpu_rcx) int ret, i; u64 err; =20 - page =3D alloc_page(GFP_KERNEL_ACCOUNT); + page =3D tdx_alloc_control_page(); if (!page) return -ENOMEM; tdx->vp.tdvpr_page =3D page; @@ -2968,7 +2963,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u6= 4 vcpu_rcx) } =20 for (i =3D 0; i < kvm_tdx->td.tdcx_nr_pages; i++) { - page =3D alloc_page(GFP_KERNEL_ACCOUNT); + page =3D tdx_alloc_control_page(); if (!page) { ret =3D -ENOMEM; goto free_tdcx; @@ -2990,7 +2985,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u6= 4 vcpu_rcx) * method, but the rest are freed here. */ for (; i < kvm_tdx->td.tdcx_nr_pages; i++) { - __free_page(tdx->vp.tdcx_pages[i]); + tdx_free_control_page(tdx->vp.tdcx_pages[i]); tdx->vp.tdcx_pages[i] =3D NULL; } return -EIO; @@ -3018,16 +3013,14 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, = u64 vcpu_rcx) =20 free_tdcx: for (i =3D 0; i < kvm_tdx->td.tdcx_nr_pages; i++) { - if (tdx->vp.tdcx_pages[i]) - __free_page(tdx->vp.tdcx_pages[i]); + tdx_free_control_page(tdx->vp.tdcx_pages[i]); tdx->vp.tdcx_pages[i] =3D NULL; } kfree(tdx->vp.tdcx_pages); tdx->vp.tdcx_pages =3D NULL; =20 free_tdvpr: - if (tdx->vp.tdvpr_page) - __free_page(tdx->vp.tdvpr_page); + tdx_free_control_page(tdx->vp.tdvpr_page); tdx->vp.tdvpr_page =3D NULL; tdx->vp.tdvpr_pa =3D 0; =20 --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7AFA33A71A; Thu, 3 Sep 2026 01:51:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400298; cv=none; b=gwJnZt/77ZcUW4hhnumDK4P7QBYApcccKzoQHQ7qnGcTrgJXhzXPcPh0RruxRQUXyxxI/RpHahd3x6QXPv21i5H3MeP0cEZUu8l7ERzSM8wxwyjC12fahMsY/+bWUjC0I/WYWwd38pvphNlig9Gve/fS2nrwpc++reiVsfRTVBg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400298; c=relaxed/simple; bh=iA901QX0ICW3/nK/d5Wp8EzNVv2A6U90sa7YtRVw97w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vsrvzm30uDb9bIo8snuwNJHCTahcgMuEtdf7GW0pebAcqU2CDX0fK9TmLlx9nF4lvFrbw2wJGjTN+D45VDLQ9/O/uEHpRYZOTozw3hJu68MCYJmbFeqknpsFCnFULY3x/TjqJOBLF+9FiFI8BGbsQ7B8lYRYx/9Cz3sngsEYZGM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nGOPTdPa; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nGOPTdPa" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400296; x=1819936296; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iA901QX0ICW3/nK/d5Wp8EzNVv2A6U90sa7YtRVw97w=; b=nGOPTdPaKOxHVXAFq1u35HqWN5UYHd9QQVxanGxeuKY7Lgi1wQwu1GwB EWAnacVx3lConqc/m6Y23ilSApD22bB5uJh9I4wvlyh7Gw9aU8j8q91xM jNeHYZnqLkBw5uAqnVXyuSvh7LtJdONgLcqNZ+DokZRjqlxwFoJ9w9A3L x1ifWcKefncFx9w5qovt0/jxemJMMMEQGR10jSQqXHxxEg4XgLW3ClLix Wz07feV1IFxZ0MjPh6M1WAHOx/aHf2mOqIEKiFIWIBKjLkEBrzPmUFX0M 7bMxEi/Gta7BW2l/y/CzlCYmOb7Tq5U09idBl8xSZkQRdDnRcep9fEXfA w==; X-CSE-ConnectionGUID: zdYN8dmUQTumo75BDsyJ1Q== X-CSE-MsgGUID: ElnYxqMOSzexYlnhxPiiXA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469176" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469176" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:23 -0700 X-CSE-ConnectionGUID: XYjQ4mRLRxeK8xyNUJi1Vg== X-CSE-MsgGUID: uuAKoUD8Trqhy2WPZdifyg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770232" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu , Dave Hansen Subject: [PATCH v10 07/11] x86/virt/tdx: Add APIs to support Dynamic PAMT ops from KVM's fault path Date: Wed, 2 Sep 2026 18:51:09 -0700 Message-ID: <20260903015113.93343-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When handling an EPT violation, KVM holds a spinlock while manipulating the EPT. Before entering the spinlock it doesn't know how many EPT page tables will need to be installed or whether a huge page will be used. For this reason it allocates a worst case number of page tables that it might need as part of servicing the EPT violation. Under Dynamic PAMT (DPAMT) these pre-allocated pages will potentially need to have DPAMT backing pages installed for them. KVM already has helpers to manage topping up page caches before taking the MMU lock, but they cannot be passed from KVM to arch/x86 code. The problem of how and when to install the DPAMT backing pages for the pages given to the TDX module during the fault path has had a lot of design attempts. - Extracting KVM's MMU caches requires too much inlined code added to headers. - A few varieties of installing DPAMT backing when allocating the S-EPT page tables. (see links) - Using mempool_t to transfer the pages between KVM and arch/x86 doesn't work because the component is designed more around maintaining a pool of pages, rather than topping up a continually drained cache. So don't do these as they all had various problems. Instead just create a small simple data structure to use for handing a pre-allocated list of pages between KVM and arch/x86 code. Model this on KVM's existing MMU memory caches. Add a tdx_pamt_cache arg to tdx_pamt_get() so it can draw pages from a cache when needed. Not all DPAMT page installations will happen under spinlock, for example TD and vCPU scoped control pages. So have tdx_pamt_get() maintain the existing behavior of allocating from the page allocator when NULL is passed for the struct tdx_pamt_cache arg. This prevents excess allocations for cases where it can be avoided. Export the new helpers for KVM. AI was used under supervision to review code and workshop logs. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: Binbin Wu Reviewed-by: Chao Gao Reviewed-by: Yan Zhao Reviewed-by: Tony Lindgren Reviewed-by: Nikolay Borisov Reviewed-by: Dave Hansen Acked-by: Sohil Mehta Link: https://lore.kernel.org/kvm/aXENNKjAKTM9UJNH@google.com/ Link: https://lore.kernel.org/kvm/20260129011517.3545883-20-seanjc@google.c= om/ Link: https://lore.kernel.org/kvm/aYW5CbUvZrLogsWF@yzhao56-desk.sh.intel.co= m/ --- v10: - Change "Dynamic PAMT" to "DPAMT" in comments, and in the second reference in the logs. (Dave) - x86/tdx -> x86/virt/tdx in title to match the others (AI nit checker) v7: - Log/comment tweaks (Yan) - Drop Assisted-by tag and cover AI use in log (Dave) v6: - Filled out log from Sean's series --- arch/x86/include/asm/tdx.h | 16 +++++++++- arch/x86/virt/vmx/tdx/tdx.c | 61 ++++++++++++++++++++++++++++++++++--- 2 files changed, 71 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index f7442ad20e46d..8c7839d61296d 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -120,7 +120,21 @@ static inline bool tdx_supports_runtime_update(const s= truct tdx_sys_info *sysinf =20 bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo); =20 -int tdx_pamt_get(kvm_pfn_t pfn); +/* Simple structure for pre-allocating DPAMT pages outside of spinlocks. */ +struct tdx_pamt_cache { + struct list_head page_list; + int cnt; +}; + +static inline void tdx_init_pamt_cache(struct tdx_pamt_cache *cache) +{ + INIT_LIST_HEAD(&cache->page_list); + cache->cnt =3D 0; +} + +void tdx_free_pamt_cache(struct tdx_pamt_cache *cache); +int tdx_topup_pamt_cache(struct tdx_pamt_cache *cache, unsigned long npage= s); +int tdx_pamt_get(kvm_pfn_t pfn, struct tdx_pamt_cache *cache); void tdx_pamt_put(kvm_pfn_t pfn); =20 int tdx_guest_keyid_alloc(void); diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index c347600a0aabb..39865a2da5822 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -2050,12 +2050,33 @@ bool tdx_supports_dynamic_pamt(const struct tdx_sys= _info *sysinfo) return false; } =20 -static int alloc_pamt_array(struct page **pamt_pages) +static struct page *tdx_alloc_page_pamt_cache(struct tdx_pamt_cache *cache) +{ + struct page *page; + + page =3D list_first_entry_or_null(&cache->page_list, struct page, lru); + if (page) { + list_del(&page->lru); + cache->cnt--; + } + + return page; +} + +static struct page *alloc_dpamt_page(struct tdx_pamt_cache *cache) +{ + if (cache) + return tdx_alloc_page_pamt_cache(cache); + + return alloc_page(GFP_KERNEL_ACCOUNT); +} + +static int alloc_pamt_array(struct page **pamt_pages, struct tdx_pamt_cach= e *cache) { int i, j; =20 for (i =3D 0; i < TDX_DPAMT_ENTRY_PAGE_CNT; i++) { - pamt_pages[i] =3D alloc_page(GFP_KERNEL_ACCOUNT); + pamt_pages[i] =3D alloc_dpamt_page(cache); if (!pamt_pages[i]) goto err; } @@ -2132,7 +2153,7 @@ static u64 tdh_phymem_pamt_remove(kvm_pfn_t pfn, stru= ct page **pamt_pages) static DEFINE_SPINLOCK(dpamt_lock); =20 /* Bump DPAMT refcount for the given pfn and allocate DPAMT backing if nee= ded. */ -int tdx_pamt_get(kvm_pfn_t pfn) +int tdx_pamt_get(kvm_pfn_t pfn, struct tdx_pamt_cache *cache) { struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT]; atomic_t *dpamt_refcount; @@ -2142,7 +2163,7 @@ int tdx_pamt_get(kvm_pfn_t pfn) if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) return 0; =20 - ret =3D alloc_pamt_array(pamt_pages); + ret =3D alloc_pamt_array(pamt_pages, cache); if (ret) return ret; =20 @@ -2220,6 +2241,36 @@ void tdx_pamt_put(kvm_pfn_t pfn) } EXPORT_SYMBOL_FOR_KVM(tdx_pamt_put); =20 +void tdx_free_pamt_cache(struct tdx_pamt_cache *cache) +{ + struct page *page; + + while ((page =3D tdx_alloc_page_pamt_cache(cache))) + __free_page(page); +} +EXPORT_SYMBOL_FOR_KVM(tdx_free_pamt_cache); + +int tdx_topup_pamt_cache(struct tdx_pamt_cache *cache, unsigned long npage= s) +{ + if (WARN_ON_ONCE(!tdx_supports_dynamic_pamt(&tdx_sysinfo))) + return 0; + + npages *=3D TDX_DPAMT_ENTRY_PAGE_CNT; + + while (cache->cnt < npages) { + struct page *page =3D alloc_page(GFP_KERNEL_ACCOUNT); + + if (!page) + return -ENOMEM; + + list_add(&page->lru, &cache->page_list); + cache->cnt++; + } + + return 0; +} +EXPORT_SYMBOL_FOR_KVM(tdx_topup_pamt_cache); + /* * Return a page that can be gifted to the TDX module for use as a "contro= l" * page, i.e. pages that are used for control structures for a given TDX @@ -2233,7 +2284,7 @@ struct page *tdx_alloc_control_page(void) if (!page) return NULL; =20 - if (tdx_pamt_get(page_to_pfn(page))) { + if (tdx_pamt_get(page_to_pfn(page), NULL)) { __free_page(page); return NULL; } --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2655D30E835; Thu, 3 Sep 2026 01:51:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400296; cv=none; b=h8kH5iJR13TMHjVfhHt2thdHoMgShjMaxe640kq7sVZcFMJx+B0QhEVr9qPR+jf2d/fTr8VbC0wHP8LmpiJpSuE51R7aoA69vGkqAxdukAN4xDEBQUvfxOQkDFUJObCdXqUgFhsfsyarKq20ixqVVZPgAuE1WAW3BEuj6y7t/o8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400296; c=relaxed/simple; bh=yvaxJRKIvppfFL8wWPTvkEtg5yGvkXX0L2nrA19bQus=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cPHMZ+W4weOOSyrIs1DtL3lsXk0KQhsVXxjDgScp5bRG07Kx6kNzTw1SGCf80axLirVY7LxErp6ndre6IVJDOVYplILUNB2qjQLLUmTXKm3cv3qzvhdQj0wH+QzKDfAgg+kc74M6uqKygp7ghdMc9zVu1adUKUIKqYDNGQRdpQU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=cDVwKV+7; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="cDVwKV+7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400293; x=1819936293; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yvaxJRKIvppfFL8wWPTvkEtg5yGvkXX0L2nrA19bQus=; b=cDVwKV+7rw6fnr8WbVGqgTShPgnLhzSwlvePJXmTk/RlfdagZkj/IjpM nmwJ3fvDDuMygDG7kQLq1JRpz+/fcdIh+I8qHeVtmhFebxcAh1LoVerd+ a5hzV6wJbprahXFPCXVFMOQcf8xBjVugaB7vNIw6ECXnqICveXMprIRN1 LFfFiLX3GOsqjftcjcvAOU1vuLTnDNU9q8TsJaHSnpBZqjxsunaJxpqrs /x9XqCbQj3b/P+M5qwSUmQ5jpRuxdnqedupEc3JDigTq1ceVtK/ycc9bV sbBFehniUEgwpwJJCP7uAZpE6ynIZcEkBAVkgN0ubc78bSoRuEAypPc9O A==; X-CSE-ConnectionGUID: 1a7m6T2xRWeLNilSwaCoRg== X-CSE-MsgGUID: 4spGsPIXSHyPPF50fMBJGA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469179" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469179" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:23 -0700 X-CSE-ConnectionGUID: 4L8LDhUeRk6RCJmUDqiG2g== X-CSE-MsgGUID: BC/lyfAIS5qcE8GfpR7m+Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770235" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu , Dave Hansen Subject: [PATCH v10 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory Date: Wed, 2 Sep 2026 18:51:10 -0700 Message-ID: <20260903015113.93343-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kirill A. Shutemov" Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE). To access the per-vCPU PAMT caches without plumbing @vcpu throughout the TDP MMU, begrudgingly use kvm_get_running_vcpu() to get the vCPU, and bug the VM if KVM attempts to set an S-EPT leaf without an active vCPU. KVM only supports creating _new_ mappings in page (pre)fault paths, all of which require an active vCPU. The PAMT memory holds metadata for TDX protected memory. With Dynamic PAMT, PAMT_4K is allocated on demand. The kernel supplies the TDX module with a few pages that cover 2MB of host physical memory. Releases are balanced via tdx_pamt_put(): every control-page free goes through tdx_free_control_page(), and guest data pages are put directly on the successful tdh_mem_page_remove() path and in the tdx_mem_page_add/aug() error path. Signed-off-by: Kirill A. Shutemov Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson [rick: enhance log, reviewing, rebase, with help from AI tooling] Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Tony Lindgren Reviewed-by: Dave Hansen Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta Acked-by: Sean Christopherson --- v8: - Remove extra new line around *topup_external_cache() (Sean) - Make +1/-1 logic clearer by just coding the easily optimizable math. (Nikolay, Sean) - Do KVM_BUG_ON() on tdx_pamt_get() return. (Sean) v7: - Don't do to_tdx() before NULL check for readability (Binbin, Sean) - Fixup tags (Sean) - Export tdx_supports_dynamic_pamt() since it's used in KVM here and no longer an static inline. --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 4 ++ arch/x86/kvm/vmx/tdx.c | 64 ++++++++++++++++++++++++++---- arch/x86/kvm/vmx/tdx.h | 2 + arch/x86/virt/vmx/tdx/tdx.c | 1 + 6 files changed, 65 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-= x86-ops.h index e213c9ae3e301..5c358c40eae8a 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -99,6 +99,7 @@ KVM_X86_OP_OPTIONAL_RET0(tdp_has_smep) KVM_X86_OP(load_mmu_pgd) KVM_X86_OP_OPTIONAL_RET0(set_external_spte) KVM_X86_OP_OPTIONAL(free_external_spt) +KVM_X86_OP_OPTIONAL_RET0(topup_external_cache) KVM_X86_OP(has_wbinvd_exit) KVM_X86_OP(get_l2_tsc_offset) KVM_X86_OP(get_l2_tsc_multiplier) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 683bb8bf43a94..57d37491c7c09 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1644,6 +1644,7 @@ struct kvm_x86_ops { /* Update external page tables for page table about to be freed. */ void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp); =20 + int (*topup_external_cache)(struct kvm_vcpu *vcpu, int min_nr_spts); =20 bool (*has_wbinvd_exit)(void); =20 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 064ecc33b9267..2816861ca279c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -617,6 +617,10 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vc= pu, bool maybe_indirect) PT64_ROOT_MAX_LEVEL); if (r) return r; + + r =3D kvm_x86_call(topup_external_cache)(vcpu, PT64_ROOT_MAX_LEVEL); + if (r) + return r; } r =3D kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache, PT64_ROOT_MAX_LEVEL); diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 3592596b5649a..7c79f4d4bf99a 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -681,6 +681,8 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu) if (!irqchip_split(vcpu->kvm)) return -EINVAL; =20 + tdx_init_pamt_cache(&tdx->pamt_cache); + fpstate_set_confidential(&vcpu->arch.guest_fpu); vcpu->arch.apic->guest_apic_protected =3D true; INIT_LIST_HEAD(&tdx->vt.pi_wakeup_list); @@ -866,6 +868,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu) struct vcpu_tdx *tdx =3D to_tdx(vcpu); int i; =20 + tdx_free_pamt_cache(&tdx->pamt_cache); + if (vcpu->cpu !=3D -1) { KVM_BUG_ON(tdx->state =3D=3D VCPU_TD_STATE_INITIALIZED, vcpu->kvm); tdx_flush_vp_on_cpu(vcpu); @@ -1618,6 +1622,17 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t r= oot_hpa, int pgd_level) td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa); } =20 +static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min_nr= _spts) +{ + /* + * Minus one page to exclude the root SPT, but plus one page for a + * possible 4KB private mapping. + */ + min_nr_spts +=3D -1 + 1; + + return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min_nr_spts); +} + static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level leve= l, kvm_pfn_t pfn) { @@ -1676,16 +1691,28 @@ static struct page *tdx_spte_to_sept_pt(struct kvm = *kvm, gfn_t gfn, static int tdx_sept_map_nonleaf_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, u64 new_spte) { + struct kvm_vcpu *vcpu =3D kvm_get_running_vcpu(); gpa_t gpa =3D gfn_to_gpa(gfn); u64 err, entry, level_state; struct page *sept_pt; + int ret; + + if (KVM_BUG_ON(!vcpu, kvm)) + return -EIO; =20 sept_pt =3D tdx_spte_to_sept_pt(kvm, gfn, new_spte, level); if (!sept_pt) return -EIO; =20 + ret =3D tdx_pamt_get(page_to_pfn(sept_pt), &to_tdx(vcpu)->pamt_cache); + if (KVM_BUG_ON(ret, kvm)) + return ret; + err =3D tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, level, sept_pt, &entry, &level_state); + if (err) + tdx_pamt_put(page_to_pfn(sept_pt)); + if (unlikely(tdx_operand_busy(err))) return -EBUSY; =20 @@ -1698,8 +1725,13 @@ static int tdx_sept_map_nonleaf_spte(struct kvm *kvm= , gfn_t gfn, static int tdx_sept_map_leaf_spte(struct kvm *kvm, gfn_t gfn, enum pg_leve= l level, u64 new_spte) { + struct kvm_vcpu *vcpu =3D kvm_get_running_vcpu(); struct kvm_tdx *kvm_tdx =3D to_kvm_tdx(kvm); kvm_pfn_t pfn =3D spte_to_pfn(new_spte); + int ret; + + if (KVM_BUG_ON(!vcpu, kvm)) + return -EIO; =20 /* TODO: handle large pages. */ if (KVM_BUG_ON(level !=3D PG_LEVEL_4K, kvm)) @@ -1707,6 +1739,10 @@ static int tdx_sept_map_leaf_spte(struct kvm *kvm, g= fn_t gfn, enum pg_level leve =20 WARN_ON_ONCE((new_spte & VMX_EPT_RWX_MASK) !=3D VMX_EPT_RWX_MASK); =20 + ret =3D tdx_pamt_get(pfn, &to_tdx(vcpu)->pamt_cache); + if (KVM_BUG_ON(ret, kvm)) + return ret; + /* * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() * before kvm_tdx->state. Userspace must not be allowed to pre-fault @@ -1719,10 +1755,15 @@ static int tdx_sept_map_leaf_spte(struct kvm *kvm, = gfn_t gfn, enum pg_level leve * If the TD isn't finalized/runnable, then userspace is initializing * the VM image via KVM_TDX_INIT_MEM_REGION; ADD the page to the TD. */ - if (unlikely(kvm_tdx->state !=3D TD_STATE_RUNNABLE)) - return tdx_mem_page_add(kvm, gfn, level, pfn); + if (likely(kvm_tdx->state =3D=3D TD_STATE_RUNNABLE)) + ret =3D tdx_mem_page_aug(kvm, gfn, level, pfn); + else + ret =3D tdx_mem_page_add(kvm, gfn, level, pfn); =20 - return tdx_mem_page_aug(kvm, gfn, level, pfn); + if (ret) + tdx_pamt_put(pfn); + + return ret; } =20 /* @@ -1819,6 +1860,7 @@ static int tdx_sept_remove_leaf_spte(struct kvm *kvm,= gfn_t gfn, return -EIO; =20 tdx_quirk_reset_paddr(PFN_PHYS(pfn), PAGE_SIZE); + tdx_pamt_put(pfn); return 0; } =20 @@ -1862,6 +1904,8 @@ static int tdx_sept_set_private_spte(struct kvm *kvm,= gfn_t gfn, u64 old_spte, */ static void tdx_sept_free_private_spt(struct kvm *kvm, struct kvm_mmu_page= *sp) { + struct page *sept_pt =3D virt_to_page(sp->external_spt); + /* * KVM doesn't (yet) zap page table pages in mirror page table while * TD is active, though guest pages mapped in mirror page table could be @@ -1875,15 +1919,15 @@ static void tdx_sept_free_private_spt(struct kvm *k= vm, struct kvm_mmu_page *sp) * the page to prevent the kernel from accessing the encrypted page. */ if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) || - tdx_reclaim_page(virt_to_page(sp->external_spt))) + tdx_reclaim_page(sept_pt)) goto out; =20 /* - * Immediately free the S-EPT page because RCU-time free is unnecessary - * after TDH.PHYMEM.PAGE.RECLAIM ensures there are no outstanding - * readers. + * Immediately free the S-EPT page as the TDX subsystem doesn't support + * freeing pages from RCU callbacks, and more importantly because + * TDH.PHYMEM.PAGE.RECLAIM ensures there are no outstanding readers. */ - free_page((unsigned long)sp->external_spt); + tdx_free_control_page(sept_pt); out: sp->external_spt =3D NULL; } @@ -3480,6 +3524,10 @@ int __init tdx_hardware_setup(void) =20 vt_x86_ops.set_external_spte =3D tdx_sept_set_private_spte; vt_x86_ops.free_external_spt =3D tdx_sept_free_private_spt; + + if (tdx_supports_dynamic_pamt(tdx_sysinfo)) + vt_x86_ops.topup_external_cache =3D tdx_topup_external_pamt_cache; + vt_x86_ops.protected_apic_has_interrupt =3D tdx_protected_apic_has_interr= upt; return 0; =20 diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index ac8323a68b163..fd368e3ee060b 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -72,6 +72,8 @@ struct vcpu_tdx { =20 u64 map_gpa_next; u64 map_gpa_end; + + struct tdx_pamt_cache pamt_cache; }; =20 void tdh_vp_rd_failed(struct vcpu_tdx *tdx, char *uclass, u32 field, u64 e= rr); diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 39865a2da5822..ff00ee6d5705a 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -2049,6 +2049,7 @@ bool tdx_supports_dynamic_pamt(const struct tdx_sys_i= nfo *sysinfo) /* To be enabled when kernel is ready. */ return false; } +EXPORT_SYMBOL_FOR_KVM(tdx_supports_dynamic_pamt); =20 static struct page *tdx_alloc_page_pamt_cache(struct tdx_pamt_cache *cache) { --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99320314B63; Thu, 3 Sep 2026 01:51:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400298; cv=none; b=lE2e0doJigXuS5M5xr7FJYDEe8ujx6dg4Lur+XGEVfV/OXmCg5VLuYzsGk5547XWcQ4xYcfBJNgZoaS9kJGVRPwTMqks+OQkcSL+uhc+QUHedNgxtUBi+K0GmrJ/7GdnPXtIUNzaYprqXV/K3FDzl+te7X4V8y7cOx6SXyYKFDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400298; c=relaxed/simple; bh=dtu1ggg5VIypqWoDzB4J7tyX0oD/kIqG7vXK2qDzpJE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=smakK/7tnf1OUsRT1EOE3Ozbw/17XhEzruekWk4tlaLhrG4vJIIRtxGi3XCs3Ay2RkBEK/nRwE8jFk9ycTTvfGEB5PiMD2MHMODbWbce5DHdNFYEIcAZF8XJMO4esqWv/HsgJwjzGMV3CCxBfKh29P7oR4g4djDz9p9roe7oEN8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=aKgH04P5; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="aKgH04P5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400296; x=1819936296; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dtu1ggg5VIypqWoDzB4J7tyX0oD/kIqG7vXK2qDzpJE=; b=aKgH04P5aTG0XMYUqGhGHMXwpc069Y3lTUPYGlH88P03oJzkMlz1XjH2 KuXk7L0gRKYeouL+s+g0VF37qGVyKyH7QIHnOEhoCDtpgWGIyIMRoYft4 L6sgkcqhbFXuxYJylYxz1FGk+Zcolzy1obwuJ/XKcolZ9AfTj+65fgmXW WbKeZ9wq8jK64bxDmH8MbsWf7MoOBWmeQOWVXUn4AYlXuMeyOjmOXIu5d zNOmGVVqhpEt8tA8CssDTVuHgysevkU+3lUQMvKSFZ80qIuXqxjmceky9 IMD4yafjKpXQJ9HPcrdDD/CxxrudCuyGYMh0ydxCWzoLd957a15i23apN g==; X-CSE-ConnectionGUID: P1ZWqpjZRk2sgWU0OOoYsA== X-CSE-MsgGUID: 4xVhqUdgQL+dq/4LjOjHDA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469184" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469184" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:24 -0700 X-CSE-ConnectionGUID: S0y2GxwPSI6AMG6uNKS59g== X-CSE-MsgGUID: fVqLM6QtTWq1a7Bjr3cajQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770238" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning Subject: [PATCH v10 09/11] x86/virt/tdx: Enable Dynamic PAMT Date: Wed, 2 Sep 2026 18:51:11 -0700 Message-ID: <20260903015113.93343-10-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The Physical Address Metadata Table (PAMT) holds TDX metadata for physical memory and must be allocated by the kernel during TDX module initialization. Dynamic PAMT (DPAMT) is a TDX module feature that can reduce this memory use by allocating part of the PAMT dynamically. The TDX module exposes whether DPAMT is supported via a bit in the 'features0' metadata. Unfortunately, the TDX module exposes the feature as supported even when it does not support using it with the number of keyids currently configured in the BIOS. Since no TDX modules exist today with that issue fixed, make the feature default off to prevent users from upgrading their kernel and encountering TDX erroring out when trying to enable DPAMT. For the decision of whether to make it a boot time option and/or compile time option, consider that DPAMT's memory savings are significant enough to make it a good default configuration. That is most TDX users should want it unless they have strange keyid configurations. The feature increases the kernel size by 2KB (when TDX is configured in the build). All pieces are in place to enable DPAMT if it is supported and the user passes a kernel parameter. AI was used to review code. It made a style suggestion. Based on a patch originally by Kiryl Shutsemau. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Tony Lindgren Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta --- v10: - Change "Dynamic PAMT" to "DPAMT" at the second reference in the logs. (Dave) - Adjust tdx_supports_dynamic_pamt() to check multiple conditions in a more tip style. (AI nit checker) v8: - Order tdx_dpamt in kernel-parameters.txt (Sohil) - Make tdx_enable_dpamt static (Sashiko) v7: - Add kernel parameter following some twists and turns, deriving originally from a comment by (Chao) --- .../admin-guide/kernel-parameters.txt | 7 ++++++ arch/x86/include/asm/tdx.h | 1 + arch/x86/virt/vmx/tdx/tdx.c | 23 +++++++++++++++++-- 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 68647ff4bdd24..f32aa58f744a1 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7582,6 +7582,13 @@ Kernel parameters =20 tdfx=3D [HW,DRM] =20 + tdx_dpamt=3D + [X86] Controls whether TDX will use Dynamic PAMT + to save memory, when supported. + + Valid parameters: "on", "off" + Default: "off" + test_suspend=3D [SUSPEND] Format: { "mem" | "standby" | "freeze" }[,N] Specify "mem" (for Suspend-to-RAM) or "standby" (for diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 8c7839d61296d..e186dfe5bf885 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -36,6 +36,7 @@ /* Bit definitions of TDX_FEATURES0 metadata field */ #define TDX_FEATURES0_TD_PRESERVING BIT_ULL(1) #define TDX_FEATURES0_NO_RBP_MOD BIT_ULL(18) +#define TDX_FEATURES0_DYNAMIC_PAMT BIT_ULL(36) =20 #ifndef __ASSEMBLER__ =20 diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index ff00ee6d5705a..3daa8c63f9c51 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -50,6 +50,8 @@ /* Number of DPAMT pages to be provided to TDX module per 2MB region of PA= */ #define TDX_DPAMT_ENTRY_PAGE_CNT 2 =20 +static bool tdx_enable_dpamt __ro_after_init; + struct tdx_module_state { bool initialized; bool sysinit_done; @@ -1031,6 +1033,8 @@ static __init int construct_tdmrs(struct list_head *t= mb_list, return ret; } =20 +#define TDX_SYS_CONFIG_DYNAMIC_PAMT BIT(16) + static __init int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid) { @@ -1059,6 +1063,12 @@ static __init int config_tdx_module(struct tdmr_info= _list *tdmr_list, args.rcx =3D __pa(tdmr_pa_array); args.rdx =3D tdmr_list->nr_consumed_tdmrs; args.r8 =3D global_keyid; + + if (tdx_supports_dynamic_pamt(&tdx_sysinfo)) { + pr_info("Enable Dynamic PAMT\n"); + args.r8 |=3D TDX_SYS_CONFIG_DYNAMIC_PAMT; + } + ret =3D seamcall_prerr(TDH_SYS_CONFIG, &args); =20 /* Free the array as it is not required anymore. */ @@ -2046,8 +2056,10 @@ EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid); =20 bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo) { - /* To be enabled when kernel is ready. */ - return false; + if (!tdx_enable_dpamt) + return false; + + return sysinfo->features.tdx_features0 & TDX_FEATURES0_DYNAMIC_PAMT; } EXPORT_SYMBOL_FOR_KVM(tdx_supports_dynamic_pamt); =20 @@ -2308,6 +2320,13 @@ void tdx_free_control_page(struct page *page) } EXPORT_SYMBOL_FOR_KVM(tdx_free_control_page); =20 +static int __init tdx_dpamt_setup(char *str) +{ + return kstrtobool(str, &tdx_enable_dpamt) =3D=3D 0; +} + +__setup("tdx_dpamt=3D", tdx_dpamt_setup); + void tdx_sys_disable(void) { struct tdx_module_args args =3D {}; --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 064AA33EB17; Thu, 3 Sep 2026 01:51:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400299; cv=none; b=RkqLCEo1rY9A2hyrCGNc3o2EUN2XIxYFUvg35TXssDG5N6cNEw7w0Sl8a3gyT+eZSgh07C2ocfY2CwwwCW9A+5ykcskkJAAHYHt10M9WIkZqHRnIJ+y+kRniYKWl3Ivf+ZCnVPJS86h/FCt4L7rZMdcWG9LvdNNig4bj10CJRNI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400299; c=relaxed/simple; bh=/CB9ippxrpOsGJgcDiFoqTZKAywhBcGUCX3drlUDM9o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A5kbY+R+XQqlZ4jC3FBRxHFHhFr6r7DetdLWo1bxa1UNO+Imojx1wO1qc7ww+EsJy7NfGA+hFNWjT3ZwxswdMpwOCNi+NPp9oztE4F2DJop4bp0wuYOSxv8qaCe28bsLda0dtJvd5AwteaN2JOzvOkbhKVsmN1gH/E9h4PWPGh0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=OP1nE7N8; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="OP1nE7N8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400297; x=1819936297; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/CB9ippxrpOsGJgcDiFoqTZKAywhBcGUCX3drlUDM9o=; b=OP1nE7N86jl5x6P7Zf5J9f88yC92DKQogYbjNxjNnWDggQI4i9XvCJR5 gvBs892/AVPfJ/XZHdMvtgHj3b+kpehYgvOnxzoFnN0jIDpg2lhK3OTeo xojZdC6A/ax+HKD+xMyiBd3h/9rkoZCfOzef6+4TLC0a9iHh90lJSieat pTiDX2IgYNjp0sU+jnzKzd98mPpKWRj0wAvez+eqiDb4kYRBB7uzIh3tE aCPKsoINxb/EV6C+J0A3n5aijilQYbcS3nrJS5/yc/VKkbmRy9/FyYXlz QSu8G3LsVUOgn5+vvsdPh0ua7GQvtte1YjW41d4fjtS8UL3a+70bmft2S Q==; X-CSE-ConnectionGUID: TQPxrIwTQSmOUHarVrPhnQ== X-CSE-MsgGUID: M8QjSOtdS5SmUw9lcasxHA== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469185" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469185" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:24 -0700 X-CSE-ConnectionGUID: IDGYm3eoThaIO9Cc5yXNNw== X-CSE-MsgGUID: YVKeJmFjRyicB83AcbjztA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770242" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Binbin Wu Subject: [PATCH v10 10/11] Documentation/x86: Add documentation for TDX's Dynamic PAMT Date: Wed, 2 Sep 2026 18:51:12 -0700 Message-ID: <20260903015113.93343-11-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Expand TDX documentation to include information on the Dynamic PAMT feature. The new section explains PAMT support in the TDX module and how Dynamic PAMT affects the kernel memory use. Some of the docs verbiage was provided by Dave Hansen (see link). AI was used under supervision to review the docs. Based on a patch originally by Kiryl Shutsemau. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Binbin Wu Reviewed-by: Tony Lindgren Acked-by: Sohil Mehta Link: https://lore.kernel.org/all/8e40862e-891a-4cad-8bb8-06ad25ad6061@inte= l.com/ --- v10: - Updated verbiage from Dave, but unify 4k to 4KB to be consistent with the rest of the series. v7: - Spell out PAMT acronym (Binbin) - Drop Assisted-by tag and cover AI use in log (Dave) - Add info about kernel parameter v6: - Add missing word (Binbin) - Use "::" instead of ":" - Make format of dmesg example accurate --- .../admin-guide/kernel-parameters.txt | 3 +++ Documentation/arch/x86/tdx.rst | 27 +++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index f32aa58f744a1..e7558b8382acf 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7589,6 +7589,9 @@ Kernel parameters Valid parameters: "on", "off" Default: "off" =20 + For details see: + Documentation/arch/x86/tdx.rst + test_suspend=3D [SUSPEND] Format: { "mem" | "standby" | "freeze" }[,N] Specify "mem" (for Suspend-to-RAM) or "standby" (for diff --git a/Documentation/arch/x86/tdx.rst b/Documentation/arch/x86/tdx.rst index 3303499ad4c6f..de09967ce8d8c 100644 --- a/Documentation/arch/x86/tdx.rst +++ b/Documentation/arch/x86/tdx.rst @@ -200,6 +200,33 @@ reflects the TCB of the currently running TDX module a= nd therefore changes after an update. By contrast, TEE_TCB_SVN reflects the TCB at TD launch time and is not affected. =20 +Dynamic PAMT +------------ + +The Physical Address Metadata Table (PAMT) is metadata in which the TDX +module keeps data about each physical page (think struct page). Space +for it is allocated by the VMM, consumes up to about 0.4% of system +memory and needs to be supplied to the TDX module when the TDX module is +first loaded. + +Dynamic PAMT is an add-on feature that allows a VMM to dynamically +allocate the part of the PAMT which tracks 4KB pages. This reduces the +amount of memory that TDX consumes while TDs are not in use. + +When Dynamic PAMT is in use, dmesg shows it like:: + + [..] virt/tdx: Enable Dynamic PAMT + [..] virt/tdx: 10092 KB allocated for PAMT + [..] virt/tdx: TDX-Module initialized + +Dynamic PAMT is only enabled when supported and the ``tdx_dpamt=3D`` kernel +parameter is set to "on". The feature is off by default because TDX module +internal details prevent Dynamic PAMT from working on all keyid partitioni= ng +configurations. When the TDX module is fixed to include these constraints = in +its enumeration of Dynamic PAMT support, kernel support can be changed to +default on. For more information, consult the Intel TDX documentation about +Dynamic PAMT. + TDX Interaction to Other Kernel Components ------------------------------------------ =20 --=20 2.55.0 From nobody Sat Sep 26 08:37:43 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B153E345EC1; Thu, 3 Sep 2026 01:51:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400301; cv=none; b=X5JiPIRZ9H7Hq1hSy4Y/vNm7zV62/fuYfZXFk9A3k56he5AQ+KAzOjGk+TkREq7w2Z9t4WhgnByjLGHSDD+6/9fR7K5yoKWNTXSuPYisuNeCHCvx6IWNeMz3fXAHk/IU74crQAuzNPhHFAFVnPe8ockdnLsyOucEr48Quy0S9Fg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788400301; c=relaxed/simple; bh=NEuIFUQir2yViFwLjPozotQd36++IpvCcKGowJE+yzo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tAumX/UavEozfc2S1zWnV35z37tYx1iD/dm+15TFT81UEWhUidmiWARD5xCM2uAIsvYfpcV1Sd2OMgp1Lou0UHZ2Cls6SCuquzyuDSBYO6vzn+XXAoJW0N+sGDWaRMIxmCXdv0ZGFLRU5SQcf4vp4vsLqQgMlJxGX4LeNrKWj2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=dt2QOSZO; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="dt2QOSZO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1788400299; x=1819936299; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NEuIFUQir2yViFwLjPozotQd36++IpvCcKGowJE+yzo=; b=dt2QOSZOMFE7PTj8hKMtfrqyHyL3vjmRrsVpbR4N7lMbtwgLDf+wd9iD dQ+NjWN3No6mkk49mtmXy6Z5lmefaeJ8RuPKxwyU6r9izsLZzz1bMNkws m2bZqKOtNhYPEXjfLAXMcO4Ecqtuts6Fo0EIKIyiZSZY9O/WLMSrYa7+c eLwtDPVg/FWSG+5VByRyDOgdf9xhEiDOu2ET7goc/QEKQv3f6o0SeAa+D zeolWbE9KZjndZ5LltdqKwQAA39z/hV+ROLFE5uQjpLEFQYJtXWs7W/Sc 3T9EUFSXsPbPgYEzwjmk0c0xb1ozcXl8pEAYc2JTT3Fd1DifChO7H2pSN g==; X-CSE-ConnectionGUID: hdY2HlSyRoe5k002fnYu1g== X-CSE-MsgGUID: HXPL2Ck/RU2oKMSoY51mgg== X-IronPort-AV: E=McAfee;i="6800,10657,11894"; a="99469188" X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="99469188" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:24 -0700 X-CSE-ConnectionGUID: 97J0hKp1QWSAiiTRn7u93Q== X-CSE-MsgGUID: kBfcBs+eRTeNq0N8bTiJEw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,258,1779174000"; d="scan'208";a="307770245" Received: from rpedgeco-desk.jf.intel.com ([10.88.27.135]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Sep 2026 18:51:22 -0700 From: Rick Edgecombe To: bp@alien8.de, dave.hansen@intel.com, hpa@zytor.com, kas@kernel.org, kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com, tglx@kernel.org, vannapurve@google.com, x86@kernel.org, chao.gao@intel.com, yan.y.zhao@intel.com, kai.huang@intel.com, tony.lindgren@linux.intel.com, binbin.wu@intel.com, sohil.mehta@intel.com Cc: rick.p.edgecombe@intel.com, Hongyu Ning , Dave Hansen Subject: [PATCH v10 11/11] x86/virt/tdx: Optimize tdx_pamt_get/put() Date: Wed, 2 Sep 2026 18:51:13 -0700 Message-ID: <20260903015113.93343-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903015113.93343-1-rick.p.edgecombe@intel.com> References: <20260903015113.93343-1-rick.p.edgecombe@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The Dynamic PAMT (DPAMT) get/put helpers use a global spinlock to serialize all refcount updates and SEAMCALL invocations. This gives correct behavior for concurrent callers, but leads to contention. It is especially bad from the KVM side, which is designed to allow faulting in EPT under a shared lock. With the global spinlock, not only is the lock an exclusive one, but it is for all TDs instead of just a single one. But taking the global lock each time is actually unnecessary. Only the 0->1 and 1->0 refcount transitions actually need the lock (to pair with SEAMCALLs that actually add and remove with the DPAMT pages). The common case of incrementing or decrementing a non-zero refcount can be done locklessly. So create a fast and slow path. Check the refcount outside the lock and only take it for the slow path (0->1 and 1->0 transitions). On the put side make the refcount adjustment and lock taking atomic so if a 'get' happens between them, it doesn't cause the DPAMT to be freed incorrectly. On the get side there is no technique for doing the refcount adjustment and lock atomically, so check the refcount again inside the lock. AI was used under supervision to collect/apply feedback, review code and workshop logs. It assisted in identifying/evaluating the stale conditionals for the races resolved from the atomic_dec_and_lock() change. Separate from atomic_dec_and_lock() fallout, it suggested to change atomic_inc() to atomic_set(pamt_refcount, 1) in the put error path for the sake of being more precise, which Kiryl had also suggested in the past. The model also suggested updated comments following the atomic_dec_and_lock() change based on some directed prompting. The comments were subsequently edited or further prompted for fine tuning. Based on a patch originally by Kiryl Shutsemau. Signed-off-by: Rick Edgecombe Tested-by: Hongyu Ning Reviewed-by: Chao Gao Reviewed-by: Tony Lindgren Reviewed-by: Nikolay Borisov Reviewed-by: Dave Hansen Reviewed-by: Vishal Annapurve Acked-by: Sohil Mehta --- v10: - Change "Dynamic PAMT" to "DPAMT" in comments. (Dave) v7: - Drop Assisted-by tag and cover AI use in log. (Dave) - Move to end of the series. - Use atomic_inc_not_zero() in this patch inside the spin_lock(), as suggested on the non-optimized patch by (Dave). v6: - Fix "tdx_pamt_add()" typo to "tdx_pamt_get()" in lost-race comment - Fix error path bug: set ret =3D -EIO and use WARN_ON_ONCE() instead of pr_err() for unexpected PAMT.ADD failures (Sean) - Use "set the refcount 0->1" wording to match atomic_set() usage - Wrap comments to 80 columns - Switch to atomic_dec_and_lock() and remove handling of races that are no longer needed as a result. Adjust comments as appropriate. (Dave) - Adjustments from dropping error helper patches --- arch/x86/virt/vmx/tdx/tdx.c | 44 ++++++++++++++++++++++++------------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 3daa8c63f9c51..2acab6e5f5f87 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -2176,28 +2176,41 @@ int tdx_pamt_get(kvm_pfn_t pfn, struct tdx_pamt_cac= he *cache) if (!tdx_supports_dynamic_pamt(&tdx_sysinfo)) return 0; =20 - ret =3D alloc_pamt_array(pamt_pages, cache); - if (ret) - return ret; - dpamt_refcount =3D tdx_find_dpamt_refcount(pfn); =20 - spin_lock(&dpamt_lock); - /* * If the DPAMT entry is already added (i.e. refcount >=3D 1), * then just increment the refcount. */ + if (atomic_inc_not_zero(dpamt_refcount)) + return 0; + + ret =3D alloc_pamt_array(pamt_pages, cache); + if (ret) + return ret; + + spin_lock(&dpamt_lock); + + /* + * Unlike tdx_pamt_put() which uses atomic_dec_and_lock() to + * atomically handle the 1->0 transition, the get side has no + * equivalent combined primitive for 0->1. Recheck under the + * lock since another get may have already done the 0->1 + * transition after both saw atomic_inc_not_zero() fail. + */ if (atomic_inc_not_zero(dpamt_refcount)) goto out_free; =20 - /* Try to add the PAMT page and take the refcount 0->1. */ tdx_status =3D tdh_phymem_pamt_add(pfn, pamt_pages); if (WARN_ON_ONCE(tdx_status !=3D TDX_SUCCESS)) { ret =3D -EIO; goto out_free; } =20 + /* + * The refcount is zero, and this locked path is the + * only way to increase it from 0->1. + */ atomic_set(dpamt_refcount, 1); spin_unlock(&dpamt_lock); return 0; @@ -2222,17 +2235,13 @@ void tdx_pamt_put(kvm_pfn_t pfn) =20 dpamt_refcount =3D tdx_find_dpamt_refcount(pfn); =20 - spin_lock(&dpamt_lock); /* * If there is more than 1 reference on the DPAMT entry, don't * remove it yet. Just decrement the refcount. */ - if (atomic_read(dpamt_refcount) > 1) { - atomic_dec(dpamt_refcount); - goto out_unlock; - } + if (!atomic_dec_and_lock(dpamt_refcount, &dpamt_lock)) + return; =20 - /* Try to remove the pamt page and take the refcount 1->0. */ tdx_status =3D tdh_phymem_pamt_remove(pfn, pamt_pages); =20 /* @@ -2242,10 +2251,15 @@ void tdx_pamt_put(kvm_pfn_t pfn) * failure indicates a kernel bug, memory is being leaked, and * the dangling DPAMT entry may cause future operations to fail. */ - if (WARN_ON_ONCE(tdx_status !=3D TDX_SUCCESS)) + if (WARN_ON_ONCE(tdx_status !=3D TDX_SUCCESS)) { + /* + * atomic_dec_and_lock() already decremented it to 0, + * but the DPAMT entry still exists since REMOVE failed. + */ + atomic_set(dpamt_refcount, 1); goto out_unlock; + } =20 - atomic_set(dpamt_refcount, 0); spin_unlock(&dpamt_lock); free_pamt_array(pamt_pages); return; --=20 2.55.0