From nobody Sat Feb 7 22:07:20 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4552E651AC for ; Tue, 19 Mar 2024 01:48:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812940; cv=none; b=IM8eNuP558ESbWNICkXlKlBo4oPMNzW/I/zioBSPyqertqztGLtmkGu1rUo/4A4Fv1Iu+OSdRDbZsTky9B/BOiM2OvXnw2pHgEKX60znNWaP6X1Z90ecjRBSgctTy6ozx+LYrR7XezraX0O8u92T5P5D2NYq2vs0tTz+HYBs8iA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812940; c=relaxed/simple; bh=FlX7CgjifVeRrQ/QQNoSfgHgX0lV/1ajjGELuX3hAgc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=msOw8tZ/P/ekBMxJjkj7FOev/TMXVgQBYQzqpKn9lOM55Rgf+oiwezxvQYOyKFyinF9Dj33Nt78lkJF9BXPCJ/EitdyxXa27ivhsvGS6I2ENJOvhSMWWT83AZP/02/OantdL/CSJPwIbd5Bacd2PfafKJ/WcZpBWzb1kxKsbsNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=N6AMwDdo; arc=none smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="N6AMwDdo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710812938; x=1742348938; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FlX7CgjifVeRrQ/QQNoSfgHgX0lV/1ajjGELuX3hAgc=; b=N6AMwDdoMvcY4K6ojIsH2BY8ivkMOJg0FxOashOPMU2WIaJ84U5xVMiB DX2/B6aGE8Mc3w6HqHGK101h3fdLJQCaQqFf8SSCoI/BY0mgOEBmZ005z vOHRxnCYgTXoCUgAsZv2UM0bSv4lXcAT96ynwQZEnA2Yiwx2SH5x58imP 6LnTxoJdyfhGunxuHPiIkNTpvhjjoTJjzGTA+/yGa2uUOISro2Zj1JVgb QYcXh/V+PfriAJ3UIsUB+jEq93vf4VrClc9bsJtaAGle9rxjnS/x0CWIk cseJCyy0J18LS5uBx0xfIWPjcsEhIx9trqKUAPHktU02MbrL0AydN6/+L g==; X-IronPort-AV: E=McAfee;i="6600,9927,11017"; a="5776877" X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="5776877" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:48:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="13697326" Received: from server.sh.intel.com ([10.239.53.117]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:48:54 -0700 From: Kai Huang To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, dave.hansen@intel.com, bp@alien8.de, kirill.shutemov@linux.intel.com, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, luto@kernel.org, peterz@infradead.org, rick.p.edgecombe@intel.com, thomas.lendacky@amd.com, ashish.kalra@amd.com, chao.gao@intel.com, bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com Subject: [PATCH v2 1/5] x86/kexec: do unconditional WBINVD in stop_this_cpu() Date: Tue, 19 Mar 2024 01:48:44 +0000 Message-Id: <0b7223b39510a51fdb164a847deab5778de25fb2.1710811610.git.kai.huang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" TL;DR: Change to do unconditional WBINVD in stop_this_cpu() to cover kexec support for both AMD SME and Intel TDX, despite there _was_ some issue preventing doing so but now has got fixed. Long version: Both SME and TDX can leave caches in incoherent state due to memory encryption. During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel. Currently, for SME the kernel only does WBINVD in stop_this_cpu() when the kernel determines the hardware supports SME. To support TDX, one option is to extend that specific check to cover both SME and TDX. However, instead of sprinkling around vendor-specific checks, it's better to just do unconditional WBINVD. Kexec() is a slow path, and it is acceptable to have an additional WBINVD in order to have simple and easy to maintain code. Note: Historically, there _was_ an issue preventing doing unconditional WBINVD but that has been fixed. When SME kexec() support was initially added in commit bba4ed011a52: ("x86/mm, kexec: Allow kexec to be used with SME") WBINVD was done unconditionally. However since then some issues were reported that different Intel systems would hang or reset due to that commit. To try to fix, a later commit f23d74f6c66c: ("x86/mm: Rework wbinvd, hlt operation in stop_this_cpu()") then changed to only do WBINVD when hardware supports SME. While this commit made the reported issues go away, it didn't pinpoint the root cause. Also, it didn't handle a corner case[*] correctly, which resulted in the reveal of the root cause and the final fix by commit 1f5e7eb7868e: ("x86/smp: Make stop_other_cpus() more robust") See [1][2] for more information. Further testing of doing unconditional WBINVD based on the above fix on the problematic machines (that issues were originally reported) confirmed the issues couldn't be reproduced. See [3][4] for more information. Therefore, it is safe to do unconditional WBINVD now. [*] The commit didn't check whether the CPUID leaf is available or not. Making unsupported CPUID leaf on Intel returns garbage resulting in unintended WBINVD which caused some issue (followed by the analysis and the reveal of the final root cause). The corner case was independently fixed by commit 9b040453d444: ("x86/smp: Dont access non-existing CPUID leaf") [1]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642b= Q3fTBbSAVg@mail.gmail.com/T/#m300f3f9790850b5daa20a71abcc200ae8d94a12a [2]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642b= Q3fTBbSAVg@mail.gmail.com/T/#ma7263a7765483db0dabdeef62a1110940e634846 [3]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642b= Q3fTBbSAVg@mail.gmail.com/T/#mc043191f2ff860d649c8466775dc61ac1e0ae320 [4]: https://lore.kernel.org/lkml/CALu+AoQKmeixJdkO07t7BtttN7v3RM4_aBKi642b= Q3fTBbSAVg@mail.gmail.com/T/#md23f1a8f6afcc59fa2b0ac1967f18e418e24347c Signed-off-by: Kai Huang Suggested-by: Borislav Petkov Cc: Tom Lendacky Cc: Dave Young --- arch/x86/kernel/process.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b8441147eb5e..b375f069dd2a 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -813,19 +813,14 @@ void __noreturn stop_this_cpu(void *dummy) mcheck_cpu_clear(c); =20 /* - * Use wbinvd on processors that support SME. This provides support - * for performing a successful kexec when going from SME inactive - * to SME active (or vice-versa). The cache must be cleared so that - * if there are entries with the same physical address, both with and - * without the encryption bit, they don't race each other when flushed - * and potentially end up with the wrong entry being committed to - * memory. + * The kernel could leave caches in incoherent state on SME/TDX + * capable platforms. Flush cache to avoid silent memory + * corruption for these platforms. * - * Test the CPUID bit directly because the machine might've cleared - * X86_FEATURE_SME due to cmdline options. + * stop_this_cpu() is not a fast path, just do unconditional + * WBINVD for simplicity. */ - if (c->extended_cpuid_level >=3D 0x8000001f && (cpuid_eax(0x8000001f) & B= IT(0))) - native_wbinvd(); + native_wbinvd(); =20 /* * This brings a cache line back and dirties it, but --=20 2.34.1 From nobody Sat Feb 7 22:07:20 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 355C5657BF for ; Tue, 19 Mar 2024 01:49:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812944; cv=none; b=pzbFSFZ2nhyuDTAT0x5t1UXcv4+UN0CuBEzcbitXKiZqa+hAeioe6GE0n2QrKJ4ZD+scvX+cO4MBnbNVBYAr6vEpC6AEZy3knoyKsuXMqcvJQnmoa0nTG3UhOA+34eboQHm+iXNCDFTMIewXWK30vInz1OHcz61aYtld5DcK7S8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812944; c=relaxed/simple; bh=xaBDCWG/HU1lwB4kGooE2OZgFghxRiDJBIv/7/4jW88=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AjO+augTJlLjP00413rohVu2LAfGlu//hJQ+8Xx3KhHde/9MlJuC7FMx6TDcZxuu7bwJeILEbFUwRbwoKfIZMQFzWTm1ZaLc3Jr6tFo97TxxxCKRctBnI5t9CSxJ8r1HEzkDZLc6p2hZ3u0vVyAFC0WSmYXu7KJEA44zpAStowI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=iQdl4HDZ; arc=none smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="iQdl4HDZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710812942; x=1742348942; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xaBDCWG/HU1lwB4kGooE2OZgFghxRiDJBIv/7/4jW88=; b=iQdl4HDZNunFdZFQhvosmzuewoBV/kFV1+DY+mvo99l1OuTdi8gsvmzF 4DobtzglGiHJA4oNM8ORAkvPxPrmpwnvfplhuRm0PwOBmu7fX10u+L2Qe ORfM3rNt3MuDC6ZqAzpI4Y4uCCwO4xbWeFSAw1OYOmy2s9sxU3spKMDFP ZDlzYhjAR8vAl/C42nq05UDwBlqkFGbg+4tlc3pwMC4g3ExpFVx7s4eFX AGjsjtWjmxJ0nk2fD2FyNy/Ynz8loZH4Nshlf4fUBUbrHEJ6qf8l2pTyX joNwKinqX7gHCzFzMCtJgPH+p8egSaHnvuEfHn5OAs4umf6qdQMnt7vWt w==; X-IronPort-AV: E=McAfee;i="6600,9927,11017"; a="5776892" X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="5776892" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="13697336" Received: from server.sh.intel.com ([10.239.53.117]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:48:58 -0700 From: Kai Huang To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, dave.hansen@intel.com, bp@alien8.de, kirill.shutemov@linux.intel.com, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, luto@kernel.org, peterz@infradead.org, rick.p.edgecombe@intel.com, thomas.lendacky@amd.com, ashish.kalra@amd.com, chao.gao@intel.com, bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com Subject: [PATCH v2 2/5] x86/kexec: do unconditional WBINVD in relocate_kernel() Date: Tue, 19 Mar 2024 01:48:45 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Both SME and TDX can leave caches in incoherent state due to memory encryption. During kexec, the caches must be flushed before jumping to the second kernel to avoid silent memory corruption to the second kernel. During kexec, the WBINVD in stop_this_cpu() flushes caches for all remote cpus when they are being stopped. For SME, the WBINVD in relocate_kernel() flushes the cache for the last running cpu (which is executing the kexec). Similarly, for TDX after stopping all remote cpus with cache flushed, to support kexec, the kernel needs to flush cache for the last running cpu. Make the WBINVD in the relocate_kernel() unconditional to cover both SME and TDX. Signed-off-by: Kai Huang Suggested-by: Borislav Petkov Cc: Tom Lendacky Cc: Dave Young --- arch/x86/include/asm/kexec.h | 3 +-- arch/x86/kernel/machine_kexec_64.c | 3 +-- arch/x86/kernel/relocate_kernel_64.S | 13 +++---------- 3 files changed, 5 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h index 91ca9a9ee3a2..9754794242ad 100644 --- a/arch/x86/include/asm/kexec.h +++ b/arch/x86/include/asm/kexec.h @@ -127,8 +127,7 @@ unsigned long relocate_kernel(unsigned long indirection_page, unsigned long page_list, unsigned long start_address, - unsigned int preserve_context, - unsigned int host_mem_enc_active); + unsigned int preserve_context); #endif =20 #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_k= exec_64.c index bc0a5348b4a6..b9a632479b36 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -357,8 +357,7 @@ void machine_kexec(struct kimage *image) image->start =3D relocate_kernel((unsigned long)image->head, (unsigned long)page_list, image->start, - image->preserve_context, - cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)); + image->preserve_context); =20 #ifdef CONFIG_KEXEC_JUMP if (image->preserve_context) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocat= e_kernel_64.S index 56cab1bb25f5..66b628686dbc 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -50,7 +50,6 @@ SYM_CODE_START_NOALIGN(relocate_kernel) * %rsi page_list * %rdx start address * %rcx preserve_context - * %r8 host_mem_enc_active */ =20 /* Save the CPU context, used for jumping back */ @@ -78,9 +77,6 @@ SYM_CODE_START_NOALIGN(relocate_kernel) pushq $0 popfq =20 - /* Save SME active flag */ - movq %r8, %r12 - /* * get physical address of control page now * this is impossible after page table switch @@ -160,14 +156,11 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) movq %r9, %cr3 =20 /* - * If SME is active, there could be old encrypted cache line - * entries that will conflict with the now unencrypted memory - * used by kexec. Flush the caches before copying the kernel. + * The kernel could leave caches in incoherent state on SME/TDX + * capable platforms. Just do unconditional cache flush to avoid + * silent memory corruption to the new kernel for these platforms. */ - testq %r12, %r12 - jz 1f wbinvd -1: =20 movq %rcx, %r11 call swap_pages --=20 2.34.1 From nobody Sat Feb 7 22:07:20 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45B0765BA0 for ; Tue, 19 Mar 2024 01:49:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812948; cv=none; b=YWUD1vr7mg6Cy+9xMolmQ6qTLrAqrzcPYcl5x6mJl+GFe0LODPYBoSOAfNbMsZvmRRC9ald1v+z5XFoIf4AbZ40F9mUckYEBpYTegxtGi/9Snr9a1xiZvRWtUu1Ns8Zj5pcGVmhkm5vni7av/jXNxd3NmUyGeDbN3XKdQHdB6nw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812948; c=relaxed/simple; bh=hMVS5xLN7SybAyi7eh+e9lC94o21AAJJbCO+iURGvB8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=r7EvmHFhkwJdks3HJvHBQ7oiu0IyNyuPtuR0/uhKZPORV2zofFnkWlfk0GPMinRT6xQwsv0S3FCQ5iQ6uhqHatXkM1In9yLYw2UYClZLy5yldOQNFWvevZ/ghEZWKOjehe7u0A78Vs+0VCZmujwEqq4Crj18A4I2FqMNm1uuRU0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZVisq1T7; arc=none smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZVisq1T7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710812946; x=1742348946; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hMVS5xLN7SybAyi7eh+e9lC94o21AAJJbCO+iURGvB8=; b=ZVisq1T7i0a/wkzmxxR8YMXdScXWHOYec5BxJJf1hqcyh7T0wOPPjhv9 5Le9MMP4kpoUEiyVC450YqqiyPJl1R0IoOg0deN2VUsp1AZLGjMOpOU5l l6IhEcVaSi23gQdxi5DceJh6AIKXaG2KQL4+kQLf4f4o6Semf/hgE88Dl 4kN7egBuO36Ydh+XCCfRzdy2ZKWTDPK52N1bBObfHjpjQ2pWjb36/b5JU CgQSzQmJgo+oYxafwpfX/ZxrpYFLt40gp8HgVoquom7fNdufV/4bIGZlL nUx5SK5s4B56D6hRD0svb3keVAMsyuLOtxb93zLL66LY9W3hCc8PIuuD8 g==; X-IronPort-AV: E=McAfee;i="6600,9927,11017"; a="5776907" X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="5776907" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="13697344" Received: from server.sh.intel.com ([10.239.53.117]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:02 -0700 From: Kai Huang To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, dave.hansen@intel.com, bp@alien8.de, kirill.shutemov@linux.intel.com, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, luto@kernel.org, peterz@infradead.org, rick.p.edgecombe@intel.com, thomas.lendacky@amd.com, ashish.kalra@amd.com, chao.gao@intel.com, bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com Subject: [PATCH v2 3/5] x86/kexec: Reset TDX private memory on platforms with TDX erratum Date: Tue, 19 Mar 2024 01:48:46 +0000 Message-Id: <848cf96640be6ea8132ab173b83f5ea13b88a88e.1710811610.git.kai.huang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" TL;DR: On the platforms with TDX "partial write machine check" erratum, during kexec, convert TDX private memory back to normal before jumping to the second kernel to avoid the second kernel seeing potential unexpected machine check. Long version: The first few generations of TDX hardware have an erratum. A partial write to a TDX private memory cacheline will silently "poison" the line. Subsequent reads will consume the poison and generate a machine check. According to the TDX hardware spec, neither of these things should have happened. =3D=3D Background =3D=3D Virtually all kernel memory accesses operations happen in full cachelines. In practice, writing a "byte" of memory usually reads a 64 byte cacheline of memory, modifies it, then writes the whole line back. Those operations do not trigger this problem. This problem is triggered by "partial" writes where a write transaction of less than cacheline lands at the memory controller. The CPU does these via non-temporal write instructions (like MOVNTI), or through UC/WC memory mappings. The issue can also be triggered away from the CPU by devices doing partial writes via DMA. =3D=3D Problem =3D=3D A fast warm reset doesn't reset TDX private memory. Kexec() can also boot into the new kernel directly. Thus if the old kernel has left any TDX private pages on the platform with this erratum, the new kernel might get unexpected machine check. Note that w/o this erratum any kernel read/write on TDX private memory should never cause machine check, thus it's OK for the old kernel to leave TDX private pages as is. =3D=3D Solution =3D=3D In short, with this erratum, the kernel needs to explicitly convert all TDX private pages back to normal to give the new kernel a clean slate after kexec(). The BIOS is also expected to disable fast warm reset as a workaround to this erratum, thus this implementation doesn't try to reset TDX private memory for the reboot case in the kernel but depends on the BIOS to enable the workaround. Convert TDX private pages back to normal (using MOVDIR64B to clear these pages) after all remote cpus have been stopped and cache flush has been done on all cpus, when no more TDX activity can happen further. Do it in machine_kexec() to cover both normal kexec, and crash kexec. For now TDX private memory can only be PAMT pages. It would be ideal to cover all types of TDX private memory here, but there are practical problems to do so: 1) There's no existing infrastructure to track TDX private pages; 2) It's not feasible to query the TDX module about page type, because VMX, which making SEAMCALL requires, has already been disabled; 3) Even if it is feasible to query the TDX module, the result may not be accurate. E.g., the remote CPU could be stopped right before MOVDIR64B. One temporary solution is to blindly convert all memory pages, but it's problematic to do so too, because not all pages are mapped as writable in the direct mapping. It can be done by switching to the identical mapping created for kexec(), or a new page table, but the complexity looks overkill. Therefore, rather than doing something dramatic, only reset PAMT pages here. Leave resetting other TDX private pages as a future work when they become possible to exist. Signed-off-by: Kai Huang Reviewed-by: Kirill A. Shutemov --- v1 -> v2: - Remove using reboot notifier to stop TDX module as it doesn't cover crash kexec. Change to use a variable with barrier instead. (Rick) - Introduce kexec_save_processor_start() to make code better, and make the comment around calling site of tdx_reset_memory() more concise. (Dave) - Mention cache for all other cpus have been flushed around native_wbinvd() in tdx_reset_memory(). (Dave) - Remove the extended alternaties discussion from the comment, but leave it in the changelog. Point out what does current code do and point out risk. (Dave) --- arch/x86/include/asm/tdx.h | 2 + arch/x86/kernel/machine_kexec_64.c | 27 ++++++++-- arch/x86/virt/vmx/tdx/tdx.c | 79 ++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index eba178996d84..ed3ac9a8a079 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -116,11 +116,13 @@ static inline u64 sc_retry(sc_func_t func, u64 fn, int tdx_cpu_enable(void); int tdx_enable(void); const char *tdx_dump_mce_info(struct mce *m); +void tdx_reset_memory(void); #else static inline void tdx_init(void) { } static inline int tdx_cpu_enable(void) { return -ENODEV; } static inline int tdx_enable(void) { return -ENODEV; } static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; } +static inline void tdx_reset_memory(void) { } #endif /* CONFIG_INTEL_TDX_HOST */ =20 #endif /* !__ASSEMBLY__ */ diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_k= exec_64.c index b9a632479b36..fdc2ffdc7d90 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -28,6 +28,7 @@ #include #include #include +#include =20 #ifdef CONFIG_ACPI /* @@ -288,6 +289,14 @@ void machine_kexec_cleanup(struct kimage *image) free_transition_pgtable(image); } =20 +static void kexec_save_processor_start(struct kimage *image) +{ +#ifdef CONFIG_KEXEC_JUMP + if (image->preserve_context) + save_processor_state(); +#endif +} + /* * Do not allocate memory (or fail in any way) in machine_kexec(). * We are past the point of no return, committed to rebooting now. @@ -298,10 +307,20 @@ void machine_kexec(struct kimage *image) void *control_page; int save_ftrace_enabled; =20 -#ifdef CONFIG_KEXEC_JUMP - if (image->preserve_context) - save_processor_state(); -#endif + kexec_save_processor_start(image); + + /* + * Convert TDX private memory back to normal (when needed) to + * avoid the second kernel potentially seeing unexpected machine + * check. + * + * However skip this when preserve_context is on. By reaching + * here, TDX (if ever got enabled by the kernel) has survived + * from the suspend when preserve_context is on, and it can + * continue to work after jumping back from the second kernel. + */ + if (!image->preserve_context) + tdx_reset_memory(); =20 save_ftrace_enabled =3D __ftrace_enabled_save(); =20 diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 4d6826a76f78..1443d486858a 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -53,6 +53,8 @@ static DEFINE_MUTEX(tdx_module_lock); /* All TDX-usable memory regions. Protected by mem_hotplug_lock. */ static LIST_HEAD(tdx_memlist); =20 +static bool tdx_may_have_private_memory __read_mostly; + typedef void (*sc_err_func_t)(u64 fn, u64 err, struct tdx_module_args *arg= s); =20 static inline void seamcall_err(u64 fn, u64 err, struct tdx_module_args *a= rgs) @@ -1097,6 +1099,18 @@ static int init_tdmrs(struct tdmr_info_list *tdmr_li= st) return 0; } =20 +static void mark_may_have_private_memory(bool may) +{ + tdx_may_have_private_memory =3D may; + + /* + * Ensure update to tdx_may_have_private_memory is visible to all + * cpus. This ensures when any remote cpu reads it as true, the + * 'tdx_tdmr_list' must be stable for reading PAMTs. + */ + smp_wmb(); +} + static int init_tdx_module(void) { struct tdx_tdmr_sysinfo tdmr_sysinfo; @@ -1142,6 +1156,12 @@ static int init_tdx_module(void) if (ret) goto err_reset_pamts; =20 + /* + * Starting from this point the system is possible to have + * TDX private memory. + */ + mark_may_have_private_memory(true); + /* Initialize TDMRs to complete the TDX module initialization */ ret =3D init_tdmrs(&tdx_tdmr_list); if (ret) @@ -1173,6 +1193,7 @@ static int init_tdx_module(void) * as suggested by the TDX spec. */ tdmrs_reset_pamt_all(&tdx_tdmr_list); + mark_may_have_private_memory(false); err_free_pamts: tdmrs_free_pamt_all(&tdx_tdmr_list); err_free_tdmrs: @@ -1490,3 +1511,61 @@ void __init tdx_init(void) =20 check_tdx_erratum(); } + +void tdx_reset_memory(void) +{ + if (!boot_cpu_has(X86_FEATURE_TDX_HOST_PLATFORM)) + return; + + /* + * Converting TDX private pages back to normal must be done + * when there's no TDX activity anymore on all remote cpus. + * Verify this is only called when all remote cpus have + * been stopped. + */ + WARN_ON_ONCE(num_online_cpus() !=3D 1); + + /* + * Kernel read/write to TDX private memory doesn't cause + * machine check on hardware w/o this erratum. + */ + if (!boot_cpu_has_bug(X86_BUG_TDX_PW_MCE)) + return; + + /* + * Nothing to convert if it's not possible to have any TDX + * private pages. + */ + if (!tdx_may_have_private_memory) + return; + + /* + * Ensure the 'tdx_tdmr_list' is stable for reading PAMTs + * when tdx_may_have_private_memory reads true, paired with + * the smp_wmb() in mark_may_have_private_memory(). + */ + smp_rmb(); + + /* + * All remote cpus have been stopped, and their caches have + * been flushed in stop_this_cpu(). Now flush cache for the + * last running cpu _before_ converting TDX private pages. + */ + native_wbinvd(); + + /* + * It's ideal to cover all types of TDX private pages here, but + * currently there's no unified way to tell whether a given page + * is TDX private page or not. + * + * Just convert PAMT pages now, as currently TDX private pages + * can only be PAMT pages. + * + * TODO: + * + * This leaves all other types of TDX private pages undealt + * with. They must be handled in _some_ way when they become + * possible to exist. + */ + tdmrs_reset_pamt_all(&tdx_tdmr_list); +} --=20 2.34.1 From nobody Sat Feb 7 22:07:20 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EBCE65BAE for ; Tue, 19 Mar 2024 01:49:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812951; cv=none; b=J1ApnT5Dbzmnk+wfNx7GaDEcDDdvhrgxpSME2dwRmhhMEWw945PX8x7bPQO3fhOTc2LOFuwXSaLviPYGP5sP/pYgiyboFEfbjvlbCMfvyqAXivtbD8rCRcypfqckvlao/XR7w3HCap3cLrp8ODtqOWL7BBxssquzbHoi5v7HrOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812951; c=relaxed/simple; bh=MNoeTtcQYX9pC2/kmVoMbgA9lvpq5nltcpL+l2VqpG4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tvcByHMqsz+qzNGRiTCKCJf6pyg0HXCiH5otTmNn6W6MCsUVT0dqtHQU+1DAmtz2hyu0pT1MgY2y1iELNUzzJXRj2m0msbhDS+saFGLf0B7iexhJ/8KJukLc4cVlS4As5Db9/hHfctvfryxJY4pEe77yrC7VP9PkfyfgEP/J65Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=WfDqQdyd; arc=none smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="WfDqQdyd" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710812950; x=1742348950; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MNoeTtcQYX9pC2/kmVoMbgA9lvpq5nltcpL+l2VqpG4=; b=WfDqQdydJKvVyToz/Lag8WIWrNU4jHjQ1mffOImj7NicRdYA3ze5kPOQ TSKYY5ZdtKZ9hoN+TqHkNU5tD3mSIfKgwWmzxdAIG/lvRAjX62O/DWJjn 4izAApUWCrFK+FSwQW8x9rlJgj9CnwO6dU+km1idgwG1em9/uh2sb2U+R 94Rn029WKO7oVnRzXLQ+NxuhLJIbJF1rqopDZ/vxqfrcNEYM7acb29T/l o2FdHl/C+b1h2oedo9N4FNAsnwdCcWiHqPW7kOzBC64SNriMBUeG2VsyT KzGwUqIcyqb1hi2ZGAajl3+p7jCCtXFgXdtQunsD3Nymaou015xVyjtrR g==; X-IronPort-AV: E=McAfee;i="6600,9927,11017"; a="5776918" X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="5776918" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="13697354" Received: from server.sh.intel.com ([10.239.53.117]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:06 -0700 From: Kai Huang To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, dave.hansen@intel.com, bp@alien8.de, kirill.shutemov@linux.intel.com, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, luto@kernel.org, peterz@infradead.org, rick.p.edgecombe@intel.com, thomas.lendacky@amd.com, ashish.kalra@amd.com, chao.gao@intel.com, bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com Subject: [PATCH v2 4/5] x86/virt/tdx: Remove the !KEXEC_CORE dependency Date: Tue, 19 Mar 2024 01:48:47 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now TDX host can work with kexec(). Remove the !KEXEC_CORE dependency. Signed-off-by: Kai Huang --- arch/x86/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index e88f6f7b6b41..89b65e2e1115 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1970,7 +1970,6 @@ config INTEL_TDX_HOST depends on X86_X2APIC select ARCH_KEEP_MEMBLOCK depends on CONTIG_ALLOC - depends on !KEXEC_CORE depends on X86_MCE help Intel Trust Domain Extensions (TDX) protects guest VMs from malicious --=20 2.34.1 From nobody Sat Feb 7 22:07:20 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C92665E03 for ; Tue, 19 Mar 2024 01:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812956; cv=none; b=jkQPP2GDBx6yBhbOEHiCnmZK/p7jpibPoYTcCYBNjTeZ9X7XN0crAP1DPeGExw3mrPZd4dg8AQTl+6zZuqE0LnAIkNElqFoWvqWLIOEa871Z1s0mIeQIY4UbfFxiU8e9uWArO4FRFgXZTeyI9wu7xgRRBHjv9LoPBlGVL6/Mty8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710812956; c=relaxed/simple; bh=XhEDGjpkkBWdU1+76Autwe4cmMRpk96DQ5negFxNrS8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JY2ILKvNJ0YexTVpN+N1XDYEltc2MyQ1s6VgnKocvrD/jL9MXO77GYSC11I0vcPlXMqJ3tcGY9fajD44isVf+BHa0GlxaUEXqsjrCPSp2CIwieXX3LNmjrooQekgdxnmehYqW09yFZXASamXdQPryA9SWgcy3TsKkNNKfcu0CSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=THuYTleB; arc=none smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="THuYTleB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710812954; x=1742348954; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XhEDGjpkkBWdU1+76Autwe4cmMRpk96DQ5negFxNrS8=; b=THuYTleBd/ZOwOJS+Vg2vD7S4edSmn6O+1/5Pll3aHpL0UMc/hzYCtBk UBeVYyLhQpA1qAD1v2DmPevU9r+fqV1BP8pzEIUSSNRWLWzcaxH4+4DpQ XEa0WP4JDT2XW7QLOmToOb+33Sngbrva9Qpi7sYoS0xLK9mFmrEyw93JY fdj5AThvN1679qaGLI3EYEwl36uIxWxxqJAhGPS31zzNmbVTVJsXYMQwb 6ZRORVyjRyeptjNwP32dQJ/UdPp7OZZOjRu2+2WIYfKVSfu+EBw4ptJXl 85h0MleXyGmRpsdjS88tudNF/KZT72vcYeNsn3q/pY0MOUSnfUFOt3umt g==; X-IronPort-AV: E=McAfee;i="6600,9927,11017"; a="5776932" X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="5776932" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,135,1708416000"; d="scan'208";a="13697367" Received: from server.sh.intel.com ([10.239.53.117]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Mar 2024 18:49:10 -0700 From: Kai Huang To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, dave.hansen@intel.com, bp@alien8.de, kirill.shutemov@linux.intel.com, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, luto@kernel.org, peterz@infradead.org, rick.p.edgecombe@intel.com, thomas.lendacky@amd.com, ashish.kalra@amd.com, chao.gao@intel.com, bhe@redhat.com, nik.borisov@suse.com, pbonzini@redhat.com, seanjc@google.com Subject: [PATCH v2 5/5] x86/virt/tdx: Add TDX memory reset notifier to reset other private pages Date: Tue, 19 Mar 2024 01:48:48 +0000 Message-Id: <7ba5b283e6406790a929f61645445bf6997bd334.1710811610.git.kai.huang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" TL;DR: To cover both normal kexec and crash kexec, add a TDX specific memory reset notifier to let "in-kernel TDX users" use their own way to convert TDX private pages (that they manage respectively) in tdx_reset_memory(). Long version: On the platforms with TDX "partial write machine check" erratum, during kexec, the kernel needs to convert TDX private memory back to normal before jumping to the second kernel to avoid the second kernel seeing potential machine check. For now tdx_reset_memory() only resets PAMT pages. KVM will be the first in-kernel TDX user to support running TDX guests, and by then other TDX private pages will start to exist. They need to be covered too. Currently the kernel doesn't have a unified way to tell whether a given page is TDX private page or not. One choice is to add such unified way, and there are couple of options to do it: 1) Use a bitmap, or Xarray, etc to track TDX private page for all PFNs; 2) Use a "software-only" bit in the direct-mapping PTE to mark a given page is TDX private page; 3) Use a new flag in 'struct page' to mark TDX private page; 4) ... potential other ways. Option 1) consumes additional memory. E.g., if using bitmap, the overhead is "number of total RAM pages / 8" bytes. Option 2) would cause splitting large-page mapping to 4K mapping in the direct mapping when one page is allocated as TDX private page, and cause additional TLB flush etc. It's not ideal for such use case. Option 3) apparently contradicts to the effort to reduce the use of the flags of 'struct page'. None of above is ideal. Therefore, instead of providing a unified way to tell whether a given page is TDX private page or not, leave "resetting TDX private pages" to the "in-kernel user" of TDX. This is motivated by the fact that KVM is already maintaining an Xarray to track "memory attributes (e.g., private or shared)" for each GFN for each guest. Thus KVM can use its own way to find all TDX private pages that it manages and convert them back to normal. For the normal kexec the reboot notifier could be used, but it doesn't cover the cash kexec. Add a TDX specific memory reset notifier to achieve this. The in-kernel TDX users will need to register their own notifiers to reset TDX private pages. Call these notifiers in tdx_reset_memory() right before resetting PAMT pages. KVM will be the first user of this notifier. Export the "register" and "unregister" APIs for KVM to use. Signed-off-by: Kai Huang --- arch/x86/include/asm/tdx.h | 14 ++++++++++++ arch/x86/virt/vmx/tdx/tdx.c | 45 +++++++++++++++++++++++++++---------- 2 files changed, 47 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index ed3ac9a8a079..7c2c0a0b9754 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -117,12 +117,26 @@ int tdx_cpu_enable(void); int tdx_enable(void); const char *tdx_dump_mce_info(struct mce *m); void tdx_reset_memory(void); + +struct notifier_block; + +int tdx_register_memory_reset_notifier(struct notifier_block *nb); +void tdx_unregister_memory_reset_notifier(struct notifier_block *nb); #else static inline void tdx_init(void) { } static inline int tdx_cpu_enable(void) { return -ENODEV; } static inline int tdx_enable(void) { return -ENODEV; } static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; } static inline void tdx_reset_memory(void) { } + +struct notifier_block; + +static inline int tdx_register_memory_reset_notifier(struct notifier_block= *nb) +{ + return -EOPNOTSUPP; +} +static inline void tdx_unregister_memory_reset_notifier( + struct notifier_block *nb) { } #endif /* CONFIG_INTEL_TDX_HOST */ =20 #endif /* !__ASSEMBLY__ */ diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 1443d486858a..a3daea33c59f 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -55,6 +56,8 @@ static LIST_HEAD(tdx_memlist); =20 static bool tdx_may_have_private_memory __read_mostly; =20 +static BLOCKING_NOTIFIER_HEAD(tdx_memory_reset_chain); + typedef void (*sc_err_func_t)(u64 fn, u64 err, struct tdx_module_args *arg= s); =20 static inline void seamcall_err(u64 fn, u64 err, struct tdx_module_args *a= rgs) @@ -1512,6 +1515,27 @@ void __init tdx_init(void) check_tdx_erratum(); } =20 +int tdx_register_memory_reset_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&tdx_memory_reset_chain, nb); +} +EXPORT_SYMBOL_GPL(tdx_register_memory_reset_notifier); + +void tdx_unregister_memory_reset_notifier(struct notifier_block *nb) +{ + blocking_notifier_chain_unregister(&tdx_memory_reset_chain, nb); +} +EXPORT_SYMBOL_GPL(tdx_unregister_memory_reset_notifier); + +static int notify_reset_memory(void) +{ + int ret; + + ret =3D blocking_notifier_call_chain(&tdx_memory_reset_chain, 0, NULL); + + return notifier_to_errno(ret); +} + void tdx_reset_memory(void) { if (!boot_cpu_has(X86_FEATURE_TDX_HOST_PLATFORM)) @@ -1554,18 +1578,15 @@ void tdx_reset_memory(void) native_wbinvd(); =20 /* - * It's ideal to cover all types of TDX private pages here, but - * currently there's no unified way to tell whether a given page - * is TDX private page or not. - * - * Just convert PAMT pages now, as currently TDX private pages - * can only be PAMT pages. - * - * TODO: - * - * This leaves all other types of TDX private pages undealt - * with. They must be handled in _some_ way when they become - * possible to exist. + * Tell all in-kernel TDX users to reset TDX private pages + * that they manage. + */ + if (notify_reset_memory()) + pr_err("Failed to reset all TDX private pages.\n"); + + /* + * The only remaining TDX private pages are PAMT pages. + * Reset them. */ tdmrs_reset_pamt_all(&tdx_tdmr_list); } --=20 2.34.1