From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A75B2EC096 for ; Mon, 1 Sep 2025 16:09:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742985; cv=none; b=dHzcsozc0uZmbVXW3gPlwl/1dyB+lUV8qtLtfOwH/sjJd5c13Tqx0Y2b5amnUV8HmkosfcUTMwrZU0ZYtJpmqpYmhbyZ/eyQweDqm/gb8On0MSbRMZrZr6cMfezsKE4I2RqGBItEPnc6tKRuVzPGXP7o9OLMPwj2eaOGyEIozT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742985; c=relaxed/simple; bh=8o9MSeGyjka2YIkOKa8nP2VBMBZdX1wxLIA+kQOxqmY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YbtDnaNG/KotDcMEPAu1Bvc3GnrgPNMHNrvDPeb29jKNArzd+giwdts6bJkmkiWYKFkRQXk5xvX+U4+cIFLhVDM39LBSqiTejtgeZ0Qtv7JNCKXm51Zi+QGJCoLazRLk75cL/p/qVdxQ6XJHGwcI0ENGDRAFWdHuqZVyU0Ld85c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=UgcAyHFb; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="UgcAyHFb" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756742982; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/ILG8lOYeP81n1upiKDtnLYC19g1Qye1JLya30e5LpQ=; b=UgcAyHFbt5ommCarxCXIgasjfhj5L8Fk+B58RbA/w11yKM4GlfRbXNtLZgEg2PNO6XX9J1 aGLTQq7QHuPaDI1bw344DATofJKC4XijvQlZavlHvw1Fa5HNnIgXJwrqTlakP3+1vWt02i Ftyrkp4NyxEp8nzhtQQWZlxy67hOqQk= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-683-oKOHndDoP36ewngBwgvvKw-1; Mon, 01 Sep 2025 12:09:41 -0400 X-MC-Unique: oKOHndDoP36ewngBwgvvKw-1 X-Mimecast-MFC-AGG-ID: oKOHndDoP36ewngBwgvvKw_1756742981 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4b31825af12so52656421cf.0 for ; Mon, 01 Sep 2025 09:09:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756742980; x=1757347780; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/ILG8lOYeP81n1upiKDtnLYC19g1Qye1JLya30e5LpQ=; b=uFnhllDZS3IUN5SKde4C/XXkLAzPQrQA2eldR8WZd84Y0+tf2m5vKmxPBzkkjJ6RSo C4uUM406IDqBA2LBIeKUIJ+Y1+CjO/WT54A96ITPzVBtDBFdleVS7HiqFLpzOhqkKQI+ 6mJRhgYco2MlbOtXm1t41y/jwcqkiFl19tYrTdN4iumcml+kzGTwHSFvpl3jYfjUn6Bs dhL94GsCaR+3YUipkORpVDchU2ERRzzJTWSPwIP7+jD2q5mG7y5XIqJs3v21awbpHiK9 HPAyc3TDTg9q/XEtXm+0y+M/FC+rlwp4ip/uEd+gAEXJqLnCEs9dLLSB4DB9xQj7s+mm QJgA== X-Gm-Message-State: AOJu0YwX41Vt2PYg2V4rsXA7sMqRPNLV84JkYOyqpVMLB4/DS9upwwGv 5KOhsftZ2rhXWzePOYYlfI3EO2qf4XsEw5CrlMbydVxXa5RPrr12RT/OtiCy43MkpcDiyU4l6EH s092Ze+wdgUCToMaIxkJJ0ZYsMTiyEat1JKaTC3FtI+JkDco3HZkxuP4gbDEox60aJFjOggP4XM uLsp3mxRS2dtEFsHDbeEPAPm6ibh1+kjhZO/iBNT6d5ma2crH/nA== X-Gm-Gg: ASbGncsRhOuXQWcjJY7PZKv0rNsREZWI8MUfPU3RLNWe5zblCnENJLq75Vv7LpgbLcd KsRfwcoBXan4vrp2z/uXBZPxlgB4ANYRgAPekZZxDtfj/RQtfTzlfYdsP8K34LJYzZegTnXRbp9 Z6IjttImzOKALfAlFVT3oK32QpusUIl6P4iCqRzpfUeuF+t2tE+EgPJORoxhezXkF5cqfvFjc5g G2BcVvqnAmUoEmVTAvlrXN7aq8kO7A8QzN8qMQvJxzkq8IpXDSFHNq5Y038pl4ARJ+NClREt9GU apNhgHSxz6kxGPpIwDOYc7rTy4IUeGgKAvRm6XsMUf7FXJmRUK3mUtg7gc2U/qerDx/KmNX6klH +CgyAqCMplU00CwVnNBi3qkvsXuD/3zl6topswJ48VDxDmVQvn7N+i4LYw7S5MN4UZuah X-Received: by 2002:a05:622a:138b:b0:4b0:6c7c:a955 with SMTP id d75a77b69052e-4b31da15393mr88333101cf.35.1756742980259; Mon, 01 Sep 2025 09:09:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDti1nmXO9Q+sLEIfixu7lgNBZ2vr81Q1E5oR04YXTrIW3a1NX5lesAlwj4ITitS6AR6+hhQ== X-Received: by 2002:a05:622a:138b:b0:4b0:6c7c:a955 with SMTP id d75a77b69052e-4b31da15393mr88332171cf.35.1756742979465; Mon, 01 Sep 2025 09:09:39 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7fc14849559sm686458885a.41.2025.09.01.09.09.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:09:38 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com Subject: [PATCH 1/7] x86/kexec: Consolidate relocate_kernel() function parameters Date: Mon, 1 Sep 2025 18:09:24 +0200 Message-ID: <20250901160930.1785244-2-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang During kexec, the kernel jumps to the new kernel in relocate_kernel(), which is implemented in assembly and both 32-bit and 64-bit have their own version. Currently, for both 32-bit and 64-bit, the last two parameters of the relocate_kernel() are both 'unsigned int' but actually they only convey a boolean, i.e., one bit information. The 'unsigned int' has enough space to carry two bits information therefore there's no need to pass the two booleans in two separate 'unsigned int'. Consolidate the last two function parameters of relocate_kernel() into a single 'unsigned int' and pass flags instead. Only consolidate the 64-bit version albeit the similar optimization can be done for the 32-bit version too. Don't bother changing the 32-bit version while it is working (since assembly code change is required). Signed-off-by: Kai Huang Reviewed-by: Tom Lendacky Reviewed-by: Borislav Petkov (AMD) Reviewed-by: David Woodhouse Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/kexec.h | 12 ++++++++++-- arch/x86/kernel/machine_kexec_64.c | 22 +++++++++++++--------- arch/x86/kernel/relocate_kernel_64.S | 25 +++++++++++++++---------- 3 files changed, 38 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h index f2ad77929d6e..12cebbcdb6c8 100644 --- a/arch/x86/include/asm/kexec.h +++ b/arch/x86/include/asm/kexec.h @@ -13,6 +13,15 @@ # define KEXEC_DEBUG_EXC_HANDLER_SIZE 6 /* PUSHI, PUSHI, 2-byte JMP */ #endif =20 +#ifdef CONFIG_X86_64 + +#include + +#define RELOC_KERNEL_PRESERVE_CONTEXT BIT(0) +#define RELOC_KERNEL_HOST_MEM_ENC_ACTIVE BIT(1) + +#endif + # define KEXEC_CONTROL_PAGE_SIZE 4096 # define KEXEC_CONTROL_CODE_MAX_SIZE 2048 =20 @@ -121,8 +130,7 @@ typedef unsigned long relocate_kernel_fn(unsigned long indirection_page, unsigned long pa_control_page, unsigned long start_address, - unsigned int preserve_context, - unsigned int host_mem_enc_active); + unsigned int flags); #endif extern relocate_kernel_fn relocate_kernel; #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_k= exec_64.c index 8593760c255a..fdd04b5bb70e 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -384,16 +384,10 @@ void __nocfi machine_kexec(struct kimage *image) { unsigned long reloc_start =3D (unsigned long)__relocate_kernel_start; relocate_kernel_fn *relocate_kernel_ptr; - unsigned int host_mem_enc_active; + unsigned int relocate_kernel_flags; int save_ftrace_enabled; void *control_page; =20 - /* - * This must be done before load_segments() since if call depth tracking - * is used then GS must be valid to make any function calls. - */ - host_mem_enc_active =3D cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT); - #ifdef CONFIG_KEXEC_JUMP if (image->preserve_context) save_processor_state(); @@ -427,6 +421,17 @@ void __nocfi machine_kexec(struct kimage *image) */ relocate_kernel_ptr =3D control_page + (unsigned long)relocate_kernel - r= eloc_start; =20 + relocate_kernel_flags =3D 0; + if (image->preserve_context) + relocate_kernel_flags |=3D RELOC_KERNEL_PRESERVE_CONTEXT; + + /* + * This must be done before load_segments() since if call depth tracking + * is used then GS must be valid to make any function calls. + */ + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) + relocate_kernel_flags |=3D RELOC_KERNEL_HOST_MEM_ENC_ACTIVE; + /* * The segment registers are funny things, they have both a * visible and an invisible part. Whenever the visible part is @@ -443,8 +448,7 @@ void __nocfi machine_kexec(struct kimage *image) image->start =3D relocate_kernel_ptr((unsigned long)image->head, virt_to_phys(control_page), image->start, - image->preserve_context, - host_mem_enc_active); + relocate_kernel_flags); =20 #ifdef CONFIG_KEXEC_JUMP if (image->preserve_context) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocat= e_kernel_64.S index ea604f4d0b52..26e945f85d19 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -66,8 +66,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel) * %rdi indirection_page * %rsi pa_control_page * %rdx start address - * %rcx preserve_context - * %r8 host_mem_enc_active + * %rcx flags: RELOC_KERNEL_* */ =20 /* Save the CPU context, used for jumping back */ @@ -111,7 +110,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel) /* save indirection list for jumping back */ movq %rdi, pa_backup_pages_map(%rip) =20 - /* Save the preserve_context to %r11 as swap_pages clobbers %rcx. */ + /* Save the flags to %r11 as swap_pages clobbers %rcx. */ movq %rcx, %r11 =20 /* setup a new stack at the end of the physical control page */ @@ -129,9 +128,8 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) /* * %rdi indirection page * %rdx start address - * %r8 host_mem_enc_active * %r9 page table page - * %r11 preserve_context + * %r11 flags: RELOC_KERNEL_* * %r13 original CR4 when relocate_kernel() was invoked */ =20 @@ -204,7 +202,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) * entries that will conflict with the now unencrypted memory * used by kexec. Flush the caches before copying the kernel. */ - testq %r8, %r8 + testb $RELOC_KERNEL_HOST_MEM_ENC_ACTIVE, %r11b jz .Lsme_off wbinvd .Lsme_off: @@ -220,7 +218,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) movq %cr3, %rax movq %rax, %cr3 =20 - testq %r11, %r11 /* preserve_context */ + testb $RELOC_KERNEL_PRESERVE_CONTEXT, %r11b jnz .Lrelocate =20 /* @@ -273,7 +271,13 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) ANNOTATE_NOENDBR andq $PAGE_MASK, %r8 lea PAGE_SIZE(%r8), %rsp - movl $1, %r11d /* Ensure preserve_context flag is set */ + /* + * Ensure RELOC_KERNEL_PRESERVE_CONTEXT flag is set so that + * swap_pages() can swap pages correctly. Note all other + * RELOC_KERNEL_* flags passed to relocate_kernel() are not + * restored. + */ + movl $RELOC_KERNEL_PRESERVE_CONTEXT, %r11d call swap_pages movq kexec_va_control_page(%rip), %rax 0: addq $virtual_mapped - 0b, %rax @@ -321,7 +325,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) UNWIND_HINT_END_OF_STACK /* * %rdi indirection page - * %r11 preserve_context + * %r11 flags: RELOC_KERNEL_* */ movq %rdi, %rcx /* Put the indirection_page in %rcx */ xorl %edi, %edi @@ -357,7 +361,8 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) movq %rdi, %rdx /* Save destination page to %rdx */ movq %rsi, %rax /* Save source page to %rax */ =20 - testq %r11, %r11 /* Only actually swap for ::preserve_context */ + /* Only actually swap for ::preserve_context */ + testb $RELOC_KERNEL_PRESERVE_CONTEXT, %r11b jz .Lnoswap =20 /* copy source page to swap page */ --=20 2.51.0 From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2C4F2EC0B6 for ; Mon, 1 Sep 2025 16:09:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742991; cv=none; b=URc35snlK7uOWB3AWA6/tOkOAcfKGuAygcFIj8+v6XJDGOlaIK4mwTyT6y7iOLeslK8vHHPC5cFG9ltLErCOYIVkw1FfuKS6oTNjQFi4JI/q/OruF9PM82fBLsCvwRGGlz4dQxqODmch3wy3otqYwRKM6mv2ukvUuUbzIVE60yA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742991; c=relaxed/simple; bh=jpKHYGmCtueZ8D8VBUsMZT8ONSweBcCS83ya6hNTJpk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V4b5VqS7OpG+G9x0bKH1ufe1pSvFTcX+ga30rKlrmJC+JNoNiYdnZHNkfDGtJNXcnLJxE59+P5lMjLsoiWhf193vNDUqL9q81U3gLXk7lPaxX0+umSVNe4KV2bGTVCJ7O8MvLGMQ8trRLx5DtNpcQFp2+SfLvqBZjZIzXfjjves= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=eUXzeH4P; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="eUXzeH4P" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756742987; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U3TbFIMcvGAQ1Z4BqKhz2/agDzUx20cT3mtJIvL86Nc=; b=eUXzeH4PXEeQSwBFXzVkpiIF9tdPVVaQbs9fLJmBXv8D6HIsn9fXMAY/ATgz2ec09G89oJ mmd45LtZGzTQI4Y8C8fpR4q5W35GIjylMKFVbnjMPum86AyMDKH0i0Jb3VuowWioTZa+ZP 0KNCfw/wx/NQ0LcJD70EotPsUNwhhfw= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-551-mv_iNKzqMp68zJvy0sDPCg-1; Mon, 01 Sep 2025 12:09:46 -0400 X-MC-Unique: mv_iNKzqMp68zJvy0sDPCg-1 X-Mimecast-MFC-AGG-ID: mv_iNKzqMp68zJvy0sDPCg_1756742986 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7f2942a1aa1so1278244785a.0 for ; Mon, 01 Sep 2025 09:09:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756742985; x=1757347785; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U3TbFIMcvGAQ1Z4BqKhz2/agDzUx20cT3mtJIvL86Nc=; b=rgCKvevkCBDZeqodWvE7avC8oVANkwjyNMjAO87ySLgsfCHGJM7aaOrAktXr/DlHGP MIcMVxsQncPTWk9JRr28YMuafWfvgndiQy9boUkXIewxqr50WyalmFPrBk+GvCAFOPKb TWgZ8mO0QcOwSPukE1iQzDSTyswJ1pLF6uwB8dJhVkHsReSLP4WjLIxvv+fPI5Ror8z3 rIc7dkoc0VpqABVIU9/l0pnQ+rfbttXtkdOT2ze0uO+MgSf1vkf95NuMsAzgIXAEZsDP DimLBZU5XUW4UKfG7RfN2a6/9LJf9M47UL+yLhGkpWbIGxZhZgDGGYDIyE6d3bZLvCJR o7fg== X-Gm-Message-State: AOJu0YyGNcwdjXcRdnUvwZd8Eg4aDYzv4479oHgxxyy5D608KOfbyu6q pb7VGtsWcfvjiWx6rjrKILm5gtC0JuOeKKQAACauC2oCPw/O2biDqDFEU/P1t7aNR0E5tT0jQv4 kMipLs0S1D1UoIvOvjHAOaeUhyI6I4Z2I/tFeJLQIR5NRMIEjGPqxK9fA3Kv6FgcFkUpZ8gekaT eYAQmDXtKld5EZ9010Co1cbd0wxeMM+QWLinIvQ0UiyF4hVAIOQQ== X-Gm-Gg: ASbGnctGf0YzXwHaSSGhZbukBOzEbut/ANcskr9wqLRMmRvP3W5TCZ9n41MdpJ5h/lI y5pBXlXerA+DBVrzymVI03rpWttCvfyfwGaTLuVcA/Gb3Yj8HeLeF/4dFxqEdR77bp0VHzCu5bi YyVBtZydEjqilQnfs6rAvM6KuSrQ8+Ycc8FY01gd6WSKxAveDvmyipHj7JxsyJxRhfqVu39zNsm ZzSjZaeg/7WEHOCX8rPO8DnKuZndiQIY4ou8c1T9V/IO9PBR8myszIw+4VxXpyM3ldSaLhr8D4p DIdEgxtB0K7uh80d8LsUWa0QC2Ofx5WvIl0sJL+HjSUdTpkbyijlUArINfTQ/CT/r8HpBnTo3kR AsZti49xyK69wggpfkoypymYyaZDPwsiUOz95vVZ8hchdoQ5ZPnVFiXjNjsu5SKIcBnw3 X-Received: by 2002:a05:620a:45a3:b0:7fb:e684:bc0d with SMTP id af79cd13be357-7fed772f970mr655313285a.38.1756742984976; Mon, 01 Sep 2025 09:09:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH4rUO01RPiRDYjEO824d/2PmRfL+5SSdY+BjxsONhN7FDIfaGb+sktB53si5IfTFykSqNyLw== X-Received: by 2002:a05:620a:45a3:b0:7fb:e684:bc0d with SMTP id af79cd13be357-7fed772f970mr655305985a.38.1756742984156; Mon, 01 Sep 2025 09:09:44 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4b30b67f435sm64472141cf.29.2025.09.01.09.09.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:09:43 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com Subject: [PATCH 2/7] x86/sme: Use percpu boolean to control WBINVD during kexec Date: Mon, 1 Sep 2025 18:09:25 +0200 Message-ID: <20250901160930.1785244-3-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang TL;DR: Prepare to unify how TDX and SME do cache flushing during kexec by making a percpu boolean control whether to do the WBINVD. -- Background -- On SME platforms, dirty cacheline aliases with and without encryption bit can coexist, and the CPU can flush them back to memory in random order. During kexec, the caches must be flushed before jumping to the new kernel otherwise the dirty cachelines could silently corrupt the memory used by the new kernel due to different encryption property. TDX also needs a cache flush during kexec for the same reason. It would be good to have a generic way to flush the cache instead of scattering checks for each feature all around. When SME is enabled, the kernel basically encrypts all memory including the kernel itself and a simple memory write from the kernel could dirty cachelines. Currently, the kernel uses WBINVD to flush the cache for SME during kexec in two places: 1) the one in stop_this_cpu() for all remote CPUs when the kexec-ing CPU stops them; 2) the one in the relocate_kernel() where the kexec-ing CPU jumps to the new kernel. -- Solution -- Unlike SME, TDX can only dirty cachelines when it is used (i.e., when SEAMCALLs are performed). Since there are no more SEAMCALLs after the aforementioned WBINVDs, leverage this for TDX. To unify the approach for SME and TDX, use a percpu boolean to indicate the cache may be in an incoherent state and needs flushing during kexec, and set the boolean for SME. TDX can then leverage it. While SME could use a global flag (since it's enabled at early boot and enabled on all CPUs), the percpu flag fits TDX better: The percpu flag can be set when a CPU makes a SEAMCALL, and cleared when another WBINVD on the CPU obviates the need for a kexec-time WBINVD. Saving kexec-time WBINVD is valuable, because there is an existing race[*] where kexec could proceed while another CPU is active. WBINVD could make this race worse, so it's worth skipping it when possible. -- Side effect to SME -- Today the first WBINVD in the stop_this_cpu() is performed when SME is *supported* by the platform, and the second WBINVD is done in relocate_kernel() when SME is *activated* by the kernel. Make things simple by changing to do the second WBINVD when the platform supports SME. This allows the kernel to simply turn on this percpu boolean when bringing up a CPU by checking whether the platform supports SME. No other functional change intended. [*] The aforementioned race: During kexec native_stop_other_cpus() is called to stop all remote CPUs before jumping to the new kernel. native_stop_other_cpus() firstly sends normal REBOOT vector IPIs to stop remote CPUs and waits them to stop. If that times out, it sends NMI to stop the CPUs that are still alive. The race happens when native_stop_other_cpus() has to send NMIs and could potentially result in the system hang (for more information please see [1]). Link: https://lore.kernel.org/kvm/b963fcd60abe26c7ec5dc20b42f1a2ebbcc72397.= 1750934177.git.kai.huang@intel.com/ [1] Signed-off-by: Kai Huang Reviewed-by: Tom Lendacky Tested-by: Tom Lendacky Reviewed-by: Borislav Petkov (AMD) Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/kexec.h | 4 ++-- arch/x86/include/asm/processor.h | 2 ++ arch/x86/kernel/cpu/amd.c | 17 +++++++++++++++++ arch/x86/kernel/machine_kexec_64.c | 14 ++++++++++---- arch/x86/kernel/process.c | 24 +++++++++++------------- arch/x86/kernel/relocate_kernel_64.S | 13 ++++++++++--- 6 files changed, 52 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h index 12cebbcdb6c8..5cfb27f26583 100644 --- a/arch/x86/include/asm/kexec.h +++ b/arch/x86/include/asm/kexec.h @@ -17,8 +17,8 @@ =20 #include =20 -#define RELOC_KERNEL_PRESERVE_CONTEXT BIT(0) -#define RELOC_KERNEL_HOST_MEM_ENC_ACTIVE BIT(1) +#define RELOC_KERNEL_PRESERVE_CONTEXT BIT(0) +#define RELOC_KERNEL_CACHE_INCOHERENT BIT(1) =20 #endif =20 diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index bde58f6510ac..a24c7805acdb 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -731,6 +731,8 @@ void __noreturn stop_this_cpu(void *dummy); void microcode_check(struct cpuinfo_x86 *prev_info); void store_cpu_caps(struct cpuinfo_x86 *info); =20 +DECLARE_PER_CPU(bool, cache_state_incoherent); + enum l1tf_mitigations { L1TF_MITIGATION_OFF, L1TF_MITIGATION_AUTO, diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index a6f88ca1a6b4..5398db4dedb4 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -545,6 +545,23 @@ static void early_detect_mem_encrypt(struct cpuinfo_x8= 6 *c) { u64 msr; =20 + /* + * Mark using WBINVD is needed during kexec on processors that + * support SME. This provides support for performing a successful + * kexec when going from SME inactive to SME active (or vice-versa). + * + * The cache must be cleared so that if there are entries with the + * same physical address, both with and without the encryption bit, + * they don't race each other when flushed and potentially end up + * with the wrong entry being committed to memory. + * + * Test the CPUID bit directly because with mem_encrypt=3Doff the + * BSP will clear the X86_FEATURE_SME bit and the APs will not + * see it set after that. + */ + if (c->extended_cpuid_level >=3D 0x8000001f && (cpuid_eax(0x8000001f) & B= IT(0))) + __this_cpu_write(cache_state_incoherent, true); + /* * BIOS support is required for SME and SEV. * For SME: If BIOS has enabled SME then adjust x86_phys_bits by diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_k= exec_64.c index fdd04b5bb70e..34c303a92eaf 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -29,6 +29,7 @@ #include #include #include +#include =20 #ifdef CONFIG_ACPI /* @@ -426,11 +427,11 @@ void __nocfi machine_kexec(struct kimage *image) relocate_kernel_flags |=3D RELOC_KERNEL_PRESERVE_CONTEXT; =20 /* - * This must be done before load_segments() since if call depth tracking - * is used then GS must be valid to make any function calls. + * This must be done before load_segments() since it resets + * GS to 0 and percpu data needs the correct GS to work. */ - if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) - relocate_kernel_flags |=3D RELOC_KERNEL_HOST_MEM_ENC_ACTIVE; + if (this_cpu_read(cache_state_incoherent)) + relocate_kernel_flags |=3D RELOC_KERNEL_CACHE_INCOHERENT; =20 /* * The segment registers are funny things, they have both a @@ -441,6 +442,11 @@ void __nocfi machine_kexec(struct kimage *image) * * Take advantage of this here by force loading the segments, * before the GDT is zapped with an invalid value. + * + * load_segments() resets GS to 0. Don't make any function call + * after here since call depth tracking uses percpu variables to + * operate (relocate_kernel() is explicitly ignored by call depth + * tracking). */ load_segments(); =20 diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 1b7960cf6eb0..f2bbbeef5477 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -88,6 +88,16 @@ EXPORT_PER_CPU_SYMBOL(cpu_tss_rw); DEFINE_PER_CPU(bool, __tss_limit_invalid); EXPORT_PER_CPU_SYMBOL_GPL(__tss_limit_invalid); =20 +/* + * The cache may be in an incoherent state and needs flushing during kexec. + * E.g., on SME/TDX platforms, dirty cacheline aliases with and without + * encryption bit(s) can coexist and the cache needs to be flushed before + * booting to the new kernel to avoid the silent memory corruption due to + * dirty cachelines with different encryption property being written back + * to the memory. + */ +DEFINE_PER_CPU(bool, cache_state_incoherent); + /* * this gets called so that we can store lazy state into memory and copy t= he * current task into the new thread. @@ -827,19 +837,7 @@ void __noreturn stop_this_cpu(void *dummy) disable_local_APIC(); mcheck_cpu_clear(c); =20 - /* - * Use wbinvd on processors that support SME. This provides support - * for performing a successful kexec when going from SME inactive - * to SME active (or vice-versa). The cache must be cleared so that - * if there are entries with the same physical address, both with and - * without the encryption bit, they don't race each other when flushed - * and potentially end up with the wrong entry being committed to - * memory. - * - * Test the CPUID bit directly because the machine might've cleared - * X86_FEATURE_SME due to cmdline options. - */ - if (c->extended_cpuid_level >=3D 0x8000001f && (cpuid_eax(0x8000001f) & B= IT(0))) + if (this_cpu_read(cache_state_incoherent)) wbinvd(); =20 /* diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocat= e_kernel_64.S index 26e945f85d19..11e20bb13aca 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -198,14 +198,21 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) movq %r9, %cr3 =20 /* + * If the memory cache is in incoherent state, e.g., due to + * memory encryption, do WBINVD to flush cache. + * * If SME is active, there could be old encrypted cache line * entries that will conflict with the now unencrypted memory * used by kexec. Flush the caches before copying the kernel. + * + * Note SME sets this flag to true when the platform supports + * SME, so the WBINVD is performed even SME is not activated + * by the kernel. But this has no harm. */ - testb $RELOC_KERNEL_HOST_MEM_ENC_ACTIVE, %r11b - jz .Lsme_off + testb $RELOC_KERNEL_CACHE_INCOHERENT, %r11b + jz .Lnowbinvd wbinvd -.Lsme_off: +.Lnowbinvd: =20 call swap_pages =20 --=20 2.51.0 From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C164A2FB622 for ; Mon, 1 Sep 2025 16:09:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742994; cv=none; b=Qvw3V9OreEysb+mka9X7RsTYjJVO388/NSgxWNaPM7PnDRyj29yp2Ek0hA18vBgww1VoZQ8VZ2/qIqgb1rZs2rJ19COlMcRwukRJkm7zwVJbHX9i/OpdyDJO13O8/TxNgjhXl3jLrGWTDYueFEwzx/VuSOnELT+G/imNKeSlZRM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742994; c=relaxed/simple; bh=jOAHCUQOI95A6NhD1bdRc7Ks4QP6hIwvLd9f4ISXn+8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tyjQ6OGzWXiSh0sAr5ZnHoggBkyyVi/RM083AOKdnoBW0xJ89pVxd1utPvHeEn7F0wk/cEiOKqd6GzmFJ95pX2gA8IjWYGLapgXqGZsjJyxLIOCIZZrZdqrtfFB3xQk0k9EPavkeY3vFRDaUiloiObsHQ9c4aN4SdVIsG+jmYQo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gXTA5KjE; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gXTA5KjE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756742991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vlmYPrJMvaurL156kfyVvz/kxSmkc+/T+EJ8CZ+U2zk=; b=gXTA5KjE5Jxy/vqnORcKy3qCGatsS9dlgWT1SBJUIf26Ctn+ctfFWv478ogJ/fUrAKYEr5 MpqkKIBQSmN9tmPxT5Ev5ohZvucvxbpvpUi/9ES+I0MqXTxk57V2qj9anvxizxCsVL5vfR LSM/nHygS5wfNWa0CFOL4w2a7/Cw5xM= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-621-EQO_dQyOPlSWfCoVRtXwNg-1; Mon, 01 Sep 2025 12:09:51 -0400 X-MC-Unique: EQO_dQyOPlSWfCoVRtXwNg-1 X-Mimecast-MFC-AGG-ID: EQO_dQyOPlSWfCoVRtXwNg_1756742990 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-70de0bdb600so77132776d6.1 for ; Mon, 01 Sep 2025 09:09:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756742990; x=1757347790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vlmYPrJMvaurL156kfyVvz/kxSmkc+/T+EJ8CZ+U2zk=; b=vSbGR2NCZ9I8F1SIh+tBL6C421roqx/wV+piC2fvpituvg95f+0RzOj8PqDzhZQp6k ZPqUW2mH/DYy278gIy4wn1KOPcu0MF1H0VLQKWYsrMABKEgljLGm7Pv+BHvAzLwkzcZu onKbAzxU4dbetszWRQ3M7ujD8Vw6T4Gd9wlDv4lFJxWJrVZn9yTDkZk27JSTnAU8gQkS 3ITkv0OSoVlKFH6deSJSNdNOhSnLudckjWwxUgzdOVLNTj12ZTO/JE80Sgm+fvBBczwp c3nuBCDkj50jKEh2WiOynwD/mps+V69AUarvgD22YMFr3tarjLYRHe1Fr698XCO2G1iK 0eRg== X-Gm-Message-State: AOJu0YyMsIFCXX4Jj8l+hxExgwvX5vZFRUOp9nmrNCx0gPE5+o19WwAu Fgyb5uB9yAE/S1ooNNQZb2X6VDwxyu6niAmPfKJxcokCeH7TAzSPLtTORHK3CFONps8BESjt8Sh qpl6oFDD3FK/KlkD9PE0Va+3uGPlbcScWHmWF6IKjC/1G+f+sH8wzTo7hUUNkaNbrS9l5gbxsaM CIdSHbwrVc/B9U0NT64CaApDdeq/H8AwCPYmSnKTARzFn+wCKpgA== X-Gm-Gg: ASbGncsVZrA/jYsOlt/+YRpVGwgQu8W5sLQz19DpVfgsiezR2tNZwxBTFuqWIXEMm+i x31vxnO8ysvnAeliL/6H9/lvrlwR8+f/OYDmKPtHbhW5s/hw0YG0f1yj2GpTzv2sHqk7l4iirDC FS1E4yXv9Xcq2EPyETvjhFKiQXfMs8fmYS2nVjVCVG3+9+N4EDxncFqKk3HRsU2fK9gKPLunUBM sSpQqY2SDVcXTfpWCcIbR/60wmSRx+Xm+UJCOzb6WcUUjvZdjV7PUezCMUi4rTdm7z8tL2/vDWK EySsUpVCzm/EoXnpQwlH8gQ8x/qHYCFh86dbRvGxKITfiY8D3HyziCfcQMx9KC4CF5QBnMOIIFZ lXqNfY4YoLDX0fWPOczKAYtP8b6JT6zRsVB96cyG5kzP+Kl/xaYU7doKwonGM+sAKWmTI X-Received: by 2002:a05:6214:1c8e:b0:70d:9e0a:5ac0 with SMTP id 6a1803df08f44-70fac8dd487mr82341306d6.57.1756742989490; Mon, 01 Sep 2025 09:09:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE3atxEoR4Y3UwAcVItWfUxbMXJnqefXOjBiiJOc9QLCFCXra+INStGLPsCJ1tjig0u8r6LRA== X-Received: by 2002:a05:6214:1c8e:b0:70d:9e0a:5ac0 with SMTP id 6a1803df08f44-70fac8dd487mr82340626d6.57.1756742988809; Mon, 01 Sep 2025 09:09:48 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-70fb260d977sm42473206d6.20.2025.09.01.09.09.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:09:48 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com Subject: [PATCH 3/7] x86/virt/tdx: Mark memory cache state incoherent when making SEAMCALL Date: Mon, 1 Sep 2025 18:09:26 +0200 Message-ID: <20250901160930.1785244-4-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang On TDX platforms, dirty cacheline aliases with and without encryption bits can coexist, and the cpu can flush them back to memory in random order. During kexec, the caches must be flushed before jumping to the new kernel otherwise the dirty cachelines could silently corrupt the memory used by the new kernel due to different encryption property. A percpu boolean is used to mark whether the cache of a given CPU may be in an incoherent state, and the kexec performs WBINVD on the CPUs with that boolean turned on. For TDX, only the TDX module or the TDX guests can generate dirty cachelines of TDX private memory, i.e., they are only generated when the kernel does a SEAMCALL. Set that boolean when the kernel does SEAMCALL so that kexec can flush the cache correctly. The kernel provides both the __seamcall*() assembly functions and the seamcall*() wrapper ones which additionally handle running out of entropy error in a loop. Most of the SEAMCALLs are called using the seamcall*(), except TDH.VP.ENTER and TDH.PHYMEM.PAGE.RDMD which are called using __seamcall*() variant directly. To cover the two special cases, add a new __seamcall_dirty_cache() helper which only sets the percpu boolean and calls the __seamcall*(), and change the special cases to use the new helper. To cover all other SEAMCALLs, change seamcall*() to call the new helper. For the SEAMCALLs invoked via seamcall*(), they can be made from both task context and IRQ disabled context. Given SEAMCALL is just a lengthy instruction (e.g., thousands of cycles) from kernel's point of view and preempt_{disable|enable}() is cheap compared to it, just unconditionally disable preemption during setting the boolean and making SEAMCALL. Signed-off-by: Kai Huang Tested-by: Farrah Chen Reviewed-by: Chao Gao Reviewed-by: Rick Edgecombe Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/tdx.h | 25 ++++++++++++++++++++++++- arch/x86/virt/vmx/tdx/tdx.c | 4 ++-- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 57b46f05ff97..c178360c1fb1 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -102,10 +102,31 @@ u64 __seamcall_ret(u64 fn, struct tdx_module_args *ar= gs); u64 __seamcall_saved_ret(u64 fn, struct tdx_module_args *args); void tdx_init(void); =20 +#include #include +#include =20 typedef u64 (*sc_func_t)(u64 fn, struct tdx_module_args *args); =20 +static __always_inline u64 __seamcall_dirty_cache(sc_func_t func, u64 fn, + struct tdx_module_args *args) +{ + lockdep_assert_preemption_disabled(); + + /* + * SEAMCALLs are made to the TDX module and can generate dirty + * cachelines of TDX private memory. Mark cache state incoherent + * so that the cache can be flushed during kexec. + * + * This needs to be done before actually making the SEAMCALL, + * because kexec-ing CPU could send NMI to stop remote CPUs, + * in which case even disabling IRQ won't help here. + */ + this_cpu_write(cache_state_incoherent, true); + + return func(fn, args); +} + static __always_inline u64 sc_retry(sc_func_t func, u64 fn, struct tdx_module_args *args) { @@ -113,7 +134,9 @@ static __always_inline u64 sc_retry(sc_func_t func, u64= fn, u64 ret; =20 do { - ret =3D func(fn, args); + preempt_disable(); + ret =3D __seamcall_dirty_cache(func, fn, args); + preempt_enable(); } while (ret =3D=3D TDX_RND_NO_ENTROPY && --retry); =20 return ret; diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 823850399bb7..2abf53ed59c8 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -1268,7 +1268,7 @@ static bool paddr_is_tdx_private(unsigned long phys) return false; =20 /* Get page type from the TDX module */ - sret =3D __seamcall_ret(TDH_PHYMEM_PAGE_RDMD, &args); + sret =3D __seamcall_dirty_cache(__seamcall_ret, TDH_PHYMEM_PAGE_RDMD, &ar= gs); =20 /* * The SEAMCALL will not return success unless there is a @@ -1524,7 +1524,7 @@ noinstr __flatten u64 tdh_vp_enter(struct tdx_vp *td,= struct tdx_module_args *ar { args->rcx =3D tdx_tdvpr_pa(td); =20 - return __seamcall_saved_ret(TDH_VP_ENTER, args); + return __seamcall_dirty_cache(__seamcall_saved_ret, TDH_VP_ENTER, args); } EXPORT_SYMBOL_GPL(tdh_vp_enter); =20 --=20 2.51.0 From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBE653064AC for ; Mon, 1 Sep 2025 16:09:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742999; cv=none; b=fDRI0wnKT1JRhwIn05V4TLOeowI/4hahRrpu5VyTYZJhTbu52haLfqK89iawddilcJCyjDm6zopKnUZ053vVDBhmrAhh8o0NNATpCTVvu+YUD6Ps+x4U/qdDbpwzjCExB0d9YXl0/EQCIymbEUDCwJWdEEDpQYXXltOsoAnCpmw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756742999; c=relaxed/simple; bh=Q3PKDbHD38isJx3I8Ber0baejrO6TVXVWQsXQu3Gx6Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HWEgI2VpjyQtHf/7V4rRVQIIdmD4ucNc/cgmmNRKs2F6Wr3C5HwB99Ar9ljP/CBxbPRlYEKxTX2WgvvX9B7srvUNnPXRU9DsiccY3Uas2yJY1yKuIg5SDY91YqFi6lXei5GJfqwYpAmCSd+QqjB8hzJT6siXFOJuXL9bCsyZd60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=J1COYYbn; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="J1COYYbn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756742996; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iUjmNo6Stszk1jMfX7esmLkjaDgZtcc668a9zyeu+wg=; b=J1COYYbnQbOmL8Z+2KKLVbZMq36JM3Ot6Mrr/QLEZDNqYhxpYS74BxSkrl0be5A3FQp/Uw c0Ryavm9JNkV5DTf/XhVdpoatl3Hl2S54DMkNfkn8Wkf/hIBPJBi+wwkqGiRNmhvP5hKS8 vxnZwPiewEVKSuTXJG0S+/jeAOKfgTA= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-468-iCDlmBNNPRul0qNg695MGw-1; Mon, 01 Sep 2025 12:09:55 -0400 X-MC-Unique: iCDlmBNNPRul0qNg695MGw-1 X-Mimecast-MFC-AGG-ID: iCDlmBNNPRul0qNg695MGw_1756742995 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4b311db2c76so10972271cf.2 for ; Mon, 01 Sep 2025 09:09:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756742994; x=1757347794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iUjmNo6Stszk1jMfX7esmLkjaDgZtcc668a9zyeu+wg=; b=C1lBTALZb6xyz2lT8W+itftU8VWqoEKlyS94xAlGSzgL8lNKwp1RxpO60XLlzye6zm f0gpfZI1Ff2CtfUx0Ehf1hUR3TYm+9MqW68D/lhRGFHN8CghRxxyPPDpE0YgoSmGfLdA Y0gGNKuZdy34+4M5qhatl9ivzuwqaPiX0ioZkisolYnR6ztJYSZnu7Yj54TH/ZtrvTgq AkqQfEV+Z4VAwgu65ZN72b908LpoRQ0WYaSgwL2/tLKYroEdL9JUo28g2FxecX7o4n+P nh6gJ2Xbyo+VlvnvSKkEQPRg1j1jQZLLo807m+65M+AbxdTRDF7DCnVbCF/vOuyDdj2F /uDw== X-Gm-Message-State: AOJu0YyCpopypHjZagOEjLX5vK6GC/AwUJkXe9xr3jXoV1qfRNeodl0U 5rEaghsb83fhOaFkPh5spvMIHbOLsvozFdH42hBIPCMEdtz3w7q051hw4iGtpufviIDvWEB1w7J H4V+TZfCy+F4zqwAMZqFuDsgAw4TobUdXs/xwDIELG5sd0Rk23nVCzcr6YGMtsJ7k4mhmiQvToi B/l8H18faHY8U+kZablZLpbycuGCoNUwmlN8TghoEVJVhzYavNWA== X-Gm-Gg: ASbGncuNGkjHscteaEHL32ltPyBSr2LymAoH/cbDGJlvmFh7ZM/iqbyKWi+J1Ysgq4s S9/2x1WxcZdNyokjcOiWFB8MYnMEx7bYrHWMgJJMlgUIDb2iUJ4GId2Q2KdRjHVSf28aW8fPgck 6B8ayXZF4PF/67Iw/9sN3nV0J220Xi2f9Ujjple+AGBMn36pYn4OyMMW79FgrWH1uedHYfPjnfb j0Jkx+WRknfsRsU+7GW+hKkYzJ5dR4PyYeEAbWtvG2S2w8+lKyzg+0oN8e+c9opaesSI5HDTnFS IVp8Pf4NgIxWv+XrxWazJy8WWvKI27/Z6AKC9o09tmEHUlLOms7HLPWHP5L2SJoHHOw+FZ2nT0z SpUxRh/YvctnMPtkPapnpgTu6Ot5jqaWB9JcpnfG2DzKy9JIBiZin/fX8vD8YkCOIo/UY X-Received: by 2002:a05:622a:2cb:b0:4b3:d28:c96 with SMTP id d75a77b69052e-4b31d7f062emr105567981cf.13.1756742994202; Mon, 01 Sep 2025 09:09:54 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF0H+Me2WNr8OP1G6r+ikLDYpI0KnL49DOiTcekuqb2wap6IvnGdRHVwSMHQzA8c3JlD+sHkA== X-Received: by 2002:a05:622a:2cb:b0:4b3:d28:c96 with SMTP id d75a77b69052e-4b31d7f062emr105567311cf.13.1756742993721; Mon, 01 Sep 2025 09:09:53 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7fc1643ac5esm681629485a.68.2025.09.01.09.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:09:53 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com, Binbin Wu Subject: [PATCH 4/7] x86/kexec: Disable kexec/kdump on platforms with TDX partial write erratum Date: Mon, 1 Sep 2025 18:09:27 +0200 Message-ID: <20250901160930.1785244-5-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang Some early TDX-capable platforms have an erratum: A kernel partial write (a write transaction of less than cacheline lands at memory controller) to TDX private memory poisons that memory, and a subsequent read triggers a machine check. On those platforms, the old kernel must reset TDX private memory before jumping to the new kernel, otherwise the new kernel may see unexpected machine check. Currently the kernel doesn't track which page is a TDX private page. For simplicity just fail kexec/kdump for those platforms. Leverage the existing machine_kexec_prepare() to fail kexec/kdump by adding the check of the presence of the TDX erratum (which is only checked for if the kernel is built with TDX host support). This rejects kexec/kdump when the kernel is loading the kexec/kdump kernel image. The alternative is to reject kexec/kdump when the kernel is jumping to the new kernel. But for kexec this requires adding a new check (e.g., arch_kexec_allowed()) in the common code to fail kernel_kexec() at early stage. Kdump (crash_kexec()) needs similar check, but it's hard to justify because crash_kexec() is not supposed to abort. It's feasible to further relax this limitation, i.e., only fail kexec when TDX is actually enabled by the kernel. But this is still a half measure compared to resetting TDX private memory so just do the simplest thing for now. The impact to userspace is the users will get an error when loading the kexec/kdump kernel image: kexec_load failed: Operation not supported This might be confusing to the users, thus also print the reason in the dmesg: [..] kexec: Not allowed on platform with tdx_pw_mce bug. Signed-off-by: Kai Huang Tested-by: Farrah Chen Reviewed-by: Rick Edgecombe Reviewed-by: Binbin Wu Signed-off-by: Paolo Bonzini --- arch/x86/kernel/machine_kexec_64.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_k= exec_64.c index 34c303a92eaf..201137b98fb8 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -347,6 +347,22 @@ int machine_kexec_prepare(struct kimage *image) unsigned long reloc_end =3D (unsigned long)__relocate_kernel_end; int result; =20 + /* + * Some early TDX-capable platforms have an erratum. A kernel + * partial write (a write transaction of less than cacheline + * lands at memory controller) to TDX private memory poisons that + * memory, and a subsequent read triggers a machine check. + * + * On those platforms the old kernel must reset TDX private + * memory before jumping to the new kernel otherwise the new + * kernel may see unexpected machine check. For simplicity + * just fail kexec/kdump on those platforms. + */ + if (boot_cpu_has_bug(X86_BUG_TDX_PW_MCE)) { + pr_info_once("Not allowed on platform with tdx_pw_mce bug\n"); + return -EOPNOTSUPP; + } + /* Setup the identity mapped 64bit page table */ result =3D init_pgtable(image, __pa(control_page)); if (result) --=20 2.51.0 From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 333012E6CA3 for ; Mon, 1 Sep 2025 16:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756743003; cv=none; b=H7FiUN6RasOdlbY/APTF7kx5qSQaTTKb30q+dIMV8f+7LYinzSQfZD3cdylCkwQjm9YOxAevux7q8ZdHxGcbb/D3pmXXlFZjN4iZKz617gnRso2+wiNy1nVbZkHzVvoiD0NUhY0GEXujvRBlrFnBtNVxO7V+95cTg/ThhNzUDyw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756743003; c=relaxed/simple; bh=JGm5CCLptQwOuSwtvIdC2c2wUIwgPDkpfBMNALrs+Ds=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E1Y+aWKJPWjAjls9rF8AOHG+SWWpKp+26SIb4ltjGWBS15SeYvTK5RArKCnlcshDVG/sYKYJgFPhd8N+1SOC9piIrLGnoVxnoqv6m8c6HVZ10Qp6nh4INPy0BETRZCKqNn4EwbotZ1c53nEjbbWqfeTyUaaibhpxKUK7kNXm0u0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Dx70m/eP; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Dx70m/eP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756743001; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BK4OUWj9KISSOnqHrSCdtAWi7t1NEdnZ5iAAINpO6Ag=; b=Dx70m/ePUiAMmCSRwEBSiGVYSXG38DbAXJc1fHeh289PPttbRELlk48dnY0CWvpJkCXw96 42XRBkUsvTo/5sGYr0K+m4EulMS27Aaw2UilPzofA7j6RE+EWhOYBouFUk78PZsaEA0v/s 0ulheogS6eTM918FHL1sFNn5IT6SF2E= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-hA1VLcFgMmqgl9z8rA1VQA-1; Mon, 01 Sep 2025 12:10:00 -0400 X-MC-Unique: hA1VLcFgMmqgl9z8rA1VQA-1 X-Mimecast-MFC-AGG-ID: hA1VLcFgMmqgl9z8rA1VQA_1756743000 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7e870614b86so1366461785a.2 for ; Mon, 01 Sep 2025 09:10:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756742999; x=1757347799; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BK4OUWj9KISSOnqHrSCdtAWi7t1NEdnZ5iAAINpO6Ag=; b=SDIROgocVn7SZtD/tYJDecZYEKvx00zMR+MCSGOYEwecACZR74y2GZyxGd4qUBlJSo 2Ts+EWLrB0XINzm2pG4PGpBetYWT0Bp2kQql2/ylgMoQK1LkjrBEaZhc0HB4t/pKNuh6 uVVQNn8RnrxHZ8y68BMm511pc/lM+kbj3feDfYAJZN/R2YZCJ+Rm4zD6Oc8TDNL8V817 vxPohZdFLBb6Zdze28usHn0YBhM6mGyLhk25pwNf+KbYNdORnJi8K17eMF3AV6LAeKvp V/aESgrgdYcVTlo9WM5FMrlchlya17j2zp6mgAPHKpqQk1k876FofLkwzSA2EBLb/6s2 5nNg== X-Gm-Message-State: AOJu0YzaTqqGm6GyZQMRsQ7J6tgM+0C7bI3SSy431ZMD+bItDby9g4q3 z+oL2nYrwU1e6HmTlTjZV8xELseVKScmKndPKzuM1jir5RTvaeLgvA8YvmULPutHwg3jlgm4Cw+ xfYm2d87PJ/xB3ZzODxLhjA0tadRVrob/a/KU7ejCF8tUzLb0htb9IbPi+C2BoKyIu3X8c9PvU+ qVNW0rPwlQZkVstUA09pfgO4/YmPc257pmXEhnJYUavzhWn4hOIg== X-Gm-Gg: ASbGncuyepwHlAS3GW+e7ktYn7ZelfuX4vIPS88Ff6TGsKwb2Qg851CGyMbBRAvICeb asbb7MirRFBwohos/uOnSiFYAmCNcc6jBRoPWZrq4rYgda7A8ZVra2t+W6ju/kAjqrVcYcKr0n6 QIRj+2znX8YoVZWD9cDGVsyNj/io2ukw8hKPC2EJk9Xmtb5vlzxbMO930Pze5gTfrB+soeNTJOF r714jV2ObUJnUN5DTJ2EMoya0DdWlqsUN/yOLeRO4mjDIh2ShTKhF1FrqhrzG9zCXQfSFF6CvKk JJY6eek/oDfW0jyTSnvhbOZe/c7bbjjqwAni25zZZKMQ165l+IQPiQRHDBDxs0EQ/xQfibA7Rwl GXYO2v7BfRC1O/d2/11oGUbP1oOzDLqxVbL+BMtJbdCos2iZKAqbpJ2oM5JRky9HQ5v2q X-Received: by 2002:a05:620a:44ca:b0:7f0:b17:cb6a with SMTP id af79cd13be357-7ff2b5a761emr961318785a.57.1756742998801; Mon, 01 Sep 2025 09:09:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGWFuIM6z/6h3ooA05y4N+Coc7uXzQ+I+d1s24RIzvqtIGoYRX2aMLPreLhFfFBviPAkLZ8sA== X-Received: by 2002:a05:620a:44ca:b0:7f0:b17:cb6a with SMTP id af79cd13be357-7ff2b5a761emr961312285a.57.1756742998331; Mon, 01 Sep 2025 09:09:58 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4b30b54cc37sm61618191cf.12.2025.09.01.09.09.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:09:57 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com Subject: [PATCH 5/7] x86/virt/tdx: Remove the !KEXEC_CORE dependency Date: Mon, 1 Sep 2025 18:09:28 +0200 Message-ID: <20250901160930.1785244-6-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang During kexec it is now guaranteed that all dirty cachelines of TDX private memory are flushed before jumping to the new kernel. The TDX private memory from the old kernel will remain as TDX private memory in the new kernel, but it is OK because kernel read/write to TDX private memory will never cause machine check, except on the platforms with the TDX partial write erratum, which has already been handled. It is safe to allow kexec to work together with TDX now. Remove the !KEXEC_CORE dependency. Signed-off-by: Kai Huang Tested-by: Farrah Chen Reviewed-by: Rick Edgecombe Signed-off-by: Paolo Bonzini --- arch/x86/Kconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 348a193a3ede..217982814bd7 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1895,7 +1895,6 @@ config INTEL_TDX_HOST depends on X86_X2APIC select ARCH_KEEP_MEMBLOCK depends on CONTIG_ALLOC - depends on !KEXEC_CORE depends on X86_MCE help Intel Trust Domain Extensions (TDX) protects guest VMs from malicious --=20 2.51.0 From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B683C2E6CA3 for ; Mon, 1 Sep 2025 16:10:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756743009; cv=none; b=EqXwyeTzpZCWcAAGnp74NlkQ1v8B1DwQlzT1zv8e1q2rwUNTph1Oc9IqIhKgsIOpmpFjc9zFXqedPJx6USgWkuhww62jIvZ5rkbSiKXQZeez7NGHWh8EoGkXBpu/H/r1MnTVDo0mPrLboHUqvQ6oZCYjpIhONy93USYV4bKeM2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756743009; c=relaxed/simple; bh=j6vAFM4b+QjRpby8SIZgqTZ2iRbpFgSXYQUKj1TwaXI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gIVeo/2fNVMP1D/1KMeq09T8Sa4ZrbFbyHGciO4WG178pSMw66S8kWrjjHK1CXTfvI/3bu3pxsyN2hcay+9FazcQ7s/CNX7uSepAbfkSkEBMRYj7qYMgGa9/7k3jN8cPiy5TvE/wOiDBUDBZztnjh81nWMxDUMM5TbZWFoDeyNA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GJBD4gII; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GJBD4gII" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756743006; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dtdq00SxSPn8DNlbTBMbLDu8kopEQRB52XTmsM+6vew=; b=GJBD4gIIcgxs+eYtWCDkjb2qIPyjc/RzEU7VF4FNOsmBPizlZcKDiGkehFqkCk3dKfqzgn ZxMNwi8k0xJD0WTn52K6zm2pkyyLE7ybPqMhig77uzjDzAbyyDK0AQf73Eocc7fXwzm4wg jpsPakbksN3ZScVRlXJQmt/7spgvRvY= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-480-RObLf88QMsSqecvAPs8a5g-1; Mon, 01 Sep 2025 12:10:05 -0400 X-MC-Unique: RObLf88QMsSqecvAPs8a5g-1 X-Mimecast-MFC-AGG-ID: RObLf88QMsSqecvAPs8a5g_1756743004 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-70f9ef271a4so52269806d6.0 for ; Mon, 01 Sep 2025 09:10:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756743004; x=1757347804; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dtdq00SxSPn8DNlbTBMbLDu8kopEQRB52XTmsM+6vew=; b=QP6JCC5Ya4I9JIP2SKERdG0yRQgTyjrPYeDK51pEHMX5C4PGOW31VWGvAJf6Vl4ZkD 5U7gzdgUegCjuGLitLFSheUSIN3UBG4rkQxzUkptD6116TW4XvYT5BBxDFc8QEhabVhf F73j5Kc3MHeptPpYCXplBDQ4brIMcV3OwGnCiaYJWTOhOaYog2mhjE9reFL8qMtDlCUm 6o5DP+4uCdbUiASIxP08z75pPDYa2bZjOxBz1Ni0FAWudKMUB3V6KIbJf9daBQ4H+cC9 9N/joARuNLZoU9yA3QypFYUzEEQ/Iz1GmP8e4WNdUFNyFN+q866MCRruJiLddzT8mpMd MLUg== X-Gm-Message-State: AOJu0YyHreU3BXTf9D9CiKzBJjeEeVFrpG643QrbumB1FV4cPlKD3wN/ Oo+cUw7oCTK538H/uoYpxY9UMzFCfSjVf+qgtA2buGMxSlJAigCoMbxo+l6SgyMiMCQ5QEt5rz/ X7Wl/8tkSTOytaAFhNb6fW4aYfjS/m9ShTjKjijLlHN/F1vaLBYR0e8b5B1NvzgVh868i8fe9Cx 1g3O/35YR+hdqKPwV4Bc7njCF69NZp4+DxsZfTwHxU+SlLKyPFLA== X-Gm-Gg: ASbGncsCvzKPwaxWFsEvqfM8eRnwy2h1Ytw1tbnvDV/LqJfFdToBByhWr4PZZ0xeV9j kpozJ1BEm0YRug5UMzsI/zPHVx1zvtNpCzm6QiMAh6AEQuxV6WCUDwQLinMLrCXzlT4oTLERFOq k/HBhCHET1bcRPTLEJFZ90DbJVzNPxhwfjnCF9vu4CHNuScdEZqbRtc3iwWzFzgNranPf50lL24 DKtQcndxAGo7W86puGW9tYw86GBSsvFEdxoJu23nLrcVeESJkwRxdYmzAGVXXIW9KFysvVe0lbW dsEEAbP792sYgMQzzFh2OyyaGK7Tt4/oW0G3boQvT6H0uoHhDiUXVY2+WWVpP60StlbUzIMNeWW DjLPoB97cgn+t8IA/Dr4Zq7AxcI5DVBvVXKtnDLipsuiW/z5MQflFerQDZUnvjQbw5caM X-Received: by 2002:ad4:55f0:0:b0:70f:abbb:609a with SMTP id 6a1803df08f44-70fabbb65f3mr84998806d6.19.1756743003657; Mon, 01 Sep 2025 09:10:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF5ZhmrDjiVHmX4k410H1KEfDTJpsrmWdSsgPge/0tAl2l7w6yZq46ZoMj1qfypUN8EaSKecw== X-Received: by 2002:ad4:55f0:0:b0:70f:abbb:609a with SMTP id 6a1803df08f44-70fabbb65f3mr84997956d6.19.1756743002916; Mon, 01 Sep 2025 09:10:02 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-70fb2837f9esm41837066d6.48.2025.09.01.09.09.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:10:02 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com Subject: [PATCH 6/7] x86/virt/tdx: Update the kexec section in the TDX documentation Date: Mon, 1 Sep 2025 18:09:29 +0200 Message-ID: <20250901160930.1785244-7-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang TDX host kernel now supports kexec/kdump. Update the documentation to reflect that. Opportunistically, remove the parentheses in "Kexec()" and move this section under the "Erratum" section because the updated "Kexec" section now refers to that erratum. Signed-off-by: Kai Huang Tested-by: Farrah Chen Reviewed-by: Rick Edgecombe Signed-off-by: Paolo Bonzini --- Documentation/arch/x86/tdx.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Documentation/arch/x86/tdx.rst b/Documentation/arch/x86/tdx.rst index 719043cd8b46..61670e7df2f7 100644 --- a/Documentation/arch/x86/tdx.rst +++ b/Documentation/arch/x86/tdx.rst @@ -142,13 +142,6 @@ but depends on the BIOS to behave correctly. Note TDX works with CPU logical online/offline, thus the kernel still allows to offline logical CPU and online it again. =20 -Kexec() -~~~~~~~ - -TDX host support currently lacks the ability to handle kexec. For -simplicity only one of them can be enabled in the Kconfig. This will be -fixed in the future. - Erratum ~~~~~~~ =20 @@ -171,6 +164,13 @@ If the platform has such erratum, the kernel prints ad= ditional message in machine check handler to tell user the machine check may be caused by kernel bug on TDX private memory. =20 +Kexec +~~~~~~~ + +Currently kexec doesn't work on the TDX platforms with the aforementioned +erratum. It fails when loading the kexec kernel image. Otherwise it +works normally. + Interaction vs S3 and deeper states ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 --=20 2.51.0 From nobody Fri Oct 3 11:22:29 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E7BD320A2F for ; Mon, 1 Sep 2025 16:10:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756743016; cv=none; b=FQ7KJKA51OxnaddncHLsdEJ84BQT0O6pRGCAL3s8DXZaBJsEZCiv1v+IV48Fbf+ImYcMLrzXwoQO9S6Q8QexPb8yFZw1Rxw8L8gyBhaQcE2DenVLCph+m9dqpbAq1ea2CA7VebgqDT80dFNvcymaPJ9sZhefQIwgxRaSVvhSa5g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756743016; c=relaxed/simple; bh=Uo10WvJDZhKL6JBEoQLyO2D0kgnDiGHkYMbNhBPE5D4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rIXv1r/yF29vtdzcvasX0AhGPjhMvjTWnBO4pt5R0pdXYyDLKYwKQjU2YEmPC84h/jcUvF0Ze1dtINgxDf5bEss3RgNg9639iswNmfOrneW8U9iZswOfif5dlIODY5BPhUSaG03x+K2f9eMqBAvPXrCkX+bN9gh0X3ubfwcgwMQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ni8Sbvzi; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ni8Sbvzi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1756743012; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ay5gC/rJKyMKcqoVVau41lZbwoCz+kjLe1Ep7vUjxrI=; b=Ni8SbvzigbGyMJSLQk16IoJcqD0+rIMxxx6bLM5tvLc1VFmFMeqML4flzjvIMDySH9fHOu qWTIo0bFC2majc2D9dUnkoKzqML58/HQ4oepyxv66DzAvxMtoHgwvdUhUg80f57co3rWkt n4H4YmmqGOyotNEGpCfcNUsMzlKjYqo= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-167-YG3oUaRHOZWiO0Swj90NdA-1; Mon, 01 Sep 2025 12:10:11 -0400 X-MC-Unique: YG3oUaRHOZWiO0Swj90NdA-1 X-Mimecast-MFC-AGG-ID: YG3oUaRHOZWiO0Swj90NdA_1756743009 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-70dfd87a763so46602616d6.2 for ; Mon, 01 Sep 2025 09:10:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756743008; x=1757347808; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ay5gC/rJKyMKcqoVVau41lZbwoCz+kjLe1Ep7vUjxrI=; b=TQ2P0bOGdEC5qJysL3l13EuCs1GOERN6PF5fTg89FnKZaV38F8dXaMeTDgy9G40NW3 eDMigY+N9oF5LHrzAw+oBrijtR9XhP7FcM2gHoJ/66pNSOlX7Mifq7IJ1yyR2UsQ9yns 0wcDERu/I/XisvZHQ5a46Uw/naBHqNzuhL+qJsNNz1+sU6Ol3Pp01vpe2syCpbQj/weI 28A+cVKv53SKQyjHtJniET4hz9e+ZGNCn9mZhAAvkdUH1uqnNzUvPNL5yjjl8X3qhjYo YeikIoVyVPANVDhmqwyZlocDTUAkOW45u8s9tXaEdMz4Q5a9XRcr4P2QXnGNmLtScBrS 9DfQ== X-Gm-Message-State: AOJu0YwQTtKuvfHr61Vy8sfmTtd/omnLC/Wbv0WQyExHY+cG8Eif9oVK P4qcGaY62M7AP2kZpaufecFOqtUCTtONAyhc6NppaachM/B7ma17ldqjHLJtkdxGvgdvDz6A1xW LyJxVtTTUOuU1luEMfk0SEY5or4m0LAebULNbqFq5/XHQYfQplxH2CHqWbxYFumPRoN/Ck+BhUl fpERWGQjwUv9x3u3Wya1VdHZRk6I+1ELYU3TVEau8C6rfvPP/KOw== X-Gm-Gg: ASbGncs7P5P8y4x6Q8CsSPzIAy76kM0MqQJZsYmYPiAuwwRXa/zGfDBchT2zAV9hzIt otVN/+Ey6Db+QVd1sNGYOLkY3R4kRVRl/UUEovCPXvAkY1Ln+KqjeKoD0l8MarwZAYQWKDeYtFg Cp6e+lHUb0fdnIMqlxYybv2mcrfl+u7Bc1DbcgGHlwFX8JJqP2JEHcv7DTqX/V/0t3ez8U704XJ KOI/6ohdLdj6pTflh9gz/wd1p14xq5QTJtuktKw/u1KYUzPQeM+71kpyeNFaviMRdEDYDfoG/RK Ub4Dw4h/6UANiwfRojuvQHnrEjhC9uFhbRj1ya6mFTUD02swMX/7je5MaJVHn1r+CIdFZvBjtBi zv0fn16JtEdj/pxTpcyxjC84qokGXrUwQoiWOzVWEA0HsG326ARaO/kapniRsYshxnNwS X-Received: by 2002:a05:6214:5195:b0:70f:5a6d:a253 with SMTP id 6a1803df08f44-70fac8db3c8mr75709036d6.49.1756743008454; Mon, 01 Sep 2025 09:10:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGUuYwiXfYyBhb47Z0hN01SMvdEEJPNPj4R4dXKcCWUWWe/rDi3MWAMqA91oA+pVPmIsSEo7A== X-Received: by 2002:a05:6214:5195:b0:70f:5a6d:a253 with SMTP id 6a1803df08f44-70fac8db3c8mr75708306d6.49.1756743007561; Mon, 01 Sep 2025 09:10:07 -0700 (PDT) Received: from [10.201.49.111] (nat-pool-mxp-t.redhat.com. [149.6.153.186]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-70fb283781asm42015296d6.38.2025.09.01.09.10.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Sep 2025 09:10:06 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: dave.hansen@intel.com, bp@alien8.de, tglx@linutronix.de, peterz@infradead.org, mingo@redhat.com, hpa@zytor.com, thomas.lendacky@amd.com, x86@kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, dwmw@amazon.co.uk, kai.huang@intel.com, seanjc@google.com, reinette.chatre@intel.com, isaku.yamahata@intel.com, dan.j.williams@intel.com, ashish.kalra@amd.com, nik.borisov@suse.com, chao.gao@intel.com, sagis@google.com, farrah.chen@intel.com Subject: [PATCH 7/7] KVM: TDX: Explicitly do WBINVD when no more TDX SEAMCALLs Date: Mon, 1 Sep 2025 18:09:30 +0200 Message-ID: <20250901160930.1785244-8-pbonzini@redhat.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250901160930.1785244-1-pbonzini@redhat.com> References: <20250901160930.1785244-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kai Huang On TDX platforms, during kexec, the kernel needs to make sure there are no dirty cachelines of TDX private memory before booting to the new kernel to avoid silent memory corruption to the new kernel. To do this, the kernel has a percpu boolean to indicate whether the cache of a CPU may be in incoherent state. During kexec, namely in stop_this_cpu(), the kernel does WBINVD if that percpu boolean is true. TDX turns on that percpu boolean on a CPU when the kernel does SEAMCALL, Thus making sure the cache will be flushed during kexec. However, kexec has a race condition that, while remaining extremely rare, would be more likely in the presence of a relatively long operation such as WBINVD. In particular, the kexec-ing CPU invokes native_stop_other_cpus() to stop all remote CPUs before booting to the new kernel. native_stop_other_cpus() then sends a REBOOT vector IPI to remote CPUs and waits for them to stop; if that times out, it also sends NMIs to the still-alive CPUs and waits again for them to stop. If the race happens, kexec proceeds before all CPUs have processed the NMI and stopped[1], and the system hangs. But after tdx_disable_virtualization_cpu(), no more TDX activity can happen on this cpu. When kexec is enabled, flush the cache explicitly at that point; this moves the WBINVD to an earlier stage than stop_this_cpus(), avoiding a possibly lengthy operation at a time where it could cause this race. [1] https://lore.kernel.org/kvm/b963fcd60abe26c7ec5dc20b42f1a2ebbcc72397.17= 50934177.git.kai.huang@intel.com/ Signed-off-by: Kai Huang Acked-by: Paolo Bonzini Tested-by: Farrah Chen [Make the new function a stub for !CONFIG_KEXEC_CORE. - Paolo] Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/tdx.h | 6 ++++++ arch/x86/kvm/vmx/tdx.c | 10 ++++++++++ arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index c178360c1fb1..6120461bd5ff 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -228,5 +228,11 @@ static inline const char *tdx_dump_mce_info(struct mce= *m) { return NULL; } static inline const struct tdx_sys_info *tdx_get_sysinfo(void) { return NU= LL; } #endif /* CONFIG_INTEL_TDX_HOST */ =20 +#ifdef CONFIG_KEXEC_CORE +void tdx_cpu_flush_cache_for_kexec(void); +#else +static inline void tdx_cpu_flush_cache_for_kexec(void) { } +#endif + #endif /* !__ASSEMBLER__ */ #endif /* _ASM_X86_TDX_H */ diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index f457b2e578b2..04b6d332c1af 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -423,6 +423,16 @@ void tdx_disable_virtualization_cpu(void) tdx_flush_vp(&arg); } local_irq_restore(flags); + + /* + * Flush cache now if kexec is possible: this is necessary to avoid + * having dirty private memory cachelines when the new kernel boots, + * but WBINVD is a relatively expensive operation and doing it during + * kexec can exacerbate races in native_stop_other_cpus(). Do it + * now, since this is a safe moment and there is going to be no more + * TDX activity on this CPU from this point on. + */ + tdx_cpu_flush_cache_for_kexec(); } =20 #define TDX_SEAMCALL_RETRIES 10000 diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c index 2abf53ed59c8..330b560313af 100644 --- a/arch/x86/virt/vmx/tdx/tdx.c +++ b/arch/x86/virt/vmx/tdx/tdx.c @@ -1872,3 +1872,22 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct pag= e *page) return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args); } EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_hkid); + +#ifdef CONFIG_KEXEC_CORE +void tdx_cpu_flush_cache_for_kexec(void) +{ + lockdep_assert_preemption_disabled(); + + if (!this_cpu_read(cache_state_incoherent)) + return; + + /* + * Private memory cachelines need to be clean at the time of + * kexec. Write them back now, as the caller promises that + * there should be no more SEAMCALLs on this CPU. + */ + wbinvd(); + this_cpu_write(cache_state_incoherent, false); +} +EXPORT_SYMBOL_GPL(tdx_cpu_flush_cache_for_kexec); +#endif --=20 2.51.0