Touching privately mapped GPA that is not properly converted to private
with MapGPA and accepted leads to an unrecoverable exit to VMM.

load_unaligned_zeropad() can touch memory that is not owned by the
caller, but just happened to next after the owned memory.
This load_unaligned_zeropad() behaviour makes it important when kernel
asks VMM to convert a GPA from shared to private or back. Kernel must
never have a page mapped into direct mapping (and aliases) as private
when the GPA is already converted to shared or when GPA is not yet
converted to private.

load_unaligned_zeropad() can touch memory that is not owned by the
caller, but just happens to be next after the owned memory. This
load_unaligned_zeropad() behavior makes it important when the kernel
asks VMM to convert a GPA from shared to private or back. The kernel
must never have a page mapped into direct mapping (and aliases) as
private when the GPA is already converted to shared or when the GPA is
not yet converted to private.

guest.enc_status_change_prepare() is called before adjusting direct
mapping and therefore is responsible for converting the memory to
private.

guest.enc_status_change_finish() is called after adjusting direct
mapping and it converts the memory to shared.

It is okay to have a shared mapping of memory that is not properly
converted. handle_mmio() knows how to deal with load_unaligned_zeropad()
stepping on it.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Fixes: 7dbde7631629 ("x86/mm/cpa: Add support for TDX shared memory")
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 arch/x86/coco/tdx/tdx.c | 64 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 61 insertions(+), 3 deletions(-)

diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
index e146b599260f..f6213a10de3a 100644
--- a/arch/x86/coco/tdx/tdx.c
+++ b/arch/x86/coco/tdx/tdx.c
@@ -840,6 +840,30 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc)
 	return true;
 }
 
+static bool tdx_enc_status_change_prepare(unsigned long vaddr, int numpages,
+					  bool enc)
+{
+	/*
+	 * Only handle shared->private conversion here.
+	 * See the comment in tdx_early_init().
+	 */
+	if (enc)
+		return tdx_enc_status_changed(vaddr, numpages, enc);
+	return true;
+}
+
+static bool tdx_enc_status_change_finish(unsigned long vaddr, int numpages,
+					 bool enc)
+{
+	/*
+	 * Only handle private->shared conversion here.
+	 * See the comment in tdx_early_init().
+	 */
+	if (!enc)
+		return tdx_enc_status_changed(vaddr, numpages, enc);
+	return true;
+}
+
 void __init tdx_early_init(void)
 {
 	u64 cc_mask;
@@ -867,9 +891,43 @@ void __init tdx_early_init(void)
 	 */
 	physical_mask &= cc_mask - 1;
 
-	x86_platform.guest.enc_cache_flush_required = tdx_cache_flush_required;
-	x86_platform.guest.enc_tlb_flush_required   = tdx_tlb_flush_required;
-	x86_platform.guest.enc_status_change_finish = tdx_enc_status_changed;
+	/*
+	 * Touching privately mapped GPA that is not properly converted to
+	 * private with MapGPA and accepted leads to an unrecoverable exit
+	 * to VMM.
+	 *
+	 * load_unaligned_zeropad() can touch memory that is not owned by the
+	 * caller, but just happened to next after the owned memory.
+	 * This load_unaligned_zeropad() behaviour makes it important when
+	 * kernel asks VMM to convert a GPA from shared to private or back.
+	 * Kernel must never have a page mapped into direct mapping (and
+	 * aliases) as private when the GPA is already converted to shared or
+	 * when GPA is not yet converted to private.
+	 *
+	 * load_unaligned_zeropad() can touch memory that is not owned by the
+	 * caller, but just happens to be next after the owned memory. This
+	 * load_unaligned_zeropad() behavior makes it important when the kernel
+	 * asks VMM to convert a GPA from shared to private or back. The kernel
+	 * must never have a page mapped into direct mapping (and aliases) as
+	 * private when the GPA is already converted to shared or when the GPA
+	 * is not yet converted to private.
+	 *
+	 * guest.enc_status_change_prepare() is called before adjusting direct
+	 * mapping and therefore is responsible for converting the memory to
+	 * private.
+	 *
+	 * guest.enc_status_change_finish() is called after adjusting direct
+	 * mapping and it converts the memory to shared.
+	 *
+	 * It is okay to have a shared mapping of memory that is not properly
+	 * converted. handle_mmio() knows how to deal with
+	 * load_unaligned_zeropad() stepping on it.
+	 */
+	x86_platform.guest.enc_status_change_prepare = tdx_enc_status_change_prepare;
+	x86_platform.guest.enc_status_change_finish  = tdx_enc_status_change_finish;
+
+	x86_platform.guest.enc_cache_flush_required  = tdx_cache_flush_required;
+	x86_platform.guest.enc_tlb_flush_required    = tdx_tlb_flush_required;
 
 	pr_info("Guest detected\n");
 }
-- 
2.39.3

On 6/6/23 02:56, Kirill A. Shutemov wrote:
> load_unaligned_zeropad() can touch memory that is not owned by the
> caller, but just happened to next after the owned memory.
> This load_unaligned_zeropad() behaviour makes it important when kernel
> asks VMM to convert a GPA from shared to private or back. Kernel must
> never have a page mapped into direct mapping (and aliases) as private
> when the GPA is already converted to shared or when GPA is not yet
> converted to private.
> 
> load_unaligned_zeropad() can touch memory that is not owned by the
> caller, but just happens to be next after the owned memory. This
> load_unaligned_zeropad() behavior makes it important when the kernel
> asks VMM to convert a GPA from shared to private or back. The kernel
> must never have a page mapped into direct mapping (and aliases) as
> private when the GPA is already converted to shared or when the GPA is
> not yet converted to private.

Heh, that must be really important info to have it in the changelog twice!

I'll fix it up when I apply it.

On Tue, Jun 06, 2023 at 11:14:29AM -0700, Dave Hansen wrote:
> On 6/6/23 02:56, Kirill A. Shutemov wrote:
> > load_unaligned_zeropad() can touch memory that is not owned by the
> > caller, but just happened to next after the owned memory.
> > This load_unaligned_zeropad() behaviour makes it important when kernel
> > asks VMM to convert a GPA from shared to private or back. Kernel must
> > never have a page mapped into direct mapping (and aliases) as private
> > when the GPA is already converted to shared or when GPA is not yet
> > converted to private.
> > 
> > load_unaligned_zeropad() can touch memory that is not owned by the
> > caller, but just happens to be next after the owned memory. This
> > load_unaligned_zeropad() behavior makes it important when the kernel
> > asks VMM to convert a GPA from shared to private or back. The kernel
> > must never have a page mapped into direct mapping (and aliases) as
> > private when the GPA is already converted to shared or when the GPA is
> > not yet converted to private.
> 
> Heh, that must be really important info to have it in the changelog twice!
> 
> I'll fix it up when I apply it.

Ouch. Please fix the comment in the code too.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

On 6/6/23 11:37, Kirill A. Shutemov wrote:
>> Heh, that must be really important info to have it in the changelog twice!
>>
>> I'll fix it up when I apply it.
> Ouch. Please fix the comment in the code too.

I couldn't help myself and rewrote the changelog and comment.  Please
let me know if it looks OK:

> https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/tdx

On Tue, Jun 06, 2023 at 04:29:15PM -0700, Dave Hansen wrote:
> On 6/6/23 11:37, Kirill A. Shutemov wrote:
> >> Heh, that must be really important info to have it in the changelog twice!
> >>
> >> I'll fix it up when I apply it.
> > Ouch. Please fix the comment in the code too.
> 
> I couldn't help myself and rewrote the changelog and comment.  Please
> let me know if it looks OK:
> 
> > https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/tdx

Looks good. Thank you.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov