In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
accepted before use. The first access triggers a fatal "SEPT entry in
PENDING state" EPT violation and KVM terminates the guest.

Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
calls tdx_accept_memory() for the range being onlined.

The notifier returns NOTIFY_BAD on acceptance failure, preventing the
memory from going online.

Assisted-by: Claude:claude-opus-4-6
Reported-by: Chenyi Qiang <chenyi.qiang@intel.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
index 7b2833705d475..89f90bc303258 100644
--- a/arch/x86/coco/tdx/tdx.c
+++ b/arch/x86/coco/tdx/tdx.c
@@ -8,6 +8,7 @@
 #include <linux/export.h>
 #include <linux/io.h>
 #include <linux/kexec.h>
+#include <linux/memory.h>
 #include <asm/coco.h>
 #include <asm/tdx.h>
 #include <asm/vmx.h>
@@ -1194,3 +1195,40 @@ void __init tdx_early_init(void)
 
 	tdx_announce();
 }
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static int tdx_guest_memory_notifier(struct notifier_block *nb,
+				     unsigned long action, void *v)
+{
+	struct memory_notify *mn = v;
+	phys_addr_t start, end;
+
+	if (action != MEM_GOING_ONLINE)
+		return NOTIFY_OK;
+
+	start = PFN_PHYS(mn->start_pfn);
+	end = start + PFN_PHYS(mn->nr_pages);
+
+	if (!tdx_accept_memory(start, end)) {
+		pr_err("Failed to accept memory [0x%llx, 0x%llx)\n",
+		       (unsigned long long)start,
+		       (unsigned long long)end);
+		return NOTIFY_BAD;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block tdx_guest_memory_nb = {
+	.notifier_call = tdx_guest_memory_notifier,
+};
+
+static int __init tdx_guest_memory_init(void)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
+		return 0;
+
+	return register_memory_notifier(&tdx_guest_memory_nb);
+}
+core_initcall(tdx_guest_memory_init);
+#endif

-- 
2.53.0

On Tue, Mar 24, 2026 at 07:21:48PM +0400, Marc-André Lureau wrote:
> In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> accepted before use. The first access triggers a fatal "SEPT entry in
> PENDING state" EPT violation and KVM terminates the guest.
> 
> Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> calls tdx_accept_memory() for the range being onlined.
> 
> The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> memory from going online.
> 
> Assisted-by: Claude:claude-opus-4-6
> Reported-by: Chenyi Qiang <chenyi.qiang@intel.com>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
> 
> diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
> index 7b2833705d475..89f90bc303258 100644
> --- a/arch/x86/coco/tdx/tdx.c
> +++ b/arch/x86/coco/tdx/tdx.c
> @@ -8,6 +8,7 @@
>  #include <linux/export.h>
>  #include <linux/io.h>
>  #include <linux/kexec.h>
> +#include <linux/memory.h>
>  #include <asm/coco.h>
>  #include <asm/tdx.h>
>  #include <asm/vmx.h>
> @@ -1194,3 +1195,40 @@ void __init tdx_early_init(void)
>  
>  	tdx_announce();
>  }
> +
> +#ifdef CONFIG_MEMORY_HOTPLUG
> +static int tdx_guest_memory_notifier(struct notifier_block *nb,
> +				     unsigned long action, void *v)
> +{
> +	struct memory_notify *mn = v;
> +	phys_addr_t start, end;
> +
> +	if (action != MEM_GOING_ONLINE)
> +		return NOTIFY_OK;
> +
> +	start = PFN_PHYS(mn->start_pfn);
> +	end = start + PFN_PHYS(mn->nr_pages);
> +
> +	if (!tdx_accept_memory(start, end)) {
> +		pr_err("Failed to accept memory [0x%llx, 0x%llx)\n",
> +		       (unsigned long long)start,
> +		       (unsigned long long)end);
> +		return NOTIFY_BAD;
> +	}
> +
> +	return NOTIFY_OK;
> +}
> +
> +static struct notifier_block tdx_guest_memory_nb = {
> +	.notifier_call = tdx_guest_memory_notifier,
> +};
> +
> +static int __init tdx_guest_memory_init(void)
> +{
> +	if (!cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
> +		return 0;
> +
> +	return register_memory_notifier(&tdx_guest_memory_nb);
> +}
If I read the code correctly,

online_pages
  1. memory_notify(MEM_GOING_ONLINE, &mem_arg);
  2. online_pages_range(pfn, nr_pages); 
       (*online_page_callback)(page, order);
          generic_online_page
	      __free_pages_core(page, order, MEMINIT_HOTPLUG);

In __free_pages_core(), there's accept_memory() already:

    if (page_contains_unaccepted(page, order)) {
          if (order == MAX_PAGE_ORDER && __free_unaccepted(page))
               return;

         accept_memory(page_to_phys(page), PAGE_SIZE << order);
    }

__free_unaccepted() also adds the pages to the unaccepted_pages list, so
cond_accept_memory() will accept the memory later:

So, is it because the virtio mem sets online_page_callback to
virtio_mem_online_page_cb, which doesn't invoke __free_pages_core() properly?

Or am I missing something that makes the memory notifier approach necessary?

Thanks
Yan

> +core_initcall(tdx_guest_memory_init);
> +#endif
> 
> -- 
> 2.53.0
> 
> 

Hi

On Fri, Mar 27, 2026 at 1:09 PM Yan Zhao <yan.y.zhao@intel.com> wrote:
>
> On Tue, Mar 24, 2026 at 07:21:48PM +0400, Marc-André Lureau wrote:
> > In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> > accepted before use. The first access triggers a fatal "SEPT entry in
> > PENDING state" EPT violation and KVM terminates the guest.
> >
> > Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> > calls tdx_accept_memory() for the range being onlined.
> >
> > The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> > memory from going online.
> >
> > Assisted-by: Claude:claude-opus-4-6
> > Reported-by: Chenyi Qiang <chenyi.qiang@intel.com>
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 38 insertions(+)
> >
> > diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
> > index 7b2833705d475..89f90bc303258 100644
> > --- a/arch/x86/coco/tdx/tdx.c
> > +++ b/arch/x86/coco/tdx/tdx.c
> > @@ -8,6 +8,7 @@
> >  #include <linux/export.h>
> >  #include <linux/io.h>
> >  #include <linux/kexec.h>
> > +#include <linux/memory.h>
> >  #include <asm/coco.h>
> >  #include <asm/tdx.h>
> >  #include <asm/vmx.h>
> > @@ -1194,3 +1195,40 @@ void __init tdx_early_init(void)
> >
> >       tdx_announce();
> >  }
> > +
> > +#ifdef CONFIG_MEMORY_HOTPLUG
> > +static int tdx_guest_memory_notifier(struct notifier_block *nb,
> > +                                  unsigned long action, void *v)
> > +{
> > +     struct memory_notify *mn = v;
> > +     phys_addr_t start, end;
> > +
> > +     if (action != MEM_GOING_ONLINE)
> > +             return NOTIFY_OK;
> > +
> > +     start = PFN_PHYS(mn->start_pfn);
> > +     end = start + PFN_PHYS(mn->nr_pages);
> > +
> > +     if (!tdx_accept_memory(start, end)) {
> > +             pr_err("Failed to accept memory [0x%llx, 0x%llx)\n",
> > +                    (unsigned long long)start,
> > +                    (unsigned long long)end);
> > +             return NOTIFY_BAD;
> > +     }
> > +
> > +     return NOTIFY_OK;
> > +}
> > +
> > +static struct notifier_block tdx_guest_memory_nb = {
> > +     .notifier_call = tdx_guest_memory_notifier,
> > +};
> > +
> > +static int __init tdx_guest_memory_init(void)
> > +{
> > +     if (!cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
> > +             return 0;
> > +
> > +     return register_memory_notifier(&tdx_guest_memory_nb);
> > +}
> If I read the code correctly,
>
> online_pages
>   1. memory_notify(MEM_GOING_ONLINE, &mem_arg);
>   2. online_pages_range(pfn, nr_pages);
>        (*online_page_callback)(page, order);
>           generic_online_page
>               __free_pages_core(page, order, MEMINIT_HOTPLUG);
>
> In __free_pages_core(), there's accept_memory() already:
>
>     if (page_contains_unaccepted(page, order)) {
>           if (order == MAX_PAGE_ORDER && __free_unaccepted(page))
>                return;
>
>          accept_memory(page_to_phys(page), PAGE_SIZE << order);
>     }
>
> __free_unaccepted() also adds the pages to the unaccepted_pages list, so
> cond_accept_memory() will accept the memory later:
>
> So, is it because the virtio mem sets online_page_callback to
> virtio_mem_online_page_cb, which doesn't invoke __free_pages_core() properly?
>
> Or am I missing something that makes the memory notifier approach necessary?

virtio-mem doesn't modify efi_unaccepted_memory bitmap (populated by
TDVF code when the VM is started)

On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> accepted before use. The first access triggers a fatal "SEPT entry in
> PENDING state" EPT violation and KVM terminates the guest.
> 
> Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> calls tdx_accept_memory() for the range being onlined.
> 
> The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> memory from going online.

Does this depend on patch 1 somehow?

Hi

On Wed, Mar 25, 2026 at 2:04 AM Edgecombe, Rick P
<rick.p.edgecombe@intel.com> wrote:
>
> On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> > In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> > accepted before use. The first access triggers a fatal "SEPT entry in
> > PENDING state" EPT violation and KVM terminates the guest.
> >
> > Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> > calls tdx_accept_memory() for the range being onlined.
> >
> > The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> > memory from going online.
>
> Does this depend on patch 1 somehow?

Yes, if I plug, unplug and plug again I get this without PATCH 1:
[root@rhel10-server ~]# [ 5707.392231] virtio_mem virtio5: plugged
size: 0x80000000
[ 5707.395583] virtio_mem virtio5: requested size: 0x0

[root@rhel10-server ~]# [ 5714.648501] virtio_mem virtio5: plugged
size: 0x2e00000
[ 5714.651808] virtio_mem virtio5: requested size: 0x80000000
[ 5714.676296] tdx: Failed to accept memory [0x108000000, 0x110000000)
[ 5714.683980] tdx: Failed to accept memory [0x110000000, 0x118000000)
[ 5714.686997] tdx: Failed to accept memory [0x140000000, 0x148000000)
[ 5714.689989] tdx: Failed to accept memory [0x128000000, 0x130000000)
[ 5714.694981] tdx: Failed to accept memory [0x148000000, 0x150000000)
[ 5714.704064] tdx: Failed to accept memory [0x138000000, 0x140000000)
[ 5714.710144] tdx: Failed to accept memory [0x118000000, 0x120000000)
[ 5714.722532] tdx: Failed to accept memory [0x130000000, 0x138000000)

My understanding is that QEMU should eventually unplug the memory and
PUNCH_HOLE then KVM should TDH.MEM.PAGE.REMOVE, but that doesn't seem
to happen. Is this strictly required? According to the specification,
it may not be.

On 3/25/2026 6:29 PM, Marc-André Lureau wrote:
> Hi
> 
> On Wed, Mar 25, 2026 at 2:04 AM Edgecombe, Rick P
> <rick.p.edgecombe@intel.com> wrote:
>>
>> On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
>>> In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
>>> accepted before use. The first access triggers a fatal "SEPT entry in
>>> PENDING state" EPT violation and KVM terminates the guest.
>>>
>>> Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
>>> calls tdx_accept_memory() for the range being onlined.
>>>
>>> The notifier returns NOTIFY_BAD on acceptance failure, preventing the
>>> memory from going online.
>>
>> Does this depend on patch 1 somehow?
> 
> Yes, if I plug, unplug and plug again I get this without PATCH 1:
> [root@rhel10-server ~]# [ 5707.392231] virtio_mem virtio5: plugged
> size: 0x80000000
> [ 5707.395583] virtio_mem virtio5: requested size: 0x0
> 
> [root@rhel10-server ~]# [ 5714.648501] virtio_mem virtio5: plugged
> size: 0x2e00000
> [ 5714.651808] virtio_mem virtio5: requested size: 0x80000000
> [ 5714.676296] tdx: Failed to accept memory [0x108000000, 0x110000000)
> [ 5714.683980] tdx: Failed to accept memory [0x110000000, 0x118000000)
> [ 5714.686997] tdx: Failed to accept memory [0x140000000, 0x148000000)
> [ 5714.689989] tdx: Failed to accept memory [0x128000000, 0x130000000)
> [ 5714.694981] tdx: Failed to accept memory [0x148000000, 0x150000000)
> [ 5714.704064] tdx: Failed to accept memory [0x138000000, 0x140000000)
> [ 5714.710144] tdx: Failed to accept memory [0x118000000, 0x120000000)
> [ 5714.722532] tdx: Failed to accept memory [0x130000000, 0x138000000)
> 
> My understanding is that QEMU should eventually unplug the memory and
> PUNCH_HOLE then KVM should TDH.MEM.PAGE.REMOVE, but that doesn't seem
> to happen. 

I guess it doesn't happen because virtio-mem in QEMU only PUNCH_HOLE the
shared memory by ram_block_discard_range() but it doesn't touch the private
memory which should be discarded by ram_block_discard_guest_memfd_range().

Is this strictly required? According to the specification,
> it may not be.
> 
> 

On 3/27/26 04:05, Chenyi Qiang wrote:
> 
> 
> On 3/25/2026 6:29 PM, Marc-André Lureau wrote:
>> Hi
>>
>> On Wed, Mar 25, 2026 at 2:04 AM Edgecombe, Rick P
>> <rick.p.edgecombe@intel.com> wrote:
>>>
>>>
>>> Does this depend on patch 1 somehow?
>>
>> Yes, if I plug, unplug and plug again I get this without PATCH 1:
>> [root@rhel10-server ~]# [ 5707.392231] virtio_mem virtio5: plugged
>> size: 0x80000000
>> [ 5707.395583] virtio_mem virtio5: requested size: 0x0
>>
>> [root@rhel10-server ~]# [ 5714.648501] virtio_mem virtio5: plugged
>> size: 0x2e00000
>> [ 5714.651808] virtio_mem virtio5: requested size: 0x80000000
>> [ 5714.676296] tdx: Failed to accept memory [0x108000000, 0x110000000)
>> [ 5714.683980] tdx: Failed to accept memory [0x110000000, 0x118000000)
>> [ 5714.686997] tdx: Failed to accept memory [0x140000000, 0x148000000)
>> [ 5714.689989] tdx: Failed to accept memory [0x128000000, 0x130000000)
>> [ 5714.694981] tdx: Failed to accept memory [0x148000000, 0x150000000)
>> [ 5714.704064] tdx: Failed to accept memory [0x138000000, 0x140000000)
>> [ 5714.710144] tdx: Failed to accept memory [0x118000000, 0x120000000)
>> [ 5714.722532] tdx: Failed to accept memory [0x130000000, 0x138000000)
>>
>> My understanding is that QEMU should eventually unplug the memory and
>> PUNCH_HOLE then KVM should TDH.MEM.PAGE.REMOVE, but that doesn't seem
>> to happen. 
> 
> I guess it doesn't happen because virtio-mem in QEMU only PUNCH_HOLE the
> shared memory by ram_block_discard_range() but it doesn't touch the private
> memory which should be discarded by ram_block_discard_guest_memfd_range().
> 
> Is this strictly required? According to the specification,

So far nobody specified how virtio-mem should behave in a CoCo environment.

I assume that we need enhancements on the driver and the device side.

In Linux, we should not be accepting memory during memory
onlining/offlining through notifiers, as we might only hot(un)plug parts
of a memory block etc.

We need some explicit calls into the core before we hand hotplugged
memory to the core, and before we hand back unplugged memory to the device.

In QEMU, I would similarly assume that we might have to perform some
additional work when converting memory blocks. *maybe* that would just
be done by the guest that converts memory from private to shared before
unplug etc.

-- 
Cheers,

David

On Wed, 2026-03-25 at 14:29 +0400, Marc-André Lureau wrote:
> > Does this depend on patch 1 somehow?
> 
> Yes, if I plug, unplug and plug again I get this without PATCH 1:
> [root@rhel10-server ~]# [ 5707.392231] virtio_mem virtio5: plugged
> size: 0x80000000
> [ 5707.395583] virtio_mem virtio5: requested size: 0x0
> 
> [root@rhel10-server ~]# [ 5714.648501] virtio_mem virtio5: plugged
> size: 0x2e00000
> [ 5714.651808] virtio_mem virtio5: requested size: 0x80000000
> [ 5714.676296] tdx: Failed to accept memory [0x108000000,
> 0x110000000)
> [ 5714.683980] tdx: Failed to accept memory [0x110000000,
> 0x118000000)
> [ 5714.686997] tdx: Failed to accept memory [0x140000000,
> 0x148000000)
> [ 5714.689989] tdx: Failed to accept memory [0x128000000,
> 0x130000000)
> [ 5714.694981] tdx: Failed to accept memory [0x148000000,
> 0x150000000)
> [ 5714.704064] tdx: Failed to accept memory [0x138000000,
> 0x140000000)
> [ 5714.710144] tdx: Failed to accept memory [0x118000000,
> 0x120000000)
> [ 5714.722532] tdx: Failed to accept memory [0x130000000,
> 0x138000000)
> 
> My understanding is that QEMU should eventually unplug the memory and
> PUNCH_HOLE then KVM should TDH.MEM.PAGE.REMOVE, but that doesn't seem
> to happen. Is this strictly required? According to the specification,
> it may not be.

Ah, I see now! So the problem is not that the kernel is accidentally
re-accepting the memory. It's that host userspace is not actually
removing the memory during unplug. Hmm. Why not fix userspace then? If
the memory is unplugged it should not be usable anymore by the guest.
If it is still accessible then it seems kind of like a bug, no?

And! This totally justifies the warning. If the error is ignored, the
guest would think the memory is zeroed, but it could have old data in
it. It's exactly the kind of tricks a VMM could play to attack the
guest.

Another option could be to perform a TDG.MEM.PAGE.RELEASE TDCALL from
the guest when it unplugs the memory, to put it in an unaccepted state.
This would be more robust to buggy VMM behavior. But working around
buggy VM behavior would need a high bar.

Il mer 25 mar 2026, 18:21 Edgecombe, Rick P
<rick.p.edgecombe@intel.com> ha scritto:
>
> Ah, I see now! So the problem is not that the kernel is accidentally
> re-accepting the memory. It's that host userspace is not actually
> removing the memory during unplug. Hmm. Why not fix userspace then? If
> the memory is unplugged it should not be usable anymore by the guest.
> If it is still accessible then it seems kind of like a bug, no?
>
> And! This totally justifies the warning. If the error is ignored, the
> guest would think the memory is zeroed, but it could have old data in
> it. It's exactly the kind of tricks a VMM could play to attack the
> guest.
>
> Another option could be to perform a TDG.MEM.PAGE.RELEASE TDCALL from
> the guest when it unplugs the memory, to put it in an unaccepted state.
> This would be more robust to buggy VMM behavior. But working around
> buggy VM behavior would need a high bar.

Wouldn't it actually be a very low bar? Just from these two paragraphs
of yours, it's clear that the line between buggy and malicious is
fine, in fact I think userspace should not care at all about removing
the memory. Only the guest cares about acceptance state.

Doing a RELEASE TDCALL seems more robust and not hard.

Paolo

Hi Paolo!

On Thu, 2026-03-26 at 19:25 +0100, Paolo Bonzini wrote:
> > Another option could be to perform a TDG.MEM.PAGE.RELEASE TDCALL from
> > the guest when it unplugs the memory, to put it in an unaccepted state.
> > This would be more robust to buggy VMM behavior. But working around
> > buggy VM behavior would need a high bar.
> 
> Wouldn't it actually be a very low bar? Just from these two paragraphs
> of yours, it's clear that the line between buggy and malicious is
> fine, in fact I think userspace should not care at all about removing
> the memory. Only the guest cares about acceptance state.
> 
> Doing a RELEASE TDCALL seems more robust and not hard.

I mean I guess the contract is a bit fuzzy. The reason why I was thinking it was
a host userspace bug is because the conventional bare metal behavior of
unplugging memory should be that it is no longer accessible, right? If the guest
could still use the unplugged memory, it could be surprising for userspace and
the guest. Also, ideally I'd think the behavior wouldn't cover up guest bugs
where it tried to keep using the memory. So forgetting about TDX, isn't it
better behavior in general for unplugging memory, to actually pull it from the
guest? Did I look at that wrong?

As for the bar to change the guest, I was first imagining it would be the size
of the accept memory plumbing. Which was not a small effort and has had a steady
stream of bugs to squash where the accept was missed.

But I didn't actually POC anything to check the scope so maybe that was a bit
hasty. Should we do a POC? But considering the scope, I wonder if SNP has the
same problem.

On 3/26/26 4:40 PM, Edgecombe, Rick P wrote:
> Hi Paolo!
> 
> On Thu, 2026-03-26 at 19:25 +0100, Paolo Bonzini wrote:
>>> Another option could be to perform a TDG.MEM.PAGE.RELEASE TDCALL from
>>> the guest when it unplugs the memory, to put it in an unaccepted state.
>>> This would be more robust to buggy VMM behavior. But working around
>>> buggy VM behavior would need a high bar.
>>
>> Wouldn't it actually be a very low bar? Just from these two paragraphs
>> of yours, it's clear that the line between buggy and malicious is
>> fine, in fact I think userspace should not care at all about removing
>> the memory. Only the guest cares about acceptance state.
>>
>> Doing a RELEASE TDCALL seems more robust and not hard.
> 
> I mean I guess the contract is a bit fuzzy. The reason why I was thinking it was
> a host userspace bug is because the conventional bare metal behavior of
> unplugging memory should be that it is no longer accessible, right? If the guest
> could still use the unplugged memory, it could be surprising for userspace and
> the guest. Also, ideally I'd think the behavior wouldn't cover up guest bugs
> where it tried to keep using the memory. So forgetting about TDX, isn't it
> better behavior in general for unplugging memory, to actually pull it from the
> guest? Did I look at that wrong?
> 
> As for the bar to change the guest, I was first imagining it would be the size
> of the accept memory plumbing. Which was not a small effort and has had a steady
> stream of bugs to squash where the accept was missed.
> 
> But I didn't actually POC anything to check the scope so maybe that was a bit
> hasty. Should we do a POC? But considering the scope, I wonder if SNP has the
> same problem.

SNP likely has an analogous issue too.
Failing to switch states on remove will cause that RMP entry to remain
validated. A malicious hypervisor could then remap this GPA to another HPA
which would put this in the Guest-Invalid state. On re-hotplug if we ignore
errors suggested by Patch 1 (in our case that'd be PVALIDATE_FAIL_NOUPDATE
error likely), we could have two RMP entries for the same GPA and both being
validated. This is dangerous because hypervisor could swap these at will.

Would it not be better to have this information in the unaccepted bitmap which
we could explicitly query to accept/unaccept?

For ACPI hardware-style hotplug I was working with the UEFI side on a POC to
reflect SRAT hotplug windows in UEFI_UNACCEPTED_MEMORY using
EFI_MEMORY_HOT_PLUGGABLE attribute and working to modify that spec. I’m less
sure what this description for virtio-mem would look like and if it'd be
possible to do this early-boot.

Thanks,
--Pratik

On Mon, 2026-03-30 at 11:10 -0400, Pratik R. Sampat wrote:
> SNP likely has an analogous issue too.
> Failing to switch states on remove will cause that RMP entry to
> remain validated. A malicious hypervisor could then remap this GPA to
> another HPA which would put this in the Guest-Invalid state. On re-
> hotplug if we ignore errors suggested by Patch 1 (in our case that'd
> be PVALIDATE_FAIL_NOUPDATE error likely), we could have two RMP
> entries for the same GPA and both being validated. This is dangerous
> because hypervisor could swap these at will.

Oh, I was just wondering if we could just zero the page on accept
failure for the case of already accepted. Handle the issue internally
and actually go back to something like patch 1. Will it work for SNP?

> 
> Would it not be better to have this information in the unaccepted
> bitmap which we could explicitly query to accept/unaccept?

It makes me think about shared memory too. Should the unplug event also
signal the host to reset the memory to private? If the VMM is actually
not adjusting the guest mapping for a unplug/re-plug then the memory
would come back as shared.

But it really starts to feel like work the host should be doing.

> 
> For ACPI hardware-style hotplug I was working with the UEFI side on a
> POC to reflect SRAT hotplug windows in UEFI_UNACCEPTED_MEMORY using
> EFI_MEMORY_HOT_PLUGGABLE attribute and working to modify that spec.
> I’m less sure what this description for virtio-mem would look like
> and if it'd be possible to do this early-boot.

> On Mon, 2026-03-30 at 11:10 -0400, Pratik R. Sampat wrote:
> > SNP likely has an analogous issue too.
> > Failing to switch states on remove will cause that RMP entry to
> > remain validated. A malicious hypervisor could then remap this GPA to
> > another HPA which would put this in the Guest-Invalid state. On re-
> > hotplug if we ignore errors suggested by Patch 1 (in our case that'd
> > be PVALIDATE_FAIL_NOUPDATE error likely), we could have two RMP
> > entries for the same GPA and both being validated. This is dangerous
> > because hypervisor could swap these at will.
> 
> Oh, I was just wondering if we could just zero the page on accept
> failure for the case of already accepted. Handle the issue internally
> and actually go back to something like patch 1. Will it work for SNP?

I don't know about SNP, but if you are proposing to zero the page on
double acceptance, this is not great from security pov. It creates a
predictable behaviour primitive for the host to zero any data inside
the confidential guest and it can be misused (think of zeroing out a
page containing a cryptographic key). 

On Thu, 2026-04-02 at 08:18 +0000, Reshetova, Elena wrote:
> > Oh, I was just wondering if we could just zero the page on accept
> > failure for the case of already accepted. Handle the issue
> > internally
> > and actually go back to something like patch 1. Will it work for
> > SNP?
> 
> I don't know about SNP, but if you are proposing to zero the page on
> double acceptance, this is not great from security pov.

Accept does zero the memory already. So the guest side operation is
doing an operation that says "make this memory usable in an known state
of zeros". And the operation complies. What is the difference?

>  It creates a
> predictable behaviour primitive for the host to zero any data inside
> the confidential guest and it can be misused (think of zeroing out a
> page containing a cryptographic key).

If the host can trigger an accept somehow in the guest (via something
like this or other issue), then the host can also remove, then AUG the
page from the S-EPT. This will result in a normal accept which also
zeros the page.

So the part about whether a triggered accept succeeds or returns an
already accepted error is already under the control of the host. I.e.,
if we don't have the zeroing behavior, the host can already cause the
page to get zeroed. So I don't think anything is regressed. Both come
down to how careful the guest is about what it accepts.

> On Thu, 2026-04-02 at 08:18 +0000, Reshetova, Elena wrote:
> > > Oh, I was just wondering if we could just zero the page on accept
> > > failure for the case of already accepted. Handle the issue
> > > internally
> > > and actually go back to something like patch 1. Will it work for
> > > SNP?
> >
> > I don't know about SNP, but if you are proposing to zero the page on
> > double acceptance, this is not great from security pov.
> 
> Accept does zero the memory already. So the guest side operation is
> doing an operation that says "make this memory usable in an known state
> of zeros". And the operation complies. What is the difference?

The difference is that you do it in a re-accept case. 

> 
> >  It creates a
> > predictable behaviour primitive for the host to zero any data inside
> > the confidential guest and it can be misused (think of zeroing out a
> > page containing a cryptographic key).
> 
> If the host can trigger an accept somehow in the guest (via something
> like this or other issue), then the host can also remove, then AUG the
> page from the S-EPT. This will result in a normal accept which also
> zeros the page.

Yes, that's why the guest currently does not allow accepting a page that
has already been accepted.  
> 
> So the part about whether a triggered accept succeeds or returns an
> already accepted error is already under the control of the host. I.e.,
> if we don't have the zeroing behavior, the host can already cause the
> page to get zeroed. So I don't think anything is regressed. Both come
> down to how careful the guest is about what it accepts.

Yes, and my point is that we should not allow guest to freely double
accepting ever. 
For any use case that requires releasing memory and accepting it back,
it should be explicit action by the guest to track that memory has been
"released" (under correct and safe conditions) and then it is ok to accept
it back (even if it doesnt mean physically accepting it) and in this case it
is ok (and even strongly desired) to zero the page to simulate the normal
accept behaviour.  

On Fri, 2026-04-03 at 10:37 +0000, Reshetova, Elena wrote:
> > > > So the part about whether a triggered accept succeeds or returns an
> > > > already accepted error is already under the control of the host. > >
> > > > I.e., if we don't have the zeroing behavior, the host can already > >
> > > > cause the page to get zeroed. So I don't think anything is > >
> > > > regressed. Both come down to how careful the guest is about what it > >
> > > > accepts.
> > 
> > Yes, and my point is that we should not allow guest to freely double
> > accepting ever. 
> > For any use case that requires releasing memory and accepting it > back, it
> > should be explicit action by the guest to track that memory > has been
> > "released" (under correct and safe conditions) and then it > is ok to accept
> > it back (even if it doesnt mean physically accepting > it) and in this case
> > it is ok (and even strongly desired) to zero the > page to simulate the
> > normal accept behaviour. 

Hmm, it doesn't seem like you engaged with my point. Or at least I'm not
following what is exposed?

So I'm going to assume you agree that this procedure would not open up any
specific new capabilities for the host that don't exist today. And instead you
are just saying that the guest should have infrastructure to not double accept
memory in the first place.

But the problem here is not that the guest losing track of the accept state
actually. It is that the guest relies on the host to actually zap the S-EPT
before re-plugging memory at the same physical address space. So the guest is
tracking that the memory is released correctly. Better tracking will not help.
It relies on host behavior to not hit a double accept.

TDX connect will use this "unaccept" seamcall, so I asked Zhenzhong (Cced) how
much of what we need for that solution will just get added for TDX connect
anyway. It seems like we should make sure the same solution will work for both
SNP and TDX and keep the options open at this stage.

On Wed, 2026-04-01 at 08:39 -0700, Edgecombe, Rick P wrote:
> > Would it not be better to have this information in the unaccepted
> > bitmap which we could explicitly query to accept/unaccept?
> 
> It makes me think about shared memory too. Should the unplug event
> also signal the host to reset the memory to private? If the VMM is
> actually not adjusting the guest mapping for a unplug/re-plug then
> the memory would come back as shared.
> 
> But it really starts to feel like work the host should be doing.

Although if memory was able to be unplugged, the Linux guest would have
reset the memory to private when it was done with the memory. But from
a interface perspective, it seems weird to require the guest to signal
this because private/shared state is ultimately controlled by the host.
Unlike accept state where the guest gets some say.

On Thu, Mar 26, 2026 at 08:40:06PM +0000, Edgecombe, Rick P wrote:
> Hi Paolo!
> 
> On Thu, 2026-03-26 at 19:25 +0100, Paolo Bonzini wrote:
> > > Another option could be to perform a TDG.MEM.PAGE.RELEASE TDCALL from
> > > the guest when it unplugs the memory, to put it in an unaccepted state.
> > > This would be more robust to buggy VMM behavior. But working around
> > > buggy VM behavior would need a high bar.
> > 
> > Wouldn't it actually be a very low bar? Just from these two paragraphs
> > of yours, it's clear that the line between buggy and malicious is
> > fine, in fact I think userspace should not care at all about removing
> > the memory. Only the guest cares about acceptance state.
> > 
> > Doing a RELEASE TDCALL seems more robust and not hard.
> 
> I mean I guess the contract is a bit fuzzy. The reason why I was thinking it was
> a host userspace bug is because the conventional bare metal behavior of
> unplugging memory should be that it is no longer accessible, right? If the guest
> could still use the unplugged memory, it could be surprising for userspace and
> the guest. Also, ideally I'd think the behavior wouldn't cover up guest bugs
> where it tried to keep using the memory. So forgetting about TDX, isn't it
> better behavior in general for unplugging memory, to actually pull it from the
> guest? Did I look at that wrong?
> 
> As for the bar to change the guest, I was first imagining it would be the size
> of the accept memory plumbing. Which was not a small effort and has had a steady
> stream of bugs to squash where the accept was missed.
> 
> But I didn't actually POC anything to check the scope so maybe that was a bit
> hasty. Should we do a POC? But considering the scope, I wonder if SNP has the
> same problem.

Doing RELEASE will be required with TDX Connect in the picture.
Otherwise userspace wouldn't be able to pull the memory out of TD.
So, let's do it and drop the first patch.

We can suggest that userspace actually remove the memory, but I don't
think it should be part of the contract. Userspace might have a reasons
to keep the memory around.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov