A single stubs page is initialized with 0xcc and re-used, with multiple
CPUs each using a portion of the shared page. In cpu_smpboot_free(),
each stubs area is checked against 0xcc. When all are set to 0xcc, the
page is freed.
Booting a system with smt=0, CPU0 is initially setup, allocating the
stubs page and initializing to 0xcc. When more CPUs are brought up,
CPU1 is initialized and then immediately brough offline as it is the
sibling of CPU0. Since the page was initially memset with 0xcc,
cpu_smpboot_free() finds all stubs as 0xcc and frees the page.
However, the page is still assigned to CPU0 and continues to be assigned
to other CPUs.
Meanwhile the page can be reallocated, which can lead to misbehavior.
The particular instance was the stubs page re-used as a page table which
later faulted when the entry was all 0xcc.
Change to initializing the page as 0xd6/STUB_BUF_FREE, and initializing
individual stubs as 0xcc/STUB_BUF_USED. 0xd6 now indicates unused, and
0xcc indicates used/assigned. When freeing a CPU, the stub is set to
0xd6, and the page is freed if all stubs are 0xd6. Initializing with
STUB_BUF_FREE lets cpu_smpboot_free() a page that was only ever
partially used.
0xd6/UDB is a 1 byte invalid opcode, which is similar to the existing
use of 0xcc. 0xd6 is used to identify bug frames, but the stub addr
(e.g. 0xffff82d07fffe000) fails the is_active_kernel_text() check. It
should be okay to use here.
Fixes: 7a66ac8d1633 ("x86: move syscall trampolines off the stack")
Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
---
It would be nice to use get_page()/put_page() to let count_info handle
reference counting, but they require an owning domain.
The listed Fixes introduced the use of 0xcc, but the smt commit may have
made it more problematic.
Fixes: d8f974f1a646 ("x86: command line option to avoid use of secondary hyper-threads")
---
xen/arch/x86/smpboot.c | 21 ++++++++++++++++++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c
index ff05955bae..a5d485b732 100644
--- a/xen/arch/x86/smpboot.c
+++ b/xen/arch/x86/smpboot.c
@@ -643,10 +643,16 @@ static int do_boot_cpu(int apicid, int cpu)
#define STUB_BUF_CPU_OFFS(cpu) (((cpu) & (STUBS_PER_PAGE - 1)) * STUB_BUF_SIZE)
+/* Fill values indicating state of stub. */
+#define STUB_BUF_USED 0xcc
+#define STUB_BUF_FREE 0xd6
+
unsigned long alloc_stub_page(unsigned int cpu, unsigned long *mfn)
{
+ unsigned char *stub_page;
unsigned long stub_va;
struct page_info *pg;
+ bool initialize = false;
BUILD_BUG_ON(STUBS_PER_PAGE & (STUBS_PER_PAGE - 1));
@@ -661,7 +667,7 @@ unsigned long alloc_stub_page(unsigned int cpu, unsigned long *mfn)
if ( !pg )
return 0;
- unmap_domain_page(memset(__map_domain_page(pg), 0xcc, PAGE_SIZE));
+ initialize = true;
}
stub_va = XEN_VIRT_END - FIXADDR_X_SIZE - (cpu + 1) * PAGE_SIZE;
@@ -675,6 +681,14 @@ unsigned long alloc_stub_page(unsigned int cpu, unsigned long *mfn)
else if ( !*mfn )
*mfn = mfn_x(page_to_mfn(pg));
+ stub_page = __map_domain_page(pg);
+ /* Newly allocated page is marked entirely unused. */
+ if ( initialize )
+ memset(stub_page, STUB_BUF_FREE, PAGE_SIZE);
+ /* Specific CPU is marked used. */
+ memset(stub_page + STUB_BUF_CPU_OFFS(cpu), STUB_BUF_USED, STUB_BUF_SIZE);
+ unmap_domain_page(stub_page);
+
return stub_va;
}
@@ -992,9 +1006,10 @@ static void cpu_smpboot_free(unsigned int cpu, bool remove)
unsigned char *stub_page = map_domain_page(mfn);
unsigned int i;
- memset(stub_page + STUB_BUF_CPU_OFFS(cpu), 0xcc, STUB_BUF_SIZE);
+ memset(stub_page + STUB_BUF_CPU_OFFS(cpu), STUB_BUF_FREE,
+ STUB_BUF_SIZE);
for ( i = 0; i < STUBS_PER_PAGE; ++i )
- if ( stub_page[i * STUB_BUF_SIZE] != 0xcc )
+ if ( stub_page[i * STUB_BUF_SIZE] != STUB_BUF_FREE )
break;
unmap_domain_page(stub_page);
destroy_xen_mappings(per_cpu(stubs.addr, cpu) & PAGE_MASK,
--
2.54.0On 26/05/2026 9:31 pm, Jason Andryuk wrote:
> A single stubs page is initialized with 0xcc and re-used, with multiple
> CPUs each using a portion of the shared page. In cpu_smpboot_free(),
> each stubs area is checked against 0xcc. When all are set to 0xcc, the
> page is freed.
>
> Booting a system with smt=0, CPU0 is initially setup, allocating the
> stubs page and initializing to 0xcc. When more CPUs are brought up,
> CPU1 is initialized and then immediately brough offline as it is the
> sibling of CPU0. Since the page was initially memset with 0xcc,
> cpu_smpboot_free() finds all stubs as 0xcc and frees the page.
> However, the page is still assigned to CPU0 and continues to be assigned
> to other CPUs.
It's more complicated than this.
With CONFIG_PV (and !opt_fred in 4.22 which is perhaps newer than you're
testing), the LSTAR and CSTAR stubs guarantee that the 0xcc's are
overwritten with real instructions.
In !CONFIG_PV, the 0xcc's only get overwritten by the exception recovery
selftests (CPU0 only, and gated on CONFIG_SELF_TESTS), and "complicated"
instructions in the emulator (which in your safety environment, you
likely have compiled out).
So, in your environment, I think you probably can exclude the stubs
entirely and trim even more LoC.
>
> Meanwhile the page can be reallocated, which can lead to misbehavior.
> The particular instance was the stubs page re-used as a page table which
> later faulted when the entry was all 0xcc.
>
> Change to initializing the page as 0xd6/STUB_BUF_FREE, and initializing
> individual stubs as 0xcc/STUB_BUF_USED. 0xd6 now indicates unused, and
> 0xcc indicates used/assigned. When freeing a CPU, the stub is set to
> 0xd6, and the page is freed if all stubs are 0xd6. Initializing with
> STUB_BUF_FREE lets cpu_smpboot_free() a page that was only ever
> partially used.
>
> 0xd6/UDB is a 1 byte invalid opcode, which is similar to the existing
> use of 0xcc. 0xd6 is used to identify bug frames, but the stub addr
> (e.g. 0xffff82d07fffe000) fails the is_active_kernel_text() check. It
> should be okay to use here.
>
> Fixes: 7a66ac8d1633 ("x86: move syscall trampolines off the stack")
> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
> ---
> It would be nice to use get_page()/put_page() to let count_info handle
> reference counting, but they require an owning domain.
>
> The listed Fixes introduced the use of 0xcc, but the smt commit may have
> made it more problematic.
> Fixes: d8f974f1a646 ("x86: command line option to avoid use of secondary hyper-threads")
Honestly, I dislike all of this "try to free data for going-offline
CPUs". It is both complex and a non-stop source of bugs for tantamount
to 0 benefit.
On x86, we must boot all CPUs we find in the MADT. You're seeing this
behaviour already. This is because if an #MC hits any group of CPUs
where any CR4.MCE=0, it's an instant reset.
For this reason, firmware doesn't hand APs over to the OS in the
Wait-for-SIPI state (which resets CR4 to 0); they're in MWAIT or IO-wait
typically these days, using firmware provided stacks. But firmware
cannot handle an #MC intended for the OS, so the OS must set up stacks
and at least an NMI and #MC handler even for those CPUs not wanting to run.
This what park_offline_cpus is trying to do, and while it's set for
Intel and clear for AMD, I'm pretty sure this is a bug on AMD because
you can still get MCEs with core-scope groups.
Beyond that, smt=0 is an emergency bodge for speculation safety, which
is always better done by changing SMT settings in the firmware.
xen-hptool is useful for testing but it's not a thing anyone uses in a
production system.
ACPI CPU hot-add does exist in virtual environments, but hot-remove is
theoretical at best. I've not seen any evidence of ACPI hotplug
actually working on Xen, and I think the chances that it does are slim;
it requires AML execution, and is right in the middle of the split-brain
problem with physical vs virtual details that dom0 suffers.
So, lets just allocate the stubs and "leak" them in testing scenarios.
It removes bugs and removes code, and has no effect on well-configured
systems (where cpu offline is not used in practice).
~Andrew
On 2026-05-26 18:03, Andrew Cooper wrote:
> On 26/05/2026 9:31 pm, Jason Andryuk wrote:
>> A single stubs page is initialized with 0xcc and re-used, with multiple
>> CPUs each using a portion of the shared page. In cpu_smpboot_free(),
>> each stubs area is checked against 0xcc. When all are set to 0xcc, the
>> page is freed.
>>
>> Booting a system with smt=0, CPU0 is initially setup, allocating the
>> stubs page and initializing to 0xcc. When more CPUs are brought up,
>> CPU1 is initialized and then immediately brough offline as it is the
>> sibling of CPU0. Since the page was initially memset with 0xcc,
>> cpu_smpboot_free() finds all stubs as 0xcc and frees the page.
>> However, the page is still assigned to CPU0 and continues to be assigned
>> to other CPUs.
>
> It's more complicated than this.
>
> With CONFIG_PV (and !opt_fred in 4.22 which is perhaps newer than you're
> testing), the LSTAR and CSTAR stubs guarantee that the 0xcc's are
> overwritten with real instructions.
>
> In !CONFIG_PV, the 0xcc's only get overwritten by the exception recovery
> selftests (CPU0 only, and gated on CONFIG_SELF_TESTS), and "complicated"
> instructions in the emulator (which in your safety environment, you
> likely have compiled out).
>
> So, in your environment, I think you probably can exclude the stubs
> entirely and trim even more LoC.
Thanks. Ok, my build was !CONFIG_PV, so 0xcc's were not overwritten.
The fault happened before the self tests ran.
>>
>> Meanwhile the page can be reallocated, which can lead to misbehavior.
>> The particular instance was the stubs page re-used as a page table which
>> later faulted when the entry was all 0xcc.
>>
>> Change to initializing the page as 0xd6/STUB_BUF_FREE, and initializing
>> individual stubs as 0xcc/STUB_BUF_USED. 0xd6 now indicates unused, and
>> 0xcc indicates used/assigned. When freeing a CPU, the stub is set to
>> 0xd6, and the page is freed if all stubs are 0xd6. Initializing with
>> STUB_BUF_FREE lets cpu_smpboot_free() a page that was only ever
>> partially used.
>>
>> 0xd6/UDB is a 1 byte invalid opcode, which is similar to the existing
>> use of 0xcc. 0xd6 is used to identify bug frames, but the stub addr
>> (e.g. 0xffff82d07fffe000) fails the is_active_kernel_text() check. It
>> should be okay to use here.
>>
>> Fixes: 7a66ac8d1633 ("x86: move syscall trampolines off the stack")
>> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
>> ---
>> It would be nice to use get_page()/put_page() to let count_info handle
>> reference counting, but they require an owning domain.
>>
>> The listed Fixes introduced the use of 0xcc, but the smt commit may have
>> made it more problematic.
>> Fixes: d8f974f1a646 ("x86: command line option to avoid use of secondary hyper-threads")
>
> Honestly, I dislike all of this "try to free data for going-offline
> CPUs". It is both complex and a non-stop source of bugs for tantamount
> to 0 benefit.
>
> On x86, we must boot all CPUs we find in the MADT. You're seeing this
> behaviour already. This is because if an #MC hits any group of CPUs
> where any CR4.MCE=0, it's an instant reset.
>
> For this reason, firmware doesn't hand APs over to the OS in the
> Wait-for-SIPI state (which resets CR4 to 0); they're in MWAIT or IO-wait
> typically these days, using firmware provided stacks. But firmware
> cannot handle an #MC intended for the OS, so the OS must set up stacks
> and at least an NMI and #MC handler even for those CPUs not wanting to run.
>
> This what park_offline_cpus is trying to do, and while it's set for
> Intel and clear for AMD, I'm pretty sure this is a bug on AMD because
> you can still get MCEs with core-scope groups.
>
>
> Beyond that, smt=0 is an emergency bodge for speculation safety, which
> is always better done by changing SMT settings in the firmware.
I was asked to check something with SMT disabled, and I could not find
SMT in my firmware. I looked for a while without success, and then set
smt=0 :/
> xen-hptool is useful for testing but it's not a thing anyone uses in a
> production system.
>
> ACPI CPU hot-add does exist in virtual environments, but hot-remove is
> theoretical at best. I've not seen any evidence of ACPI hotplug
> actually working on Xen, and I think the chances that it does are slim;
> it requires AML execution, and is right in the middle of the split-brain
> problem with physical vs virtual details that dom0 suffers.
>
>
> So, lets just allocate the stubs and "leak" them in testing scenarios.
> It removes bugs and removes code, and has no effect on well-configured
> systems (where cpu offline is not used in practice).
Ok.
Thanks,
Jason
On 2026-05-27 10:14, Jason Andryuk wrote: > On 2026-05-26 18:03, Andrew Cooper wrote: >> On 26/05/2026 9:31 pm, Jason Andryuk wrote: >>> A single stubs page is initialized with 0xcc and re-used, with multiple >>> CPUs each using a portion of the shared page. In cpu_smpboot_free(), >>> each stubs area is checked against 0xcc. When all are set to 0xcc, the >>> page is freed. >>> >>> Booting a system with smt=0, CPU0 is initially setup, allocating the >>> stubs page and initializing to 0xcc. When more CPUs are brought up, >>> CPU1 is initialized and then immediately brough offline as it is the >>> sibling of CPU0. Since the page was initially memset with 0xcc, >>> cpu_smpboot_free() finds all stubs as 0xcc and frees the page. >>> However, the page is still assigned to CPU0 and continues to be assigned >>> to other CPUs. >> >> It's more complicated than this. >> >> With CONFIG_PV (and !opt_fred in 4.22 which is perhaps newer than you're >> testing), the LSTAR and CSTAR stubs guarantee that the 0xcc's are >> overwritten with real instructions. >> >> In !CONFIG_PV, the 0xcc's only get overwritten by the exception recovery >> selftests (CPU0 only, and gated on CONFIG_SELF_TESTS), and "complicated" >> instructions in the emulator (which in your safety environment, you >> likely have compiled out). >> >> So, in your environment, I think you probably can exclude the stubs >> entirely and trim even more LoC. > > Thanks. Ok, my build was !CONFIG_PV, so 0xcc's were not overwritten. > The fault happened before the self tests ran. Correction: It was after the self tests ran and during dom0 construction. (XEN) Pagetable walk from ffff830842652008: (XEN) L4[0x106] = 8000000079c72063 ffffffffffffffff (XEN) L3[0x021] = 0000000079ff3063 ffffffffffffffff (XEN) L2[0x013] = 000000085680f063 ffffffffffffffff (XEN) L1[0x052] = cccccccccccccccc ffffffffffffffff It looks like the page is reallocated after free-ing, so after CPU1 is down. The re-use would write the page with PTEs. However, when later CPUs are brought down, their portion of the stubs page is overwritten with 0xcc. I think that is how the page, as a page table, is corrupted. Regards, Jason
© 2016 - 2026 Red Hat, Inc.