tboot_shutdown() calls into tboot to perform the actual system shutdown.
tboot isn't built with endbr annotations, and Xen has CET-IBT enabled on
newer hardware.  shutdown_entry isn't annotated with endbr and Xen
faults:

Panic on CPU 0:
CONTROL-FLOW PROTECTION FAULT: #CP[0003] endbranch

And Xen hangs at this point.

Disabling CET-IBT let Xen and tboot power off, but reboot was
perfoming a poweroff instead of a warm reboot.  Disabling all of CET,
i.e. shadow stacks as well, lets tboot reboot properly.

Fixes: cdbe2b0a1aec ("x86: Enable CET Indirect Branch Tracking")
Signed-off-by: Jason Andryuk <jandryuk@gmail.com>
---
Without this fix, Xen subsequently hangs:

Reboot in five seconds...
[VT-D] IOMMU1: QI wait descriptor taking too long
 IQA = 484897000
 IQH = 0
 IQT = 820

with no futher output.
---
 xen/arch/x86/tboot.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xen/arch/x86/tboot.c b/xen/arch/x86/tboot.c
index 90f6e805a9..86c4c22cac 100644
--- a/xen/arch/x86/tboot.c
+++ b/xen/arch/x86/tboot.c
@@ -353,6 +353,16 @@ void tboot_shutdown(uint32_t shutdown_type)
         tboot_gen_xenheap_integrity(g_tboot_shared->s3_key, &xenheap_mac);
     }
 
+    /*
+     * Disable CET - tboot may not be built with endbr, and it doesn't support
+     * shadow stacks.
+     */
+    if ( read_cr4() & X86_CR4_CET )
+    {
+        wrmsrl(MSR_S_CET, 0);
+        write_cr4(read_cr4() & ~X86_CR4_CET);
+    }
+
     /*
      * During early boot, we can be called by panic before idle_vcpu[0] is
      * setup, but in that case we don't need to change page tables.
-- 
2.41.0

On 8/15/23 12:11, Jason Andryuk wrote:
> tboot_shutdown() calls into tboot to perform the actual system shutdown.
> tboot isn't built with endbr annotations, and Xen has CET-IBT enabled on
> newer hardware.  shutdown_entry isn't annotated with endbr and Xen
> faults:
> 
> Panic on CPU 0:
> CONTROL-FLOW PROTECTION FAULT: #CP[0003] endbranch
> 
> And Xen hangs at this point.
> 
> Disabling CET-IBT let Xen and tboot power off, but reboot was
> perfoming a poweroff instead of a warm reboot.  Disabling all of CET,
> i.e. shadow stacks as well, lets tboot reboot properly.
> 
> Fixes: cdbe2b0a1aec ("x86: Enable CET Indirect Branch Tracking")
> Signed-off-by: Jason Andryuk <jandryuk@gmail.com>
> ---
> Without this fix, Xen subsequently hangs:
> 
> Reboot in five seconds...
> [VT-D] IOMMU1: QI wait descriptor taking too long
>   IQA = 484897000
>   IQH = 0
>   IQT = 820
> 
> with no futher output.
> ---
>   xen/arch/x86/tboot.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/xen/arch/x86/tboot.c b/xen/arch/x86/tboot.c
> index 90f6e805a9..86c4c22cac 100644
> --- a/xen/arch/x86/tboot.c
> +++ b/xen/arch/x86/tboot.c
> @@ -353,6 +353,16 @@ void tboot_shutdown(uint32_t shutdown_type)
>           tboot_gen_xenheap_integrity(g_tboot_shared->s3_key, &xenheap_mac);
>       }
>   
> +    /*
> +     * Disable CET - tboot may not be built with endbr, and it doesn't support
> +     * shadow stacks.
> +     */
> +    if ( read_cr4() & X86_CR4_CET )
> +    {
> +        wrmsrl(MSR_S_CET, 0);
> +        write_cr4(read_cr4() & ~X86_CR4_CET);
> +    }
> +
>       /*
>        * During early boot, we can be called by panic before idle_vcpu[0] is
>        * setup, but in that case we don't need to change page tables.

Reviewed-by: Daniel P. Smith <dpsmith@apertussolutions.com>

On 15/08/2023 5:11 pm, Jason Andryuk wrote:
> tboot_shutdown() calls into tboot to perform the actual system shutdown.
> tboot isn't built with endbr annotations, and Xen has CET-IBT enabled on
> newer hardware.  shutdown_entry isn't annotated with endbr and Xen
> faults:
>
> Panic on CPU 0:
> CONTROL-FLOW PROTECTION FAULT: #CP[0003] endbranch
>
> And Xen hangs at this point.
>
> Disabling CET-IBT let Xen and tboot power off, but reboot was
> perfoming a poweroff instead of a warm reboot.  Disabling all of CET,
> i.e. shadow stacks as well, lets tboot reboot properly.
>
> Fixes: cdbe2b0a1aec ("x86: Enable CET Indirect Branch Tracking")
> Signed-off-by: Jason Andryuk <jandryuk@gmail.com>

:sadpanda:

I guess this is the least bad option going.

Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>