From nobody Sun May 3 14:26:26 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1777662673; cv=none; d=zohomail.com; s=zohoarc; b=nqGr39d7suwVZ5GCRNfj9tdvdKyO4oK7GKvDdApyXWht8jUYO/o85Fj6LNUX6BYzPoZNTBbsIWpd2vIzIicMtos7dtxDkodoVpLPhKnCAVDhc0fpG/bcARasLHCJdemsp1EbOHbRvM32DtoAr+v4+qC15jMoqEmzBrKjU+Hq9xc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1777662673; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=6aloAp7jHnwZ3WXl4y8dNSH+C8JTIu98fW6pgEvDoyk=; b=BHcO4e+P9cVokCGnqA/2X8QJrtuVtcY+c5pMdyIjZIwF5kXlmE/ZbNZZ8HYND6HLHRmSy9uLWMpazdjn4NXXsQGbfISPNQGaWabsh+npcuXHPi8pEqzqDU9HBnsbVgGssBG2qS9WHHXYbR9YHCYITTtIGM6wni3517Tz5z0xRlI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1777662673507468.71471427826725; Fri, 1 May 2026 12:11:13 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1298878.1573852 (Exim 4.92) (envelope-from ) id 1wItFy-00069I-ND; Fri, 01 May 2026 19:10:34 +0000 Received: by outflank-mailman (output) from mailman id 1298878.1573852; Fri, 01 May 2026 19:10:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1wItFy-00069B-JU; Fri, 01 May 2026 19:10:34 +0000 Received: by outflank-mailman (input) for mailman id 1298878; Fri, 01 May 2026 19:10:33 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1wItFx-000695-0l for xen-devel@lists.xenproject.org; Fri, 01 May 2026 19:10:33 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1wItFv-00Bf5u-UY for xen-devel@lists.xenproject.org; Fri, 01 May 2026 21:10:31 +0200 Received: from [10.42.69.8] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 69f4faa7-e002-0a2a0a5209dd-0a2a450891a0-0 for ; Fri, 01 May 2026 21:10:31 +0200 Received: from [209.85.128.51] (helo=mail-wm1-f51.google.com) by tlsNG-c1860d.mxtls.expurgate.net with ESMTPS (eXpurgate 4.56.1) (envelope-from ) id 69f4faa7-63b5-0a2a45080019-d1558033e02a-3 for ; Fri, 01 May 2026 21:10:31 +0200 Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-488b0e1b870so35222345e9.2 for ; Fri, 01 May 2026 12:10:31 -0700 (PDT) Received: from localhost.localdomain (host-78-146-242-105.as13285.net. [78.146.242.105]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48a82308d77sm170781835e9.14.2026.05.01.12.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 May 2026 12:10:30 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=citrix.com header.i="@citrix.com" header.h="Content-Transfer-Encoding:MIME-Version:Message-Id:Date:Subject:Cc:To:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1777662631; x=1778267431; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=6aloAp7jHnwZ3WXl4y8dNSH+C8JTIu98fW6pgEvDoyk=; b=MrpYICRWH+glcx43E3o4WWDFW35MwCs3nei7y4a2DNjQILvJ1VAfS9eY0vZtn16sc6 oklq/sJgiooxr1W/Qoz7BHqEgUqhW9cMzC22wFn1MoIfQKagX0Iz2BKwKawDnGWin0zG 7lf8zKjKNgJpPMBCNMx+TYJHg6jEdUKcn2lTU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777662631; x=1778267431; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=6aloAp7jHnwZ3WXl4y8dNSH+C8JTIu98fW6pgEvDoyk=; b=np4KkM+3XOd3im+CwrEuNBHt2KqxpF9QFBn1FwUz54qKZ+K/47z5h0FyDcc3drdQe6 QWH5WNqGEI1YFk0PihsRnQaz6pRgZzTtfoiOpAVA8AUqSO4SMa1IQIHiwS7PFZ3M7KMX h/yWaYKQKdikanUzLkyi9e22+zy2jqak/IeAb77zJRpwxWrRGjStP/Ymel871z/AY2mS YwrWkon2eybLHOrOjSibozvCC7q4wu0OUqTZqsOfzTTrTJzUskiGHxfo1p3+7xmWt4H6 GFKATc6MR5qSXBLwUgWWI/0GSZ6cWK3MN3XX7nEwcneQQ2CCDH3Cra/HnzNW8vOgpFeI 0Yhg== X-Gm-Message-State: AOJu0YxOW3um6WJ2p4V9hf5ZSqubAiQClEDlv+83BUDVt0NQdpk4W2Y1 hHEtGWFBlvwQ8zQBbLvQD3mb83CEFpIRRmocyLJ9Q/eFOiqmZaQxmhJxdIdChAzSvDQdYGYqGXx Na/Od X-Gm-Gg: AeBDievIpviGp64p20h+JuvxNoO7vzcUzF7UYAqXl9ruzBqs9J902KPQbe2bBOfaOVU +/hRVqp2tJa0PD+vsa4LjCyO7XqslrKcrIt05MI3Z8UijZyiLKavQwZvTGnGVmJuNtM+DxVyick QpZeUx5BHa4eiywWx28jmpCEAWFmxNvCMSloL8NrUPJNxAOdSP5GlHOHwJLkoNUH8O+pdD7lEYG OCK74RMUM0WSIhlxuARHJEmPlix8BytDGSkTNw8DEBSBA900GIpYK94vqqjxrfAGn0yPciVAE+q 32eBdj9Edz1F34LDIaAuIEFozSjBTWp5RP4nXjKWwm+m8LsAmsvJf68W36TUaynYa7eW1tBzdP6 FOaao79YZnNOjIlfwlyb8SolY7UE7MkW8QknLeCu0MW70bKT3bpqlym/ly2qLB0e0s/1rSgBnX0 pupO3d7VkVc9e6tBoOu+26E9ux9S5szjmj3hntci0JLOFpPa8ojFodRhGB6EYa5IYRF0vQhKt8x 4PwpaLR6MD/kyM= X-Received: by 2002:a05:600c:608a:b0:488:acbc:b2e with SMTP id 5b1f17b1804b1-48a9865d2ddmr5509405e9.17.1777662630692; Fri, 01 May 2026 12:10:30 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Teddy Astie Subject: [PATCH] x86/boot: Disable interrupts when establishing SSP Date: Fri, 1 May 2026 20:10:28 +0100 Message-Id: <20260501191028.1250225-1-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-purgate-ID: tlsNG-c1860d/1777662631-C0D63DB1-924C0444/0/0 X-purgate-type: clean X-purgate-size: 4465 X-ZohoMail-DKIM: pass (identity @citrix.com) X-ZM-MESSAGEID: 1777662675614158500 Gitlab CI reported a crash on boot on Alder Lake hardware. The bug is years old, making it an incredibly rare occurance: (XEN) *** DOUBLE FAULT *** (XEN) ----[ Xen-4.22-unstable x86_64 debug=3Dy ubsan=3Dy Not tainted ]= ---- (XEN) CPU: 0 (XEN) RIP: e008:[] arch/x86/setup.c#reinit_bsp_stack= +0xfa/0x160 (XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor (XEN) rax: 0000000000000007 rbx: ffff83049a4b0000 rcx: 00000000000006= a2 (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 00000000000000= 00 (XEN) rbp: ffff83049a4b7f00 rsp: ffff83049a4b7ef8 r8: ffff830497e470= 00 (XEN) r9: 00000000ffffffff r10: 00000000900c2121 r11: 000000009a3929= 56 (XEN) r12: ffff830497e47000 r13: ffff830497e49f40 r14: 00000000000000= 00 (XEN) r15: ffff82d0407dad10 cr0: 0000000080050033 cr4: 0000000000f526= e0 (XEN) cr3: 0000000043c16000 cr2: fffffffffffffffc (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 00000000000000= 00 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 (XEN) Xen code around (arch/x86/setup.c#reinit_bsp_sta= ck+0xfa/0x160): (XEN) 00 b9 a2 06 00 00 0f 30 <80> 3d 71 26 f1 ff 00 74 3e 48 8d 93 f8 5= f 00 00 (XEN) Valid stack range: ffff83049a4b6000-ffff83049a4b8000, sp=3Dffff8304= 9a4b7ef8, tss.rsp0=3Dffff83049a4b7fb0 (XEN) No stack overflow detected. Skipping stack trace. (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) DOUBLE FAULT -- system shutdown (XEN) **************************************** This is on the instruction boundary after enabling CET (writing MSR_S_CET) = and prior to establishing SSP. Despite identifying this as a critical window where any fault was deadly (the CPU tries to push a shadow stack frame at 0, hence the CR2 value wrapping around to the top of the address space), I clearly forgot that this meant interrupts too, which are enabled. Along with regular interrupts, NMIs are a problem. Unlike other cases need= ing NMI safety, we can't use a self NMI and callback, as the stack needs to be empty at the point of enabling Shadow Stacks. Disable interrupts, and turn off the watchdog if it's configured. Note that watchdog_{en,dis}able() do not work here. They cause the watchdog NMI to be ignored; they do not inhibit the generation of NMIs. Fixes: b60ab42db2f0 ("x86/shstk: Activate Supervisor Shadow Stacks") Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Teddy Astie The only way I can think of doing this in NMI context is to have the NMI LR= ET off the NMI stack back to the interrupted context. But this is horrible to arrange, not to mention different between IDT and FRED. Also, the {disable,setup}_lapic_nmi_watchdog() API is horrible but I don't have time to make it sane right now, and this needs backporting a long way. https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/2494069238 --- xen/arch/x86/setup.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index d041cbd5f6f1..95ac36beab37 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c @@ -908,6 +908,19 @@ static void __init noreturn reinit_bsp_stack(void) =20 if ( cpu_has_xen_shstk ) { + bool watchdog =3D (nmi_watchdog =3D=3D NMI_LOCAL_APIC); + + /* + * Between enabling CET and establishing SSP, any fault or interru= pt + * is fatal. We must arrange for none to happen. + * + * TODO: Figure out how to perform CET enablement in NMI context, + * given the constraint that the stack must be empty. + */ + if ( watchdog ) + disable_lapic_nmi_watchdog(); + local_irq_disable(); + wrmsrl(MSR_S_CET, xen_msr_s_cet_value()); =20 /* @@ -932,6 +945,13 @@ static void __init noreturn reinit_bsp_stack(void) } else asm volatile ( "setssbsy" ::: "memory" ); + + local_irq_enable(); + if ( watchdog ) + { + nmi_watchdog =3D NMI_LOCAL_APIC; + setup_apic_nmi_watchdog(); + } } =20 reset_stack_and_jump(init_done); base-commit: 61f957d48c78df6c5254b6f54d6170d3bd3d717e --=20 2.39.5