From nobody Fri Dec 19 07:21:12 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1747156324; cv=none; d=zohomail.com; s=zohoarc; b=jt7h4pr5etzWMuoTdMCzBurLn9kpZ522KWG2fI8AtKWk9TlgMzaLxj9fKD79luw4Mx/21NOazgY4Kls+dQqxr3ztBkFhiL/sCxoQTKj2nq6sGr/C4V6lq3mzLX9I3zlAYVg4kdgowO+cr+b0Axa5irvqDiycgQXkVwF2fooGg+Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747156324; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=LFjLGOzbWGs49PySjNsQW0fv0Cb3jzfBKePmEV573e8=; b=jMRmxyT/u5YQscXAyhf9YPSSDVCWj61lKE/YhcI/Zc7LLQlRxkc/lB8Z660DWT5/zb8hzCwIfcUTcMyk0p3Ktl7lcEqYRm5L5CHWzNPjMZ3AF4ndKmyyQq0BDer+P1UQzVK1xZe7iMrVbZcoPix/81PW8E1zYGEu2HVGL2TKjVg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1747156324315240.0700357480839; Tue, 13 May 2025 10:12:04 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.983322.1369703 (Exim 4.92) (envelope-from ) id 1uEtAU-0005t2-BQ; Tue, 13 May 2025 17:11:50 +0000 Received: by outflank-mailman (output) from mailman id 983322.1369703; Tue, 13 May 2025 17:11:50 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uEtAU-0005su-88; Tue, 13 May 2025 17:11:50 +0000 Received: by outflank-mailman (input) for mailman id 983322; Tue, 13 May 2025 17:11:48 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uEt5t-0003Mm-Ls for xen-devel@lists.xenproject.org; Tue, 13 May 2025 17:07:05 +0000 Received: from 13.mo582.mail-out.ovh.net (13.mo582.mail-out.ovh.net [188.165.56.124]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id b0798e6e-301c-11f0-9eb6-5ba50f476ded; Tue, 13 May 2025 19:07:02 +0200 (CEST) Received: from director7.ghost.mail-out.ovh.net (unknown [10.109.139.225]) by mo582.mail-out.ovh.net (Postfix) with ESMTP id 4ZxjZG3qKJz1f6K for ; Tue, 13 May 2025 17:07:02 +0000 (UTC) Received: from ghost-submission-5b5ff79f4f-tzcrg (unknown [10.110.118.108]) by director7.ghost.mail-out.ovh.net (Postfix) with ESMTPS id D9C691FD39; Tue, 13 May 2025 17:07:01 +0000 (UTC) Received: from 3mdeb.com ([37.59.142.99]) by ghost-submission-5b5ff79f4f-tzcrg with ESMTPSA id /APTKjV8I2g1KAEAShSnmA (envelope-from ); Tue, 13 May 2025 17:07:01 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: b0798e6e-301c-11f0-9eb6-5ba50f476ded Authentication-Results: garm.ovh; auth=pass (GARM-99G00304773c23-a1d9-40a9-a6f8-85607a0d1f7b, 0F27B6D195039ACFBDF5EC7F2AC12BEA7E98F15C) smtp.auth=sergii.dmytruk@3mdeb.com X-OVh-ClientIp: 176.111.181.178 From: Sergii Dmytruk To: xen-devel@lists.xenproject.org Cc: Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , trenchboot-devel@googlegroups.com Subject: [PATCH v2 14/22] x86/boot: choose AP stack based on APIC ID Date: Tue, 13 May 2025 20:05:51 +0300 Message-ID: <391ebc0e786cc94e2e558bfb91383705e8998b35.1747155790.git.sergii.dmytruk@3mdeb.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Ovh-Tracer-Id: 8950341310297519260 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: -100 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdeftdegieejucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhephffvvefufffkofgjfhgggfestdekredtredttdenucfhrhhomhepufgvrhhgihhiucffmhihthhruhhkuceoshgvrhhgihhirdgumhihthhruhhkseefmhguvggsrdgtohhmqeenucggtffrrghtthgvrhhnpeevleeiheduleelgfelgeeiveetgeduhfehffefgffhledtvefhledvgfekfefhueenucffohhmrghinhephhgvrggurdhssgdpthhrrghmphholhhinhgvrdhssgdpgiekiegpieegrdhssgenucfkphepuddvjedrtddrtddruddpudejiedrudduuddrudekuddrudejkedpfeejrdehledrudegvddrleelnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepuddvjedrtddrtddruddpmhgrihhlfhhrohhmpehsvghrghhiihdrughmhihtrhhukhesfehmuggvsgdrtghomhdpnhgspghrtghpthhtohepuddprhgtphhtthhopeigvghnqdguvghvvghlsehlihhsthhsrdigvghnphhrohhjvggtthdrohhrghdpoffvtefjohhsthepmhhoheekvdgmpdhmohguvgepshhmthhpohhuth DKIM-Signature: a=rsa-sha256; bh=LFjLGOzbWGs49PySjNsQW0fv0Cb3jzfBKePmEV573e8=; c=relaxed/relaxed; d=3mdeb.com; h=From; s=ovhmo3617313-selector1; t=1747156022; v=1; b=S4CmfvIlPVHOUxOBRXERak2n0KoEmq/QTHvFLaDEZGwxGkxEA6jb29CJ6y4iVxrpNtVWYJg0 vDQVc9nY8fUbCoyOyUBDoKAdVobWaDRsRbVBlYQxbdMFudZd3ijCPjpDVNwcD8zBR/UgFA08EEW lJXa2+YqinoEcRQ5YBXb0SmfMUgUufosT9AIdNOsZyFDBWtvZ7BE5/JItvg8ONh9iyU808kvuL+ kRtax6AzJCRRfVpmdfZ/psAnxAZSh85dhIGwZKjDCcSZFNec564dFhjbUXJS6KESgBzge2MvECe w+yZWgj5bPhdtP6CFF3q6/hGSx7++p0fQ9zK6erWt7s7Q== X-ZohoMail-DKIM: pass (identity @3mdeb.com) X-ZM-MESSAGEID: 1747156325047019000 Content-Type: text/plain; charset="utf-8" From: Krystian Hebel This is made as the first step of making parallel AP bring-up possible. It should be enough for pre-C code. Parallel AP bring-up is necessary because TXT by design releases all APs at once. In addition to that it reduces number of IPIs (and more importantly, delays between them) required to start all logical processors. This results in significant reduction of boot time, even when DRTM is not used, with performance gain growing with the number of logical CPUs. Signed-off-by: Krystian Hebel Signed-off-by: Sergii Dmytruk --- xen/arch/x86/boot/head.S | 1 + xen/arch/x86/boot/trampoline.S | 21 +++++++++++++++++++++ xen/arch/x86/boot/x86_64.S | 28 +++++++++++++++++++++++++++- xen/arch/x86/include/asm/apicdef.h | 4 ++++ xen/arch/x86/include/asm/msr-index.h | 3 +++ xen/arch/x86/setup.c | 7 +++++++ 6 files changed, 63 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/boot/head.S b/xen/arch/x86/boot/head.S index 9a272155e9..7376fa85d5 100644 --- a/xen/arch/x86/boot/head.S +++ b/xen/arch/x86/boot/head.S @@ -8,6 +8,7 @@ #include #include #include +#include #include #include =20 diff --git a/xen/arch/x86/boot/trampoline.S b/xen/arch/x86/boot/trampoline.S index a92e399fbe..ed593acc46 100644 --- a/xen/arch/x86/boot/trampoline.S +++ b/xen/arch/x86/boot/trampoline.S @@ -71,6 +71,27 @@ trampoline_protmode_entry: mov $X86_CR4_PAE,%ecx mov %ecx,%cr4 =20 + /* + * Get APIC ID while we're in non-paged mode. Start by checking if + * x2APIC is enabled. + */ + mov $MSR_APIC_BASE, %ecx + rdmsr + test $APIC_BASE_EXTD, %eax + jnz .Lx2apic + + /* Not x2APIC, read from MMIO */ + and $APIC_BASE_ADDR_MASK, %eax + mov APIC_ID(%eax), %esp + shr $24, %esp + jmp 1f + +.Lx2apic: + mov $(MSR_X2APIC_FIRST + (APIC_ID >> MSR_X2APIC_SHIFT)), %ecx + rdmsr + mov %eax, %esp +1: + /* Load pagetable base register. */ mov $sym_offs(idle_pg_table),%eax add bootsym_rel(trampoline_xen_phys_start,4,%eax) diff --git a/xen/arch/x86/boot/x86_64.S b/xen/arch/x86/boot/x86_64.S index 08ae97e261..ac33576d8f 100644 --- a/xen/arch/x86/boot/x86_64.S +++ b/xen/arch/x86/boot/x86_64.S @@ -15,7 +15,33 @@ ENTRY(__high_start) mov $XEN_MINIMAL_CR4,%rcx mov %rcx,%cr4 =20 - mov stack_start(%rip),%rsp + test %ebx,%ebx + cmovz stack_start(%rip), %rsp + jz .L_stack_set + + /* APs only: get stack base from APIC ID saved in %esp. */ + mov $-1, %rax + lea x86_cpu_to_apicid(%rip), %rcx +1: + add $1, %rax + cmp $NR_CPUS, %eax + jb 2f + hlt +2: + cmp %esp, (%rcx, %rax, 4) + jne 1b + + /* %eax is now Xen CPU index. */ + lea stack_base(%rip), %rcx + mov (%rcx, %rax, 8), %rsp + + test %rsp,%rsp + jnz 1f + hlt +1: + add $(STACK_SIZE - CPUINFO_sizeof), %rsp + +.L_stack_set: =20 /* Reset EFLAGS (subsumes CLI and CLD). */ pushq $0 diff --git a/xen/arch/x86/include/asm/apicdef.h b/xen/arch/x86/include/asm/= apicdef.h index 63dab01dde..e093a2aa3c 100644 --- a/xen/arch/x86/include/asm/apicdef.h +++ b/xen/arch/x86/include/asm/apicdef.h @@ -121,6 +121,10 @@ =20 #define MAX_IO_APICS 128 =20 +#ifndef __ASSEMBLY__ + extern bool x2apic_enabled; =20 +#endif /* !__ASSEMBLY__ */ + #endif diff --git a/xen/arch/x86/include/asm/msr-index.h b/xen/arch/x86/include/as= m/msr-index.h index 22d9e76e55..794cf44abe 100644 --- a/xen/arch/x86/include/asm/msr-index.h +++ b/xen/arch/x86/include/asm/msr-index.h @@ -169,6 +169,9 @@ #define MSR_X2APIC_FIRST 0x00000800 #define MSR_X2APIC_LAST 0x000008ff =20 +/* MSR offset can be obtained by shifting MMIO offset this number of bits = to the right. */ +#define MSR_X2APIC_SHIFT 4 + #define MSR_X2APIC_TPR 0x00000808 #define MSR_X2APIC_PPR 0x0000080a #define MSR_X2APIC_EOI 0x0000080b diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index 479d2d744e..8e79d4be23 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c @@ -2096,6 +2096,7 @@ void asmlinkage __init noreturn __start_xen(void) */ if ( !pv_shim ) { + /* Separate loop to make parallel AP bringup possible. */ for_each_present_cpu ( i ) { /* Set up cpu_to_node[]. */ @@ -2103,6 +2104,12 @@ void asmlinkage __init noreturn __start_xen(void) /* Set up node_to_cpumask based on cpu_to_node[]. */ numa_add_cpu(i); =20 + if ( stack_base[i] =3D=3D NULL ) + stack_base[i] =3D cpu_alloc_stack(i); + } + + for_each_present_cpu ( i ) + { if ( (park_offline_cpus || num_online_cpus() < max_cpus) && !cpu_online(i) ) { --=20 2.49.0