From nobody Wed Oct 8 21:30:05 2025 Received: from cstnet.cn (smtp81.cstnet.cn [159.226.251.81]) (using TLSv1.2 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9ECBD253950; Tue, 24 Jun 2025 08:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=159.226.251.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750752367; cv=none; b=IKldt2kJsQgDfGKloAM+Czr2eZ8e9xqizApwqacVnm2gYMtjo4G/7ngFJY5EJUnveuC8Uh1yHkuc5snFEun1HW/GOtiOF/9lVk/h18iCuDzx0+pmWv42nSvIa83JYP8oVkhgE8lS/jM2G4SvGARzZgOgmOka5zhpxugoAfmkVqY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750752367; c=relaxed/simple; bh=n500nFU0GBgArjcDyeafUU0aiuWxGbNhBge2jWYqoDg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=sDSdqNHKZBjvbx27+tV4iQHBSN6InVnn+FcPA00teiSV98EGXTrzAsfiWVWfcj2N5s/dYit0+fOjsSnPn+UlB7zfxYri7f765PUbA349dQabDnrjhLkMa0EEZZdaw1+lOTHDH2bF/F1yu7G2RRr01xGqwEX+EfIO9L+Kr9Sai28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=iscas.ac.cn; spf=pass smtp.mailfrom=iscas.ac.cn; arc=none smtp.client-ip=159.226.251.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=iscas.ac.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=iscas.ac.cn Received: from [127.0.0.2] (unknown [210.73.43.2]) by APP-03 (Coremail) with SMTP id rQCowADn_FRSXFpoFc3eCA--.42565S2; Tue, 24 Jun 2025 16:05:39 +0800 (CST) From: Vivian Wang Date: Tue, 24 Jun 2025 16:04:46 +0800 Subject: [PATCH] riscv: cpu_ops_sbi: Use static array for boot_data Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250624-riscv-hsm-boot-data-array-v1-1-50b5eeafbe61@iscas.ac.cn> X-B4-Tracking: v=1; b=H4sIAB1cWmgC/x3MQQqDQAxA0atI1gbGVEv1KuIiOrFmUUcSkRbx7 h1cvsX/J7iYikNXnGByqGtaM6qygGnh9S2oMRsoUBOeVKOpTwcu/sExpR0j74xsxj+s4qOl0L7 GSAS530xm/d7vfriuP7OT6gJrAAAA X-Change-ID: 20250624-riscv-hsm-boot-data-array-1d392098bd22 To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Alexandre Ghiti , Atish Patra , Anup Patel Cc: Vivian Wang , Han Gao , Yao Zi , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Vivian Wang X-Mailer: b4 0.14.2 X-CM-TRANSID: rQCowADn_FRSXFpoFc3eCA--.42565S2 X-Coremail-Antispam: 1UD129KBjvJXoW3XFWkZrWkJr4xZF47uw17Jrb_yoW7CFW8pF y5Jr1UWF4kXr1UAw1xtry5ur15Ar1DA3W3JFn7tF1rAF1UGr1UJr1Ut3yIka98tFy8JFy7 tr4DZa1xKFyDJaDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUU9E14x267AKxVW8JVW5JwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2ocxC64kIII0Yj41l84x0c7CEw4AK67xGY2AK02 1l84ACjcxK6xIIjxv20xvE14v26ryj6F1UM28EF7xvwVC0I7IYx2IY6xkF7I0E14v26F4j 6r4UJwA2z4x0Y4vEx4A2jsIE14v26F4UJVW0owA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_Gc CE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx0E 2Ix0cI8IcVAFwI0_JrI_JrylYx0Ex4A2jsIE14v26r4j6F4UMcvjeVCFs4IE7xkEbVWUJV W8JwACjcxG0xvY0x0EwIxGrwACjI8F5VA0II8E6IAqYI8I648v4I1lFIxGxcIEc7CjxVA2 Y2ka0xkIwI1lc7CjxVAaw2AFwI0_Jw0_GFylc2xSY4AK67AK6r43MxAIw28IcxkI7VAKI4 8JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xv wVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjx v20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20E Y4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267 AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7VUjWSotUUUUU== X-CM-SenderInfo: pzdqw2pxlnt03j6l2u1dvotugofq/ Since commit 6b9f29b81b15 ("riscv: Enable pcpu page first chunk allocator"), if NUMA is enabled, the page percpu allocator may be used on very sparse configurations, or when requested on boot with percpu_alloc=3Dpage. In that case, percpu data gets put in the vmalloc area. However, sbi_hsm_hart_start() needs the physical address of a sbi_hart_boot_data, and simply assumes that __pa() would work. This causes the just started hart to immediately access an invalid address and hang. Fortunately, struct sbi_hart_boot_data is not too large, so we can simply allocate an array for boot_data statically, putting it in the kernel image. This fixes NUMA=3Dy SMP boot on Sophgo SG2042. To reproduce on QEMU: Set CONFIG_NUMA=3Dy and CONFIG_DEBUG_VIRTUAL=3Dy, then run with: qemu-system-riscv64 -M virt -smp 2 -nographic \ -kernel arch/riscv/boot/Image \ -append "percpu_alloc=3Dpage" Kernel output: [ 0.000000] Booting Linux on hartid 0 [ 0.000000] Linux version 6.16.0-rc1 (dram@sakuya) (riscv64-unknown-linu= x-gnu-gcc (GCC) 14.2.1 20250322, GNU ld (GNU Binutils) 2.44) #11 SMP Tue Ju= n 24 14:56:22 CST 2025 ... [ 0.000000] percpu: 28 4K pages/cpu s85784 r8192 d20712 ... [ 0.083192] smp: Bringing up secondary CPUs ... [ 0.086722] ------------[ cut here ]------------ [ 0.086849] virt_to_phys used for non-linear address: (____ptrval____) (= 0xff2000000001d080) [ 0.088001] WARNING: CPU: 0 PID: 1 at arch/riscv/mm/physaddr.c:14 __virt= _to_phys+0xae/0xe8 [ 0.088376] Modules linked in: [ 0.088656] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.16.0-rc1 = #11 NONE [ 0.088833] Hardware name: riscv-virtio,qemu (DT) [ 0.088948] epc : __virt_to_phys+0xae/0xe8 [ 0.089001] ra : __virt_to_phys+0xae/0xe8 [ 0.089037] epc : ffffffff80021eaa ra : ffffffff80021eaa sp : ff20000000= 04bbc0 [ 0.089057] gp : ffffffff817f49c0 tp : ff60000001d60000 t0 : 5f6f745f74= 726976 [ 0.089076] t1 : 0000000000000076 t2 : 705f6f745f747269 s0 : ff20000000= 04bbe0 [ 0.089095] s1 : ff2000000001d080 a0 : 0000000000000000 a1 : 0000000000= 000000 [ 0.089113] a2 : 0000000000000000 a3 : 0000000000000000 a4 : 0000000000= 000000 [ 0.089131] a5 : 0000000000000000 a6 : 0000000000000000 a7 : 0000000000= 000000 [ 0.089155] s2 : ffffffff8130dc00 s3 : 0000000000000001 s4 : 0000000000= 000001 [ 0.089174] s5 : ffffffff8185eff8 s6 : ff2000007f1eb000 s7 : ffffffff80= 02a2ec [ 0.089193] s8 : 0000000000000001 s9 : 0000000000000001 s10: 0000000000= 000000 [ 0.089211] s11: 0000000000000000 t3 : ffffffff8180a9f7 t4 : ffffffff81= 80a9f7 [ 0.089960] t5 : ffffffff8180a9f8 t6 : ff2000000004b9d8 [ 0.089984] status: 0000000200000120 badaddr: ffffffff80021eaa cause: 00= 00000000000003 [ 0.090101] [] __virt_to_phys+0xae/0xe8 [ 0.090228] [] sbi_cpu_start+0x6e/0xe8 [ 0.090247] [] __cpu_up+0x1e/0x8c [ 0.090260] [] bringup_cpu+0x42/0x258 [ 0.090277] [] cpuhp_invoke_callback+0xe0/0x40c [ 0.090292] [] __cpuhp_invoke_callback_range+0x68/0xfc [ 0.090320] [] _cpu_up+0x11a/0x244 [ 0.090334] [] cpu_up+0x52/0x90 [ 0.090384] [] bringup_nonboot_cpus+0x78/0x118 [ 0.090411] [] smp_init+0x34/0xb8 [ 0.090425] [] kernel_init_freeable+0x148/0x2e4 [ 0.090442] [] kernel_init+0x1e/0x14c [ 0.090455] [] ret_from_fork_kernel+0xe/0xf0 [ 0.090471] [] ret_from_fork_kernel_asm+0x16/0x18 [ 0.090560] ---[ end trace 0000000000000000 ]--- [ 1.179875] CPU1: failed to come online [ 1.190324] smp: Brought up 1 node, 1 CPU Cc: stable@vger.kernel.org Reported-by: Han Gao Fixes: 9a2451f18663 ("RISC-V: Avoid using per cpu array for ordered booting= ") Signed-off-by: Vivian Wang Reviewed-by: Alexandre Ghiti Tested-by: Alexandre Ghiti --- arch/riscv/kernel/cpu_ops_sbi.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/cpu_ops_sbi.c b/arch/riscv/kernel/cpu_ops_sb= i.c index e6fbaaf549562d6e9ca63a66371441fe8b230cb3..87d6559448039cdd2e8a604a19b= d832ab5a98fc2 100644 --- a/arch/riscv/kernel/cpu_ops_sbi.c +++ b/arch/riscv/kernel/cpu_ops_sbi.c @@ -18,10 +18,10 @@ const struct cpu_operations cpu_ops_sbi; =20 /* * Ordered booting via HSM brings one cpu at a time. However, cpu hotplug = can - * be invoked from multiple threads in parallel. Define a per cpu data + * be invoked from multiple threads in parallel. Define an array of boot d= ata * to handle that. */ -static DEFINE_PER_CPU(struct sbi_hart_boot_data, boot_data); +static struct sbi_hart_boot_data boot_data[NR_CPUS]; =20 static int sbi_hsm_hart_start(unsigned long hartid, unsigned long saddr, unsigned long priv) @@ -67,7 +67,7 @@ static int sbi_cpu_start(unsigned int cpuid, struct task_= struct *tidle) unsigned long boot_addr =3D __pa_symbol(secondary_start_sbi); unsigned long hartid =3D cpuid_to_hartid_map(cpuid); unsigned long hsm_data; - struct sbi_hart_boot_data *bdata =3D &per_cpu(boot_data, cpuid); + struct sbi_hart_boot_data *bdata =3D &boot_data[cpuid]; =20 /* Make sure tidle is updated */ smp_mb(); --- base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494 change-id: 20250624-riscv-hsm-boot-data-array-1d392098bd22 Best regards, --=20 Vivian "dramforever" Wang