[v2] powerpc/kexec: Enable SMT before waking offline CPUs

[PATCH v2] powerpc/kexec: Enable SMT before waking offline CPUs

Posted by Nysal Jan K.A. 3 months, 1 week ago

If SMT is disabled or a partial SMT state is enabled, when a new kernel
image is loaded for kexec, on reboot the following warning is observed:

kexec: Waking offline cpu 228.
WARNING: CPU: 0 PID: 9062 at arch/powerpc/kexec/core_64.c:223 kexec_prepare_cpus+0x1b0/0x1bc
[snip]
 NIP kexec_prepare_cpus+0x1b0/0x1bc
 LR  kexec_prepare_cpus+0x1a0/0x1bc
 Call Trace:
  kexec_prepare_cpus+0x1a0/0x1bc (unreliable)
  default_machine_kexec+0x160/0x19c
  machine_kexec+0x80/0x88
  kernel_kexec+0xd0/0x118
  __do_sys_reboot+0x210/0x2c4
  system_call_exception+0x124/0x320
  system_call_vectored_common+0x15c/0x2ec

This occurs as add_cpu() fails due to cpu_bootable() returning false for
CPUs that fail the cpu_smt_thread_allowed() check or non primary
threads if SMT is disabled.

Fix the issue by enabling SMT and resetting the number of SMT threads to
the number of threads per core, before attempting to wake up all present
CPUs.

Fixes: 38253464bc82 ("cpu/SMT: Create topology_smt_thread_allowed()")
Reported-by: Sachin P Bappalige <sachinpb@linux.ibm.com>
Cc: stable@vger.kernel.org # v6.6+
Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
Signed-off-by: Nysal Jan K.A. <nysal@linux.ibm.com>
---
 arch/powerpc/kexec/core_64.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
index 222aa326dace..825ab8a88f18 100644
--- a/arch/powerpc/kexec/core_64.c
+++ b/arch/powerpc/kexec/core_64.c
@@ -202,6 +202,23 @@ static void kexec_prepare_cpus_wait(int wait_state)
 	mb();
 }
 
+
+/*
+ * The add_cpu() call in wake_offline_cpus() can fail as cpu_bootable()
+ * returns false for CPUs that fail the cpu_smt_thread_allowed() check
+ * or non primary threads if SMT is disabled. Re-enable SMT and set the
+ * number of SMT threads to threads per core.
+ */
+static void kexec_smt_reenable(void)
+{
+#if defined(CONFIG_SMP) && defined(CONFIG_HOTPLUG_SMT)
+	lock_device_hotplug();
+	cpu_smt_num_threads = threads_per_core;
+	cpu_smt_control = CPU_SMT_ENABLED;
+	unlock_device_hotplug();
+#endif
+}
+
 /*
  * We need to make sure each present CPU is online.  The next kernel will scan
  * the device tree and assume primary threads are online and query secondary
@@ -216,6 +233,8 @@ static void wake_offline_cpus(void)
 {
 	int cpu = 0;
 
+	kexec_smt_reenable();
+
 	for_each_present_cpu(cpu) {
 		if (!cpu_online(cpu)) {
 			printk(KERN_INFO "kexec: Waking offline cpu %d.\n",
-- 
2.47.3

Re: [PATCH v2] powerpc/kexec: Enable SMT before waking offline CPUs

Posted by Madhavan Srinivasan 1 month, 1 week ago

On Tue, 28 Oct 2025 16:25:12 +0530, Nysal Jan K.A. wrote:
> If SMT is disabled or a partial SMT state is enabled, when a new kernel
> image is loaded for kexec, on reboot the following warning is observed:
> 
> kexec: Waking offline cpu 228.
> WARNING: CPU: 0 PID: 9062 at arch/powerpc/kexec/core_64.c:223 kexec_prepare_cpus+0x1b0/0x1bc
> [snip]
>  NIP kexec_prepare_cpus+0x1b0/0x1bc
>  LR  kexec_prepare_cpus+0x1a0/0x1bc
>  Call Trace:
>   kexec_prepare_cpus+0x1a0/0x1bc (unreliable)
>   default_machine_kexec+0x160/0x19c
>   machine_kexec+0x80/0x88
>   kernel_kexec+0xd0/0x118
>   __do_sys_reboot+0x210/0x2c4
>   system_call_exception+0x124/0x320
>   system_call_vectored_common+0x15c/0x2ec
> 
> [...]

Applied to powerpc/fixes.

[1/1] powerpc/kexec: Enable SMT before waking offline CPUs
      https://git.kernel.org/powerpc/c/c2296a1e42418556efbeb5636c4fa6aa6106713a

cheers

Re: [PATCH v2] powerpc/kexec: Enable SMT before waking offline CPUs

Posted by Sourabh Jain 1 month, 2 weeks ago


On 28/10/25 16:25, Nysal Jan K.A. wrote:
> If SMT is disabled or a partial SMT state is enabled, when a new kernel
> image is loaded for kexec, on reboot the following warning is observed:
>
> kexec: Waking offline cpu 228.
> WARNING: CPU: 0 PID: 9062 at arch/powerpc/kexec/core_64.c:223 kexec_prepare_cpus+0x1b0/0x1bc
> [snip]
>   NIP kexec_prepare_cpus+0x1b0/0x1bc
>   LR  kexec_prepare_cpus+0x1a0/0x1bc
>   Call Trace:
>    kexec_prepare_cpus+0x1a0/0x1bc (unreliable)
>    default_machine_kexec+0x160/0x19c
>    machine_kexec+0x80/0x88
>    kernel_kexec+0xd0/0x118
>    __do_sys_reboot+0x210/0x2c4
>    system_call_exception+0x124/0x320
>    system_call_vectored_common+0x15c/0x2ec
>
> This occurs as add_cpu() fails due to cpu_bootable() returning false for
> CPUs that fail the cpu_smt_thread_allowed() check or non primary
> threads if SMT is disabled.
>
> Fix the issue by enabling SMT and resetting the number of SMT threads to
> the number of threads per core, before attempting to wake up all present
> CPUs.
>
> Fixes: 38253464bc82 ("cpu/SMT: Create topology_smt_thread_allowed()")
> Reported-by: Sachin P Bappalige <sachinpb@linux.ibm.com>
> Cc: stable@vger.kernel.org # v6.6+
> Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
> Signed-off-by: Nysal Jan K.A. <nysal@linux.ibm.com>
> ---
>   arch/powerpc/kexec/core_64.c | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
>
> diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
> index 222aa326dace..825ab8a88f18 100644
> --- a/arch/powerpc/kexec/core_64.c
> +++ b/arch/powerpc/kexec/core_64.c
> @@ -202,6 +202,23 @@ static void kexec_prepare_cpus_wait(int wait_state)
>   	mb();
>   }
>   
> +
> +/*
> + * The add_cpu() call in wake_offline_cpus() can fail as cpu_bootable()
> + * returns false for CPUs that fail the cpu_smt_thread_allowed() check
> + * or non primary threads if SMT is disabled. Re-enable SMT and set the
> + * number of SMT threads to threads per core.
> + */
> +static void kexec_smt_reenable(void)
> +{
> +#if defined(CONFIG_SMP) && defined(CONFIG_HOTPLUG_SMT)
> +	lock_device_hotplug();
> +	cpu_smt_num_threads = threads_per_core;
> +	cpu_smt_control = CPU_SMT_ENABLED;
> +	unlock_device_hotplug();
> +#endif
> +}
> +
>   /*
>    * We need to make sure each present CPU is online.  The next kernel will scan
>    * the device tree and assume primary threads are online and query secondary
> @@ -216,6 +233,8 @@ static void wake_offline_cpus(void)
>   {
>   	int cpu = 0;
>   
> +	kexec_smt_reenable();
> +
>   	for_each_present_cpu(cpu) {
>   		if (!cpu_online(cpu)) {
>   			printk(KERN_INFO "kexec: Waking offline cpu %d.\n",

The fix looks good to me. I also tested it with QEMU using different
SMT values with the kexec_load and kexec_file_load system calls, and no
warnings were observed during kexec.

Feel free to add:
Reviewed-by: Sourabh Jain <sourabhjain@linux.ibm.com>

Re: [PATCH v2] powerpc/kexec: Enable SMT before waking offline CPUs

Posted by Shrikanth Hegde 3 months, 1 week ago

Hi Nysal.

On 10/28/25 4:25 PM, Nysal Jan K.A. wrote:
> If SMT is disabled or a partial SMT state is enabled, when a new kernel
> image is loaded for kexec, on reboot the following warning is observed:
> 
> kexec: Waking offline cpu 228.
> WARNING: CPU: 0 PID: 9062 at arch/powerpc/kexec/core_64.c:223 kexec_prepare_cpus+0x1b0/0x1bc
> [snip]
>   NIP kexec_prepare_cpus+0x1b0/0x1bc
>   LR  kexec_prepare_cpus+0x1a0/0x1bc
>   Call Trace:
>    kexec_prepare_cpus+0x1a0/0x1bc (unreliable)
>    default_machine_kexec+0x160/0x19c
>    machine_kexec+0x80/0x88
>    kernel_kexec+0xd0/0x118
>    __do_sys_reboot+0x210/0x2c4
>    system_call_exception+0x124/0x320
>    system_call_vectored_common+0x15c/0x2ec
> 
> This occurs as add_cpu() fails due to cpu_bootable() returning false for
> CPUs that fail the cpu_smt_thread_allowed() check or non primary
> threads if SMT is disabled.
> 
> Fix the issue by enabling SMT and resetting the number of SMT threads to
> the number of threads per core, before attempting to wake up all present
> CPUs.
> 
> Fixes: 38253464bc82 ("cpu/SMT: Create topology_smt_thread_allowed()")
> Reported-by: Sachin P Bappalige <sachinpb@linux.ibm.com>
> Cc: stable@vger.kernel.org # v6.6+
> Reviewed-by: Srikar Dronamraju <srikar@linux.ibm.com>
> Signed-off-by: Nysal Jan K.A. <nysal@linux.ibm.com>
> ---
>   arch/powerpc/kexec/core_64.c | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
> 
> diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
> index 222aa326dace..825ab8a88f18 100644
> --- a/arch/powerpc/kexec/core_64.c
> +++ b/arch/powerpc/kexec/core_64.c
> @@ -202,6 +202,23 @@ static void kexec_prepare_cpus_wait(int wait_state)
>   	mb();
>   }
>   
> +
> +/*
> + * The add_cpu() call in wake_offline_cpus() can fail as cpu_bootable()
> + * returns false for CPUs that fail the cpu_smt_thread_allowed() check
> + * or non primary threads if SMT is disabled. Re-enable SMT and set the
> + * number of SMT threads to threads per core.
> + */
> +static void kexec_smt_reenable(void)
> +{
> +#if defined(CONFIG_SMP) && defined(CONFIG_HOTPLUG_SMT)
> +	lock_device_hotplug();

I was looking at usage of lock_device_hotplug, looks like a good candidate for
guard() use case. Could be done on its own patch/series.

> +	cpu_smt_num_threads = threads_per_core;
> +	cpu_smt_control = CPU_SMT_ENABLED;
> +	unlock_device_hotplug();
> +#endif
> +}


Will this work too? It might be better since we anyway going to bring that CPU up
by doing add_cpu afterwords.

	cpu_smt_num_threads = threads_per_core;
	cpuhp_smt_enable()

> +
>   /*
>    * We need to make sure each present CPU is online.  The next kernel will scan
>    * the device tree and assume primary threads are online and query secondary
> @@ -216,6 +233,8 @@ static void wake_offline_cpus(void)
>   {
>   	int cpu = 0;
>   
> +	kexec_smt_reenable();
> +

If we do above, just change the below logic to complain if any present CPU is offline.

>   	for_each_present_cpu(cpu) {
>   		if (!cpu_online(cpu)) {
>   			printk(KERN_INFO "kexec: Waking offline cpu %d.\n",

Re: [PATCH v2] powerpc/kexec: Enable SMT before waking offline CPUs

Posted by Nysal Jan K.A. 3 months, 1 week ago

hi Shrikanth,

On Tue, Oct 28, 2025 at 10:56:05PM +0530, Shrikanth Hegde wrote:
> Hi Nysal.
> 
> On 10/28/25 4:25 PM, Nysal Jan K.A. wrote:

[snip]

> > --- a/arch/powerpc/kexec/core_64.c
> > +++ b/arch/powerpc/kexec/core_64.c
> > @@ -202,6 +202,23 @@ static void kexec_prepare_cpus_wait(int wait_state)
> >   	mb();
> >   }
> > +
> > +/*
> > + * The add_cpu() call in wake_offline_cpus() can fail as cpu_bootable()
> > + * returns false for CPUs that fail the cpu_smt_thread_allowed() check
> > + * or non primary threads if SMT is disabled. Re-enable SMT and set the
> > + * number of SMT threads to threads per core.
> > + */
> > +static void kexec_smt_reenable(void)
> > +{
> > +#if defined(CONFIG_SMP) && defined(CONFIG_HOTPLUG_SMT)
> > +	lock_device_hotplug();
> 
> I was looking at usage of lock_device_hotplug, looks like a good candidate for
> guard() use case. Could be done on its own patch/series.
> 

Agree, we can look at it as a separate patch.

> > +	cpu_smt_num_threads = threads_per_core;
> > +	cpu_smt_control = CPU_SMT_ENABLED;
> > +	unlock_device_hotplug();
> > +#endif
> > +}
> 
> 
> Will this work too? It might be better since we anyway going to bring that CPU up
> by doing add_cpu afterwords.
> 
> 	cpu_smt_num_threads = threads_per_core;
> 	cpuhp_smt_enable()
> 

There is some reasoning in 4d37cc2dc3df, which made the switch to use the core
device API, against calling cpu_up() directly. The other issue is 
cpuhp_smt_enable() can skip bringing up a CPU in certain cases, for example
when a core is offline.

> > +
> >   /*
> >    * We need to make sure each present CPU is online.  The next kernel will scan
> >    * the device tree and assume primary threads are online and query secondary
> > @@ -216,6 +233,8 @@ static void wake_offline_cpus(void)
> >   {
> >   	int cpu = 0;
> > +	kexec_smt_reenable();
> > +
> 
> If we do above, just change the below logic to complain if any present CPU is offline.
> 
> >   	for_each_present_cpu(cpu) {
> >   		if (!cpu_online(cpu)) {
> >   			printk(KERN_INFO "kexec: Waking offline cpu %d.\n",
> 

Thanks for the review.

--Nysal