I'm taking over the series Michael sent previously [1] which is smartly
reviewing the initial series I sent [2].  This series is addressing the
comments sent by Thomas and me on the Michael's one.

Here is a short introduction to the issue this series is addressing:

When a new CPU is added, the kernel is activating all its threads. This
leads to weird, but functional, result when adding CPU on a SMT 4 system
for instance.

Here the newly added CPU 1 has 8 threads while the other one has 4 threads
active (system has been booted with the 'smt-enabled=4' kernel option):

ltcden3-lp12:~ # ppc64_cpu --info
Core   0:    0*    1*    2*    3*    4     5     6     7
Core   1:    8*    9*   10*   11*   12*   13*   14*   15*

This mixed SMT level may confused end users and/or some applications.

There is no SMT level recorded in the kernel (common code), neither in user
space, as far as I know. Such a level is helpful when adding new CPU or
when optimizing the energy efficiency (when reactivating CPUs).

When SMP and HOTPLUG_SMT are defined, this series is adding a new SMT level
(cpu_smt_num_threads) and few callbacks allowing the architecture code to
fine control this value, setting a max and a "at boot" level, and
controling whether a thread should be onlined or not.


v2:
  As Thomas suggested,
    Reword some commit's description
    Remove topology_smt_supported()
    Remove topology_smt_threads_supported()
    Introduce CONFIG_SMT_NUM_THREADS_DYNAMIC
    Remove switch() in __store_smt_control()
  Update kernel-parameters.txt

[1] https://lore.kernel.org/linuxppc-dev/20230524155630.794584-1-mpe@ellerman.id.au/
[2] https://lore.kernel.org/linuxppc-dev/20230331153905.31698-1-ldufour@linux.ibm.com/

Laurent Dufour (1):
  cpu/SMT: Remove topology_smt_supported()

Michael Ellerman (8):
  cpu/SMT: Move SMT prototypes into cpu_smt.h
  cpu/SMT: Move smt/control simple exit cases earlier
  cpu/SMT: Store the current/max number of threads
  cpu/SMT: Create topology_smt_thread_allowed()
  cpu/SMT: Allow enabling partial SMT states via sysfs
  powerpc/pseries: Initialise CPU hotplug callbacks earlier
  powerpc: Add HOTPLUG_SMT support
  powerpc/pseries: Honour current SMT state when DLPAR onlining CPUs

 .../ABI/testing/sysfs-devices-system-cpu      |   1 +
 .../admin-guide/kernel-parameters.txt         |   4 +-
 arch/Kconfig                                  |   3 +
 arch/powerpc/Kconfig                          |   2 +
 arch/powerpc/include/asm/topology.h           |  15 +++
 arch/powerpc/kernel/smp.c                     |   8 +-
 arch/powerpc/platforms/pseries/hotplug-cpu.c  |  30 +++--
 arch/powerpc/platforms/pseries/pseries.h      |   2 +
 arch/powerpc/platforms/pseries/setup.c        |   2 +
 arch/x86/include/asm/topology.h               |   4 +-
 arch/x86/kernel/cpu/bugs.c                    |   3 +-
 arch/x86/kernel/smpboot.c                     |   8 --
 include/linux/cpu.h                           |  25 +---
 include/linux/cpu_smt.h                       |  33 +++++
 kernel/cpu.c                                  | 116 ++++++++++++++----
 15 files changed, 185 insertions(+), 71 deletions(-)
 create mode 100644 include/linux/cpu_smt.h

-- 
2.41.0

> On 28-Jun-2023, at 3:35 PM, Laurent Dufour <ldufour@linux.ibm.com> wrote:
> 
> I'm taking over the series Michael sent previously [1] which is smartly
> reviewing the initial series I sent [2].  This series is addressing the
> comments sent by Thomas and me on the Michael's one.
> 
> Here is a short introduction to the issue this series is addressing:
> 
> When a new CPU is added, the kernel is activating all its threads. This
> leads to weird, but functional, result when adding CPU on a SMT 4 system
> for instance.
> 
> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
> active (system has been booted with the 'smt-enabled=4' kernel option):
> 
> ltcden3-lp12:~ # ppc64_cpu --info
> Core   0:    0*    1*    2*    3*    4     5     6     7
> Core   1:    8*    9*   10*   11*   12*   13*   14*   15*
> 
> This mixed SMT level may confused end users and/or some applications.
> 

Thanks for the patches Laurent.

Is the SMT level retained even when dynamically changing SMT values?
I am observing difference in behaviour with and without smt-enabled
kernel command line option.

When smt-enabled= option is specified SMT level is retained across 
cpu core remove and add.

Without this option but changing SMT level during runtime using
ppc64_cpu —smt=<level>, the SMT level is not retained after
cpu core add.

[root@ltcden8-lp8 ~]# ppc64_cpu —smt=4
[root@ltcden8-lp8 ~]# ppc64_cpu --info
Core   0:    0*    1*    2*    3*    4     5     6     7   
Core   1:    8*    9*   10*   11*   12    13    14    15   
Core   2:   16*   17*   18*   19*   20    21    22    23   
Core   3:   24*   25*   26*   27*   28    29    30    31   

Remove a core, SMT level is retained.

[root@ltcden8-lp8 ~]# ppc64_cpu --info
Core   0:    0*    1*    2*    3*    4     5     6     7  
Core   1:    8*    9*   10*   11*   12    13    14    15  
Core   2:   16*   17*   18*   19*   20    21    22    23  
[root@ltcden8-lp8 ~]#  

Add 3 cores, SMT level is not retained.

[  496.600648] Fallback order for Node 1: 1 0 
[  496.600655] Built 1 zonelists, mobility grouping on.  Total pages: 1228159
[  496.600676] Policy zone: Normal
[  496.661173] WARNING: workqueue cpumask: online intersect > possible intersect
[  499.530646] Fallback order for Node 3: 3 0 
[  499.530655] Built 2 zonelists, mobility grouping on.  Total pages: 1228159
[  499.530675] Policy zone: Normal

ppc64_cpu --info
Core   0:    0*    1*    2*    3*    4     5     6     7  
Core   1:    8*    9*   10*   11*   12    13    14    15  
Core   2:   16*   17*   18*   19*   20    21    22    23  
Core   3:   24*   25*   26*   27*   28*   29*   30*   31* 
Core   4:   32*   33*   34*   35*   36*   37*   38*   39* 
Core   5:   40*   41*   42*   43*   44*   45*   46*   47*
 [root@ltcden8-lp8 ~]# 

- Sachin

Sachin Sant <sachinp@linux.ibm.com> writes:
>> On 28-Jun-2023, at 3:35 PM, Laurent Dufour <ldufour@linux.ibm.com> wrote:
>> 
>> I'm taking over the series Michael sent previously [1] which is smartly
>> reviewing the initial series I sent [2].  This series is addressing the
>> comments sent by Thomas and me on the Michael's one.
>> 
>> Here is a short introduction to the issue this series is addressing:
>> 
>> When a new CPU is added, the kernel is activating all its threads. This
>> leads to weird, but functional, result when adding CPU on a SMT 4 system
>> for instance.
>> 
>> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
>> active (system has been booted with the 'smt-enabled=4' kernel option):
>> 
>> ltcden3-lp12:~ # ppc64_cpu --info
>> Core   0:    0*    1*    2*    3*    4     5     6     7
>> Core   1:    8*    9*   10*   11*   12*   13*   14*   15*
>> 
>> This mixed SMT level may confused end users and/or some applications.
>> 
>
> Thanks for the patches Laurent.
>
> Is the SMT level retained even when dynamically changing SMT values?
> I am observing difference in behaviour with and without smt-enabled
> kernel command line option.
>
> When smt-enabled= option is specified SMT level is retained across 
> cpu core remove and add.
>
> Without this option but changing SMT level during runtime using
> ppc64_cpu —smt=<level>, the SMT level is not retained after
> cpu core add.

That's because ppc64_cpu is not using the sysfs SMT control file, it's
just onlining/offlining threads manually.

If you run:
 $ ppc64_cpu --smt=4 

And then also do:

 $ echo 4 > /sys/devices/system/cpu/smt/control

It should work as expected?

ppc64_cpu will need to be updated to do that automatically.

cheers

Le 29/06/2023 à 13:10, Michael Ellerman a écrit :
> Sachin Sant <sachinp@linux.ibm.com> writes:
>>> On 28-Jun-2023, at 3:35 PM, Laurent Dufour <ldufour@linux.ibm.com> wrote:
>>>
>>> I'm taking over the series Michael sent previously [1] which is smartly
>>> reviewing the initial series I sent [2].  This series is addressing the
>>> comments sent by Thomas and me on the Michael's one.
>>>
>>> Here is a short introduction to the issue this series is addressing:
>>>
>>> When a new CPU is added, the kernel is activating all its threads. This
>>> leads to weird, but functional, result when adding CPU on a SMT 4 system
>>> for instance.
>>>
>>> Here the newly added CPU 1 has 8 threads while the other one has 4 threads
>>> active (system has been booted with the 'smt-enabled=4' kernel option):
>>>
>>> ltcden3-lp12:~ # ppc64_cpu --info
>>> Core   0:    0*    1*    2*    3*    4     5     6     7
>>> Core   1:    8*    9*   10*   11*   12*   13*   14*   15*
>>>
>>> This mixed SMT level may confused end users and/or some applications.
>>>
>>
>> Thanks for the patches Laurent.
>>
>> Is the SMT level retained even when dynamically changing SMT values?
>> I am observing difference in behaviour with and without smt-enabled
>> kernel command line option.
>>
>> When smt-enabled= option is specified SMT level is retained across
>> cpu core remove and add.
>>
>> Without this option but changing SMT level during runtime using
>> ppc64_cpu —smt=<level>, the SMT level is not retained after
>> cpu core add.
> 
> That's because ppc64_cpu is not using the sysfs SMT control file, it's
> just onlining/offlining threads manually.
> 
> If you run:
>   $ ppc64_cpu --smt=4
> 
> And then also do:
> 
>   $ echo 4 > /sys/devices/system/cpu/smt/control
> 
> It should work as expected?
> 
> ppc64_cpu will need to be updated to do that automatically.

Hi Sachin and Michael,

Yes, ppc64_cpu will need an update, and I have a patch ready to be sent 
once this series will be accepted.

By the way, I've a fix for the build issue reported against the patch 
6/9. I'll send a v3 soon.

Cheers,

Laurent.

>> 
>> Without this option but changing SMT level during runtime using
>> ppc64_cpu —smt=<level>, the SMT level is not retained after
>> cpu core add.
> 
> That's because ppc64_cpu is not using the sysfs SMT control file, it's
> just onlining/offlining threads manually.
> 
> If you run:
> $ ppc64_cpu --smt=4 
> 
> And then also do:
> 
> $ echo 4 > /sys/devices/system/cpu/smt/control
> 
> It should work as expected?

Thanks Michael. Yes this works. The SMT level is preserved
after a core add.

- Sachin