If there are shared processor LPARs, underlying Hypervisor can have more
virtual cores to handle than actual physical cores.
Starting with Power 9, a big core (aka SMT8 core) has 2 nearly
independent thread groups. On a shared processors LPARs, it helps to
pack threads to lesser number of cores so that the overall system
performance and utilization improves. PowerVM schedules at a big core
level. Hence packing to fewer cores helps.
For example: Lets says there are two 8-core Shared LPARs that are
actually sharing a 8 Core shared physical pool, each running 8 threads
each. Then Consolidating 8 threads to 4 cores on each LPAR would help
them to perform better. This is because each of the LPAR will get
100% time to run applications and there will no switching required by
the Hypervisor.
To achieve this, enable SD_ASYM_PACKING flag at CACHE, MC and DIE level
when the system is running in shared processor mode and has big cores.
Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
---
Changelog:
v3 -> v4:
- Dont use splpar_asym_pack with SMT
- Conflict resolution due to rebase
(DIE changed to PKG)
v2 -> v3:
- Handle comments from Michael Ellerman.
- Rework using existing cpu_has_features static key
v1->v2: Using Jump label instead of a variable.
arch/powerpc/kernel/smp.c | 37 +++++++++++++++++++++++++++++--------
1 file changed, 29 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index ab691c89d787..69a3262024f1 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -993,16 +993,20 @@ static bool shared_caches;
/* cpumask of CPUs with asymmetric SMT dependency */
static int powerpc_smt_flags(void)
{
- int flags = SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES;
+ if (!cpu_has_feature(CPU_FTR_ASYM_SMT))
+ return SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES;
- if (cpu_has_feature(CPU_FTR_ASYM_SMT)) {
- printk_once(KERN_INFO "Enabling Asymmetric SMT scheduling\n");
- flags |= SD_ASYM_PACKING;
- }
- return flags;
+ return SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING;
}
#endif
+/*
+ * On shared processor LPARs scheduled on a big core (which has two or more
+ * independent thread groups per core), prefer lower numbered CPUs, so
+ * that workload consolidates to lesser number of cores.
+ */
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(splpar_asym_pack);
+
/*
* P9 has a slightly odd architecture where pairs of cores share an L2 cache.
* This topology makes it *much* cheaper to migrate tasks between adjacent cores
@@ -1011,9 +1015,20 @@ static int powerpc_smt_flags(void)
*/
static int powerpc_shared_cache_flags(void)
{
+ if (static_branch_unlikely(&splpar_asym_pack))
+ return SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING;
+
return SD_SHARE_PKG_RESOURCES;
}
+static int powerpc_shared_proc_flags(void)
+{
+ if (static_branch_unlikely(&splpar_asym_pack))
+ return SD_ASYM_PACKING;
+
+ return 0;
+}
+
/*
* We can't just pass cpu_l2_cache_mask() directly because
* returns a non-const pointer and the compiler barfs on that.
@@ -1050,8 +1065,8 @@ static struct sched_domain_topology_level powerpc_topology[] = {
{ cpu_smt_mask, powerpc_smt_flags, SD_INIT_NAME(SMT) },
#endif
{ shared_cache_mask, powerpc_shared_cache_flags, SD_INIT_NAME(CACHE) },
- { cpu_mc_mask, SD_INIT_NAME(MC) },
- { cpu_cpu_mask, SD_INIT_NAME(PKG) },
+ { cpu_mc_mask, powerpc_shared_proc_flags, SD_INIT_NAME(MC) },
+ { cpu_cpu_mask, powerpc_shared_proc_flags, SD_INIT_NAME(PKG) },
{ NULL, },
};
@@ -1686,7 +1701,13 @@ static void __init fixup_topology(void)
{
int i;
+ if (is_shared_processor() && has_big_cores)
+ static_branch_enable(&splpar_asym_pack);
+
#ifdef CONFIG_SCHED_SMT
+ if (cpu_has_feature(CPU_FTR_ASYM_SMT))
+ pr_info_once("Enabling Asymmetric SMT scheduling\n");
+
if (has_big_cores) {
pr_info("Big cores detected but using small core scheduling\n");
powerpc_topology[smt_idx].mask = smallcore_smt_mask;
--
2.31.1Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes:
> If there are shared processor LPARs, underlying Hypervisor can have more
> virtual cores to handle than actual physical cores.
>
> Starting with Power 9, a big core (aka SMT8 core) has 2 nearly
> independent thread groups. On a shared processors LPARs, it helps to
> pack threads to lesser number of cores so that the overall system
> performance and utilization improves. PowerVM schedules at a big core
> level. Hence packing to fewer cores helps.
>
> For example: Lets says there are two 8-core Shared LPARs that are
> actually sharing a 8 Core shared physical pool, each running 8 threads
> each. Then Consolidating 8 threads to 4 cores on each LPAR would help
> them to perform better. This is because each of the LPAR will get
> 100% time to run applications and there will no switching required by
> the Hypervisor.
>
> To achieve this, enable SD_ASYM_PACKING flag at CACHE, MC and DIE level
> when the system is running in shared processor mode and has big cores.
>
> Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
> ---
> Changelog:
> v3 -> v4:
> - Dont use splpar_asym_pack with SMT
> - Conflict resolution due to rebase
> (DIE changed to PKG)
> v2 -> v3:
> - Handle comments from Michael Ellerman.
> - Rework using existing cpu_has_features static key
> v1->v2: Using Jump label instead of a variable.
>
> arch/powerpc/kernel/smp.c | 37 +++++++++++++++++++++++++++++--------
> 1 file changed, 29 insertions(+), 8 deletions(-)
>
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index ab691c89d787..69a3262024f1 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -993,16 +993,20 @@ static bool shared_caches;
> /* cpumask of CPUs with asymmetric SMT dependency */
> static int powerpc_smt_flags(void)
> {
> - int flags = SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES;
> + if (!cpu_has_feature(CPU_FTR_ASYM_SMT))
> + return SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES;
>
> - if (cpu_has_feature(CPU_FTR_ASYM_SMT)) {
> - printk_once(KERN_INFO "Enabling Asymmetric SMT scheduling\n");
> - flags |= SD_ASYM_PACKING;
> - }
> - return flags;
> + return SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING;
> }
> #endif
>
Only relevant change there is dropping printk_once(). Rest of the
changes are not needed?
-aneesh
* Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> [2023-11-15 12:05:22]:
> Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes:
>
> >
> > arch/powerpc/kernel/smp.c | 37 +++++++++++++++++++++++++++++--------
> > 1 file changed, 29 insertions(+), 8 deletions(-)
> >
> > diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> > index ab691c89d787..69a3262024f1 100644
> > --- a/arch/powerpc/kernel/smp.c
> > +++ b/arch/powerpc/kernel/smp.c
> > @@ -993,16 +993,20 @@ static bool shared_caches;
> > /* cpumask of CPUs with asymmetric SMT dependency */
> > static int powerpc_smt_flags(void)
> > {
> > - int flags = SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES;
> > + if (!cpu_has_feature(CPU_FTR_ASYM_SMT))
> > + return SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES;
> >
> > - if (cpu_has_feature(CPU_FTR_ASYM_SMT)) {
> > - printk_once(KERN_INFO "Enabling Asymmetric SMT scheduling\n");
> > - flags |= SD_ASYM_PACKING;
> > - }
> > - return flags;
> > + return SD_SHARE_CPUCAPACITY | SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING;
> > }
> > #endif
> >
>
> Only relevant change there is dropping printk_once(). Rest of the
> changes are not needed?
>
> -aneesh
If you are looking at just this hunk, then yes its moving the printk_once to
another function.
--
Thanks and Regards
Srikar Dronamraju
Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes: > If there are shared processor LPARs, underlying Hypervisor can have more > virtual cores to handle than actual physical cores. > > Starting with Power 9, a big core (aka SMT8 core) has 2 nearly > independent thread groups. On a shared processors LPARs, it helps to > pack threads to lesser number of cores so that the overall system > performance and utilization improves. PowerVM schedules at a big core > level. Hence packing to fewer cores helps. > > For example: Lets says there are two 8-core Shared LPARs that are > actually sharing a 8 Core shared physical pool, each running 8 threads > each. Then Consolidating 8 threads to 4 cores on each LPAR would help > them to perform better. This is because each of the LPAR will get > 100% time to run applications and there will no switching required by > the Hypervisor. > Will this patch consolidate things to first 8 threads or just the one Big core? /me continues to look at other patches and wonder whether 4/5 should come before this? > > To achieve this, enable SD_ASYM_PACKING flag at CACHE, MC and DIE level > when the system is running in shared processor mode and has big cores. > > Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com> -aneesh
* Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> [2023-11-15 10:57:08]: > Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes: > > > If there are shared processor LPARs, underlying Hypervisor can have more > > virtual cores to handle than actual physical cores. > > > > Starting with Power 9, a big core (aka SMT8 core) has 2 nearly > > independent thread groups. On a shared processors LPARs, it helps to > > pack threads to lesser number of cores so that the overall system > > performance and utilization improves. PowerVM schedules at a big core > > level. Hence packing to fewer cores helps. > > > > For example: Lets says there are two 8-core Shared LPARs that are > > actually sharing a 8 Core shared physical pool, each running 8 threads > > each. Then Consolidating 8 threads to 4 cores on each LPAR would help > > them to perform better. This is because each of the LPAR will get > > 100% time to run applications and there will no switching required by > > the Hypervisor. > > > > Will this patch consolidate things to first 8 threads or just the one > Big core? /me continues to look at other patches and wonder whether 4/5 > should come before this? It will consolidate 1 thread per small core aka SMT domain or 2 threads per Big core. If the load is such that there are more unbound threads than SMT domains, asym packing will not kick-in. 4/5 would make sense only once we enable asym_packing above SMT domain. > > > > > > To achieve this, enable SD_ASYM_PACKING flag at CACHE, MC and DIE level > > when the system is running in shared processor mode and has big cores. > > > > Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com> > > > -aneesh -- Thanks and Regards Srikar Dronamraju
© 2016 - 2025 Red Hat, Inc.