Sleeping inside arm_pmu_acpi_cpu_starting() while running a preemp_rt
kernel was reported at [1], and a solution was suggested at [2].

It seems [2] doesn't work because of the following:
"""PREPARE: The callbacks are invoked on a control CPU before the
hotplugged CPU is started up or after the hotplugged CPU has died."""

Indeed the 'prepare' callbacks are executed on the control CPU and
this CPU cannot remotely read the cpuid (cf read_cpuid_id()) of the
other CPUs.

Another solution:
1. count the number of different PMU IRQs (#IRQs)
2. allocate #IRQs pmu structures. There is at most #IRQs PMUs
3. in arm_pmu_acpi_cpu_starting() subcalls, after checking the cpuid,
   decide to use or re-use a pre-allocated pmu structure.
   Thus the pre-allocated pmu struct can be seen as a pool and all
   the struct might not be used.
4. free the unused pmu structures when probing
is suggested in this patch.

Freeing the unused pmu structures (4.) could maybe be done in a
CPUHP_AP_ONLINE_DYN hotplug callback instead, but only one CPU is
needed to search/free unused pre-allocated pmu structures.


This bug is the last remaining among the ones reported at [1]:
- [SPLAT 2/3] irqchip/gic-v3-its: Sleeping spinlocks down gic_reserve_range()
  Fixed with [3].
- [SPLAT 3/3] gpio: dwapb: Sleeping spinlocks down IRQ mapping
  Fixed with [4].

[1] https://lore.kernel.org/all/20210810134127.1394269-2-valentin.schneider@arm.com
[2] https://lore.kernel.org/all/87y299oyyq.ffs@tglx
[3] https://lore.kernel.org/lkml/20211027151506.2085066-3-valentin.schneider@arm.com/
[4 ]https://lore.kernel.org/all/20220419012810.88417-1-schspa@gmail.com/

Pierre Gondois (1):
  arm_pmu: acpi: Pre-allocate pmu structures

 drivers/perf/arm_pmu.c       |  17 +-----
 drivers/perf/arm_pmu_acpi.c  | 114 ++++++++++++++++++++++++++++++-----
 include/linux/perf/arm_pmu.h |   1 -
 3 files changed, 103 insertions(+), 29 deletions(-)

-- 
2.25.1

On Mon, Sep 12, 2022 at 05:51:03PM +0200, Pierre Gondois wrote:
> Sleeping inside arm_pmu_acpi_cpu_starting() while running a preemp_rt
> kernel was reported at [1], and a solution was suggested at [2].
> 
> It seems [2] doesn't work because of the following:
> """PREPARE: The callbacks are invoked on a control CPU before the
> hotplugged CPU is started up or after the hotplugged CPU has died."""
> 
> Indeed the 'prepare' callbacks are executed on the control CPU and
> this CPU cannot remotely read the cpuid (cf read_cpuid_id()) of the
> other CPUs.
> 
> Another solution:
> 1. count the number of different PMU IRQs (#IRQs)
> 2. allocate #IRQs pmu structures. There is at most #IRQs PMUs
> 3. in arm_pmu_acpi_cpu_starting() subcalls, after checking the cpuid,
>    decide to use or re-use a pre-allocated pmu structure.
>    Thus the pre-allocated pmu struct can be seen as a pool and all
>    the struct might not be used.
> 4. free the unused pmu structures when probing
> is suggested in this patch.
> 
> Freeing the unused pmu structures (4.) could maybe be done in a
> CPUHP_AP_ONLINE_DYN hotplug callback instead, but only one CPU is
> needed to search/free unused pre-allocated pmu structures.

I vaguely remember Mark R having a look at this at the time, so I'd like
his Ack on your approach before I queue anything.

Will