arch/x86/events/rapl.c | 418 +++++++++++++++++++++++++++++------------ 1 file changed, 297 insertions(+), 121 deletions(-)
Currently the energy-cores event in the power PMU aggregates energy consumption data at a package level. On the other hand the core energy RAPL counter in AMD CPUs has a core scope (which means the energy consumption is recorded separately for each core). Earlier efforts to add the core event in the power PMU had failed [1], due to the difference in the scope of these two events. Hence, there is a need for a new core scope PMU. This patchset adds a new "power_per_core" PMU alongside the existing "power" PMU, which will be responsible for collecting the new "energy-per-core" event. Tested the package level and core level PMU counters with workloads pinned to different CPUs. Results with workload pinned to CPU 1 in Core 1 on an AMD Zen4 Genoa machine: $ perf stat -a --per-core -e power_per_core/energy-per-core/ sleep 1 Performance counter stats for 'system wide': S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C1 1 5.72 Joules power_per_core/energy-per-core/ S0-D0-C2 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C3 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C4 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C5 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C6 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C7 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C8 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C9 1 0.02 Joules power_per_core/energy-per-core/ S0-D0-C10 1 0.02 Joules power_per_core/energy-per-core/ [1]: https://lore.kernel.org/lkml/3e766f0e-37d4-0f82-3868-31b14228868d@linux.intel.com/ This patchset applies cleanly on top of v6.10-rc4 as well as latest tip/master. v2 changes: * Patches 6,7,8 added to split some changes out of the last patch * Use container_of to get the rapl_pmus from event variable (Rui) * Set PERF_EV_CAP_READ_ACTIVE_PKG flag only for pkg scope PMU (Rui) * Use event id 0x1 for energy-per-core event (Rui) * Use PERF_RAPL_PER_CORE bit instead of adding a new flag to check for per-core counter hw support (Rui) Dhananjay Ugwekar (9): perf/x86/rapl: Fix the energy-pkg event for AMD CPUs perf/x86/rapl: Rename rapl_pmu variables perf/x86/rapl: Make rapl_model struct global perf/x86/rapl: Move cpumask variable to rapl_pmus struct perf/x86/rapl: Add wrapper for online/offline functions perf/x86/rapl: Add an argument to the cleanup and init functions perf/x86/rapl: Modify the generic variable names to *_pkg* perf/x86/rapl: Remove the global variable rapl_msrs perf/x86/rapl: Add per-core energy counter support for AMD CPUs arch/x86/events/rapl.c | 418 +++++++++++++++++++++++++++++------------ 1 file changed, 297 insertions(+), 121 deletions(-) -- 2.34.1
Hello Dhananjay,
On 6/20/2024 6:26 PM, Dhananjay Ugwekar wrote:
> Currently the energy-cores event in the power PMU aggregates energy
> consumption data at a package level. On the other hand the core energy
> RAPL counter in AMD CPUs has a core scope (which means the energy
> consumption is recorded separately for each core). Earlier efforts to add
> the core event in the power PMU had failed [1], due to the difference in
> the scope of these two events. Hence, there is a need for a new core scope
> PMU.
>
> This patchset adds a new "power_per_core" PMU alongside the existing
> "power" PMU, which will be responsible for collecting the new
> "energy-per-core" event.
>
> Tested the package level and core level PMU counters with workloads
> pinned to different CPUs.
>
> Results with workload pinned to CPU 1 in Core 1 on an AMD Zen4 Genoa
> machine:
>
> $ perf stat -a --per-core -e power_per_core/energy-per-core/ sleep 1
When testing this on a 2P 3rd Generation EPYC System (2 x 64/128T), I
ran into an issue where it seems like the energy reporting for the
system is coming from the second socket. Following are the CPUs on each
socket of the system:
Node 0: 0-63, 128-191
Node 1: 64-127, 192-255
Following are the experiments I ran:
$ # Run a busy loop on each thread of the first socket
$ for i in `seq 0 63` `seq 128 191`; do taskset -c $i ~/scripts/loop & done
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5
S0-D0-C0 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C1 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C0 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C1 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/
From the energy data, it looks as if the system is entirely idle.
If I repeat the same, pinning the running busy loop on the threads of
second socket, I see the following:
$ # Run a busy loop on each thread of the second socket
$ for i in `seq 64 127` `seq 192 255`; do taskset -c $i ~/scripts/loop & done
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5
S0-D0-C0 1 11.79 Joules power_per_core/energy-per-core/
S0-D0-C1 1 11.80 Joules power_per_core/energy-per-core/
S0-D0-C2 1 11.90 Joules power_per_core/energy-per-core/
S0-D0-C3 1 11.88 Joules power_per_core/energy-per-core/
...
S0-D0-C63 1 11.76 Joules power_per_core/energy-per-core/
S1-D1-C0 1 11.81 Joules power_per_core/energy-per-core/
S1-D1-C1 1 11.80 Joules power_per_core/energy-per-core/
S1-D1-C2 1 11.90 Joules power_per_core/energy-per-core/
S1-D1-C3 1 11.88 Joules power_per_core/energy-per-core/
...
S1-D1-C63 1 11.76 Joules power_per_core/energy-per-core/
The whole system seems to be busy this time around. I've verified that
only half the system is busy using htop in either case.
Running some more experiments, I see the following:
$ taskset -c 1 ~/scripts/loop& # First thread from Core 1, Socket
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5
S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C1 1 0.21 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.20 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/
...
(Seemingly idle system)
$ taskset -c 65 ~/scripts/loop&
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5
S0-D0-C0 1 0.01 Joules power_per_core/energy-per-core/
S0-D0-C1 1 16.73 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C0 1 0.01 Joules power_per_core/energy-per-core/
S1-D1-C1 1 16.73 Joules power_per_core/energy-per-core/
S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/
(Core 1 from both sockets look busy reporting identical energy
values)
Hope it helps narrow down the issue.
>
> Performance counter stats for 'system wide':
>
> S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C1 1 5.72 Joules power_per_core/energy-per-core/
> S0-D0-C2 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C3 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C4 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C5 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C6 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C7 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C8 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C9 1 0.02 Joules power_per_core/energy-per-core/
> S0-D0-C10 1 0.02 Joules power_per_core/energy-per-core/
>
> [1]: https://lore.kernel.org/lkml/3e766f0e-37d4-0f82-3868-31b14228868d@linux.intel.com/
>
> This patchset applies cleanly on top of v6.10-rc4 as well as latest
> tip/master.
P.S. I tested these changes on top of tip:perf/core
>
> [..snip..]
>
--
Thanks and Regards,
Prateek
Hello Prateek, On 6/21/2024 1:54 PM, K Prateek Nayak wrote: > Hello Dhananjay, > > On 6/20/2024 6:26 PM, Dhananjay Ugwekar wrote: >> Currently the energy-cores event in the power PMU aggregates energy >> consumption data at a package level. On the other hand the core energy >> RAPL counter in AMD CPUs has a core scope (which means the energy >> consumption is recorded separately for each core). Earlier efforts to add >> the core event in the power PMU had failed [1], due to the difference in >> the scope of these two events. Hence, there is a need for a new core scope >> PMU. >> >> This patchset adds a new "power_per_core" PMU alongside the existing >> "power" PMU, which will be responsible for collecting the new >> "energy-per-core" event. >> >> Tested the package level and core level PMU counters with workloads >> pinned to different CPUs. >> >> Results with workload pinned to CPU 1 in Core 1 on an AMD Zen4 Genoa >> machine: >> >> $ perf stat -a --per-core -e power_per_core/energy-per-core/ sleep 1 > > When testing this on a 2P 3rd Generation EPYC System (2 x 64/128T), I > ran into an issue where it seems like the energy reporting for the > system is coming from the second socket. Following are the CPUs on each > socket of the system: > > Node 0: 0-63, 128-191 > Node 1: 64-127, 192-255 > > Following are the experiments I ran: > > $ # Run a busy loop on each thread of the first socket > $ for i in `seq 0 63` `seq 128 191`; do taskset -c $i ~/scripts/loop & done > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C0 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C1 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/ > > From the energy data, it looks as if the system is entirely idle. > > If I repeat the same, pinning the running busy loop on the threads of > second socket, I see the following: > > $ # Run a busy loop on each thread of the second socket > $ for i in `seq 64 127` `seq 192 255`; do taskset -c $i ~/scripts/loop & done > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 11.79 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 11.80 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 11.90 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 11.88 Joules power_per_core/energy-per-core/ > ... > S0-D0-C63 1 11.76 Joules power_per_core/energy-per-core/ > S1-D1-C0 1 11.81 Joules power_per_core/energy-per-core/ > S1-D1-C1 1 11.80 Joules power_per_core/energy-per-core/ > S1-D1-C2 1 11.90 Joules power_per_core/energy-per-core/ > S1-D1-C3 1 11.88 Joules power_per_core/energy-per-core/ > ... > S1-D1-C63 1 11.76 Joules power_per_core/energy-per-core/ > > The whole system seems to be busy this time around. I've verified that > only half the system is busy using htop in either case. > > Running some more experiments, I see the following: > > $ taskset -c 1 ~/scripts/loop& # First thread from Core 1, Socket > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 0.21 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.20 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > (Seemingly idle system) > > > $ taskset -c 65 ~/scripts/loop& > $ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5 > > S0-D0-C0 1 0.01 Joules power_per_core/energy-per-core/ > S0-D0-C1 1 16.73 Joules power_per_core/energy-per-core/ > S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/ > S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C0 1 0.01 Joules power_per_core/energy-per-core/ > S1-D1-C1 1 16.73 Joules power_per_core/energy-per-core/ > S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/ > S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/ > ... > S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/ > > (Core 1 from both sockets look busy reporting identical energy > values) > > Hope it helps narrow down the issue. I think my assumption that topology_core_id() will return a unique core ID across the system might not be correct. It seems the core ID is unique only within a package, will fix this in the next version. Thanks a lot for testing and helping narrow down the issue! Regards, Dhananjay > >> >> Performance counter stats for 'system wide': >> >> S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C1 1 5.72 Joules power_per_core/energy-per-core/ >> S0-D0-C2 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C3 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C4 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C5 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C6 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C7 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C8 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C9 1 0.02 Joules power_per_core/energy-per-core/ >> S0-D0-C10 1 0.02 Joules power_per_core/energy-per-core/ >> >> [1]: https://lore.kernel.org/lkml/3e766f0e-37d4-0f82-3868-31b14228868d@linux.intel.com/ >> >> This patchset applies cleanly on top of v6.10-rc4 as well as latest >> tip/master. > > P.S. I tested these changes on top of tip:perf/core > >> >> [..snip..] >> >
© 2016 - 2025 Red Hat, Inc.