Frame pointer callchains are not supported on s390. Ignore the option
and print a warning.
Signed-off-by: Ian Rogers <irogers@google.com>
---
v2: Only disable user callchains as AI is telling me native "kernel"
callchains are supported on s390.
---
tools/perf/util/evsel.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index bd14d9bbc91f..fa21b48cba86 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
attr->exclude_callchain_user = 1;
}
+ if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
+ !evsel->core.attr.exclude_callchain_user) {
+ pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
+ evsel->core.attr.exclude_callchain_user = 1;
+ }
+
if (param->defer && !attr->exclude_callchain_user)
attr->defer_callchain = 1;
}
--
2.53.0.851.ga537e3e6e9-googOn 3/12/26 07:16, Ian Rogers wrote:
> Frame pointer callchains are not supported on s390. Ignore the option
> and print a warning.
>
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
> v2: Only disable user callchains as AI is telling me native "kernel"
> callchains are supported on s390.
> ---
> tools/perf/util/evsel.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index bd14d9bbc91f..fa21b48cba86 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
> @@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> attr->exclude_callchain_user = 1;
> }
>
> + if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
> + !evsel->core.attr.exclude_callchain_user) {
> + pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
> + evsel->core.attr.exclude_callchain_user = 1;
> + }
> +
> if (param->defer && !attr->exclude_callchain_user)
> attr->defer_callchain = 1;
> }
Ian, thanks very much.
Your patch set helps a lot. However there is a small nit (which is mandatory). Please add these lines
evsel->core.attr.sample_type &= ~PERF_SAMPLE_CALLCHAIN;
evsel->core.attr.sample_type &= ~PERF_SAMPLE_REGS_USER;
evsel->core.attr.sample_type &= ~PERF_SAMPLE_STACK_USER;
to the new if(EM_HOST == ...) above.
The s390 CPU Measurement sampling device driver does not check on the attr.core.exclude_callchain_user
member, but on the sample_type bit mask. It returns -EOPNOTSUPP when this bit PERF_SAMPLE_CALLCHAIN
is set. This solves the invocation with command line flag -g as in
# ./perf record -v -e cycles -g -- perf test -w noploop
...
perf record: Captured and wrote 0.183 MB perf.data ]
Also I discovered that the fallback when using --call-graph dwarf command line flag still fails:
# ./perf record -v -e cycles --call-graph dwarf -- perf test -w noploop
...
Warning:
Trying to fall back to excluding guest samples
Error:
Failure to open event 'cycles:H' on PMU 'cpum_cf' which will be removed.
cycles:H: PMU Hardware doesn't support sampling overflow-interrupts. Try 'perf stat'
Error:
Failure to open any events for recording.
The reason is in __evsel__config_callchain() which calls evsel__set_sample_bit(evsel, CALLCHAIN)
and sets the PERF_SAMPLE_CALLCHAIN bit in evsel->core.attr.sample_type. It also sets the
member attr->exclude_callchain_user = 1 and sets bits REGS_USER and _STACK_USER.
All three bits are not supported by s390.
I have modified your 2nd patch and appended it.
I find all these bits in sample_type and the attr.exclude_XXX stuff very confusing. If there
is a more consistant way of checking these feature, please let me know.
Thanks again for looking into this.
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Wolfgang Wendt
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
On Thu, Mar 12, 2026 at 5:45 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
>
> On 3/12/26 07:16, Ian Rogers wrote:
> > Frame pointer callchains are not supported on s390. Ignore the option
> > and print a warning.
> >
> > Signed-off-by: Ian Rogers <irogers@google.com>
> > ---
> > v2: Only disable user callchains as AI is telling me native "kernel"
> > callchains are supported on s390.
> > ---
> > tools/perf/util/evsel.c | 6 ++++++
> > 1 file changed, 6 insertions(+)
> >
> > diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> > index bd14d9bbc91f..fa21b48cba86 100644
> > --- a/tools/perf/util/evsel.c
> > +++ b/tools/perf/util/evsel.c
> > @@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> > attr->exclude_callchain_user = 1;
> > }
> >
> > + if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
> > + !evsel->core.attr.exclude_callchain_user) {
> > + pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
> > + evsel->core.attr.exclude_callchain_user = 1;
> > + }
> > +
> > if (param->defer && !attr->exclude_callchain_user)
> > attr->defer_callchain = 1;
> > }
>
> Ian, thanks very much.
> Your patch set helps a lot. However there is a small nit (which is mandatory). Please add these lines
>
> evsel->core.attr.sample_type &= ~PERF_SAMPLE_CALLCHAIN;
> evsel->core.attr.sample_type &= ~PERF_SAMPLE_REGS_USER;
> evsel->core.attr.sample_type &= ~PERF_SAMPLE_STACK_USER;
So these lines are dropping callchain from the sample_type which means
the kernel stack won't be sampled. AI was telling me this worked, but
I'm guess it was wrong. I think rather than this it is just cleaner
never to set the bits in the perf_event_attr, more like what v1 of the
patch did:
https://lore.kernel.org/lkml/20260312031928.1494864-3-irogers@google.com/
> to the new if(EM_HOST == ...) above.
> The s390 CPU Measurement sampling device driver does not check on the attr.core.exclude_callchain_user
> member, but on the sample_type bit mask. It returns -EOPNOTSUPP when this bit PERF_SAMPLE_CALLCHAIN
> is set. This solves the invocation with command line flag -g as in
> # ./perf record -v -e cycles -g -- perf test -w noploop
> ...
> perf record: Captured and wrote 0.183 MB perf.data ]
Right because the callchain was removed from all the samples. We can't
fix old kernels (other than by using fix tags); is there a possibility
of adding the exclude_callchain_user to the s390 perf driver for the
sake of kernel callchains? It seems better than providing no
callchain.
> Also I discovered that the fallback when using --call-graph dwarf command line flag still fails:
> # ./perf record -v -e cycles --call-graph dwarf -- perf test -w noploop
> ...
> Warning:
> Trying to fall back to excluding guest samples
> Error:
> Failure to open event 'cycles:H' on PMU 'cpum_cf' which will be removed.
> cycles:H: PMU Hardware doesn't support sampling overflow-interrupts. Try 'perf stat'
> Error:
> Failure to open any events for recording.
>
> The reason is in __evsel__config_callchain() which calls evsel__set_sample_bit(evsel, CALLCHAIN)
> and sets the PERF_SAMPLE_CALLCHAIN bit in evsel->core.attr.sample_type. It also sets the
> member attr->exclude_callchain_user = 1 and sets bits REGS_USER and _STACK_USER.
> All three bits are not supported by s390.
I'm confused by this and your previous testing that showed the
`--call-graph dwarf` worked. You need the sampled registers for dwarf
unwinding to provide initial register values for the unwinder.
> I have modified your 2nd patch and appended it.
>
> I find all these bits in sample_type and the attr.exclude_XXX stuff very confusing. If there
> is a more consistant way of checking these feature, please let me know.
Ok, let me check it out.
> Thanks again for looking into this.
Thanks,
Ian
> --
> Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> --
> IBM Deutschland Research & Development GmbH
>
> Vorsitzender des Aufsichtsrats: Wolfgang Wendt
>
> Geschäftsführung: David Faller
>
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
On Thu, Mar 12, 2026 at 8:54 AM Ian Rogers <irogers@google.com> wrote:
>
> On Thu, Mar 12, 2026 at 5:45 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
> >
> > On 3/12/26 07:16, Ian Rogers wrote:
> > > Frame pointer callchains are not supported on s390. Ignore the option
> > > and print a warning.
> > >
> > > Signed-off-by: Ian Rogers <irogers@google.com>
> > > ---
> > > v2: Only disable user callchains as AI is telling me native "kernel"
> > > callchains are supported on s390.
> > > ---
> > > tools/perf/util/evsel.c | 6 ++++++
> > > 1 file changed, 6 insertions(+)
> > >
> > > diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> > > index bd14d9bbc91f..fa21b48cba86 100644
> > > --- a/tools/perf/util/evsel.c
> > > +++ b/tools/perf/util/evsel.c
> > > @@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> > > attr->exclude_callchain_user = 1;
> > > }
> > >
> > > + if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
> > > + !evsel->core.attr.exclude_callchain_user) {
> > > + pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
> > > + evsel->core.attr.exclude_callchain_user = 1;
> > > + }
> > > +
> > > if (param->defer && !attr->exclude_callchain_user)
> > > attr->defer_callchain = 1;
> > > }
> >
> > Ian, thanks very much.
> > Your patch set helps a lot. However there is a small nit (which is mandatory). Please add these lines
> >
> > evsel->core.attr.sample_type &= ~PERF_SAMPLE_CALLCHAIN;
> > evsel->core.attr.sample_type &= ~PERF_SAMPLE_REGS_USER;
> > evsel->core.attr.sample_type &= ~PERF_SAMPLE_STACK_USER;
>
> So these lines are dropping callchain from the sample_type which means
> the kernel stack won't be sampled. AI was telling me this worked, but
> I'm guess it was wrong. I think rather than this it is just cleaner
> never to set the bits in the perf_event_attr, more like what v1 of the
> patch did:
> https://lore.kernel.org/lkml/20260312031928.1494864-3-irogers@google.com/
>
> > to the new if(EM_HOST == ...) above.
> > The s390 CPU Measurement sampling device driver does not check on the attr.core.exclude_callchain_user
> > member, but on the sample_type bit mask. It returns -EOPNOTSUPP when this bit PERF_SAMPLE_CALLCHAIN
> > is set. This solves the invocation with command line flag -g as in
> > # ./perf record -v -e cycles -g -- perf test -w noploop
> > ...
> > perf record: Captured and wrote 0.183 MB perf.data ]
>
> Right because the callchain was removed from all the samples. We can't
> fix old kernels (other than by using fix tags); is there a possibility
> of adding the exclude_callchain_user to the s390 perf driver for the
> sake of kernel callchains? It seems better than providing no
> callchain.
>
> > Also I discovered that the fallback when using --call-graph dwarf command line flag still fails:
> > # ./perf record -v -e cycles --call-graph dwarf -- perf test -w noploop
> > ...
> > Warning:
> > Trying to fall back to excluding guest samples
> > Error:
> > Failure to open event 'cycles:H' on PMU 'cpum_cf' which will be removed.
> > cycles:H: PMU Hardware doesn't support sampling overflow-interrupts. Try 'perf stat'
> > Error:
> > Failure to open any events for recording.
> >
> > The reason is in __evsel__config_callchain() which calls evsel__set_sample_bit(evsel, CALLCHAIN)
> > and sets the PERF_SAMPLE_CALLCHAIN bit in evsel->core.attr.sample_type. It also sets the
> > member attr->exclude_callchain_user = 1 and sets bits REGS_USER and _STACK_USER.
> > All three bits are not supported by s390.
>
> I'm confused by this and your previous testing that showed the
> `--call-graph dwarf` worked. You need the sampled registers for dwarf
> unwinding to provide initial register values for the unwinder.
>
> > I have modified your 2nd patch and appended it.
> >
> > I find all these bits in sample_type and the attr.exclude_XXX stuff very confusing. If there
> > is a more consistant way of checking these feature, please let me know.
I forgot to mention, yeah the exclude thing is maddening. It takes
about 100 lines to convert the command line modifiers to those in the
perf_event_attr, there's a priority to them, and so on:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/parse-events.c#n1759
Fwiw, I was thinking for patch 1 of holding onto the parsed modifiers
so that we could reset the excludes based on them when switching to
the software event.
> Ok, let me check it out.
So looking at the cpum_cf driver it fails events for having sampling enabled:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_cf.c#n859
```
if (is_sampling_event(event)) /* No sampling support */
```
and the cpum_sf driver fails for any kind of callchain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_sf.c#n839
```
static bool is_callchain_event(struct perf_event *event)
{
u64 sample_type = event->attr.sample_type;
return sample_type & (PERF_SAMPLE_CALLCHAIN | PERF_SAMPLE_REGS_USER |
PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_STACK_USER);
}
static int cpumsf_pmu_event_init(struct perf_event *event)
{
int err;
/* No support for taken branch sampling */
/* No support for callchain, stacks and registers */
if (has_branch_stack(event) || is_callchain_event(event))
return -EOPNOTSUPP;
```
Perhaps there is an oversight in the cpum_cf driver wrt branch stacks
(LBR on x86). The PERF_SAMPLE_CALLCHAIN bit is set for perf call-graph
options currently:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/evsel.c#n1024
I think cpum_sf is the PMU we care about for the default cycles event?
Since callchains of any flavor don't work with cpum_sf we can do a few
things:
1) Disable the callchain and allow hardware sampling to continue,
2) Switch to a software event like cpu-clock,
3) Fail for the callchain option with this PMU, which is currently
happening anyway.
I dislike option 3 because it requires special s390 logic for many
tests, and we lose testing coverage of hardware events. Option 1 is a
smaller patch, an early return in __evsel__config_callchain if on
s390. Option 2 feels most like what the user would want given they
asked for a callchain. We could change evlist__new_default to take a
"with callchain" boolean and on s390 that could switch the event to a
software one. Only two non-test callers use evlist__new__default
(record and top), so it isn't a huge change.
Wdyt Thomas, option 1 or 2?
Thanks,
Ian
> > Thanks again for looking into this.
>
> Thanks,
> Ian
>
> > --
> > Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> > --
> > IBM Deutschland Research & Development GmbH
> >
> > Vorsitzender des Aufsichtsrats: Wolfgang Wendt
> >
> > Geschäftsführung: David Faller
> >
> > Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
On 3/12/26 17:46, Ian Rogers wrote:
> On Thu, Mar 12, 2026 at 8:54 AM Ian Rogers <irogers@google.com> wrote:
>>
>> On Thu, Mar 12, 2026 at 5:45 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
>>>
>>> On 3/12/26 07:16, Ian Rogers wrote:
>>>> Frame pointer callchains are not supported on s390. Ignore the option
>>>> and print a warning.
>>>>
>>>> Signed-off-by: Ian Rogers <irogers@google.com>
>>>> ---
>>>> v2: Only disable user callchains as AI is telling me native "kernel"
>>>> callchains are supported on s390.
>>>> ---
>>>> tools/perf/util/evsel.c | 6 ++++++
>>>> 1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
>>>> index bd14d9bbc91f..fa21b48cba86 100644
>>>> --- a/tools/perf/util/evsel.c
>>>> +++ b/tools/perf/util/evsel.c
>>>> @@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
>>>> attr->exclude_callchain_user = 1;
>>>> }
>>>>
>>>> + if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
>>>> + !evsel->core.attr.exclude_callchain_user) {
>>>> + pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
>>>> + evsel->core.attr.exclude_callchain_user = 1;
>>>> + }
>>>> +
>>>> if (param->defer && !attr->exclude_callchain_user)
>>>> attr->defer_callchain = 1;
>>>> }
>>>
>>> Ian, thanks very much.
>>> Your patch set helps a lot. However there is a small nit (which is mandatory). Please add these lines
>>>
>>> evsel->core.attr.sample_type &= ~PERF_SAMPLE_CALLCHAIN;
>>> evsel->core.attr.sample_type &= ~PERF_SAMPLE_REGS_USER;
>>> evsel->core.attr.sample_type &= ~PERF_SAMPLE_STACK_USER;
>>
>> So these lines are dropping callchain from the sample_type which means
>> the kernel stack won't be sampled. AI was telling me this worked, but
>> I'm guess it was wrong. I think rather than this it is just cleaner
>> never to set the bits in the perf_event_attr, more like what v1 of the
>> patch did:
>> https://lore.kernel.org/lkml/20260312031928.1494864-3-irogers@google.com/
>>
Let me try to answer your and Namhyung questions in one reply.
s390 has many PMUs, there is one for counting (cpum_cf) and one for sampling (cpum_sf),
which are totally different hardware components.
Hardware sampling
1. writes into a large buffer, data is timestamp and IP (instruction pointer), no registers, no PID.
2. when buffer gets full an interrupt occurs and the interrupt handler scans the
hardware samples and converts valid samples to perf format and saves them in perf ring buffer.
This means we can not reconstruct register values for an individual sample. Current Pid is saved by
other means somewhere and added to perf data during perf ring buffer write.
For s390 when we need callchains, we always use event cpu-clock, never hardware event cycles.
>>> to the new if(EM_HOST == ...) above.
>>> The s390 CPU Measurement sampling device driver does not check on the attr.core.exclude_callchain_user
>>> member, but on the sample_type bit mask. It returns -EOPNOTSUPP when this bit PERF_SAMPLE_CALLCHAIN
>>> is set. This solves the invocation with command line flag -g as in
>>> # ./perf record -v -e cycles -g -- perf test -w noploop
>>> ...
>>> perf record: Captured and wrote 0.183 MB perf.data ]
>>
>> Right because the callchain was removed from all the samples. We can't
>> fix old kernels (other than by using fix tags); is there a possibility
>> of adding the exclude_callchain_user to the s390 perf driver for the
>> sake of kernel callchains? It seems better than providing no
>> callchain.
>>
>>> Also I discovered that the fallback when using --call-graph dwarf command line flag still fails:
>>> # ./perf record -v -e cycles --call-graph dwarf -- perf test -w noploop
>>> ...
>>> Warning:
>>> Trying to fall back to excluding guest samples
>>> Error:
>>> Failure to open event 'cycles:H' on PMU 'cpum_cf' which will be removed.
>>> cycles:H: PMU Hardware doesn't support sampling overflow-interrupts. Try 'perf stat'
>>> Error:
>>> Failure to open any events for recording.
>>>
>>> The reason is in __evsel__config_callchain() which calls evsel__set_sample_bit(evsel, CALLCHAIN)
>>> and sets the PERF_SAMPLE_CALLCHAIN bit in evsel->core.attr.sample_type. It also sets the
>>> member attr->exclude_callchain_user = 1 and sets bits REGS_USER and _STACK_USER.
>>> All three bits are not supported by s390.
>>
>> I'm confused by this and your previous testing that showed the
>> `--call-graph dwarf` worked. You need the sampled registers for dwarf
>> unwinding to provide initial register values for the unwinder.
>>
Right, as explained shortly above, we have to use '-e cpu-clock --call-graph dwarf'
to get valid call chain data.
>>> I have modified your 2nd patch and appended it.
>>>
>>> I find all these bits in sample_type and the attr.exclude_XXX stuff very confusing. If there
>>> is a more consistant way of checking these feature, please let me know.
>
> I forgot to mention, yeah the exclude thing is maddening. It takes
> about 100 lines to convert the command line modifiers to those in the
> perf_event_attr, there's a priority to them, and so on:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/parse-events.c#n1759
> Fwiw, I was thinking for patch 1 of holding onto the parsed modifiers
> so that we could reset the excludes based on them when switching to
> the software event.
>
>> Ok, let me check it out.
>
> So looking at the cpum_cf driver it fails events for having sampling enabled:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_cf.c#n859
> ```
> if (is_sampling_event(event)) /* No sampling support */
> ```
> and the cpum_sf driver fails for any kind of callchain:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_sf.c#n839
> ```
> static bool is_callchain_event(struct perf_event *event)
> {
> u64 sample_type = event->attr.sample_type;
>
> return sample_type & (PERF_SAMPLE_CALLCHAIN | PERF_SAMPLE_REGS_USER |
> PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_STACK_USER);
> }
>
> static int cpumsf_pmu_event_init(struct perf_event *event)
> {
> int err;
>
> /* No support for taken branch sampling */
> /* No support for callchain, stacks and registers */
> if (has_branch_stack(event) || is_callchain_event(event))
> return -EOPNOTSUPP;
> ```
> Perhaps there is an oversight in the cpum_cf driver wrt branch stacks
> (LBR on x86). The PERF_SAMPLE_CALLCHAIN bit is set for perf call-graph
> options currently:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/evsel.c#n1024
>
> I think cpum_sf is the PMU we care about for the default cycles event?
Right this PMU does sampling on s390
> Since callchains of any flavor don't work with cpum_sf we can do a few
> things:
> 1) Disable the callchain and allow hardware sampling to continue,
> 2) Switch to a software event like cpu-clock,
> 3) Fail for the callchain option with this PMU, which is currently
> happening anyway.
@Namhyung:
It fails to open event cycles for PMU cpum_sf.
>
> I dislike option 3 because it requires special s390 logic for many
> tests, and we lose testing coverage of hardware events. Option 1 is a
> smaller patch, an early return in __evsel__config_callchain if on
> s390. Option 2 feels most like what the user would want given they
> asked for a callchain. We could change evlist__new_default to take a
> "with callchain" boolean and on s390 that could switch the event to a
> software one. Only two non-test callers use evlist__new__default
> (record and top), so it isn't a huge change.
>
> Wdyt Thomas, option 1 or 2?
>
> Thanks,
> Ian
I actually prefer option 2. when call chains are requested by the user
switch to cpu-clock event.
Thanks a lot for your patience.
--
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Wolfgang Wendt
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
On Fri, Mar 13, 2026 at 10:46:49AM +0100, Thomas Richter wrote:
> On 3/12/26 17:46, Ian Rogers wrote:
> > On Thu, Mar 12, 2026 at 8:54 AM Ian Rogers <irogers@google.com> wrote:
> >>
> >> On Thu, Mar 12, 2026 at 5:45 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
> >>>
> >>> On 3/12/26 07:16, Ian Rogers wrote:
> >>>> Frame pointer callchains are not supported on s390. Ignore the option
> >>>> and print a warning.
> >>>>
> >>>> Signed-off-by: Ian Rogers <irogers@google.com>
> >>>> ---
> >>>> v2: Only disable user callchains as AI is telling me native "kernel"
> >>>> callchains are supported on s390.
> >>>> ---
> >>>> tools/perf/util/evsel.c | 6 ++++++
> >>>> 1 file changed, 6 insertions(+)
> >>>>
> >>>> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> >>>> index bd14d9bbc91f..fa21b48cba86 100644
> >>>> --- a/tools/perf/util/evsel.c
> >>>> +++ b/tools/perf/util/evsel.c
> >>>> @@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> >>>> attr->exclude_callchain_user = 1;
> >>>> }
> >>>>
> >>>> + if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
> >>>> + !evsel->core.attr.exclude_callchain_user) {
> >>>> + pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
> >>>> + evsel->core.attr.exclude_callchain_user = 1;
> >>>> + }
> >>>> +
> >>>> if (param->defer && !attr->exclude_callchain_user)
> >>>> attr->defer_callchain = 1;
> >>>> }
> >>>
> >>> Ian, thanks very much.
> >>> Your patch set helps a lot. However there is a small nit (which is mandatory). Please add these lines
> >>>
> >>> evsel->core.attr.sample_type &= ~PERF_SAMPLE_CALLCHAIN;
> >>> evsel->core.attr.sample_type &= ~PERF_SAMPLE_REGS_USER;
> >>> evsel->core.attr.sample_type &= ~PERF_SAMPLE_STACK_USER;
> >>
> >> So these lines are dropping callchain from the sample_type which means
> >> the kernel stack won't be sampled. AI was telling me this worked, but
> >> I'm guess it was wrong. I think rather than this it is just cleaner
> >> never to set the bits in the perf_event_attr, more like what v1 of the
> >> patch did:
> >> https://lore.kernel.org/lkml/20260312031928.1494864-3-irogers@google.com/
> >>
>
> Let me try to answer your and Namhyung questions in one reply.
> s390 has many PMUs, there is one for counting (cpum_cf) and one for sampling (cpum_sf),
> which are totally different hardware components.
> Hardware sampling
> 1. writes into a large buffer, data is timestamp and IP (instruction pointer), no registers, no PID.
> 2. when buffer gets full an interrupt occurs and the interrupt handler scans the
> hardware samples and converts valid samples to perf format and saves them in perf ring buffer.
> This means we can not reconstruct register values for an individual sample. Current Pid is saved by
> other means somewhere and added to perf data during perf ring buffer write.
Thanks for the explanation!
Sounds like ARM SPE. Off-topic but I just wonder if it's possible to
adjust the buffer size so that it can fire for each sample.
>
> For s390 when we need callchains, we always use event cpu-clock, never hardware event cycles.
I see.
>
>
> >>> to the new if(EM_HOST == ...) above.
> >>> The s390 CPU Measurement sampling device driver does not check on the attr.core.exclude_callchain_user
> >>> member, but on the sample_type bit mask. It returns -EOPNOTSUPP when this bit PERF_SAMPLE_CALLCHAIN
> >>> is set. This solves the invocation with command line flag -g as in
> >>> # ./perf record -v -e cycles -g -- perf test -w noploop
> >>> ...
> >>> perf record: Captured and wrote 0.183 MB perf.data ]
> >>
> >> Right because the callchain was removed from all the samples. We can't
> >> fix old kernels (other than by using fix tags); is there a possibility
> >> of adding the exclude_callchain_user to the s390 perf driver for the
> >> sake of kernel callchains? It seems better than providing no
> >> callchain.
> >>
> >>> Also I discovered that the fallback when using --call-graph dwarf command line flag still fails:
> >>> # ./perf record -v -e cycles --call-graph dwarf -- perf test -w noploop
> >>> ...
> >>> Warning:
> >>> Trying to fall back to excluding guest samples
> >>> Error:
> >>> Failure to open event 'cycles:H' on PMU 'cpum_cf' which will be removed.
> >>> cycles:H: PMU Hardware doesn't support sampling overflow-interrupts. Try 'perf stat'
> >>> Error:
> >>> Failure to open any events for recording.
> >>>
> >>> The reason is in __evsel__config_callchain() which calls evsel__set_sample_bit(evsel, CALLCHAIN)
> >>> and sets the PERF_SAMPLE_CALLCHAIN bit in evsel->core.attr.sample_type. It also sets the
> >>> member attr->exclude_callchain_user = 1 and sets bits REGS_USER and _STACK_USER.
> >>> All three bits are not supported by s390.
> >>
> >> I'm confused by this and your previous testing that showed the
> >> `--call-graph dwarf` worked. You need the sampled registers for dwarf
> >> unwinding to provide initial register values for the unwinder.
> >>
>
> Right, as explained shortly above, we have to use '-e cpu-clock --call-graph dwarf'
> to get valid call chain data.
>
> >>> I have modified your 2nd patch and appended it.
> >>>
> >>> I find all these bits in sample_type and the attr.exclude_XXX stuff very confusing. If there
> >>> is a more consistant way of checking these feature, please let me know.
> >
> > I forgot to mention, yeah the exclude thing is maddening. It takes
> > about 100 lines to convert the command line modifiers to those in the
> > perf_event_attr, there's a priority to them, and so on:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/parse-events.c#n1759
> > Fwiw, I was thinking for patch 1 of holding onto the parsed modifiers
> > so that we could reset the excludes based on them when switching to
> > the software event.
> >
> >> Ok, let me check it out.
> >
> > So looking at the cpum_cf driver it fails events for having sampling enabled:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_cf.c#n859
> > ```
> > if (is_sampling_event(event)) /* No sampling support */
> > ```
> > and the cpum_sf driver fails for any kind of callchain:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_sf.c#n839
> > ```
> > static bool is_callchain_event(struct perf_event *event)
> > {
> > u64 sample_type = event->attr.sample_type;
> >
> > return sample_type & (PERF_SAMPLE_CALLCHAIN | PERF_SAMPLE_REGS_USER |
> > PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_STACK_USER);
> > }
> >
> > static int cpumsf_pmu_event_init(struct perf_event *event)
> > {
> > int err;
> >
> > /* No support for taken branch sampling */
> > /* No support for callchain, stacks and registers */
> > if (has_branch_stack(event) || is_callchain_event(event))
> > return -EOPNOTSUPP;
> > ```
> > Perhaps there is an oversight in the cpum_cf driver wrt branch stacks
> > (LBR on x86). The PERF_SAMPLE_CALLCHAIN bit is set for perf call-graph
> > options currently:
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/evsel.c#n1024
> >
> > I think cpum_sf is the PMU we care about for the default cycles event?
>
> Right this PMU does sampling on s390
>
> > Since callchains of any flavor don't work with cpum_sf we can do a few
> > things:
> > 1) Disable the callchain and allow hardware sampling to continue,
> > 2) Switch to a software event like cpu-clock,
> > 3) Fail for the callchain option with this PMU, which is currently
> > happening anyway.
>
> @Namhyung:
> It fails to open event cycles for PMU cpum_sf.
Ok, I assume it'd reject events with PERF_SAMPLE_CALLCHAIN.
>
> >
> > I dislike option 3 because it requires special s390 logic for many
> > tests, and we lose testing coverage of hardware events. Option 1 is a
> > smaller patch, an early return in __evsel__config_callchain if on
> > s390. Option 2 feels most like what the user would want given they
> > asked for a callchain. We could change evlist__new_default to take a
> > "with callchain" boolean and on s390 that could switch the event to a
> > software one. Only two non-test callers use evlist__new__default
> > (record and top), so it isn't a huge change.
> >
> > Wdyt Thomas, option 1 or 2?
> >
> > Thanks,
> > Ian
>
> I actually prefer option 2. when call chains are requested by the user
> switch to cpu-clock event.
I think the current code will do the job already. But we missed to
remove some exclude_* bits before that. And moving callchain setting
from FP to DWARF is a separate issue.
>
> Thanks a lot for your patience.
The same goes to you. :)
Thanks,
Namhyung
>
> --
> Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> --
> IBM Deutschland Research & Development GmbH
>
> Vorsitzender des Aufsichtsrats: Wolfgang Wendt
>
> Geschäftsführung: David Faller
>
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Hello,
On Thu, Mar 12, 2026 at 09:46:54AM -0700, Ian Rogers wrote:
> On Thu, Mar 12, 2026 at 8:54 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Thu, Mar 12, 2026 at 5:45 AM Thomas Richter <tmricht@linux.ibm.com> wrote:
> > >
> > > On 3/12/26 07:16, Ian Rogers wrote:
> > > > Frame pointer callchains are not supported on s390. Ignore the option
> > > > and print a warning.
> > > >
> > > > Signed-off-by: Ian Rogers <irogers@google.com>
> > > > ---
> > > > v2: Only disable user callchains as AI is telling me native "kernel"
> > > > callchains are supported on s390.
> > > > ---
> > > > tools/perf/util/evsel.c | 6 ++++++
> > > > 1 file changed, 6 insertions(+)
> > > >
> > > > diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> > > > index bd14d9bbc91f..fa21b48cba86 100644
> > > > --- a/tools/perf/util/evsel.c
> > > > +++ b/tools/perf/util/evsel.c
> > > > @@ -1076,6 +1076,12 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> > > > attr->exclude_callchain_user = 1;
> > > > }
> > > >
> > > > + if (EM_HOST == EM_S390 && (evsel->core.attr.sample_type & PERF_SAMPLE_CALLCHAIN) &&
> > > > + !evsel->core.attr.exclude_callchain_user) {
> > > > + pr_warning("Excluding user callchains that are not supported on s390. Try '--call-graph dwarf'\n");
> > > > + evsel->core.attr.exclude_callchain_user = 1;
> > > > + }
> > > > +
> > > > if (param->defer && !attr->exclude_callchain_user)
> > > > attr->defer_callchain = 1;
> > > > }
> > >
> > > Ian, thanks very much.
> > > Your patch set helps a lot. However there is a small nit (which is mandatory). Please add these lines
> > >
> > > evsel->core.attr.sample_type &= ~PERF_SAMPLE_CALLCHAIN;
> > > evsel->core.attr.sample_type &= ~PERF_SAMPLE_REGS_USER;
> > > evsel->core.attr.sample_type &= ~PERF_SAMPLE_STACK_USER;
> >
> > So these lines are dropping callchain from the sample_type which means
> > the kernel stack won't be sampled. AI was telling me this worked, but
> > I'm guess it was wrong. I think rather than this it is just cleaner
> > never to set the bits in the perf_event_attr, more like what v1 of the
> > patch did:
> > https://lore.kernel.org/lkml/20260312031928.1494864-3-irogers@google.com/
> >
> > > to the new if(EM_HOST == ...) above.
> > > The s390 CPU Measurement sampling device driver does not check on the attr.core.exclude_callchain_user
> > > member, but on the sample_type bit mask. It returns -EOPNOTSUPP when this bit PERF_SAMPLE_CALLCHAIN
> > > is set. This solves the invocation with command line flag -g as in
> > > # ./perf record -v -e cycles -g -- perf test -w noploop
> > > ...
> > > perf record: Captured and wrote 0.183 MB perf.data ]
> >
> > Right because the callchain was removed from all the samples. We can't
> > fix old kernels (other than by using fix tags); is there a possibility
> > of adding the exclude_callchain_user to the s390 perf driver for the
> > sake of kernel callchains? It seems better than providing no
> > callchain.
> >
> > > Also I discovered that the fallback when using --call-graph dwarf command line flag still fails:
> > > # ./perf record -v -e cycles --call-graph dwarf -- perf test -w noploop
> > > ...
> > > Warning:
> > > Trying to fall back to excluding guest samples
> > > Error:
> > > Failure to open event 'cycles:H' on PMU 'cpum_cf' which will be removed.
> > > cycles:H: PMU Hardware doesn't support sampling overflow-interrupts. Try 'perf stat'
> > > Error:
> > > Failure to open any events for recording.
> > >
> > > The reason is in __evsel__config_callchain() which calls evsel__set_sample_bit(evsel, CALLCHAIN)
> > > and sets the PERF_SAMPLE_CALLCHAIN bit in evsel->core.attr.sample_type. It also sets the
> > > member attr->exclude_callchain_user = 1 and sets bits REGS_USER and _STACK_USER.
> > > All three bits are not supported by s390.
> >
> > I'm confused by this and your previous testing that showed the
> > `--call-graph dwarf` worked. You need the sampled registers for dwarf
> > unwinding to provide initial register values for the unwinder.
> >
> > > I have modified your 2nd patch and appended it.
> > >
> > > I find all these bits in sample_type and the attr.exclude_XXX stuff very confusing. If there
> > > is a more consistant way of checking these feature, please let me know.
>
> I forgot to mention, yeah the exclude thing is maddening. It takes
> about 100 lines to convert the command line modifiers to those in the
> perf_event_attr, there's a priority to them, and so on:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/parse-events.c#n1759
> Fwiw, I was thinking for patch 1 of holding onto the parsed modifiers
> so that we could reset the excludes based on them when switching to
> the software event.
>
> > Ok, let me check it out.
>
> So looking at the cpum_cf driver it fails events for having sampling enabled:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_cf.c#n859
> ```
> if (is_sampling_event(event)) /* No sampling support */
> ```
> and the cpum_sf driver fails for any kind of callchain:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/s390/kernel/perf_cpum_sf.c#n839
> ```
> static bool is_callchain_event(struct perf_event *event)
> {
> u64 sample_type = event->attr.sample_type;
>
> return sample_type & (PERF_SAMPLE_CALLCHAIN | PERF_SAMPLE_REGS_USER |
> PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_STACK_USER);
> }
>
> static int cpumsf_pmu_event_init(struct perf_event *event)
> {
> int err;
>
> /* No support for taken branch sampling */
> /* No support for callchain, stacks and registers */
> if (has_branch_stack(event) || is_callchain_event(event))
> return -EOPNOTSUPP;
> ```
> Perhaps there is an oversight in the cpum_cf driver wrt branch stacks
> (LBR on x86). The PERF_SAMPLE_CALLCHAIN bit is set for perf call-graph
> options currently:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/perf/util/evsel.c#n1024
>
> I think cpum_sf is the PMU we care about for the default cycles event?
> Since callchains of any flavor don't work with cpum_sf we can do a few
> things:
> 1) Disable the callchain and allow hardware sampling to continue,
> 2) Switch to a software event like cpu-clock,
> 3) Fail for the callchain option with this PMU, which is currently
> happening anyway.
I'm confused. Is it failing to open any events with callchains or just
events from cpum_sf PMU (with callchains)?
Thanks,
Namhyung
>
> I dislike option 3 because it requires special s390 logic for many
> tests, and we lose testing coverage of hardware events. Option 1 is a
> smaller patch, an early return in __evsel__config_callchain if on
> s390. Option 2 feels most like what the user would want given they
> asked for a callchain. We could change evlist__new_default to take a
> "with callchain" boolean and on s390 that could switch the event to a
> software one. Only two non-test callers use evlist__new__default
> (record and top), so it isn't a huge change.
>
> Wdyt Thomas, option 1 or 2?
>
> Thanks,
> Ian
>
> > > Thanks again for looking into this.
> >
> > Thanks,
> > Ian
> >
> > > --
> > > Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
> > > --
> > > IBM Deutschland Research & Development GmbH
> > >
> > > Vorsitzender des Aufsichtsrats: Wolfgang Wendt
> > >
> > > Geschäftsführung: David Faller
> > >
> > > Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
© 2016 - 2026 Red Hat, Inc.