perf/test: perf test 86 fails on s390 repo linux-next

[PATCH] perf/test: perf test 86 fails on s390 repo linux-next

Posted by Thomas Richter 2 months, 2 weeks ago

Command perf test 86 fails on s390 using linux-next repository:
 # perf test -F 86
 ping 868299 [007] 28248.013596: probe_libc:inet_pton_1: (3ff95948020)
 3ff95948020 inet_pton+0x0 (inlined)
 3ff9595e6e7 text_to_binary_address+0x1007 (inlined)
 3ff9595e6e7 gaih_inet+0x1007 (inlined)
 FAIL: expected backtrace entry \
	 "main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$"
	 got "3ff9595e6e7 gaih_inet+0x1007 (inlined)"
 86: probe libc's inet_pton & backtrace it with ping  : FAILED!
 #

The root cause is a new stack layout, some functions have been added
as seen below. Add text_to_binary_address and friends to the
list of expected functions.

 # perf script | tac | grep -m1 '^ping' -B9 | tac
 ping  866856 [007] 25979.494921: probe_libc:inet_pton: (3ff8ec48020)
	     3ff8ec48020 inet_pton+0x0 (inlined)
	     3ff8ec5e6e7 text_to_binary_address+0x1007 (inlined)
	     3ff8ec5e6e7 gaih_inet+0x1007 (inlined)
	     3ff8ec5e6e7 getaddrinfo+0x1007 (/usr/lib64/libc.so.6)
	     2aa3fe04bf5 main+0xff5 (/usr/bin/ping)
	     3ff8eb34a5b __libc_start_call_main+0x8b (/usr/lib64/libc.so.6)
	     3ff8eb34b5d __libc_start_main@GLIBC_2.2+0xad (inlined)
	     2aa3fe06a1f [unknown] (/usr/bin/ping)

 #

Output after:
 # perf test -F 86
 86: probe libc's inet_pton & backtrace it with ping  : Ok
 #

Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
---
 tools/perf/tests/shell/record+probe_libc_inet_pton.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
index f38c8ead0b03..bc6e2fe1d999 100755
--- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
+++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
@@ -40,6 +40,8 @@ trace_libc_inet_pton_backtrace() {
 	case "$(uname -m)" in
 	s390x)
 		eventattr='call-graph=dwarf,max-stack=4'
+		echo "text_to_binary_address\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
+		echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
 		echo "(__GI_)?getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
 		echo "main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$" >> $expected
 		;;
-- 
2.46.0

Re: [PATCH] perf/test: perf test 86 fails on s390 repo linux-next

Posted by Namhyung Kim 2 months ago

Hello,

On Mon, Sep 16, 2024 at 02:57:18PM +0200, Thomas Richter wrote:
> Command perf test 86 fails on s390 using linux-next repository:
>  # perf test -F 86
>  ping 868299 [007] 28248.013596: probe_libc:inet_pton_1: (3ff95948020)
>  3ff95948020 inet_pton+0x0 (inlined)
>  3ff9595e6e7 text_to_binary_address+0x1007 (inlined)
>  3ff9595e6e7 gaih_inet+0x1007 (inlined)
>  FAIL: expected backtrace entry \
> 	 "main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$"
> 	 got "3ff9595e6e7 gaih_inet+0x1007 (inlined)"
>  86: probe libc's inet_pton & backtrace it with ping  : FAILED!
>  #
> 
> The root cause is a new stack layout, some functions have been added
> as seen below. Add text_to_binary_address and friends to the
> list of expected functions.
> 
>  # perf script | tac | grep -m1 '^ping' -B9 | tac
>  ping  866856 [007] 25979.494921: probe_libc:inet_pton: (3ff8ec48020)
> 	     3ff8ec48020 inet_pton+0x0 (inlined)
> 	     3ff8ec5e6e7 text_to_binary_address+0x1007 (inlined)
> 	     3ff8ec5e6e7 gaih_inet+0x1007 (inlined)
> 	     3ff8ec5e6e7 getaddrinfo+0x1007 (/usr/lib64/libc.so.6)
> 	     2aa3fe04bf5 main+0xff5 (/usr/bin/ping)
> 	     3ff8eb34a5b __libc_start_call_main+0x8b (/usr/lib64/libc.so.6)
> 	     3ff8eb34b5d __libc_start_main@GLIBC_2.2+0xad (inlined)
> 	     2aa3fe06a1f [unknown] (/usr/bin/ping)

Is it because of a kernel change?  What about old kernels then?

> 
>  #
> 
> Output after:
>  # perf test -F 86
>  86: probe libc's inet_pton & backtrace it with ping  : Ok
>  #
> 
> Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
> ---
>  tools/perf/tests/shell/record+probe_libc_inet_pton.sh | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> index f38c8ead0b03..bc6e2fe1d999 100755
> --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> @@ -40,6 +40,8 @@ trace_libc_inet_pton_backtrace() {
>  	case "$(uname -m)" in
>  	s390x)
>  		eventattr='call-graph=dwarf,max-stack=4'
> +		echo "text_to_binary_address\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
> +		echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected

Is it possible to make it this part conditional and only have it for the
new kernels?

Thanks,
Namhyung


>  		echo "(__GI_)?getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
>  		echo "main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$" >> $expected
>  		;;
> -- 
> 2.46.0
>

Re: [PATCH] perf/test: perf test 86 fails on s390 repo linux-next

Posted by Thomas Richter 2 months ago

On 9/27/24 00:42, Namhyung Kim wrote:
> Hello,
> 
> On Mon, Sep 16, 2024 at 02:57:18PM +0200, Thomas Richter wrote:
>> Command perf test 86 fails on s390 using linux-next repository:
>>  # perf test -F 86
>>  ping 868299 [007] 28248.013596: probe_libc:inet_pton_1: (3ff95948020)
>>  3ff95948020 inet_pton+0x0 (inlined)
>>  3ff9595e6e7 text_to_binary_address+0x1007 (inlined)
>>  3ff9595e6e7 gaih_inet+0x1007 (inlined)
>>  FAIL: expected backtrace entry \
>> 	 "main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$"
>> 	 got "3ff9595e6e7 gaih_inet+0x1007 (inlined)"
>>  86: probe libc's inet_pton & backtrace it with ping  : FAILED!
>>  #
>>
>> The root cause is a new stack layout, some functions have been added
>> as seen below. Add text_to_binary_address and friends to the
>> list of expected functions.
>>
>>  # perf script | tac | grep -m1 '^ping' -B9 | tac
>>  ping  866856 [007] 25979.494921: probe_libc:inet_pton: (3ff8ec48020)
>> 	     3ff8ec48020 inet_pton+0x0 (inlined)
>> 	     3ff8ec5e6e7 text_to_binary_address+0x1007 (inlined)
>> 	     3ff8ec5e6e7 gaih_inet+0x1007 (inlined)
>> 	     3ff8ec5e6e7 getaddrinfo+0x1007 (/usr/lib64/libc.so.6)
>> 	     2aa3fe04bf5 main+0xff5 (/usr/bin/ping)
>> 	     3ff8eb34a5b __libc_start_call_main+0x8b (/usr/lib64/libc.so.6)
>> 	     3ff8eb34b5d __libc_start_main@GLIBC_2.2+0xad (inlined)
>> 	     2aa3fe06a1f [unknown] (/usr/bin/ping)
> 
> Is it because of a kernel change?  What about old kernels then?
> 
>>
>>  #
>>
>> Output after:
>>  # perf test -F 86
>>  86: probe libc's inet_pton & backtrace it with ping  : Ok
>>  #
>>
>> Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
>> ---
>>  tools/perf/tests/shell/record+probe_libc_inet_pton.sh | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
>> index f38c8ead0b03..bc6e2fe1d999 100755
>> --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
>> +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
>> @@ -40,6 +40,8 @@ trace_libc_inet_pton_backtrace() {
>>  	case "$(uname -m)" in
>>  	s390x)
>>  		eventattr='call-graph=dwarf,max-stack=4'
>> +		echo "text_to_binary_address\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
>> +		echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
> 
> Is it possible to make it this part conditional and only have it for the
> new kernels?
> 
> Thanks,
> Namhyung
> 
> 
>>

I think this is more related to glibc and has nothing to do with the kernel version.
It happened after I did a dnf update.

-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: [PATCH] perf/test: perf test 86 fails on s390 repo linux-next

Posted by Namhyung Kim 1 month, 4 weeks ago

On Mon, Sep 30, 2024 at 02:30:39PM +0200, Thomas Richter wrote:
> On 9/27/24 00:42, Namhyung Kim wrote:
> > Hello,
> > 
> > On Mon, Sep 16, 2024 at 02:57:18PM +0200, Thomas Richter wrote:
> >> Command perf test 86 fails on s390 using linux-next repository:
> >>  # perf test -F 86
> >>  ping 868299 [007] 28248.013596: probe_libc:inet_pton_1: (3ff95948020)
> >>  3ff95948020 inet_pton+0x0 (inlined)
> >>  3ff9595e6e7 text_to_binary_address+0x1007 (inlined)
> >>  3ff9595e6e7 gaih_inet+0x1007 (inlined)
> >>  FAIL: expected backtrace entry \
> >> 	 "main\+0x[[:xdigit:]]+[[:space:]]\(.*/bin/ping.*\)$"
> >> 	 got "3ff9595e6e7 gaih_inet+0x1007 (inlined)"
> >>  86: probe libc's inet_pton & backtrace it with ping  : FAILED!
> >>  #
> >>
> >> The root cause is a new stack layout, some functions have been added
> >> as seen below. Add text_to_binary_address and friends to the
> >> list of expected functions.
> >>
> >>  # perf script | tac | grep -m1 '^ping' -B9 | tac
> >>  ping  866856 [007] 25979.494921: probe_libc:inet_pton: (3ff8ec48020)
> >> 	     3ff8ec48020 inet_pton+0x0 (inlined)
> >> 	     3ff8ec5e6e7 text_to_binary_address+0x1007 (inlined)
> >> 	     3ff8ec5e6e7 gaih_inet+0x1007 (inlined)
> >> 	     3ff8ec5e6e7 getaddrinfo+0x1007 (/usr/lib64/libc.so.6)
> >> 	     2aa3fe04bf5 main+0xff5 (/usr/bin/ping)
> >> 	     3ff8eb34a5b __libc_start_call_main+0x8b (/usr/lib64/libc.so.6)
> >> 	     3ff8eb34b5d __libc_start_main@GLIBC_2.2+0xad (inlined)
> >> 	     2aa3fe06a1f [unknown] (/usr/bin/ping)
> > 
> > Is it because of a kernel change?  What about old kernels then?
> > 
> >>
> >>  #
> >>
> >> Output after:
> >>  # perf test -F 86
> >>  86: probe libc's inet_pton & backtrace it with ping  : Ok
> >>  #
> >>
> >> Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
> >> ---
> >>  tools/perf/tests/shell/record+probe_libc_inet_pton.sh | 2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >> diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> >> index f38c8ead0b03..bc6e2fe1d999 100755
> >> --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> >> +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> >> @@ -40,6 +40,8 @@ trace_libc_inet_pton_backtrace() {
> >>  	case "$(uname -m)" in
> >>  	s390x)
> >>  		eventattr='call-graph=dwarf,max-stack=4'
> >> +		echo "text_to_binary_address\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
> >> +		echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
> > 
> > Is it possible to make it this part conditional and only have it for the
> > new kernels?
> 
> I think this is more related to glibc and has nothing to do with the kernel version.
> It happened after I did a dnf update.

Ok, then I'm afraid fixing this would break others with old glibc.
Can you please update the regex to cover both cases?

Thanks,
Namhyung