5.19-rc1 build fails at scripts/check-local-export

Posted by Tetsuo Handa 3 years, 10 months ago

Hello.

Commit 31cb50b5590fe911 ("kbuild: check static EXPORT_SYMBOL* by script
instead of modpost") introduced scripts/check-local-export but it fails
in my CentOS 7 environment.

$ rpm -q binutils bash
binutils-2.27-44.base.el7_9.1.x86_64
bash-4.2.46-35.el7_9.x86_64
$ make
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_64.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_x32.h
  SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
  SYSHDR  arch/x86/include/generated/asm/unistd_32_ia32.h
  SYSHDR  arch/x86/include/generated/asm/unistd_64_x32.h
  SYSTBL  arch/x86/include/generated/asm/syscalls_64.h
  HOSTCC  scripts/basic/fixdep
  HOSTCC  arch/x86/tools/relocs_32.o
  HOSTCC  arch/x86/tools/relocs_64.o
  HOSTCC  arch/x86/tools/relocs_common.o
  HOSTLD  arch/x86/tools/relocs
  HOSTCC  scripts/genksyms/genksyms.o
  YACC    scripts/genksyms/parse.tab.[ch]
  HOSTCC  scripts/genksyms/parse.tab.o
  LEX     scripts/genksyms/lex.lex.c
  HOSTCC  scripts/genksyms/lex.lex.o
  HOSTLD  scripts/genksyms/genksyms
  HOSTCC  scripts/bin2c
  HOSTCC  scripts/kallsyms
  HOSTCC  scripts/sorttable
  HOSTCC  scripts/asn1_compiler
  HOSTCC  scripts/mod/mk_elfconfig
  CC      scripts/mod/empty.o
./scripts/check-local-export: line 54: wait: pid 17328 is not a child of this shell
make[2]: *** [scripts/mod/empty.o] Error 127
make[2]: *** Deleting file `scripts/mod/empty.o'
make[1]: *** [prepare0] Error 2
make: *** [__sub-make] Error 2

Is below change OK for you?

diff --git a/scripts/check-local-export b/scripts/check-local-export
index da745e2743b7..1631c79558b7 100755
--- a/scripts/check-local-export
+++ b/scripts/check-local-export
@@ -51,7 +51,7 @@ do
 done < <(${NM} ${1} 2>/dev/null || { echo "${0}: ${NM} failed" >&2; false; } )
 
 # Catch error in the process substitution
-wait $!
+wait $! 2>/dev/null || :
 
 for name in "${export_symbols[@]}"
 do

Re: 5.19-rc1 build fails at scripts/check-local-export

Posted by Michael Ellerman 3 years, 10 months ago

Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> writes:
> Hello.
>
> Commit 31cb50b5590fe911 ("kbuild: check static EXPORT_SYMBOL* by script
> instead of modpost") introduced scripts/check-local-export but it fails
> in my CentOS 7 environment.
>
> $ rpm -q binutils bash
> binutils-2.27-44.base.el7_9.1.x86_64
> bash-4.2.46-35.el7_9.x86_64

I'm also seeing this, on Ubuntu 16.04.

$ bash --version
GNU bash, version 4.3.48(1)-release (powerpc64le-unknown-linux-gnu)

> $ make
...
>   HOSTCC  scripts/mod/mk_elfconfig
>   CC      scripts/mod/empty.o
> ./scripts/check-local-export: line 54: wait: pid 17328 is not a child of this shell
> make[2]: *** [scripts/mod/empty.o] Error 127
> make[2]: *** Deleting file `scripts/mod/empty.o'
> make[1]: *** [prepare0] Error 2
> make: *** [__sub-make] Error 2
>
> Is below change OK for you?
>
> diff --git a/scripts/check-local-export b/scripts/check-local-export
> index da745e2743b7..1631c79558b7 100755
> --- a/scripts/check-local-export
> +++ b/scripts/check-local-export
> @@ -51,7 +51,7 @@ do
>  done < <(${NM} ${1} 2>/dev/null || { echo "${0}: ${NM} failed" >&2; false; } )
>  
>  # Catch error in the process substitution
> -wait $!
> +wait $! 2>/dev/null || :
>  
>  for name in "${export_symbols[@]}"
>  do

That fixes it for me.

Tested-by: Michael Ellerman <mpe@ellerman.id.au>

cheers

[PATCH] kbuild: fix build failure by scripts/check-local-export

Posted by Tetsuo Handa 3 years, 10 months ago

scripts/check-local-export fails with some versions of bash.

    CC      scripts/mod/empty.o
  ./scripts/check-local-export: line 54: wait: pid 17328 is not a child of this shell
  make[2]: *** [scripts/mod/empty.o] Error 127
  make[2]: *** Deleting file `scripts/mod/empty.o'
  make[1]: *** [prepare0] Error 2
  make: *** [__sub-make] Error 2

Avoid use of bash's built-in wait command, by saving the output from
nm command into a temporary variable.

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Fixes: 31cb50b5590fe911 ("kbuild: check static EXPORT_SYMBOL* by script instead of modpost")
---
 scripts/check-local-export | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)

diff --git a/scripts/check-local-export b/scripts/check-local-export
index da745e2743b7..5e46bc37a635 100755
--- a/scripts/check-local-export
+++ b/scripts/check-local-export
@@ -11,9 +11,20 @@ set -e
 declare -A symbol_types
 declare -a export_symbols
 
+function die
+{
+    echo "$1" >&2
+    exit 1
+}
+
 exit_code=0
 
-while read value type name
+# If there is no symbol in the object, ${NM} (both GNU nm and llvm-nm)
+# shows 'no symbols' diagnostic and exits with 0. Saving such line into
+# symbol_types is fine because export_symbols will remain empty.
+result=$(${NM} -- ${1} 2>&1) || die "${result}"
+
+echo "${result}" | while read value type name
 do
 	# Skip the line if the number of fields is less than 3.
 	#
@@ -37,21 +48,7 @@ do
 	if [[ ${name} == __ksymtab_* ]]; then
 		export_symbols+=(${name#__ksymtab_})
 	fi
-
-	# If there is no symbol in the object, ${NM} (both GNU nm and llvm-nm)
-	# shows 'no symbols' diagnostic (but exits with 0). It is harmless and
-	# hidden by '2>/dev/null'. However, it suppresses real error messages
-	# as well. Add a hand-crafted error message here.
-	#
-	# Use --quiet instead of 2>/dev/null when we upgrade the minimum version
-	# of binutils to 2.37, llvm to 13.0.0.
-	#
-	# Then, the following line will be really simple:
-	#   done < <(${NM} --quiet ${1})
-done < <(${NM} ${1} 2>/dev/null || { echo "${0}: ${NM} failed" >&2; false; } )
-
-# Catch error in the process substitution
-wait $!
+done
 
 for name in "${export_symbols[@]}"
 do
-- 
2.18.4

Re: [PATCH] kbuild: fix build failure by scripts/check-local-export

Posted by kernel test robot 3 years, 10 months ago

Hi Tetsuo,

I love your patch! Yet something to improve:

[auto build test ERROR on masahiroy-kbuild/for-next]
[also build test ERROR on linus/master v5.19-rc1 next-20220606]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/intel-lab-lkp/linux/commits/Tetsuo-Handa/kbuild-fix-build-failure-by-scripts-check-local-export/20220606-234446
base:   https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git for-next
config: arm-mxs_defconfig (https://download.01.org/0day-ci/archive/20220607/202206070434.3wvNWfJZ-lkp@intel.com/config)
compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project b92436efcb7813fc481b30f2593a4907568d917a)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install arm cross compiling tool for clang build
        # apt-get install binutils-arm-linux-gnueabi
        # https://github.com/intel-lab-lkp/linux/commit/7047234a52a99a58113ebe0502e1c227af2b6c61
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Tetsuo-Handa/kbuild-fix-build-failure-by-scripts-check-local-export/20220606-234446
        git checkout 7047234a52a99a58113ebe0502e1c227af2b6c61
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm prepare

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   scripts/genksyms/parse.y: warning: 9 shift/reduce conflicts [-Wconflicts-sr]
   scripts/genksyms/parse.y: warning: 5 reduce/reduce conflicts [-Wconflicts-rr]
   scripts/genksyms/parse.y: note: rerun with option '-Wcounterexamples' to generate conflict counterexamples
>> llvm-nm: error: : unknown argument '--'
   make[2]: *** [scripts/Makefile.build:249: scripts/mod/empty.o] Error 1
   make[2]: *** Deleting file 'scripts/mod/empty.o'
   make[2]: Target '__build' not remade because of errors.
   make[1]: *** [Makefile:1195: prepare0] Error 2
   make[1]: Target 'prepare' not remade because of errors.
   make: *** [Makefile:219: __sub-make] Error 2
   make: Target 'prepare' not remade because of errors.

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

[PATCH v2] kbuild: fix build failure by scripts/check-local-export

Posted by Tetsuo Handa 3 years, 10 months ago

scripts/check-local-export fails with some versions of bash.

    CC      scripts/mod/empty.o
  ./scripts/check-local-export: line 54: wait: pid 17328 is not a child of this shell
  make[2]: *** [scripts/mod/empty.o] Error 127
  make[2]: *** Deleting file `scripts/mod/empty.o'
  make[1]: *** [prepare0] Error 2
  make: *** [__sub-make] Error 2

Avoid use of bash's built-in wait command, by saving the output from
nm command into a temporary variable.

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Fixes: 31cb50b5590fe911 ("kbuild: check static EXPORT_SYMBOL* by script instead of modpost")
---
Changes in v2:
  llvm-nm can't use end-of-options argument, reported-by kernel test robot <lkp@intel.com>

 scripts/check-local-export | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)

diff --git a/scripts/check-local-export b/scripts/check-local-export
index da745e2743b7..850abc150855 100755
--- a/scripts/check-local-export
+++ b/scripts/check-local-export
@@ -11,9 +11,20 @@ set -e
 declare -A symbol_types
 declare -a export_symbols
 
+function die
+{
+    echo "$1" >&2
+    exit 1
+}
+
 exit_code=0
 
-while read value type name
+# If there is no symbol in the object, ${NM} (both GNU nm and llvm-nm)
+# shows 'no symbols' diagnostic and exits with 0. Saving such line into
+# symbol_types is fine because export_symbols will remain empty.
+result=$(${NM} ${1} 2>&1) || die "${result}"
+
+echo "${result}" | while read value type name
 do
 	# Skip the line if the number of fields is less than 3.
 	#
@@ -37,21 +48,7 @@ do
 	if [[ ${name} == __ksymtab_* ]]; then
 		export_symbols+=(${name#__ksymtab_})
 	fi
-
-	# If there is no symbol in the object, ${NM} (both GNU nm and llvm-nm)
-	# shows 'no symbols' diagnostic (but exits with 0). It is harmless and
-	# hidden by '2>/dev/null'. However, it suppresses real error messages
-	# as well. Add a hand-crafted error message here.
-	#
-	# Use --quiet instead of 2>/dev/null when we upgrade the minimum version
-	# of binutils to 2.37, llvm to 13.0.0.
-	#
-	# Then, the following line will be really simple:
-	#   done < <(${NM} --quiet ${1})
-done < <(${NM} ${1} 2>/dev/null || { echo "${0}: ${NM} failed" >&2; false; } )
-
-# Catch error in the process substitution
-wait $!
+done
 
 for name in "${export_symbols[@]}"
 do
-- 
2.18.4

Re: [PATCH v2] kbuild: fix build failure by scripts/check-local-export

Posted by Masahiro Yamada 3 years, 10 months ago

On Tue, Jun 7, 2022 at 7:13 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> scripts/check-local-export fails with some versions of bash.
>
>     CC      scripts/mod/empty.o
>   ./scripts/check-local-export: line 54: wait: pid 17328 is not a child of this shell
>   make[2]: *** [scripts/mod/empty.o] Error 127
>   make[2]: *** Deleting file `scripts/mod/empty.o'
>   make[1]: *** [prepare0] Error 2
>   make: *** [__sub-make] Error 2
>
> Avoid use of bash's built-in wait command, by saving the output from
> nm command into a temporary variable.


This patch does not work because you did not avoid
running the while-loop in a subshell.

It is well described in  this page:
https://riptutorial.com/bash/example/26955/to-avoid-usage-of-a-sub-shell



I will send a working patch with a proper commit log.

The part "Saving such line into symbol_types is fine because export_symbols
will remain empty." seems OK with me.
(I was searching for an elegant solution for this, but
I could not come up with a better one.)






> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Fixes: 31cb50b5590fe911 ("kbuild: check static EXPORT_SYMBOL* by script instead of modpost")
> ---









--
Best Regards

Masahiro Yamada

Re: [PATCH v2] kbuild: fix build failure by scripts/check-local-export

Posted by Tetsuo Handa 3 years, 10 months ago

On 2022/06/07 17:34, Masahiro Yamada wrote:
> This patch does not work because you did not avoid
> running the while-loop in a subshell.
> 
> It is well described in  this page:
> https://riptutorial.com/bash/example/26955/to-avoid-usage-of-a-sub-shell
> 

I didn't know that. Then, adding below diff will work.

@@ -24,7 +24,7 @@ exit_code=0
 # symbol_types is fine because export_symbols will remain empty.
 result=$(${NM} ${1} 2>&1) || die "${result}"

-echo "${result}" | while read value type name
+while read value type name
 do
        # Skip the line if the number of fields is less than 3.
        #
@@ -48,7 +48,9 @@ do
        if [[ ${name} == __ksymtab_* ]]; then
                export_symbols+=(${name#__ksymtab_})
        fi
-done
+done <<EOF
+"${result}"
+EOF

 for name in "${export_symbols[@]}"
 do

> 
> 
> I will send a working patch with a proper commit log.

OK. "[PATCH] scripts/check-local-export: avoid 'wait $!' for process substitution" works.

Thank you.