1. Patchew
  2. QEMU
  3. [RFC v4 00/11] Native Library Calls
  4. View patch

[RFC v4 01/11] build: Implement logic for sharing cross-building config files

Yeqi Fu posted 11 patches 2 years, 6 months ago
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, Thomas Huth <thuth@redhat.com>, Richard Henderson <richard.henderson@linaro.org>, Riku Voipio <riku.voipio@iki.fi>, Laurent Vivier <laurent@vivier.eu>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Jiaxun Yang <jiaxun.yang@flygoat.com>, Peter Maydell <peter.maydell@linaro.org>, Eduardo Habkost <eduardo@habkost.net>, Aurelien Jarno <aurelien@aurel32.net>, Aleksandar Rikalo <aleksandar.rikalo@syrmia.com>
There is a newer version of this series
[RFC v4 01/11] build: Implement logic for sharing cross-building config files
Posted by Yeqi Fu 2 years, 6 months ago 
Signed-off-by: Yeqi Fu <fufuyqqqqqq@gmail.com>
---
 configure | 57 +++++++++++++++++++++++++++++++++----------------------
 1 file changed, 34 insertions(+), 23 deletions(-)

diff --git a/configure b/configure
index 2b41c49c0d..a076583141 100755
--- a/configure
+++ b/configure
@@ -1751,56 +1751,67 @@ if test "$ccache_cpp2" = "yes"; then
   echo "export CCACHE_CPP2=y" >> $config_host_mak
 fi
 
-# tests/tcg configuration
-(config_host_mak=tests/tcg/config-host.mak
-mkdir -p tests/tcg
-echo "# Automatically generated by configure - do not modify" > $config_host_mak
-echo "SRC_PATH=$source_path" >> $config_host_mak
-echo "HOST_CC=$host_cc" >> $config_host_mak
+# Prepare the config files for cross building.
+# This process generates 'cross-build/<target>/config-target.mak' files.
+# These files are then symlinked to the directories that need them which
+# including the TCG tests (tests/tcg/<target>) and the libnative library
+# for linux-user (common/native/<target>/).
+mkdir -p cross-build
 
-# versioned checked in the main config_host.mak above
-if test -n "$gdb_bin"; then
-    echo "HAVE_GDB_BIN=$gdb_bin" >> $config_host_mak
-fi
-if test "$plugins" = "yes" ; then
-    echo "CONFIG_PLUGIN=y" >> $config_host_mak
-fi
-
-tcg_tests_targets=
 for target in $target_list; do
   arch=${target%%-*}
-
   case $target in
     xtensa*-linux-user)
-      # the toolchain is not complete with headers, only build softmmu tests
+      # the toolchain for tests/tcg is not complete with headers
       continue
       ;;
     *-softmmu)
-      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
       qemu="qemu-system-$arch"
       ;;
     *-linux-user|*-bsd-user)
       qemu="qemu-$arch"
       ;;
   esac
-
   if probe_target_compiler $target || test -n "$container_image"; then
       test -n "$container_image" && build_static=y
-      mkdir -p "tests/tcg/$target"
-      config_target_mak=tests/tcg/$target/config-target.mak
-      ln -sf "$source_path/tests/tcg/Makefile.target" "tests/tcg/$target/Makefile"
+      mkdir -p "cross-build/$target"
+      config_target_mak=cross-build/$target/config-target.mak
       echo "# Automatically generated by configure - do not modify" > "$config_target_mak"
       echo "TARGET_NAME=$arch" >> "$config_target_mak"
       echo "TARGET=$target" >> "$config_target_mak"
-      write_target_makefile "build-tcg-tests-$target" >> "$config_target_mak"
+      write_target_makefile "$target" >> "$config_target_mak"
       echo "BUILD_STATIC=$build_static" >> "$config_target_mak"
       echo "QEMU=$PWD/$qemu" >> "$config_target_mak"
 
+      # get the interpreter prefix and the path of libnative required for native call tests
+      if [ -d "/usr/$(echo "$target_cc" | sed 's/-gcc//')" ]; then
+          echo "LD_PREFIX=/usr/$(echo "$target_cc" | sed 's/-gcc//')" >> "$config_target_mak"
+      fi
+
       # will GDB work with these binaries?
       if test "${gdb_arches#*$arch}" != "$gdb_arches"; then
           echo "HOST_GDB_SUPPORTS_ARCH=y" >> "$config_target_mak"
       fi
+  fi
+done
+
+# tests/tcg configuration
+(mkdir -p tests/tcg
+# create a symlink to the config-host.mak file in the tests/tcg
+ln -srf $config_host_mak tests/tcg/config-host.mak
+
+tcg_tests_targets=
+for target in $target_list; do
+  case $target in
+    *-softmmu)
+      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
+      ;;
+  esac
 
+  if test -f cross-build/$target/config-target.mak; then
+      mkdir -p "tests/tcg/$target"
+      ln -srf cross-build/$target/config-target.mak tests/tcg/$target/config-target.mak
+      ln -sf $source_path/tests/tcg/Makefile.target tests/tcg/$target/Makefile
       echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> Makefile.prereqs
       tcg_tests_targets="$tcg_tests_targets $target"
   fi
-- 
2.34.1
Re: [RFC v4 01/11] build: Implement logic for sharing cross-building config files
Posted by Alex Bennée 2 years, 6 months ago 
Yeqi Fu <fufuyqqqqqq@gmail.com> writes:

> Signed-off-by: Yeqi Fu <fufuyqqqqqq@gmail.com>
> ---
>  configure | 57 +++++++++++++++++++++++++++++++++----------------------
>  1 file changed, 34 insertions(+), 23 deletions(-)
>
> diff --git a/configure b/configure
> index 2b41c49c0d..a076583141 100755
> --- a/configure
> +++ b/configure
<snip>
>  
> +      # get the interpreter prefix and the path of libnative required for native call tests
> +      if [ -d "/usr/$(echo "$target_cc" | sed 's/-gcc//')" ]; then
> +          echo "LD_PREFIX=/usr/$(echo "$target_cc" | sed 's/-gcc//')" >> "$config_target_mak"
> +      fi
> +

We need some gating and testing here because for mips on my system we
fallback to docker:

  cat tests/tcg/mips-linux-user/config-target.mak
  # Automatically generated by configure - do not modify
  TARGET_NAME=mips
  TARGET=mips-linux-user
  EXTRA_CFLAGS=
  CC=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-gcc -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  CCAS=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-gcc -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  AR=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-ar -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  AS=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-as -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  LD=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-ld -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  NM=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-nm -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  OBJCOPY=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-objcopy -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  RANLIB=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-ranlib -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  STRIP=/home/alex/lsrc/qemu.git/builds/debug/pyvenv/bin/python3 -B /home/alex/lsrc/qemu.git/tests/docker/docker.py --engine docker cc --cc mips-linux-gnu-strip -i qemu/debian-mips-cross -s /home/alex/lsrc/qemu.git --
  BUILD_STATIC=y
  QEMU=/home/alex/lsrc/qemu.git/builds/debug/qemu-mips
  LD_PREFIX=/usr/
  HOST_GDB_SUPPORTS_ARCH=y
  LIBNATIVE=/home/alex/lsrc/qemu.git/builds/debug/common-user/native/mips-linux-user/libnative.so

but still set LD_PREFIX. We should at least check there is some sort of
ld.so in the LD_PREFIX path to indicate the loader is available.


>        # will GDB work with these binaries?
>        if test "${gdb_arches#*$arch}" != "$gdb_arches"; then
>            echo "HOST_GDB_SUPPORTS_ARCH=y" >> "$config_target_mak"
>        fi
> +  fi
> +done
> +
> +# tests/tcg configuration
> +(mkdir -p tests/tcg
> +# create a symlink to the config-host.mak file in the tests/tcg
> +ln -srf $config_host_mak tests/tcg/config-host.mak
> +
> +tcg_tests_targets=
> +for target in $target_list; do
> +  case $target in
> +    *-softmmu)
> +      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
> +      ;;
> +  esac
>  
> +  if test -f cross-build/$target/config-target.mak; then
> +      mkdir -p "tests/tcg/$target"
> +      ln -srf cross-build/$target/config-target.mak tests/tcg/$target/config-target.mak
> +      ln -sf $source_path/tests/tcg/Makefile.target tests/tcg/$target/Makefile
>        echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> Makefile.prereqs
>        tcg_tests_targets="$tcg_tests_targets $target"
>    fi


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [RFC v4 01/11] build: Implement logic for sharing cross-building config files
Posted by Alex Bennée 2 years, 6 months ago 
Yeqi Fu <fufuyqqqqqq@gmail.com> writes:

> Signed-off-by: Yeqi Fu <fufuyqqqqqq@gmail.com>
> ---
>  configure | 57 +++++++++++++++++++++++++++++++++----------------------
>  1 file changed, 34 insertions(+), 23 deletions(-)
>
> diff --git a/configure b/configure
> index 2b41c49c0d..a076583141 100755
> --- a/configure
> +++ b/configure
> @@ -1751,56 +1751,67 @@ if test "$ccache_cpp2" = "yes"; then
>    echo "export CCACHE_CPP2=y" >> $config_host_mak
>  fi
>  
> -# tests/tcg configuration
> -(config_host_mak=tests/tcg/config-host.mak
> -mkdir -p tests/tcg
> -echo "# Automatically generated by configure - do not modify" > $config_host_mak
> -echo "SRC_PATH=$source_path" >> $config_host_mak
> -echo "HOST_CC=$host_cc" >> $config_host_mak
> +# Prepare the config files for cross building.
> +# This process generates 'cross-build/<target>/config-target.mak' files.
> +# These files are then symlinked to the directories that need them which
> +# including the TCG tests (tests/tcg/<target>) and the libnative library
> +# for linux-user (common/native/<target>/).
> +mkdir -p cross-build
>  
> -# versioned checked in the main config_host.mak above
> -if test -n "$gdb_bin"; then
> -    echo "HAVE_GDB_BIN=$gdb_bin" >> $config_host_mak
> -fi
> -if test "$plugins" = "yes" ; then
> -    echo "CONFIG_PLUGIN=y" >> $config_host_mak
> -fi

I think there is a merge conflict here because a bunch of the
config-host.mak output has been squashed. This disabled plugins and gdb
testing.

> -
> -tcg_tests_targets=
>  for target in $target_list; do
>    arch=${target%%-*}
> -
>    case $target in
>      xtensa*-linux-user)
> -      # the toolchain is not complete with headers, only build softmmu tests
> +      # the toolchain for tests/tcg is not complete with headers
>        continue
>        ;;
>      *-softmmu)
> -      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue

We still want to skip linking tests/tcg/foo-softmmu/config-target.mak
when there are no softmmu tests to build (only a few targets currently
have softmmu tests). I think this is triggering failures like:

  ➜  make run-tcg-tests-m68k-softmmu V=1
  make -C tests/tcg/m68k-softmmu 
  make[1]: Entering directory '/home/alex/lsrc/qemu.git/builds/all/tests/tcg/m68k-softmmu'
  make[1]: Nothing to be done for 'all'.
  make[1]: Leaving directory '/home/alex/lsrc/qemu.git/builds/all/tests/tcg/m68k-softmmu'
  make -C tests/tcg/m68k-softmmu  SPEED=quick run
  make[1]: Entering directory '/home/alex/lsrc/qemu.git/builds/all/tests/tcg/m68k-softmmu'
  make[1]: *** No rule to make target 'hello', needed by 'run-plugin-hello-with-libbb.so'.  Stop.
  make[1]: Leaving directory '/home/alex/lsrc/qemu.git/builds/all/tests/tcg/m68k-softmmu'
  make: *** [/home/alex/lsrc/qemu.git/tests/Makefile.include:56: run-tcg-tests-m68k-softmmu] Erro

>        qemu="qemu-system-$arch"
>        ;;
>      *-linux-user|*-bsd-user)
>        qemu="qemu-$arch"
>        ;;
>    esac
> -
>    if probe_target_compiler $target || test -n "$container_image"; then
>        test -n "$container_image" && build_static=y
> -      mkdir -p "tests/tcg/$target"
> -      config_target_mak=tests/tcg/$target/config-target.mak
> -      ln -sf "$source_path/tests/tcg/Makefile.target" "tests/tcg/$target/Makefile"
> +      mkdir -p "cross-build/$target"
> +      config_target_mak=cross-build/$target/config-target.mak
>        echo "# Automatically generated by configure - do not modify" > "$config_target_mak"
>        echo "TARGET_NAME=$arch" >> "$config_target_mak"
>        echo "TARGET=$target" >> "$config_target_mak"
> -      write_target_makefile "build-tcg-tests-$target" >> "$config_target_mak"
> +      write_target_makefile "$target" >> "$config_target_mak"
>        echo "BUILD_STATIC=$build_static" >> "$config_target_mak"
>        echo "QEMU=$PWD/$qemu" >> "$config_target_mak"
>  
> +      # get the interpreter prefix and the path of libnative required for native call tests
> +      if [ -d "/usr/$(echo "$target_cc" | sed 's/-gcc//')" ]; then
> +          echo "LD_PREFIX=/usr/$(echo "$target_cc" | sed 's/-gcc//')" >> "$config_target_mak"
> +      fi
> +

We should only emit LD_PREFIX for -user targets.

>        # will GDB work with these binaries?
>        if test "${gdb_arches#*$arch}" != "$gdb_arches"; then
>            echo "HOST_GDB_SUPPORTS_ARCH=y" >> "$config_target_mak"
>        fi
> +  fi
> +done
> +
> +# tests/tcg configuration
> +(mkdir -p tests/tcg
> +# create a symlink to the config-host.mak file in the tests/tcg
> +ln -srf $config_host_mak tests/tcg/config-host.mak
> +
> +tcg_tests_targets=
> +for target in $target_list; do
> +  case $target in
> +    *-softmmu)
> +      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
> +      ;;
> +  esac
>  
> +  if test -f cross-build/$target/config-target.mak; then
> +      mkdir -p "tests/tcg/$target"
> +      ln -srf cross-build/$target/config-target.mak tests/tcg/$target/config-target.mak
> +      ln -sf $source_path/tests/tcg/Makefile.target tests/tcg/$target/Makefile
>        echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> Makefile.prereqs
>        tcg_tests_targets="$tcg_tests_targets $target"
>    fi


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [RFC v4 01/11] build: Implement logic for sharing cross-building config files
Posted by Alex Bennée 2 years, 6 months ago 
Alex Bennée <alex.bennee@linaro.org> writes:

> Yeqi Fu <fufuyqqqqqq@gmail.com> writes:
>
>> Signed-off-by: Yeqi Fu <fufuyqqqqqq@gmail.com>
>> ---
>>  configure | 57 +++++++++++++++++++++++++++++++++----------------------
>>  1 file changed, 34 insertions(+), 23 deletions(-)
>>
>> diff --git a/configure b/configure
>> index 2b41c49c0d..a076583141 100755
>> --- a/configure
>> +++ b/configure
>> @@ -1751,56 +1751,67 @@ if test "$ccache_cpp2" = "yes"; then
>>    echo "export CCACHE_CPP2=y" >> $config_host_mak
>>  fi
>>  
>> -# tests/tcg configuration
>> -(config_host_mak=tests/tcg/config-host.mak
>> -mkdir -p tests/tcg
>> -echo "# Automatically generated by configure - do not modify" > $config_host_mak
>> -echo "SRC_PATH=$source_path" >> $config_host_mak
>> -echo "HOST_CC=$host_cc" >> $config_host_mak
>> +# Prepare the config files for cross building.
>> +# This process generates 'cross-build/<target>/config-target.mak' files.
>> +# These files are then symlinked to the directories that need them which
>> +# including the TCG tests (tests/tcg/<target>) and the libnative library
>> +# for linux-user (common/native/<target>/).
>> +mkdir -p cross-build
>>  
>> -# versioned checked in the main config_host.mak above
>> -if test -n "$gdb_bin"; then
>> -    echo "HAVE_GDB_BIN=$gdb_bin" >> $config_host_mak
>> -fi
>> -if test "$plugins" = "yes" ; then
>> -    echo "CONFIG_PLUGIN=y" >> $config_host_mak
>> -fi
>
> I think there is a merge conflict here because a bunch of the
> config-host.mak output has been squashed. This disabled plugins and gdb
> testing.

Ahh I see now this was intentional because we symlink however it was
lost in the noise of the diff. As Manos pointed out detailing the
movement in the commit message aids reviewers in tracing what has
changed.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [RFC v4 01/11] build: Implement logic for sharing cross-building config files
Posted by Manos Pitsidianakis 2 years, 6 months ago 
This patch needs a detailed commit message, since it's not obvious why 
these changes are made at all. It'd also be helpful for reviewing.

General style comment for shell scripts: Always put curly braces around 
variables even if they are unnecessary. a $source_path could become 
$source_pathPREFIX in the future and instead of ${source_path} it would 
expand to ${source_pathPREFIX}.

On Tue, 08 Aug 2023 16:17, Yeqi Fu <fufuyqqqqqq@gmail.com> wrote:
>+tcg_tests_targets=
>+for target in $target_list; do
>+  case $target in
>+    *-softmmu)
>+      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
>+      ;;
>+  esac
> 
>+  if test -f cross-build/$target/config-target.mak; then

targets will never have spaces but I'd still double quote ${target} for 
consistency and style


>+      mkdir -p "tests/tcg/$target"
>+      ln -srf cross-build/$target/config-target.mak tests/tcg/$target/config-target.mak
>+      ln -sf $source_path/tests/tcg/Makefile.target tests/tcg/$target/Makefile

This ln definitely needs double quoting.

>       echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> Makefile.prereqs
>       tcg_tests_targets="$tcg_tests_targets $target"
>   fi
>-- 
>2.34.1
>
>

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.