When determining the endiandness of the target architecture we're
building for a small program is compiled, which in an obfuscated
way declares two strings. Then, we look which string is in
correct order (using strings binary) and deduct the endiandness.
But using the strings binary is problematic, because it's part of
toolchain (strings is just a symlink to
x86_64-pc-linux-gnu-strings or llvm-strings). And when
(cross-)compiling, it requires users to set the symlink to the
correct toolchain.

Fortunately, we have a better alternative anyways. We can mimic
what compiler.h is already doing: comparing __BYTE_ORDER__
against values for little/big endiandness.

Bug: https://bugs.gentoo.org/876933
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
---

v2 of:

https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg02054.html

diff to v1:
- Fixed reversed logic
- Ditched custom compiler macros in favor of __BYTE_ORDER__

 configure | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/configure b/configure
index 45ee6f4eb3..2ac26c6978 100755
--- a/configure
+++ b/configure
@@ -1423,30 +1423,31 @@ if test "$tcg" = "enabled"; then
     git_submodules="$git_submodules tests/fp/berkeley-softfloat-3"
 fi
 
-# ---
+##########################################
 # big/little endian test
 cat > $TMPC << EOF
-#include <stdio.h>
-short big_endian[] = { 0x4269, 0x4765, 0x4e64, 0x4961, 0x4e00, 0, };
-short little_endian[] = { 0x694c, 0x7454, 0x654c, 0x6e45, 0x6944, 0x6e41, 0, };
-int main(int argc, char *argv[])
-{
-    return printf("%s %s\n", (char *)big_endian, (char *)little_endian);
-}
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+# error LITTLE
+#endif
+int main(void) { return 0; }
 EOF
 
 if compile_prog ; then
-    if strings -a $TMPE | grep -q BiGeNdIaN ; then
-        bigendian="yes"
-    elif strings -a $TMPE | grep -q LiTtLeEnDiAn ; then
-        bigendian="no"
-    else
-        echo big/little test failed
-        exit 1
-    fi
+  bigendian="yes"
 else
+  cat > $TMPC << EOF
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+# error BIG
+#endif
+int main(void) { return 0; }
+EOF
+
+  if compile_prog ; then
+    bigendian="no"
+  else
     echo big/little test failed
     exit 1
+  fi
 fi
 
 ##########################################
-- 
2.35.1

On 10/13/22 15:43, Michal Privoznik wrote:
> When determining the endiandness of the target architecture we're
> building for a small program is compiled, which in an obfuscated
> way declares two strings. Then, we look which string is in
> correct order (using strings binary) and deduct the endiandness.
> But using the strings binary is problematic, because it's part of
> toolchain (strings is just a symlink to
> x86_64-pc-linux-gnu-strings or llvm-strings). And when
> (cross-)compiling, it requires users to set the symlink to the
> correct toolchain.
> 
> Fortunately, we have a better alternative anyways. We can mimic
> what compiler.h is already doing: comparing __BYTE_ORDER__
> against values for little/big endiandness.
> 
> Bug: https://bugs.gentoo.org/876933
> Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
> ---
> 
> v2 of:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg02054.html
> 
> diff to v1:
> - Fixed reversed logic
> - Ditched custom compiler macros in favor of __BYTE_ORDER__
> 
>   configure | 33 +++++++++++++++++----------------
>   1 file changed, 17 insertions(+), 16 deletions(-)
> 
> diff --git a/configure b/configure
> index 45ee6f4eb3..2ac26c6978 100755
> --- a/configure
> +++ b/configure
> @@ -1423,30 +1423,31 @@ if test "$tcg" = "enabled"; then
>       git_submodules="$git_submodules tests/fp/berkeley-softfloat-3"
>   fi
>   
> -# ---
> +##########################################
>   # big/little endian test
>   cat > $TMPC << EOF
> -#include <stdio.h>
> -short big_endian[] = { 0x4269, 0x4765, 0x4e64, 0x4961, 0x4e00, 0, };
> -short little_endian[] = { 0x694c, 0x7454, 0x654c, 0x6e45, 0x6944, 0x6e41, 0, };
> -int main(int argc, char *argv[])
> -{
> -    return printf("%s %s\n", (char *)big_endian, (char *)little_endian);
> -}
> +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__

#if defined __BYTE_ORDER__ && ...

> +# error LITTLE
> +#endif
> +int main(void) { return 0; }
>   EOF
>   
>   if compile_prog ; then

And then you need to check if it *fails* (i.e. "if ! compile_prog ; then 
bigendian=no", and likewise below for the big-endian test).  This way if 
the macro is not defined both tests will compile successfully and you'll 
print the error here:

> +  if compile_prog ; then
> +    bigendian="no"
> +  else
>       echo big/little test failed
>       exit 1
> +  fi
>   fi

You can test it by misspelling __BYTE_ORDER__ intentionally.

Thanks,

Paolo