Hi Julien and Stefano

I found that the "cc-option/cc-option-add" is not working for "-march=xxx" option on a few very common aarch64 compilers.
Here is what I got when trying to compile XEN on r82 platform.
```
diff --git a/xen/arch/arm/arch.mk b/xen/arch/arm/arch.mk
index 11caec86ba..d2d71baef4 100644
--- a/xen/arch/arm/arch.mk
+++ b/xen/arch/arm/arch.mk
@@ -10,6 +10,10 @@ $(call cc-option-add,CFLAGS,CC,-Wnested-externs)
CFLAGS-$(CONFIG_ARM_32) += -msoft-float
CFLAGS-$(CONFIG_ARM_32) += -mcpu=cortex-a15

+#ifeq ($(CONFIG_ARM64_V8R),y)
+    CFLAGS-$(CONFIG_ARM_64) += $(call cc-option,$(CC),-march=armv8-r,-march=armv8.4-a)
+#endif
+
CFLAGS-$(CONFIG_ARM_64) += -mcpu=generic
CFLAGS-$(CONFIG_ARM_64) += -mgeneral-regs-only # No fp registers etc
$(call cc-option-add,CFLAGS-$(CONFIG_ARM_64),CC,-mno-outline-atomics)
```
My gcc compiler version is as follows:
```
Using built-in specs.
COLLECT_GCC=aarch64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/aarch64-linux-gnu/9/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04'
--with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,
go,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9
--enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
--enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin
--enable-default-pie --with-system-zlib --without-target-system-zlib --enable-libpth-m2 --enable-multiarch
--enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu- --includedir=/usr/aarch64-linux-gnu/include
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
```
However. even if my gcc compiler is indeed not supporting "-march=armv8-r", above check will not identify it
"unrecognized option" and seek to the "-march=armv8.4-a" as expected.
See the resulting errors, it still uses "-march=armv8-r":
```
make[3]: Entering directory '/home/penny/FVP/r82/fvp_aemv8r_cortexr82_0.0_6347_gcc6.4/xen/xen'
aarch64-linux-gnu-gcc -MMD -MP -MF ./.asm-offsets.s.d -DBUILD_ID -fno-strict-aliasing -std=gnu99 -Wall
-Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs
-O1 -fno-omit-frame-pointer -nostdinc -fno-builtin -fno-common -Werror -Wredundant-decls -Wno-pointer-arith -Wvla
-pipe -D__XEN__ -include /home/penny/FVP/r82/fvp_aemv8r_cortexr82_0.0_6347_gcc6.4/xen/xen/include/xen/config.h
-g -march=armv8-r -mgeneral-regs-only -mno-outline-atomics -I/home/penny/FVP/r82/fvp_aemv8r_cortexr82_0.0_6347_gcc6.4
/xen/xen/include -fno-stack-protector -fno-exceptions -fno-asynchronous-unwind-tables -fcf-protection=none
-Wnested-externs '-D__OBJECT_FILE__="asm-offsets.s"' -S -g0 -o asm-offsets.s.new -MQ asm-offsets.s arch/arm/arm64/asm-offsets.c
cc1: error: unknown value 'armv8-r' for '-march'
cc1: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a armv8.5-a; did you mean 'armv8-a'?
make[3]: *** [Makefile:413: asm-offsets.s] Error 1
```
And the reason is that in the implementation of "cc-option": To handle this we do a test compile, passing the option-under-test,
on a code fragment that will always produce a warning (integer assigned to pointer). We then grep for the option-under-test in
the compiler's output, the presence of which would indicate an "unrecognized command-line option" warning/error.
```
cc-option = $(shell if test -z "`echo 'void*p=1;' | \
                     $(1) $(2) -S -o /dev/null -x c - 2>&1 | grep -- $(2) -`"; \
                     then echo "$(2)"; else echo "$(3)"; fi ;)
```
Here, that is, we are trying to grep "-march=armv8-r" in output message of the test compile, and since the output
message(unknown value 'armv8-r' for '-march') doesn't contain the COMPLETE string, it thought the compiler supported this
option, but it actually not...

Tracing back to the commit about the implementation of "cc-option", it discards the linux method(using exit code to tell) for dealing with
disable-warning options.
See https://github.com/xen-project/xen/commit/28f8fb7d2b3fde2f5cbe5526ac4f1c932e3f5d26 for more details.

To fix this issue, I could loose the "grep check", instead trying to grep the COMPLETE option string("-march=armv8-r"), only partial(armv8-r) is
enough. But, hmmm, the way of greping output message to check is kinds of reliable, IMO, since there is no standard for that.
What do you suggest?

Cheers

--
Penny Zheng

On 07.09.2021 15:03, Penny Zheng wrote:
> Hi Julien and Stefano
> 
> I found that the "cc-option/cc-option-add" is not working for "-march=xxx" option on a few very common aarch64 compilers.
> Here is what I got when trying to compile XEN on r82 platform.
> ```
> diff --git a/xen/arch/arm/arch.mk b/xen/arch/arm/arch.mk
> index 11caec86ba..d2d71baef4 100644
> --- a/xen/arch/arm/arch.mk
> +++ b/xen/arch/arm/arch.mk
> @@ -10,6 +10,10 @@ $(call cc-option-add,CFLAGS,CC,-Wnested-externs)
> CFLAGS-$(CONFIG_ARM_32) += -msoft-float
> CFLAGS-$(CONFIG_ARM_32) += -mcpu=cortex-a15
> 
> +#ifeq ($(CONFIG_ARM64_V8R),y)
> +    CFLAGS-$(CONFIG_ARM_64) += $(call cc-option,$(CC),-march=armv8-r,-march=armv8.4-a)
> +#endif
> +
> CFLAGS-$(CONFIG_ARM_64) += -mcpu=generic
> CFLAGS-$(CONFIG_ARM_64) += -mgeneral-regs-only # No fp registers etc
> $(call cc-option-add,CFLAGS-$(CONFIG_ARM_64),CC,-mno-outline-atomics)
> ```
> My gcc compiler version is as follows:
> ```
> Using built-in specs.
> COLLECT_GCC=aarch64-linux-gnu-gcc
> COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/aarch64-linux-gnu/9/lto-wrapper
> Target: aarch64-linux-gnu
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04'
> --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,
> go,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9
> --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext
> --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu
> --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
> --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin
> --enable-default-pie --with-system-zlib --without-target-system-zlib --enable-libpth-m2 --enable-multiarch
> --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=x86_64-linux-gnu
> --host=x86_64-linux-gnu --target=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu- --includedir=/usr/aarch64-linux-gnu/include
> Thread model: posix
> gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
> ```
> However. even if my gcc compiler is indeed not supporting "-march=armv8-r", above check will not identify it
> "unrecognized option" and seek to the "-march=armv8.4-a" as expected.
> See the resulting errors, it still uses "-march=armv8-r":
> ```
> make[3]: Entering directory '/home/penny/FVP/r82/fvp_aemv8r_cortexr82_0.0_6347_gcc6.4/xen/xen'
> aarch64-linux-gnu-gcc -MMD -MP -MF ./.asm-offsets.s.d -DBUILD_ID -fno-strict-aliasing -std=gnu99 -Wall
> -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs
> -O1 -fno-omit-frame-pointer -nostdinc -fno-builtin -fno-common -Werror -Wredundant-decls -Wno-pointer-arith -Wvla
> -pipe -D__XEN__ -include /home/penny/FVP/r82/fvp_aemv8r_cortexr82_0.0_6347_gcc6.4/xen/xen/include/xen/config.h
> -g -march=armv8-r -mgeneral-regs-only -mno-outline-atomics -I/home/penny/FVP/r82/fvp_aemv8r_cortexr82_0.0_6347_gcc6.4
> /xen/xen/include -fno-stack-protector -fno-exceptions -fno-asynchronous-unwind-tables -fcf-protection=none
> -Wnested-externs '-D__OBJECT_FILE__="asm-offsets.s"' -S -g0 -o asm-offsets.s.new -MQ asm-offsets.s arch/arm/arm64/asm-offsets.c
> cc1: error: unknown value 'armv8-r' for '-march'
> cc1: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a armv8.5-a; did you mean 'armv8-a'?
> make[3]: *** [Makefile:413: asm-offsets.s] Error 1
> ```
> And the reason is that in the implementation of "cc-option": To handle this we do a test compile, passing the option-under-test,
> on a code fragment that will always produce a warning (integer assigned to pointer). We then grep for the option-under-test in
> the compiler's output, the presence of which would indicate an "unrecognized command-line option" warning/error.
> ```
> cc-option = $(shell if test -z "`echo 'void*p=1;' | \
>                      $(1) $(2) -S -o /dev/null -x c - 2>&1 | grep -- $(2) -`"; \
>                      then echo "$(2)"; else echo "$(3)"; fi ;)
> ```
> Here, that is, we are trying to grep "-march=armv8-r" in output message of the test compile, and since the output
> message(unknown value 'armv8-r' for '-march') doesn't contain the COMPLETE string, it thought the compiler supported this
> option, but it actually not...
> 
> Tracing back to the commit about the implementation of "cc-option", it discards the linux method(using exit code to tell) for dealing with
> disable-warning options.
> See https://github.com/xen-project/xen/commit/28f8fb7d2b3fde2f5cbe5526ac4f1c932e3f5d26 for more details.
> 
> To fix this issue, I could loose the "grep check", instead trying to grep the COMPLETE option string("-march=armv8-r"), only partial(armv8-r) is
> enough. But, hmmm, the way of greping output message to check is kinds of reliable, IMO, since there is no standard for that.

Since we need to be careful about not relaxing this too much, my
immediate suggestion here would be to split option strings at =
signs, and then use grep to check that every resulting piece is
present in the error message coming from the compiler. Afaict
this should cover your case and might also make a few others
less fragile. Unfortunately I expect this will make for quite a
bit more complicated a make macro. I wonder whether Linux hasn't
run into the same problem at some point, and has found some more
elegant and/or robust solution.

Jan