gitlab: fix failing TSAN job in CI

[PATCH 3/3] gitlab: use 'setarch -R' to workaround tsan bug

Posted by Daniel P. Berrangé 6 months, 2 weeks ago

The TSAN job started failing when gitlab rolled out their latest
release. The root cause is a change in the Google COS version used
on shared runners. This brings a kernel running with

 vm.mmap_rnd_bits = 31

which is incompatible with TSAN in LLVM < 18, which only supports
upto '28'. LLVM 18 can support upto '30', and failing that will
re-exec itself to turn off VA randomization.

Our LLVM is too old for now, but we can run with 'setarch -R make ..'
to turn off VA randomization ourselves.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index bab6194564..d864562628 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -575,6 +575,9 @@ tsan-build:
     CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
           --enable-trace-backends=ust --disable-slirp
     TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
+    # Remove when we switch to a distro with clang >= 18
+    # https://github.com/google/sanitizers/issues/1716
+    MAKE: setarch -R make
 
 # gcov is a GCC features
 gcov:
-- 
2.43.0

Re: [PATCH 3/3] gitlab: use 'setarch -R' to workaround tsan bug

Posted by Daniel P. Berrangé 6 months, 2 weeks ago

On Mon, May 13, 2024 at 12:15:51PM +0100, Daniel P. Berrangé wrote:
> The TSAN job started failing when gitlab rolled out their latest
> release. The root cause is a change in the Google COS version used
> on shared runners. This brings a kernel running with
> 
>  vm.mmap_rnd_bits = 31
> 
> which is incompatible with TSAN in LLVM < 18, which only supports
> upto '28'. LLVM 18 can support upto '30', and failing that will
> re-exec itself to turn off VA randomization.
> 
> Our LLVM is too old for now, but we can run with 'setarch -R make ..'
> to turn off VA randomization ourselves.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml | 3 +++
>  1 file changed, 3 insertions(+)

Example job showing this working:

  https://gitlab.com/berrange/qemu/-/jobs/6824465594

 > 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index bab6194564..d864562628 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -575,6 +575,9 @@ tsan-build:
>      CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
>            --enable-trace-backends=ust --disable-slirp
>      TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
> +    # Remove when we switch to a distro with clang >= 18
> +    # https://github.com/google/sanitizers/issues/1716
> +    MAKE: setarch -R make
>  
>  # gcov is a GCC features
>  gcov:
> -- 
> 2.43.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/3] gitlab: use 'setarch -R' to workaround tsan bug

Posted by Thomas Huth 6 months, 2 weeks ago

On 13/05/2024 13.15, Daniel P. Berrangé wrote:
> The TSAN job started failing when gitlab rolled out their latest
> release. The root cause is a change in the Google COS version used
> on shared runners. This brings a kernel running with
> 
>   vm.mmap_rnd_bits = 31
> 
> which is incompatible with TSAN in LLVM < 18, which only supports
> upto '28'. LLVM 18 can support upto '30', and failing that will
> re-exec itself to turn off VA randomization.
> 
> Our LLVM is too old for now, but we can run with 'setarch -R make ..'
> to turn off VA randomization ourselves.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index bab6194564..d864562628 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -575,6 +575,9 @@ tsan-build:
>       CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
>             --enable-trace-backends=ust --disable-slirp
>       TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
> +    # Remove when we switch to a distro with clang >= 18
> +    # https://github.com/google/sanitizers/issues/1716
> +    MAKE: setarch -R make

Thanks for tackling this!

Reviewed-by: Thomas Huth <thuth@redhat.com>