The cross-i686-tci CI job is persistently flaky with various tests
hitting timeouts.  One theory for why this is happening is that we're
running too many tests in parallel and so sometimes a test gets
starved of CPU and isn't able to complete within the timeout.

Set the MESON_TESTTHREADS environment variable to 1 for this job;
this will cause 'meson test' to run only one test at a time.

(Note that this relies on the change to meson2make that makes it
honour MESON_TESTTHREADS; otherwise it will have no effect.)

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
Seems worth a try -- if this doesn't have an effect then
we can revert it, but we'll at least have determined what
the problem isn't...
---
 .gitlab-ci.d/crossbuilds.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index cb499e4ee0d..ca1db011b11 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -70,6 +70,9 @@ cross-i686-tci:
     ACCEL: tcg-interpreter
     EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm
     MAKE_CHECK_ARGS: check check-tcg
+    # Force 'meson test' to run only one test at once, to
+    # see whether this reduces the flakiness of this CI job.
+    MESON_TESTTHREADS: 1
 
 cross-mipsel-system:
   extends: .cross_system_build_job
-- 
2.34.1

On 06/09/2024 20.07, Peter Maydell wrote:
> The cross-i686-tci CI job is persistently flaky with various tests
> hitting timeouts.  One theory for why this is happening is that we're
> running too many tests in parallel and so sometimes a test gets
> starved of CPU and isn't able to complete within the timeout.
> 
> Set the MESON_TESTTHREADS environment variable to 1 for this job;
> this will cause 'meson test' to run only one test at a time.
> 
> (Note that this relies on the change to meson2make that makes it
> honour MESON_TESTTHREADS; otherwise it will have no effect.)
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
> Seems worth a try -- if this doesn't have an effect then
> we can revert it, but we'll at least have determined what
> the problem isn't...
> ---
>   .gitlab-ci.d/crossbuilds.yml | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index cb499e4ee0d..ca1db011b11 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -70,6 +70,9 @@ cross-i686-tci:
>       ACCEL: tcg-interpreter
>       EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm
>       MAKE_CHECK_ARGS: check check-tcg
> +    # Force 'meson test' to run only one test at once, to
> +    # see whether this reduces the flakiness of this CI job.
> +    MESON_TESTTHREADS: 1

Can't we simply add "-j1" to the MAKE_CHECK_ARGS line?

According to the man-page of "make":

  "If there is more than one -j option, the last one is effective."

So adding a -j1 should override the previous setting, I think.

  Thomas

On Tue, 10 Sept 2024 at 13:13, Thomas Huth <thuth@redhat.com> wrote:
>
> On 06/09/2024 20.07, Peter Maydell wrote:
> > The cross-i686-tci CI job is persistently flaky with various tests
> > hitting timeouts.  One theory for why this is happening is that we're
> > running too many tests in parallel and so sometimes a test gets
> > starved of CPU and isn't able to complete within the timeout.
> >
> > Set the MESON_TESTTHREADS environment variable to 1 for this job;
> > this will cause 'meson test' to run only one test at a time.
> >
> > (Note that this relies on the change to meson2make that makes it
> > honour MESON_TESTTHREADS; otherwise it will have no effect.)
> >
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> > ---
> > Seems worth a try -- if this doesn't have an effect then
> > we can revert it, but we'll at least have determined what
> > the problem isn't...
> > ---
> >   .gitlab-ci.d/crossbuilds.yml | 3 +++
> >   1 file changed, 3 insertions(+)
> >
> > diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> > index cb499e4ee0d..ca1db011b11 100644
> > --- a/.gitlab-ci.d/crossbuilds.yml
> > +++ b/.gitlab-ci.d/crossbuilds.yml
> > @@ -70,6 +70,9 @@ cross-i686-tci:
> >       ACCEL: tcg-interpreter
> >       EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm
> >       MAKE_CHECK_ARGS: check check-tcg
> > +    # Force 'meson test' to run only one test at once, to
> > +    # see whether this reduces the flakiness of this CI job.
> > +    MESON_TESTTHREADS: 1
>
> Can't we simply add "-j1" to the MAKE_CHECK_ARGS line?
>
> According to the man-page of "make":
>
>   "If there is more than one -j option, the last one is effective."
>
> So adding a -j1 should override the previous setting, I think.

You would also be relying on the rune in mtest2make to
pick the last -j option in the list, but luckily I think
it already does that:

.mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword
-j1 $(filter -j%, $(MAKEFLAGS)))))

thanks
-- PMM