tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI

[PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI

Thomas Huth posted 1 patch 2 years, 5 months ago

Download mbox

Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20230710144348.239441-1-thuth@redhat.com

Maintainers: Radoslaw Biernacki <rad@semihalf.com>, Peter Maydell <peter.maydell@linaro.org>, Leif Lindholm <quic_llindhol@quicinc.com>, Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>, Cleber Rosa <crosa@redhat.com>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>

tests/avocado/machine_aarch64_sbsaref.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Expand all Fold all

[PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI

Posted by Thomas Huth 2 years, 5 months ago

The test fails occasionally, see e.g.:

 https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
 https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290

Disable it at least in the gitlab-CI to avoid failing CI
pipelines due to this problem.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/avocado/machine_aarch64_sbsaref.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index cce6ef9f65..dbd364b537 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -8,8 +8,7 @@
 
 import os
 
-from avocado import skip
-from avocado import skipUnless
+from avocado import skipIf
 from avocado.utils import archive
 
 from avocado_qemu import QemuSystemTest
@@ -76,6 +75,7 @@ def fetch_firmware(self):
             "sbsa-ref",
         )
 
+    @skipIf(os.getenv('GITLAB_CI'), 'Test does not work reliably')
     def test_sbsaref_edk2_firmware(self):
         """
         :avocado: tags=cpu:cortex-a57
-- 
2.39.3

Re: [PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI

Posted by Peter Maydell 2 years, 5 months ago

On Mon, 10 Jul 2023 at 15:43, Thomas Huth <thuth@redhat.com> wrote:
>
> The test fails occasionally, see e.g.:
>
>  https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
>  https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290
>
> Disable it at least in the gitlab-CI to avoid failing CI
> pipelines due to this problem.

I'm not really a fan of disabling flaky tests only in the
gitlab CI. It means that plain "make check-avocado"
is useless to the average developer because it can
have all manner of broken or flaky tests in it that
never get tested by our CI. In fact I've had to develop
the habit of running "GITLAB_CI=1 make check-avocado"
so I can run the CI tests that we think work and not
get distracted by intermittent failures.

If we want "disable this test except for if you're
trying to investigate why it's flaky" then we should
have the test be opt-in, not opt-out, the same way
we do with QEMU_TEST_FLAKY_TESTS in the qtests.

thanks
-- PMM

Re: [PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI

Posted by Thomas Huth 2 years, 5 months ago

On 10/07/2023 16.54, Peter Maydell wrote:
> On Mon, 10 Jul 2023 at 15:43, Thomas Huth <thuth@redhat.com> wrote:
>>
>> The test fails occasionally, see e.g.:
>>
>>   https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
>>   https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290
>>
>> Disable it at least in the gitlab-CI to avoid failing CI
>> pipelines due to this problem.
> 
> I'm not really a fan of disabling flaky tests only in the
> gitlab CI. It means that plain "make check-avocado"
> is useless to the average developer because it can
> have all manner of broken or flaky tests in it that
> never get tested by our CI. In fact I've had to develop
> the habit of running "GITLAB_CI=1 make check-avocado"
> so I can run the CI tests that we think work and not
> get distracted by intermittent failures.
> 
> If we want "disable this test except for if you're
> trying to investigate why it's flaky" then we should
> have the test be opt-in, not opt-out, the same way
> we do with QEMU_TEST_FLAKY_TESTS in the qtests.

Agreed, this makes perfectly sense. Actually, I was looking for something 
like this in the tests/avocado directory, but I did not find anything 
similar, so I went for the GITLAB_CI variable ...

I'll send a v2 with QEMU_TEST_FLAKY_TESTS instead (and document it in the 
docs directory).

  Thomas