[PATCH v2] tests/avocado: Disable the test_sbsaref_edk2_firmware by default

Thomas Huth posted 1 patch 2 years, 5 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20230710170155.7192-1-thuth@redhat.com
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>, Radoslaw Biernacki <rad@semihalf.com>, Peter Maydell <peter.maydell@linaro.org>, Leif Lindholm <quic_llindhol@quicinc.com>, Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>, Cleber Rosa <crosa@redhat.com>
docs/devel/testing.rst                   | 5 +++++
tests/avocado/machine_aarch64_sbsaref.py | 2 +-
2 files changed, 6 insertions(+), 1 deletion(-)
[PATCH v2] tests/avocado: Disable the test_sbsaref_edk2_firmware by default
Posted by Thomas Huth 2 years, 5 months ago
The test fails occasionally, see e.g.:

 https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
 https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290

It also fails on my laptop in ca. 50% of all runs. Thus disable it by
default by using the QEMU_TEST_FLAKY_TESTS environment variable to fence
it (which we also already use in flaky qtests). While we're at it, also
document this variable in docs/devel/testing.rst.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 v2: Use QEMU_TEST_FLAKY_TESTS instead of GITLAB_CI as sugested by PMM

 docs/devel/testing.rst                   | 5 +++++
 tests/avocado/machine_aarch64_sbsaref.py | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index fb651eae11..b6ad21bed1 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1384,6 +1384,11 @@ variable as shown on the code snippet below to skip the test:
   def test(self):
       do_something()
 
+QEMU_TEST_FLAKY_TESTS
+^^^^^^^^^^^^^^^^^^^^^
+Some tests are not working reliably and thus are disabled by default.
+Set this environment variable to enable them.
+
 Uninstalling Avocado
 ~~~~~~~~~~~~~~~~~~~~
 
diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index cce6ef9f65..a794245e7e 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -8,7 +8,6 @@
 
 import os
 
-from avocado import skip
 from avocado import skipUnless
 from avocado.utils import archive
 
@@ -76,6 +75,7 @@ def fetch_firmware(self):
             "sbsa-ref",
         )
 
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is not reliable')
     def test_sbsaref_edk2_firmware(self):
         """
         :avocado: tags=cpu:cortex-a57
-- 
2.39.3