The nature of block I/O tests is such that there can be unexpected false
positive failures in certain scenarios that have not been encountered
before, and sometimes non-deterministic failures that are hard to
reproduce.
Before enabling the I/O tests as gating jobs in CI, there needs to be a
mechanism to dynamically mark tests as skipped, without having to commit
code changes.
This introduces the QEMU_TEST_IO_SKIP environment variable that is set
to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
variable can be set as a GitLab CI pipeline variable to temporarily
disable a test while problems are being debugged.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
docs/devel/testing/main.rst | 7 +++++++
tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 910ec11ab4..bb90ea5f15 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -285,6 +285,13 @@ that are specific to certain cache mode.
More options are supported by the ``./check`` script, run ``./check -h`` for
help.
+If a test program is known to be broken, it can be disabled by setting
+the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
+be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
+leading component can be omitted to skip the test for all formats and
+protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
+will skip for LUKS only, and ``185`` and ``iov-padding`` for all.
+
Writing a new test case
~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index dbe2dddc32..ecb5d4529f 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
self._stack: contextlib.ExitStack
+ self.skip = {}
+ for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
+ rule = rule.strip()
+ if rule == "":
+ continue
+ if ":" in rule:
+ fmt, name = rule.split(":")
+ if fmt in ("", env.imgfmt, env.imgproto):
+ self.skip[name] = True
+ else:
+ self.skip[rule] = True
+
def __enter__(self) -> 'TestRunner':
self._stack = contextlib.ExitStack()
self._stack.enter_context(self.env)
@@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
description='No qualified output '
f'(expected {f_reference})')
+ if f_test.name in self.skip:
+ return TestResult(status='not run',
+ description='Listed in QEMU_TEST_IO_SKIP')
+
args = [str(f_test.resolve())]
env = self.env.prepare_subprocess(args)
--
2.52.0