tests/qtest: Run migration subtests via meson

[RFC PATCH 0/2] tests/qtest: Run migration subtests via meson

Fabiano Rosas posted 2 patches 10 months ago

Download series mbox

Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20230703144700.13536-1-farosas@suse.de

Maintainers: Yonggang Luo <luoyonggang@gmail.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>, Laurent Vivier <lvivier@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>

.gitlab-ci.d/windows.yml                |  2 +-
tests/qtest/gen_migration_tests_list.py | 12 ++++++++
tests/qtest/meson.build                 | 40 +++++++++++++++++++------
3 files changed, 44 insertions(+), 10 deletions(-)
create mode 100644 tests/qtest/gen_migration_tests_list.py

Expand all Fold all

[RFC PATCH 0/2] tests/qtest: Run migration subtests via meson

Posted by Fabiano Rosas 10 months ago

We've had some issues with the migration-test failing in CI and giving
no clue as to which migration sub-test has failed. I think it would be
an improvement to have each subtest listed individually.

With this we can see which migration test has failed, can set timeouts
individually and can run 'make check-migration' to run only the
migration tests.

It does however increase the amount of lines emitted with make check,
so let me know if this is an issue.

I had to include some hacking due to meson, but I think it is simple
enough and worth the benefits.

CI run: https://gitlab.com/farosas/qemu/-/pipelines/919324510

Fabiano Rosas (2):
  tests/qtest: Add a script to gather migration tests list
  tests/qtest: Pass migration tests individually to meson

 .gitlab-ci.d/windows.yml                |  2 +-
 tests/qtest/gen_migration_tests_list.py | 12 ++++++++
 tests/qtest/meson.build                 | 40 +++++++++++++++++++------
 3 files changed, 44 insertions(+), 10 deletions(-)
 create mode 100644 tests/qtest/gen_migration_tests_list.py

-- 
2.35.3

Re: [RFC PATCH 0/2] tests/qtest: Run migration subtests via meson

Posted by Daniel P. Berrangé 10 months ago

On Mon, Jul 03, 2023 at 11:46:58AM -0300, Fabiano Rosas wrote:
> We've had some issues with the migration-test failing in CI and giving
> no clue as to which migration sub-test has failed. I think it would be
> an improvement to have each subtest listed individually.

If the migration-test (or any test) actually fails, the meson test
log should show us exactly which test case failed already. We had
some jobs which didn't publish the log as an artifact, but I think
we finally have those all fixed.

If the migration-test hangs, however, then we get stuck until the
GitLab CI job timeout and get no useful info. For this scenario
we need to enable timeouts at the meson level, which are currently
all disabled. I've got a series here to address this:

  https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg00305.html

It needs 

> 
> With this we can see which migration test has failed, can set timeouts
> individually and can run 'make check-migration' to run only the
> migration tests.

IMHO focusing just on migration-test is undesirable, as while this
has been one of the least reliable ones, other tests do also exhibit
hard to debug failures for much the same reasons. IOW, we need
improved debugging across the board rather than specialcased solutions.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|