[RFC PATCH 0/4] Replace custom test harness with "meson test"

Paolo Bonzini posted 4 patches 2 years, 6 months ago
Test checkpatch failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20211015100718.17828-1-pbonzini@redhat.com
Maintainers: Aurelien Jarno <aurelien@aurel32.net>, Willian Rampazzo <willianr@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, Kevin Wolf <kwolf@redhat.com>, Peter Maydell <peter.maydell@linaro.org>, Hanna Reitz <hreitz@redhat.com>, Cleber Rosa <crosa@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, "Philippe Mathieu-Daudé" <f4bug@amsat.org>
There is a newer version of this series
.gitlab-ci.d/crossbuild-template.yml |   2 +-
Makefile                             |   3 +-
configure                            |   3 -
meson.build                          |   5 +-
scripts/mtest2make.py                | 111 +++-----
scripts/tap-driver.pl                | 379 ---------------------------
scripts/tap-merge.pl                 | 111 --------
tests/Makefile.include               |  15 +-
tests/check-block.sh                 |  28 +-
tests/fp/meson.build                 |   2 +-
tests/meson.build                    |   1 +
tests/qemu-iotests/check             |   6 +-
tests/qemu-iotests/meson.build       |  14 +
tests/qemu-iotests/testenv.py        |  30 +--
tests/qemu-iotests/testrunner.py     |  48 ++--
15 files changed, 121 insertions(+), 637 deletions(-)
delete mode 100755 scripts/tap-driver.pl
delete mode 100755 scripts/tap-merge.pl
create mode 100644 tests/qemu-iotests/meson.build
[RFC PATCH 0/4] Replace custom test harness with "meson test"
Posted by Paolo Bonzini 2 years, 6 months ago
Hi all,

Starting with Meson 0.57, "meson test" has all features of QEMU's
makefile-based harness and more.  In particular, some features that
were added to reach feature parity are:

* print reproducer command line for each failed test right below the test

* keep the output of multiple (non-TAP) tests together in verbose mode,
  similar to "make --output-sync target"

* report on TAP subtests as they are encountered

It also includes nicer handling of test interruption, logging of the run
in the meson-logs/ subdirectory, and a progress report/spinner.  For
these reasons it would be nice to adopt it and remove the Perl scripts
that we have to present TAP output nicely.  While at it, this series
also changes qemu-iotests to be described in tests/qemu-iotests/meson.build
and it replaces the current "-makecheck" output style with TAP.

This is an RFC just to let people try it out and because I still haven't
sorted out timeouts (e.g.  OpenBSD is insanely slow on some tests).
But in theory, if you want to try it out, the only things you need to
know are:

* while "make check" does not timeout tests like it doesn't right now,
  "meson test" does respect timeouts, so you could get failures because
  of that.  That can and probably will change from RFC to final.

* CTRL+C will only interrupt the longest running test.  Pressing
  CTRL+C repeatedly three times (which you would likely do anyway,
  that's how things work) interrupts the whole run

* Right now "make check-block" only does a single test run just like
  "../tests/check-block.sh", but it would be possible to add the thorough
  suite to "meson test --suite block" as well.

* If you were using make check-report.tap and similar, they are replaced
  by targets like make check-report.junit.xml.  This is because Gitlab
  is able to parse the resulting XML and include on the website a report
  of which tests failed.

An example of non-verbose "make check" output is available at
https://gitlab.com/bonzini/qemu/-/jobs/1680980620.  A verbose run
instead is like https://asciinema.org/a/e5irnEszSnAheOHM30exbo3F6
(does not include check-block).

Paolo

Paolo Bonzini (4):
  build: use "meson test" as the test harness
  build: make check-block a meson test
  check-block: replace -makecheck with TAP output
  configure: remove dead EXESUF variable

 .gitlab-ci.d/crossbuild-template.yml |   2 +-
 Makefile                             |   3 +-
 configure                            |   3 -
 meson.build                          |   5 +-
 scripts/mtest2make.py                | 111 +++-----
 scripts/tap-driver.pl                | 379 ---------------------------
 scripts/tap-merge.pl                 | 111 --------
 tests/Makefile.include               |  15 +-
 tests/check-block.sh                 |  28 +-
 tests/fp/meson.build                 |   2 +-
 tests/meson.build                    |   1 +
 tests/qemu-iotests/check             |   6 +-
 tests/qemu-iotests/meson.build       |  14 +
 tests/qemu-iotests/testenv.py        |  30 +--
 tests/qemu-iotests/testrunner.py     |  48 ++--
 15 files changed, 121 insertions(+), 637 deletions(-)
 delete mode 100755 scripts/tap-driver.pl
 delete mode 100755 scripts/tap-merge.pl
 create mode 100644 tests/qemu-iotests/meson.build

-- 
2.31.1


Re: [RFC PATCH 0/4] Replace custom test harness with "meson test"
Posted by Thomas Huth 2 years, 6 months ago
On 15/10/2021 12.07, Paolo Bonzini wrote:
> Hi all,
> 
> Starting with Meson 0.57, "meson test" has all features of QEMU's
> makefile-based harness and more.

I just gave it a try, and basically I like this ... but I also encountered 
two issues:

> * CTRL+C will only interrupt the longest running test.  Pressing
>    CTRL+C repeatedly three times (which you would likely do anyway,
>    that's how things work) interrupts the whole run

I tried this, and while hitting CTRL-C multiple times brought me back to the 
shell prompt, the remaining tests kept getting started in the background 
instead of getting stopped ... something is still fishy here, I think.

> * Right now "make check-block" only does a single test run just like
>    "../tests/check-block.sh", but it would be possible to add the thorough
>    suite to "meson test --suite block" as well.

The output of the iotests is also not optimal yet... when running "make 
check SPEED=slow", the iotests are run multiple times with different target 
image types, but each run prints the same "▶ 1/1 test 001   OK" etc. to the 
console, so it's hard to say which target type  is currently exercised. 
Would it be possible to include the target image type here, e.g. something like:

▶ 1/1 test-qcow2 001                       OK

?

  Thomas


Re: [RFC PATCH 0/4] Replace custom test harness with "meson test"
Posted by Paolo Bonzini 2 years, 6 months ago
On 18/10/21 11:51, Thomas Huth wrote:
>> * CTRL+C will only interrupt the longest running test.  Pressing
>>    CTRL+C repeatedly three times (which you would likely do anyway,
>>    that's how things work) interrupts the whole run
> 
> I tried this, and while hitting CTRL-C multiple times brought me back to 
> the shell prompt, the remaining tests kept getting started in the 
> background instead of getting stopped ... something is still fishy here, 
> I think.

Ok, I checked that out.  Looks like CTRL+C magic and "make -j" are 
incompatible. :/  So this will have to wait a bit more, but in the 
meanwhile people can already use "meson test" if they want.

>> * Right now "make check-block" only does a single test run just like
>>    "../tests/check-block.sh", but it would be possible to add the 
>> thorough
>>    suite to "meson test --suite block" as well.
> 
> The output of the iotests is also not optimal yet... when running "make 
> check SPEED=slow", the iotests are run multiple times with different 
> target image types, but each run prints the same "▶ 1/1 test 001   OK" 
> etc. to the console, so it's hard to say which target type  is currently 
> exercised. Would it be possible to include the target image type here, 
> e.g. something like:

Yes, that's trivial:

diff --git a/tests/qemu-iotests/testrunner.py 
b/tests/qemu-iotests/testrunner.py
index 3ef14af1fa..45debc1928 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -163,11 +163,11 @@ def test_print_one_line(self, test: str, 
starttime: str,

          if self.tap:
              if status == 'pass':
-                print(f'ok test {test}')
+                print(f'ok {self.env.imgfmt} {test}')
              elif status == 'fail':
-                print(f'not ok test {test}')
+                print(f'not ok {self.env.imgfmt} {test}')
              elif status == 'not run':
-                print(f'ok test {test} # SKIP')
+                print(f'ok {self.env.imgfmt} {test} # SKIP')
              return

          if lasttime:

In fact, that's exactly what was printed in the non-TAP case.  Thanks 
for the feedback, even though it was bad! :)

Paolo