scripts/gtester-cat | 26 -- scripts/tap-driver.pl | 386 +++++++++++++++++++++++++++ scripts/tap-merge.pl | 108 ++++++++ tests/Makefile.include | 54 ++-- tests/cdrom-test.c | 2 +- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/debian-amd64.docker | 1 + tests/docker/dockerfiles/debian-ports.docker | 1 + tests/docker/dockerfiles/debian-sid.docker | 1 + tests/docker/dockerfiles/debian8.docker | 1 + tests/docker/dockerfiles/debian9.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/ubuntu.docker | 1 + tests/migration-test.c | 8 +- 14 files changed, 540 insertions(+), 52 deletions(-) delete mode 100755 scripts/gtester-cat create mode 100755 scripts/tap-driver.pl create mode 100755 scripts/tap-merge.pl
gtester is deprecated by upstream glib and it does not support tests that call g_test_skip in some glib stable releases. glib suggests instead using Automake's TAP support. We do not support Automake, but we can copy the code that beautifies the TAP output and use it. I chose to use the Perl copy rather than the shell/awk one, in order to reuse Perl's TAP parsing package, but I'm open to suggestions about which language to use. Thanks, Paolo Paolo Bonzini (2): test: execute g_test_run when tests are skipped test: replace gtester with a TAP driver scripts/gtester-cat | 26 -- scripts/tap-driver.pl | 386 +++++++++++++++++++++++++++ scripts/tap-merge.pl | 108 ++++++++ tests/Makefile.include | 54 ++-- tests/cdrom-test.c | 2 +- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/debian-amd64.docker | 1 + tests/docker/dockerfiles/debian-ports.docker | 1 + tests/docker/dockerfiles/debian-sid.docker | 1 + tests/docker/dockerfiles/debian8.docker | 1 + tests/docker/dockerfiles/debian9.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/ubuntu.docker | 1 + tests/migration-test.c | 8 +- 14 files changed, 540 insertions(+), 52 deletions(-) delete mode 100755 scripts/gtester-cat create mode 100755 scripts/tap-driver.pl create mode 100755 scripts/tap-merge.pl -- 1.8.3.1
On 11/29/18 11:45 AM, Paolo Bonzini wrote: > gtester is deprecated by upstream glib and it does not support tests > that call g_test_skip in some glib stable releases. > > glib suggests instead using Automake's TAP support. We do not support > Automake, but we can copy the code that beautifies the TAP output and > use it. I chose to use the Perl copy rather than the shell/awk one, > in order to reuse Perl's TAP parsing package, but I'm open to suggestions > about which language to use. I'm less familiar with the TAP protocol than I'd like to admit, but I did find: https://blog.gtk.org/2018/07/11/news-from-glib-2-58/ which corroborates your claim that switching to Automake's TAP parser is indeed a recommended approach to avoid the now-deprecated gtester. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 2018-11-29 21:43, Eric Blake wrote: > On 11/29/18 11:45 AM, Paolo Bonzini wrote: >> gtester is deprecated by upstream glib and it does not support tests >> that call g_test_skip in some glib stable releases. >> >> glib suggests instead using Automake's TAP support. We do not support >> Automake, but we can copy the code that beautifies the TAP output and >> use it. I chose to use the Perl copy rather than the shell/awk one, >> in order to reuse Perl's TAP parsing package, but I'm open to suggestions >> about which language to use. > > I'm less familiar with the TAP protocol than I'd like to admit, but I > did find: > > https://blog.gtk.org/2018/07/11/news-from-glib-2-58/ > > which corroborates your claim that switching to Automake's TAP parser is > indeed a recommended approach to avoid the now-deprecated gtester. Hmm, do we really have to maintain our own version of a test runner now? There's already some effort going on to use avocado in "make check-acceptance" ... maybe we could use avocado as replacement for gtester, too? Thomas
On 30/11/18 08:21, Thomas Huth wrote: > On 2018-11-29 21:43, Eric Blake wrote: >> On 11/29/18 11:45 AM, Paolo Bonzini wrote: >>> gtester is deprecated by upstream glib and it does not support tests >>> that call g_test_skip in some glib stable releases. >>> >>> glib suggests instead using Automake's TAP support. We do not support >>> Automake, but we can copy the code that beautifies the TAP output and >>> use it. I chose to use the Perl copy rather than the shell/awk one, >>> in order to reuse Perl's TAP parsing package, but I'm open to suggestions >>> about which language to use. >> >> I'm less familiar with the TAP protocol than I'd like to admit, but I >> did find: >> >> https://blog.gtk.org/2018/07/11/news-from-glib-2-58/ >> >> which corroborates your claim that switching to Automake's TAP parser is >> indeed a recommended approach to avoid the now-deprecated gtester. > > Hmm, do we really have to maintain our own version of a test runner now? Well, that's what you get for using a custom build system but it's "just" an output beautifier really. I could just "grep -ve ^# -e ^ok" but it wouldn't look very nice (plus things that start as 1-line shell scripts end up being the same as the thing you're trying to replace). I don't think it will be a big maintenance headache---just like most other files in scripts/, some of which have had exactly zero commits since they were added. In fact, Automake has hardly changed its driver since the first commit. > There's already some effort going on to use avocado in "make > check-acceptance" ... maybe we could use avocado as replacement for > gtester, too? Avocado does not have the required functionality, unfortunately. It can produce TAP, but not consume it. Also, Python libraries for TAP do not abound and are generally not shipped with the OS (apart from Perl, which basically invented TAP, the language with the most TAP libraries seem to be JavaScript!). Paolo
On 11/30/18 4:58 AM, Paolo Bonzini wrote: > On 30/11/18 08:21, Thomas Huth wrote: >> On 2018-11-29 21:43, Eric Blake wrote: >>> On 11/29/18 11:45 AM, Paolo Bonzini wrote: >>>> gtester is deprecated by upstream glib and it does not support tests >>>> that call g_test_skip in some glib stable releases. >>>> >>>> glib suggests instead using Automake's TAP support. We do not support >>>> Automake, but we can copy the code that beautifies the TAP output and >>>> use it. I chose to use the Perl copy rather than the shell/awk one, >>>> in order to reuse Perl's TAP parsing package, but I'm open to suggestions >>>> about which language to use. >>> >>> I'm less familiar with the TAP protocol than I'd like to admit, but I >>> did find: >>> >>> https://blog.gtk.org/2018/07/11/news-from-glib-2-58/ >>> >>> which corroborates your claim that switching to Automake's TAP parser is >>> indeed a recommended approach to avoid the now-deprecated gtester. >> >> Hmm, do we really have to maintain our own version of a test runner now? > > Well, that's what you get for using a custom build system but it's > "just" an output beautifier really. I could just "grep -ve ^# -e ^ok" > but it wouldn't look very nice (plus things that start as 1-line shell > scripts end up being the same as the thing you're trying to replace). > > I don't think it will be a big maintenance headache---just like most > other files in scripts/, some of which have had exactly zero commits > since they were added. In fact, Automake has hardly changed its driver > since the first commit. > >> There's already some effort going on to use avocado in "make >> check-acceptance" ... maybe we could use avocado as replacement for >> gtester, too? > > Avocado does not have the required functionality, unfortunately. It can > produce TAP, but not consume it. Also, Python libraries for TAP do not > abound and are generally not shipped with the OS (apart from Perl, which > basically invented TAP, the language with the most TAP libraries seem to > be JavaScript!). > Avocado can't currently consume TAP (it produces it, though), but it has some integration with glib based tests: https://avocado-framework.readthedocs.io/en/66.0/optional_plugins/glib.html But, there may be bits missing to fully replae gtests. Given that we have been adding features to Avocado based on QEMU requirements, we can certainly look at adding missing pieces there. If this sounds like something you'd be interested in, I can come up with an integration proposal. > Paolo > Cheers, - Cleber.
On 2018-11-30 15:47, Cleber Rosa wrote: > > > On 11/30/18 4:58 AM, Paolo Bonzini wrote: >> On 30/11/18 08:21, Thomas Huth wrote: >>> On 2018-11-29 21:43, Eric Blake wrote: >>>> On 11/29/18 11:45 AM, Paolo Bonzini wrote: >>>>> gtester is deprecated by upstream glib and it does not support tests >>>>> that call g_test_skip in some glib stable releases. >>>>> >>>>> glib suggests instead using Automake's TAP support. We do not support >>>>> Automake, but we can copy the code that beautifies the TAP output and >>>>> use it. I chose to use the Perl copy rather than the shell/awk one, >>>>> in order to reuse Perl's TAP parsing package, but I'm open to suggestions >>>>> about which language to use. >>>> >>>> I'm less familiar with the TAP protocol than I'd like to admit, but I >>>> did find: >>>> >>>> https://blog.gtk.org/2018/07/11/news-from-glib-2-58/ >>>> >>>> which corroborates your claim that switching to Automake's TAP parser is >>>> indeed a recommended approach to avoid the now-deprecated gtester. >>> >>> Hmm, do we really have to maintain our own version of a test runner now? >> >> Well, that's what you get for using a custom build system but it's >> "just" an output beautifier really. I could just "grep -ve ^# -e ^ok" >> but it wouldn't look very nice (plus things that start as 1-line shell >> scripts end up being the same as the thing you're trying to replace). >> >> I don't think it will be a big maintenance headache---just like most >> other files in scripts/, some of which have had exactly zero commits >> since they were added. In fact, Automake has hardly changed its driver >> since the first commit. OK, fair. >>> There's already some effort going on to use avocado in "make >>> check-acceptance" ... maybe we could use avocado as replacement for >>> gtester, too? >> >> Avocado does not have the required functionality, unfortunately. It can >> produce TAP, but not consume it. Also, Python libraries for TAP do not >> abound and are generally not shipped with the OS (apart from Perl, which >> basically invented TAP, the language with the most TAP libraries seem to >> be JavaScript!). > > Avocado can't currently consume TAP (it produces it, though), but it has > some integration with glib based tests: > > https://avocado-framework.readthedocs.io/en/66.0/optional_plugins/glib.html > > But, there may be bits missing to fully replae gtests. Given that we > have been adding features to Avocado based on QEMU requirements, we can > certainly look at adding missing pieces there. If this sounds like > something you'd be interested in, I can come up with an integration > proposal. FWIW, it seems basically to work, indeed: $ export QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 $ avocado run --loaders glib -- tests/pxe-test tests/cdrom-test JOB ID : 67657631c2f1f80e34d90fe0c2c05e05aab6f1ba JOB LOG : /home/thuth/avocado/job-results/job-2018-11-30T10.02-6765763/job.log (01/12) tests/pxe-test:/x86_64/pxe/ipv4/pc/e1000: PASS (6.46 s) (02/12) tests/pxe-test:/x86_64/pxe/ipv4/pc/virtio-net-pci: PASS (6.47 s) (03/12) tests/pxe-test:/x86_64/pxe/ipv4/q35/e1000e: PASS (6.46 s) (04/12) tests/pxe-test:/x86_64/pxe/ipv4/q35/virtio-net-pci: PASS (6.46 s) (05/12) tests/cdrom-test:/x86_64/cdrom/boot/default: PASS (0.25 s) (06/12) tests/cdrom-test:/x86_64/cdrom/boot/virtio-scsi: PASS (0.25 s) (07/12) tests/cdrom-test:/x86_64/cdrom/boot/isapc: PASS (0.24 s) (08/12) tests/cdrom-test:/x86_64/cdrom/boot/am53c974: PASS (0.25 s) (09/12) tests/cdrom-test:/x86_64/cdrom/boot/dc390: PASS (0.25 s) (10/12) tests/cdrom-test:/x86_64/cdrom/boot/lsi53c895a: PASS (0.26 s) (11/12) tests/cdrom-test:/x86_64/cdrom/boot/megasas: PASS (0.15 s) (12/12) tests/cdrom-test:/x86_64/cdrom/boot/megasas-gen2: PASS (0.15 s) RESULTS : PASS 12 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 28.31 s JOB HTML : /home/thuth/avocado/job-results/job-2018-11-30T10.02-6765763/results.html Thomas
On Thu, Nov 29, 2018 at 02:43:03PM -0600, Eric Blake wrote: > On 11/29/18 11:45 AM, Paolo Bonzini wrote: > > gtester is deprecated by upstream glib and it does not support tests > > that call g_test_skip in some glib stable releases. > > > > glib suggests instead using Automake's TAP support. We do not support > > Automake, but we can copy the code that beautifies the TAP output and > > use it. I chose to use the Perl copy rather than the shell/awk one, > > in order to reuse Perl's TAP parsing package, but I'm open to suggestions > > about which language to use. > > I'm less familiar with the TAP protocol than I'd like to admit, but I did > find: > > https://blog.gtk.org/2018/07/11/news-from-glib-2-58/ > > which corroborates your claim that switching to Automake's TAP parser is > indeed a recommended approach to avoid the now-deprecated gtester. FYI, for docs about the TAP protocol see: https://testanything.org/tap-specification.html I've always liked the TAP protocol approach to test output where you print an ok/not ok status for each piece of the test, as opposed to assertion style tests where you just abort at the first piece that fails. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 30/11/18 10:54, Daniel P. Berrangé wrote: > FYI, for docs about the TAP protocol see: > > https://testanything.org/tap-specification.html > > I've always liked the TAP protocol approach to test output where you print > an ok/not ok status for each piece of the test, as opposed to assertion > style tests where you just abort at the first piece that fails. FWIW, this is not what this series does. Each ok/not ok line corresponds to one assertion-style test. Paolo
On Fri, Nov 30, 2018 at 10:56:01AM +0100, Paolo Bonzini wrote: > On 30/11/18 10:54, Daniel P. Berrangé wrote: > > FYI, for docs about the TAP protocol see: > > > > https://testanything.org/tap-specification.html > > > > I've always liked the TAP protocol approach to test output where you print > > an ok/not ok status for each piece of the test, as opposed to assertion > > style tests where you just abort at the first piece that fails. > > FWIW, this is not what this series does. Each ok/not ok line > corresponds to one assertion-style test. Oh sure, that's the way they've integrated it into the glib test APIs. I just meant that in terms of the output format, I like its flexibility in this manner. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
© 2016 - 2024 Red Hat, Inc.