From nobody Thu May 2 02:38:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1543513670254358.9203360966461; Thu, 29 Nov 2018 09:47:50 -0800 (PST) Received: from localhost ([::1]:55542 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gSQPk-0006gt-Fc for importer@patchew.org; Thu, 29 Nov 2018 12:47:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56855) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gSQNg-0005fT-D3 for qemu-devel@nongnu.org; Thu, 29 Nov 2018 12:45:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gSQNc-0004S5-Bf for qemu-devel@nongnu.org; Thu, 29 Nov 2018 12:45:40 -0500 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:34658) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gSQNc-0004R8-22 for qemu-devel@nongnu.org; Thu, 29 Nov 2018 12:45:36 -0500 Received: by mail-wr1-x443.google.com with SMTP id j2so2797321wrw.1 for ; Thu, 29 Nov 2018 09:45:35 -0800 (PST) Received: from 640k.lan ([93.56.166.5]) by smtp.gmail.com with ESMTPSA id 185sm3895997wmx.41.2018.11.29.09.45.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Nov 2018 09:45:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=jX10hc4ppAS7/NYYJxBTrVQ8um1Mb3yK+l1j6jcoV0k=; b=VQLA6v5iP+bEY1w4KknCFzRtt0X5AXZOtCiXdzQiedu7Z7o4lOp4LuW8HvEEx5KOPj ZgDymSe5/bKXN29iWTvq4WtIM5voWQ38gNLe1utNMkpwPVRTUoen5bY991HgcOkqEdQx hO0XNuvXGerGE5dxE2IF8f/dlKFoHXU5SqNAiLBJUXgjj6nmEF0Q+NQ79amFqXGoBb7s 6IIhqZDROQBRWAX0E/zeUo8TZBivjJ/UlbE7e0hknSu52mcH7h6nG7myxaXu65duzGIy A9A7JxCNLkECdc24TnrJWyzYxIa6u7W4JqcTJI3Lm+4WMX8SWQSRv1uoG6ZAQFzCV88S fpvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=jX10hc4ppAS7/NYYJxBTrVQ8um1Mb3yK+l1j6jcoV0k=; b=fsRWjRVPzaMJRiQfJTqv5/lGZ6AQ4O/Mf7Y1JAobiZBdaSP2UrGFeH1raKUGf+2B3p 3hPL4qGZoUtyNnPEZphedQbbBTeKF3dSm21gqq+lZjw8/HUtEUtfUJZmqeylcmT0lmrH 3vI9SUvHtqmXGn+3QNbtXXLUCME235vy9kHB7M0jGHyABlv3qCPFXuBuoWxlypMhiM4a IT9SMzY7EG6EwzvleUa/72GnF4LUpvJxtGgOjDQghvAByAC1kmy4l53Q0JX7MVSZEhYl o4h+kNiOiFQ+UlIAp26nhrnRjW7yG0RZT1K86DBuJMRmTEJb+05G/v3XrHOEBMK66Sop xSIw== X-Gm-Message-State: AA+aEWZiZyql0WPF9tghFip+BmpKhvgC/5Riy+5Vis7v8ALvsOyxBUKr Q3Ud3Y34x7sNXvAuS5f67LQXPzH/ X-Google-Smtp-Source: AFSGD/X2K1SGXZIY+CV0c11F9dyif4od8g7EaWiBG+SDNSFmofaLvsA+eYHGvlSxG7Qi3k7oE0KdpA== X-Received: by 2002:adf:ec11:: with SMTP id x17mr2318292wrn.88.1543513534367; Thu, 29 Nov 2018 09:45:34 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Thu, 29 Nov 2018 18:45:30 +0100 Message-Id: <1543513531-1151-2-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1543513531-1151-1-git-send-email-pbonzini@redhat.com> References: <1543513531-1151-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::443 Subject: [Qemu-devel] [PATCH 1/2] test: execute g_test_run when tests are skipped X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Sometimes a test's main() function recognizes that the environment does not support the test, and therefore exits. In this case, we still should run g_test_run() so that a TAP harness will print the test plan ("1..0") and the test will be marked as skipped. Signed-off-by: Paolo Bonzini Reviewed-by: Eric Blake Reviewed-by: Thomas Huth --- tests/cdrom-test.c | 2 +- tests/migration-test.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/cdrom-test.c b/tests/cdrom-test.c index 9b43dc9..14bd981 100644 --- a/tests/cdrom-test.c +++ b/tests/cdrom-test.c @@ -169,7 +169,7 @@ int main(int argc, char **argv) =20 if (exec_genisoimg(genisocheck)) { /* genisoimage not available - so can't run tests */ - return 0; + return g_test_run(); } =20 ret =3D prepare_image(arch, isoimage); diff --git a/tests/migration-test.c b/tests/migration-test.c index 06ca506..8352612 100644 --- a/tests/migration-test.c +++ b/tests/migration-test.c @@ -789,7 +789,7 @@ int main(int argc, char **argv) g_test_init(&argc, &argv, NULL); =20 if (!ufd_version_check()) { - return 0; + return g_test_run(); } =20 /* @@ -800,7 +800,7 @@ int main(int argc, char **argv) if (g_str_equal(qtest_get_arch(), "ppc64") && access("/sys/module/kvm_hv", F_OK)) { g_test_message("Skipping test: kvm_hv not available"); - return 0; + return g_test_run(); } =20 /* @@ -811,11 +811,11 @@ int main(int argc, char **argv) #if defined(HOST_S390X) if (access("/dev/kvm", R_OK | W_OK)) { g_test_message("Skipping test: kvm not available"); - return 0; + return g_test_run(); } #else g_test_message("Skipping test: Need s390x host to work properly"); - return 0; + return g_test_run(); #endif } =20 --=20 1.8.3.1 From nobody Thu May 2 02:38:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1543514002853640.9052337166404; Thu, 29 Nov 2018 09:53:22 -0800 (PST) Received: from localhost ([::1]:55572 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gSQV7-0003U1-Ca for importer@patchew.org; Thu, 29 Nov 2018 12:53:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56877) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gSQNh-0005gB-9x for qemu-devel@nongnu.org; Thu, 29 Nov 2018 12:45:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gSQNe-0004UJ-0F for qemu-devel@nongnu.org; Thu, 29 Nov 2018 12:45:41 -0500 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:46050) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gSQNd-0004SI-E9 for qemu-devel@nongnu.org; Thu, 29 Nov 2018 12:45:37 -0500 Received: by mail-wr1-x443.google.com with SMTP id v6so2754969wrr.12 for ; Thu, 29 Nov 2018 09:45:37 -0800 (PST) Received: from 640k.lan ([93.56.166.5]) by smtp.gmail.com with ESMTPSA id 185sm3895997wmx.41.2018.11.29.09.45.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Nov 2018 09:45:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=32hXvMxLkk9eychGVKbBOmWfMMATmJqX9zpXzm7rPrs=; b=exCBQRfVTk5v8kbP1QEQlohiMWCnzmmVuaGhIQ9t/CuNw60J1k56iB6nZwP5HKyreo 6RVRYLT1HBzwKSUiK2Tcw+5ws33tNY25owf9QgYIAgLYawyP2rBzIMkE9Iu5nG9fH7qe N9lHLG4v6og2pQSAo8EvChDpAvueH9qDdiRXlobo3hckxvsCzbHWM0rGaYtrv3RfrtlQ NEigj/5nvFGjg2M+fJpSk4BWB/cOHFk4DIZNDsuVueyJaz1fwzlRyFxOjct0IhGsJDpj 9Ch3V3AMLvuheNdleh+ULfctLGds5v7YEPHrE9Aju4Z8nEkwB9W75UiDHCO5HwCYNi6s lavA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=32hXvMxLkk9eychGVKbBOmWfMMATmJqX9zpXzm7rPrs=; b=kHbrlxeslaf9h/bj2V4Wwkol5ssrSh+T2Rlx9HElDnIHbRZeNgSKwzD14mhBHeqBVR B4BxEsATHGTfWkJxrzxuu1KFAepyN0F3YRgKfpU/Y0gbnfR15CEfpbduczK0jsHp8aG/ e1/fP7tQekcHkCAS3uF6NZmNQ4/7q3ehgxZtNUi1zbjN5xa4bpRj/IatRFeqdYS6ciEY 0gaW3dJdS3ZEQAxxTVSlCUnLp/lzMrO+vEiBsIPjut0mV/XVby94VTNErPjDcK3ZTtQ9 f7Bf3osbZo7Xw9zIu1y/DHZwbzXi2EPXpJqDx8om4mYvTaHqNTU0dW0SPq5zFLBkDhWL c+Jg== X-Gm-Message-State: AA+aEWZW9KIgi14Fffgmc2Bygc3p1R6qDTC8zJ1S2vnOJPKMlxlOixqv cos7HY2fWqM9P7ss5Fn3C6Qda8ai X-Google-Smtp-Source: AFSGD/Ui5bOyrk40cc38sPw/jJzEkHnKlJkxbJfKGMlXBN8xypbguBpay2UhFwG0+6Bq3a+Bx2279g== X-Received: by 2002:adf:dcd0:: with SMTP id x16mr2176263wrm.143.1543513535517; Thu, 29 Nov 2018 09:45:35 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Thu, 29 Nov 2018 18:45:31 +0100 Message-Id: <1543513531-1151-3-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1543513531-1151-1-git-send-email-pbonzini@redhat.com> References: <1543513531-1151-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::443 Subject: [Qemu-devel] [PATCH 2/2] test: replace gtester with a TAP driver X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" gtester is deprecated by upstream glib and it does not support tests that call g_test_skip in some glib stable releases. glib suggests instead using Automake's TAP support. We do not support Automake, but we can copy the code that beautifies the TAP output and use it. I chose to use the Perl copy rather than the shell/awk one, in order to reuse Perl's TAP parsing package, but I'm open to suggestions about which language to use. Signed-off-by: Paolo Bonzini Reviewed-by: Eric Blake --- scripts/gtester-cat | 26 -- scripts/tap-driver.pl | 386 +++++++++++++++++++++++= ++++ scripts/tap-merge.pl | 108 ++++++++ tests/Makefile.include | 54 ++-- tests/docker/dockerfiles/centos7.docker | 1 + tests/docker/dockerfiles/debian-amd64.docker | 1 + tests/docker/dockerfiles/debian-ports.docker | 1 + tests/docker/dockerfiles/debian-sid.docker | 1 + tests/docker/dockerfiles/debian8.docker | 1 + tests/docker/dockerfiles/debian9.docker | 1 + tests/docker/dockerfiles/fedora.docker | 1 + tests/docker/dockerfiles/ubuntu.docker | 1 + 12 files changed, 535 insertions(+), 47 deletions(-) delete mode 100755 scripts/gtester-cat create mode 100755 scripts/tap-driver.pl create mode 100755 scripts/tap-merge.pl diff --git a/scripts/gtester-cat b/scripts/gtester-cat deleted file mode 100755 index 061a952..0000000 --- a/scripts/gtester-cat +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/sh -# -# Copyright IBM, Corp. 2012 -# -# Authors: -# Anthony Liguori -# -# This work is licensed under the terms of the GNU GPLv2 or later. -# See the COPYING file in the top-level directory. - -cat < - - - qemu - 0.0 - rev - -EOF - -sed \ - -e '/$/d' \ - -e '//,/<\/info>/d' \ - -e '$b' \ - -e '/^<\/gtester>$/d' "$@" diff --git a/scripts/tap-driver.pl b/scripts/tap-driver.pl new file mode 100755 index 0000000..2e674c0 --- /dev/null +++ b/scripts/tap-driver.pl @@ -0,0 +1,386 @@ +#! /usr/bin/env perl +# Copyright (C) 2011-2013 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# As a special exception to the GNU General Public License, if you +# distribute this file as part of a program that contains a +# configuration script generated by Autoconf, you may include it under +# the same distribution terms that you use for the rest of that program. + +# ---------------------------------- # +# Imports, static data, and setup. # +# ---------------------------------- # + +use warnings FATAL =3D> 'all'; +use strict; +use Getopt::Long (); +use TAP::Parser; +use Term::ANSIColor qw(:constants); + +my $VERSION =3D '2012-02-01.19'; # UTC + +my $ME =3D "tap-driver.pl"; + +my $USAGE =3D <<'END'; +Usage: + tap-driver [--color=3D{always|never|auto}] + [--verbose] +END + +my $HELP =3D "$ME: TAP-aware test driver for QEMU testsuite harness." . + "\n" . $USAGE; + +# It's important that NO_PLAN evaluates "false" as a boolean. +use constant NO_PLAN =3D> 0; +use constant EARLY_PLAN =3D> 1; +use constant LATE_PLAN =3D> 2; + +# ------------------- # +# Global variables. # +# ------------------- # + +my $testno =3D 0; # Number of test results seen so far. +my $bailed_out =3D 0; # Whether a "Bail out!" directive has been seen. + +# Whether the TAP plan has been seen or not, and if yes, which kind +# it is ("early" is seen before any test result, "late" otherwise). +my $plan_seen =3D NO_PLAN; + +# ----------------- # +# Option parsing. # +# ----------------- # + +my %cfg =3D ( + "color" =3D> 0, + "verbose" =3D> 0, +); + +my $diag_string =3D "#"; +my $color =3D "auto"; + +# Perl's Getopt::Long allows options to take optional arguments after a sp= ace. +# Prevent --color by itself from consuming other arguments +foreach (@ARGV) { + if ($_ eq "--color" || $_ eq "-color") { + $_ =3D "--color=3D$color"; + } +} + +Getopt::Long::GetOptions + ( + 'help' =3D> sub { print $HELP; exit 0; }, + 'version' =3D> sub { print "$ME $VERSION\n"; exit 0; }, + 'color=3Ds' =3D> \$color, + 'verbose' =3D> sub { $cfg{"verbose"} =3D 1; }, + ) or exit 1; + +if ($color =3D~ /^always$/i) { + $cfg{'color'} =3D 1; +} elsif ($color =3D~ /^never$/i) { + $cfg{'color'} =3D 0; +} elsif ($color =3D~ /^auto$/i) { + $cfg{'color'} =3D (-t STDOUT); +} else { + die "Invalid color mode: $color\n"; +} + +# ------------- # +# Prototypes. # +# ------------- # + +sub bool_opt ($$); +sub colored ($$); +sub decorate_result ($); +sub extract_tap_comment ($); +sub handle_tap_bailout ($); +sub handle_tap_plan ($); +sub handle_tap_result ($); +sub is_null_string ($); +sub main (@); +sub report ($;$); +sub stringify_result_obj ($); +sub testsuite_error ($); + +# -------------- # +# Subroutines. # +# -------------- # + +sub bool_opt ($$) +{ + my ($opt, $val) =3D @_; + if ($val =3D~ /^(?:y|yes)\z/i) + { + $cfg{$opt} =3D 1; + } + elsif ($val =3D~ /^(?:n|no)\z/i) + { + $cfg{$opt} =3D 0; + } + else + { + die "$ME: invalid argument '$val' for option '$opt'\n"; + } +} + +# If the given string is undefined or empty, return true, otherwise +# return false. This function is useful to avoid pitfalls like: +# if ($message) { print "$message\n"; } +# which wouldn't print anything if $message is the literal "0". +sub is_null_string ($) +{ + my $str =3D shift; + return ! (defined $str and length $str); +} + +sub stringify_result_obj ($) +{ + my $result_obj =3D shift; + if ($result_obj->is_unplanned || $result_obj->number !=3D $testno) + { + return "ERROR"; + } + elsif ($plan_seen =3D=3D LATE_PLAN) + { + return "ERROR"; + } + elsif (!$result_obj->directive) + { + return $result_obj->is_ok ? "PASS" : "FAIL"; + } + elsif ($result_obj->has_todo) + { + return $result_obj->is_actual_ok ? "XPASS" : "XFAIL"; + } + elsif ($result_obj->has_skip) + { + return $result_obj->is_ok ? "SKIP" : "FAIL"; + } + die "$ME: INTERNAL ERROR"; # NOTREACHED +} + +sub colored ($$) +{ + my ($color_string, $text) =3D @_; + return $color_string . $text . RESET; +} + +sub decorate_result ($) +{ + my $result =3D shift; + return $result unless $cfg{"color"}; + my %color_for_result =3D + ( + "ERROR" =3D> MAGENTA, + "PASS" =3D> GREEN, + "XPASS" =3D> RED, + "FAIL" =3D> BOLD.RED, + "XFAIL" =3D> BOLD.GREEN, + "SKIP" =3D> BLUE, + ); + if (my $color =3D $color_for_result{$result}) + { + return colored ($color, $result); + } + else + { + return $result; # Don't colorize unknown stuff. + } +} + +sub report ($;$) +{ + my ($msg, $result, $explanation) =3D (undef, @_); + if ($result =3D~ /^(?:X?(?:PASS|FAIL)|SKIP|ERROR)/) + { + # Output on console might be colorized. + $msg =3D decorate_result($result); + } + elsif ($result eq "#") + { + $msg =3D " "; + } + else + { + die "$ME: INTERNAL ERROR"; # NOTREACHED + } + $msg .=3D " $explanation" if defined $explanation; + print $msg . "\n"; +} + +sub testsuite_error ($) +{ + report "ERROR", "- $_[0]"; +} + +sub handle_tap_result ($) +{ + $testno++; + my $result_obj =3D shift; + + my $test_result =3D stringify_result_obj $result_obj; + my $string =3D $result_obj->number; + + my $description =3D $result_obj->description; + $string .=3D " $description" + unless is_null_string $description; + + if ($plan_seen =3D=3D LATE_PLAN) + { + $string .=3D " # AFTER LATE PLAN"; + } + elsif ($result_obj->is_unplanned) + { + $string .=3D " # UNPLANNED"; + } + elsif ($result_obj->number !=3D $testno) + { + $string .=3D " # OUT-OF-ORDER (expecting $testno)"; + } + elsif (my $directive =3D $result_obj->directive) + { + $string .=3D " # $directive"; + my $explanation =3D $result_obj->explanation; + $string .=3D " $explanation" + unless is_null_string $explanation; + } + + report $test_result, $string; +} + +sub handle_tap_plan ($) +{ + my $plan =3D shift; + if ($plan_seen) + { + # Error, only one plan per stream is acceptable. + testsuite_error "multiple test plans"; + return; + } + # The TAP plan can come before or after *all* the TAP results; we speak + # respectively of an "early" or a "late" plan. If we see the plan line + # after at least one TAP result has been seen, assume we have a late + # plan; in this case, any further test result seen after the plan will + # be flagged as an error. + $plan_seen =3D ($testno >=3D 1 ? LATE_PLAN : EARLY_PLAN); + # If $testno > 0, we have an error ("too many tests run") that will be + # automatically dealt with later, so don't worry about it here. If + # $plan_seen is true, we have an error due to a repeated plan, and that + # has already been dealt with above. Otherwise, we have a valid "plan + # with SKIP" specification, and should report it as a particular kind + # of SKIP result. + if ($plan->directive && $testno =3D=3D 0) + { + my $explanation =3D is_null_string ($plan->explanation) ? + undef : "- " . $plan->explanation; + report "SKIP", $explanation; + } +} + +sub handle_tap_bailout ($) +{ + my ($bailout, $msg) =3D ($_[0], "Bail out!"); + $bailed_out =3D 1; + $msg .=3D " " . $bailout->explanation + unless is_null_string $bailout->explanation; + testsuite_error $msg; +} + +sub extract_tap_comment ($) +{ + my $line =3D shift; + if (index ($line, $diag_string) =3D=3D 0) + { + # Strip leading `$diag_string' from `$line'. + $line =3D substr ($line, length ($diag_string)); + # And strip any leading and trailing whitespace left. + $line =3D~ s/(?:^\s*|\s*$)//g; + # Return what is left (if any). + return $line; + } + return ""; +} + +sub main (@) +{ + my $iterator =3D TAP::Parser::Iterator::Stream->new(\*STDIN); + my $parser =3D TAP::Parser->new ({iterator =3D> $iterator }); + + while (defined (my $cur =3D $parser->next)) + { + # Parsing of TAP input should stop after a "Bail out!" directive. + next if $bailed_out; + + if ($cur->is_plan) + { + handle_tap_plan ($cur); + } + elsif ($cur->is_test) + { + handle_tap_result ($cur); + } + elsif ($cur->is_bailout) + { + handle_tap_bailout ($cur); + } + elsif ($cfg{"verbose"}) + { + my $comment =3D extract_tap_comment ($cur->raw); + report "#", "$comment" if length $comment; + } + } + # A "Bail out!" directive should cause us to ignore any following TAP + # error, as well as a non-zero exit status from the TAP producer. + if (!$bailed_out) + { + if (!$plan_seen) + { + testsuite_error "missing test plan"; + } + elsif ($parser->tests_planned !=3D $parser->tests_run) + { + my ($planned, $run) =3D ($parser->tests_planned, $parser->tests_= run); + my $bad_amount =3D $run > $planned ? "many" : "few"; + testsuite_error (sprintf "too %s tests run (expected %d, got %d)= ", + $bad_amount, $planned, $run); + } + } +} + +# ----------- # +# Main code. # +# ----------- # + +main @ARGV; + +# Local Variables: +# perl-indent-level: 2 +# perl-continued-statement-offset: 2 +# perl-continued-brace-offset: 0 +# perl-brace-offset: 0 +# perl-brace-imaginary-offset: 0 +# perl-label-offset: -2 +# cperl-indent-level: 2 +# cperl-brace-offset: 0 +# cperl-continued-brace-offset: 0 +# cperl-label-offset: -2 +# cperl-extra-newline-before-brace: t +# cperl-merge-trailing-else: nil +# cperl-continued-statement-offset: 2 +# eval: (add-hook 'write-file-hooks 'time-stamp) +# time-stamp-start: "my $VERSION =3D " +# time-stamp-format: "'%:y-%02m-%02d.%02H'" +# time-stamp-time-zone: "UTC" +# time-stamp-end: "; # UTC" +# End: diff --git a/scripts/tap-merge.pl b/scripts/tap-merge.pl new file mode 100755 index 0000000..7e0de90 --- /dev/null +++ b/scripts/tap-merge.pl @@ -0,0 +1,108 @@ +#! /usr/bin/env perl +# Copyright (C) 2018 Red Hat, Inc. +# +# Author: Paolo Bonzini +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# ---------------------------------- # +# Imports, static data, and setup. # +# ---------------------------------- # + +use warnings FATAL =3D> 'all'; +use strict; +use Getopt::Long (); +use TAP::Parser; + +my $ME =3D "tap-merge.pl"; +my $VERSION =3D "2018-11-29"; + +my $HELP =3D "$ME: merge multiple TAP inputs from stdin."; + +# ----------------- # +# Option parsing. # +# ----------------- # + +Getopt::Long::GetOptions + ( + 'help' =3D> sub { print $HELP; exit 0; }, + 'version' =3D> sub { print "$ME $VERSION\n"; exit 0; }, + ); + +# -------------- # +# Subroutines. # +# -------------- # + +sub main () +{ + my $iterator =3D TAP::Parser::Iterator::Stream->new(\*STDIN); + my $parser =3D TAP::Parser->new ({iterator =3D> $iterator }); + my $testno =3D 0; # Number of test results seen so far. + my $bailed_out =3D 0; # Whether a "Bail out!" directive has been seen. + + while (defined (my $cur =3D $parser->next)) + { + if ($cur->is_bailout) + { + $bailed_out =3D 1; + next; + } + if ($cur->is_plan) + { + $bailed_out =3D 0; + next; + } + if ($cur->is_test) + { + $bailed_out =3D 0 if $cur->number =3D=3D 1; + $testno++; + $cur =3D TAP::Parser::Result::Test->new({ + ok =3D> $cur->ok, + test_num =3D> $testno, + directive =3D> $cur->directive, + explanation =3D> $cur->explanation, + description =3D> $cur->description + }); + } + print $cur->as_string . "\n" unless $bailed_out; + } + print "1..$testno\n"; +} + +# ----------- # +# Main code. # +# ----------- # + +main; + +# Local Variables: +# perl-indent-level: 2 +# perl-continued-statement-offset: 2 +# perl-continued-brace-offset: 0 +# perl-brace-offset: 0 +# perl-brace-imaginary-offset: 0 +# perl-label-offset: -2 +# cperl-indent-level: 2 +# cperl-brace-offset: 0 +# cperl-continued-brace-offset: 0 +# cperl-label-offset: -2 +# cperl-extra-newline-before-brace: t +# cperl-merge-trailing-else: nil +# cperl-continued-statement-offset: 2 +# eval: (add-hook 'write-file-hooks 'time-stamp) +# time-stamp-start: "my $VERSION =3D " +# time-stamp-format: "'%:y-%02m-%02d.%02H'" +# time-stamp-time-zone: "UTC" +# time-stamp-end: "; # UTC" +# End: diff --git a/tests/Makefile.include b/tests/Makefile.include index fb0b449..d27e137 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -799,41 +799,53 @@ tests/test-qga$(EXESUF): qemu-ga$(EXESUF) tests/test-qga$(EXESUF): tests/test-qga.o $(qtest-obj-y) =20 SPEED =3D quick -GTESTER_OPTIONS =3D -k $(if $(V),--verbose,-q) -GCOV_OPTIONS =3D -n $(if $(V),-f,) =20 # gtester tests, possibly with verbose output =20 +do_test_human =3D \ + $(call quiet-command, \ + { $(foreach COMMAND, $1, \ + $(if $(V), echo $(notdir $(COMMAND)):;) \ + MALLOC_PERTURB_=3D$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))}= \ + $2 $(COMMAND) -m=3D$(SPEED) -k $(if $(V), --tap | ./scripts/tap-dri= ver.pl, \ + >/dev/null );) }, \ + "TEST","$@") + +do_test_tap =3D \ + $(call quiet-command, \ + { $(foreach COMMAND, $1, \ + MALLOC_PERTURB_=3D$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))}= \ + $2 $(COMMAND) -m=3D$(SPEED) -k --tap | sed "s/^[a-z][a-z]* [0-9]* /= &$(notdir $(COMMAND)) /" ;) } \ + | $(SRC_PATH)/scripts/tap-merge.pl \ + $(if $(V), | tee "$@" | ./scripts/tap-driver.pl, >"$@"), \ + "TAP","$@") + .PHONY: $(patsubst %, check-qtest-%, $(QTEST_TARGETS)) $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: subdir-%-so= ftmmu $(check-qtest-y) - $(call quiet-command,QTEST_QEMU_BINARY=3D$*-softmmu/qemu-system-$* \ - QTEST_QEMU_IMG=3Dqemu-img$(EXESUF) \ - MALLOC_PERTURB_=3D$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \ - gtester $(GTESTER_OPTIONS) -m=3D$(SPEED) $(check-qtest-$*-y) $(check-qte= st-generic-y),"GTESTER","$@") + $(call do_test_human,$(check-qtest-$*-y) $(check-qtest-generic-y), \ + QTEST_QEMU_BINARY=3D$*-softmmu/qemu-system-$* \ + QTEST_QEMU_IMG=3Dqemu-img$(EXESUF)) =20 .PHONY: $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)) $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: % - $(call quiet-command, \ - MALLOC_PERTURB_=3D$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \ - gtester $(GTESTER_OPTIONS) -m=3D$(SPEED) $*,"GTESTER","$*") + $(call do_test_human, $*) =20 -# gtester tests with XML output +# gtester tests with TAP output =20 -$(patsubst %, check-report-qtest-%.xml, $(QTEST_TARGETS)): check-report-qt= est-%.xml: $(check-qtest-y) - $(call quiet-command,QTEST_QEMU_BINARY=3D$*-softmmu/qemu-system-$* \ - QTEST_QEMU_IMG=3Dqemu-img$(EXESUF) \ - gtester -q $(GTESTER_OPTIONS) -o $@ -m=3D$(SPEED) $(check-qtest-$*-y) $= (check-qtest-generic-y),"GTESTER","$@") +$(patsubst %, check-report-qtest-%.tap, $(QTEST_TARGETS)): check-report-qt= est-%.tap: $(check-qtest-y) + $(call do_test_tap, $(check-qtest-$*-y) $(check-qtest-generic-y), \ + QTEST_QEMU_BINARY=3D$*-softmmu/qemu-system-$* \ + QTEST_QEMU_IMG=3Dqemu-img$(EXESUF)) =20 -check-report-unit.xml: $(check-unit-y) - $(call quiet-command,gtester -q $(GTESTER_OPTIONS) -o $@ -m=3D$(SPEED) $^= ,"GTESTER","$@") +check-report-unit.tap: $(check-unit-y) + $(call do_test_tap,$^, \ + QTEST_QEMU_BINARY=3D$*-softmmu/qemu-system-$* \ + QTEST_QEMU_IMG=3Dqemu-img$(EXESUF)) =20 # Reports and overall runs =20 -check-report.xml: $(patsubst %,check-report-qtest-%.xml, $(QTEST_TARGETS))= check-report-unit.xml - $(call quiet-command,$(SRC_PATH)/scripts/gtester-cat $^ > $@,"GEN","$@") - -check-report.html: check-report.xml - $(call quiet-command,gtester-report $< > $@,"GEN","$@") +check-report.tap: $(patsubst %,check-report-qtest-%.tap, $(QTEST_TARGETS))= check-report-unit.tap + $(call quiet-command,$(SRC_PATH)/scripts/tap-merge.py $^ > $@,"GEN","$@") =20 # Per guest TCG tests =20 diff --git a/tests/docker/dockerfiles/centos7.docker b/tests/docker/dockerf= iles/centos7.docker index 0a04bfb..e0f18f5 100644 --- a/tests/docker/dockerfiles/centos7.docker +++ b/tests/docker/dockerfiles/centos7.docker @@ -22,6 +22,7 @@ ENV PACKAGES \ mesa-libEGL-devel \ mesa-libgbm-devel \ nettle-devel \ + perl-Test-Harness \ pixman-devel \ SDL-devel \ spice-glib-devel \ diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/do= ckerfiles/debian-amd64.docker index 24b113b..c66e341 100644 --- a/tests/docker/dockerfiles/debian-amd64.docker +++ b/tests/docker/dockerfiles/debian-amd64.docker @@ -16,6 +16,7 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ liblzo2-dev \ librdmacm-dev \ libsnappy-dev \ + libtest-harness-perl \ libvte-dev =20 # virgl diff --git a/tests/docker/dockerfiles/debian-ports.docker b/tests/docker/do= ckerfiles/debian-ports.docker index e05a9a9..514ab53 100644 --- a/tests/docker/dockerfiles/debian-ports.docker +++ b/tests/docker/dockerfiles/debian-ports.docker @@ -29,6 +29,7 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ flex \ gettext \ git \ + libtest-harness-perl \ pkg-config \ psmisc \ python \ diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dock= erfiles/debian-sid.docker index 9a3d168..b30cbe7 100644 --- a/tests/docker/dockerfiles/debian-sid.docker +++ b/tests/docker/dockerfiles/debian-sid.docker @@ -26,6 +26,7 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ ca-certificates \ flex \ git \ + libtest-harness-perl \ pkg-config \ psmisc \ python \ diff --git a/tests/docker/dockerfiles/debian8.docker b/tests/docker/dockerf= iles/debian8.docker index 5294563..cdc3f11 100644 --- a/tests/docker/dockerfiles/debian8.docker +++ b/tests/docker/dockerfiles/debian8.docker @@ -29,6 +29,7 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ gettext \ git \ gnupg \ + libtest-harness-perl \ pkg-config \ python-minimal =20 diff --git a/tests/docker/dockerfiles/debian9.docker b/tests/docker/dockerf= iles/debian9.docker index 154ae2a..9561d4f 100644 --- a/tests/docker/dockerfiles/debian9.docker +++ b/tests/docker/dockerfiles/debian9.docker @@ -24,6 +24,7 @@ RUN DEBIAN_FRONTEND=3Dnoninteractive eatmydata \ flex \ gettext \ git \ + libtest-harness-perl \ pkg-config \ psmisc \ python \ diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfi= les/fedora.docker index 0c4eb9e..1d0e3dc 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -70,6 +70,7 @@ ENV PACKAGES \ nss-devel \ numactl-devel \ perl \ + perl-Test-Harness \ pixman-devel \ python3 \ PyYAML \ diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfi= les/ubuntu.docker index 36e2b17..229add5 100644 --- a/tests/docker/dockerfiles/ubuntu.docker +++ b/tests/docker/dockerfiles/ubuntu.docker @@ -45,6 +45,7 @@ ENV PACKAGES flex bison \ libspice-protocol-dev \ libspice-server-dev \ libssh2-1-dev \ + libtest-harness-perl \ libusb-1.0-0-dev \ libusbredirhost-dev \ libvdeplug-dev \ --=20 1.8.3.1