From nobody Tue Feb 10 05:39:55 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1598006061; cv=none; d=zohomail.com; s=zohoarc; b=GmT5+WK9YtX7m94BE2PqPZaONEVrpJ6NqsUHRzKkCpZaUpbwI23Ou4C9hNjOYA2O1iSgvlh1Ajyb6CThsy4j6f0w9aKEtfg5bO3FZHxp+oqD7SQ24+GpRUxlzrTzMGVr7Tbg4BLbKpxnNQjmk+Qh5vukN+6SCwi+Ahu4Gk55vnI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598006061; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=PtV7VLP3ii69wiqNVBJxmBFeD723UgNJUixvpYytDdQ=; b=WjYr39sO6FXFpNZdMg9WpMTLMes0L86hwZYQtiDQY7MoyQV7N2f+Q9eFPBNn+KqeOECnZfOWTEHj+iqiincMaUT1DEH/Spt5EOJNuTN2zjP+HFXBDD/y5Cex4YtSVuDsm26EaA7xgii4UcpoiffaiZ0OBDphovLb2/H2VKNHjqg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598006061521829.7348391990324; Fri, 21 Aug 2020 03:34:21 -0700 (PDT) Received: from localhost ([::1]:51914 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k94NI-0004Ew-4x for importer@patchew.org; Fri, 21 Aug 2020 06:34:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59622) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k94DJ-00019e-MW for qemu-devel@nongnu.org; Fri, 21 Aug 2020 06:24:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:50236) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k94DG-0001UN-6V for qemu-devel@nongnu.org; Fri, 21 Aug 2020 06:24:01 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-54-VLpqcKKWPgiEVqvGFvcxmw-1; Fri, 21 Aug 2020 06:23:55 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CF43E191E2A8 for ; Fri, 21 Aug 2020 10:23:54 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9E1727AECF for ; Fri, 21 Aug 2020 10:23:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1598005437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PtV7VLP3ii69wiqNVBJxmBFeD723UgNJUixvpYytDdQ=; b=Q6+RUYIHS62RBRRmbcSs+DZn9hAb1vVhJNg40eNmChOATrRK0Si35+rxluamCz78c3i3Wk vbGznkssInRNyXks7I/1Te4R4o3b3gRKssAOCuhciMng/Cb9Cj7tEfkejMcHseM10IBTud j2NK5N6/IVcRwAviBmW2uABqrsPXAlA= X-MC-Unique: VLpqcKKWPgiEVqvGFvcxmw-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PULL v8 025/152] meson: add testsuite Makefile generator Date: Fri, 21 Aug 2020 06:21:22 -0400 Message-Id: <20200821102329.29777-26-pbonzini@redhat.com> In-Reply-To: <20200821102329.29777-1-pbonzini@redhat.com> References: <20200821102329.29777-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=pbonzini@redhat.com X-Mimecast-Spam-Score: 0.0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=63.128.21.124; envelope-from=pbonzini@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/21 02:43:55 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=-1, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Rules to execute tests are generated by a simple Python program that integrates into the existing "make check" mechanism. This provides familiarity for developers, and also allows piecewise conversion of the testsuite Makefiles to meson. The generated rules are based on QEMU's existing test harness Makefile and TAP parser. Signed-off-by: Paolo Bonzini --- Makefile | 10 +++- scripts/mtest2make.py | 102 +++++++++++++++++++++++++++++++++++++++++ tests/Makefile.include | 1 - 3 files changed, 111 insertions(+), 2 deletions(-) create mode 100644 scripts/mtest2make.py diff --git a/Makefile b/Makefile index 81c9642a30..5f9aae6e3e 100644 --- a/Makefile +++ b/Makefile @@ -68,6 +68,14 @@ Makefile.ninja: build.ninja ninjatool =20 ${ninja-targets-c_COMPILER} ${ninja-targets-cpp_COMPILER}: .var.command += =3D -MP =20 +# If MESON is empty, the rule will be re-evaluated after Makefiles are +# reread (and MESON won't be empty anymore). +ifneq ($(MESON),) +Makefile.mtest: build.ninja scripts/mtest2make.py + $(MESON) introspect --tests | $(PYTHON) scripts/mtest2make.py > $@ +-include Makefile.mtest +endif + .git-submodule-status: git-submodule-update config-host.mak =20 # Check that we're not trying to do an out-of-tree build from @@ -825,7 +833,7 @@ distclean: clean ninja-distclean rm -f roms/seabios/config.mak roms/vgabios/config.mak rm -f qemu-plugins-ld.symbols qemu-plugins-ld64.symbols rm -rf meson-private meson-logs meson-info compile_commands.json - rm -f Makefile.ninja ninjatool ninjatool.stamp + rm -f Makefile.ninja ninjatool ninjatool.stamp Makefile.mtest rm -f config.log rm -f linux-headers/asm rm -f docs/version.texi diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py new file mode 100644 index 0000000000..bdb257bbd9 --- /dev/null +++ b/scripts/mtest2make.py @@ -0,0 +1,102 @@ +#! /usr/bin/env python3 + +# Create Makefile targets to run tests, from Meson's test introspection da= ta. +# +# Author: Paolo Bonzini + +from collections import defaultdict +import json +import os +import shlex +import sys + +class Suite(object): + def __init__(self): + self.tests =3D list() + self.slow_tests =3D list() + self.executables =3D set() + +print(''' +SPEED =3D quick + +# $1 =3D test command, $2 =3D test name +.test-human-tap =3D $1 < /dev/null | ./scripts/tap-driver.pl --test-name= =3D"$2" $(if $(V),,--show-failures-only) +.test-human-exitcode =3D $1 < /dev/null +.test-tap-tap =3D $1 < /dev/null | sed "s/^[a-z][a-z]* [0-9]*/& $2/" || tr= ue +.test-tap-exitcode =3D printf "%s\\n" 1..1 "`$1 < /dev/null > /dev/null ||= echo "not "`ok 1 $2" +.test.print =3D echo $(if $(V),'$1','Running test $2') >&3 +.test.env =3D MALLOC_PERTURB_=3D$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 2= 55 + 1))} + +# $1 =3D test name, $2 =3D test target (human or tap) +.test.run =3D $(call .test.print,$(.test.cmd.$1),$(.test.name.$1)) && $(ca= ll .test-$2-$(.test.driver.$1),$(.test.cmd.$1),$(.test.name.$1)) + +define .test.human_k + @exec 3>&1; rc=3D0; $(foreach TEST, $1, $(call .test.run,$(TEST),h= uman) || rc=3D$$?;) \\ + exit $$rc +endef +define .test.human_no_k + $(foreach TEST, $1, @exec 3>&1; $(call .test.run,$(TEST),human) +) +endef +.test.human =3D \\ + $(if $(findstring k, $(MAKEFLAGS)), $(.test.human_k), $(.test.huma= n_no_k)) + +define .test.tap + @exec 3>&1; { $(foreach TEST, $1, $(call .test.run,$(TEST),tap); )= } \\ + | ./scripts/tap-merge.pl | tee "$@" \\ + | ./scripts/tap-driver.pl $(if $(V),, --show-failures-only) +endef +''') + +suites =3D defaultdict(Suite) +i =3D 0 +for test in json.load(sys.stdin): + env =3D ' '.join(('%s=3D%s' % (shlex.quote(k), shlex.quote(v)) + for k, v in test['env'].items())) + executable =3D os.path.relpath(test['cmd'][0]) + if test['workdir'] is not None: + test['cmd'][0] =3D os.path.relpath(test['cmd'][0], test['workdir']) + else: + test['cmd'][0] =3D executable + cmd =3D '$(.test.env) %s %s' % (env, ' '.join((shlex.quote(x) for x in= test['cmd']))) + if test['workdir'] is not None: + cmd =3D '(cd %s && %s)' % (shlex.quote(test['workdir']), cmd) + driver =3D test['protocol'] if 'protocol' in test else 'exitcode' + + i +=3D 1 + print('.test.name.%d :=3D %s' % (i, test['name'])) + print('.test.driver.%d :=3D %s' % (i, driver)) + print('.test.cmd.%d :=3D %s' % (i, cmd)) + + test_suites =3D test['suite'] or ['default'] + is_slow =3D any(s.endswith('-slow') for s in test_suites) + for s in test_suites: + # The suite name in the introspection info is "PROJECT:SUITE" + s =3D s.split(':')[1] + if s.endswith('-slow'): + s =3D s[:-5] + if is_slow: + suites[s].slow_tests.append(i) + else: + suites[s].tests.append(i) + suites[s].executables.add(executable) + +print('.PHONY: check check-report.tap') +print('check:') +print('check-report.tap:') +print('\t@cat $^ | scripts/tap-merge.pl >$@') +for name, suite in suites.items(): + executables =3D ' '.join(suite.executables) + slow_test_numbers =3D ' '.join((str(x) for x in suite.slow_tests)) + test_numbers =3D ' '.join((str(x) for x in suite.tests)) + print('.test.suite-quick.%s :=3D %s' % (name, test_numbers)) + print('.test.suite-slow.%s :=3D $(.test.suite-quick.%s) %s' % (name, n= ame, slow_test_numbers)) + print('check-build: %s' % executables) + print('.PHONY: check-%s' % name) + print('.PHONY: check-report-%s.tap' % name) + print('check: check-%s' % name) + print('check-%s: all %s' % (name, executables)) + print('\t$(call .test.human, $(.test.suite-$(SPEED).%s))' % (name, )) + print('check-report.tap: check-report-%s.tap' % name) + print('check-report-%s.tap: %s' % (name, executables)) + print('\t$(call .test.tap, $(.test.suite-$(SPEED).%s))' % (name, )) diff --git a/tests/Makefile.include b/tests/Makefile.include index c7e4646ded..ad54100369 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -674,7 +674,6 @@ check-report-unit.tap: $(check-unit-y) # Reports and overall runs =20 check-report.tap: $(patsubst %,check-report-qtest-%.tap, $(QTEST_TARGETS))= check-report-unit.tap - $(call quiet-command, cat $^ | scripts/tap-merge.pl >$@,"GEN","$@") =20 # FPU Emulation tests (aka softfloat) # --=20 2.26.2