From nobody Thu Nov 6 08:25:36 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539792647118107.35492637489756; Wed, 17 Oct 2018 09:10:47 -0700 (PDT) Received: from localhost ([::1]:37994 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoPF-0003Kb-AZ for importer@patchew.org; Wed, 17 Oct 2018 12:10:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59492) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoKr-0008FR-Ql for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCoKq-0003nw-J4 for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:13 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49216) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gCoKm-0003Ri-NU for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:10 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8039230C9EEB; Wed, 17 Oct 2018 16:06:01 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-123-183.rdu2.redhat.com [10.10.123.183]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EEA185E7C9; Wed, 17 Oct 2018 16:05:59 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Wed, 17 Oct 2018 12:05:38 -0400 Message-Id: <20181017160539.12316-3-crosa@redhat.com> In-Reply-To: <20181017160539.12316-1-crosa@redhat.com> References: <20181017160539.12316-1-crosa@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Wed, 17 Oct 2018 16:06:01 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 2/3] Acceptance tests: add make rule for running them X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Caio Carrara , Laszlo Ersek Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The acceptance (aka functional, aka Avocado-based) tests are Python files located in "tests/acceptance" that need to be run with the Avocado libs and test runner. Let's provide a convenient way for QEMU developers to run them, by making use of the tests-venv with the required setup. Also, while the Avocado test runner will take care of creating a location to save test results to, it was understood that it's better if the results are kept within the build tree. Signed-off-by: Cleber Rosa Acked-by: Stefan Hajnoczi --- docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++----- tests/Makefile.include | 21 +++++++++++++++++++-- tests/requirements.txt | 1 + 3 files changed, 58 insertions(+), 7 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 727c4019b5..e04feaccb5 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily: - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.ht= ml#avocado.Test - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.u= tils.html =20 -Installation ------------- +Running tests +------------- =20 -To install Avocado and its dependencies, run: +You can run the acceptance tests simply by executing: + +.. code:: + + make check-acceptance + +This involves the automatic creation of Python virtual environment +within the build tree (at ``tests/venv``) which will have all the +right dependencies, and will save tests results also within the +build tree (at ``tests/results``). + +Note: the build environment must be using a Python 3 stack, and have +the ``venv`` and ``pip`` packages installed. If necessary, make sure +``configure`` is called with ``--python=3D`` and that those modules are +available. On Debian and Ubuntu based systems, depending on the +specific version, they may be on packages named ``python3-venv`` and +``python3-pip``. + +The scripts installed inside the virtual environment may be used +without an "activation". For instance, the Avocado test runner +may be invoked by running: + + .. code:: + + tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ + +Manual Installation +------------------- + +To manually install Avocado and its dependencies, run: =20 .. code:: =20 @@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine. Uninstalling Avocado -------------------- =20 -If you've followed the installation instructions above, you can easily -uninstall Avocado. Start by listing the packages you have installed:: +If you've followed the manual installation instructions above, you can +easily uninstall Avocado. Start by listing the packages you have +installed:: =20 pip list --user =20 And remove any package you want with:: =20 pip uninstall + +If you've used ``make check-acceptance``, the Python virtual environment w= here +Avocado is installed will be cleaned up as part of ``make check-clean``. diff --git a/tests/Makefile.include b/tests/Makefile.include index 4b88a119e0..406a9bdaf8 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -11,6 +11,7 @@ check-help: @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-block Run block tests" @echo " $(MAKE) check-tcg Run TCG tests" + @echo " $(MAKE) check-acceptance Run all acceptance (functional) test= s" @echo " $(MAKE) check-report.html Generates an HTML test report" @echo " $(MAKE) check-venv Creates a Python venv for tests" @echo " $(MAKE) check-clean Clean the tests" @@ -1020,10 +1021,15 @@ check-decodetree: =20 # Python venv for running tests =20 -.PHONY: check-venv +.PHONY: check-venv check-acceptance =20 TESTS_VENV_DIR=3D$(BUILD_DIR)/tests/venv TESTS_VENV_REQ=3D$(SRC_PATH)/tests/requirements.txt +TESTS_RESULTS_DIR=3D$(BUILD_DIR)/tests/results +# Controls the output generated by Avocado when running tests. +# Any number of command separated loggers are accepted. For more +# information please refer to "avocado --help". +AVOCADO_SHOW=3Dnone =20 PYTHON_VER :=3D $(findstring Python 3,$(shell $(PYTHON) -V)) ifeq ($(PYTHON_VER),Python 3) @@ -1040,8 +1046,19 @@ $(TESTS_VENV_DIR): $(error "venv directory for tests requires Python 3") endif =20 +$(TESTS_RESULTS_DIR): + $(call quiet-command, mkdir -p $@, \ + MKDIR, $@) + check-venv: $(TESTS_VENV_DIR) =20 +check-acceptance: check-venv $(TESTS_RESULTS_DIR) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m avocado \ + --show=3D$(AVOCADO_SHOW) run --job-results-dir=3D$(TESTS_RESUL= TS_DIR) \ + --failfast=3Don $(SRC_PATH)/tests/acceptance, \ + "AVOCADO", "tests/acceptance") + # Consolidated targets =20 .PHONY: check-qapi-schema check-qtest check-unit check check-clean @@ -1055,7 +1072,7 @@ check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(tar= get)-y)) $(check-qtest-generic-y)) rm -f tests/test-qapi-gen-timestamp - rm -rf $(TESTS_VENV_DIR) + rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) =20 clean: check-clean =20 diff --git a/tests/requirements.txt b/tests/requirements.txt index d39f9d1576..64c6e27a94 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,3 +1,4 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 +avocado-framework=3D=3D65.0 --=20 2.17.1