From nobody Thu Nov 6 06:16:04 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539792605235169.28974245414202; Wed, 17 Oct 2018 09:10:05 -0700 (PDT) Received: from localhost ([::1]:37984 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoOa-0002oh-6B for importer@patchew.org; Wed, 17 Oct 2018 12:10:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59465) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoKq-0008Ek-LE for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCoKk-0003cs-OK for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:10 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52964) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gCoKh-0003P8-Gq for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:05 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C291A7E9FD; Wed, 17 Oct 2018 16:05:59 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-123-183.rdu2.redhat.com [10.10.123.183]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 38FAD5E7B1; Wed, 17 Oct 2018 16:05:58 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Wed, 17 Oct 2018 12:05:37 -0400 Message-Id: <20181017160539.12316-2-crosa@redhat.com> In-Reply-To: <20181017160539.12316-1-crosa@redhat.com> References: <20181017160539.12316-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Wed, 17 Oct 2018 16:05:59 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 1/3] Bootstrap Python venv for tests X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Caio Carrara , Laszlo Ersek Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" A number of QEMU tests are written in Python, and may benefit from an untainted Python venv. By using make rules, tests that depend on specific Python libs can set that rule as a requirement, along with rules that require the presence or installation of specific libraries. The tests/venv-requirements.txt is supposed to contain the Python requirements that should be added to the venv created by check-venv. Signed-off-by: Cleber Rosa Tested-by: Philippe Mathieu-Daud=C3=A9 Acked-by: Stefan Hajnoczi Acked-by: Wainer dos Santos Moschetta Reviewed-by: Caio Carrara --- tests/Makefile.include | 26 ++++++++++++++++++++++++++ tests/requirements.txt | 3 +++ 2 files changed, 29 insertions(+) create mode 100644 tests/requirements.txt diff --git a/tests/Makefile.include b/tests/Makefile.include index 5eadfd52f9..4b88a119e0 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -12,6 +12,7 @@ check-help: @echo " $(MAKE) check-block Run block tests" @echo " $(MAKE) check-tcg Run TCG tests" @echo " $(MAKE) check-report.html Generates an HTML test report" + @echo " $(MAKE) check-venv Creates a Python venv for tests" @echo " $(MAKE) check-clean Clean the tests" @echo @echo "Please note that HTML reports do not regenerate if the unit tests" @@ -1017,6 +1018,30 @@ check-decodetree: ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \ TEST, decodetree.py) =20 +# Python venv for running tests + +.PHONY: check-venv + +TESTS_VENV_DIR=3D$(BUILD_DIR)/tests/venv +TESTS_VENV_REQ=3D$(SRC_PATH)/tests/requirements.txt + +PYTHON_VER :=3D $(findstring Python 3,$(shell $(PYTHON) -V)) +ifeq ($(PYTHON_VER),Python 3) +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ) + $(call quiet-command, \ + $(PYTHON) -m venv --system-site-packages $@, \ + VENV, $@) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV= _REQ), \ + PIP, $(TESTS_VENV_REQ)) + $(call quiet-command, touch $@) +else +$(TESTS_VENV_DIR): + $(error "venv directory for tests requires Python 3") +endif + +check-venv: $(TESTS_VENV_DIR) + # Consolidated targets =20 .PHONY: check-qapi-schema check-qtest check-unit check check-clean @@ -1030,6 +1055,7 @@ check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(tar= get)-y)) $(check-qtest-generic-y)) rm -f tests/test-qapi-gen-timestamp + rm -rf $(TESTS_VENV_DIR) =20 clean: check-clean =20 diff --git a/tests/requirements.txt b/tests/requirements.txt new file mode 100644 index 0000000000..d39f9d1576 --- /dev/null +++ b/tests/requirements.txt @@ -0,0 +1,3 @@ +# Add Python module requirements, one per line, to be installed +# in the tests/venv Python virtual environment. For more info, +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1 --=20 2.17.1 From nobody Thu Nov 6 06:16:04 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539792647118107.35492637489756; Wed, 17 Oct 2018 09:10:47 -0700 (PDT) Received: from localhost ([::1]:37994 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoPF-0003Kb-AZ for importer@patchew.org; Wed, 17 Oct 2018 12:10:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59492) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoKr-0008FR-Ql for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCoKq-0003nw-J4 for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:13 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49216) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gCoKm-0003Ri-NU for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:10 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8039230C9EEB; Wed, 17 Oct 2018 16:06:01 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-123-183.rdu2.redhat.com [10.10.123.183]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EEA185E7C9; Wed, 17 Oct 2018 16:05:59 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Wed, 17 Oct 2018 12:05:38 -0400 Message-Id: <20181017160539.12316-3-crosa@redhat.com> In-Reply-To: <20181017160539.12316-1-crosa@redhat.com> References: <20181017160539.12316-1-crosa@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Wed, 17 Oct 2018 16:06:01 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 2/3] Acceptance tests: add make rule for running them X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Caio Carrara , Laszlo Ersek Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The acceptance (aka functional, aka Avocado-based) tests are Python files located in "tests/acceptance" that need to be run with the Avocado libs and test runner. Let's provide a convenient way for QEMU developers to run them, by making use of the tests-venv with the required setup. Also, while the Avocado test runner will take care of creating a location to save test results to, it was understood that it's better if the results are kept within the build tree. Signed-off-by: Cleber Rosa Acked-by: Stefan Hajnoczi Reviewed-by: Caio Carrara --- docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++----- tests/Makefile.include | 21 +++++++++++++++++++-- tests/requirements.txt | 1 + 3 files changed, 58 insertions(+), 7 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 727c4019b5..e04feaccb5 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily: - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.ht= ml#avocado.Test - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.u= tils.html =20 -Installation ------------- +Running tests +------------- =20 -To install Avocado and its dependencies, run: +You can run the acceptance tests simply by executing: + +.. code:: + + make check-acceptance + +This involves the automatic creation of Python virtual environment +within the build tree (at ``tests/venv``) which will have all the +right dependencies, and will save tests results also within the +build tree (at ``tests/results``). + +Note: the build environment must be using a Python 3 stack, and have +the ``venv`` and ``pip`` packages installed. If necessary, make sure +``configure`` is called with ``--python=3D`` and that those modules are +available. On Debian and Ubuntu based systems, depending on the +specific version, they may be on packages named ``python3-venv`` and +``python3-pip``. + +The scripts installed inside the virtual environment may be used +without an "activation". For instance, the Avocado test runner +may be invoked by running: + + .. code:: + + tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ + +Manual Installation +------------------- + +To manually install Avocado and its dependencies, run: =20 .. code:: =20 @@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine. Uninstalling Avocado -------------------- =20 -If you've followed the installation instructions above, you can easily -uninstall Avocado. Start by listing the packages you have installed:: +If you've followed the manual installation instructions above, you can +easily uninstall Avocado. Start by listing the packages you have +installed:: =20 pip list --user =20 And remove any package you want with:: =20 pip uninstall + +If you've used ``make check-acceptance``, the Python virtual environment w= here +Avocado is installed will be cleaned up as part of ``make check-clean``. diff --git a/tests/Makefile.include b/tests/Makefile.include index 4b88a119e0..406a9bdaf8 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -11,6 +11,7 @@ check-help: @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-block Run block tests" @echo " $(MAKE) check-tcg Run TCG tests" + @echo " $(MAKE) check-acceptance Run all acceptance (functional) test= s" @echo " $(MAKE) check-report.html Generates an HTML test report" @echo " $(MAKE) check-venv Creates a Python venv for tests" @echo " $(MAKE) check-clean Clean the tests" @@ -1020,10 +1021,15 @@ check-decodetree: =20 # Python venv for running tests =20 -.PHONY: check-venv +.PHONY: check-venv check-acceptance =20 TESTS_VENV_DIR=3D$(BUILD_DIR)/tests/venv TESTS_VENV_REQ=3D$(SRC_PATH)/tests/requirements.txt +TESTS_RESULTS_DIR=3D$(BUILD_DIR)/tests/results +# Controls the output generated by Avocado when running tests. +# Any number of command separated loggers are accepted. For more +# information please refer to "avocado --help". +AVOCADO_SHOW=3Dnone =20 PYTHON_VER :=3D $(findstring Python 3,$(shell $(PYTHON) -V)) ifeq ($(PYTHON_VER),Python 3) @@ -1040,8 +1046,19 @@ $(TESTS_VENV_DIR): $(error "venv directory for tests requires Python 3") endif =20 +$(TESTS_RESULTS_DIR): + $(call quiet-command, mkdir -p $@, \ + MKDIR, $@) + check-venv: $(TESTS_VENV_DIR) =20 +check-acceptance: check-venv $(TESTS_RESULTS_DIR) + $(call quiet-command, \ + $(TESTS_VENV_DIR)/bin/python -m avocado \ + --show=3D$(AVOCADO_SHOW) run --job-results-dir=3D$(TESTS_RESUL= TS_DIR) \ + --failfast=3Don $(SRC_PATH)/tests/acceptance, \ + "AVOCADO", "tests/acceptance") + # Consolidated targets =20 .PHONY: check-qapi-schema check-qtest check-unit check check-clean @@ -1055,7 +1072,7 @@ check-clean: rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(tar= get)-y)) $(check-qtest-generic-y)) rm -f tests/test-qapi-gen-timestamp - rm -rf $(TESTS_VENV_DIR) + rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) =20 clean: check-clean =20 diff --git a/tests/requirements.txt b/tests/requirements.txt index d39f9d1576..64c6e27a94 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,3 +1,4 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 +avocado-framework=3D=3D65.0 --=20 2.17.1 From nobody Thu Nov 6 06:16:04 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539792533165415.94345111468283; Wed, 17 Oct 2018 09:08:53 -0700 (PDT) Received: from localhost ([::1]:37980 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoNP-0001X1-66 for importer@patchew.org; Wed, 17 Oct 2018 12:08:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59615) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gCoLH-00006v-TO for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gCoLD-0004RR-SS for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:39 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44036) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gCoL9-00045P-PZ for qemu-devel@nongnu.org; Wed, 17 Oct 2018 12:06:34 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5E8AD81DE9; Wed, 17 Oct 2018 16:06:23 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-123-183.rdu2.redhat.com [10.10.123.183]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AC1395DF49; Wed, 17 Oct 2018 16:06:01 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Wed, 17 Oct 2018 12:05:39 -0400 Message-Id: <20181017160539.12316-4-crosa@redhat.com> In-Reply-To: <20181017160539.12316-1-crosa@redhat.com> References: <20181017160539.12316-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Wed, 17 Oct 2018 16:06:23 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 3/3] Travis support for the acceptance tests X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Caio Carrara , Laszlo Ersek Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" This enables the execution of the acceptance tests on Travis. Because the Travis environment is based on Ubuntu Trusty, it requires the python3-pip and python3.4-venv packages. Signed-off-by: Cleber Rosa Tested-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Alex Benn=C3=A9e Acked-by: Stefan Hajnoczi Reviewed-by: Caio Carrara --- .travis.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.travis.yml b/.travis.yml index 95be6ec59f..aa49c7b114 100644 --- a/.travis.yml +++ b/.travis.yml @@ -117,6 +117,14 @@ matrix: - env: CONFIG=3D"--target-list=3Dx86_64-softmmu" python: - "3.6" + # Acceptance (Functional) tests + - env: CONFIG=3D"--python=3D/usr/bin/python3 --target-list=3Dx86_64-so= ftmmu" + TEST_CMD=3D"make AVOCADO_SHOW=3Dapp check-acceptance" + addons: + apt: + packages: + - python3-pip + - python3.4-venv # Using newer GCC with sanitizers - addons: apt: --=20 2.17.1