From nobody Tue Nov 11 03:18:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561097578; cv=none; d=zoho.com; s=zohoarc; b=CLsGkAmMKSORaxHeGBq92JSS0nrMR90/+u6GV0xPb8rs5pMLDdAFJoZmhPXJhFbQjyPnK31cxdHzqy9zRuCzhEgBRo8sls8dP/x4OB2nV4MCrvVGBsEf93/8AxiK1FgtgGbA/4v3mIxQ8CM6bOtx4InlCyyIWYMz6acrJ9k061o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561097578; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=zkQkyYT7K+2E8laIQP7cFQT8389cbfs+VmMKp6PtsN8=; b=IHxLWQMEe5VTr8awBgL5WeceQ/IJuzOr43cJmx6QHoxozgya/p1b0nlHaxU96Dm3beexpkX073DFQIQLhuztoUpY6edVmWZ++dTCgbDrYJpyAJYFdwfZM947wHdJjCE81RWLCey+sEjyI22DahqYmVJqblha8C0SAMpw5uQ2QB4= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561097578810789.739492890273; Thu, 20 Jun 2019 23:12:58 -0700 (PDT) Received: from localhost ([::1]:54962 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1heCn8-0007Rv-7b for importer@patchew.org; Fri, 21 Jun 2019 02:12:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47109) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1heCks-0005C7-GW for qemu-devel@nongnu.org; Fri, 21 Jun 2019 02:10:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1heCkb-00049U-6T for qemu-devel@nongnu.org; Fri, 21 Jun 2019 02:10:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41860) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1heCka-0003JO-Ul for qemu-devel@nongnu.org; Fri, 21 Jun 2019 02:10:17 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4528B85539; Fri, 21 Jun 2019 06:09:44 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-120-204.rdu2.redhat.com [10.10.120.204]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7E42F5D772; Fri, 21 Jun 2019 06:09:36 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Fri, 21 Jun 2019 02:09:24 -0400 Message-Id: <20190621060925.16214-2-crosa@redhat.com> In-Reply-To: <20190621060925.16214-1-crosa@redhat.com> References: <20190621060925.16214-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 21 Jun 2019 06:09:45 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/2] Acceptance tests: exclude "flaky" tests X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , Aleksandar Rikalo , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Wainer dos Santos Moschetta , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Aurelien Jarno Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" It's a fact that some tests may not be 100% reliable in all environments. While it's a tough call to remove a useful test that from the tree because it may fail every 1/100th time (or so), having human attention drawn to known issues is very bad for humans and for the projects they manage. As a compromise solution, this marks tests that are known to have issues, or that exercises known issues in QEMU or other components, and excludes them from the entry point. As a consequence, tests marked as "flaky" will not be executed as part of "make check-acceptance". Because such tests should be forgiven but never be forgotten, it's possible to list them with (assuming "make check-venv" or "make check-acceptance" has already initiatilized the venv): $ ./tests/venv/bin/avocado list -t flaky tests/acceptance The current list of tests marked as flaky are a result of running the entire set of acceptance tests around 20 times. The results were then processed with a helper script[1]. That either confirmed known issues (in the case of aarch64 and arm)[2] or revealed new ones (mips). This also bumps the Avocado version to one that includes a fix to the parsing of multiple and mix "key:val" and simple tag values. [1] https://raw.githubusercontent.com/avocado-framework/avocado/master/cont= rib/scripts/summarize-job-failures.py [2] https://bugs.launchpad.net/qemu/+bug/1829779 Signed-off-by: Cleber Rosa --- docs/devel/testing.rst | 17 +++++++++++++++++ tests/Makefile.include | 6 +++++- tests/acceptance/boot_linux_console.py | 2 ++ tests/acceptance/linux_ssh_mips_malta.py | 2 ++ tests/requirements.txt | 2 +- 5 files changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index da2d0fc964..ff4d8e2e1c 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -574,6 +574,23 @@ may be invoked by running: =20 tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ =20 +Tagging tests +------------- + +flaky +~~~~~ + +If a test is known to fail intermittently, even if only every one +hundredth time, it's highly advisable to mark it as a flaky test. +This will prevent these individual tests from failing much larger +jobs, will avoid human interaction and time wasted to verify a known +issue, and worse of all, can lead to the discredit of automated +testing. + +To mark a test as flaky, add to its docstring.:: + + :avocado: tags=3Dflaky + Manual Installation ------------------- =20 diff --git a/tests/Makefile.include b/tests/Makefile.include index db750dd6d0..4c97da2878 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -1125,7 +1125,11 @@ TESTS_RESULTS_DIR=3D$(BUILD_DIR)/tests/results # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help". AVOCADO_SHOW=3Dapp -AVOCADO_TAGS=3D$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET= _DIRS))) + +# Additional tags that are added to each occurence of "--filter-by-tags" +AVOCADO_EXTRA_TAGS :=3D ,-flaky + +AVOCADO_TAGS=3D$(patsubst %-softmmu,--filter-by-tags=3Darch:%$(AVOCADO_EXT= RA_TAGS), $(filter %-softmmu,$(TARGET_DIRS))) =20 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2) $(TESTS_VENV_DIR): $(TESTS_VENV_REQ) diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot= _linux_console.py index 32159503e9..6bd5c1ab53 100644 --- a/tests/acceptance/boot_linux_console.py +++ b/tests/acceptance/boot_linux_console.py @@ -249,6 +249,7 @@ class BootLinuxConsole(Test): """ :avocado: tags=3Darch:aarch64 :avocado: tags=3Dmachine:virt + :avocado: tags=3Dflaky """ kernel_url =3D ('https://download.fedoraproject.org/pub/fedora/lin= ux/' 'releases/29/Everything/aarch64/os/images/pxeboot/vm= linuz') @@ -270,6 +271,7 @@ class BootLinuxConsole(Test): """ :avocado: tags=3Darch:arm :avocado: tags=3Dmachine:virt + :avocado: tags=3Dflaky """ kernel_url =3D ('https://download.fedoraproject.org/pub/fedora/lin= ux/' 'releases/29/Everything/armhfp/os/images/pxeboot/vml= inuz') diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/li= nux_ssh_mips_malta.py index aafb0c39f6..ae70b658e0 100644 --- a/tests/acceptance/linux_ssh_mips_malta.py +++ b/tests/acceptance/linux_ssh_mips_malta.py @@ -208,6 +208,7 @@ class LinuxSSH(Test): :avocado: tags=3Dmachine:malta :avocado: tags=3Dendian:big :avocado: tags=3Ddevice:pcnet32 + :avocado: tags=3Dflaky """ kernel_url =3D ('https://people.debian.org/~aurel32/qemu/mips/' 'vmlinux-3.2.0-4-5kc-malta') @@ -222,6 +223,7 @@ class LinuxSSH(Test): :avocado: tags=3Dmachine:malta :avocado: tags=3Dendian:little :avocado: tags=3Ddevice:pcnet32 + :avocado: tags=3Dflaky """ kernel_url =3D ('https://people.debian.org/~aurel32/qemu/mipsel/' 'vmlinux-3.2.0-4-5kc-malta') diff --git a/tests/requirements.txt b/tests/requirements.txt index 3ae0e29ad7..58d63d171f 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,5 +1,5 @@ # Add Python module requirements, one per line, to be installed # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 -avocado-framework=3D=3D68.0 +avocado-framework=3D=3D69.1 paramiko --=20 2.21.0 From nobody Tue Nov 11 03:18:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561097755; cv=none; d=zoho.com; s=zohoarc; b=licyLMF2nx7AKn7oTZoMmwxwgY1HUeQCuWCBxb70RDBu0fqESO+lr19JA1BFsIiyK4yNjbDCs5PqK5CGBBr1jC9lfa6GG1YPrHLQzU/UQaRFHhTwagYHYmcQOjUKG11AidVZcK2C2x5trkF/+mXmMEEfG4T3xBwrazcuuYovgg0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561097755; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=JFdKWWTD1gA5PUPxn/aRTV0heQP+Id1Ni0NKEDV9Bfw=; b=V5DTL3ZY7vQkJ1ZEV8QQUUwRDdxElijx1B3JqSLXBxyEU+FSYZT04gCqOh05QFIoDXb+K7pfIRQeqYML9Bsz4ym6hZUT7vdW0qOzNGkb/gwzbIHbdGXwuxaflXcUm94U2u5eEHXRmBKUNuy5Br5bR5suff5p0dbNSysmFa4uGWg= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1561097755278326.9527426628273; Thu, 20 Jun 2019 23:15:55 -0700 (PDT) Received: from localhost ([::1]:55009 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1heCq0-0001zs-Bz for importer@patchew.org; Fri, 21 Jun 2019 02:15:52 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47109) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1heCkw-0005C7-Uc for qemu-devel@nongnu.org; Fri, 21 Jun 2019 02:10:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1heCkX-00046Y-2M for qemu-devel@nongnu.org; Fri, 21 Jun 2019 02:10:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:13970) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1heCkV-0003YM-O0 for qemu-devel@nongnu.org; Fri, 21 Jun 2019 02:10:12 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1006C81F1B; Fri, 21 Jun 2019 06:09:52 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-120-204.rdu2.redhat.com [10.10.120.204]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 734735B683; Fri, 21 Jun 2019 06:09:44 +0000 (UTC) From: Cleber Rosa To: qemu-devel@nongnu.org Date: Fri, 21 Jun 2019 02:09:25 -0400 Message-Id: <20190621060925.16214-3-crosa@redhat.com> In-Reply-To: <20190621060925.16214-1-crosa@redhat.com> References: <20190621060925.16214-1-crosa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Fri, 21 Jun 2019 06:09:53 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/2] Acceptance tests: add SPICE protocol check X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , Aleksandar Rikalo , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Wainer dos Santos Moschetta , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Aurelien Jarno Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" This fires a QEMU binary with SPICE enabled, and does a basic handshake, doing a basic client/server interaction and protocol validation. Signed-off-by: Cleber Rosa --- .travis.yml | 5 +++- tests/Makefile.include | 6 +++++ tests/acceptance/spice.py | 54 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 tests/acceptance/spice.py diff --git a/.travis.yml b/.travis.yml index aeb9b211cd..6c9257a459 100644 --- a/.travis.yml +++ b/.travis.yml @@ -231,7 +231,7 @@ matrix: =20 # Acceptance (Functional) tests - env: - - CONFIG=3D"--python=3D/usr/bin/python3 --target-list=3Dx86_64-sof= tmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmm= u,alpha-softmmu" + - CONFIG=3D"--python=3D/usr/bin/python3 --enable-spice --target-li= st=3Dx86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softm= mu,s390x-softmmu,alpha-softmmu" - TEST_CMD=3D"make check-acceptance" after_failure: - cat tests/results/latest/job.log @@ -240,6 +240,9 @@ matrix: packages: - python3-pip - python3.5-venv + - libspice-protocol-dev + - libspice-server-dev + # Using newer GCC with sanitizers - addons: apt: diff --git a/tests/Makefile.include b/tests/Makefile.include index 4c97da2878..7fc2d28099 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -1129,6 +1129,12 @@ AVOCADO_SHOW=3Dapp # Additional tags that are added to each occurence of "--filter-by-tags" AVOCADO_EXTRA_TAGS :=3D ,-flaky =20 +# At last one test require spice to be enabled, allow it to be excluded +# if it's not enabled +ifneq ($(findstring y,"$(CONFIG_SPICE)"),y) +AVOCADO_EXTRA_TAGS :=3D $(AVOCADO_EXTRA_TAGS),-spice +endif + AVOCADO_TAGS=3D$(patsubst %-softmmu,--filter-by-tags=3Darch:%$(AVOCADO_EXT= RA_TAGS), $(filter %-softmmu,$(TARGET_DIRS))) =20 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2) diff --git a/tests/acceptance/spice.py b/tests/acceptance/spice.py new file mode 100644 index 0000000000..aa22b1992d --- /dev/null +++ b/tests/acceptance/spice.py @@ -0,0 +1,54 @@ +# Simple functional tests for SPICE functionality +# +# Copyright (c) 2019 Red Hat, Inc. +# +# Author: +# Cleber Rosa +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. + +import socket +import struct + +from avocado_qemu import Test +from avocado.utils.network import find_free_port + + +class Spice(Test): + + def test_protocol(self): + """ + :avocado: tags=3Dquick + :avocado: tags=3Dspice + """ + port =3D find_free_port(5001, 5500, sequent=3DFalse) + self.vm.add_args('-nodefaults', '-S', + '-spice', 'port=3D%d,disable-ticketing' % port) + self.vm.launch() + + RED_MAGIC =3D 0x51444552 + MAJOR_VERSION =3D 0x2 + + client =3D socket.socket(socket.AF_INET, socket.SOCK_STREAM) + client.connect(('127.0.0.1', port)) + red_link_mess =3D struct.pack('<10I', + RED_MAGIC, # magic + MAJOR_VERSION, # major version + 0x0, # minor version + 0x18, # size in bytes from h= ere + 0x0, # connection id + 0x1, # channel type RED_CHA= NNEL_MAIN + 0x0, # channel id + 0x0, # number of common caps + 0x0, # number of channel ca= ps + 0x14) # caps offset from size + client.send(red_link_mess) + + RED_LINK_REPLY_BASE_FMT =3D '<5I' # magic, major, minor, size, er= ror + red_link_reply =3D client.recv(struct.calcsize(RED_LINK_REPLY_BASE= _FMT)) + (magic, major, minor, size, error) =3D struct.unpack_from(RED_LINK= _REPLY_BASE_FMT, + red_link_r= eply) + self.assertEqual(magic, RED_MAGIC, "Mismatch of MAGIC number") + self.assertEqual(major, MAJOR_VERSION, "Mismatch of major protocol= version") + self.assertEqual(error, 0x0, "Unexpected error reported by server") --=20 2.21.0