[PULL 0/9] testing updates (tests/vm and acceptance tweaks)

Alex Bennée posted 9 patches 5 years, 8 months ago
Only 2 patches received!
configure                 | 13 ++++++++++
.travis.yml               |  6 ++++-
tests/requirements.txt    |  2 +-
tests/vm/Makefile.include | 16 +++++++++----
tests/vm/basevm.py        | 61 ++++++++++++++++++++++++++++++++++++++++++++---
tests/vm/centos           | 33 +------------------------
tests/vm/ubuntu.i386      | 37 +---------------------------
7 files changed, 91 insertions(+), 77 deletions(-)
[PULL 0/9] testing updates (tests/vm and acceptance tweaks)
Posted by Alex Bennée 5 years, 8 months ago
The following changes since commit abfa865014ab17941eb1fcb7cc2fa293a25843c4:

  Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-virtiofs-20200303' into staging (2020-03-03 15:20:12 +0000)

are available in the Git repository at:

  https://github.com/stsquad/qemu.git tags/pull-testing-040320-1

for you to fetch changes up to c2e09ad8cdf4705a91eb6c20a9f3d4d90a0f46aa:

  travis.yml: install python3 numpy and opencv libraries (2020-03-04 09:57:33 +0000)

----------------------------------------------------------------
Testing updates

  - some clean-ups for tests/vm
  - enable tools build for MacOSX
  - bump avocado to a newer version
  - bump travis env for avocado

----------------------------------------------------------------
Alex Bennée (3):
      configure: detect and report genisoimage
      tests/acceptance: bump avocado requirements to 76.0
      travis.yml: install python3 numpy and opencv libraries

Laurent Vivier (1):
      travis: enable tools build on OS X

Robert Foley (5):
      tests/vm: use $(PYTHON) consistently
      tests/vm: Debug mode shows ssh output.
      tests/vm: increased max timeout for vm boot.
      tests/vm: give wait_ssh() option to wait for root
      tests/vm: Added gen_cloud_init_iso() to basevm.py

 configure                 | 13 ++++++++++
 .travis.yml               |  6 ++++-
 tests/requirements.txt    |  2 +-
 tests/vm/Makefile.include | 16 +++++++++----
 tests/vm/basevm.py        | 61 ++++++++++++++++++++++++++++++++++++++++++++---
 tests/vm/centos           | 33 +------------------------
 tests/vm/ubuntu.i386      | 37 +---------------------------
 7 files changed, 91 insertions(+), 77 deletions(-)

-- 
2.20.1


Re: [PULL 0/9] testing updates (tests/vm and acceptance tweaks)
Posted by Peter Maydell 5 years, 8 months ago
On Wed, 4 Mar 2020 at 10:01, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The following changes since commit abfa865014ab17941eb1fcb7cc2fa293a25843c4:
>
>   Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-virtiofs-20200303' into staging (2020-03-03 15:20:12 +0000)
>
> are available in the Git repository at:
>
>   https://github.com/stsquad/qemu.git tags/pull-testing-040320-1
>
> for you to fetch changes up to c2e09ad8cdf4705a91eb6c20a9f3d4d90a0f46aa:
>
>   travis.yml: install python3 numpy and opencv libraries (2020-03-04 09:57:33 +0000)
>
> ----------------------------------------------------------------
> Testing updates
>
>   - some clean-ups for tests/vm
>   - enable tools build for MacOSX
>   - bump avocado to a newer version
>   - bump travis env for avocado
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.0
for any user-visible changes.

-- PMM

[PULL 1/9] tests/vm: use $(PYTHON) consistently
Posted by Alex Bennée 5 years, 8 months ago
From: Robert Foley <robert.foley@linaro.org>

Change Makefile.include to use $(PYTHON) so for vm-boot-ssh to be
consistent with other cases like vm-build.

Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200219163537.22098-2-robert.foley@linaro.org>
Message-Id: <20200303150622.20133-2-alex.bennee@linaro.org>

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 9e7c46a4735..778e5067554 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -80,7 +80,7 @@ vm-boot-serial-%: $(IMAGES_DIR)/%.img
 
 vm-boot-ssh-%: $(IMAGES_DIR)/%.img
 	$(call quiet-command, \
-		$(SRC_PATH)/tests/vm/$* \
+		$(PYTHON) $(SRC_PATH)/tests/vm/$* \
 		$(if $(J),--jobs $(J)) \
 		--image "$<" \
 		--interactive \
-- 
2.20.1


[PULL 2/9] tests/vm: Debug mode shows ssh output.
Posted by Alex Bennée 5 years, 8 months ago
From: Robert Foley <robert.foley@linaro.org>

Add changes to tests/vm/basevm.py so that during debug mode we show ssh output.

Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200219163537.22098-3-robert.foley@linaro.org>
Message-Id: <20200303150622.20133-3-alex.bennee@linaro.org>

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 4dee6647e6e..c99725b8c0d 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -120,11 +120,16 @@ class BaseVM(object):
         return fname
 
     def _ssh_do(self, user, cmd, check):
-        ssh_cmd = ["ssh", "-q", "-t",
+        ssh_cmd = ["ssh",
+                   "-t",
                    "-o", "StrictHostKeyChecking=no",
                    "-o", "UserKnownHostsFile=" + os.devnull,
                    "-o", "ConnectTimeout=1",
                    "-p", self.ssh_port, "-i", self._ssh_key_file]
+        # If not in debug mode, set ssh to quiet mode to
+        # avoid printing the results of commands.
+        if not self.debug:
+            ssh_cmd.append("-q")
         for var in self.envvars:
             ssh_cmd += ['-o', "SendEnv=%s" % var ]
         assert not isinstance(cmd, str)
-- 
2.20.1


[PULL 3/9] tests/vm: increased max timeout for vm boot.
Posted by Alex Bennée 5 years, 8 months ago
From: Robert Foley <robert.foley@linaro.org>

Add change to increase timeout waiting for VM to boot.
Needed for some emulation cases where it can take longer
than 5 minutes to boot.

Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200219163537.22098-4-robert.foley@linaro.org>
Message-Id: <20200303150622.20133-4-alex.bennee@linaro.org>

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index c99725b8c0d..5ca445e29af 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -57,6 +57,10 @@ class BaseVM(object):
     poweroff = "poweroff"
     # enable IPv6 networking
     ipv6 = True
+    # Scale up some timeouts under TCG.
+    # 4 is arbitrary, but greater than 2,
+    # since we found we need to wait more than twice as long.
+    tcg_ssh_timeout_multiplier = 4
     def __init__(self, debug=False, vcpus=None):
         self._guest = None
         self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-",
@@ -309,6 +313,9 @@ class BaseVM(object):
         sys.stderr.write("### %s ...\n" % text)
 
     def wait_ssh(self, seconds=300):
+        # Allow more time for VM to boot under TCG.
+        if not kvm_available(self.arch):
+            seconds *= self.tcg_ssh_timeout_multiplier
         starttime = datetime.datetime.now()
         endtime = starttime + datetime.timedelta(seconds=seconds)
         guest_up = False
-- 
2.20.1


[PULL 4/9] tests/vm: give wait_ssh() option to wait for root
Posted by Alex Bennée 5 years, 8 months ago
From: Robert Foley <robert.foley@linaro.org>

Allow wait_ssh to wait for root user to be ready.
This solves the issue where we perform a wait_ssh()
successfully, but the root user is not yet ready
to be logged in.

Signed-off-by: Robert Foley <robert.foley@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Peter Puhov <peter.puhov@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200219163537.22098-5-robert.foley@linaro.org>
Message-Id: <20200303150622.20133-5-alex.bennee@linaro.org>

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 5ca445e29af..7f268922685 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -312,7 +312,7 @@ class BaseVM(object):
     def print_step(self, text):
         sys.stderr.write("### %s ...\n" % text)
 
-    def wait_ssh(self, seconds=300):
+    def wait_ssh(self, wait_root=False, seconds=300):
         # Allow more time for VM to boot under TCG.
         if not kvm_available(self.arch):
             seconds *= self.tcg_ssh_timeout_multiplier
@@ -320,7 +320,10 @@ class BaseVM(object):
         endtime = starttime + datetime.timedelta(seconds=seconds)
         guest_up = False
         while datetime.datetime.now() < endtime:
-            if self.ssh("exit 0") == 0:
+            if wait_root and self.ssh_root("exit 0") == 0:
+                guest_up = True
+                break
+            elif self.ssh("exit 0") == 0:
                 guest_up = True
                 break
             seconds = (endtime - datetime.datetime.now()).total_seconds()
-- 
2.20.1


[PULL 6/9] travis: enable tools build on OS X
Posted by Alex Bennée 5 years, 8 months ago
From: Laurent Vivier <lvivier@redhat.com>

As we can build tools on OS X we should check we don't break build
when we submit new codes.

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200302154630.45620-3-lvivier@redhat.com>
Message-Id: <20200303150622.20133-7-alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index 9867272177d..70a24bf2fc8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -273,6 +273,7 @@ jobs:
 
     - name: "OSX Xcode 10.3"
       env:
+        - BASE_CONFIG="--disable-docs --enable-tools"
         - CONFIG="--target-list=i386-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,x86_64-softmmu"
       os: osx
       osx_image: xcode10.3
-- 
2.20.1


[PULL 8/9] tests/acceptance: bump avocado requirements to 76.0
Posted by Alex Bennée 5 years, 8 months ago
If we want to use @skipUnless decorations on the class we need a
newer version of avocado.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200303150622.20133-9-alex.bennee@linaro.org>

diff --git a/tests/requirements.txt b/tests/requirements.txt
index a2a587223a9..f4f1736a086 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,4 +1,4 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==72.0
+avocado-framework==76.0
-- 
2.20.1


[PULL 9/9] travis.yml: install python3 numpy and opencv libraries
Posted by Alex Bennée 5 years, 8 months ago
These are used for the acceptance framebuffer tests to count Tux. As
we need slightly newer python3 for opencv we bump up to bionic.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Message-Id: <20200303150622.20133-10-alex.bennee@linaro.org>

diff --git a/.travis.yml b/.travis.yml
index 70a24bf2fc8..b92798ac3b9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -313,6 +313,7 @@ jobs:
 
     # Acceptance (Functional) tests
     - name: "GCC check-acceptance"
+      dist: bionic
       env:
         - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
         - TEST_CMD="make check-acceptance"
@@ -323,7 +324,9 @@ jobs:
           packages:
             - python3-pil
             - python3-pip
-            - python3.5-venv
+            - python3-numpy
+            - python3-opencv
+            - python3-venv
             - rpm2cpio
             - tesseract-ocr
             - tesseract-ocr-eng
-- 
2.20.1