[Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console

Philippe Mathieu-Daudé posted 7 patches 5 years, 11 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20180419164642.9536-1-f4bug@amsat.org
Test checkpatch failed
Test docker-build@min-glib passed
Test docker-mingw@fedora passed
Test s390x passed
scripts/qemu.py                               |  50 ++-
tests/avocado/README.rst                      | 132 ++++++
tests/avocado/avocado_qemu/__init__.py        |   0
tests/avocado/avocado_qemu/test.py            | 419 ++++++++++++++++++
tests/avocado/parameters.yaml                 |  19 +
tests/avocado/test_info_memdev_host_nodes.py  |  66 +++
tests/avocado/test_linux-boot-console.py      | 180 ++++++++
.../parameters.yaml                           |   5 +
tests/avocado/test_nec-usb-xhci.py            |  63 +++
.../test_nec-usb-xhci.py.data/parameters.yaml |   4 +
tests/avocado/test_numa_hotplug.py            | 120 +++++
tests/avocado/test_ovmf_with_240_vcpus.py     |  70 +++
.../parameters.yaml                           |   2 +
tests/avocado/variants.yaml                   |  62 +++
14 files changed, 1187 insertions(+), 5 deletions(-)
create mode 100644 tests/avocado/README.rst
create mode 100644 tests/avocado/avocado_qemu/__init__.py
create mode 100644 tests/avocado/avocado_qemu/test.py
create mode 100644 tests/avocado/parameters.yaml
create mode 100644 tests/avocado/test_info_memdev_host_nodes.py
create mode 100644 tests/avocado/test_linux-boot-console.py
create mode 100644 tests/avocado/test_linux-boot-console.py.data/parameters.yaml
create mode 100644 tests/avocado/test_nec-usb-xhci.py
create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml
create mode 100644 tests/avocado/test_numa_hotplug.py
create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py
create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml
create mode 100644 tests/avocado/variants.yaml
[Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Posted by Philippe Mathieu-Daudé 5 years, 11 months ago
Hi,

While previously working on a Super I/O refactor, I encountered some problems
at runtime, after building the codebase successfully and running qtests.
I had to manually start to boot different guests and check the bootlog.

I wanted to give a try at Avocado which seems designed to simplify that kind
of functional tests.

I applied Amador Pahim work following Cleber Rosa documentation from
http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
however I had to modify few things to parse the boot console.
Since his work is not merged, I included it in this series.

The tests simply expect to find a string reported by Linux printk when a
device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
"registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
device are correctly detected:


    self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
    self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
    self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
    self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)

Example of the tests output:

    $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml 
    JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
    JOB LOG    : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
     (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
     (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
     (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
     (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
     (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
     (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
    RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 13.31 s

Please apologize my ugly Python, this series is just a proof-of-concept :)
I couldn't figure out how to use the @skipUnless(correct arch) decorator.

Regards,

Phil.

Amador Pahim (1):
  AVOCADO_QEMU: Snapshot commit

Philippe Mathieu-Daudé (6):
  avocado: Update python scripts to upstream codebase
  qemu.py: Check console arch is supported before calling mktemp()
  qemu.py: Avoid deprecated tempfile.mktemp()
  avocado: Add an optional flag 'login' to get_console()
  avocado: Add a test parsing Linux kernel booting console
  avocado: Add tests booting the Malta machine

 scripts/qemu.py                               |  50 ++-
 tests/avocado/README.rst                      | 132 ++++++
 tests/avocado/avocado_qemu/__init__.py        |   0
 tests/avocado/avocado_qemu/test.py            | 419 ++++++++++++++++++
 tests/avocado/parameters.yaml                 |  19 +
 tests/avocado/test_info_memdev_host_nodes.py  |  66 +++
 tests/avocado/test_linux-boot-console.py      | 180 ++++++++
 .../parameters.yaml                           |   5 +
 tests/avocado/test_nec-usb-xhci.py            |  63 +++
 .../test_nec-usb-xhci.py.data/parameters.yaml |   4 +
 tests/avocado/test_numa_hotplug.py            | 120 +++++
 tests/avocado/test_ovmf_with_240_vcpus.py     |  70 +++
 .../parameters.yaml                           |   2 +
 tests/avocado/variants.yaml                   |  62 +++
 14 files changed, 1187 insertions(+), 5 deletions(-)
 create mode 100644 tests/avocado/README.rst
 create mode 100644 tests/avocado/avocado_qemu/__init__.py
 create mode 100644 tests/avocado/avocado_qemu/test.py
 create mode 100644 tests/avocado/parameters.yaml
 create mode 100644 tests/avocado/test_info_memdev_host_nodes.py
 create mode 100644 tests/avocado/test_linux-boot-console.py
 create mode 100644 tests/avocado/test_linux-boot-console.py.data/parameters.yaml
 create mode 100644 tests/avocado/test_nec-usb-xhci.py
 create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml
 create mode 100644 tests/avocado/test_numa_hotplug.py
 create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py
 create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml
 create mode 100644 tests/avocado/variants.yaml

-- 
2.17.0


Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Posted by no-reply@patchew.org 5 years, 11 months ago
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180419164642.9536-1-f4bug@amsat.org
Subject: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 t [tag update]            patchew/1524135103-30477-1-git-send-email-frederic.konrad@adacore.com -> patchew/1524135103-30477-1-git-send-email-frederic.konrad@adacore.com
 * [new tag]               patchew/20180419164642.9536-1-f4bug@amsat.org -> patchew/20180419164642.9536-1-f4bug@amsat.org
Switched to a new branch 'test'
137b1f8943 avocado: Add tests booting the Malta machine
ec4a19f56b avocado: Add a test parsing Linux kernel booting console
dc618b369b avocado: Add an optional flag 'login' to get_console()
19a3666713 qemu.py: Avoid deprecated tempfile.mktemp()
99bb4ac92b qemu.py: Check console arch is supported before calling mktemp()
163318b813 avocado: Update python scripts to upstream codebase
621512b86b AVOCADO_QEMU: Snapshot commit

=== OUTPUT BEGIN ===
Checking PATCH 1/7: AVOCADO_QEMU: Snapshot commit...
WARNING: line over 80 characters
#412: FILE: tests/avocado/avocado_qemu/test.py:121:
+    :raise QEMULoginProcessTerminatedError: If the client terminates during login

WARNING: line over 80 characters
#512: FILE: tests/avocado/avocado_qemu/test.py:221:
+            raise QEMULoginProcessTerminatedError(details.status, details.output)

WARNING: line over 80 characters
#789: FILE: tests/avocado/test_info_memdev_host_nodes.py:49:
+        cmd = 'object_add memory-backend-ram,id=mem1,host-nodes=0,size=2G,policy=bind'

WARNING: line over 80 characters
#841: FILE: tests/avocado/test_nec-usb-xhci.py:29:
+        self.vm.args.extend(['-device', 'nec-usb-xhci,id=xhci1,bus=bridge1,addr=0x3'])

ERROR: line over 90 characters
#842: FILE: tests/avocado/test_nec-usb-xhci.py:30:
+        self.vm.args.extend(['-drive', 'file=%s,format=raw,id=drive_usb,if=none' % usbdevice])

ERROR: line over 90 characters
#843: FILE: tests/avocado/test_nec-usb-xhci.py:31:
+        self.vm.args.extend(['-device', 'usb-storage,drive=drive_usb,id=device_usb,bus=xhci1.0'])

ERROR: line over 90 characters
#1039: FILE: tests/avocado/test_ovmf_with_240_vcpus.py:22:
+                                         default='/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd')

WARNING: line over 80 characters
#1041: FILE: tests/avocado/test_ovmf_with_240_vcpus.py:24:
+                                         default='/usr/share/edk2/ovmf/OVMF_VARS.fd')

total: 3 errors, 5 warnings, 1132 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 2/7: avocado: Update python scripts to upstream codebase...
Checking PATCH 3/7: qemu.py: Check console arch is supported before calling mktemp()...
Checking PATCH 4/7: qemu.py: Avoid deprecated tempfile.mktemp()...
Checking PATCH 5/7: avocado: Add an optional flag 'login' to get_console()...
Checking PATCH 6/7: avocado: Add a test parsing Linux kernel booting console...
ERROR: line over 90 characters
#72: FILE: tests/avocado/test_linux-boot-console.py:53:
+        return 'http://archive.debian.org/debian/dists/lenny/main/installer-alpha/current/images/cdrom/vmlinuz'

ERROR: line over 90 characters
#81: FILE: tests/avocado/test_linux-boot-console.py:62:
+        self.vm._args.extend(['-chardev', 'socket,id=srm,server,nowait,path=' + self.console_path])

WARNING: line over 80 characters
#95: FILE: tests/avocado/test_linux-boot-console.py:76:
+        console = self.vm.get_console(console_address=self.console_path, login=False)

ERROR: line over 90 characters
#97: FILE: tests/avocado/test_linux-boot-console.py:78:
+        bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=30.0)[1]

total: 3 errors, 1 warnings, 91 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 7/7: avocado: Add tests booting the Malta machine...
WARNING: line over 80 characters
#30: FILE: tests/avocado/test_linux-boot-console.py:99:
+        return 'http://people.debian.org/~aurel32/qemu/mips/vmlinux-2.6.32-5-4kc-malta'

ERROR: line over 90 characters
#39: FILE: tests/avocado/test_linux-boot-console.py:108:
+        self.vm._args.extend(['-chardev', 'socket,id=uart0,server,nowait,path=' + self.console_path])

WARNING: line over 80 characters
#51: FILE: tests/avocado/test_linux-boot-console.py:120:
+        console = self.vm.get_console(console_address=self.console_path, login=False)

ERROR: line over 90 characters
#53: FILE: tests/avocado/test_linux-boot-console.py:122:
+        bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=6.0)[1]

WARNING: line over 80 characters
#67: FILE: tests/avocado/test_linux-boot-console.py:136:
+# FIXME this is a copy of TestMips4kcMaltaBoot2_6 with a different the kernel url

WARNING: line over 80 characters
#76: FILE: tests/avocado/test_linux-boot-console.py:145:
+        return 'http://people.debian.org/~aurel32/qemu/mips/vmlinux-3.2.0-4-4kc-malta'

ERROR: line over 90 characters
#85: FILE: tests/avocado/test_linux-boot-console.py:154:
+        self.vm._args.extend(['-chardev', 'socket,id=uart0,server,nowait,path=' + self.console_path])

WARNING: line over 80 characters
#97: FILE: tests/avocado/test_linux-boot-console.py:166:
+        console = self.vm.get_console(console_address=self.console_path, login=False)

ERROR: line over 90 characters
#99: FILE: tests/avocado/test_linux-boot-console.py:168:
+        bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=6.0)[1]

total: 4 errors, 5 warnings, 100 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Posted by Cleber Rosa 5 years, 11 months ago

On 04/19/2018 12:46 PM, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> While previously working on a Super I/O refactor, I encountered some problems
> at runtime, after building the codebase successfully and running qtests.
> I had to manually start to boot different guests and check the bootlog.
> 
> I wanted to give a try at Avocado which seems designed to simplify that kind
> of functional tests.
> 
> I applied Amador Pahim work following Cleber Rosa documentation from
> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
> however I had to modify few things to parse the boot console.
> Since his work is not merged, I included it in this series.
> 
> The tests simply expect to find a string reported by Linux printk when a
> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
> device are correctly detected:
> 
> 
>     self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>     self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>     self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>     self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
> 
> Example of the tests output:
> 
>     $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml 
>     JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
>     JOB LOG    : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>      (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
>      (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
>      (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
>      (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
>      (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
>      (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 13.31 s
> 
> Please apologize my ugly Python, this series is just a proof-of-concept :)
> I couldn't figure out how to use the @skipUnless(correct arch) decorator.
> 
> Regards,
> 
> Phil.
> 

Hi Philippe,

Thanks for this interesting RFC/PoC.  As someone who has spent more time
"on the other" side of this code, this perspective is invaluable.  I'll
respond to the major feature utilization points here, and then proceed
to review the individual patches.

One of the distinguishing features of Avocado is the ability to reuse
test code in different scenarios (thus reducing the amount of duplicated
code while still improving the overall coverage).  I noticed you grasped
the overall usage pattern of the "YAML to Mux" plugin, by means of the
"parameter.yaml".

This is an alternative proposal for the "parameter.yaml" file:

--- "parameter.yaml" STARTS HERE
architechture: !mux
   alpha:
      arch: alpha
      machine_type: clipper
      kernel_url:
http://archive.debian.org/debian/dists/lenny/main/installer-alpha/current/images/cdrom/vmlinuz
   mips:
     arch: mips
     machine_type: malta
     check_pci: True
     kernel_versions: !mux
        2_6:
           kernel_url:
http://people.debian.org/~aurel32/qemu/mips/vmlinux-2.6.32-5-4kc-malta
        3_2:
           kernel_url:
http://people.debian.org/~aurel32/qemu/mips/vmlinux-3.2.0-4-4kc-malta
--- "parameter.yaml" ENDS HERE

This file content produces the following tree (run it with "avocado -c
-t -m parameter.yaml"):

Multiplex tree representation:
 ┗━━ run
      ┗━━ architechture
           ╠══ alpha
           ║     → arch: alpha
           ║     → machine_type: clipper
           ║     → kernel_url:
http://archive.debian.org/debian/dists/lenny/main/installer-alpha/current/images/cdrom/vmlinuz
           ╚══ mips
                ┃   → check_pci: True
                ┃   → arch: mips
                ┃   → machine_type: malta
                ┗━━ kernel_versions
                     ╠══ 2_6
                     ║     → kernel_url:
http://people.debian.org/~aurel32/qemu/mips/vmlinux-2.6.32-5-4kc-malta
                     ╚══ 3_2
                           → kernel_url:
http://people.debian.org/~aurel32/qemu/mips/vmlinux-3.2.0-4-4kc-malta


And it generates the following variants (run it with "avocado -m
parameters.yaml"):

Multiplex variants (3):
Variant alpha-deaa:  /run/architechture/alpha
Variant 2_6-6f08:    /run/architechture/mips/kernel_versions/2_6
Variant 3_2-e556:    /run/architechture/mips/kernel_versions/3_2

With these variants, it's possible to have a single code test
implementation.  Here's my alternative implementation of your tests:

--- "test_linux-boot-console.py" STARTS HERE
import os
import tempfile
import gzip

from avocado_qemu import test
from avocado.utils import process


class TestBootCheckDevices(test.QemuTest):
    """
    :avocado: enable
    """
    def get_kernel_uncompressed(self):
        kernel = self.fetch_asset(self.params.get("kernel_url"))
        try:
            process.run("gzip -l %s" % kernel)
        except process.CmdError:
            # only deal with gzip compressed kernels here
            return kernel

        kernel_uncompressed_path = kernel + ".uncompressed"
        with gzip.open(kernel, 'rb') as kernel_gzip:
            with open(kernel_uncompressed_path, 'wb') as
kernel_uncompressed:
                kernel_uncompressed.write(kernel_gzip.read())
        return kernel_uncompressed_path

    def setUp(self):
        kernel_path = self.get_kernel_uncompressed()
        self.console_path = tempfile.mkstemp()[1]
        self.vm._args.extend(['-m', '64'])
        self.vm._args.extend(['-kernel', kernel_path])
        self.vm._args.extend(['-append', '"console=ttyS0 printk.time=0"'])
        self.vm._args.extend(['-chardev',
'socket,id=srm,server,nowait,path=' + self.console_path])
        self.vm._args.extend(['-serial', 'chardev:srm'])
        # This kernel crashes without VGA display
        self.vm._args.extend(['-vga', 'std'])
        self.vm.launch(self.console_path)
        console = self.vm.get_console(console_address=self.console_path,
login=False)
        # no filesystem provided on purpose, wait for the Kernel panic
        self.bootlog = console.read_until_any_line_matches(["Kernel
panic - not syncing: VFS: Unable to mount root fs"], timeout=30.0)[1]
        console.close()

    def test(self):
        # check Super I/O
        self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A',
self.bootlog)
        self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A',
self.bootlog)
        self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', self.bootlog)
        self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', self.bootlog)
        # check PCI
        if self.params.get('check_pci', default=False):
            self.assertIn(u'registered as PCnet/PCI II 79C970A',
self.bootlog)

    def tearDown(self):
        self.vm.shutdown()
        os.remove(self.console_path)
--- "test_linux-boot-console.py" ENDS HERE

A few points about this code:

1) get_kernel_uncompressed() is a prime candidate for a utility
function, even at the "avocado.utils.archive" level (a better version,
of course).  It's one of those things that arise from writing more
diverse code (again, thanks for this experiment).

2) setUp() still contains a lot of boiler plate code, and can certainly
have some of that moved to "avocado_qemu".  For instance, the
"avocado_qemu" code already knows how to deal with some parameters, such
as "arch" and "machine_type", so "memory", "kernel", etc could be
handled in a similar fashion.

3) It wasn't clear to me if manually creating a serial device and using
it as a console was a hard requirement of the test.  The
QEMUMachine._create_console() method on the "scripts/qemu.py" file
already has some code to create many console types.

Finally, running this code + parameters (by using "avocado run -m
test_linux-boot-console.py.data/parameters.yaml --
test_linux-boot-console.py") results in:

JOB ID     : bb7585de42035a062ddda539801743267aa4431a
JOB LOG    :
/home/cleber/avocado/job-results/job-2018-04-30T18.46-bb7585d/job.log
 (1/3) test_linux-boot-console.py:TestBootCheckDevices.test;alpha-deaa:
PASS (3.04 s)
 (2/3) test_linux-boot-console.py:TestBootCheckDevices.test;2_6-6f08:
PASS (2.83 s)
 (3/3) test_linux-boot-console.py:TestBootCheckDevices.test;3_2-e556:
PASS (2.86 s)
RESULTS    : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME   : 9.86 s
JOB HTML   :
/home/cleber/avocado/job-results/job-2018-04-30T18.46-bb7585d/results.html

The resulting "job.log" file can be seen at:

https://paste.fedoraproject.org/paste/G4X-z5PcmMdmmIVBDKVWVg/raw

Now, unto individual commits.
- Cleber.

> Amador Pahim (1):
>   AVOCADO_QEMU: Snapshot commit
> 
> Philippe Mathieu-Daudé (6):
>   avocado: Update python scripts to upstream codebase
>   qemu.py: Check console arch is supported before calling mktemp()
>   qemu.py: Avoid deprecated tempfile.mktemp()
>   avocado: Add an optional flag 'login' to get_console()
>   avocado: Add a test parsing Linux kernel booting console
>   avocado: Add tests booting the Malta machine
> 
>  scripts/qemu.py                               |  50 ++-
>  tests/avocado/README.rst                      | 132 ++++++
>  tests/avocado/avocado_qemu/__init__.py        |   0
>  tests/avocado/avocado_qemu/test.py            | 419 ++++++++++++++++++
>  tests/avocado/parameters.yaml                 |  19 +
>  tests/avocado/test_info_memdev_host_nodes.py  |  66 +++
>  tests/avocado/test_linux-boot-console.py      | 180 ++++++++
>  .../parameters.yaml                           |   5 +
>  tests/avocado/test_nec-usb-xhci.py            |  63 +++
>  .../test_nec-usb-xhci.py.data/parameters.yaml |   4 +
>  tests/avocado/test_numa_hotplug.py            | 120 +++++
>  tests/avocado/test_ovmf_with_240_vcpus.py     |  70 +++
>  .../parameters.yaml                           |   2 +
>  tests/avocado/variants.yaml                   |  62 +++
>  14 files changed, 1187 insertions(+), 5 deletions(-)
>  create mode 100644 tests/avocado/README.rst
>  create mode 100644 tests/avocado/avocado_qemu/__init__.py
>  create mode 100644 tests/avocado/avocado_qemu/test.py
>  create mode 100644 tests/avocado/parameters.yaml
>  create mode 100644 tests/avocado/test_info_memdev_host_nodes.py
>  create mode 100644 tests/avocado/test_linux-boot-console.py
>  create mode 100644 tests/avocado/test_linux-boot-console.py.data/parameters.yaml
>  create mode 100644 tests/avocado/test_nec-usb-xhci.py
>  create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml
>  create mode 100644 tests/avocado/test_numa_hotplug.py
>  create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py
>  create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml
>  create mode 100644 tests/avocado/variants.yaml
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Posted by Philippe Mathieu-Daudé 5 years, 11 months ago
Cross-posting qemu-devel + avocado-devel.

> While previously working on a Super I/O refactor, I encountered some problems
> at runtime, after building the codebase successfully and running qtests.
> I had to manually start to boot different guests and check the bootlog.
> 
> I wanted to give a try at Avocado which seems designed to simplify that kind
> of functional tests.
> 
> I applied Amador Pahim work following Cleber Rosa documentation from
> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
> however I had to modify few things to parse the boot console.
> Since his work is not merged, I included it in this series.
> 
> The tests simply expect to find a string reported by Linux printk when a
> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
> device are correctly detected:
> 
> 
>     self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>     self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>     self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>     self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
> 
> Example of the tests output:
> 
>     $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml 
>     JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
>     JOB LOG    : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>      (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
>      (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
>      (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
>      (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
>      (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
>      (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 13.31 s
> 
> Please apologize my ugly Python, this series is just a proof-of-concept :)
> I couldn't figure out how to use the @skipUnless(correct arch) decorator.

Eduardo asked me to share my first impressions after trying this
framework with QEMU.

So far it seems more designed to run tests _inside_ a qemu instance
(think user-space), eventually testing packages/scripts.

My tests are targeting the machine model itself, if the devices are
correctly instantiated and so.
The fastest approach was to check the Linux kernel bootlog, but if you
look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
executed before the kernel. Casually this firmware also send information
on the console, but what if the console is not accessible?
We can use a chardev for the ioport80 POST, but some firmwares post boot
events via I2C, SPI, CAN...

To improve the testing, my idea is to use the Tracing framework.
The machine trace events would be logged in a db and avogado would
verify some of the trace events, did they occur? in the correct sequence
order? with the correct arguments? We can also check relative
timestamps, is this sequence timing fast enough?

I plan to use the gdb API to insert breakpoints and follow part
execution flow, eventually injecting (faulty) events.

The multi-arch/machines support is a bit weak yet.

I'm personally more interested in automatically testing [real world]
firmware or full machine boot process, the kind of integration testing I
can not do with qtests.

I liked the possibility to generate coredumps, or the replay function.

I also liked to be able to write a test on how a machine boots, in ~20
LOC (see TestAlphaClipperBoot2_6).

The storage API is OK to fetch a full VM image, but not a single file
like a kernel or a flash image.

Enough for my second try, good work :)

I hope this was helpful.

Regards,

Phil.

Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Posted by Philippe Mathieu-Daudé 5 years, 11 months ago
> Cross-posting qemu-devel + avocado-devel.
> 
>> While previously working on a Super I/O refactor, I encountered some problems
>> at runtime, after building the codebase successfully and running qtests.
>> I had to manually start to boot different guests and check the bootlog.
>>
>> I wanted to give a try at Avocado which seems designed to simplify that kind
>> of functional tests.
>>
>> I applied Amador Pahim work following Cleber Rosa documentation from
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
>> however I had to modify few things to parse the boot console.
>> Since his work is not merged, I included it in this series.
>>
>> The tests simply expect to find a string reported by Linux printk when a
>> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
>> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
>> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
>> device are correctly detected:
>>
>>
>>     self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>>     self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>>     self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>>     self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
>>
>> Example of the tests output:
>>
>>     $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml 
>>     JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
>>     JOB LOG    : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>>      (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
>>      (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
>>      (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
>>      (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
>>      (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
>>      (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
>>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>     JOB TIME   : 13.31 s
>>
>> Please apologize my ugly Python, this series is just a proof-of-concept :)
>> I couldn't figure out how to use the @skipUnless(correct arch) decorator.
> 
> Eduardo asked me to share my first impressions after trying this
> framework with QEMU.
> 
> So far it seems more designed to run tests _inside_ a qemu instance
> (think user-space), eventually testing packages/scripts.

Current examples use the same pattern, which I felt a bit restricting:

[QemuTest]
  .setUp: prepare VM arguments, launch the VM

  .test_*: run tests in the running VM

  .tearDown: shutdown the VM and cleanup dangling files

> My tests are targeting the machine model itself, if the devices are
> correctly instantiated and so.
> The fastest approach was to check the Linux kernel bootlog, but if you
> look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
> executed before the kernel. Casually this firmware also send information
> on the console, but what if the console is not accessible?
> We can use a chardev for the ioport80 POST, but some firmwares post boot
> events via I2C, SPI, CAN...
> 
> To improve the testing, my idea is to use the Tracing framework.
> The machine trace events would be logged in a db and avogado would
> verify some of the trace events, did they occur? in the correct sequence
> order? with the correct arguments? We can also check relative
> timestamps, is this sequence timing fast enough?
> 
> I plan to use the gdb API to insert breakpoints and follow part
> execution flow, eventually injecting (faulty) events.
> 
> The multi-arch/machines support is a bit weak yet.
> 
> I'm personally more interested in automatically testing [real world]
> firmware or full machine boot process, the kind of integration testing I
> can not do with qtests.
> 
> I liked the possibility to generate coredumps, or the replay function.
> 
> I also liked to be able to write a test on how a machine boots, in ~20
> LOC (see TestAlphaClipperBoot2_6).
> 
> The storage API is OK to fetch a full VM image, but not a single file
> like a kernel or a flash image

Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Posted by Cleber Rosa 5 years, 11 months ago

On 04/20/2018 12:17 AM, Philippe Mathieu-Daudé wrote:
> Cross-posting qemu-devel + avocado-devel.
> 
>> While previously working on a Super I/O refactor, I encountered some problems
>> at runtime, after building the codebase successfully and running qtests.
>> I had to manually start to boot different guests and check the bootlog.
>>
>> I wanted to give a try at Avocado which seems designed to simplify that kind
>> of functional tests.
>>
>> I applied Amador Pahim work following Cleber Rosa documentation from
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
>> however I had to modify few things to parse the boot console.
>> Since his work is not merged, I included it in this series.
>>
>> The tests simply expect to find a string reported by Linux printk when a
>> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
>> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
>> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
>> device are correctly detected:
>>
>>
>>     self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>>     self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>>     self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>>     self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
>>
>> Example of the tests output:
>>
>>     $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml 
>>     JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
>>     JOB LOG    : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>>      (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
>>      (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
>>      (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
>>      (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
>>      (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
>>      (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
>>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>     JOB TIME   : 13.31 s
>>
>> Please apologize my ugly Python, this series is just a proof-of-concept :)
>> I couldn't figure out how to use the @skipUnless(correct arch) decorator.
> 
> Eduardo asked me to share my first impressions after trying this
> framework with QEMU.
> 
> So far it seems more designed to run tests _inside_ a qemu instance
> (think user-space), eventually testing packages/scripts.
> 

The first tests written indeed involved interacting with the guest OS
more often than not.  I tried to show one example that did not envolve a
guest here:

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03468.html

Just as short exercise, this is another example (sorry for the lack of
creativity):

class CpuSmp(test.QemuTest):
    """
    :avocado: enable
    """
    def test(self):
        smp_max = int(self.params.get('smp_max', default=255))
        self.vm._args.extend(['-smp', '%s' % smp_max])
        self.vm.launch()
        res = self.vm.qmp('query-cpus-fast')
        self.assertEqual(len(res[u'return']), smp_max)

> My tests are targeting the machine model itself, if the devices are
> correctly instantiated and so.
> The fastest approach was to check the Linux kernel bootlog, but if you
> look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
> executed before the kernel. Casually this firmware also send information
> on the console, but what if the console is not accessible?
> We can use a chardev for the ioport80 POST, but some firmwares post boot
> events via I2C, SPI, CAN...
> 
> To improve the testing, my idea is to use the Tracing framework.
> The machine trace events would be logged in a db and avogado would
> verify some of the trace events, did they occur? in the correct sequence
> order? with the correct arguments? We can also check relative
> timestamps, is this sequence timing fast enough?
> 

Right, I think I see where you're going here.

For the record, the Avocado test runner has the ability to check the
test output against a previously recorded file, and that can determine
the test status.  Maybe a second implementation of the same concept can
be applied to your idea.

> I plan to use the gdb API to insert breakpoints and follow part
> execution flow, eventually injecting (faulty) events.
> 

Just to be clear, do you mean the Avocado GDB API?

> The multi-arch/machines support is a bit weak yet.
> 

Yes.  One of the ways I believe the multi-arch/machines support can be
significantly improved is by using some level of parameter
standardization, together with tools to generate the intended test
"matrix".  On a quick experiment with "pict" (an indepedent
combinatorial testing tool) on this matter, I came up with:

---

arch: x86_64, ppc64, ppcemb, ppc, aarch64, alpha, s390x
machine_type: pc, q35, pseries-2.12, virt, 40p, bamboo, g3beige, mac99,
mpc8544ds, ppce500, prep, sam460ex, clipper, s390-ccw-virtio-2.12
machine_accel: kvm, tcg

smp_max: 1, 4, 8, 15, 248, 255

IF [arch] = "x86_64" THEN
   [machine_type] = "q35" OR
   [machine_type] = "pc";

IF [arch] = "ppc" THEN
   [machine_type] = "40p" OR
   [machine_type] = "bamboo" OR
   [machine_type] = "g3beige" OR
   [machine_type] = "mac99" OR
   [machine_type] = "mpc8544ds" OR
   [machine_type] = "ppce500" OR
   [machine_type] = "prep" OR
   [machine_type] = "sam460ex";

IF [arch] = "alpha" THEN [machine_type] = "clipper";
IF [arch] = "ppc64" THEN [machine_type] = "pseries-2.12";
IF [arch] = "ppcemb" THEN [machine_type] = "bamboo";
IF [arch] = "aarch64" THEN [machine_type] = "virt";
IF [arch] = "s390x" THEN [machine_type] = "s390-ccw-virtio-2.12";

# On my machine, only x86_64 can do KVM
IF NOT [arch] = "x86_64" THEN [machine_accel] = "tcg";

IF [machine_type] = "q35" THEN [smp_max] = 255;
IF [machine_type] = "pc" THEN [smp_max] = 255;
IF [machine_type] = "s390-ccw-virtio-2.12" THEN [smp_max] = 248;
IF [machine_type] = "virt" THEN [smp_max] = 8;
IF [machine_type] = "mpc8544ds" THEN [smp_max] = 15;
IF [machine_type] = "clipper" THEN [smp_max] = 4;
IF [machine_type] = "40p" THEN [smp_max] = 1;
IF [machine_type] = "bamboo" THEN [smp_max] = 1;
IF [machine_type] = "g3beige" THEN [smp_max] = 1;
IF [machine_type] = "mac99" THEN [smp_max] = 1;
IF [machine_type] = "sam460ex" THEN [smp_max] = 1;
IF [machine_type] = "ppce500" THEN [smp_max] = 1;
IF [machine_type] = "prep" THEN [smp_max] = 1;

---

With this *very partial* definition of the arch/machine support in QEMU,
a single test can be run in 22 different scenarios.  I've used it to run
the 4 VNC tests and the one CpuSmp test printed above, resulting in 110
different test executions:

   https://cleber.fedorapeople.org/results.html

The first three parameters are already supported by avocado_qemu, and
more can follow.

And to be fully honest, "avocado_qemu" has more weaknesses than
strengthens at the moment.  That's partially because it's so young (and
immature), but, can be made into our ideal grown "being".

> I'm personally more interested in automatically testing [real world]
> firmware or full machine boot process, the kind of integration testing I
> can not do with qtests.
> 
> I liked the possibility to generate coredumps, or the replay function.
> 
> I also liked to be able to write a test on how a machine boots, in ~20
> LOC (see TestAlphaClipperBoot2_6).
> 
> The storage API is OK to fetch a full VM image, but not a single file
> like a kernel or a flash image.
> 

As pointed elsewhere, fetching/caching is probably OK at this point.
But handling the (optional) gzip compression can use some TLC.

> Enough for my second try, good work :)
> 
> I hope this was helpful.
> 
> Regards,
> 
> Phil.
> 

Thanks for the review!
- Cleber.