scripts/qemu.py | 50 ++- tests/avocado/README.rst | 132 ++++++ tests/avocado/avocado_qemu/__init__.py | 0 tests/avocado/avocado_qemu/test.py | 419 ++++++++++++++++++ tests/avocado/parameters.yaml | 19 + tests/avocado/test_info_memdev_host_nodes.py | 66 +++ tests/avocado/test_linux-boot-console.py | 180 ++++++++ .../parameters.yaml | 5 + tests/avocado/test_nec-usb-xhci.py | 63 +++ .../test_nec-usb-xhci.py.data/parameters.yaml | 4 + tests/avocado/test_numa_hotplug.py | 120 +++++ tests/avocado/test_ovmf_with_240_vcpus.py | 70 +++ .../parameters.yaml | 2 + tests/avocado/variants.yaml | 62 +++ 14 files changed, 1187 insertions(+), 5 deletions(-) create mode 100644 tests/avocado/README.rst create mode 100644 tests/avocado/avocado_qemu/__init__.py create mode 100644 tests/avocado/avocado_qemu/test.py create mode 100644 tests/avocado/parameters.yaml create mode 100644 tests/avocado/test_info_memdev_host_nodes.py create mode 100644 tests/avocado/test_linux-boot-console.py create mode 100644 tests/avocado/test_linux-boot-console.py.data/parameters.yaml create mode 100644 tests/avocado/test_nec-usb-xhci.py create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml create mode 100644 tests/avocado/test_numa_hotplug.py create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml create mode 100644 tests/avocado/variants.yaml
Hi, While previously working on a Super I/O refactor, I encountered some problems at runtime, after building the codebase successfully and running qtests. I had to manually start to boot different guests and check the bootlog. I wanted to give a try at Avocado which seems designed to simplify that kind of functional tests. I applied Amador Pahim work following Cleber Rosa documentation from http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html, however I had to modify few things to parse the boot console. Since his work is not merged, I included it in this series. The tests simply expect to find a string reported by Linux printk when a device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A" and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network device are correctly detected: self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog) self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog) self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog) self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog) Example of the tests output: $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910 JOB LOG : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s) (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s) (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s) (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s) (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s) (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s) RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 13.31 s Please apologize my ugly Python, this series is just a proof-of-concept :) I couldn't figure out how to use the @skipUnless(correct arch) decorator. Regards, Phil. Amador Pahim (1): AVOCADO_QEMU: Snapshot commit Philippe Mathieu-Daudé (6): avocado: Update python scripts to upstream codebase qemu.py: Check console arch is supported before calling mktemp() qemu.py: Avoid deprecated tempfile.mktemp() avocado: Add an optional flag 'login' to get_console() avocado: Add a test parsing Linux kernel booting console avocado: Add tests booting the Malta machine scripts/qemu.py | 50 ++- tests/avocado/README.rst | 132 ++++++ tests/avocado/avocado_qemu/__init__.py | 0 tests/avocado/avocado_qemu/test.py | 419 ++++++++++++++++++ tests/avocado/parameters.yaml | 19 + tests/avocado/test_info_memdev_host_nodes.py | 66 +++ tests/avocado/test_linux-boot-console.py | 180 ++++++++ .../parameters.yaml | 5 + tests/avocado/test_nec-usb-xhci.py | 63 +++ .../test_nec-usb-xhci.py.data/parameters.yaml | 4 + tests/avocado/test_numa_hotplug.py | 120 +++++ tests/avocado/test_ovmf_with_240_vcpus.py | 70 +++ .../parameters.yaml | 2 + tests/avocado/variants.yaml | 62 +++ 14 files changed, 1187 insertions(+), 5 deletions(-) create mode 100644 tests/avocado/README.rst create mode 100644 tests/avocado/avocado_qemu/__init__.py create mode 100644 tests/avocado/avocado_qemu/test.py create mode 100644 tests/avocado/parameters.yaml create mode 100644 tests/avocado/test_info_memdev_host_nodes.py create mode 100644 tests/avocado/test_linux-boot-console.py create mode 100644 tests/avocado/test_linux-boot-console.py.data/parameters.yaml create mode 100644 tests/avocado/test_nec-usb-xhci.py create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml create mode 100644 tests/avocado/test_numa_hotplug.py create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml create mode 100644 tests/avocado/variants.yaml -- 2.17.0
Hi, This series seems to have some coding style problems. See output below for more information: Type: series Message-id: 20180419164642.9536-1-f4bug@amsat.org Subject: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console === TEST SCRIPT BEGIN === #!/bin/bash BASE=base n=1 total=$(git log --oneline $BASE.. | wc -l) failed=0 git config --local diff.renamelimit 0 git config --local diff.renames True git config --local diff.algorithm histogram commits="$(git log --format=%H --reverse $BASE..)" for c in $commits; do echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..." if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then failed=1 echo fi n=$((n+1)) done exit $failed === TEST SCRIPT END === Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 From https://github.com/patchew-project/qemu t [tag update] patchew/1524135103-30477-1-git-send-email-frederic.konrad@adacore.com -> patchew/1524135103-30477-1-git-send-email-frederic.konrad@adacore.com * [new tag] patchew/20180419164642.9536-1-f4bug@amsat.org -> patchew/20180419164642.9536-1-f4bug@amsat.org Switched to a new branch 'test' 137b1f8943 avocado: Add tests booting the Malta machine ec4a19f56b avocado: Add a test parsing Linux kernel booting console dc618b369b avocado: Add an optional flag 'login' to get_console() 19a3666713 qemu.py: Avoid deprecated tempfile.mktemp() 99bb4ac92b qemu.py: Check console arch is supported before calling mktemp() 163318b813 avocado: Update python scripts to upstream codebase 621512b86b AVOCADO_QEMU: Snapshot commit === OUTPUT BEGIN === Checking PATCH 1/7: AVOCADO_QEMU: Snapshot commit... WARNING: line over 80 characters #412: FILE: tests/avocado/avocado_qemu/test.py:121: + :raise QEMULoginProcessTerminatedError: If the client terminates during login WARNING: line over 80 characters #512: FILE: tests/avocado/avocado_qemu/test.py:221: + raise QEMULoginProcessTerminatedError(details.status, details.output) WARNING: line over 80 characters #789: FILE: tests/avocado/test_info_memdev_host_nodes.py:49: + cmd = 'object_add memory-backend-ram,id=mem1,host-nodes=0,size=2G,policy=bind' WARNING: line over 80 characters #841: FILE: tests/avocado/test_nec-usb-xhci.py:29: + self.vm.args.extend(['-device', 'nec-usb-xhci,id=xhci1,bus=bridge1,addr=0x3']) ERROR: line over 90 characters #842: FILE: tests/avocado/test_nec-usb-xhci.py:30: + self.vm.args.extend(['-drive', 'file=%s,format=raw,id=drive_usb,if=none' % usbdevice]) ERROR: line over 90 characters #843: FILE: tests/avocado/test_nec-usb-xhci.py:31: + self.vm.args.extend(['-device', 'usb-storage,drive=drive_usb,id=device_usb,bus=xhci1.0']) ERROR: line over 90 characters #1039: FILE: tests/avocado/test_ovmf_with_240_vcpus.py:22: + default='/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd') WARNING: line over 80 characters #1041: FILE: tests/avocado/test_ovmf_with_240_vcpus.py:24: + default='/usr/share/edk2/ovmf/OVMF_VARS.fd') total: 3 errors, 5 warnings, 1132 lines checked Your patch has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS. Checking PATCH 2/7: avocado: Update python scripts to upstream codebase... Checking PATCH 3/7: qemu.py: Check console arch is supported before calling mktemp()... Checking PATCH 4/7: qemu.py: Avoid deprecated tempfile.mktemp()... Checking PATCH 5/7: avocado: Add an optional flag 'login' to get_console()... Checking PATCH 6/7: avocado: Add a test parsing Linux kernel booting console... ERROR: line over 90 characters #72: FILE: tests/avocado/test_linux-boot-console.py:53: + return 'http://archive.debian.org/debian/dists/lenny/main/installer-alpha/current/images/cdrom/vmlinuz' ERROR: line over 90 characters #81: FILE: tests/avocado/test_linux-boot-console.py:62: + self.vm._args.extend(['-chardev', 'socket,id=srm,server,nowait,path=' + self.console_path]) WARNING: line over 80 characters #95: FILE: tests/avocado/test_linux-boot-console.py:76: + console = self.vm.get_console(console_address=self.console_path, login=False) ERROR: line over 90 characters #97: FILE: tests/avocado/test_linux-boot-console.py:78: + bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=30.0)[1] total: 3 errors, 1 warnings, 91 lines checked Your patch has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS. Checking PATCH 7/7: avocado: Add tests booting the Malta machine... WARNING: line over 80 characters #30: FILE: tests/avocado/test_linux-boot-console.py:99: + return 'http://people.debian.org/~aurel32/qemu/mips/vmlinux-2.6.32-5-4kc-malta' ERROR: line over 90 characters #39: FILE: tests/avocado/test_linux-boot-console.py:108: + self.vm._args.extend(['-chardev', 'socket,id=uart0,server,nowait,path=' + self.console_path]) WARNING: line over 80 characters #51: FILE: tests/avocado/test_linux-boot-console.py:120: + console = self.vm.get_console(console_address=self.console_path, login=False) ERROR: line over 90 characters #53: FILE: tests/avocado/test_linux-boot-console.py:122: + bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=6.0)[1] WARNING: line over 80 characters #67: FILE: tests/avocado/test_linux-boot-console.py:136: +# FIXME this is a copy of TestMips4kcMaltaBoot2_6 with a different the kernel url WARNING: line over 80 characters #76: FILE: tests/avocado/test_linux-boot-console.py:145: + return 'http://people.debian.org/~aurel32/qemu/mips/vmlinux-3.2.0-4-4kc-malta' ERROR: line over 90 characters #85: FILE: tests/avocado/test_linux-boot-console.py:154: + self.vm._args.extend(['-chardev', 'socket,id=uart0,server,nowait,path=' + self.console_path]) WARNING: line over 80 characters #97: FILE: tests/avocado/test_linux-boot-console.py:166: + console = self.vm.get_console(console_address=self.console_path, login=False) ERROR: line over 90 characters #99: FILE: tests/avocado/test_linux-boot-console.py:168: + bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=6.0)[1] total: 4 errors, 5 warnings, 100 lines checked Your patch has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS. === OUTPUT END === Test command exited with code: 1 --- Email generated automatically by Patchew [http://patchew.org/]. Please send your feedback to patchew-devel@redhat.com
On 04/19/2018 12:46 PM, Philippe Mathieu-Daudé wrote: > Hi, > > While previously working on a Super I/O refactor, I encountered some problems > at runtime, after building the codebase successfully and running qtests. > I had to manually start to boot different guests and check the bootlog. > > I wanted to give a try at Avocado which seems designed to simplify that kind > of functional tests. > > I applied Amador Pahim work following Cleber Rosa documentation from > http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html, > however I had to modify few things to parse the boot console. > Since his work is not merged, I included it in this series. > > The tests simply expect to find a string reported by Linux printk when a > device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A" > and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such > "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network > device are correctly detected: > > > self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog) > self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog) > self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog) > self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog) > > Example of the tests output: > > $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml > JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910 > JOB LOG : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log > (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s) > (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s) > (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s) > (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s) > (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s) > (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s) > RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 13.31 s > > Please apologize my ugly Python, this series is just a proof-of-concept :) > I couldn't figure out how to use the @skipUnless(correct arch) decorator. > > Regards, > > Phil. > Hi Philippe, Thanks for this interesting RFC/PoC. As someone who has spent more time "on the other" side of this code, this perspective is invaluable. I'll respond to the major feature utilization points here, and then proceed to review the individual patches. One of the distinguishing features of Avocado is the ability to reuse test code in different scenarios (thus reducing the amount of duplicated code while still improving the overall coverage). I noticed you grasped the overall usage pattern of the "YAML to Mux" plugin, by means of the "parameter.yaml". This is an alternative proposal for the "parameter.yaml" file: --- "parameter.yaml" STARTS HERE architechture: !mux alpha: arch: alpha machine_type: clipper kernel_url: http://archive.debian.org/debian/dists/lenny/main/installer-alpha/current/images/cdrom/vmlinuz mips: arch: mips machine_type: malta check_pci: True kernel_versions: !mux 2_6: kernel_url: http://people.debian.org/~aurel32/qemu/mips/vmlinux-2.6.32-5-4kc-malta 3_2: kernel_url: http://people.debian.org/~aurel32/qemu/mips/vmlinux-3.2.0-4-4kc-malta --- "parameter.yaml" ENDS HERE This file content produces the following tree (run it with "avocado -c -t -m parameter.yaml"): Multiplex tree representation: ┗━━ run ┗━━ architechture ╠══ alpha ║ → arch: alpha ║ → machine_type: clipper ║ → kernel_url: http://archive.debian.org/debian/dists/lenny/main/installer-alpha/current/images/cdrom/vmlinuz ╚══ mips ┃ → check_pci: True ┃ → arch: mips ┃ → machine_type: malta ┗━━ kernel_versions ╠══ 2_6 ║ → kernel_url: http://people.debian.org/~aurel32/qemu/mips/vmlinux-2.6.32-5-4kc-malta ╚══ 3_2 → kernel_url: http://people.debian.org/~aurel32/qemu/mips/vmlinux-3.2.0-4-4kc-malta And it generates the following variants (run it with "avocado -m parameters.yaml"): Multiplex variants (3): Variant alpha-deaa: /run/architechture/alpha Variant 2_6-6f08: /run/architechture/mips/kernel_versions/2_6 Variant 3_2-e556: /run/architechture/mips/kernel_versions/3_2 With these variants, it's possible to have a single code test implementation. Here's my alternative implementation of your tests: --- "test_linux-boot-console.py" STARTS HERE import os import tempfile import gzip from avocado_qemu import test from avocado.utils import process class TestBootCheckDevices(test.QemuTest): """ :avocado: enable """ def get_kernel_uncompressed(self): kernel = self.fetch_asset(self.params.get("kernel_url")) try: process.run("gzip -l %s" % kernel) except process.CmdError: # only deal with gzip compressed kernels here return kernel kernel_uncompressed_path = kernel + ".uncompressed" with gzip.open(kernel, 'rb') as kernel_gzip: with open(kernel_uncompressed_path, 'wb') as kernel_uncompressed: kernel_uncompressed.write(kernel_gzip.read()) return kernel_uncompressed_path def setUp(self): kernel_path = self.get_kernel_uncompressed() self.console_path = tempfile.mkstemp()[1] self.vm._args.extend(['-m', '64']) self.vm._args.extend(['-kernel', kernel_path]) self.vm._args.extend(['-append', '"console=ttyS0 printk.time=0"']) self.vm._args.extend(['-chardev', 'socket,id=srm,server,nowait,path=' + self.console_path]) self.vm._args.extend(['-serial', 'chardev:srm']) # This kernel crashes without VGA display self.vm._args.extend(['-vga', 'std']) self.vm.launch(self.console_path) console = self.vm.get_console(console_address=self.console_path, login=False) # no filesystem provided on purpose, wait for the Kernel panic self.bootlog = console.read_until_any_line_matches(["Kernel panic - not syncing: VFS: Unable to mount root fs"], timeout=30.0)[1] console.close() def test(self): # check Super I/O self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', self.bootlog) self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', self.bootlog) self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', self.bootlog) self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', self.bootlog) # check PCI if self.params.get('check_pci', default=False): self.assertIn(u'registered as PCnet/PCI II 79C970A', self.bootlog) def tearDown(self): self.vm.shutdown() os.remove(self.console_path) --- "test_linux-boot-console.py" ENDS HERE A few points about this code: 1) get_kernel_uncompressed() is a prime candidate for a utility function, even at the "avocado.utils.archive" level (a better version, of course). It's one of those things that arise from writing more diverse code (again, thanks for this experiment). 2) setUp() still contains a lot of boiler plate code, and can certainly have some of that moved to "avocado_qemu". For instance, the "avocado_qemu" code already knows how to deal with some parameters, such as "arch" and "machine_type", so "memory", "kernel", etc could be handled in a similar fashion. 3) It wasn't clear to me if manually creating a serial device and using it as a console was a hard requirement of the test. The QEMUMachine._create_console() method on the "scripts/qemu.py" file already has some code to create many console types. Finally, running this code + parameters (by using "avocado run -m test_linux-boot-console.py.data/parameters.yaml -- test_linux-boot-console.py") results in: JOB ID : bb7585de42035a062ddda539801743267aa4431a JOB LOG : /home/cleber/avocado/job-results/job-2018-04-30T18.46-bb7585d/job.log (1/3) test_linux-boot-console.py:TestBootCheckDevices.test;alpha-deaa: PASS (3.04 s) (2/3) test_linux-boot-console.py:TestBootCheckDevices.test;2_6-6f08: PASS (2.83 s) (3/3) test_linux-boot-console.py:TestBootCheckDevices.test;3_2-e556: PASS (2.86 s) RESULTS : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 9.86 s JOB HTML : /home/cleber/avocado/job-results/job-2018-04-30T18.46-bb7585d/results.html The resulting "job.log" file can be seen at: https://paste.fedoraproject.org/paste/G4X-z5PcmMdmmIVBDKVWVg/raw Now, unto individual commits. - Cleber. > Amador Pahim (1): > AVOCADO_QEMU: Snapshot commit > > Philippe Mathieu-Daudé (6): > avocado: Update python scripts to upstream codebase > qemu.py: Check console arch is supported before calling mktemp() > qemu.py: Avoid deprecated tempfile.mktemp() > avocado: Add an optional flag 'login' to get_console() > avocado: Add a test parsing Linux kernel booting console > avocado: Add tests booting the Malta machine > > scripts/qemu.py | 50 ++- > tests/avocado/README.rst | 132 ++++++ > tests/avocado/avocado_qemu/__init__.py | 0 > tests/avocado/avocado_qemu/test.py | 419 ++++++++++++++++++ > tests/avocado/parameters.yaml | 19 + > tests/avocado/test_info_memdev_host_nodes.py | 66 +++ > tests/avocado/test_linux-boot-console.py | 180 ++++++++ > .../parameters.yaml | 5 + > tests/avocado/test_nec-usb-xhci.py | 63 +++ > .../test_nec-usb-xhci.py.data/parameters.yaml | 4 + > tests/avocado/test_numa_hotplug.py | 120 +++++ > tests/avocado/test_ovmf_with_240_vcpus.py | 70 +++ > .../parameters.yaml | 2 + > tests/avocado/variants.yaml | 62 +++ > 14 files changed, 1187 insertions(+), 5 deletions(-) > create mode 100644 tests/avocado/README.rst > create mode 100644 tests/avocado/avocado_qemu/__init__.py > create mode 100644 tests/avocado/avocado_qemu/test.py > create mode 100644 tests/avocado/parameters.yaml > create mode 100644 tests/avocado/test_info_memdev_host_nodes.py > create mode 100644 tests/avocado/test_linux-boot-console.py > create mode 100644 tests/avocado/test_linux-boot-console.py.data/parameters.yaml > create mode 100644 tests/avocado/test_nec-usb-xhci.py > create mode 100644 tests/avocado/test_nec-usb-xhci.py.data/parameters.yaml > create mode 100644 tests/avocado/test_numa_hotplug.py > create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py > create mode 100644 tests/avocado/test_ovmf_with_240_vcpus.py.data/parameters.yaml > create mode 100644 tests/avocado/variants.yaml > -- Cleber Rosa [ Sr Software Engineer - Virtualization Team - Red Hat ] [ Avocado Test Framework - avocado-framework.github.io ] [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
Cross-posting qemu-devel + avocado-devel. > While previously working on a Super I/O refactor, I encountered some problems > at runtime, after building the codebase successfully and running qtests. > I had to manually start to boot different guests and check the bootlog. > > I wanted to give a try at Avocado which seems designed to simplify that kind > of functional tests. > > I applied Amador Pahim work following Cleber Rosa documentation from > http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html, > however I had to modify few things to parse the boot console. > Since his work is not merged, I included it in this series. > > The tests simply expect to find a string reported by Linux printk when a > device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A" > and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such > "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network > device are correctly detected: > > > self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog) > self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog) > self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog) > self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog) > > Example of the tests output: > > $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml > JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910 > JOB LOG : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log > (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s) > (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s) > (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s) > (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s) > (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s) > (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s) > RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 13.31 s > > Please apologize my ugly Python, this series is just a proof-of-concept :) > I couldn't figure out how to use the @skipUnless(correct arch) decorator. Eduardo asked me to share my first impressions after trying this framework with QEMU. So far it seems more designed to run tests _inside_ a qemu instance (think user-space), eventually testing packages/scripts. My tests are targeting the machine model itself, if the devices are correctly instantiated and so. The fastest approach was to check the Linux kernel bootlog, but if you look at the TestAlphaClipperBoot console you'll see a boot "firmware" is executed before the kernel. Casually this firmware also send information on the console, but what if the console is not accessible? We can use a chardev for the ioport80 POST, but some firmwares post boot events via I2C, SPI, CAN... To improve the testing, my idea is to use the Tracing framework. The machine trace events would be logged in a db and avogado would verify some of the trace events, did they occur? in the correct sequence order? with the correct arguments? We can also check relative timestamps, is this sequence timing fast enough? I plan to use the gdb API to insert breakpoints and follow part execution flow, eventually injecting (faulty) events. The multi-arch/machines support is a bit weak yet. I'm personally more interested in automatically testing [real world] firmware or full machine boot process, the kind of integration testing I can not do with qtests. I liked the possibility to generate coredumps, or the replay function. I also liked to be able to write a test on how a machine boots, in ~20 LOC (see TestAlphaClipperBoot2_6). The storage API is OK to fetch a full VM image, but not a single file like a kernel or a flash image. Enough for my second try, good work :) I hope this was helpful. Regards, Phil.
> Cross-posting qemu-devel + avocado-devel. > >> While previously working on a Super I/O refactor, I encountered some problems >> at runtime, after building the codebase successfully and running qtests. >> I had to manually start to boot different guests and check the bootlog. >> >> I wanted to give a try at Avocado which seems designed to simplify that kind >> of functional tests. >> >> I applied Amador Pahim work following Cleber Rosa documentation from >> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html, >> however I had to modify few things to parse the boot console. >> Since his work is not merged, I included it in this series. >> >> The tests simply expect to find a string reported by Linux printk when a >> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A" >> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such >> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network >> device are correctly detected: >> >> >> self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog) >> self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog) >> self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog) >> self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog) >> >> Example of the tests output: >> >> $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml >> JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910 >> JOB LOG : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log >> (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s) >> (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s) >> (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s) >> (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s) >> (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s) >> (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s) >> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 >> JOB TIME : 13.31 s >> >> Please apologize my ugly Python, this series is just a proof-of-concept :) >> I couldn't figure out how to use the @skipUnless(correct arch) decorator. > > Eduardo asked me to share my first impressions after trying this > framework with QEMU. > > So far it seems more designed to run tests _inside_ a qemu instance > (think user-space), eventually testing packages/scripts. Current examples use the same pattern, which I felt a bit restricting: [QemuTest] .setUp: prepare VM arguments, launch the VM .test_*: run tests in the running VM .tearDown: shutdown the VM and cleanup dangling files > My tests are targeting the machine model itself, if the devices are > correctly instantiated and so. > The fastest approach was to check the Linux kernel bootlog, but if you > look at the TestAlphaClipperBoot console you'll see a boot "firmware" is > executed before the kernel. Casually this firmware also send information > on the console, but what if the console is not accessible? > We can use a chardev for the ioport80 POST, but some firmwares post boot > events via I2C, SPI, CAN... > > To improve the testing, my idea is to use the Tracing framework. > The machine trace events would be logged in a db and avogado would > verify some of the trace events, did they occur? in the correct sequence > order? with the correct arguments? We can also check relative > timestamps, is this sequence timing fast enough? > > I plan to use the gdb API to insert breakpoints and follow part > execution flow, eventually injecting (faulty) events. > > The multi-arch/machines support is a bit weak yet. > > I'm personally more interested in automatically testing [real world] > firmware or full machine boot process, the kind of integration testing I > can not do with qtests. > > I liked the possibility to generate coredumps, or the replay function. > > I also liked to be able to write a test on how a machine boots, in ~20 > LOC (see TestAlphaClipperBoot2_6). > > The storage API is OK to fetch a full VM image, but not a single file > like a kernel or a flash image
On 04/20/2018 12:17 AM, Philippe Mathieu-Daudé wrote: > Cross-posting qemu-devel + avocado-devel. > >> While previously working on a Super I/O refactor, I encountered some problems >> at runtime, after building the codebase successfully and running qtests. >> I had to manually start to boot different guests and check the bootlog. >> >> I wanted to give a try at Avocado which seems designed to simplify that kind >> of functional tests. >> >> I applied Amador Pahim work following Cleber Rosa documentation from >> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html, >> however I had to modify few things to parse the boot console. >> Since his work is not merged, I included it in this series. >> >> The tests simply expect to find a string reported by Linux printk when a >> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A" >> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such >> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network >> device are correctly detected: >> >> >> self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog) >> self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog) >> self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog) >> self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog) >> >> Example of the tests output: >> >> $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml >> JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910 >> JOB LOG : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log >> (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s) >> (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s) >> (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s) >> (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s) >> (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s) >> (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s) >> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 >> JOB TIME : 13.31 s >> >> Please apologize my ugly Python, this series is just a proof-of-concept :) >> I couldn't figure out how to use the @skipUnless(correct arch) decorator. > > Eduardo asked me to share my first impressions after trying this > framework with QEMU. > > So far it seems more designed to run tests _inside_ a qemu instance > (think user-space), eventually testing packages/scripts. > The first tests written indeed involved interacting with the guest OS more often than not. I tried to show one example that did not envolve a guest here: https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03468.html Just as short exercise, this is another example (sorry for the lack of creativity): class CpuSmp(test.QemuTest): """ :avocado: enable """ def test(self): smp_max = int(self.params.get('smp_max', default=255)) self.vm._args.extend(['-smp', '%s' % smp_max]) self.vm.launch() res = self.vm.qmp('query-cpus-fast') self.assertEqual(len(res[u'return']), smp_max) > My tests are targeting the machine model itself, if the devices are > correctly instantiated and so. > The fastest approach was to check the Linux kernel bootlog, but if you > look at the TestAlphaClipperBoot console you'll see a boot "firmware" is > executed before the kernel. Casually this firmware also send information > on the console, but what if the console is not accessible? > We can use a chardev for the ioport80 POST, but some firmwares post boot > events via I2C, SPI, CAN... > > To improve the testing, my idea is to use the Tracing framework. > The machine trace events would be logged in a db and avogado would > verify some of the trace events, did they occur? in the correct sequence > order? with the correct arguments? We can also check relative > timestamps, is this sequence timing fast enough? > Right, I think I see where you're going here. For the record, the Avocado test runner has the ability to check the test output against a previously recorded file, and that can determine the test status. Maybe a second implementation of the same concept can be applied to your idea. > I plan to use the gdb API to insert breakpoints and follow part > execution flow, eventually injecting (faulty) events. > Just to be clear, do you mean the Avocado GDB API? > The multi-arch/machines support is a bit weak yet. > Yes. One of the ways I believe the multi-arch/machines support can be significantly improved is by using some level of parameter standardization, together with tools to generate the intended test "matrix". On a quick experiment with "pict" (an indepedent combinatorial testing tool) on this matter, I came up with: --- arch: x86_64, ppc64, ppcemb, ppc, aarch64, alpha, s390x machine_type: pc, q35, pseries-2.12, virt, 40p, bamboo, g3beige, mac99, mpc8544ds, ppce500, prep, sam460ex, clipper, s390-ccw-virtio-2.12 machine_accel: kvm, tcg smp_max: 1, 4, 8, 15, 248, 255 IF [arch] = "x86_64" THEN [machine_type] = "q35" OR [machine_type] = "pc"; IF [arch] = "ppc" THEN [machine_type] = "40p" OR [machine_type] = "bamboo" OR [machine_type] = "g3beige" OR [machine_type] = "mac99" OR [machine_type] = "mpc8544ds" OR [machine_type] = "ppce500" OR [machine_type] = "prep" OR [machine_type] = "sam460ex"; IF [arch] = "alpha" THEN [machine_type] = "clipper"; IF [arch] = "ppc64" THEN [machine_type] = "pseries-2.12"; IF [arch] = "ppcemb" THEN [machine_type] = "bamboo"; IF [arch] = "aarch64" THEN [machine_type] = "virt"; IF [arch] = "s390x" THEN [machine_type] = "s390-ccw-virtio-2.12"; # On my machine, only x86_64 can do KVM IF NOT [arch] = "x86_64" THEN [machine_accel] = "tcg"; IF [machine_type] = "q35" THEN [smp_max] = 255; IF [machine_type] = "pc" THEN [smp_max] = 255; IF [machine_type] = "s390-ccw-virtio-2.12" THEN [smp_max] = 248; IF [machine_type] = "virt" THEN [smp_max] = 8; IF [machine_type] = "mpc8544ds" THEN [smp_max] = 15; IF [machine_type] = "clipper" THEN [smp_max] = 4; IF [machine_type] = "40p" THEN [smp_max] = 1; IF [machine_type] = "bamboo" THEN [smp_max] = 1; IF [machine_type] = "g3beige" THEN [smp_max] = 1; IF [machine_type] = "mac99" THEN [smp_max] = 1; IF [machine_type] = "sam460ex" THEN [smp_max] = 1; IF [machine_type] = "ppce500" THEN [smp_max] = 1; IF [machine_type] = "prep" THEN [smp_max] = 1; --- With this *very partial* definition of the arch/machine support in QEMU, a single test can be run in 22 different scenarios. I've used it to run the 4 VNC tests and the one CpuSmp test printed above, resulting in 110 different test executions: https://cleber.fedorapeople.org/results.html The first three parameters are already supported by avocado_qemu, and more can follow. And to be fully honest, "avocado_qemu" has more weaknesses than strengthens at the moment. That's partially because it's so young (and immature), but, can be made into our ideal grown "being". > I'm personally more interested in automatically testing [real world] > firmware or full machine boot process, the kind of integration testing I > can not do with qtests. > > I liked the possibility to generate coredumps, or the replay function. > > I also liked to be able to write a test on how a machine boots, in ~20 > LOC (see TestAlphaClipperBoot2_6). > > The storage API is OK to fetch a full VM image, but not a single file > like a kernel or a flash image. > As pointed elsewhere, fetching/caching is probably OK at this point. But handling the (optional) gzip compression can use some TLC. > Enough for my second try, good work :) > > I hope this was helpful. > > Regards, > > Phil. > Thanks for the review! - Cleber.
© 2016 - 2024 Red Hat, Inc.