[v1] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

[Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

Posted by Fam Zheng 6 years, 8 months ago

Build tests in one 32 bit Linux guest and three BSD images are defined in this
series. This is a more managable way than the manually maintained virtual
machines in patchew. Also, one big advantage of ephemeral VMs over long running
guests is the reduced RAM usage of host, which makes it possible to have one
host test all these BSD variants and probably more.

The BSD guest templates are manually prepared following

https://wiki.qemu.org/Hosts/BSD

as it is not easy to automate. (The ideal approach is like the ubuntu.i386
script, which configures the guest on top of an official released image, fully
automatically.)

Need for help: "gmake check" in the added OpenBSD image fails with -ENOMEM
errors, even if I change "-m 2G" to "-m 8G" when starting VM. Ideas? And there
is a warning from ./configure about OpenBSD going to be unsupported in coming
releases, is it still the case?

Fam

Fam Zheng (9):
  gitignore: Ignore vm test images
  qemu.py: Add variable vga type
  qemu.py: Add "wait()" method
  tests: Add vm test lib
  tests: Add ubuntu.i386 image
  tests: Add FreeBSD image
  tests: Add NetBSD image
  tests: Add OpenBSD image
  Makefile: Add rules to run vm tests

 .gitignore                |   2 +
 Makefile                  |   2 +
 configure                 |   2 +-
 scripts/qemu.py           |   8 +-
 tests/vm/Makefile.include |  41 ++++++++
 tests/vm/basevm.py        | 246 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/vm/freebsd          |  44 +++++++++
 tests/vm/netbsd           |  44 +++++++++
 tests/vm/openbsd          |  45 +++++++++
 tests/vm/ubuntu.i386      |  88 +++++++++++++++++
 10 files changed, 520 insertions(+), 2 deletions(-)
 create mode 100644 tests/vm/Makefile.include
 create mode 100755 tests/vm/basevm.py
 create mode 100755 tests/vm/freebsd
 create mode 100755 tests/vm/netbsd
 create mode 100755 tests/vm/openbsd
 create mode 100755 tests/vm/ubuntu.i386

-- 
2.13.4

Re: [Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

Posted by no-reply@patchew.org 6 years, 8 months ago

Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20170816072104.24420-1-famz@redhat.com
Subject: [Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
055464f200 Makefile: Add rules to run vm tests
a51bbb970d tests: Add OpenBSD image
ad7b5b619b tests: Add NetBSD image
e80f6022ec tests: Add FreeBSD image
cc87c43c6d tests: Add ubuntu.i386 image
9bb54d6c23 tests: Add vm test lib
5c83acfced qemu.py: Add "wait()" method
64ca3a8cde qemu.py: Add variable vga type
8c873e4307 gitignore: Ignore vm test images

=== OUTPUT BEGIN ===
Checking PATCH 1/9: gitignore: Ignore vm test images...
Checking PATCH 2/9: qemu.py: Add variable vga type...
Checking PATCH 3/9: qemu.py: Add "wait()" method...
Checking PATCH 4/9: tests: Add vm test lib...
ERROR: line over 90 characters
#83: FILE: tests/vm/basevm.py:58:
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCikC46WYtXotUd0UGPz9547Aj0KqC4gk+nt4BBJm86IHgCD9FygSGX9EFutXlhz9KZIPg9Okk7+IzXRHCWI2MNvhrcjyrezKREm71z08j9iwfxY3340fY2Mo+0khwpO7bzsgzkljHIHqcOg7MgttPInVMNH/EfqpgR8EDKJuWCB2Ny+EBFN/3dAiff0X/EvKle9PUrY70EkSycnyURS8HZReEqj8lN9J5kXzA8F6jBo/0Q42Ttv6e4k5YcaDrwmLrBWLra2PCXZLNyHqXEiFkGmdXtA1Eox9gc/p4jIXim6xrPNmpN6WyrrEjaCF5xYvNv8wXkD6uSWwbHYU24lIAn qemu-vm-key

WARNING: line over 80 characters
#187: FILE: tests/vm/basevm.py:162:
+                            "file=%s,if=none,id=%s,cache=writeback,format=raw" % \

WARNING: line over 80 characters
#190: FILE: tests/vm/basevm.py:165:
+                            "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)]

total: 1 errors, 2 warnings, 246 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 5/9: tests: Add ubuntu.i386 image...
Checking PATCH 6/9: tests: Add FreeBSD image...
Checking PATCH 7/9: tests: Add NetBSD image...
Checking PATCH 8/9: tests: Add OpenBSD image...
Checking PATCH 9/9: Makefile: Add rules to run vm tests...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

Posted by Kamil Rytarowski 6 years, 8 months ago

On 16.08.2017 09:20, Fam Zheng wrote:
> Build tests in one 32 bit Linux guest and three BSD images are defined in this
> series. This is a more managable way than the manually maintained virtual
> machines in patchew. Also, one big advantage of ephemeral VMs over long running
> guests is the reduced RAM usage of host, which makes it possible to have one
> host test all these BSD variants and probably more.
> 

Thank you for your work on this!

> The BSD guest templates are manually prepared following
> 
> https://wiki.qemu.org/Hosts/BSD
> 
> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> script, which configures the guest on top of an official released image, fully
> automatically.)
> 

For the reference, inside the NetBSD infrastructure we use homegrown Anita:

http://pkgsrc.se/misc/py-anita

> Need for help: "gmake check" in the added OpenBSD image fails with -ENOMEM
> errors, even if I change "-m 2G" to "-m 8G" when starting VM. Ideas? And there
> is a warning from ./configure about OpenBSD going to be unsupported in coming
> releases, is it still the case?
> 

I'm not sure about their defaults. Please check ulimit and perhaps
/etc/login.conf.

> Fam
> 
> Fam Zheng (9):
>   gitignore: Ignore vm test images
>   qemu.py: Add variable vga type
>   qemu.py: Add "wait()" method
>   tests: Add vm test lib
>   tests: Add ubuntu.i386 image
>   tests: Add FreeBSD image
>   tests: Add NetBSD image
>   tests: Add OpenBSD image
>   Makefile: Add rules to run vm tests
> 
>  .gitignore                |   2 +
>  Makefile                  |   2 +
>  configure                 |   2 +-
>  scripts/qemu.py           |   8 +-
>  tests/vm/Makefile.include |  41 ++++++++
>  tests/vm/basevm.py        | 246 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/vm/freebsd          |  44 +++++++++
>  tests/vm/netbsd           |  44 +++++++++
>  tests/vm/openbsd          |  45 +++++++++
>  tests/vm/ubuntu.i386      |  88 +++++++++++++++++
>  10 files changed, 520 insertions(+), 2 deletions(-)
>  create mode 100644 tests/vm/Makefile.include
>  create mode 100755 tests/vm/basevm.py
>  create mode 100755 tests/vm/freebsd
>  create mode 100755 tests/vm/netbsd
>  create mode 100755 tests/vm/openbsd
>  create mode 100755 tests/vm/ubuntu.i386
>

Re: [Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

Posted by Fam Zheng 6 years, 8 months ago

On Wed, 08/16 11:24, Kamil Rytarowski wrote:
> On 16.08.2017 09:20, Fam Zheng wrote:
> > Build tests in one 32 bit Linux guest and three BSD images are defined in this
> > series. This is a more managable way than the manually maintained virtual
> > machines in patchew. Also, one big advantage of ephemeral VMs over long running
> > guests is the reduced RAM usage of host, which makes it possible to have one
> > host test all these BSD variants and probably more.
> > 
> 
> Thank you for your work on this!
> 
> > The BSD guest templates are manually prepared following
> > 
> > https://wiki.qemu.org/Hosts/BSD
> > 
> > as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> > script, which configures the guest on top of an official released image, fully
> > automatically.)
> > 
> 
> For the reference, inside the NetBSD infrastructure we use homegrown Anita:
> 
> http://pkgsrc.se/misc/py-anita

Looks cool. Is it suitable for using to implement tests/vm/netbsd? Is there a
user documentation?

Fam

Re: [Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

Posted by Kamil Rytarowski 6 years, 8 months ago

On 16.08.2017 12:13, Fam Zheng wrote:
> On Wed, 08/16 11:24, Kamil Rytarowski wrote:
>> On 16.08.2017 09:20, Fam Zheng wrote:
>>> Build tests in one 32 bit Linux guest and three BSD images are defined in this
>>> series. This is a more managable way than the manually maintained virtual
>>> machines in patchew. Also, one big advantage of ephemeral VMs over long running
>>> guests is the reduced RAM usage of host, which makes it possible to have one
>>> host test all these BSD variants and probably more.
>>>
>>
>> Thank you for your work on this!
>>
>>> The BSD guest templates are manually prepared following
>>>
>>> https://wiki.qemu.org/Hosts/BSD
>>>
>>> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
>>> script, which configures the guest on top of an official released image, fully
>>> automatically.)
>>>
>>
>> For the reference, inside the NetBSD infrastructure we use homegrown Anita:
>>
>> http://pkgsrc.se/misc/py-anita
> 
> Looks cool. Is it suitable for using to implement tests/vm/netbsd? Is there a
> user documentation?
> 
> Fam
> 

Documentation:

http://www.gson.org/netbsd/anita/

I'm not sure if it is suitable. It was designed with a different goal -
to verify the process of installation and execution of ATF (regress)
tests. The process of clean installation of NetBSD for every patch is
overkill... however this might be a source of inspiration. (I've not
been digging into the source code).

Re: [Qemu-devel] [PATCH RFC 0/9] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

Posted by Stefan Hajnoczi 6 years, 8 months ago

On Wed, Aug 16, 2017 at 03:20:55PM +0800, Fam Zheng wrote:
> Build tests in one 32 bit Linux guest and three BSD images are defined in this
> series. This is a more managable way than the manually maintained virtual
> machines in patchew. Also, one big advantage of ephemeral VMs over long running
> guests is the reduced RAM usage of host, which makes it possible to have one
> host test all these BSD variants and probably more.
> 
> The BSD guest templates are manually prepared following
> 
> https://wiki.qemu.org/Hosts/BSD
> 
> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> script, which configures the guest on top of an official released image, fully
> automatically.)
> 
> Need for help: "gmake check" in the added OpenBSD image fails with -ENOMEM
> errors, even if I change "-m 2G" to "-m 8G" when starting VM. Ideas? And there
> is a warning from ./configure about OpenBSD going to be unsupported in coming
> releases, is it still the case?
> 
> Fam
> 
> Fam Zheng (9):
>   gitignore: Ignore vm test images
>   qemu.py: Add variable vga type
>   qemu.py: Add "wait()" method
>   tests: Add vm test lib
>   tests: Add ubuntu.i386 image
>   tests: Add FreeBSD image
>   tests: Add NetBSD image
>   tests: Add OpenBSD image
>   Makefile: Add rules to run vm tests
> 
>  .gitignore                |   2 +
>  Makefile                  |   2 +
>  configure                 |   2 +-
>  scripts/qemu.py           |   8 +-
>  tests/vm/Makefile.include |  41 ++++++++
>  tests/vm/basevm.py        | 246 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/vm/freebsd          |  44 +++++++++
>  tests/vm/netbsd           |  44 +++++++++
>  tests/vm/openbsd          |  45 +++++++++
>  tests/vm/ubuntu.i386      |  88 +++++++++++++++++
>  10 files changed, 520 insertions(+), 2 deletions(-)
>  create mode 100644 tests/vm/Makefile.include
>  create mode 100755 tests/vm/basevm.py
>  create mode 100755 tests/vm/freebsd
>  create mode 100755 tests/vm/netbsd
>  create mode 100755 tests/vm/openbsd
>  create mode 100755 tests/vm/ubuntu.i386

Adding Cleber to CC since he's looking into QEMU test infrastructure
improvements.

Stefan