v3: Drop RFC.
    Add Stefan's and Kamil's reviewed-bys.
    Use optparse. [Stefan]
    Drop the VGA patch. [Paolo, Stefan]
    Improve exit/exit code/doc. [Stefan]
    Drop unused line from basevm.py. [Stefan]
    Drop "--target-list" form Makefile.
    More intelligent '-j'.
    Add README. [Stefan]

v2: - Add docstring. [Stefan]
    - Call self._load_io_lod. [Stefan]
    - Use "info usernet" and dynamic ssh_port forwarding. [Stefan]
    - Add image checksum.
    - Use os.rename() and os.makedirs(). [Stefan]
    - Fix NetBSD URL. [Kamil]

Build tests in one 32 bit Linux guest and three BSD images are defined in this
series. This is a more managable way than the manually maintained virtual
machines in patchew. Also, one big advantage of ephemeral VMs over long running
guests is the reduced RAM usage of host, which makes it possible to have one
host test all these BSD variants and probably more.

The BSD guest templates are manually prepared following

https://wiki.qemu.org/Hosts/BSD

as it is not easy to automate. (The ideal approach is like the ubuntu.i386
script, which configures the guest on top of an official released image, fully
automatically.)

Need for help: "gmake check" in the added OpenBSD image fails with -ENOMEM
errors, even if I change "-m 2G" to "-m 8G" when starting VM. Ideas? And there
is a warning from ./configure about OpenBSD going to be unsupported in coming
releases, is it still the case?

Fam

Fam Zheng (10):
  gitignore: Ignore vm test images
  qemu.py: Add "wait()" method
  tests: Add vm test lib
  tests: Add ubuntu.i386 image
  tests: Add FreeBSD image
  tests: Add NetBSD image
  tests: Add OpenBSD image
  Makefile: Add rules to run vm tests
  MAINTAINERS: Add tests/vm entry
  tests: Add README for vm tests

 .gitignore                |   2 +
 MAINTAINERS               |   1 +
 Makefile                  |   2 +
 configure                 |   2 +-
 scripts/qemu.py           |   7 ++
 tests/vm/Makefile.include |  40 +++++++
 tests/vm/README           |  63 +++++++++++
 tests/vm/basevm.py        | 278 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/vm/freebsd          |  45 ++++++++
 tests/vm/netbsd           |  45 ++++++++
 tests/vm/openbsd          |  46 ++++++++
 tests/vm/ubuntu.i386      |  88 +++++++++++++++
 12 files changed, 618 insertions(+), 1 deletion(-)
 create mode 100644 tests/vm/Makefile.include
 create mode 100644 tests/vm/README
 create mode 100755 tests/vm/basevm.py
 create mode 100755 tests/vm/freebsd
 create mode 100755 tests/vm/netbsd
 create mode 100755 tests/vm/openbsd
 create mode 100755 tests/vm/ubuntu.i386

-- 
2.13.5

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20170822044155.2451-1-famz@redhat.com
Subject: [Qemu-devel] [PATCH v3 00/10] tests: Add VM based build tests (for non-x86_64 and/or non-Linux)

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
7aa4d0521d tests: Add README for vm tests
70efee6246 MAINTAINERS: Add tests/vm entry
c446bc5831 Makefile: Add rules to run vm tests
e1fd590bf6 tests: Add OpenBSD image
e305ed3d1d tests: Add NetBSD image
2edb0605fa tests: Add FreeBSD image
e49723e391 tests: Add ubuntu.i386 image
a32addb8d9 tests: Add vm test lib
41055a6e52 qemu.py: Add "wait()" method
ccad1233b2 gitignore: Ignore vm test images

=== OUTPUT BEGIN ===
Checking PATCH 1/10: gitignore: Ignore vm test images...
Checking PATCH 2/10: qemu.py: Add "wait()" method...
Checking PATCH 3/10: tests: Add vm test lib...
ERROR: line over 90 characters
#85: FILE: tests/vm/basevm.py:60:
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCikC46WYtXotUd0UGPz9547Aj0KqC4gk+nt4BBJm86IHgCD9FygSGX9EFutXlhz9KZIPg9Okk7+IzXRHCWI2MNvhrcjyrezKREm71z08j9iwfxY3340fY2Mo+0khwpO7bzsgzkljHIHqcOg7MgttPInVMNH/EfqpgR8EDKJuWCB2Ny+EBFN/3dAiff0X/EvKle9PUrY70EkSycnyURS8HZReEqj8lN9J5kXzA8F6jBo/0Q42Ttv6e4k5YcaDrwmLrBWLra2PCXZLNyHqXEiFkGmdXtA1Eox9gc/p4jIXim6xrPNmpN6WyrrEjaCF5xYvNv8wXkD6uSWwbHYU24lIAn qemu-vm-key

WARNING: line over 80 characters
#197: FILE: tests/vm/basevm.py:172:
+                            "file=%s,if=none,id=%s,cache=writeback,format=raw" % \

WARNING: line over 80 characters
#200: FILE: tests/vm/basevm.py:175:
+                            "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)]

ERROR: line over 90 characters
#247: FILE: tests/vm/basevm.py:222:
+    VM test utility.  Exit codes: 0 = success, 1 = command line error, 2 = environment initialization failed, 3 = test command failed""")

total: 2 errors, 2 warnings, 278 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 4/10: tests: Add ubuntu.i386 image...
Checking PATCH 5/10: tests: Add FreeBSD image...
Checking PATCH 6/10: tests: Add NetBSD image...
Checking PATCH 7/10: tests: Add OpenBSD image...
Checking PATCH 8/10: Makefile: Add rules to run vm tests...
Checking PATCH 9/10: MAINTAINERS: Add tests/vm entry...
Checking PATCH 10/10: tests: Add README for vm tests...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

On 08/22/2017 12:41 AM, Fam Zheng wrote:
> v3: Drop RFC.
>     Add Stefan's and Kamil's reviewed-bys.
>     Use optparse. [Stefan]
>     Drop the VGA patch. [Paolo, Stefan]
>     Improve exit/exit code/doc. [Stefan]
>     Drop unused line from basevm.py. [Stefan]
>     Drop "--target-list" form Makefile.
>     More intelligent '-j'.
>     Add README. [Stefan]
> 
> v2: - Add docstring. [Stefan]
>     - Call self._load_io_lod. [Stefan]
>     - Use "info usernet" and dynamic ssh_port forwarding. [Stefan]
>     - Add image checksum.
>     - Use os.rename() and os.makedirs(). [Stefan]
>     - Fix NetBSD URL. [Kamil]
> 
> Build tests in one 32 bit Linux guest and three BSD images are defined in this
> series. This is a more managable way than the manually maintained virtual
> machines in patchew. Also, one big advantage of ephemeral VMs over long running
> guests is the reduced RAM usage of host, which makes it possible to have one
> host test all these BSD variants and probably more.
> 
> The BSD guest templates are manually prepared following
> 
> https://wiki.qemu.org/Hosts/BSD
> 
> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> script, which configures the guest on top of an official released image, fully
> automatically.)
> 

I replayed manually the FreeBSD VM setup, just to get a sense of how it
could be automated.  Taking a few steps back, I realized that:

 * describing how to prepare a given OS to build QEMU is a generic task,
not really bound to this VM setup
 * cloud-init, while unarguably a sound solution, is bound to not only
VMs, but to already prepared (with cloud-init support) images

What if we attempt to switch the "build environment setup" automation to
use a more generic tool such as ansible?  Having an in-tree playbook
that could be executed against a VM seems like slightly better than
syncing the WIKI pages.

I'm no expert in ansible, but I can give it that a try if you fancy the
idea.

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

On Thu, 08/24 11:06, Cleber Rosa wrote:
> I replayed manually the FreeBSD VM setup, just to get a sense of how it
> could be automated.  Taking a few steps back, I realized that:
> 
>  * describing how to prepare a given OS to build QEMU is a generic task,
> not really bound to this VM setup
>  * cloud-init, while unarguably a sound solution, is bound to not only
> VMs, but to already prepared (with cloud-init support) images
> 
> What if we attempt to switch the "build environment setup" automation to
> use a more generic tool such as ansible?  Having an in-tree playbook
> that could be executed against a VM seems like slightly better than
> syncing the WIKI pages.
> 
> I'm no expert in ansible, but I can give it that a try if you fancy the
> idea.
> 

I have no objection if you want to give it a try. However it is basically a
separate problem, and I don't think this series would depend on that. The reason
is that setting up guest ssh is not part of the "build env setup" task, but is
the trickiest part here.  As you can see in the ubuntu image, "cloud-init" is
used just for that, and the build env is simply doing a "apt-get build-dep"; if
it were for Fedora the command is a dnf counterpart.  (I probably could have
removed the dev packages in BSD templates and add code to the *bsd scripts to
locally install packages after downloading.)

Ansible is probably an overkill anyway because what we'd have in the playbooks
are command lists to install package sets. Shell scripts can do that just fine,
this is not what ansible is great for, IMO (of course I'd be wrong if there is
an ansible module called 'build-dep' that supports many OSes, I'm not an ansible
expert either).

Fam

On 25/08/2017 11:25, Fam Zheng wrote:
>> I'm no expert in ansible, but I can give it that a try if you fancy the
>> idea.
>>
> I have no objection if you want to give it a try. However it is basically a
> separate problem, and I don't think this series would depend on that. The reason
> is that setting up guest ssh is not part of the "build env setup" task, but is
> the trickiest part here. 

Indeed, and worse this is also needed for ansible.  So it's a
chicken-and-egg problem.

Paolo

I propose to rename *bsd.img.xz (like netbsd.img.xz) to
*bsd-version-arch.img.xz, like netbsd-7.1-amd64.img.xz. This will be
more verbose about the version and leave room for image upgrades in future.

On 22.08.2017 06:41, Fam Zheng wrote:
> v3: Drop RFC.
>     Add Stefan's and Kamil's reviewed-bys.
>     Use optparse. [Stefan]
>     Drop the VGA patch. [Paolo, Stefan]
>     Improve exit/exit code/doc. [Stefan]
>     Drop unused line from basevm.py. [Stefan]
>     Drop "--target-list" form Makefile.
>     More intelligent '-j'.
>     Add README. [Stefan]
> 
> v2: - Add docstring. [Stefan]
>     - Call self._load_io_lod. [Stefan]
>     - Use "info usernet" and dynamic ssh_port forwarding. [Stefan]
>     - Add image checksum.
>     - Use os.rename() and os.makedirs(). [Stefan]
>     - Fix NetBSD URL. [Kamil]
> 
> Build tests in one 32 bit Linux guest and three BSD images are defined in this
> series. This is a more managable way than the manually maintained virtual
> machines in patchew. Also, one big advantage of ephemeral VMs over long running
> guests is the reduced RAM usage of host, which makes it possible to have one
> host test all these BSD variants and probably more.
> 
> The BSD guest templates are manually prepared following
> 
> https://wiki.qemu.org/Hosts/BSD
> 
> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> script, which configures the guest on top of an official released image, fully
> automatically.)
> 
> Need for help: "gmake check" in the added OpenBSD image fails with -ENOMEM
> errors, even if I change "-m 2G" to "-m 8G" when starting VM. Ideas? And there
> is a warning from ./configure about OpenBSD going to be unsupported in coming
> releases, is it still the case?
> 
> Fam
> 
> Fam Zheng (10):
>   gitignore: Ignore vm test images
>   qemu.py: Add "wait()" method
>   tests: Add vm test lib
>   tests: Add ubuntu.i386 image
>   tests: Add FreeBSD image
>   tests: Add NetBSD image
>   tests: Add OpenBSD image
>   Makefile: Add rules to run vm tests
>   MAINTAINERS: Add tests/vm entry
>   tests: Add README for vm tests
> 
>  .gitignore                |   2 +
>  MAINTAINERS               |   1 +
>  Makefile                  |   2 +
>  configure                 |   2 +-
>  scripts/qemu.py           |   7 ++
>  tests/vm/Makefile.include |  40 +++++++
>  tests/vm/README           |  63 +++++++++++
>  tests/vm/basevm.py        | 278 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/vm/freebsd          |  45 ++++++++
>  tests/vm/netbsd           |  45 ++++++++
>  tests/vm/openbsd          |  46 ++++++++
>  tests/vm/ubuntu.i386      |  88 +++++++++++++++
>  12 files changed, 618 insertions(+), 1 deletion(-)
>  create mode 100644 tests/vm/Makefile.include
>  create mode 100644 tests/vm/README
>  create mode 100755 tests/vm/basevm.py
>  create mode 100755 tests/vm/freebsd
>  create mode 100755 tests/vm/netbsd
>  create mode 100755 tests/vm/openbsd
>  create mode 100755 tests/vm/ubuntu.i386
> 

On Tue, 08/22 16:33, Kamil Rytarowski wrote:
> I propose to rename *bsd.img.xz (like netbsd.img.xz) to
> *bsd-version-arch.img.xz, like netbsd-7.1-amd64.img.xz. This will be
> more verbose about the version and leave room for image upgrades in future.

Good idea, will update.

Fam

On 22/08/2017 06:41, Fam Zheng wrote:
> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> script, which configures the guest on top of an official released image, fully
> automatically.)

Out of curiosity, why can't the 32-bit test use containers instead?

Paolo

On 22.08.2017 15:58, Paolo Bonzini wrote:
> On 22/08/2017 06:41, Fam Zheng wrote:
>> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
>> script, which configures the guest on top of an official released image, fully
>> automatically.)
> 
> Out of curiosity, why can't the 32-bit test use containers instead?
> 
> Paolo
> 

At least this is more friendly to !Linux users.

On Tue, 08/22 15:58, Paolo Bonzini wrote:
> On 22/08/2017 06:41, Fam Zheng wrote:
> > as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> > script, which configures the guest on top of an official released image, fully
> > automatically.)
> 
> Out of curiosity, why can't the 32-bit test use containers instead?

It can.

This one is here just because what are distributed by hub.docker.com are x86_64
environments. Of course it's possible to do 32-bit builds with them, the i386
image just provides a bit more authenticity. I'm not counting on it to catch
more bugs than the mingw 32bit cross-build, but this makes a good example of
using basevm.py.

Fam

On Tue, Aug 22, 2017 at 12:41:45PM +0800, Fam Zheng wrote:
> v3: Drop RFC.
>     Add Stefan's and Kamil's reviewed-bys.
>     Use optparse. [Stefan]
>     Drop the VGA patch. [Paolo, Stefan]
>     Improve exit/exit code/doc. [Stefan]
>     Drop unused line from basevm.py. [Stefan]
>     Drop "--target-list" form Makefile.
>     More intelligent '-j'.
>     Add README. [Stefan]
> 
> v2: - Add docstring. [Stefan]
>     - Call self._load_io_lod. [Stefan]
>     - Use "info usernet" and dynamic ssh_port forwarding. [Stefan]
>     - Add image checksum.
>     - Use os.rename() and os.makedirs(). [Stefan]
>     - Fix NetBSD URL. [Kamil]
> 
> Build tests in one 32 bit Linux guest and three BSD images are defined in this
> series. This is a more managable way than the manually maintained virtual
> machines in patchew. Also, one big advantage of ephemeral VMs over long running
> guests is the reduced RAM usage of host, which makes it possible to have one
> host test all these BSD variants and probably more.
> 
> The BSD guest templates are manually prepared following
> 
> https://wiki.qemu.org/Hosts/BSD
> 
> as it is not easy to automate. (The ideal approach is like the ubuntu.i386
> script, which configures the guest on top of an official released image, fully
> automatically.)
> 
> Need for help: "gmake check" in the added OpenBSD image fails with -ENOMEM
> errors, even if I change "-m 2G" to "-m 8G" when starting VM. Ideas? And there
> is a warning from ./configure about OpenBSD going to be unsupported in coming
> releases, is it still the case?
> 
> Fam
> 
> Fam Zheng (10):
>   gitignore: Ignore vm test images
>   qemu.py: Add "wait()" method
>   tests: Add vm test lib
>   tests: Add ubuntu.i386 image
>   tests: Add FreeBSD image
>   tests: Add NetBSD image
>   tests: Add OpenBSD image
>   Makefile: Add rules to run vm tests
>   MAINTAINERS: Add tests/vm entry
>   tests: Add README for vm tests
> 
>  .gitignore                |   2 +
>  MAINTAINERS               |   1 +
>  Makefile                  |   2 +
>  configure                 |   2 +-
>  scripts/qemu.py           |   7 ++
>  tests/vm/Makefile.include |  40 +++++++
>  tests/vm/README           |  63 +++++++++++
>  tests/vm/basevm.py        | 278 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/vm/freebsd          |  45 ++++++++
>  tests/vm/netbsd           |  45 ++++++++
>  tests/vm/openbsd          |  46 ++++++++
>  tests/vm/ubuntu.i386      |  88 +++++++++++++++
>  12 files changed, 618 insertions(+), 1 deletion(-)
>  create mode 100644 tests/vm/Makefile.include
>  create mode 100644 tests/vm/README
>  create mode 100755 tests/vm/basevm.py
>  create mode 100755 tests/vm/freebsd
>  create mode 100755 tests/vm/netbsd
>  create mode 100755 tests/vm/openbsd
>  create mode 100755 tests/vm/ubuntu.i386

Aside from the comments I posted:

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>