This patch series changes the way virtual machines for test builds are
managed.  They are created locally on the developer machine now.  The
installer is booted on the serial console and the scripts walks through
the dialogs to install and configure the guest.

That takes the download.patchew.org server out of the loop and makes it
alot easier to tweak the guest images (adding build dependencies for
example).

The install scripts take care to apply host proxy settings (from *_proxy
environment variables) to the guest, so any package downloads will be
routed through the proxy and can be cached that way.  This also makes
them work behind strict firewalls.

There are also a bunch of smaller tweaks for tests/vm to fix issues I
was struggling with.  See commit messages of individual patches for
details.

Gerd Hoffmann (13):
  scripts: use git archive in archive-source
  tests/vm: send proxy environment variables over ssh
  tests/vm: use ssh with pty unconditionally
  tests/vm: run test builds on snapshot
  tests/vm: proper guest shutdown
  tests/vm: add vm-boot-{ssh,serial}-<guest> targets
  tests/vm: add DEBUG=1 to help text
  tests/vm: serial console support helpers
  tests/vm: openbsd autoinstall, using serial console
  tests/vm: freebsd autoinstall, using serial console
  tests/vm: netbsd autoinstall, using serial console
  tests/vm: fedora autoinstall, using serial console
  tests/vm: ubuntu.i386: apt proxy setup

 tests/vm/basevm.py        | 134 +++++++++++++++++++++++----
 scripts/archive-source.sh |  72 +++++++--------
 tests/vm/Makefile.include |  25 ++++-
 tests/vm/fedora           | 187 ++++++++++++++++++++++++++++++++++++++
 tests/vm/freebsd          | 175 +++++++++++++++++++++++++++++++++--
 tests/vm/netbsd           | 187 ++++++++++++++++++++++++++++++++++++--
 tests/vm/openbsd          | 154 ++++++++++++++++++++++++++++---
 tests/vm/ubuntu.i386      |   4 +
 8 files changed, 845 insertions(+), 93 deletions(-)
 create mode 100755 tests/vm/fedora

-- 
2.18.1

On 10/05/2019 12.46, Gerd Hoffmann wrote:
> This patch series changes the way virtual machines for test builds are
> managed.  They are created locally on the developer machine now.  The
> installer is booted on the serial console and the scripts walks through
> the dialogs to install and configure the guest.
> 
> That takes the download.patchew.org server out of the loop and makes it
> alot easier to tweak the guest images (adding build dependencies for
> example).
> 
> The install scripts take care to apply host proxy settings (from *_proxy
> environment variables) to the guest, so any package downloads will be
> routed through the proxy and can be cached that way.  This also makes
> them work behind strict firewalls.
> 
> There are also a bunch of smaller tweaks for tests/vm to fix issues I
> was struggling with.  See commit messages of individual patches for
> details.
> 
> Gerd Hoffmann (13):
>   scripts: use git archive in archive-source
>   tests/vm: send proxy environment variables over ssh
>   tests/vm: use ssh with pty unconditionally
>   tests/vm: run test builds on snapshot
>   tests/vm: proper guest shutdown
>   tests/vm: add vm-boot-{ssh,serial}-<guest> targets
>   tests/vm: add DEBUG=1 to help text
>   tests/vm: serial console support helpers
>   tests/vm: openbsd autoinstall, using serial console
>   tests/vm: freebsd autoinstall, using serial console
>   tests/vm: netbsd autoinstall, using serial console
>   tests/vm: fedora autoinstall, using serial console
>   tests/vm: ubuntu.i386: apt proxy setup

freebsd, netbsd and fedora targets work fine for me, so for the patches
1 - 8 and 10 - 12 :

Tested-by: Thomas Huth <thuth@redhat.com>

openbsd still fails for me:

  TEST    check-qtest-arm: tests/tmp105-test
  TEST    check-qtest-arm: tests/pca9552-test
  TEST    check-qtest-arm: tests/ds1338-test
  TEST    check-qtest-arm: tests/microbit-test
  TEST    check-qtest-arm: tests/m25p80-test
  TEST    check-qtest-arm: tests/test-arm-mptimer
  TEST    check-qtest-arm: tests/boot-serial-test
qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
Broken pipe
/home/qemu/qemu-test.Ka98K9/src/tests/libqtest.c:135: kill_qemu() tried
to terminate QEMU process but encountered exit status 1
ERROR - too few tests run (expected 2, got 0)
Abort trap (core dumped)
gmake: *** [/home/qemu/qemu-test.Ka98K9/src/tests/Makefile.include:884:
check-qtest-arm] Error 1

Brad, does current master work for you on OpenBSD? ... looking at the
history of the openbsd script, it seems like "make check" is broken on
OpenBSD since 2017 ... any chance that this could ever be fixed?

 Thomas

On 5/16/2019 9:04 PM, Thomas Huth wrote:

> On 10/05/2019 12.46, Gerd Hoffmann wrote:
>> This patch series changes the way virtual machines for test builds are
>> managed.  They are created locally on the developer machine now.  The
>> installer is booted on the serial console and the scripts walks through
>> the dialogs to install and configure the guest.
>>
>> That takes the download.patchew.org server out of the loop and makes it
>> alot easier to tweak the guest images (adding build dependencies for
>> example).
>>
>> The install scripts take care to apply host proxy settings (from *_proxy
>> environment variables) to the guest, so any package downloads will be
>> routed through the proxy and can be cached that way.  This also makes
>> them work behind strict firewalls.
>>
>> There are also a bunch of smaller tweaks for tests/vm to fix issues I
>> was struggling with.  See commit messages of individual patches for
>> details.
>>
>> Gerd Hoffmann (13):
>>    scripts: use git archive in archive-source
>>    tests/vm: send proxy environment variables over ssh
>>    tests/vm: use ssh with pty unconditionally
>>    tests/vm: run test builds on snapshot
>>    tests/vm: proper guest shutdown
>>    tests/vm: add vm-boot-{ssh,serial}-<guest> targets
>>    tests/vm: add DEBUG=1 to help text
>>    tests/vm: serial console support helpers
>>    tests/vm: openbsd autoinstall, using serial console
>>    tests/vm: freebsd autoinstall, using serial console
>>    tests/vm: netbsd autoinstall, using serial console
>>    tests/vm: fedora autoinstall, using serial console
>>    tests/vm: ubuntu.i386: apt proxy setup
> freebsd, netbsd and fedora targets work fine for me, so for the patches
> 1 - 8 and 10 - 12 :
>
> Tested-by: Thomas Huth <thuth@redhat.com>
>
> openbsd still fails for me:
>
>    TEST    check-qtest-arm: tests/tmp105-test
>    TEST    check-qtest-arm: tests/pca9552-test
>    TEST    check-qtest-arm: tests/ds1338-test
>    TEST    check-qtest-arm: tests/microbit-test
>    TEST    check-qtest-arm: tests/m25p80-test
>    TEST    check-qtest-arm: tests/test-arm-mptimer
>    TEST    check-qtest-arm: tests/boot-serial-test
> qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
> Broken pipe

How much memory is trying to be allocated here?

The default maximum data size is set to 768MB. If there is a requirement 
to go beyond
that then the default has to be adjusted in /etc/login.conf.

datasize-max and datasize-cur

default:\
         :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin 
/usr/local/bin /usr/local/sbin:\
         :umask=022:\
         :datasize-max=768M:\
         :datasize-cur=768M:\
         :maxproc-max=256:\
         :maxproc-cur=128:\
         :openfiles-max=1024:\
         :openfiles-cur=512:\

  Hi,

> > qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate memory
> > Broken pipe
> 
> How much memory is trying to be allocated here?
> 
> The default maximum data size is set to 768MB. If there is a requirement to
> go beyond
> that then the default has to be adjusted in /etc/login.conf.

Yes, adjusting the limits fixes that.

thanks,
  Gerd

I just noticed when I had replied that my e-mail was sent from a different
name, by accident, as I was testing something with my e-mail client.

On 5/18/2019 5:27 PM, Jim Payne wrote:
> On 5/16/2019 9:04 PM, Thomas Huth wrote:
>
>> On 10/05/2019 12.46, Gerd Hoffmann wrote:
>>> This patch series changes the way virtual machines for test builds are
>>> managed.  They are created locally on the developer machine now.  The
>>> installer is booted on the serial console and the scripts walks through
>>> the dialogs to install and configure the guest.
>>>
>>> That takes the download.patchew.org server out of the loop and makes it
>>> alot easier to tweak the guest images (adding build dependencies for
>>> example).
>>>
>>> The install scripts take care to apply host proxy settings (from 
>>> *_proxy
>>> environment variables) to the guest, so any package downloads will be
>>> routed through the proxy and can be cached that way.  This also makes
>>> them work behind strict firewalls.
>>>
>>> There are also a bunch of smaller tweaks for tests/vm to fix issues I
>>> was struggling with.  See commit messages of individual patches for
>>> details.
>>>
>>> Gerd Hoffmann (13):
>>>    scripts: use git archive in archive-source
>>>    tests/vm: send proxy environment variables over ssh
>>>    tests/vm: use ssh with pty unconditionally
>>>    tests/vm: run test builds on snapshot
>>>    tests/vm: proper guest shutdown
>>>    tests/vm: add vm-boot-{ssh,serial}-<guest> targets
>>>    tests/vm: add DEBUG=1 to help text
>>>    tests/vm: serial console support helpers
>>>    tests/vm: openbsd autoinstall, using serial console
>>>    tests/vm: freebsd autoinstall, using serial console
>>>    tests/vm: netbsd autoinstall, using serial console
>>>    tests/vm: fedora autoinstall, using serial console
>>>    tests/vm: ubuntu.i386: apt proxy setup
>> freebsd, netbsd and fedora targets work fine for me, so for the patches
>> 1 - 8 and 10 - 12 :
>>
>> Tested-by: Thomas Huth <thuth@redhat.com>
>>
>> openbsd still fails for me:
>>
>>    TEST    check-qtest-arm: tests/tmp105-test
>>    TEST    check-qtest-arm: tests/pca9552-test
>>    TEST    check-qtest-arm: tests/ds1338-test
>>    TEST    check-qtest-arm: tests/microbit-test
>>    TEST    check-qtest-arm: tests/m25p80-test
>>    TEST    check-qtest-arm: tests/test-arm-mptimer
>>    TEST    check-qtest-arm: tests/boot-serial-test
>> qemu-system-arm: cannot set up guest memory 'ram': Cannot allocate 
>> memory
>> Broken pipe
>
> How much memory is trying to be allocated here?
>
> The default maximum data size is set to 768MB. If there is a 
> requirement to go beyond
> that then the default has to be adjusted in /etc/login.conf.
>
> datasize-max and datasize-cur
>
> default:\
>         :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin 
> /usr/local/bin /usr/local/sbin:\
>         :umask=022:\
>         :datasize-max=768M:\
>         :datasize-cur=768M:\
>         :maxproc-max=256:\
>         :maxproc-cur=128:\
>         :openfiles-max=1024:\
>         :openfiles-cur=512:\
>
>