This series add a new acceptance test: boot an uncompressed
Linux kernel built with CONFIG_PVH, so checking the PVH
capability introduced in QEMU 4.0 works.

The test implementation depends on [1] which is likely released
on next Avocado. So that will need a version 2 of this
series to bump Avocado version.

Also I want to use this as an example of a scenario that test
assets could be better managed. As you see on patch 01 the
kernel is built at test time on localhost. While Avocado provides
an API to easily fetch and build it, the whole process takes
reasonable time - besides the fact that localhost must have
all build dependencies installed. How could it be done better?

Nonetheless I argue in favor of merging this as is, and
gradually implement corrections to improve the tests assets
management. Also if eventually the test is proven to unacceptable
slow down the Travis CI then I can add a guard to skip it.

[1] https://github.com/avocado-framework/avocado/pull/3389

Wainer dos Santos Moschetta (2):
  tests/acceptance: Add PVH boot test
  .travis.yml: Add kernel build deps for acceptance tests

 .travis.yml             |  2 ++
 tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+)
 create mode 100644 tests/acceptance/pvh.py

-- 
2.21.0

On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote:
> This series add a new acceptance test: boot an uncompressed
> Linux kernel built with CONFIG_PVH, so checking the PVH
> capability introduced in QEMU 4.0 works.
> 
> The test implementation depends on [1] which is likely released
> on next Avocado. So that will need a version 2 of this
> series to bump Avocado version.
>

Right, the Avocado bits have been merged, so unless a major reversal
comes (not expected at all), it will be on Avocado 74.0.

> Also I want to use this as an example of a scenario that test
> assets could be better managed. As you see on patch 01 the
> kernel is built at test time on localhost. While Avocado provides
> an API to easily fetch and build it, the whole process takes
> reasonable time - besides the fact that localhost must have
> all build dependencies installed. How could it be done better?
>

This is clearly not a "kernel build" test, so we should avoid building
the kernel as part of the "PVH boot" test.  The problem you expose
here is a very real, and each possible solution has its own problems,
unfortunately.

The best solution IMO would be to find a "well known" distribution of
such a kernel.  Maybe something maintained by the Xen project or one
of its commercial products?

The second best solution is to have a helper script (using the Avocado
utils API is fine) that will build a kernel and create/populate the
test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux.
The test can cancel itself if it doesn't find a kernel there.

The third best solution IMO is for this test to require a parameter
telling where the CONFIG_PVH enabled vmlinux kernel image is.

Also notice that, with a custom built kernel image (a different one
for each user), the result of the test as a general indicator to the
QEMU codebase diminishes quite a bit.

> Nonetheless I argue in favor of merging this as is, and
> gradually implement corrections to improve the tests assets
> management. Also if eventually the test is proven to unacceptable
> slow down the Travis CI then I can add a guard to skip it.
>

I'm pretty sure that building the kernel will cause a major slow down
of the Travis CI results for the "acceptance" block (and thus for the
overall results).  But, it may also time it out (there's a 50 minutes
deadline).

I hope this brings some ideas, and thanks for coming up with
interesting problems! :)

- Cleber.

> [1] https://github.com/avocado-framework/avocado/pull/3389
> 
> Wainer dos Santos Moschetta (2):
>   tests/acceptance: Add PVH boot test
>   .travis.yml: Add kernel build deps for acceptance tests
> 
>  .travis.yml             |  2 ++
>  tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 50 insertions(+)
>  create mode 100644 tests/acceptance/pvh.py
> 
> -- 
> 2.21.0
> 

Cleber Rosa <crosa@redhat.com> writes:

> On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote:
>> This series add a new acceptance test: boot an uncompressed
>> Linux kernel built with CONFIG_PVH, so checking the PVH
>> capability introduced in QEMU 4.0 works.
>> 
>> The test implementation depends on [1] which is likely released
>> on next Avocado. So that will need a version 2 of this
>> series to bump Avocado version.
>>
>
> Right, the Avocado bits have been merged, so unless a major reversal
> comes (not expected at all), it will be on Avocado 74.0.
>
>> Also I want to use this as an example of a scenario that test
>> assets could be better managed. As you see on patch 01 the
>> kernel is built at test time on localhost. While Avocado provides
>> an API to easily fetch and build it, the whole process takes
>> reasonable time - besides the fact that localhost must have
>> all build dependencies installed. How could it be done better?
>>
>
> This is clearly not a "kernel build" test, so we should avoid building
> the kernel as part of the "PVH boot" test.  The problem you expose
> here is a very real, and each possible solution has its own problems,
> unfortunately.
>
> The best solution IMO would be to find a "well known" distribution of
> such a kernel.  Maybe something maintained by the Xen project or one
> of its commercial products?
>
> The second best solution is to have a helper script (using the Avocado
> utils API is fine) that will build a kernel and create/populate the
> test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux.
> The test can cancel itself if it doesn't find a kernel there.
>
> The third best solution IMO is for this test to require a parameter
> telling where the CONFIG_PVH enabled vmlinux kernel image is.
>
> Also notice that, with a custom built kernel image (a different one
> for each user), the result of the test as a general indicator to the
> QEMU codebase diminishes quite a bit.

I can see a use case for making it easier for developers to build test
cases otherwise everyone has their own particular variant of the kernel
and buildroot/busybox which they have in their own farm. However as
Cleber has noted they make poor standardised tests given the variation
in kernel builds you can get.

That said I think there are better targets. kvm-unit-tests can be built
against a range of architectures and are fashioned as individual unit
tests for specific functionality. It would be useful to make it easy for
a developer to regenerate the test assets to re-run a test someone else
has found to fail.

>> Nonetheless I argue in favor of merging this as is, and
>> gradually implement corrections to improve the tests assets
>> management. Also if eventually the test is proven to unacceptable
>> slow down the Travis CI then I can add a guard to skip it.
>>
>
> I'm pretty sure that building the kernel will cause a major slow down
> of the Travis CI results for the "acceptance" block (and thus for the
> overall results).  But, it may also time it out (there's a 50 minutes
> deadline).
>
> I hope this brings some ideas, and thanks for coming up with
> interesting problems! :)
>
> - Cleber.
>
>> [1] https://github.com/avocado-framework/avocado/pull/3389
>> 
>> Wainer dos Santos Moschetta (2):
>>   tests/acceptance: Add PVH boot test
>>   .travis.yml: Add kernel build deps for acceptance tests
>> 
>>  .travis.yml             |  2 ++
>>  tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 50 insertions(+)
>>  create mode 100644 tests/acceptance/pvh.py
>> 
>> -- 
>> 2.21.0
>> 


-- 
Alex Bennée