qgraph API for the qtest driver framework

This series of patches introduce a different qtest driver 
organization, viewing machines, drivers and tests as node in a 
graph, each having one or multiple edges relations.

The idea is to have a framework where each test asks for a specific
driver, and the framework takes care of allocating the proper devices
required and passing the correct command line arguments to QEMU.

A node can be of four types:
- MACHINE:   for example "arm/raspi2"
- DRIVER:    for example "generic-sdhci"
- INTERFACE: for example "sdhci" (interface for all "-sdhci" drivers)
- TEST:      for example "sdhci-test", consumes an interface and tests
             the functions provided

An edge relation between two nodes (drivers or machines) X and Y can be:
- X CONSUMES Y: Y can be plugged into X
- X PRODUCES Y: X provides the interface Y
- X CONTAINS Y: Y is part of X component

Basic framework steps are the following:
- All nodes and edges are created in their respective machine/driver/test files
- The framework starts QEMU and asks for a list of available drivers
  and machines
- The framework walks the graph starting from the available machines and
  performs a Depth First Search for tests
- Once a test is found, the path is walked again and all drivers are
  allocated accordingly and the final interface is passed to the test
- The test is executed
- Unused objects are cleaned and the path discovery is continued

Depending on the QEMU binary used, only some drivers/machines will be available
and only test that are reached by them will be executed.

This work is being done as Google Summer of Code 2018 project for QEMU,
my mentors are Paolo Bonzini and Laurent Vivier.
Additional infos on the project can be found at:
https://wiki.qemu.org/Features/qtest_driver_framework

Emanuele Giuseppe Esposito (7):
  tests: qgraph API for the qtest driver framework
  tests/qgraph: pci-pc driver and interface nodes
  tests/qgraph: sdhci driver and interface nodes
  tests/qgraph: arm/raspi2 machine node
  tests/qgraph: x86_64/pc machine node
  tests/qgraph: gtest integration
  tests/qgraph: sdhci test node

 configure                        |   2 +-
 include/qemu/module.h            |   2 +
 tests/Makefile.include           |  16 +-
 tests/libqos/pci-pc.c            |  53 ++-
 tests/libqos/pci-pc.h            |   8 +
 tests/libqos/pci.c               |   8 +
 tests/libqos/qgraph.c            | 676 +++++++++++++++++++++++++++++++
 tests/libqos/qgraph.h            | 259 ++++++++++++
 tests/libqos/qgraph_extra.h      | 155 +++++++
 tests/libqos/raspi2-machine.c    |  68 ++++
 tests/libqos/sdhci.c             | 142 +++++++
 tests/libqos/sdhci.h             |  68 ++++
 tests/libqos/x86_64_pc-machine.c |  93 +++++
 tests/qos-test.c                 | 310 ++++++++++++++
 tests/sdhci-test.c               | 222 +++-------
 tests/test-qgraph.c              | 446 ++++++++++++++++++++
 16 files changed, 2357 insertions(+), 171 deletions(-)
 create mode 100644 tests/libqos/qgraph.c
 create mode 100644 tests/libqos/qgraph.h
 create mode 100644 tests/libqos/qgraph_extra.h
 create mode 100644 tests/libqos/raspi2-machine.c
 create mode 100644 tests/libqos/sdhci.c
 create mode 100644 tests/libqos/sdhci.h
 create mode 100644 tests/libqos/x86_64_pc-machine.c
 create mode 100644 tests/qos-test.c
 create mode 100644 tests/test-qgraph.c

-- 
2.17.1

On Mon, Jul 09, 2018 at 11:11:29AM +0200, Emanuele Giuseppe Esposito wrote:
> Basic framework steps are the following:
> - All nodes and edges are created in their respective machine/driver/test files
> - The framework starts QEMU and asks for a list of available drivers
>   and machines

QEMU doesn't know about qtest drivers.  Does "drivers" mean "devices"
(i.e. qemu -device \?) and they have a 1:1 correspondence with qtest
drivers?

The concept of qgraph makes sense.  Ease of use and documentation will
be critical to gain adoption.  The extra complexity makes it hard for
people writing and running tests, so it's important that it's clear
which tests will get run and why (or why not).

On 07/11/2018 04:00 PM, Stefan Hajnoczi wrote:

> On Mon, Jul 09, 2018 at 11:11:29AM +0200, Emanuele Giuseppe Esposito wrote:
>> Basic framework steps are the following:
>> - All nodes and edges are created in their respective machine/driver/test files
>> - The framework starts QEMU and asks for a list of available drivers
>>    and machines
> QEMU doesn't know about qtest drivers.  Does "drivers" mean "devices"
> (i.e. qemu -device \?) and they have a 1:1 correspondence with qtest
> drivers?
Yes, they are QEMU devices, it was a typo.
Not all qtest drivers are mapped with the QEMU devices, though. In fact, 
for now all drivers "contained" or "produced" by other nodes are not 
QEMU devices.

The others (machines and nodes that "consume"), are mapped to QEMU using 
the
{ "execute": "qom-list-types", "arguments" : { "implements" : "device" } }
for "consume" and
{ "execute": "query-machines" }
for machines.

With "consume" I mean in situation like "node X" ---> consumes --> "node 
Y",
"X" should be in the output of qom-list-types query.
> The concept of qgraph makes sense.  Ease of use and documentation will
> be critical to gain adoption.  The extra complexity makes it hard for
> people writing and running tests, so it's important that it's clear
> which tests will get run and why (or why not).
I am trying to document and makes it as easy and clean as possible :)

On Mon, Jul 09, 2018 at 11:11:29AM +0200, Emanuele Giuseppe Esposito wrote:
> This work is being done as Google Summer of Code 2018 project for QEMU,
> my mentors are Paolo Bonzini and Laurent Vivier.
> Additional infos on the project can be found at:
> https://wiki.qemu.org/Features/qtest_driver_framework

Thanks, I've finished reviewing this version.  It looks like a good
start!

The main challenge to me seems "how can we make tests simpler?".  The
presence of a new API and object model raises the bar for writing and
running tests.  I hope all qtests will use qgraph but if the complexity
is too high then qgraph may only be used in a few niche cases where
authors think it's worth abstracting machine types and others will
continue to write qtests without qgraph.

What are the next steps and do you have plans for making qgraph more
integral to libqos?

Stefan

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Mon, Jul 09, 2018 at 11:11:29AM +0200, Emanuele Giuseppe Esposito wrote:
>> This work is being done as Google Summer of Code 2018 project for QEMU,
>> my mentors are Paolo Bonzini and Laurent Vivier.
>> Additional infos on the project can be found at:
>> https://wiki.qemu.org/Features/qtest_driver_framework
>
> Thanks, I've finished reviewing this version.  It looks like a good
> start!
>
> The main challenge to me seems "how can we make tests simpler?".  The
> presence of a new API and object model raises the bar for writing and
> running tests.  I hope all qtests will use qgraph but if the complexity
> is too high then qgraph may only be used in a few niche cases where
> authors think it's worth abstracting machine types and others will
> continue to write qtests without qgraph.

The #1 reason for writing code a certain way is following existing
examples.  As long as "qtest without qgraph" is prevalent in existing
tests, it'll remain prevalent in new tests.  I'm afraid we'll need a
conversion effort.

> What are the next steps and do you have plans for making qgraph more
> integral to libqos?

Good questions.

On 18/07/2018 19:14, Markus Armbruster wrote:
>> The main challenge to me seems "how can we make tests simpler?".  The
>> presence of a new API and object model raises the bar for writing and
>> running tests.  I hope all qtests will use qgraph but if the complexity
>> is too high then qgraph may only be used in a few niche cases where
>> authors think it's worth abstracting machine types and others will
>> continue to write qtests without qgraph.
> The #1 reason for writing code a certain way is following existing
> examples.  As long as "qtest without qgraph" is prevalent in existing
> tests, it'll remain prevalent in new tests.  I'm afraid we'll need a
> conversion effort.
> 

Indeed, Emanuele is already working on converting virtio-serial tests
(virtio is possibly the largest part of libqos).  The various nop tests
are also more or less trivial to adjust.

Paolo