1. Patchew
  2. QEMU
  3. [Qemu-devel] [PATCH 0/7] Acceptance Tests: basic architecture support
  4. View patch

[Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures

Cleber Rosa posted 7 patches 7 years, 4 months ago
There is a newer version of this series
[Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Cleber Rosa 7 years, 4 months ago 
One of the Avocado features relevant to virtualization testing is the
ability to reuse tests in different scenarios, known as variants.
This adds a JSON based variants file, that can be used to run most
tests in a number of different architectures.  It can be run with:

   $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     --filter-by-tags='-x86_64' -- tests/acceptance/

Currently this covers 5 architectures, resulting in the execution
of 25 different combinations.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/variants/arch.json | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 tests/acceptance/variants/arch.json

diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json
new file mode 100644
index 0000000000..a7a2570553
--- /dev/null
+++ b/tests/acceptance/variants/arch.json
@@ -0,0 +1 @@
+[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}]
-- 
2.17.1
Re: [Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Laszlo Ersek 7 years, 4 months ago 
On 10/04/18 17:14, Cleber Rosa wrote:
> One of the Avocado features relevant to virtualization testing is the
> ability to reuse tests in different scenarios, known as variants.
> This adds a JSON based variants file, that can be used to run most
> tests in a number of different architectures.  It can be run with:
> 
>    $ avocado run \
>      --json-variants-load=tests/acceptance/variants/arch.json \
>      --filter-by-tags='-x86_64' -- tests/acceptance/
> 
> Currently this covers 5 architectures, resulting in the execution
> of 25 different combinations.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/variants/arch.json | 1 +
>  1 file changed, 1 insertion(+)
>  create mode 100644 tests/acceptance/variants/arch.json
> 
> diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json
> new file mode 100644
> index 0000000000..a7a2570553
> --- /dev/null
> +++ b/tests/acceptance/variants/arch.json
> @@ -0,0 +1 @@
> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}]
> 

(Side comment: you can parse json? :) That's awesome. Then I *am*
tempted to suggest that Philippe's work parse the firmware metadata
format, in the long run.)

Laszlo
Re: [Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Philippe Mathieu-Daudé 7 years, 4 months ago 
Hi Cleber,

On 04/10/2018 17:14, Cleber Rosa wrote:
> One of the Avocado features relevant to virtualization testing is the
> ability to reuse tests in different scenarios, known as variants.
> This adds a JSON based variants file, that can be used to run most
> tests in a number of different architectures.  It can be run with:
> 
>    $ avocado run \
>      --json-variants-load=tests/acceptance/variants/arch.json \
>      --filter-by-tags='-x86_64' -- tests/acceptance/
> 
> Currently this covers 5 architectures, resulting in the execution
> of 25 different combinations.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/variants/arch.json | 1 +
>  1 file changed, 1 insertion(+)
>  create mode 100644 tests/acceptance/variants/arch.json
> 
> diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json
> new file mode 100644
> index 0000000000..a7a2570553
> --- /dev/null
> +++ b/tests/acceptance/variants/arch.json
> @@ -0,0 +1 @@
> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}]
> 

Is this generated? (thinking about the other archs supported).

You should use some linter ;)
Re: [Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Eric Blake 7 years, 4 months ago 
On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 04/10/2018 17:14, Cleber Rosa wrote:
>> One of the Avocado features relevant to virtualization testing is the
>> ability to reuse tests in different scenarios, known as variants.
>> This adds a JSON based variants file, that can be used to run most
>> tests in a number of different architectures.  It can be run with:
>>
>>     $ avocado run \
>>       --json-variants-load=tests/acceptance/variants/arch.json \
>>       --filter-by-tags='-x86_64' -- tests/acceptance/

>> +++ b/tests/acceptance/variants/arch.json
>> @@ -0,0 +1 @@
>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64", "arch", "aarch64"]]]],"variant_id": "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc", "arch", "ppc"]]]],"variant_id": "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64", "arch", "ppc64"]]]],"variant_id": "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x", "arch", "s390x"]]]],"variant_id": "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64", "arch", "x86_64"]]]],"variant_id": "x86_64"}]
>>
> 
> Is this generated? (thinking about the other archs supported).
> 
> You should use some linter ;)

Also, that's a long line, which will probably get longer as more support 
is added. Beyond 990 bytes, it starts risking problems with corruption 
over email. It's also hard to view what changes incrementally if the 
single line changes. Is there a way to pretty-print things across 
multiple lines, for shorter lines and easier reading of future diffs?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Cleber Rosa 7 years, 4 months ago 

On 10/5/18 12:32 PM, Eric Blake wrote:
> On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote:
>> Hi Cleber,
>>
>> On 04/10/2018 17:14, Cleber Rosa wrote:
>>> One of the Avocado features relevant to virtualization testing is the
>>> ability to reuse tests in different scenarios, known as variants.
>>> This adds a JSON based variants file, that can be used to run most
>>> tests in a number of different architectures.  It can be run with:
>>>
>>>     $ avocado run \
>>>       --json-variants-load=tests/acceptance/variants/arch.json \
>>>       --filter-by-tags='-x86_64' -- tests/acceptance/
> 
>>> +++ b/tests/acceptance/variants/arch.json
>>> @@ -0,0 +1 @@
>>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64",
>>> "arch", "aarch64"]]]],"variant_id":
>>> "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc",
>>> "arch", "ppc"]]]],"variant_id":
>>> "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64",
>>> "arch", "ppc64"]]]],"variant_id":
>>> "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x",
>>> "arch", "s390x"]]]],"variant_id":
>>> "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64",
>>> "arch", "x86_64"]]]],"variant_id": "x86_64"}]
>>>
>>
>> Is this generated? (thinking about the other archs supported).
>>

It's generated and kept on every job result (jobdata/variants.json).
Basically, you'd use any varianter plugin on a job, and then you can
reuse the JSON generated on other jobs.

TBH, I tweaked this one a bit.

>> You should use some linter ;)

I missed your point here... do you mean the style is not ideal?

> 
> Also, that's a long line, which will probably get longer as more support
> is added. Beyond 990 bytes, it starts risking problems with corruption
> over email. It's also hard to view what changes incrementally if the
> single line changes. Is there a way to pretty-print things across
> multiple lines, for shorter lines and easier reading of future diffs?
> 

Yes, good point.  I'll pretty print it.

Just a disclaimer: I've chosen to use a JSON variants because it's a
core Avocado feature (doesn't require any external plugin), and the
results are 100% reproducible (the variants are static).  In the future,
we may consider also shipping (and depending) on other variants.

One idea that is being maturing (and prototype) is a native QEMU
varianter.  There's some info here:

https://trello.com/c/qW4kMw50/32-guest-abi-machine-type-cpu-model-test-cases

Regards,
- Cleber.
Re: [Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Philippe Mathieu-Daudé 7 years, 4 months ago 
On 05/10/2018 19:07, Cleber Rosa wrote:
> On 10/5/18 12:32 PM, Eric Blake wrote:
>> On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote:
>>> Hi Cleber,
>>>
>>> On 04/10/2018 17:14, Cleber Rosa wrote:
>>>> One of the Avocado features relevant to virtualization testing is the
>>>> ability to reuse tests in different scenarios, known as variants.
>>>> This adds a JSON based variants file, that can be used to run most
>>>> tests in a number of different architectures.  It can be run with:
>>>>
>>>>     $ avocado run \
>>>>       --json-variants-load=tests/acceptance/variants/arch.json \
>>>>       --filter-by-tags='-x86_64' -- tests/acceptance/
>>
>>>> +++ b/tests/acceptance/variants/arch.json
>>>> @@ -0,0 +1 @@
>>>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64",
>>>> "arch", "aarch64"]]]],"variant_id":
>>>> "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc",
>>>> "arch", "ppc"]]]],"variant_id":
>>>> "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64",
>>>> "arch", "ppc64"]]]],"variant_id":
>>>> "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x",
>>>> "arch", "s390x"]]]],"variant_id":
>>>> "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64",
>>>> "arch", "x86_64"]]]],"variant_id": "x86_64"}]
>>>>
>>>
>>> Is this generated? (thinking about the other archs supported).
>>>
> 
> It's generated and kept on every job result (jobdata/variants.json).
> Basically, you'd use any varianter plugin on a job, and then you can
> reuse the JSON generated on other jobs.
> 
> TBH, I tweaked this one a bit.
> 
>>> You should use some linter ;)
> 
> I missed your point here... do you mean the style is not ideal?

As meant "pretty printer" :)

[
  {
    "paths": [
      "/run/*"
    ],
    "variant": [
      [
        "/run/aarch64",
        [
          [
            "/run/aarch64",
            "arch",
            "aarch64"
          ]
        ]
      ]
    ],
    "variant_id": "aarch64"
  },
  ...

But since it is generated I'd rather generate it...

>>
>> Also, that's a long line, which will probably get longer as more support
>> is added. Beyond 990 bytes, it starts risking problems with corruption
>> over email. It's also hard to view what changes incrementally if the
>> single line changes. Is there a way to pretty-print things across
>> multiple lines, for shorter lines and easier reading of future diffs?
>>
> 
> Yes, good point.  I'll pretty print it.
> 
> Just a disclaimer: I've chosen to use a JSON variants because it's a
> core Avocado feature (doesn't require any external plugin), and the
> results are 100% reproducible (the variants are static).  In the future,
> we may consider also shipping (and depending) on other variants.
> 
> One idea that is being maturing (and prototype) is a native QEMU
> varianter.  There's some info here:
> 
> https://trello.com/c/qW4kMw50/32-guest-abi-machine-type-cpu-model-test-cases
> 
> Regards,
> - Cleber.
>
Re: [Qemu-devel] [PATCH 6/7] Acceptance Tests: add variants definition for architectures
Posted by Cleber Rosa 7 years, 4 months ago 

On 10/5/18 1:30 PM, Philippe Mathieu-Daudé wrote:
> On 05/10/2018 19:07, Cleber Rosa wrote:
>> On 10/5/18 12:32 PM, Eric Blake wrote:
>>> On 10/5/18 11:24 AM, Philippe Mathieu-Daudé wrote:
>>>> Hi Cleber,
>>>>
>>>> On 04/10/2018 17:14, Cleber Rosa wrote:
>>>>> One of the Avocado features relevant to virtualization testing is the
>>>>> ability to reuse tests in different scenarios, known as variants.
>>>>> This adds a JSON based variants file, that can be used to run most
>>>>> tests in a number of different architectures.  It can be run with:
>>>>>
>>>>>     $ avocado run \
>>>>>       --json-variants-load=tests/acceptance/variants/arch.json \
>>>>>       --filter-by-tags='-x86_64' -- tests/acceptance/
>>>
>>>>> +++ b/tests/acceptance/variants/arch.json
>>>>> @@ -0,0 +1 @@
>>>>> +[{"paths":["/run/*"],"variant":[["/run/aarch64",[["/run/aarch64",
>>>>> "arch", "aarch64"]]]],"variant_id":
>>>>> "aarch64"},{"paths":["/run/*"],"variant":[["/run/ppc",[["/run/ppc",
>>>>> "arch", "ppc"]]]],"variant_id":
>>>>> "ppc"},{"paths":["/run/*"],"variant":[["/run/ppc64",[["/run/ppc64",
>>>>> "arch", "ppc64"]]]],"variant_id":
>>>>> "ppc64"},{"paths":["/run/*"],"variant":[["/run/s390x",[["/run/s390x",
>>>>> "arch", "s390x"]]]],"variant_id":
>>>>> "s390x"},{"paths":["/run/*"],"variant":[["/run/x86_64",[["/run/x86_64",
>>>>> "arch", "x86_64"]]]],"variant_id": "x86_64"}]
>>>>>
>>>>
>>>> Is this generated? (thinking about the other archs supported).
>>>>
>>
>> It's generated and kept on every job result (jobdata/variants.json).
>> Basically, you'd use any varianter plugin on a job, and then you can
>> reuse the JSON generated on other jobs.
>>
>> TBH, I tweaked this one a bit.
>>
>>>> You should use some linter ;)
>>
>> I missed your point here... do you mean the style is not ideal?
> 
> As meant "pretty printer" :)
> 
> [
>   {
>     "paths": [
>       "/run/*"
>     ],
>     "variant": [
>       [
>         "/run/aarch64",
>         [
>           [
>             "/run/aarch64",
>             "arch",
>             "aarch64"
>           ]
>         ]
>       ]
>     ],
>     "variant_id": "aarch64"
>   },
>   ...
> 
> But since it is generated I'd rather generate it...

Actually, I think it's a good idea indeed to pretty print it, "python -m
json.tool" should do the job.

Thanks,
- Cleber.

> 
>>>
>>> Also, that's a long line, which will probably get longer as more support
>>> is added. Beyond 990 bytes, it starts risking problems with corruption
>>> over email. It's also hard to view what changes incrementally if the
>>> single line changes. Is there a way to pretty-print things across
>>> multiple lines, for shorter lines and easier reading of future diffs?
>>>
>>
>> Yes, good point.  I'll pretty print it.
>>
>> Just a disclaimer: I've chosen to use a JSON variants because it's a
>> core Avocado feature (doesn't require any external plugin), and the
>> results are 100% reproducible (the variants are static).  In the future,
>> we may consider also shipping (and depending) on other variants.
>>
>> One idea that is being maturing (and prototype) is a native QEMU
>> varianter.  There's some info here:
>>
>> https://trello.com/c/qW4kMw50/32-guest-abi-machine-type-cpu-model-test-cases
>>
>> Regards,
>> - Cleber.
>>

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.