The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should maybe try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset so
far, but this already proves that many tests only need small modifi-
cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
classes (e.g. based on cloud-init images or using SSH) really depend
on the Avocado framework, so we'd need a solution for those if we
want to continue using them. One solution might be to simply use the
required functions from avocado.utils for these tests, and still run
them via the meson test runner instead, but that needs some further
investigation that will be done later.


Now if you want to try out these patches: Apply the patches, then
recompile and then run:

 make check-functional

You can also run single targets e.g. with:

 make check-functional-ppc

You can also run the tests without any test runner now by
setting the PYTHONPATH environment variable to the "python" folder
of your source tree, and by specifying the build directory via
QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

 export PYTHONPATH=$HOME/qemu/python
 export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
 export PYTHONPATH=$HOME/qemu/build
 ~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under
tests/functional/<testname> - console log and general logs will
be put in separate files there.

Still to be done: Update the documentation for this new test framework.

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so
  many things that I also did not dare to pick up the Reviewed-bys
  from the RFC

Thomas Huth (11):
  tests/functional: Add base classes for the upcoming pytest-based tests
  tests/functional: Convert simple avocado tests into standalone python
    tests
  tests/functional: Convert avocado tests that just need a small
    adjustment
  tests/functional: Add python-based tests to the meson build system
  tests/functional: Implement fetch_asset() method for downloading
    assets
  tests/functional: Convert some tests that download files via
    fetch_asset()
  tests/functional: Add a function for extracting files from an archive
  tests/functional: Convert some avocado tests that needed
    avocado.utils.archive
  tests/functional: Set up logging
  tests/functional: Convert the s390x avocado tests into standalone
    tests
  gitlab-ci: Add "check-functional" to the build tests

 MAINTAINERS                                   |  22 +-
 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +--
 pythondeps.toml                               |   3 +-
 tests/Makefile.include                        |  18 +-
 tests/functional/meson.build                  | 112 +++++
 tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  28 ++
 .../test_arm_canona1100.py}                   |  21 +-
 .../test_arm_n8x0.py}                         |  25 +-
 .../test_avr_mega2560.py}                     |  11 +-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  16 +-
 .../test_mem_addr_space.py}                   |  53 +--
 .../test_mips64el_loongson3v.py}              |  26 +-
 .../test_netdev_ethtool.py}                   |  32 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
 .../test_ppc_74xx.py}                         |  74 ++--
 .../test_ppc_bamboo.py}                       |  23 +-
 .../test_s390x_ccw_virtio.py}                 |  32 +-
 .../test_s390x_topology.py}                   |  70 ++--
 .../test_sparc64_sun4u.py}                    |  25 +-
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_version.py}                   |   8 +-
 tests/meson.build                             |   1 +
 28 files changed, 795 insertions(+), 320 deletions(-)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
 mode change 100644 => 100755

-- 
2.45.2

On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
> 
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
> 
> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
> 
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
> 
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
> 
> 
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
> 
>  make check-functional
> 
> You can also run single targets e.g. with:
> 
>  make check-functional-ppc
> 
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
> 
>  export PYTHONPATH=$HOME/qemu/python
>  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>  export PYTHONPATH=$HOME/qemu/build
>  ~/qemu/tests/functional/test_virtio_version.py

For the whole series as is

 Tested-by: Daniel P. Berrangé <berrange@redhat.com>

as it does what you claim it does here when I tried it.

> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.

As an example, one dir name appears to be:

  __main__.MemAddrCheck.test_phybits_ok_pentium_pae

I'd rather prefer it if the dir name matched the test script
file name - in this case test_mem_addr_space.py, as I don't
want to have to lookup which class names were defined inside
each test script. We could drop the "test_" prefix from the
method name too

IOW, could we make this dir name be:

  test_mem_addr_space.phybits_ok_pentium_pae


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
...
>> So instead of trying to update the python-based test suite in QEMU
>> to a newer version of Avocado, we should maybe try to better integrate
>> it with the meson test runner instead. Indeed most tests work quite
>> nicely without the Avocado framework already, as you can see with
>> this patch series - it does not convert all tests, just a subset so
>> far, but this already proves that many tests only need small modifi-
>> cations to work without Avocado.
...
>> Now if you want to try out these patches: Apply the patches, then
>> recompile and then run:
>>
>>   make check-functional
>>
>> You can also run single targets e.g. with:
>>
>>   make check-functional-ppc
>>
>> You can also run the tests without any test runner now by
>> setting the PYTHONPATH environment variable to the "python" folder
>> of your source tree, and by specifying the build directory via
>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>
>>   export PYTHONPATH=$HOME/qemu/python
>>   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>   export PYTHONPATH=$HOME/qemu/build
>>   ~/qemu/tests/functional/test_virtio_version.py
> 
> For the whole series as is
> 
>   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
> 
> as it does what you claim it does here when I tried it.

Thanks!

>> The logs of the tests can be found in the build directory under
>> tests/functional/<testname> - console log and general logs will
>> be put in separate files there.
> 
> As an example, one dir name appears to be:
> 
>    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
> 
> I'd rather prefer it if the dir name matched the test script
> file name - in this case test_mem_addr_space.py, as I don't
> want to have to lookup which class names were defined inside
> each test script. We could drop the "test_" prefix from the
> method name too
> 
> IOW, could we make this dir name be:
> 
>    test_mem_addr_space.phybits_ok_pentium_pae

I can try to change that, indeed ... but the boilerplate code will increase 
a little bit, I guess, since I cannot simply rely on the unittest.id() 
function in that case anymore...

  Thomas

On 16/07/2024 19.03, Thomas Huth wrote:
> On 16/07/2024 18.51, Daniel P. Berrangé wrote:
>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> ...
>>> So instead of trying to update the python-based test suite in QEMU
>>> to a newer version of Avocado, we should maybe try to better integrate
>>> it with the meson test runner instead. Indeed most tests work quite
>>> nicely without the Avocado framework already, as you can see with
>>> this patch series - it does not convert all tests, just a subset so
>>> far, but this already proves that many tests only need small modifi-
>>> cations to work without Avocado.
> ...
>>> Now if you want to try out these patches: Apply the patches, then
>>> recompile and then run:
>>>
>>>   make check-functional
>>>
>>> You can also run single targets e.g. with:
>>>
>>>   make check-functional-ppc
>>>
>>> You can also run the tests without any test runner now by
>>> setting the PYTHONPATH environment variable to the "python" folder
>>> of your source tree, and by specifying the build directory via
>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>>
>>>   export PYTHONPATH=$HOME/qemu/python
>>>   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>>   export PYTHONPATH=$HOME/qemu/build
>>>   ~/qemu/tests/functional/test_virtio_version.py
>>
>> For the whole series as is
>>
>>   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>>
>> as it does what you claim it does here when I tried it.
> 
> Thanks!
> 
>>> The logs of the tests can be found in the build directory under
>>> tests/functional/<testname> - console log and general logs will
>>> be put in separate files there.
>>
>> As an example, one dir name appears to be:
>>
>>    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>>
>> I'd rather prefer it if the dir name matched the test script
>> file name - in this case test_mem_addr_space.py, as I don't
>> want to have to lookup which class names were defined inside
>> each test script. We could drop the "test_" prefix from the
>> method name too
>>
>> IOW, could we make this dir name be:
>>
>>    test_mem_addr_space.phybits_ok_pentium_pae
> 
> I can try to change that, indeed ... but the boilerplate code will increase 
> a little bit, I guess, since I cannot simply rely on the unittest.id() 
> function in that case anymore...

After looking at this for a while, I think it's maybe best to ditch the idea 
of making the .py files directly runnable and run the tests via a simple 
pycotap runner instead. Then you get proper module names:

$ pyvenv/bin/python3 -m pycotap test_virtio_version
TAP version 13
ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
1..2

  Thomas

On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
> On 16/07/2024 19.03, Thomas Huth wrote:
> > On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> > > On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> > ...
> > > > So instead of trying to update the python-based test suite in QEMU
> > > > to a newer version of Avocado, we should maybe try to better integrate
> > > > it with the meson test runner instead. Indeed most tests work quite
> > > > nicely without the Avocado framework already, as you can see with
> > > > this patch series - it does not convert all tests, just a subset so
> > > > far, but this already proves that many tests only need small modifi-
> > > > cations to work without Avocado.
> > ...
> > > > Now if you want to try out these patches: Apply the patches, then
> > > > recompile and then run:
> > > > 
> > > >   make check-functional
> > > > 
> > > > You can also run single targets e.g. with:
> > > > 
> > > >   make check-functional-ppc
> > > > 
> > > > You can also run the tests without any test runner now by
> > > > setting the PYTHONPATH environment variable to the "python" folder
> > > > of your source tree, and by specifying the build directory via
> > > > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> > > > QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
> > > > 
> > > >   export PYTHONPATH=$HOME/qemu/python
> > > >   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> > > >   export PYTHONPATH=$HOME/qemu/build
> > > >   ~/qemu/tests/functional/test_virtio_version.py
> > > 
> > > For the whole series as is
> > > 
> > >   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > as it does what you claim it does here when I tried it.
> > 
> > Thanks!
> > 
> > > > The logs of the tests can be found in the build directory under
> > > > tests/functional/<testname> - console log and general logs will
> > > > be put in separate files there.
> > > 
> > > As an example, one dir name appears to be:
> > > 
> > >    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
> > > 
> > > I'd rather prefer it if the dir name matched the test script
> > > file name - in this case test_mem_addr_space.py, as I don't
> > > want to have to lookup which class names were defined inside
> > > each test script. We could drop the "test_" prefix from the
> > > method name too
> > > 
> > > IOW, could we make this dir name be:
> > > 
> > >    test_mem_addr_space.phybits_ok_pentium_pae
> > 
> > I can try to change that, indeed ... but the boilerplate code will
> > increase a little bit, I guess, since I cannot simply rely on the
> > unittest.id() function in that case anymore...
> 
> After looking at this for a while, I think it's maybe best to ditch the idea
> of making the .py files directly runnable and run the tests via a simple
> pycotap runner instead. Then you get proper module names:

I'd really not want to loose that. To me, eliminating the test harness
entirely when debugging is the single biggest improvement of this new
approach, especially when I want to 'strace' the test without
extraneous processes.

> $ pyvenv/bin/python3 -m pycotap test_virtio_version
> TAP version 13
> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
> 1..

With the following change, you get the same output with direct
execution, by making argv look the same as you'd get when
running your pycotap example.

diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
index cc49fd4c94..3a3e65252d 100644
--- a/tests/functional/qemu_test/__init__.py
+++ b/tests/functional/qemu_test/__init__.py
@@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        unittest.main(testRunner = tr)
+        import sys
+        import os.path
+        path = os.path.basename(sys.argv[0])[:-3]
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
 
 
 class QemuSystemTest(QemuBaseTest):

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 17/07/2024 10.37, Daniel P. Berrangé wrote:
> On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
>> On 16/07/2024 19.03, Thomas Huth wrote:
>>> On 16/07/2024 18.51, Daniel P. Berrangé wrote:
>>>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
>>> ...
>>>>> So instead of trying to update the python-based test suite in QEMU
>>>>> to a newer version of Avocado, we should maybe try to better integrate
>>>>> it with the meson test runner instead. Indeed most tests work quite
>>>>> nicely without the Avocado framework already, as you can see with
>>>>> this patch series - it does not convert all tests, just a subset so
>>>>> far, but this already proves that many tests only need small modifi-
>>>>> cations to work without Avocado.
>>> ...
>>>>> Now if you want to try out these patches: Apply the patches, then
>>>>> recompile and then run:
>>>>>
>>>>>    make check-functional
>>>>>
>>>>> You can also run single targets e.g. with:
>>>>>
>>>>>    make check-functional-ppc
>>>>>
>>>>> You can also run the tests without any test runner now by
>>>>> setting the PYTHONPATH environment variable to the "python" folder
>>>>> of your source tree, and by specifying the build directory via
>>>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>>>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>>>>
>>>>>    export PYTHONPATH=$HOME/qemu/python
>>>>>    export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>>>>    export PYTHONPATH=$HOME/qemu/build
>>>>>    ~/qemu/tests/functional/test_virtio_version.py
>>>>
>>>> For the whole series as is
>>>>
>>>>    Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>
>>>> as it does what you claim it does here when I tried it.
>>>
>>> Thanks!
>>>
>>>>> The logs of the tests can be found in the build directory under
>>>>> tests/functional/<testname> - console log and general logs will
>>>>> be put in separate files there.
>>>>
>>>> As an example, one dir name appears to be:
>>>>
>>>>     __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>>>>
>>>> I'd rather prefer it if the dir name matched the test script
>>>> file name - in this case test_mem_addr_space.py, as I don't
>>>> want to have to lookup which class names were defined inside
>>>> each test script. We could drop the "test_" prefix from the
>>>> method name too
>>>>
>>>> IOW, could we make this dir name be:
>>>>
>>>>     test_mem_addr_space.phybits_ok_pentium_pae
>>>
>>> I can try to change that, indeed ... but the boilerplate code will
>>> increase a little bit, I guess, since I cannot simply rely on the
>>> unittest.id() function in that case anymore...
>>
>> After looking at this for a while, I think it's maybe best to ditch the idea
>> of making the .py files directly runnable and run the tests via a simple
>> pycotap runner instead. Then you get proper module names:
> 
> I'd really not want to loose that. To me, eliminating the test harness
> entirely when debugging is the single biggest improvement of this new
> approach, especially when I want to 'strace' the test without
> extraneous processes.
> 
>> $ pyvenv/bin/python3 -m pycotap test_virtio_version
>> TAP version 13
>> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
>> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
>> 1..
> 
> With the following change, you get the same output with direct
> execution, by making argv look the same as you'd get when
> running your pycotap example.
> 
> diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
> index cc49fd4c94..3a3e65252d 100644
> --- a/tests/functional/qemu_test/__init__.py
> +++ b/tests/functional/qemu_test/__init__.py
> @@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
>       def main():
>           tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
>                                      test_output_log = pycotap.LogMode.LogToError)
> -        unittest.main(testRunner = tr)
> +        import sys
> +        import os.path
> +        path = os.path.basename(sys.argv[0])[:-3]
> +        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])

Sweet, thank you very much, looks like this will do the job!

  Thomas

On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
>
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>

Hi Thomas,

Let me start by saying that this is my attempt to explain the context
of the lack of updates on QEMU when it comes to the Avocado update,
and point to the actual updates.

> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>

So, we've seen in the past an attempt to update Avocado from 88.1 to a
regular release, and the troubles it caused, including a revert.  My
take was that a LTS version should be used, but during this time,
Avocado experienced a rewrite and having it replacing the old
implementation in a production level project such as QEMU was tricky.
Then 103.0 LTS was released, and there was extensive work to test the
QEMU tests before that release was cut.  Additionally, there was
further work, but unfortunately not posted yet, to make use of 103.0
features in the existing tests[2].   I've tested on GitLab with tests
running in parallel, cutting job times in 1/3[2].  A side node is
that, because 103.0 is an LTS release, it will receive the needed bug
fixes and updates that we deem necessary, including things we find in
QEMU tests.  In fact, 103.1[3] is in the works.

> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>

Now I believe we can be very much in sync here.  I've thought for a
while that there's no reason for Avocado to cooperate or be compatible
with Meson.  There's no reason why users can't simply pick how the
test gets run.  In fact, with the new Avocado architecture, you don't
even need to run "avocado" to run an "avocado-instrumented" test.  You
could pretty much run "avocado-runner-avocado-instrumented" with the
right parameters through Meson.

> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>

So, I believe this type of higher level testing is something that
needs to remain, and even grow.  Speaking for Red Hat, I see the
movement of QE contributing more Avocado-VT style tests into QEMU
itself.  This means way more libraries and features that go into a
common set of utilities and features (more on that later) than it
currently exists in avocado.utils.

This brings the autils[4] initiative into the picture.  We're about
80% done with the project structure, and after that, it will be a
common utility project (such as the cloudinit and ssh) which can be
released automatically when the maintainer votes (through GitHub) that
a new release is needed.  With that, we can have all the existing QEMU
tests (and more) running in either meson or Avocado (for users that
want Avocado runner only features, such as dependency resolution,
container support, etc).

Now, if you think any of this makes sense, I'd love to cooperate with
you on an update that does both: update Avocado, and allow for the
execution of all tests through meson.

Thanks and I'm very very sorry for the delay in this update.
- Cleber.

[1] https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/avocado_test_updates_lts?ref_type=heads
[2] https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478
[3] https://github.com/avocado-framework/avocado/pull/5953
[4] https://github.com/avocado-framework/autils

On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote:
> On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> > There have been several attempts to update the test suite in QEMU
> > to a newer version of Avocado, but so far no attempt has successfully
> > been merged yet.
> >
> 
> So, we've seen in the past an attempt to update Avocado from 88.1 to a
> regular release, and the troubles it caused, including a revert.  My
> take was that a LTS version should be used, but during this time,
> Avocado experienced a rewrite and having it replacing the old
> implementation in a production level project such as QEMU was tricky.

IMHO the problems with updating avocado in QEMU are a sign that we're
suffering from excess complexity, as it shouldn't be this difficult to
update the test harness. That we need to stick on an LTS release to
protect us from further instability further suggests that we would
benefit from a much simpler solution here. 

> > Additionally, the whole "make check" test suite in QEMU is using the
> > meson test runner nowadays, so running the python-based tests via the
> > Avocodo test runner looks and feels quite like an oddball, requiring
> > the users to deal with the knowledge of multiple test runners in
> > parallel (e.g. the timeout settings work completely differently).
> >
> 
> Now I believe we can be very much in sync here.  I've thought for a
> while that there's no reason for Avocado to cooperate or be compatible
> with Meson.  There's no reason why users can't simply pick how the
> test gets run.  In fact, with the new Avocado architecture, you don't
> even need to run "avocado" to run an "avocado-instrumented" test.  You
> could pretty much run "avocado-runner-avocado-instrumented" with the
> right parameters through Meson.

Looking back to when we introduced avocado, we still had our original
home grown build system, and multiple different ways to run integrate
testing betwen our unit tests / qtests and block iotests. Adding
avocado to the mix introduced a third way to run tests. All round it
was rather a mess of inconsistent approaches to testing, but that was
a fact of life with our old home grown build/test process.

Today we have adopted a much more standardized build system via meson.
While we do still have the block iotests harness, we now expose every
test to meson individually to reduce what the intermediate harness
needs to be involved in doing. Thomas' series here is bringing the same
simplified integration to our functional tests, such that each functional
test is known to meson individually. Overall we'll now have a consistent
approach to integrating tests into meson, without intermediate harnesses
adding their own logic.

I think this will make it easier for QEMU maintainers to understand how
the tests are working, and make it easier to maintain them and troubleshoot
failures.

So overall I'm not seeing a compelling reason for us to offer users a
choice between meson & avocado how they run tests. Such choice is a maint
burden over the long term, as we need to validate that both options keep
working. Focusing on a single option reduces the complexity and maint
work and gives us more flexiblity to evolve the code over time.


> > So instead of trying to update the python-based test suite in QEMU
> > to a newer version of Avocado, we should maybe try to better integrate
> > it with the meson test runner instead. Indeed most tests work quite
> > nicely without the Avocado framework already, as you can see with
> > this patch series - it does not convert all tests, just a subset so
> > far, but this already proves that many tests only need small modifi-
> > cations to work without Avocado.
> >
> > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> > classes (e.g. based on cloud-init images or using SSH) really depend
> > on the Avocado framework, so we'd need a solution for those if we
> > want to continue using them. One solution might be to simply use the
> > required functions from avocado.utils for these tests, and still run
> > them via the meson test runner instead, but that needs some further
> > investigation that will be done later.
> >
> 
> So, I believe this type of higher level testing is something that
> needs to remain, and even grow.  Speaking for Red Hat, I see the
> movement of QE contributing more Avocado-VT style tests into QEMU
> itself.  This means way more libraries and features that go into a
> common set of utilities and features (more on that later) than it
> currently exists in avocado.utils.

The series here is showing how simple it is to adapt our existing
tests to the new execution approach, so shouldn't have a significant
impact on ability of people to contribute futher tests in future.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Mon, Jul 29, 2024 at 8:44 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote:
> > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> > > There have been several attempts to update the test suite in QEMU
> > > to a newer version of Avocado, but so far no attempt has successfully
> > > been merged yet.
> > >
> >
> > So, we've seen in the past an attempt to update Avocado from 88.1 to a
> > regular release, and the troubles it caused, including a revert.  My
> > take was that a LTS version should be used, but during this time,
> > Avocado experienced a rewrite and having it replacing the old
> > implementation in a production level project such as QEMU was tricky.
>
> IMHO the problems with updating avocado in QEMU are a sign that we're
> suffering from excess complexity, as it shouldn't be this difficult to
> update the test harness. That we need to stick on an LTS release to
> protect us from further instability further suggests that we would
> benefit from a much simpler solution here.
>

Hi Daniel,

Some other projects using Avocado have picked different, non-LTS
versions, to use as their pinned versions.  The consequences of using
rolling releases versus LTS-like releases is not something specific to
one software package IMO.  The suggestion to use LTS was to minimize
the impact on users writing and running tests, while allowing for only
needed fixes or very important features (say supporting a new Python
version) to be addressed.

> > > Additionally, the whole "make check" test suite in QEMU is using the
> > > meson test runner nowadays, so running the python-based tests via the
> > > Avocodo test runner looks and feels quite like an oddball, requiring
> > > the users to deal with the knowledge of multiple test runners in
> > > parallel (e.g. the timeout settings work completely differently).
> > >
> >
> > Now I believe we can be very much in sync here.  I've thought for a
> > while that there's no reason for Avocado to cooperate or be compatible
> > with Meson.  There's no reason why users can't simply pick how the
> > test gets run.  In fact, with the new Avocado architecture, you don't
> > even need to run "avocado" to run an "avocado-instrumented" test.  You
> > could pretty much run "avocado-runner-avocado-instrumented" with the
> > right parameters through Meson.
>
> Looking back to when we introduced avocado, we still had our original
> home grown build system, and multiple different ways to run integrate
> testing betwen our unit tests / qtests and block iotests. Adding
> avocado to the mix introduced a third way to run tests. All round it
> was rather a mess of inconsistent approaches to testing, but that was
> a fact of life with our old home grown build/test process.
>

Agreed.

> Today we have adopted a much more standardized build system via meson.
> While we do still have the block iotests harness, we now expose every
> test to meson individually to reduce what the intermediate harness
> needs to be involved in doing. Thomas' series here is bringing the same
> simplified integration to our functional tests, such that each functional
> test is known to meson individually. Overall we'll now have a consistent
> approach to integrating tests into meson, without intermediate harnesses
> adding their own logic.
>
> I think this will make it easier for QEMU maintainers to understand how
> the tests are working, and make it easier to maintain them and troubleshoot
> failures.
>

I think this is a very good way forward.  I don't think anything
(simple tests here) should depend on what they don't need to depend
on.

> So overall I'm not seeing a compelling reason for us to offer users a
> choice between meson & avocado how they run tests. Such choice is a maint
> burden over the long term, as we need to validate that both options keep
> working. Focusing on a single option reduces the complexity and maint
> work and gives us more flexiblity to evolve the code over time.
>
>

If meson is capable of doing all that can be done right now, then I
believe you're right.  No reason to use duplicate software.  Now, If
you mean that having "single option" may imply duplicating code that
already exists and is maintained elsewhere, into the QEMU repo, then
you're wrong.  This can be easily judged by how easy it is to convert
all current tests and their features (timeout, download management,
etc).

Another level of judgement that I believe would be helpful to the
decision making process here is how this impacts growing the test
coverage.  Tests that manipulate both QEMU and the guest are still
very very simple compared to what can be found on what I'll call "real
life integration testing" of QEMU (read Avocado-VT like tests). But
they present a glimpse of the problem. Having such tests requires a
good amount of code to either be used from a project/repo that
addresses that specifically, or be duplicated inside QEMU.

> > > So instead of trying to update the python-based test suite in QEMU
> > > to a newer version of Avocado, we should maybe try to better integrate
> > > it with the meson test runner instead. Indeed most tests work quite
> > > nicely without the Avocado framework already, as you can see with
> > > this patch series - it does not convert all tests, just a subset so
> > > far, but this already proves that many tests only need small modifi-
> > > cations to work without Avocado.
> > >
> > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> > > classes (e.g. based on cloud-init images or using SSH) really depend
> > > on the Avocado framework, so we'd need a solution for those if we
> > > want to continue using them. One solution might be to simply use the
> > > required functions from avocado.utils for these tests, and still run
> > > them via the meson test runner instead, but that needs some further
> > > investigation that will be done later.
> > >
> >
> > So, I believe this type of higher level testing is something that
> > needs to remain, and even grow.  Speaking for Red Hat, I see the
> > movement of QE contributing more Avocado-VT style tests into QEMU
> > itself.  This means way more libraries and features that go into a
> > common set of utilities and features (more on that later) than it
> > currently exists in avocado.utils.
>
> The series here is showing how simple it is to adapt our existing
> tests to the new execution approach, so shouldn't have a significant
> impact on ability of people to contribute futher tests in future.
>

I am on the side of the unified execution approach.  But, will it
really be simple to convert all tests?  "Real life integration" tests
too?  I would love to see that, but I can't see how that can be done
without the woes mentioned before (duplicating other code or limiting
the further toolset and capabilities of test writers).

Best regards,
- Cleber.

On 25/07/2024 16.21, Cleber Rosa wrote:
> On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
...
>> There have been several attempts to update the test suite in QEMU
>> to a newer version of Avocado, but so far no attempt has successfully
>> been merged yet.
> 
> So, we've seen in the past an attempt to update Avocado from 88.1 to a
> regular release, and the troubles it caused, including a revert.  My
> take was that a LTS version should be used, but during this time,
> Avocado experienced a rewrite and having it replacing the old
> implementation in a production level project such as QEMU was tricky.
> Then 103.0 LTS was released, and there was extensive work to test the
> QEMU tests before that release was cut.  Additionally, there was
> further work, but unfortunately not posted yet, to make use of 103.0
> features in the existing tests[2].   I've tested on GitLab with tests
> running in parallel, cutting job times in 1/3[2].  A side node is
> that, because 103.0 is an LTS release, it will receive the needed bug
> fixes and updates that we deem necessary, including things we find in
> QEMU tests.  In fact, 103.1[3] is in the works.

  Hi Cleber,

thanks for the explanation, but we really need to replace v88 rather *now* 
since v88 does not work with the latest versions of Python anymore (there is 
a work-around on Fedora fortunately, but it's completely broken on Ubuntu 
24.04 as far as I know). So even a single-threaded execution with v103 would 
have been better than waiting forever for your update to land. The problem 
with v88 being broken has been raised a couple of times already, but it's 
incredibly hard to get a response from you Avocado folks, so with hardly any 
help from the Avocado side, and nobody on the QEMU side being really 
familiar with the Avocado stuff, and with the meson test runner being used 
by all other subsystems in QEMU already, I think it's best if we continue 
with this series here.

>> Additionally, the whole "make check" test suite in QEMU is using the
>> meson test runner nowadays, so running the python-based tests via the
>> Avocodo test runner looks and feels quite like an oddball, requiring
>> the users to deal with the knowledge of multiple test runners in
>> parallel (e.g. the timeout settings work completely differently).
>>
> 
> Now I believe we can be very much in sync here.  I've thought for a
> while that there's no reason for Avocado to cooperate or be compatible
> with Meson.  There's no reason why users can't simply pick how the
> test gets run.  In fact, with the new Avocado architecture, you don't
> even need to run "avocado" to run an "avocado-instrumented" test.  You
> could pretty much run "avocado-runner-avocado-instrumented" with the
> right parameters through Meson.

Ok, good to know, we could maybe use that eventually for the tests that 
really require the Avocado framework (i.e. the cloud-init based tests).

For the others, as Daniel said in an earlier mail, it's much more convenient 
if you can also run the tests directly instead, without such a layer in 
between - that makes debugging way easier.

>> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
>> classes (e.g. based on cloud-init images or using SSH) really depend
>> on the Avocado framework, so we'd need a solution for those if we
>> want to continue using them. One solution might be to simply use the
>> required functions from avocado.utils for these tests, and still run
>> them via the meson test runner instead, but that needs some further
>> investigation that will be done later.
> 
> So, I believe this type of higher level testing is something that
> needs to remain, and even grow.  Speaking for Red Hat, I see the
> movement of QE contributing more Avocado-VT style tests into QEMU
> itself.

I didn't really see such a movement in recent times yet ... Could you point 
to an example?

>  This means way more libraries and features that go into a
> common set of utilities and features (more on that later) than it
> currently exists in avocado.utils.
> 
> This brings the autils[4] initiative into the picture.  We're about
> 80% done with the project structure, and after that, it will be a
> common utility project (such as the cloudinit and ssh) which can be
> released automatically when the maintainer votes (through GitHub) that
> a new release is needed.

autils sounds promising, but I just had a look at the repository, and there 
does not seem to be that much code available there yet, so I guess it will 
take still quite a long time 'til that's ready?

  Thomas

On Fri, Jul 26, 2024 at 6:07 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 25/07/2024 16.21, Cleber Rosa wrote:
> > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> ...
> >> There have been several attempts to update the test suite in QEMU
> >> to a newer version of Avocado, but so far no attempt has successfully
> >> been merged yet.
> >
> > So, we've seen in the past an attempt to update Avocado from 88.1 to a
> > regular release, and the troubles it caused, including a revert.  My
> > take was that a LTS version should be used, but during this time,
> > Avocado experienced a rewrite and having it replacing the old
> > implementation in a production level project such as QEMU was tricky.
> > Then 103.0 LTS was released, and there was extensive work to test the
> > QEMU tests before that release was cut.  Additionally, there was
> > further work, but unfortunately not posted yet, to make use of 103.0
> > features in the existing tests[2].   I've tested on GitLab with tests
> > running in parallel, cutting job times in 1/3[2].  A side node is
> > that, because 103.0 is an LTS release, it will receive the needed bug
> > fixes and updates that we deem necessary, including things we find in
> > QEMU tests.  In fact, 103.1[3] is in the works.
>
>   Hi Cleber,
>
> thanks for the explanation, but we really need to replace v88 rather *now*
> since v88 does not work with the latest versions of Python anymore (there is
> a work-around on Fedora fortunately, but it's completely broken on Ubuntu
> 24.04 as far as I know). So even a single-threaded execution with v103 would
> have been better than waiting forever for your update to land. The problem
> with v88 being broken has been raised a couple of times already, but it's
> incredibly hard to get a response from you Avocado folks, so with hardly any
> help from the Avocado side, and nobody on the QEMU side being really
> familiar with the Avocado stuff, and with the meson test runner being used
> by all other subsystems in QEMU already, I think it's best if we continue
> with this series here.
>

Hi Thomas,

I agree with the urgency.  I've posted the updates here:

   https://lists.gnu.org/archive/html/qemu-devel/2024-07/msg06236.html

Sorry for the delay, and I hope this gives more time to make the
evolution towards a solution that better suits QEMU.

Regards,
- Cleber.

> >> Additionally, the whole "make check" test suite in QEMU is using the
> >> meson test runner nowadays, so running the python-based tests via the
> >> Avocodo test runner looks and feels quite like an oddball, requiring
> >> the users to deal with the knowledge of multiple test runners in
> >> parallel (e.g. the timeout settings work completely differently).
> >>
> >
> > Now I believe we can be very much in sync here.  I've thought for a
> > while that there's no reason for Avocado to cooperate or be compatible
> > with Meson.  There's no reason why users can't simply pick how the
> > test gets run.  In fact, with the new Avocado architecture, you don't
> > even need to run "avocado" to run an "avocado-instrumented" test.  You
> > could pretty much run "avocado-runner-avocado-instrumented" with the
> > right parameters through Meson.
>
> Ok, good to know, we could maybe use that eventually for the tests that
> really require the Avocado framework (i.e. the cloud-init based tests).
>
> For the others, as Daniel said in an earlier mail, it's much more convenient
> if you can also run the tests directly instead, without such a layer in
> between - that makes debugging way easier.
>
> >> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> >> classes (e.g. based on cloud-init images or using SSH) really depend
> >> on the Avocado framework, so we'd need a solution for those if we
> >> want to continue using them. One solution might be to simply use the
> >> required functions from avocado.utils for these tests, and still run
> >> them via the meson test runner instead, but that needs some further
> >> investigation that will be done later.
> >
> > So, I believe this type of higher level testing is something that
> > needs to remain, and even grow.  Speaking for Red Hat, I see the
> > movement of QE contributing more Avocado-VT style tests into QEMU
> > itself.
>
> I didn't really see such a movement in recent times yet ... Could you point
> to an example?
>

Indeed, there is no public evidence of that, yet.

> >  This means way more libraries and features that go into a
> > common set of utilities and features (more on that later) than it
> > currently exists in avocado.utils.
> >
> > This brings the autils[4] initiative into the picture.  We're about
> > 80% done with the project structure, and after that, it will be a
> > common utility project (such as the cloudinit and ssh) which can be
> > released automatically when the maintainer votes (through GitHub) that
> > a new release is needed.
>
> autils sounds promising, but I just had a look at the repository, and there
> does not seem to be that much code available there yet, so I guess it will
> take still quite a long time 'til that's ready?
>

Like I said previously, we were working on the project infrastructure.
We haven't ported much code over there yet.  There are very few things
missing, and all of them have been assigned to be worked on.  It's
been tracked here:

   https://github.com/avocado-framework/autils/issues/1

My estimation is that 2-4 weeks for the project infrastructure to be complete.

Regards,
- Cleber.

>   Thomas
>

Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com> ha scritto:

> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>

I think we should take another look at that. Avocado 92 should work, though
I am not sure if it's also depending on "imp", and if I recall correctly
the problem with more recent version was more that it conflicted with
distros that didn't have it packaged. That should also not be a problem
anymore with the pythondeps.toml mechanism.

Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>

This is true.

Paolo


> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>
>
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
>
>  make check-functional
>
> You can also run single targets e.g. with:
>
>  make check-functional-ppc
>
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>
>  export PYTHONPATH=$HOME/qemu/python
>  export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>  export PYTHONPATH=$HOME/qemu/build
>  ~/qemu/tests/functional/test_virtio_version.py
>
> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.
>
> Still to be done: Update the documentation for this new test framework.
>
> RFC -> v1:
> - Now using pycotap for running the tests instead of "pytest"
> - Change the name from "tests/pytest" to "tests/functional" accordingly
> - Make it possible to run the tests directly
> - Use Python's urllib instead of wget for downloading
> - Lots of makefile / meson integration improvements
> - Converted more tests
> - Update MAINTAINERS file accordingly
> - Added a patch to run check-functional in the gitlab-CI
> - ... lots of other changes I forgot about ... in fact, I changed so
>   many things that I also did not dare to pick up the Reviewed-bys
>   from the RFC
>
> Thomas Huth (11):
>   tests/functional: Add base classes for the upcoming pytest-based tests
>   tests/functional: Convert simple avocado tests into standalone python
>     tests
>   tests/functional: Convert avocado tests that just need a small
>     adjustment
>   tests/functional: Add python-based tests to the meson build system
>   tests/functional: Implement fetch_asset() method for downloading
>     assets
>   tests/functional: Convert some tests that download files via
>     fetch_asset()
>   tests/functional: Add a function for extracting files from an archive
>   tests/functional: Convert some avocado tests that needed
>     avocado.utils.archive
>   tests/functional: Set up logging
>   tests/functional: Convert the s390x avocado tests into standalone
>     tests
>   gitlab-ci: Add "check-functional" to the build tests
>
>  MAINTAINERS                                   |  22 +-
>  .gitlab-ci.d/buildtest-template.yml           |   3 +-
>  .gitlab-ci.d/buildtest.yml                    |  60 +--
>  pythondeps.toml                               |   3 +-
>  tests/Makefile.include                        |  18 +-
>  tests/functional/meson.build                  | 112 +++++
>  tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
>  tests/functional/qemu_test/utils.py           |  28 ++
>  .../test_arm_canona1100.py}                   |  21 +-
>  .../test_arm_n8x0.py}                         |  25 +-
>  .../test_avr_mega2560.py}                     |  11 +-
>  .../test_cpu_queries.py}                      |   7 +-
>  .../test_empty_cpu_model.py}                  |   7 +-
>  .../test_info_usernet.py}                     |  11 +-
>  .../test_loongarch64_virt.py}                 |  16 +-
>  .../test_mem_addr_space.py}                   |  53 +--
>  .../test_mips64el_loongson3v.py}              |  26 +-
>  .../test_netdev_ethtool.py}                   |  32 +-
>  .../test_pc_cpu_hotplug_props.py}             |  11 +-
>  .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
>  .../test_ppc_74xx.py}                         |  74 ++--
>  .../test_ppc_bamboo.py}                       |  23 +-
>  .../test_s390x_ccw_virtio.py}                 |  32 +-
>  .../test_s390x_topology.py}                   |  70 ++--
>  .../test_sparc64_sun4u.py}                    |  25 +-
>  .../version.py => functional/test_version.py} |  13 +-
>  .../test_virtio_version.py}                   |   8 +-
>  tests/meson.build                             |   1 +
>  28 files changed, 795 insertions(+), 320 deletions(-)
>  create mode 100644 tests/functional/meson.build
>  create mode 100644 tests/functional/qemu_test/__init__.py
>  create mode 100644 tests/functional/qemu_test/utils.py
>  rename tests/{avocado/machine_arm_canona1100.py =>
> functional/test_arm_canona1100.py} (71%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py}
> (71%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py}
> (90%)
>  mode change 100644 => 100755
>  rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py}
> (89%)
>  mode change 100644 => 100755
>  rename tests/{avocado/empty_cpu_model.py =>
> functional/test_empty_cpu_model.py} (84%)
>  mode change 100644 => 100755
>  rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py}
> (87%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_loongarch.py =>
> functional/test_loongarch64_virt.py} (89%)
>  mode change 100644 => 100755
>  rename tests/{avocado/mem-addr-space-check.py =>
> functional/test_mem_addr_space.py} (92%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_mips_loongson3v.py =>
> functional/test_mips64el_loongson3v.py} (55%)
>  mode change 100644 => 100755
>  rename tests/{avocado/netdev-ethtool.py =>
> functional/test_netdev_ethtool.py} (81%)
>  mode change 100644 => 100755
>  rename tests/{avocado/pc_cpu_hotplug_props.py =>
> functional/test_pc_cpu_hotplug_props.py} (90%)
>  mode change 100644 => 100755
>  rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
>  mode change 100644 => 100755
>  rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
>  mode change 100644 => 100755
>  rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py}
> (75%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_s390_ccw_virtio.py =>
> functional/test_s390x_ccw_virtio.py} (95%)
>  mode change 100644 => 100755
>  rename tests/{avocado/s390_topology.py =>
> functional/test_s390x_topology.py} (90%)
>  mode change 100644 => 100755
>  rename tests/{avocado/machine_sparc64_sun4u.py =>
> functional/test_sparc64_sun4u.py} (60%)
>  mode change 100644 => 100755
>  rename tests/{avocado/version.py => functional/test_version.py} (78%)
>  mode change 100644 => 100755
>  rename tests/{avocado/virtio_version.py =>
> functional/test_virtio_version.py} (98%)
>  mode change 100644 => 100755
>
> --
> 2.45.2
>
>

On 16/07/2024 19.57, Paolo Bonzini wrote:
> 
> 
> Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com 
> <mailto:thuth@redhat.com>> ha scritto:
> 
>     The Avocado v88 that we use in QEMU is already on a life support
>     system: It is not supported by upstream anymore, and with the latest
>     versions of Python, it won't work anymore since it depends on the
>     "imp" module that has been removed in Python 3.12.
> 
>     There have been several attempts to update the test suite in QEMU
>     to a newer version of Avocado, but so far no attempt has successfully
>     been merged yet.
> 
> 
> I think we should take another look at that. Avocado 92 should work, though 
> I am not sure if it's also depending on "imp"

FWIW, the "imp" problem had been fixed here:

 
https://github.com/avocado-framework/avocado/commit/dfc340f1a0b0aeddbf6699e49e94b75ed080137c

That had just been released with Avocado version 99, so updating to 92 does 
not really help.

  Thomas