[PULL 00/22] Tests and misc patches

Thomas Huth posted 22 patches 2 months, 2 weeks ago
There is a newer version of this series
MAINTAINERS                                        |  12 +-
docs/devel/testing/fuzzing.rst                     |   4 +-
meson.build                                        |  26 +-
tests/qtest/fdc-test.c                             |   2 +-
.gitlab-ci.d/buildtest.yml                         |  12 +-
.../custom-runners/ubuntu-22.04-aarch64.yml        |   2 +-
.gitlab-ci.d/custom-runners/ubuntu-22.04-s390x.yml |   2 +-
.gitlab-ci.d/windows.yml                           |   3 +-
contrib/plugins/Makefile                           |   2 +-
meson_options.txt                                  |   6 +-
scripts/meson-buildoptions.sh                      |   9 +-
tests/avocado/avocado_qemu/__init__.py             |  10 -
tests/avocado/boot_linux_console.py                | 483 ---------------------
tests/avocado/machine_mips_malta.py                | 162 -------
tests/avocado/multiprocess.py                      | 102 -----
tests/docker/test-debug                            |   4 +-
tests/functional/meson.build                       |  26 +-
tests/functional/qemu_test/__init__.py             |   1 +
tests/functional/qemu_test/cmd.py                  |   6 +-
tests/functional/qemu_test/linuxkernel.py          |  41 ++
tests/functional/qemu_test/tesseract.py            |   3 +-
tests/functional/test_aarch64_raspi3.py            |  41 ++
tests/functional/test_aarch64_raspi4.py            |  99 +++++
tests/functional/test_alpha_clipper.py             |  38 ++
tests/functional/test_arm_raspi2.py                |  95 ++++
tests/functional/test_m68k_mcf5208evb.py           |  29 ++
tests/functional/test_m68k_q800.py                 |  37 ++
tests/functional/test_mips64el_fuloong2e.py        |  25 +-
tests/functional/test_mips64el_malta.py            | 186 ++++++++
tests/functional/test_mips_malta.py                |  81 ++++
tests/functional/test_mipsel_malta.py              |  96 ++++
tests/functional/test_multiprocess.py              | 100 +++++
tests/functional/test_or1k_sim.py                  |  29 ++
33 files changed, 978 insertions(+), 796 deletions(-)
delete mode 100644 tests/avocado/machine_mips_malta.py
delete mode 100644 tests/avocado/multiprocess.py
create mode 100644 tests/functional/qemu_test/linuxkernel.py
create mode 100755 tests/functional/test_aarch64_raspi3.py
create mode 100755 tests/functional/test_aarch64_raspi4.py
create mode 100755 tests/functional/test_alpha_clipper.py
create mode 100755 tests/functional/test_arm_raspi2.py
create mode 100755 tests/functional/test_m68k_mcf5208evb.py
create mode 100755 tests/functional/test_m68k_q800.py
create mode 100755 tests/functional/test_mips64el_malta.py
create mode 100755 tests/functional/test_mips_malta.py
create mode 100755 tests/functional/test_mipsel_malta.py
create mode 100755 tests/functional/test_multiprocess.py
create mode 100755 tests/functional/test_or1k_sim.py
[PULL 00/22] Tests and misc patches
Posted by Thomas Huth 2 months, 2 weeks ago
The following changes since commit 7bbadc60b58b742494555f06cd342311ddab9351:

  Merge tag 'crypto-fixes-pull-request' of https://gitlab.com/berrange/qemu into staging (2024-09-09 15:19:32 +0100)

are available in the Git repository at:

  https://gitlab.com/thuth/qemu.git tags/pull-request-2024-09-10

for you to fetch changes up to cc3e005eb22332ba277bff2e39025d0087f3795e:

  tests/functional: Fix bad usage of has_cmd (2024-09-10 13:44:42 +0200)

----------------------------------------------------------------
* Split --enable-sanitizers to --enable-{asan, ubsan}
* Build MSYS2 job using multiple CPUs
* Fix "make distclean" wrt contrib/plugins/
* Convert more Avocado tests to plain standalone functional tests
* Fix bug that breaks "make check-functional" when tesseract is missing

----------------------------------------------------------------
Beraldo Leal (1):
      MAINTAINERS: Remove myself as reviewer

Philippe Mathieu-Daudé (10):
      gitlab-ci: Build MSYS2 job using multiple CPUs
      tests/functional: Convert mips64el Fuloong2e avocado test (2/2)
      tests/functional: Convert mips64el I6400 Malta avocado tests
      tests/functional: Convert mips64el 5KEc Malta avocado tests
      tests/functional: Convert mips32el Malta YAMON avocado test
      tests/functional: Convert nanomips Malta avocado tests
      tests/functional: Convert mips32eb 4Kc Malta avocado tests
      tests/functional: Convert ARM Raspi2 avocado tests
      tests/functional: Convert Aarch64 Raspi3 avocado tests
      tests/functional: Convert Aarch64 Raspi4 avocado tests

Richard Henderson (2):
      meson: Split --enable-sanitizers to --enable-{asan, ubsan}
      meson: Move -fsanitize=undefined into normal configuraton

Thomas Huth (9):
      contrib/plugins/Makefile: Add a 'distclean' target
      MAINTAINERS: Remove myself from the Meson section
      tests/functional: Add the LinuxKernelTest for testing the Linux boot process
      tests/functional: Convert the m68k Q800 Avocado test into a functional test
      tests/functional: Convert the Alpha Clipper Avocado test
      tests/functional: Convert the m68k MCF5208EVB Avocado test
      tests/functional: Convert the or1k-sim Avocado test
      tests/functional: Convert the multiprocess avocado test into a standalone test
      tests/functional: Fix bad usage of has_cmd

 MAINTAINERS                                        |  12 +-
 docs/devel/testing/fuzzing.rst                     |   4 +-
 meson.build                                        |  26 +-
 tests/qtest/fdc-test.c                             |   2 +-
 .gitlab-ci.d/buildtest.yml                         |  12 +-
 .../custom-runners/ubuntu-22.04-aarch64.yml        |   2 +-
 .gitlab-ci.d/custom-runners/ubuntu-22.04-s390x.yml |   2 +-
 .gitlab-ci.d/windows.yml                           |   3 +-
 contrib/plugins/Makefile                           |   2 +-
 meson_options.txt                                  |   6 +-
 scripts/meson-buildoptions.sh                      |   9 +-
 tests/avocado/avocado_qemu/__init__.py             |  10 -
 tests/avocado/boot_linux_console.py                | 483 ---------------------
 tests/avocado/machine_mips_malta.py                | 162 -------
 tests/avocado/multiprocess.py                      | 102 -----
 tests/docker/test-debug                            |   4 +-
 tests/functional/meson.build                       |  26 +-
 tests/functional/qemu_test/__init__.py             |   1 +
 tests/functional/qemu_test/cmd.py                  |   6 +-
 tests/functional/qemu_test/linuxkernel.py          |  41 ++
 tests/functional/qemu_test/tesseract.py            |   3 +-
 tests/functional/test_aarch64_raspi3.py            |  41 ++
 tests/functional/test_aarch64_raspi4.py            |  99 +++++
 tests/functional/test_alpha_clipper.py             |  38 ++
 tests/functional/test_arm_raspi2.py                |  95 ++++
 tests/functional/test_m68k_mcf5208evb.py           |  29 ++
 tests/functional/test_m68k_q800.py                 |  37 ++
 tests/functional/test_mips64el_fuloong2e.py        |  25 +-
 tests/functional/test_mips64el_malta.py            | 186 ++++++++
 tests/functional/test_mips_malta.py                |  81 ++++
 tests/functional/test_mipsel_malta.py              |  96 ++++
 tests/functional/test_multiprocess.py              | 100 +++++
 tests/functional/test_or1k_sim.py                  |  29 ++
 33 files changed, 978 insertions(+), 796 deletions(-)
 delete mode 100644 tests/avocado/machine_mips_malta.py
 delete mode 100644 tests/avocado/multiprocess.py
 create mode 100644 tests/functional/qemu_test/linuxkernel.py
 create mode 100755 tests/functional/test_aarch64_raspi3.py
 create mode 100755 tests/functional/test_aarch64_raspi4.py
 create mode 100755 tests/functional/test_alpha_clipper.py
 create mode 100755 tests/functional/test_arm_raspi2.py
 create mode 100755 tests/functional/test_m68k_mcf5208evb.py
 create mode 100755 tests/functional/test_m68k_q800.py
 create mode 100755 tests/functional/test_mips64el_malta.py
 create mode 100755 tests/functional/test_mips_malta.py
 create mode 100755 tests/functional/test_mipsel_malta.py
 create mode 100755 tests/functional/test_multiprocess.py
 create mode 100755 tests/functional/test_or1k_sim.py


Re: [PULL 00/22] Tests and misc patches
Posted by Peter Maydell 2 months, 2 weeks ago
On Tue, 10 Sept 2024 at 13:37, Thomas Huth <thuth@redhat.com> wrote:
>
> The following changes since commit 7bbadc60b58b742494555f06cd342311ddab9351:
>
>   Merge tag 'crypto-fixes-pull-request' of https://gitlab.com/berrange/qemu into staging (2024-09-09 15:19:32 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/thuth/qemu.git tags/pull-request-2024-09-10
>
> for you to fetch changes up to cc3e005eb22332ba277bff2e39025d0087f3795e:
>
>   tests/functional: Fix bad usage of has_cmd (2024-09-10 13:44:42 +0200)
>
> ----------------------------------------------------------------
> * Split --enable-sanitizers to --enable-{asan, ubsan}
> * Build MSYS2 job using multiple CPUs
> * Fix "make distclean" wrt contrib/plugins/
> * Convert more Avocado tests to plain standalone functional tests
> * Fix bug that breaks "make check-functional" when tesseract is missing
>
> ----------------------------------------------------------------

On the 'make check-vm-openbsd' test I run:

595/944 qemu:func-quick+func-mips64el /
func-mips64el-mips64el_fuloong2e          ERROR            6.16s
exit status 1
602/944 qemu:func-quick+func-or1k / func-or1k-or1k_sim
           ERROR            6.91s   exit status 1

Slightly more detail further up in the log:


595/944 qemu:func-quick+func-mips64el /
func-mips64el-mips64el_fuloong2e          ERROR            6.16s
exit status 1
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
Traceback (most recent call last):
  File "/home/qemu/qemu-test.fsnflZ/src/tests/functional/test_mips64el_fuloong2e.py",
line 29, in test_linux_kernel_3_16
    deb_path = self.ASSET_KERNEL.fetch()
  File "/home/qemu/qemu-test.fsnflZ/src/tests/functional/qemu_test/asset.py",
line 135, in fetch
    if not self._check(tmp_cache_file):
  File "/home/qemu/qemu-test.fsnflZ/src/tests/functional/qemu_test/asset.py",
line 52, in _check
    checksum = subprocess.check_output(
  File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'sha256sum'


(test program exited with status code 1)


Similarly for the or1k failure.

If you want to use sha256sum in "make check" then we should
ensure that it's installed in these BSD VM images.

Also, does this mean that "make check" is now going to run
a lot of the tests that were previously in "make check-avocado"?
That seems like it might make our CI jobs take longer...

thanks
-- PMM
Re: [PULL 00/22] Tests and misc patches
Posted by Thomas Huth 2 months, 2 weeks ago
On 10/09/2024 17.38, Peter Maydell wrote:
> On Tue, 10 Sept 2024 at 13:37, Thomas Huth <thuth@redhat.com> wrote:
>>
>> The following changes since commit 7bbadc60b58b742494555f06cd342311ddab9351:
>>
>>    Merge tag 'crypto-fixes-pull-request' of https://gitlab.com/berrange/qemu into staging (2024-09-09 15:19:32 +0100)
>>
>> are available in the Git repository at:
>>
>>    https://gitlab.com/thuth/qemu.git tags/pull-request-2024-09-10
>>
>> for you to fetch changes up to cc3e005eb22332ba277bff2e39025d0087f3795e:
>>
>>    tests/functional: Fix bad usage of has_cmd (2024-09-10 13:44:42 +0200)
>>
>> ----------------------------------------------------------------
>> * Split --enable-sanitizers to --enable-{asan, ubsan}
>> * Build MSYS2 job using multiple CPUs
>> * Fix "make distclean" wrt contrib/plugins/
>> * Convert more Avocado tests to plain standalone functional tests
>> * Fix bug that breaks "make check-functional" when tesseract is missing
>>
>> ----------------------------------------------------------------
> 
> On the 'make check-vm-openbsd' test I run:
> 
> 595/944 qemu:func-quick+func-mips64el /
> func-mips64el-mips64el_fuloong2e          ERROR            6.16s
> exit status 1
> 602/944 qemu:func-quick+func-or1k / func-or1k-or1k_sim
>             ERROR            6.91s   exit status 1
> 
> Slightly more detail further up in the log:
> 
> 
> 595/944 qemu:func-quick+func-mips64el /
> func-mips64el-mips64el_fuloong2e          ERROR            6.16s
> exit status 1
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> Traceback (most recent call last):
>    File "/home/qemu/qemu-test.fsnflZ/src/tests/functional/test_mips64el_fuloong2e.py",
> line 29, in test_linux_kernel_3_16
>      deb_path = self.ASSET_KERNEL.fetch()
>    File "/home/qemu/qemu-test.fsnflZ/src/tests/functional/qemu_test/asset.py",
> line 135, in fetch
>      if not self._check(tmp_cache_file):
>    File "/home/qemu/qemu-test.fsnflZ/src/tests/functional/qemu_test/asset.py",
> line 52, in _check
>      checksum = subprocess.check_output(
>    File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
>      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
>    File "/usr/local/lib/python3.10/subprocess.py", line 503, in run
>      with Popen(*popenargs, **kwargs) as process:
>    File "/usr/local/lib/python3.10/subprocess.py", line 971, in __init__
>      self._execute_child(args, executable, preexec_fn, close_fds,
>    File "/usr/local/lib/python3.10/subprocess.py", line 1863, in _execute_child
>      raise child_exception_type(errno_num, err_msg, err_filename)
> FileNotFoundError: [Errno 2] No such file or directory: 'sha256sum'
> 
> 
> (test program exited with status code 1)
> 
> 
> Similarly for the or1k failure.
> 
> If you want to use sha256sum in "make check" then we should
> ensure that it's installed in these BSD VM images.

Ok ... I'm looking into using the hashlib from Python instead, that way 
we'll be independent from external binaries here.

> Also, does this mean that "make check" is now going to run
> a lot of the tests that were previously in "make check-avocado"?
> That seems like it might make our CI jobs take longer...

No, it will only some basic tests that don't download any assets. These 
should be the rather quick ones.

  Thomas