[PATCH 0/9] Avocado tests: filter out tests using TCG booting full blown distros

Cleber Rosa posted 9 patches 3 years, 11 months ago
Failed in applying to current master (apply log)
Test checkpatch passed
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <f4bug@amsat.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>, Cleber Rosa <crosa@redhat.com>, Aurelien Jarno <aurelien@aurel32.net>, Antony Pavlov <antonynpavlov@gmail.com>, Peter Maydell <peter.maydell@linaro.org>, Michael Rolnik <mrolnik@gmail.com>, "Edgar E. Iglesias" <edgar.iglesias@gmail.com>, Huacai Chen <chenhuacai@kernel.org>, Jiaxun Yang <jiaxun.yang@flygoat.com>, Yoshinori Sato <ysato@users.sourceforge.jp>, Halil Pasic <pasic@linux.ibm.com>, Christian Borntraeger <borntraeger@linux.ibm.com>, Eric Farman <farman@linux.ibm.com>, Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>, Fabien Chouteau <chouteau@adacore.com>, KONRAD Frederic <frederic.konrad@adacore.com>, "Hervé Poussineau" <hpoussin@reactos.org>, "Cédric Le Goater" <clg@kaod.org>, Daniel Henrique Barboza <danielhb413@gmail.com>, David Gibson <david@gibson.dropbear.id.au>, Greg Kurz <groug@kaod.org>, Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>, Paolo Bonzini <pbonzini@redhat.com>, Eric Auger <eric.auger@redhat.com>, Alexandre Iooss <erdnaxe@crans.org>, Mahmoud Mandour <ma.mandourr@gmail.com>
.gitlab-ci.d/buildtest-template.yml       |  3 ++
.gitlab-ci.d/buildtest.yml                |  9 ----
docs/devel/testing.rst                    | 22 +++++++++
tests/Makefile.include                    |  6 ++-
tests/avocado/avocado_qemu/__init__.py    | 10 ++---
tests/avocado/boot_linux.py               |  4 ++
tests/avocado/boot_linux_console.py       | 54 +++++++++++++++++++++++
tests/avocado/boot_xen.py                 |  3 ++
tests/avocado/hotplug_cpu.py              |  1 +
tests/avocado/intel_iommu.py              |  1 +
tests/avocado/linux_initrd.py             |  5 ++-
tests/avocado/linux_ssh_mips_malta.py     |  5 +++
tests/avocado/machine_arm_canona1100.py   |  1 +
tests/avocado/machine_arm_integratorcp.py |  7 ++-
tests/avocado/machine_arm_n8x0.py         |  2 +
tests/avocado/machine_avr6.py             |  1 +
tests/avocado/machine_m68k_nextcube.py    |  1 +
tests/avocado/machine_microblaze.py       |  1 +
tests/avocado/machine_mips_fuloong2e.py   |  1 +
tests/avocado/machine_mips_loongson3v.py  |  1 +
tests/avocado/machine_mips_malta.py       |  6 ++-
tests/avocado/machine_rx_gdbsim.py        |  2 +
tests/avocado/machine_s390_ccw_virtio.py  |  4 ++
tests/avocado/machine_sparc64_sun4u.py    |  1 +
tests/avocado/machine_sparc_leon3.py      |  1 +
tests/avocado/migration.py                |  4 +-
tests/avocado/multiprocess.py             |  4 ++
tests/avocado/ppc_405.py                  |  2 +
tests/avocado/ppc_bamboo.py               |  2 +
tests/avocado/ppc_mpc8544ds.py            |  1 +
tests/avocado/ppc_prep_40p.py             |  1 +
tests/avocado/ppc_pseries.py              |  1 +
tests/avocado/ppc_virtex_ml507.py         |  1 +
tests/avocado/replay_kernel.py            | 33 ++++++++++++--
tests/avocado/replay_linux.py             |  6 +--
tests/avocado/reverse_debugging.py        |  6 +--
tests/avocado/smmu.py                     |  1 +
tests/avocado/tcg_plugins.py              |  3 ++
tests/avocado/tesseract_utils.py          |  6 +--
tests/avocado/virtio-gpu.py               |  2 +
tests/avocado/virtio_check_params.py      |  3 +-
tests/avocado/virtiofs_submounts.py       |  8 ++--
tests/requirements.txt                    |  2 +-
43 files changed, 197 insertions(+), 41 deletions(-)
[PATCH 0/9] Avocado tests: filter out tests using TCG booting full blown distros
Posted by Cleber Rosa 3 years, 11 months ago
It was previously reported[1] and discussed that tests booting full
blown distros and relying on TCG would take too much time to run,
especially in the environments given by GitLab CI's shared runners.

This is an implementation of a proposal to exclude those tests from
being run by default on `make check-avocado` invocations.  To make it
extra clear, all tests are still available, but those that are tagged
with "accel:tcg" and "boots:distro", are filtered out by default on
`make check-avocado`.

This is the situation of the Avocado GitLab CI jobs with and without
the changes in this PS:

                         +------------------+--- ----------------+
                         |        Now[2]    |       Before[3]    |
+------------------------+------------------+--------------------+
|          Job           |  Length |  Tests |  Length  |  Tests  |
|         Name           | (mm:ss) |   Run  |  (mm:ss) |   Run   |
+------------------------+------------------+-------------------+
|avocado-system-alpine   |  06:33      16   |   20:30       18   |
|avocado-system-debian   |  12:06      24   |   13:05       24   |
|avocado-system-centos   |  09:58      41   |   24:15       44   |
|avocado-system-fedora   |  08:50      35   |   08:59       35   |
|avocado-system-opensuse |  08:09      38   |   27:21       42   |
|avocado-system-ubuntu   |  06:52      16   |   18:52       18   |
|avocado-cfi-x86_64      |  05:43      27   |   15:07       29   |
+------------------------+------------------+--------------------+
|TOTALS                  |  58:11     197   | 2:08:09      210   |
+------------------------+------------------+--------------------+

Assuming the jobs run in parallel, the overall wait time for all the
Avocado jobs to complete is now ~12 minutes.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg07271.html
[2] https://gitlab.com/cleber.gnu/qemu/-/pipelines/479720240
[3] https://gitlab.com/qemu-project/qemu/-/pipelines/478580581

Cleber Rosa (9):
  Avocado GitLab CI jobs: don't reset TARGETS and simplify commands
  Avocado tests: use logging namespace that is preserved in test logs
  Avocado migration test: adapt to "utils.network" API namespace change
  Avocado: bump to version 95.0
  tests/avocado/linux_ssh_mips_malta.py: add missing accel (tcg) tag
  tests/avocado/virtiofs_submounts.py: shared_dir may not exist
  Avocado tests: improve documentation on tag filtering
  Avocado tests: classify tests based on what it's booted
  Avocado tests: don't run tests with TCG that boot full blown distros

 .gitlab-ci.d/buildtest-template.yml       |  3 ++
 .gitlab-ci.d/buildtest.yml                |  9 ----
 docs/devel/testing.rst                    | 22 +++++++++
 tests/Makefile.include                    |  6 ++-
 tests/avocado/avocado_qemu/__init__.py    | 10 ++---
 tests/avocado/boot_linux.py               |  4 ++
 tests/avocado/boot_linux_console.py       | 54 +++++++++++++++++++++++
 tests/avocado/boot_xen.py                 |  3 ++
 tests/avocado/hotplug_cpu.py              |  1 +
 tests/avocado/intel_iommu.py              |  1 +
 tests/avocado/linux_initrd.py             |  5 ++-
 tests/avocado/linux_ssh_mips_malta.py     |  5 +++
 tests/avocado/machine_arm_canona1100.py   |  1 +
 tests/avocado/machine_arm_integratorcp.py |  7 ++-
 tests/avocado/machine_arm_n8x0.py         |  2 +
 tests/avocado/machine_avr6.py             |  1 +
 tests/avocado/machine_m68k_nextcube.py    |  1 +
 tests/avocado/machine_microblaze.py       |  1 +
 tests/avocado/machine_mips_fuloong2e.py   |  1 +
 tests/avocado/machine_mips_loongson3v.py  |  1 +
 tests/avocado/machine_mips_malta.py       |  6 ++-
 tests/avocado/machine_rx_gdbsim.py        |  2 +
 tests/avocado/machine_s390_ccw_virtio.py  |  4 ++
 tests/avocado/machine_sparc64_sun4u.py    |  1 +
 tests/avocado/machine_sparc_leon3.py      |  1 +
 tests/avocado/migration.py                |  4 +-
 tests/avocado/multiprocess.py             |  4 ++
 tests/avocado/ppc_405.py                  |  2 +
 tests/avocado/ppc_bamboo.py               |  2 +
 tests/avocado/ppc_mpc8544ds.py            |  1 +
 tests/avocado/ppc_prep_40p.py             |  1 +
 tests/avocado/ppc_pseries.py              |  1 +
 tests/avocado/ppc_virtex_ml507.py         |  1 +
 tests/avocado/replay_kernel.py            | 33 ++++++++++++--
 tests/avocado/replay_linux.py             |  6 +--
 tests/avocado/reverse_debugging.py        |  6 +--
 tests/avocado/smmu.py                     |  1 +
 tests/avocado/tcg_plugins.py              |  3 ++
 tests/avocado/tesseract_utils.py          |  6 +--
 tests/avocado/virtio-gpu.py               |  2 +
 tests/avocado/virtio_check_params.py      |  3 +-
 tests/avocado/virtiofs_submounts.py       |  8 ++--
 tests/requirements.txt                    |  2 +-
 43 files changed, 197 insertions(+), 41 deletions(-)

-- 
2.35.1