[PATCH v2 00/18] test/functional: improve functional test debugging & fix tuxrun

Daniel P. Berrangé posted 18 patches 1 day, 20 hours ago
docs/devel/testing/functional.rst        | 16 +++++
tests/functional/qemu_test/cmd.py        | 89 +++++++++++++++++++-----
tests/functional/qemu_test/testcase.py   | 43 +++++++++---
tests/functional/qemu_test/tuxruntest.py | 17 ++---
tests/functional/test_acpi_bits.py       | 57 +++++----------
tests/functional/test_arm_bpim2u.py      | 20 ------
tests/functional/test_arm_orangepi.py    | 27 -------
tests/functional/test_m68k_nextcube.py   |  3 +-
tests/functional/test_mips64el_malta.py  |  4 +-
tests/functional/test_virtio_gpu.py      |  3 +-
10 files changed, 150 insertions(+), 129 deletions(-)
[PATCH v2 00/18] test/functional: improve functional test debugging & fix tuxrun
Posted by Daniel P. Berrangé 1 day, 20 hours ago
This started out as a series to get rid of the many GBs of temp
files the functional tests leave behind. Then it expanded into
improving the functional test debugging by ensuring we preserve
the QEMU stdout/stderr log file created by the QEMUMachine class.
In the course of doing that I encountered some other minor points
worth fixing, and then got side tracked into looking at the tuxrun
hangs with aarch64be. Investigating the latter exposed some further
holes in the debugging story prompting yet more patches, as well as
a final solution for tuxrun. So this series does:

 * Purge all scratch files created by tests
 * Preserve the stdout/stderr log file
 * Capture debug log messages on QEMUMachine
 * Provide a QMP backdoor for debugging stuck QEMUs
 * Enhance console handling for partial line matches
 * Fix the tuxrun tests by eliminating sleeps

There's quite alot of code here, but at the same time it feels like
the kind of stuff that'll be valuable either in the 9.2 release, or
in the soon to exist 9.2 stable branch.

NB, with this series applied Thomas' tuxrun conversion to functional
testing survives 200 iterations on my machine, whereas it would
reliably hang in < 20, and often in < 10, before.

Changed in v2:

 - Changed console interaction to forbid 'failure_message'
   without 'success_message'
 - Reword console interaction log messages
 - Avoid stack trace when seeing early failure
 - Rewrote comment in acpi bits test
 - Avoid duplicate os.environ access

Daniel P. Berrangé (18):
  tests/functional: fix mips64el test to honour workdir
  tests/functional: automatically clean up scratch files after tests
  tests/functional: remove "AVOCADO" from env variable name
  tests/functional: remove todo wrt avocado.utils.wait_for
  tests/functional: remove leftover :avocado: tags
  tests/functional: remove obsolete reference to avocado bug
  tests/functional: remove comments talking about avocado
  tests/functional: honour self.workdir in ACPI bits tests
  tests/functional: put QEMUMachine logs in testcase log directory
  tests/functional: honour requested test VM name in QEMUMachine
  tests/functional: enable debug logging for QEMUMachine
  tests/functional: logs details of console interaction operations
  tests/functional: don't try to wait for the empty string
  tests/functional: require non-NULL success_message for console wait
  tests/functional: rewrite console handling to be bytewise
  tests/functional: remove time.sleep usage from tuxrun tests
  tests/functional: add a QMP backdoor for debugging stalled tests
  tests/functional: avoid accessing log_filename on earlier failures

 docs/devel/testing/functional.rst        | 16 +++++
 tests/functional/qemu_test/cmd.py        | 89 +++++++++++++++++++-----
 tests/functional/qemu_test/testcase.py   | 43 +++++++++---
 tests/functional/qemu_test/tuxruntest.py | 17 ++---
 tests/functional/test_acpi_bits.py       | 57 +++++----------
 tests/functional/test_arm_bpim2u.py      | 20 ------
 tests/functional/test_arm_orangepi.py    | 27 -------
 tests/functional/test_m68k_nextcube.py   |  3 +-
 tests/functional/test_mips64el_malta.py  |  4 +-
 tests/functional/test_virtio_gpu.py      |  3 +-
 10 files changed, 150 insertions(+), 129 deletions(-)

-- 
2.46.0


Re: [PATCH v2 00/18] test/functional: improve functional test debugging & fix tuxrun
Posted by Alex Bennée 1 day, 19 hours ago
Daniel P. Berrangé <berrange@redhat.com> writes:

> This started out as a series to get rid of the many GBs of temp
> files the functional tests leave behind. Then it expanded into
> improving the functional test debugging by ensuring we preserve
> the QEMU stdout/stderr log file created by the QEMUMachine class.
> In the course of doing that I encountered some other minor points
> worth fixing, and then got side tracked into looking at the tuxrun
> hangs with aarch64be. Investigating the latter exposed some further
> holes in the debugging story prompting yet more patches, as well as
> a final solution for tuxrun. So this series does:
>
>  * Purge all scratch files created by tests
>  * Preserve the stdout/stderr log file
>  * Capture debug log messages on QEMUMachine
>  * Provide a QMP backdoor for debugging stuck QEMUs
>  * Enhance console handling for partial line matches
>  * Fix the tuxrun tests by eliminating sleeps
>
> There's quite alot of code here, but at the same time it feels like
> the kind of stuff that'll be valuable either in the 9.2 release, or
> in the soon to exist 9.2 stable branch.
>
> NB, with this series applied Thomas' tuxrun conversion to functional
> testing survives 200 iterations on my machine, whereas it would
> reliably hang in < 20, and often in < 10, before.

Queued to testing/next, thanks.

I'll combine with plugins/next and some other misc fixes and post a
pre-PR for next week.

>
> Changed in v2:
>
>  - Changed console interaction to forbid 'failure_message'
>    without 'success_message'
>  - Reword console interaction log messages
>  - Avoid stack trace when seeing early failure
>  - Rewrote comment in acpi bits test
>  - Avoid duplicate os.environ access
>
> Daniel P. Berrangé (18):
>   tests/functional: fix mips64el test to honour workdir
>   tests/functional: automatically clean up scratch files after tests
>   tests/functional: remove "AVOCADO" from env variable name
>   tests/functional: remove todo wrt avocado.utils.wait_for
>   tests/functional: remove leftover :avocado: tags
>   tests/functional: remove obsolete reference to avocado bug
>   tests/functional: remove comments talking about avocado
>   tests/functional: honour self.workdir in ACPI bits tests
>   tests/functional: put QEMUMachine logs in testcase log directory
>   tests/functional: honour requested test VM name in QEMUMachine
>   tests/functional: enable debug logging for QEMUMachine
>   tests/functional: logs details of console interaction operations
>   tests/functional: don't try to wait for the empty string
>   tests/functional: require non-NULL success_message for console wait
>   tests/functional: rewrite console handling to be bytewise
>   tests/functional: remove time.sleep usage from tuxrun tests
>   tests/functional: add a QMP backdoor for debugging stalled tests
>   tests/functional: avoid accessing log_filename on earlier failures
>
>  docs/devel/testing/functional.rst        | 16 +++++
>  tests/functional/qemu_test/cmd.py        | 89 +++++++++++++++++++-----
>  tests/functional/qemu_test/testcase.py   | 43 +++++++++---
>  tests/functional/qemu_test/tuxruntest.py | 17 ++---
>  tests/functional/test_acpi_bits.py       | 57 +++++----------
>  tests/functional/test_arm_bpim2u.py      | 20 ------
>  tests/functional/test_arm_orangepi.py    | 27 -------
>  tests/functional/test_m68k_nextcube.py   |  3 +-
>  tests/functional/test_mips64el_malta.py  |  4 +-
>  tests/functional/test_virtio_gpu.py      |  3 +-
>  10 files changed, 150 insertions(+), 129 deletions(-)

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro