[RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)

Thomas Huth posted 2 patches 1 week, 6 days ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20250915124207.42053-1-thuth@redhat.com
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, Thomas Huth <thuth@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Zhao Liu <zhao1.liu@intel.com>
configure                                     |   2 +
meson.build                                   |   4 +
meson_options.txt                             |   2 +
scripts/meson-buildoptions.sh                 |   2 +
.../functional/aarch64/test_reverse_debug.py  |  16 +-
tests/functional/meson.build                  |   7 +
tests/functional/ppc64/test_reverse_debug.py  |  18 +-
tests/functional/reverse_debugging.py         | 235 +++++++++++-------
tests/functional/x86_64/test_reverse_debug.py |  20 +-
9 files changed, 188 insertions(+), 118 deletions(-)
[RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Thomas Huth 1 week, 6 days ago
Here's yet another attempt to remove the avocado dependency from the
reverse debugging tests: I basically took Gustavo's patches to rework
tests/functional/reverse_debugging.py, but instead of calling that
through tests/guest-debug/run-test.py and adding the cumbersome code
to support additional test execution logic, I kept our normal way of
running tests via pycotap. Instead, the essential logic for running
gdb is copied from tests/guest-debug/run-test.py into the new function
reverse_debug() that then runs gdb directly with using
tests/functional/reverse_debugging.py as the script.

Marked as an RFC since this still needs some love... The aarch64 test
seems to work already (after applying the fix for the reverse debug there
first: https://patchew.org/QEMU/20250603125459.17688-1-1844144@gmail.com/ ),
but the ppc64 and x86 tests are currently still completely broken.
Also we currently log into two different folders this way, into
tests/functional/aarch64/test_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
for the normal outer test, and into
tests/functional/aarch64/reverse_debugging.ReverseDebugging.test_reverse_debugging
for the script that is run in gdb ... it's likely ok for the aarch64
test, but the ppc64 test contains two subtests, so we need to come up
with a better solution here for the final implementation.

Gustavo Romero (2):
  tests/functional: Provide GDB to the functional tests
  tests/functional: Adapt reverse_debugging to run w/o Avocado

 configure                                     |   2 +
 meson.build                                   |   4 +
 meson_options.txt                             |   2 +
 scripts/meson-buildoptions.sh                 |   2 +
 .../functional/aarch64/test_reverse_debug.py  |  16 +-
 tests/functional/meson.build                  |   7 +
 tests/functional/ppc64/test_reverse_debug.py  |  18 +-
 tests/functional/reverse_debugging.py         | 235 +++++++++++-------
 tests/functional/x86_64/test_reverse_debug.py |  20 +-
 9 files changed, 188 insertions(+), 118 deletions(-)

-- 
2.51.0
Re: [RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Daniel P. Berrangé 1 week, 5 days ago
On Mon, Sep 15, 2025 at 02:42:05PM +0200, Thomas Huth wrote:
> Here's yet another attempt to remove the avocado dependency from the
> reverse debugging tests: I basically took Gustavo's patches to rework
> tests/functional/reverse_debugging.py, but instead of calling that
> through tests/guest-debug/run-test.py and adding the cumbersome code
> to support additional test execution logic, I kept our normal way of
> running tests via pycotap. Instead, the essential logic for running
> gdb is copied from tests/guest-debug/run-test.py into the new function
> reverse_debug() that then runs gdb directly with using
> tests/functional/reverse_debugging.py as the script.

Something I've not previously realized is that when run via GDB, we are
not honouring the Python version we chose to use with QEMU. GDB is not
actually running the python interpreter binary, instead it has linked to
the libpython.so and runs python code in-process to GDB. Thus the version
of python being used is out of our control, it is whatever the distro
chose to link GDB to. When I look back at how we've handled our min
python, vs what the distros use as system python, I think the constraint
is highly undesirable.

> Marked as an RFC since this still needs some love... The aarch64 test
> seems to work already (after applying the fix for the reverse debug there
> first: https://patchew.org/QEMU/20250603125459.17688-1-1844144@gmail.com/ ),
> but the ppc64 and x86 tests are currently still completely broken.
> Also we currently log into two different folders this way, into
> tests/functional/aarch64/test_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
> for the normal outer test, and into
> tests/functional/aarch64/reverse_debugging.ReverseDebugging.test_reverse_debugging
> for the script that is run in gdb ... it's likely ok for the aarch64
> test, but the ppc64 test contains two subtests, so we need to come up
> with a better solution here for the final implementation.

Right, this is one of the things I rather dislike with this, as it is
making debugging much more painful.

In one base.log you just get stdout from gdb with no context of what
commands are run, and in the other base.log you get the logs from the
sub-process. Well at least you would if we fixed the tests to use
self.log, instead of creating a new logger category.

Even with that fixed though, it is very difficult to correlate GDB
output in one log, with GDB commands invoke in the other log, in
order to understand why it is failing in the x86/ppc tests.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Gustavo Romero 1 week, 5 days ago
Hi Thomas,

On 9/15/25 09:42, Thomas Huth wrote:
> Here's yet another attempt to remove the avocado dependency from the
> reverse debugging tests: I basically took Gustavo's patches to rework
> tests/functional/reverse_debugging.py, but instead of calling that
> through tests/guest-debug/run-test.py and adding the cumbersome code
> to support additional test execution logic, I kept our normal way of
> running tests via pycotap. Instead, the essential logic for running
> gdb is copied from tests/guest-debug/run-test.py into the new function
> reverse_debug() that then runs gdb directly with using
> tests/functional/reverse_debugging.py as the script.


Thanks a lot for this series. It's neat that we don't need to touch
meson.build, TAP works, and no additional module is used.

Thumbs up from my side :)

I just have some minor comments about it, inline in the patches.


> Marked as an RFC since this still needs some love... The aarch64 test
> seems to work already (after applying the fix for the reverse debug there
> first: https://patchew.org/QEMU/20250603125459.17688-1-1844144@gmail.com/ ),
> but the ppc64 and x86 tests are currently still completely broken.
> Also we currently log into two different folders this way, into
> tests/functional/aarch64/test_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
> for the normal outer test, and into
> tests/functional/aarch64/reverse_debugging.ReverseDebugging.test_reverse_debugging
> for the script that is run in gdb ... it's likely ok for the aarch64
> test, but the ppc64 test contains two subtests, so we need to come up
> with a better solution here for the final implementation.

Although I don't know how to fix this last bit, I think besides the
log files nothing really changes for any arch in comparison with the
current version with Avocado in the tree.


Cheers,
Gustavo

> Gustavo Romero (2):
>    tests/functional: Provide GDB to the functional tests
>    tests/functional: Adapt reverse_debugging to run w/o Avocado
> 
>   configure                                     |   2 +
>   meson.build                                   |   4 +
>   meson_options.txt                             |   2 +
>   scripts/meson-buildoptions.sh                 |   2 +
>   .../functional/aarch64/test_reverse_debug.py  |  16 +-
>   tests/functional/meson.build                  |   7 +
>   tests/functional/ppc64/test_reverse_debug.py  |  18 +-
>   tests/functional/reverse_debugging.py         | 235 +++++++++++-------
>   tests/functional/x86_64/test_reverse_debug.py |  20 +-
>   9 files changed, 188 insertions(+), 118 deletions(-)
>
Re: [RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Alex Bennée 1 week, 6 days ago
Thomas Huth <thuth@redhat.com> writes:

> Here's yet another attempt to remove the avocado dependency from the
> reverse debugging tests: I basically took Gustavo's patches to rework
> tests/functional/reverse_debugging.py, but instead of calling that
> through tests/guest-debug/run-test.py and adding the cumbersome code
> to support additional test execution logic, I kept our normal way of
> running tests via pycotap.

Hmm I was getting:

  2025-09-15 17:10:50,798 - INFO: GDB CMD: /home/alex/src/tools/binutils-gdb.git/builds/all/install/bin/gdb -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py']" -x /home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py
  2025-09-15 17:10:50,803 - DEBUG: Using cached asset /home/alex/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
  2025-09-15 17:10:50,891 - INFO: gdb output:
   Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
  Error occurred in Python: No module named 'pycotap'

which asserted. Should gdb be seeing the pyenv PYTHONPATH or should we
add a check for python3-pycotap in configure?

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Thomas Huth 1 week, 6 days ago
On 15/09/2025 18.13, Alex Bennée wrote:
> Thomas Huth <thuth@redhat.com> writes:
> 
>> Here's yet another attempt to remove the avocado dependency from the
>> reverse debugging tests: I basically took Gustavo's patches to rework
>> tests/functional/reverse_debugging.py, but instead of calling that
>> through tests/guest-debug/run-test.py and adding the cumbersome code
>> to support additional test execution logic, I kept our normal way of
>> running tests via pycotap.
> 
> Hmm I was getting:
> 
>    2025-09-15 17:10:50,798 - INFO: GDB CMD: /home/alex/src/tools/binutils-gdb.git/builds/all/install/bin/gdb -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py']" -x /home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py
>    2025-09-15 17:10:50,803 - DEBUG: Using cached asset /home/alex/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
>    2025-09-15 17:10:50,891 - INFO: gdb output:
>     Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
>    Error occurred in Python: No module named 'pycotap'

Ah, sorry, I have it installed pycotap system-wide, too, so I did not notice 
it... I'll fix it in the next version if we decide to proceed with this 
approach instead of using one of the others.

  Thomas


Re: [RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Gustavo Romero 1 week, 5 days ago
Hi Thomas,

On 9/15/25 13:18, Thomas Huth wrote:
> On 15/09/2025 18.13, Alex Bennée wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>>
>>> Here's yet another attempt to remove the avocado dependency from the
>>> reverse debugging tests: I basically took Gustavo's patches to rework
>>> tests/functional/reverse_debugging.py, but instead of calling that
>>> through tests/guest-debug/run-test.py and adding the cumbersome code
>>> to support additional test execution logic, I kept our normal way of
>>> running tests via pycotap.
>>
>> Hmm I was getting:
>>
>>    2025-09-15 17:10:50,798 - INFO: GDB CMD: /home/alex/src/tools/binutils-gdb.git/builds/all/install/bin/gdb -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py']" -x /home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py
>>    2025-09-15 17:10:50,803 - DEBUG: Using cached asset /home/alex/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
>>    2025-09-15 17:10:50,891 - INFO: gdb output:
>>     Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
>>    Error occurred in Python: No module named 'pycotap'
> 
> Ah, sorry, I have it installed pycotap system-wide, too, so I did not notice it... I'll fix it in the next version if we decide to proceed with this approach instead of using one of the others.

See my comment in patch 1/2 about adding python_site_packages to
PYTHONPATH to fix it. I think there is no harm to the other tests
by adding it to the test_env, so I don't think it's worth taking
care of it only in reverse_debugging.py.


Cheers,
Gustavo

Re: [RFC PATCH 0/2] tests/functional: Adapt reverse_debugging to run w/o Avocado (yet another try)
Posted by Alex Bennée 1 week, 5 days ago
Thomas Huth <thuth@redhat.com> writes:

> On 15/09/2025 18.13, Alex Bennée wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>> 
>>> Here's yet another attempt to remove the avocado dependency from the
>>> reverse debugging tests: I basically took Gustavo's patches to rework
>>> tests/functional/reverse_debugging.py, but instead of calling that
>>> through tests/guest-debug/run-test.py and adding the cumbersome code
>>> to support additional test execution logic, I kept our normal way of
>>> running tests via pycotap.
>> Hmm I was getting:
>>    2025-09-15 17:10:50,798 - INFO: GDB CMD:
>> /home/alex/src/tools/binutils-gdb.git/builds/all/install/bin/gdb -q
>> -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py
>> sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py']"
>> -x /home/alex/lsrc/qemu.git/tests/functional/reverse_debugging.py
>>    2025-09-15 17:10:50,803 - DEBUG: Using cached asset /home/alex/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
>>    2025-09-15 17:10:50,891 - INFO: gdb output:
>>     Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
>>    Error occurred in Python: No module named 'pycotap'
>
> Ah, sorry, I have it installed pycotap system-wide, too, so I did not
> notice it... I'll fix it in the next version if we decide to proceed
> with this approach instead of using one of the others.

FWIW I prefer this approach.

>
>  Thomas

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro