[PATCH 0/9] testing: various docker clean-ups post meson tcg conversion

Alex Bennée posted 9 patches 2 weeks, 2 days ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20260910181504.1010916-1-alex.bennee@linaro.org
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>, Peter Maydell <peter.maydell@linaro.org>
There is a newer version of this series
configure                               |   8 +
tests/tcg/test_cc.c                     |   6 +-
meson_options.txt                       |   3 +
scripts/coverity-scan/run-coverity-scan |   2 +-
scripts/meson-buildoptions.sh           | 142 ++++++++++++++++++
tests/docker/Makefile.include           |  14 +-
tests/docker/docker.py                  | 188 +++++++++---------------
tests/lcitool/libvirt-ci                |   2 +-
tests/tcg/meson.build                   |  15 +-
9 files changed, 243 insertions(+), 137 deletions(-)
[PATCH 0/9] testing: various docker clean-ups post meson tcg conversion
Posted by Alex Bennée 2 weeks, 2 days ago
This cleans up a few bits and pieces although doesn't totally solve
the issue of an apparent stall while meson builds the docker image.
However with these changes there should be no difference between the
meson build and the equivalent "make docker-image-foo" invocation.

We also finally get rid of the ill-advised magic caching support in
docker.py.

Please review.

Alex.

Alex Bennée (8):
  scripts/meson-buildoptions.sh: update missing options
  tests/tcg: prefix meson built docker tags with qemu/
  tests/tcg: use --add-current-user when building containers
  tests/docker: add container registry to configure and use
  tests/docker: remove used _read_qemu_dockerfile
  tests/docker: strip remaining caching logic from docker.py
  tests/docker: add specific command to parser
  tests/docker: swap --quiet for --verbose

Daniel P. Berrangé (1):
  tests/tcg: improve check for working cross compilers

 configure                               |   8 +
 tests/tcg/test_cc.c                     |   6 +-
 meson_options.txt                       |   3 +
 scripts/coverity-scan/run-coverity-scan |   2 +-
 scripts/meson-buildoptions.sh           | 142 ++++++++++++++++++
 tests/docker/Makefile.include           |  14 +-
 tests/docker/docker.py                  | 188 +++++++++---------------
 tests/lcitool/libvirt-ci                |   2 +-
 tests/tcg/meson.build                   |  15 +-
 9 files changed, 243 insertions(+), 137 deletions(-)

-- 
2.47.3


Re: [PATCH 0/9] testing: various docker clean-ups post meson tcg conversion
Posted by Richard Henderson 2 weeks ago
On 9/10/26 08:14, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 
> Please review.
> 
> Alex.
> 
> Alex Bennée (8):
>    scripts/meson-buildoptions.sh: update missing options
>    tests/tcg: prefix meson built docker tags with qemu/
>    tests/tcg: use --add-current-user when building containers
>    tests/docker: add container registry to configure and use
>    tests/docker: remove used _read_qemu_dockerfile
>    tests/docker: strip remaining caching logic from docker.py
>    tests/docker: add specific command to parser
>    tests/docker: swap --quiet for --verbose

Something in here breaks things badly:

$ make
/home/rth/qemu/bld/pyvenv/bin/meson introspect --buildoptions 
/home/rth/qemu/src/meson.build | /home/rth/qemu/bld/pyvenv/bin/python3 -B \
   scripts/meson-buildoptions.py > 
/home/rth/qemu/src/scripts/meson-buildoptions.sh.tmp && mv 
/home/rth/qemu/src/scripts/meson-buildoptions.sh.tmp 
/home/rth/qemu/src/scripts/meson-buildoptions.sh
Failure in scripts/meson-buildoptions.py parsing stdin as json

../src/meson.build:96:12: ERROR: Unknown compiler(s): [['rustc']]
The following exception(s) were encountered:
Running `rustc --version` gave "[Errno 2] No such file or directory: 
'rustc'"

make: *** [Makefile:134: 
/home/rth/qemu/src/scripts/meson-buildoptions.sh] Error 1


r~

Re: [PATCH 0/9] testing: various docker clean-ups post meson tcg conversion
Posted by Alex Bennée 2 weeks ago
Richard Henderson <richard.henderson@linaro.org> writes:

> On 9/10/26 08:14, Alex Bennée wrote:
>> This cleans up a few bits and pieces although doesn't totally solve
>> the issue of an apparent stall while meson builds the docker image.
>> However with these changes there should be no difference between the
>> meson build and the equivalent "make docker-image-foo" invocation.
>> We also finally get rid of the ill-advised magic caching support in
>> docker.py.
>> Please review.
>> Alex.
>> Alex Bennée (8):
>>    scripts/meson-buildoptions.sh: update missing options
>>    tests/tcg: prefix meson built docker tags with qemu/
>>    tests/tcg: use --add-current-user when building containers
>>    tests/docker: add container registry to configure and use
>>    tests/docker: remove used _read_qemu_dockerfile
>>    tests/docker: strip remaining caching logic from docker.py
>>    tests/docker: add specific command to parser
>>    tests/docker: swap --quiet for --verbose
>
> Something in here breaks things badly:
>
> $ make
> /home/rth/qemu/bld/pyvenv/bin/meson introspect --buildoptions
> /home/rth/qemu/src/meson.build | /home/rth/qemu/bld/pyvenv/bin/python3
> -B \
>   scripts/meson-buildoptions.py >
>   /home/rth/qemu/src/scripts/meson-buildoptions.sh.tmp && mv
>   /home/rth/qemu/src/scripts/meson-buildoptions.sh.tmp
>   /home/rth/qemu/src/scripts/meson-buildoptions.sh
> Failure in scripts/meson-buildoptions.py parsing stdin as json
>
> ../src/meson.build:96:12: ERROR: Unknown compiler(s): [['rustc']]
> The following exception(s) were encountered:
> Running `rustc --version` gave "[Errno 2] No such file or directory:
> 'rustc'"
>
> make: *** [Makefile:134:
> /home/rth/qemu/src/scripts/meson-buildoptions.sh] Error 1

I think this goes away with a clean reconfigure to trigger a newer meson.

>
>
> r~

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [PATCH 0/9] testing: various docker clean-ups post meson tcg conversion
Posted by Richard Henderson 2 weeks ago
On 9/10/26 08:14, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 
> Please review.
> 
> Alex.
> 
> Alex Bennée (8):
>    scripts/meson-buildoptions.sh: update missing options
>    tests/tcg: prefix meson built docker tags with qemu/
>    tests/tcg: use --add-current-user when building containers
>    tests/docker: add container registry to configure and use
>    tests/docker: remove used _read_qemu_dockerfile
>    tests/docker: strip remaining caching logic from docker.py
>    tests/docker: add specific command to parser
>    tests/docker: swap --quiet for --verbose
I haven't had time to pull this into a local branch, but I don't suppose 
something in here fixes

> [6/572] Generating tests/tcg/alpha-softmmu-hello with a custom command
> FAILED: [code=1] tests/tcg/alpha-softmmu-hello.test 
> /home/rth/qemu/src/tests/tcg/../docker/docker.py --command podman run --run-as-current-user -w /home/rth/qemu/bld-g -v /home/rth/qemu/src:/home/rth/qemu/src debian-all-test-cross alpha-linux-gnu-gcc /home/rth/qemu/src/tests/tcg/multiarch/system/hello.c -o tests/tcg/alpha-softmmu-hello.test -static -MMD -MF tests/tcg/alpha-softmmu-hello.d -Wall -Werror -O0 -g -fno-strict-aliasing -nostdlib -ffreestanding -Wa,--noexecstack -I /home/rth/qemu/src/tests/tcg/alpha/system/../../minilib /home/rth/qemu/src/tests/tcg/alpha/system/../../minilib/printf.c ../src/tests/tcg/alpha/system/boot.S -Wl,-T ../src/tests/tcg/alpha/system/kernel.ld -mcpu=ev6 -Wl,--no-warn-rwx-segments -lgcc
> Error: workdir "/home/rth/qemu/bld-g" does not exist on container 7670fc4a955a94d801b57dc0a6f6cd13ece35536a09b40634a37a83d733af942
> Traceback (most recent call last):
>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 657, in <module>
>     sys.exit(main())
>              ~~~~^^
>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 653, in main
>     return args.cmdobj.run(args, argv)
>            ~~~~~~~~~~~~~~~^^^^^^^^^^^^
>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 395, in run
>     return Docker(args.command).run(argv, args.keep, quiet=args.quiet,
>            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>                                     as_user=args.run_as_current_user)
>                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 353, in run
>     ret = self._do_check(["run", "--rm", "--label",
>                          "com.qemu.instance.uuid=" + label] + cmd,
>                          quiet=quiet)
>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 230, in _do_check
>     return subprocess.check_call(self._command + cmd, **kwargs)
>            ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.14/subprocess.py", line 420, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['podman', 'run', '--rm', '--label', 'com.qemu.instance.uuid=9ca55b05b3e540f7a0cda353bf6f1086', '--userns=keep-id', '-u', '1000', '-w', '/home/rth/qemu/bld-g', '-v', '/home/rth/qemu/src:/home/rth/qemu/src', 'debian-all-test-cross', 'alpha-linux-gnu-gcc', '/home/rth/qemu/src/tests/tcg/multiarch/system/hello.c', '-o', 'tests/tcg/alpha-softmmu-hello.test', '-static', '-MMD', '-MF', 'tests/tcg/alpha-softmmu-hello.d', '-Wall', '-Werror', '-O0', '-g', '-fno-strict-aliasing', '-nostdlib', '-ffreestanding', '-Wa,--noexecstack', '-I', '/home/rth/qemu/src/tests/tcg/alpha/system/../../minilib', '/home/rth/qemu/src/tests/tcg/alpha/system/../../minilib/printf.c', '../src/tests/tcg/alpha/system/boot.S', '-Wl,-T', '../src/tests/tcg/alpha/system/kernel.ld', '-mcpu=ev6', '-Wl,--no-warn-rwx-segments', '-lgcc']' returned non-zero exit status 126.
> filter=--filter=label=com.qemu.instance.uuid=9ca55b05b3e540f7a0cda353bf6f1086
> ninja: build stopped: subcommand failed.
> make: *** [Makefile:170: run-ninja] Error 1

For avoidance of doubt, this is immediately after

   make docker-image-debian-all-test-cross NOCACHE=1 V=1


r~

Re: [PATCH 0/9] testing: various docker clean-ups post meson tcg conversion
Posted by Alex Bennée 2 weeks ago
Richard Henderson <richard.henderson@linaro.org> writes:

> On 9/10/26 08:14, Alex Bennée wrote:
>> This cleans up a few bits and pieces although doesn't totally solve
>> the issue of an apparent stall while meson builds the docker image.
>> However with these changes there should be no difference between the
>> meson build and the equivalent "make docker-image-foo" invocation.
>> We also finally get rid of the ill-advised magic caching support in
>> docker.py.
>> Please review.
>> Alex.
>> Alex Bennée (8):
>>    scripts/meson-buildoptions.sh: update missing options
>>    tests/tcg: prefix meson built docker tags with qemu/
>>    tests/tcg: use --add-current-user when building containers
>>    tests/docker: add container registry to configure and use
>>    tests/docker: remove used _read_qemu_dockerfile
>>    tests/docker: strip remaining caching logic from docker.py
>>    tests/docker: add specific command to parser
>>    tests/docker: swap --quiet for --verbose
> I haven't had time to pull this into a local branch, but I don't
> suppose something in here fixes
>
>> [6/572] Generating tests/tcg/alpha-softmmu-hello with a custom command
>> FAILED: [code=1] tests/tcg/alpha-softmmu-hello.test
>> /home/rth/qemu/src/tests/tcg/../docker/docker.py --command podman
>> run --run-as-current-user -w /home/rth/qemu/bld-g -v

yes - the -v is missing it's args which is fixed by the argparse patch
before I switched --quiet and --verbose which exposed the latent bug.

>> /home/rth/qemu/src:/home/rth/qemu/src debian-all-test-cross
>> alpha-linux-gnu-gcc
>> /home/rth/qemu/src/tests/tcg/multiarch/system/hello.c -o
>> tests/tcg/alpha-softmmu-hello.test -static -MMD -MF
>> tests/tcg/alpha-softmmu-hello.d -Wall -Werror -O0 -g
>> -fno-strict-aliasing -nostdlib -ffreestanding -Wa,--noexecstack -I
>> /home/rth/qemu/src/tests/tcg/alpha/system/../../minilib
>> /home/rth/qemu/src/tests/tcg/alpha/system/../../minilib/printf.c
>> ../src/tests/tcg/alpha/system/boot.S -Wl,-T
>> ../src/tests/tcg/alpha/system/kernel.ld -mcpu=ev6
>> -Wl,--no-warn-rwx-segments -lgcc
>> Error: workdir "/home/rth/qemu/bld-g" does not exist on container 7670fc4a955a94d801b57dc0a6f6cd13ece35536a09b40634a37a83d733af942
>> Traceback (most recent call last):
>>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 657, in <module>
>>     sys.exit(main())
>>              ~~~~^^
>>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 653, in main
>>     return args.cmdobj.run(args, argv)
>>            ~~~~~~~~~~~~~~~^^^^^^^^^^^^
>>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 395, in run
>>     return Docker(args.command).run(argv, args.keep, quiet=args.quiet,
>>            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>                                     as_user=args.run_as_current_user)
>>                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 353, in run
>>     ret = self._do_check(["run", "--rm", "--label",
>>                          "com.qemu.instance.uuid=" + label] + cmd,
>>                          quiet=quiet)
>>   File "/home/rth/qemu/src/tests/tcg/../docker/docker.py", line 230, in _do_check
>>     return subprocess.check_call(self._command + cmd, **kwargs)
>>            ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>   File "/usr/lib/python3.14/subprocess.py", line 420, in check_call
>>     raise CalledProcessError(retcode, cmd)
>> subprocess.CalledProcessError: Command '['podman', 'run', '--rm', '--label', 'com.qemu.instance.uuid=9ca55b05b3e540f7a0cda353bf6f1086', '--userns=keep-id', '-u', '1000', '-w', '/home/rth/qemu/bld-g', '-v', '/home/rth/qemu/src:/home/rth/qemu/src', 'debian-all-test-cross', 'alpha-linux-gnu-gcc', '/home/rth/qemu/src/tests/tcg/multiarch/system/hello.c', '-o', 'tests/tcg/alpha-softmmu-hello.test', '-static', '-MMD', '-MF', 'tests/tcg/alpha-softmmu-hello.d', '-Wall', '-Werror', '-O0', '-g', '-fno-strict-aliasing', '-nostdlib', '-ffreestanding', '-Wa,--noexecstack', '-I', '/home/rth/qemu/src/tests/tcg/alpha/system/../../minilib', '/home/rth/qemu/src/tests/tcg/alpha/system/../../minilib/printf.c', '../src/tests/tcg/alpha/system/boot.S', '-Wl,-T', '../src/tests/tcg/alpha/system/kernel.ld', '-mcpu=ev6', '-Wl,--no-warn-rwx-segments', '-lgcc']' returned non-zero exit status 126.
>> filter=--filter=label=com.qemu.instance.uuid=9ca55b05b3e540f7a0cda353bf6f1086
>> ninja: build stopped: subcommand failed.
>> make: *** [Makefile:170: run-ninja] Error 1
>
> For avoidance of doubt, this is immediately after
>
>   make docker-image-debian-all-test-cross NOCACHE=1 V=1
>
>
> r~

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro