[PATCH v2 00/15] testing: various docker clean-ups post meson tcg conversion

Alex Bennée posted 15 patches 1 week, 3 days ago
There is a newer version of this series
configure                               | 491 +++++++++++++++++++++++-
tests/tcg/test_cc.c                     |   6 +-
.gitlab-map-auto                        |   8 +-
linux-user/aarch64/Makefile.vdso        |   2 +-
linux-user/arm/Makefile.vdso            |   2 +-
linux-user/hppa/Makefile.vdso           |   2 +-
linux-user/i386/Makefile.vdso           |   2 +-
linux-user/loongarch64/Makefile.vdso    |   2 +-
linux-user/ppc/Makefile.vdso            |   2 +-
linux-user/riscv/Makefile.vdso          |   2 +-
linux-user/s390x/Makefile.vdso          |   2 +-
linux-user/sh4/Makefile.vdso            |   2 +-
linux-user/x86_64/Makefile.vdso         |   2 +-
meson_options.txt                       |  95 +----
scripts/coverity-scan/run-coverity-scan |   2 +-
scripts/gitlab-map-update               |   2 +-
tests/docker/Makefile.include           |  16 +-
tests/docker/docker.py                  | 188 ++++-----
tests/lcitool/libvirt-ci                |   2 +-
tests/tcg/meson.build                   |  71 +---
20 files changed, 595 insertions(+), 306 deletions(-)
[PATCH v2 00/15] testing: various docker clean-ups post meson tcg conversion
Posted by Alex Bennée 1 week, 3 days ago
This cleans up a few bits and pieces although doesn't totally solve
the issue of an apparent stall while meson builds the docker image.
However with these changes there should be no difference between the
meson build and the equivalent "make docker-image-foo" invocation.

We also finally get rid of the ill-advised magic caching support in
docker.py.

v2

I've restored the configure based compiler checks for so we can build
roms and the vdso blobs again. It looks like we could never build roms
with container based images so I've left it for now. Unlike the vdso
build we can't trigger a container build through Makefile.prereqs.

TODO
----

I've still got Ilya's s390 test image patches to look at but I'm
sending v2 because enough people are hitting issues.

I've got a very hacky patch which fishes the docker image names out of
the custom commands and exposes them via Makefile.mtest which will
trigger the containers to be built outside of meson although meson
will still attempt to build (and find everything cached).

I also need to look at the timestamp stuff that Pierrick has suggested
to see if that improves the caching from the registry.

Please test and review.

Alex.

Alex Bennée (13):
  Revert "configure: remove cross_cc and gdb detection logic"
  Revert "tests/tcg/meson.build: implement ./configure -cross-cc-*
    options"
  Revert "tests/tcg/meson.build: implement ./configure containers
    options"
  configure: re-purpose the tcg test compiler configs for vdso building
  tests/tcg: prefix meson built docker tags with qemu/
  tests/tcg: use --add-current-user when building containers
  tests/docker: add container registry to configure and use
  tests/docker: remove used _read_qemu_dockerfile
  tests/docker: strip remaining caching logic from docker.py
  tests/docker: add specific command to parser
  tests/docker: swap --quiet for --verbose
  tests/docker: fix brown bag typo for docker-ccache
  gitlab: update the @user id mapping file

Daniel P. Berrangé (2):
  configure: correctly honour --disable-containers
  tests/tcg: improve check for working cross compilers

 configure                               | 491 +++++++++++++++++++++++-
 tests/tcg/test_cc.c                     |   6 +-
 .gitlab-map-auto                        |   8 +-
 linux-user/aarch64/Makefile.vdso        |   2 +-
 linux-user/arm/Makefile.vdso            |   2 +-
 linux-user/hppa/Makefile.vdso           |   2 +-
 linux-user/i386/Makefile.vdso           |   2 +-
 linux-user/loongarch64/Makefile.vdso    |   2 +-
 linux-user/ppc/Makefile.vdso            |   2 +-
 linux-user/riscv/Makefile.vdso          |   2 +-
 linux-user/s390x/Makefile.vdso          |   2 +-
 linux-user/sh4/Makefile.vdso            |   2 +-
 linux-user/x86_64/Makefile.vdso         |   2 +-
 meson_options.txt                       |  95 +----
 scripts/coverity-scan/run-coverity-scan |   2 +-
 scripts/gitlab-map-update               |   2 +-
 tests/docker/Makefile.include           |  16 +-
 tests/docker/docker.py                  | 188 ++++-----
 tests/lcitool/libvirt-ci                |   2 +-
 tests/tcg/meson.build                   |  71 +---
 20 files changed, 595 insertions(+), 306 deletions(-)

-- 
2.47.3


Re: [PATCH v2 00/15] testing: various docker clean-ups post meson tcg conversion
Posted by Richard Henderson 1 week, 3 days ago
On 9/15/26 10:15, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 
> v2
> 
> I've restored the configure based compiler checks for so we can build
> roms and the vdso blobs again. It looks like we could never build roms
> with container based images so I've left it for now. Unlike the vdso
> build we can't trigger a container build through Makefile.prereqs.
> 
> TODO
> ----
> 
> I've still got Ilya's s390 test image patches to look at but I'm
> sending v2 because enough people are hitting issues.
> 
> I've got a very hacky patch which fishes the docker image names out of
> the custom commands and exposes them via Makefile.mtest which will
> trigger the containers to be built outside of meson although meson
> will still attempt to build (and find everything cached).
> 
> I also need to look at the timestamp stuff that Pierrick has suggested
> to see if that improves the caching from the registry.
> 
> Please test and review.

Normal configure allowing podman seems to work:

>   TCG tests
>     gdb                             : /usr/bin/gdb
>     container registry              : registry.gitlab.com/qemu-project/qemu
>     cc for aarch64-softmmu          : aarch64-linux-gnu-gcc
>     cc for aarch64-linux-user       : aarch64-linux-gnu-gcc
>     cc for aarch64_be-linux-user    : aarch64-linux-gnu-gcc
>     cc for alpha-softmmu            : alpha-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for alpha-linux-user         : alpha-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for arm-softmmu              : arm-linux-gnueabihf-gcc
>     cc for arm-linux-user           : arm-linux-gnueabihf-gcc
>     cc for hexagon-linux-user       : hexagon-unknown-linux-musl-clang (from 'qemu/debian-hexagon-cross' container)
>     cc for hppa-linux-user          : hppa-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for i386-softmmu             : i686-linux-gnu-gcc
>     cc for i386-linux-user          : i686-linux-gnu-gcc
>     cc for loongarch64-softmmu      : loongarch64-unknown-linux-gnu-gcc (from 'qemu/debian-loongarch-cross' container)
>     cc for loongarch64-linux-user   : loongarch64-unknown-linux-gnu-gcc (from 'qemu/debian-loongarch-cross' container)
>     cc for m68k-linux-user          : m68k-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for mips-linux-user          : mips-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for mips64-linux-user        : mips64-linux-gnuabi64-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for mips64el-linux-user      : mips64el-linux-gnuabi64-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for or1k-linux-user          : or1k-none-linux-musl-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for ppc64-linux-user         : powerpc64-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for ppc64le-linux-user       : powerpc64le-linux-gnu-gcc
>     cc for riscv64-softmmu          : riscv64-linux-gnu-gcc
>     cc for riscv64-linux-user       : riscv64-linux-gnu-gcc
>     cc for s390x-softmmu            : s390x-linux-gnu-gcc
>     cc for s390x-linux-user         : s390x-linux-gnu-gcc
>     cc for sh4-linux-user           : sh4-linux-gnu-gcc (from 'qemu/debian-all-test-cross' container)
>     cc for tricore-softmmu          : tricore-gcc (from 'qemu/debian-tricore-cross' container)
>     cc for x86_64-softmmu           : x86_64-linux-gnu-gcc
>     cc for x86_64-linux-user        : x86_64-linux-gnu-gcc
>     cc for xtensa-softmmu           : xtensa-dc232b-elf-gcc (from 'qemu/debian-xtensa-cross' container)
>     cc for xtensaeb-softmmu         : xtensa-test_kc705_be-elf-gcc (from 'qemu/debian-xtensa-cross' container)

However --disable-containers yields the exact same output ("from ... 
container" and all).  Confirmed by "make check-tcg" attempting to 
rebuild said containers.  So that's not working at all.


r~