[PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

Alex Bennée posted 17 patches 1 week, 2 days ago
configure                               | 465 +++++++++++++++++++++++-
tests/tcg/s390x/libc/stdio.h            |  11 +
tests/tcg/s390x/libc/string.h           |  12 +
.gitlab-ci.d/buildtest.yml              |   2 +-
.gitlab-map-auto                        |   8 +-
linux-user/aarch64/Makefile.vdso        |   2 +-
linux-user/arm/Makefile.vdso            |   2 +-
linux-user/hppa/Makefile.vdso           |   2 +-
linux-user/i386/Makefile.vdso           |   2 +-
linux-user/loongarch64/Makefile.vdso    |   2 +-
linux-user/ppc/Makefile.vdso            |   2 +-
linux-user/riscv/Makefile.vdso          |   2 +-
linux-user/s390x/Makefile.vdso          |   2 +-
linux-user/sh4/Makefile.vdso            |   2 +-
linux-user/x86_64/Makefile.vdso         |   2 +-
meson_options.txt                       |   3 +
scripts/coverity-scan/run-coverity-scan |   2 +-
scripts/gitlab-map-update               |   2 +-
scripts/meson-buildoptions.sh           |   3 +
tests/docker/Makefile.include           |   6 +-
tests/docker/docker.py                  | 172 +++------
tests/tcg/meson.build                   |  30 +-
tests/tcg/s390x/system/meson.build      |   2 +
23 files changed, 591 insertions(+), 147 deletions(-)
create mode 100644 tests/tcg/s390x/libc/stdio.h
create mode 100644 tests/tcg/s390x/libc/string.h
[PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)
Posted by Alex Bennée 1 week, 2 days ago
This cleans up a few bits and pieces although doesn't totally solve
the issue of an apparent stall while meson builds the docker image.
However with these changes there should be no difference between the
meson build and the equivalent "make docker-image-foo" invocation.

We also finally get rid of the ill-advised magic caching support in
docker.py.

v5
 - fixup a bunch of rebase damage
 - swap out my ccache fix for Anders
 - rename revert to "configure: ensure it also honours container configuration"
 - use Pierricks solution for avoid too many SKIPs while validating config

v4
 - added Illya's fixes for out of tree and s390x tests
 - tweaked the revert so --disable-containers still passed to meson
 - don't define tests if no compiler available to avoid excess SKIPs

v3

 - Update make2test to trigger docker build from Makefile.include.
 - add Daniels's suggested console: true patch
 - fixed up some unref vars from the reverts

v2

I've restored the configure based compiler checks for so we can build
roms and the vdso blobs again. It looks like we could never build roms
with container based images so I've left it for now. Unlike the vdso
build we can't trigger a container build through Makefile.prereqs.

TODO
----

I also need to look at the timestamp stuff that Pierrick has suggested
to see if that improves the caching from the registry. However that
can wait until after the PR.

The following still need review:

  gitlab: drop check-tcg from clang-system job
  tests/tcg: don't define tests when missing compilers

Please test and review.

Alex.

Alex Bennée (13):
  Revert "configure: remove cross_cc and gdb detection logic"
  configure: ensure it also honours container configuration
  configure: re-purpose the tcg test compiler configs for vdso building
  tests/tcg: prefix meson built docker tags with qemu/
  tests/tcg: use --add-current-user when building containers
  tests/tcg: don't define tests when missing compilers
  tests/docker: add container registry to configure and use
  tests/docker: remove used _read_qemu_dockerfile
  tests/docker: strip remaining caching logic from docker.py
  tests/docker: add specific command to parser
  gitlab: update the @user id mapping file
  tests/tcg: add console property to docker image custom target
  gitlab: drop check-tcg from clang-system job

Anders Roxell (1):
  tests/docker: fix ccache directory variable expansion

Daniel P. Berrangé (1):
  configure: correctly honour --disable-containers

Ilya Leoshkevich (2):
  tests/tcg/s390x: Provide the libc bits needed by the system tests
  tests/tcg: Fix building outside of the source tree

 configure                               | 465 +++++++++++++++++++++++-
 tests/tcg/s390x/libc/stdio.h            |  11 +
 tests/tcg/s390x/libc/string.h           |  12 +
 .gitlab-ci.d/buildtest.yml              |   2 +-
 .gitlab-map-auto                        |   8 +-
 linux-user/aarch64/Makefile.vdso        |   2 +-
 linux-user/arm/Makefile.vdso            |   2 +-
 linux-user/hppa/Makefile.vdso           |   2 +-
 linux-user/i386/Makefile.vdso           |   2 +-
 linux-user/loongarch64/Makefile.vdso    |   2 +-
 linux-user/ppc/Makefile.vdso            |   2 +-
 linux-user/riscv/Makefile.vdso          |   2 +-
 linux-user/s390x/Makefile.vdso          |   2 +-
 linux-user/sh4/Makefile.vdso            |   2 +-
 linux-user/x86_64/Makefile.vdso         |   2 +-
 meson_options.txt                       |   3 +
 scripts/coverity-scan/run-coverity-scan |   2 +-
 scripts/gitlab-map-update               |   2 +-
 scripts/meson-buildoptions.sh           |   3 +
 tests/docker/Makefile.include           |   6 +-
 tests/docker/docker.py                  | 172 +++------
 tests/tcg/meson.build                   |  30 +-
 tests/tcg/s390x/system/meson.build      |   2 +
 23 files changed, 591 insertions(+), 147 deletions(-)
 create mode 100644 tests/tcg/s390x/libc/stdio.h
 create mode 100644 tests/tcg/s390x/libc/string.h

-- 
2.47.3


Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)
Posted by Pierrick Bouvier 1 week, 2 days ago
On 9/17/2026 9:55 AM, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 
> v5
>  - fixup a bunch of rebase damage
>  - swap out my ccache fix for Anders
>  - rename revert to "configure: ensure it also honours container configuration"
>  - use Pierricks solution for avoid too many SKIPs while validating config
>

It seems that this version removes patch:
Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"

That's fine and correct, but we're missing two things:
- cross-cc-cflags and prefix should be passed to config.mak also
(similar to the merge for container options). Not strictly needed as I'm
not sure anyone needs this "extra" flexibility for roms/vdso, but needed
if we want to make sure we don't break anything for anyone.
- scripts/meson_buildoptions.sh update that was missing originally for
those meson options.

For the rest, it looks good and ready!

Regards,
Pierrick

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)
Posted by Alex Bennée 1 week, 2 days ago
Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>> This cleans up a few bits and pieces although doesn't totally solve
>> the issue of an apparent stall while meson builds the docker image.
>> However with these changes there should be no difference between the
>> meson build and the equivalent "make docker-image-foo" invocation.
>> 
>> We also finally get rid of the ill-advised magic caching support in
>> docker.py.
>> 
>> v5
>>  - fixup a bunch of rebase damage
>>  - swap out my ccache fix for Anders
>>  - rename revert to "configure: ensure it also honours container configuration"
>>  - use Pierricks solution for avoid too many SKIPs while validating config
>>
>
> It seems that this version removes patch:
> Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"
>
> That's fine and correct, but we're missing two things:
> - cross-cc-cflags and prefix should be passed to config.mak also
> (similar to the merge for container options). Not strictly needed as I'm
> not sure anyone needs this "extra" flexibility for roms/vdso, but needed
> if we want to make sure we don't break anything for anyone.
> - scripts/meson_buildoptions.sh update that was missing originally for
> those meson options.

See: 20260917185322.3457036-1-alex.bennee@linaro.org

>
> For the rest, it looks good and ready!
>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)
Posted by Alex Bennée 1 week, 2 days ago
Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>> This cleans up a few bits and pieces although doesn't totally solve
>> the issue of an apparent stall while meson builds the docker image.
>> However with these changes there should be no difference between the
>> meson build and the equivalent "make docker-image-foo" invocation.
>> 
>> We also finally get rid of the ill-advised magic caching support in
>> docker.py.
>> 
>> v5
>>  - fixup a bunch of rebase damage
>>  - swap out my ccache fix for Anders
>>  - rename revert to "configure: ensure it also honours container configuration"
>>  - use Pierricks solution for avoid too many SKIPs while validating config
>>
>
> It seems that this version removes patch:
> Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"
>
> That's fine and correct, but we're missing two things:
> - cross-cc-cflags and prefix should be passed to config.mak also
> (similar to the merge for container options). Not strictly needed as I'm
> not sure anyone needs this "extra" flexibility for roms/vdso, but needed
> if we want to make sure we don't break anything for anyone.

The rom builds are outside the main makefile anyway so they just include
random bits as they want.

> - scripts/meson_buildoptions.sh update that was missing originally for
> those meson options.

We need to do something about it because at various times
"update-buildoptions" does and doesn't change things. Sometimes there
are more options than I've been messing with. It is super confusing.

>
> For the rest, it looks good and ready!
>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro
Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)
Posted by Pierrick Bouvier 1 week, 2 days ago
On 9/17/2026 10:45 AM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>>> This cleans up a few bits and pieces although doesn't totally solve
>>> the issue of an apparent stall while meson builds the docker image.
>>> However with these changes there should be no difference between the
>>> meson build and the equivalent "make docker-image-foo" invocation.
>>>
>>> We also finally get rid of the ill-advised magic caching support in
>>> docker.py.
>>>
>>> v5
>>>  - fixup a bunch of rebase damage
>>>  - swap out my ccache fix for Anders
>>>  - rename revert to "configure: ensure it also honours container configuration"
>>>  - use Pierricks solution for avoid too many SKIPs while validating config
>>>
>>
>> It seems that this version removes patch:
>> Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"
>>
>> That's fine and correct, but we're missing two things:
>> - cross-cc-cflags and prefix should be passed to config.mak also
>> (similar to the merge for container options). Not strictly needed as I'm
>> not sure anyone needs this "extra" flexibility for roms/vdso, but needed
>> if we want to make sure we don't break anything for anyone.
> 
> The rom builds are outside the main makefile anyway so they just include
> random bits as they want.
> 
>> - scripts/meson_buildoptions.sh update that was missing originally for
>> those meson options.
> 
> We need to do something about it because at various times
> "update-buildoptions" does and doesn't change things. Sometimes there
> are more options than I've been messing with. It is super confusing.
>

Agree, in my personal experience, I never could regenerate it through
Makefile dependencies, even if it's supposed to work.

What we could do is generate it directly from meson_options.txt, instead
of meson introspect. It should be quite easy as meson_options.txt can be
"included" as a python file, and we can provide option() function to
aggregate them.

This way, our only dependency becomes python, which is already a dep
anyway, and not our full venv + meson. We could directly generate this
in configure, without any need to keep the file checked in our tree.
This way, it would always be up to date by design.

I'll let Paolo judge if that's a good idea or if I missed something
important.

>>
>> For the rest, it looks good and ready!
>>
>> Regards,
>> Pierrick
> 

Regards,
Pierrick

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)
Posted by Pierrick Bouvier 1 week, 1 day ago
On 9/17/2026 9:55 AM, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 

In addition to current series, please find here additional fixes,
notably for John's issue on Fedora:
https://lore.kernel.org/qemu-devel/20260917221117.274619-2-pierrick.bouvier@oss.qualcomm.com/T/#

It can be directly applied on top of current series.

Regards,
Pierrick