Currently the tests/docker/dockerfiles/*Dockerfile recipes are all hand
written by contributors. There is a common design pattern, but the set
of packages listed for installation leaves alot to be desired

 - There is no consistency at all across distros
 - Many potential build deps are not listed in the containers
 - Some packages are not used by QEMU at all
 - Adding new distros is an error prone task

The same applies to package lists for VMs, Cirrus CI / Travis CI, and
probably more.

This problem is not unique to QEMU, libvirt faced the exact same issues
and developed a program called "lcitool" which is part of the libvirt-ci
git repository to reduce the burden in this area.

Despite its name, this repository is not tied to libvirt, and so as well
as the 40+ libvirt git repos, it is also used by the libosinfo and
virt-viewer projects for their CI needs.

lcitool is capable of automating the installation and updating of VM
images, creation of dockerfiles and creation of standalone package
lists.

In this series I'm taking the easy step which is the generation of
dockerfiles, since that is also where the most immediate value lies
for QEMU.

The key concept in lcitool that brings a huge win in maintainability
is that there is a single file which defines a mapping between a
build pre-requisite and the native package on each targetted distro.

   https://gitlab.com/libvirt/libvirt-ci/-/blob/master/guests/lcitool/lcitool/ansible/vars/mappings.yml

A project merely has to have its list of pre-requisites enumerated

   https://gitlab.com/libvirt/libvirt-ci/-/blob/master/guests/lcitool/lcitool/ansible/vars/projects/qemu.yml

The combination of these two files is enough to generate accurate
package lists for any supported distro. Currently supported distros
are Debian (10, sid), Ubuntu (18.04, 20.04), CentOS (8, 8 Stream),
Fedora (32, 33, rawhide), OpenSUSE Leap (15.2) macOS (HomeBrew),
FreeBSD w/ Ports (11, 12, current).

At the end of this series, I have dockerfiles auto-generated for QEMU
covering Ubuntu 18.04 & 20.04, CentOS 8, Fedora 32 and OpenSUSE 15.2.

lcitool is also capable of generating dockerfiles for cross-compiled
non-x86 architectures for Debian, and for mingw32/64 for Fedora. This
is driven from the very same mapping.yml file listed above, which has
attributes to indicate whether a given dependancy should be pulled from
the native or cross build target. Again this means that we have strong
guarantee of consistent deps being used between cross containers.

I have not converted cross containers in this series though, because
the way we generated cross dockerfiles is different from how QEMU does
it. lcitool will always generate fully self-contained dockerfiles, but
QEMU currently uses layered dockerfiles for cross-builds, so all cross
builds share a common intermediate container.

I could enhance lcitool to support layered containers for cross-builds,
but before doing that I wondered how strongly people are attached to
them ? If self-contained dockerfiles are acceptable I can do that more
easily.

There is also scope for auto-generating the package lists for tests/vm
and .cirrus.yml files, but I've not attempted that here. The same
general idea appies - we just call lcitool to spit out a list of native
packages for each case.

If converting tests/vm, we would need to add more distros to lcitool
mappings.yml to convert openbsd, netbsd, haiku since libvirt does not
target those distros itself.

I have provided a tests/docker/dockerfiles/refresh script that needs
to be invoked periodically to re-generate them. eg when adding a
new distro, or when the package lists change. I have added libvirt-ci.git
as a sub-module to provide more seemless integration, but this is
possibly overkill. In libvirt*.git repos we don't bother with git
submodules for libvirt-ci.git since whomever runs it to refresh
containers just has a local checkout regardless.

Changed in v3:

 - Drop changes for CentOS 7
 - Catch up with newly added build deps
 - Add git submodule and make target for refresh

Changed in v2:

 - Remove more travis stuff from tests/docker/Makefile.include
 - Convert opensuse image to be auto-generated
 - Add SDL2_image package
 - QEMU package manifest is now officially merged in libvirt-ci.git

Daniel P. Berrangé (22):
  hw/usb/ccid: remove references to NSS
  tests/docker: don't use BUILDKIT in GitLab either
  tests/docker: use project specific container registries
  tests/docker: use explicit docker.io registry
  tests/docker: remove FEATURES env var from templates
  tests/docker: fix sorting in package lists
  tests/docker: fix mistakes in centos package lists
  tests/docker: fix mistakes in fedora package list
  tests/docker: fix mistakes in ubuntu package lists
  tests/docker: remove mingw packages from Fedora
  tests/docker: expand centos8 package list
  tests/docker: expand fedora package list
  tests/docker: expand ubuntu1804 package list
  tests/docker: expand ubuntu2004 package list
  tests/docker: expand opensuse-leap package list
  tests/docker: add script for automating container refresh
  tests/docker: auto-generate centos8 with lcitool
  tests/docker: auto-generate fedora with lcitool
  tests/docker: auto-generate ubuntu1804 with lcitool
  tests/docker: auto-generate ubuntu2004 with lcitool
  tests/docker: auto-generate opensuse-leap with lcitool
  tests/docker: remove ubuntu container

 .gitlab-ci.d/containers.yml                   |   5 -
 .gitmodules                                   |   3 +
 .travis.yml                                   |  12 +-
 docs/ccid.txt                                 |  15 +-
 docs/devel/testing.rst                        |  15 +-
 scripts/coverity-scan/coverity-scan.docker    |   1 -
 tests/docker/Makefile.include                 |  10 +
 tests/docker/common.rc                        |  19 +-
 tests/docker/docker.py                        |   4 +-
 tests/docker/dockerfiles-refresh.py           |  70 +++++
 tests/docker/dockerfiles/centos8.docker       | 165 +++++++++---
 .../dockerfiles/debian-xtensa-cross.docker    |   2 +-
 tests/docker/dockerfiles/debian10.docker      |   4 +-
 tests/docker/dockerfiles/debian11.docker      |   2 +-
 .../dockerfiles/fedora-cris-cross.docker      |   2 +-
 .../dockerfiles/fedora-i386-cross.docker      |   2 +-
 .../dockerfiles/fedora-win32-cross.docker     |   3 +-
 .../dockerfiles/fedora-win64-cross.docker     |   3 +-
 tests/docker/dockerfiles/fedora.docker        | 253 ++++++++++--------
 tests/docker/dockerfiles/opensuse-leap.docker | 180 +++++++++----
 tests/docker/dockerfiles/ubuntu.docker        |  71 -----
 tests/docker/dockerfiles/ubuntu1804.docker    | 195 ++++++++++----
 tests/docker/dockerfiles/ubuntu2004.docker    | 211 ++++++++++-----
 tests/docker/libvirt-ci                       |   1 +
 tests/docker/run                              |   3 -
 tests/docker/test-clang                       |   2 +-
 tests/docker/test-debug                       |   2 +-
 tests/docker/test-mingw                       |   3 +-
 tests/docker/test-misc                        |   2 +-
 tests/docker/test-tsan                        |   2 +-
 30 files changed, 810 insertions(+), 452 deletions(-)
 create mode 100755 tests/docker/dockerfiles-refresh.py
 delete mode 100644 tests/docker/dockerfiles/ubuntu.docker
 create mode 160000 tests/docker/libvirt-ci

-- 
2.31.1

Patchew URL: https://patchew.org/QEMU/20210513095519.1213675-1-berrange@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20210513095519.1213675-1-berrange@redhat.com
Subject: [PATCH v3 00/22] tests/docker: start using libvirt-ci's "lcitool" for dockerfiles

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20210508014802.892561-1-richard.henderson@linaro.org -> patchew/20210508014802.892561-1-richard.henderson@linaro.org
 * [new tag]         patchew/20210513095519.1213675-1-berrange@redhat.com -> patchew/20210513095519.1213675-1-berrange@redhat.com
Switched to a new branch 'test'
73a0045 tests/docker: remove ubuntu container
d6b42f9 tests/docker: auto-generate opensuse-leap with lcitool
8a2b437 tests/docker: auto-generate ubuntu2004 with lcitool
dd19e03 tests/docker: auto-generate ubuntu1804 with lcitool
931d434 tests/docker: auto-generate fedora with lcitool
8402c0d tests/docker: auto-generate centos8 with lcitool
f581c7c tests/docker: add script for automating container refresh
95ad785 tests/docker: expand opensuse-leap package list
790f77d tests/docker: expand ubuntu2004 package list
0c357e1 tests/docker: expand ubuntu1804 package list
a29a532 tests/docker: expand fedora package list
f1df3d0 tests/docker: expand centos8 package list
a7c131e tests/docker: remove mingw packages from Fedora
37ce608 tests/docker: fix mistakes in ubuntu package lists
977b03d tests/docker: fix mistakes in fedora package list
b3580ed tests/docker: fix mistakes in centos package lists
5757744 tests/docker: fix sorting in package lists
826c380 tests/docker: remove FEATURES env var from templates
fad3849 tests/docker: use explicit docker.io registry
5098f17 tests/docker: use project specific container registries
2dc7c73 tests/docker: don't use BUILDKIT in GitLab either
f10a2d6 hw/usb/ccid: remove references to NSS

=== OUTPUT BEGIN ===
1/22 Checking commit f10a2d6c78d1 (hw/usb/ccid: remove references to NSS)
2/22 Checking commit 2dc7c735b228 (tests/docker: don't use BUILDKIT in GitLab either)
3/22 Checking commit 5098f173e564 (tests/docker: use project specific container registries)
4/22 Checking commit fad3849eb138 (tests/docker: use explicit docker.io registry)
5/22 Checking commit 826c380cbbc9 (tests/docker: remove FEATURES env var from templates)
6/22 Checking commit 57577442850f (tests/docker: fix sorting in package lists)
7/22 Checking commit b3580edfbd81 (tests/docker: fix mistakes in centos package lists)
8/22 Checking commit 977b03d43f95 (tests/docker: fix mistakes in fedora package list)
9/22 Checking commit 37ce60848186 (tests/docker: fix mistakes in ubuntu package lists)
10/22 Checking commit a7c131edc9b6 (tests/docker: remove mingw packages from Fedora)
11/22 Checking commit f1df3d0229f6 (tests/docker: expand centos8 package list)
12/22 Checking commit a29a53287642 (tests/docker: expand fedora package list)
13/22 Checking commit 0c357e188a91 (tests/docker: expand ubuntu1804 package list)
14/22 Checking commit 790f77db86dc (tests/docker: expand ubuntu2004 package list)
15/22 Checking commit 95ad7854e308 (tests/docker: expand opensuse-leap package list)
16/22 Checking commit f581c7c94392 (tests/docker: add script for automating container refresh)
Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#106: 
new file mode 100755

total: 0 errors, 1 warnings, 109 lines checked

Patch 16/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
17/22 Checking commit 8402c0d0f253 (tests/docker: auto-generate centos8 with lcitool)
18/22 Checking commit 931d434d8aef (tests/docker: auto-generate fedora with lcitool)
19/22 Checking commit dd19e0388006 (tests/docker: auto-generate ubuntu1804 with lcitool)
20/22 Checking commit 8a2b437a4498 (tests/docker: auto-generate ubuntu2004 with lcitool)
ERROR: line over 90 characters
#28: FILE: tests/docker/dockerfiles-refresh.py:65:
+               "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n"]

total: 1 errors, 0 warnings, 266 lines checked

Patch 20/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

21/22 Checking commit d6b42f932077 (tests/docker: auto-generate opensuse-leap with lcitool)
22/22 Checking commit 73a00450ba05 (tests/docker: remove ubuntu container)
Use of uninitialized value $acpi_testexpected in string eq at ./scripts/checkpatch.pl line 1529.
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#35: 
deleted file mode 100644

total: 0 errors, 1 warnings, 11 lines checked

Patch 22/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20210513095519.1213675-1-berrange@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com