We have a number of container images in tests/docker/dockerfiles
that are intended to provide well defined environments for doing
test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the
CI, so that the built containers are available for later build
jobs. The containers are setup to use the GitLab container
registry as the cache, so we only pay the penalty of the full
build when the dockerfiles change. The main qemu-project/qemu
repo is used as a second cache, so that users forking QEMU will
see a fast turnaround time on their CI jobs.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
new file mode 100644
index 0000000000..ea1edbb196
--- /dev/null
+++ b/.gitlab-ci.d/containers.yml
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7
+
+amd64-centos8-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos8
+
+amd64-debian10-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian10
+
+amd64-debian11-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian11
+
+amd64-debian9-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian9
+
+amd64-debian9-mxe-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian9-mxe
+
+alpha-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64-cross
+
+amd64-debian-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64
+
+arm64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian11-container']
+  variables:
+    NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win32-cross
+
+win64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-cris-cross
+
+amd64-fedora-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora
+
+i386-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu2004
+
+amd64-ubuntu-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu
+
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9fdc752ea6..72d688875f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,10 +1,13 @@
 stages:
   - containers
+  - containers-layer2
+  - containers-layer3
   - build
 
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
-- 
2.24.1

On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.

OMG you did it! Lovely... 😍

Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't
populated on all the runners yet? Or we have to use the same
runner again to hit its cache?

> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
[...]

On Mon, Jun 22, 2020 at 05:38:16PM +0200, Philippe Mathieu-Daudé wrote:
> On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> 
> OMG you did it! Lovely... 😍
> 
> Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
> why gitlab isn't caching the docker images? the cache isn't
> populated on all the runners yet? Or we have to use the same
> runner again to hit its cache?

It is caching it but it isn't obvious what to look for.

The "docker build" command is always still run, but you'll
notice each of the actual commands in the dockerfile are
followed by a message "  ---> Using cache", instead of the
normal command output.

Compare the "amd64-debian9-container" job as an example.

Here is the output seen in the original job which actually did a real
build:

   https://gitlab.com/berrange/qemu/-/jobs/605783351

And here is the output from the pipeline you point to above:

   https://gitlab.com/berrange/qemu/-/jobs/606175855

The cached case is still taking 2 mins 30 seconds, but the original
full build took 4 mins 50 seconds.

The difference will be more noticable if we expand the list of packages
install in the images to cover more of QEMU's possible dependancies.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 6/22/20 5:46 PM, Daniel P. Berrangé wrote:
> On Mon, Jun 22, 2020 at 05:38:16PM +0200, Philippe Mathieu-Daudé wrote:
>> On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>
>> OMG you did it! Lovely... 😍
>>
>> Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
>> why gitlab isn't caching the docker images? the cache isn't
>> populated on all the runners yet? Or we have to use the same
>> runner again to hit its cache?
> 
> It is caching it but it isn't obvious what to look for.
> 
> The "docker build" command is always still run, but you'll
> notice each of the actual commands in the dockerfile are
> followed by a message "  ---> Using cache", instead of the
> normal command output.

Indeed!

> 
> Compare the "amd64-debian9-container" job as an example.
> 
> Here is the output seen in the original job which actually did a real
> build:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/605783351
> 
> And here is the output from the pipeline you point to above:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/606175855
> 
> The cached case is still taking 2 mins 30 seconds, but the original
> full build took 4 mins 50 seconds.
> 
> The difference will be more noticable if we expand the list of packages
> install in the images to cover more of QEMU's possible dependancies.

The difference is already very noticeable on my host:

$ time docker pull registry.gitlab.com/berrange/qemu/ci-centos8:latest
latest: Pulling from berrange/qemu/ci-centos8
8a29a15cefae: Pull complete
4a748adf2909: Pull complete
3c034506a458: Pull complete
cd48de325fa1: Pull complete
Digest:
sha256:a6580433f5456d7c49e2b9b796c717c79a5fa15f4eecf71bdbcffd9df6b8217a
Status: Downloaded newer image for
registry.gitlab.com/berrange/qemu/ci-centos8:latest
registry.gitlab.com/berrange/qemu/ci-centos8:latest
real    0m53.566s
user    0m0.115s
sys     0m0.081s

> 
> Regards,
> Daniel
> 

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>   .gitlab-ci.yml              |   3 +
>   2 files changed, 251 insertions(+)
>   create mode 100644 .gitlab-ci.d/containers.yml
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"

I can see this in the output:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

I have to admit that I have only little knowledge about docker ... but 
could there be an issue here? Should this be done in a different way?

> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
> +
> +amd64-centos8-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos8
> +
> +amd64-debian10-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian10
> +
> +amd64-debian11-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian11
> +
> +amd64-debian9-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian9
> +
> +amd64-debian9-mxe-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian9-mxe
> +
> +alpha-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-alpha-cross
> +
> +amd64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64-cross
> +
> +amd64-debian-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64
> +
> +arm64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-arm64-cross
> +
> +arm64-test-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian11-container']
> +  variables:
> +    NAME: debian-arm64-test-cross
> +
> +armel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armel-cross
> +
> +armhf-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armhf-cross
> +
> +hppa-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-hppa-cross
> +
> +m68k-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-m68k-cross
> +
> +mips64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64-cross
> +
> +mips64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64el-cross
> +
> +mips-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips-cross
> +
> +mipsel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mipsel-cross
> +
> +powerpc-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-powerpc-cross
> +
> +ppc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64-cross
> +
> +ppc64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64el-cross
> +
> +riscv64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-riscv64-cross
> +
> +s390x-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-s390x-cross
> +
> +sh4-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sh4-cross
> +
> +sparc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sparc64-cross
> +
> +tricore-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian-tricore-cross
> +
> +win32-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win32-cross
> +
> +win64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win64-cross
> +
> +xtensa-debian-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian-xtensa-cross
> +
> +cris-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-cris-cross
> +
> +amd64-fedora-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora
> +
> +i386-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-i386-cross
> +
> +amd64-ubuntu1804-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu1804
> +
> +amd64-ubuntu2004-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu2004
> +
> +amd64-ubuntu-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu
> +
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9fdc752ea6..72d688875f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,10 +1,13 @@
>   stages:
>     - containers
> +  - containers-layer2
> +  - containers-layer3
>     - build
>   
>   include:
>     - local: '/.gitlab-ci.d/edk2.yml'
>     - local: '/.gitlab-ci.d/opensbi.yml'
> +  - local: '/.gitlab-ci.d/containers.yml'
>   
>   .update_apt_template: &before_script_apt
>    before_script:
> 

Acked-by: Thomas Huth <thuth@redhat.com>

On Thu, Jun 25, 2020 at 11:35:54AM +0200, Thomas Huth wrote:
> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> >   .gitlab-ci.yml              |   3 +
> >   2 files changed, 251 insertions(+)
> >   create mode 100644 .gitlab-ci.d/containers.yml
> > 
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > new file mode 100644
> > index 0000000000..ea1edbb196
> > --- /dev/null
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -0,0 +1,248 @@
> > +
> > +
> > +.container_job_template: &container_job_definition
> > +  image: docker:stable
> > +  stage: containers
> > +  services:
> > +    - docker:dind
> > +  before_script:
> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > +    - docker info
> > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> 
> I can see this in the output:
> 
> WARNING! Using --password via the CLI is insecure. Use --password-stdin.
> 
> I have to admit that I have only little knowledge about docker ... but could
> there be an issue here? Should this be done in a different way?

In general the warning is correct, because other users on the same
host can see the process CLI args, and thus the password is susceptible
to snooping.

In this case, however, it is a non-issue. This is running inside a docker
container already which has a PID namespace. Thus the only things that
can see our password on the CLI are things inside our own container
which already all have the env variable, and processes running in host
OS context which are only things GitLab admins control. So there's no
data leakage to anyone who doesn't already have access to the password

This particular docker login command is the documented solution:

  https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 06/25/20 11:50, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 11:35:54AM +0200, Thomas Huth wrote:
>> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>>   .gitlab-ci.yml              |   3 +
>>>   2 files changed, 251 insertions(+)
>>>   create mode 100644 .gitlab-ci.d/containers.yml
>>>
>>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>>> new file mode 100644
>>> index 0000000000..ea1edbb196
>>> --- /dev/null
>>> +++ b/.gitlab-ci.d/containers.yml
>>> @@ -0,0 +1,248 @@
>>> +
>>> +
>>> +.container_job_template: &container_job_definition
>>> +  image: docker:stable
>>> +  stage: containers
>>> +  services:
>>> +    - docker:dind
>>> +  before_script:
>>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>>> +    - docker info
>>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>>
>> I can see this in the output:
>>
>> WARNING! Using --password via the CLI is insecure. Use --password-stdin.
>>
>> I have to admit that I have only little knowledge about docker ... but could
>> there be an issue here? Should this be done in a different way?
> 
> In general the warning is correct, because other users on the same
> host can see the process CLI args, and thus the password is susceptible
> to snooping.
> 
> In this case, however, it is a non-issue. This is running inside a docker
> container already which has a PID namespace. Thus the only things that
> can see our password on the CLI are things inside our own container
> which already all have the env variable, and processes running in host
> OS context which are only things GitLab admins control. So there's no
> data leakage to anyone who doesn't already have access to the password
> 
> This particular docker login command is the documented solution:
> 
>   https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

(

Purely theoretically, we could use a "here string":

docker [...] --password-stdin <<< "$CI_REGISTRY_PASSWORD"

The password is then not exposed on any process's command line; it's a
(bash) shell redirection. (It's not in POSIX.)

)

Thanks
Laszlo

On Mon, Jun 22, 2020 at 04:33:17PM +0100, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml

Thinking about this at 3am with insomnia, I started considering possible
ways this could result in CI failures. I came up with

 - Distro repos are unavialable due to transient network/mirror problems
 
 - Distro pushes bad packages or changes something that invalidates QEMU
   assumptions

The first one can hit at any time, but I think that we're reasonably well
insulated from it due to our usage of caching. Only a small number of CI
jobs are going to actually trigger a full rebuild of an image, most will
just hit the cache.

The second one is probably not likely, *provided* we only use stable branches
of distros. It looks like we're ok in that regard as we're not using Debian
Sid, or Fedora rawhide for any images currently.

If we did want to reduce the risk though, we could mark some of the
container jobs as non-fatal. We could mark the subset of images that are
not actually required by later build jobs that we currently have.

> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
> +
> +amd64-centos8-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos8
> +
> +amd64-debian10-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian10
> +
> +amd64-debian11-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian11
> +
> +amd64-debian9-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian9
> +
> +amd64-debian9-mxe-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian9-mxe
> +
> +alpha-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-alpha-cross
> +
> +amd64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64-cross
> +
> +amd64-debian-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64
> +
> +arm64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-arm64-cross
> +
> +arm64-test-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian11-container']
> +  variables:
> +    NAME: debian-arm64-test-cross
> +
> +armel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armel-cross
> +
> +armhf-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armhf-cross
> +
> +hppa-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-hppa-cross
> +
> +m68k-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-m68k-cross
> +
> +mips64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64-cross
> +
> +mips64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64el-cross
> +
> +mips-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips-cross
> +
> +mipsel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mipsel-cross
> +
> +powerpc-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-powerpc-cross
> +
> +ppc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64-cross
> +
> +ppc64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64el-cross
> +
> +riscv64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-riscv64-cross
> +
> +s390x-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-s390x-cross
> +
> +sh4-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sh4-cross
> +
> +sparc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sparc64-cross
> +
> +tricore-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian-tricore-cross
> +
> +win32-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win32-cross
> +
> +win64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win64-cross
> +
> +xtensa-debian-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian-xtensa-cross
> +
> +cris-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-cris-cross
> +
> +amd64-fedora-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora
> +
> +i386-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-i386-cross
> +
> +amd64-ubuntu1804-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu1804
> +
> +amd64-ubuntu2004-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu2004
> +
> +amd64-ubuntu-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu
> +
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9fdc752ea6..72d688875f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,10 +1,13 @@
>  stages:
>    - containers
> +  - containers-layer2
> +  - containers-layer3
>    - build
>  
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
>    - local: '/.gitlab-ci.d/opensbi.yml'
> +  - local: '/.gitlab-ci.d/containers.yml'
>  
>  .update_apt_template: &before_script_apt
>   before_script:
> -- 
> 2.24.1
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jun 22, 2020 at 04:33:17PM +0100, Daniel P. Berrangé wrote:
>> We have a number of container images in tests/docker/dockerfiles
>> that are intended to provide well defined environments for doing
>> test builds. We want our CI system to use these containers too.
>> 
>> This introduces builds of all of them as the first stage in the
>> CI, so that the built containers are available for later build
>> jobs. The containers are setup to use the GitLab container
>> registry as the cache, so we only pay the penalty of the full
>> build when the dockerfiles change. The main qemu-project/qemu
>> repo is used as a second cache, so that users forking QEMU will
>> see a fast turnaround time on their CI jobs.
>> 
>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>  .gitlab-ci.yml              |   3 +
>>  2 files changed, 251 insertions(+)
>>  create mode 100644 .gitlab-ci.d/containers.yml
>
> Thinking about this at 3am with insomnia, I started considering possible
> ways this could result in CI failures. I came up with
>
>  - Distro repos are unavialable due to transient network/mirror problems
>  
>  - Distro pushes bad packages or changes something that invalidates QEMU
>    assumptions
>
> The first one can hit at any time, but I think that we're reasonably well
> insulated from it due to our usage of caching. Only a small number of CI
> jobs are going to actually trigger a full rebuild of an image, most will
> just hit the cache.
>
> The second one is probably not likely, *provided* we only use stable branches
> of distros. It looks like we're ok in that regard as we're not using Debian
> Sid, or Fedora rawhide for any images currently.

We use debian11 (which I think is debian-testing) for one of the aarch64
compilers because we need fairlyu bleeding edge gcc's but we won't go
back to sid anytime soon.

>
> If we did want to reduce the risk though, we could mark some of the
> container jobs as non-fatal. We could mark the subset of images that are
> not actually required by later build jobs that we currently have.
>
>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>> new file mode 100644
>> index 0000000000..ea1edbb196
>> --- /dev/null
>> +++ b/.gitlab-ci.d/containers.yml
>> @@ -0,0 +1,248 @@
>> +
>> +
>> +.container_job_template: &container_job_definition
>> +  image: docker:stable
>> +  stage: containers
>> +  services:
>> +    - docker:dind
>> +  before_script:
>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>> +    - docker info
>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>> +  script:
>> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
>> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
>> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
>> +    - docker push "$TAG"
>> +  after_script:
>> +    - docker logout
>> +
>> +amd64-centos7-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: centos7
>> +
>> +amd64-centos8-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: centos8
>> +
>> +amd64-debian10-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian10
>> +
>> +amd64-debian11-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian11
>> +
>> +amd64-debian9-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian9
>> +
>> +amd64-debian9-mxe-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian9-container']
>> +  variables:
>> +    NAME: debian9-mxe
>> +
>> +alpha-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-alpha-cross
>> +
>> +amd64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-amd64-cross
>> +
>> +amd64-debian-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-amd64
>> +
>> +arm64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-arm64-cross
>> +
>> +arm64-test-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian11-container']
>> +  variables:
>> +    NAME: debian-arm64-test-cross
>> +
>> +armel-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-armel-cross
>> +
>> +armhf-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-armhf-cross
>> +
>> +hppa-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-hppa-cross
>> +
>> +m68k-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-m68k-cross
>> +
>> +mips64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips64-cross
>> +
>> +mips64el-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips64el-cross
>> +
>> +mips-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips-cross
>> +
>> +mipsel-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mipsel-cross
>> +
>> +powerpc-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-powerpc-cross
>> +
>> +ppc64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-ppc64-cross
>> +
>> +ppc64el-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-ppc64el-cross
>> +
>> +riscv64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-riscv64-cross
>> +
>> +s390x-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-s390x-cross
>> +
>> +sh4-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-sh4-cross
>> +
>> +sparc64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-sparc64-cross
>> +
>> +tricore-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian9-container']
>> +  variables:
>> +    NAME: debian-tricore-cross
>> +
>> +win32-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer3
>> +  needs: ['amd64-debian9-mxe-container']
>> +  variables:
>> +    NAME: debian-win32-cross
>> +
>> +win64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer3
>> +  needs: ['amd64-debian9-mxe-container']
>> +  variables:
>> +    NAME: debian-win64-cross
>> +
>> +xtensa-debian-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian-xtensa-cross
>> +
>> +cris-fedora-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora-cris-cross
>> +
>> +amd64-fedora-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora
>> +
>> +i386-fedora-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora-i386-cross
>> +
>> +amd64-ubuntu1804-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu1804
>> +
>> +amd64-ubuntu2004-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu2004
>> +
>> +amd64-ubuntu-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu
>> +
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 9fdc752ea6..72d688875f 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,10 +1,13 @@
>>  stages:
>>    - containers
>> +  - containers-layer2
>> +  - containers-layer3
>>    - build
>>  
>>  include:
>>    - local: '/.gitlab-ci.d/edk2.yml'
>>    - local: '/.gitlab-ci.d/opensbi.yml'
>> +  - local: '/.gitlab-ci.d/containers.yml'
>>  
>>  .update_apt_template: &before_script_apt
>>   before_script:
>> -- 
>> 2.24.1
>> 
>
> Regards,
> Daniel


-- 
Alex Bennée

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>   .gitlab-ci.yml              |   3 +
>   2 files changed, 251 insertions(+)
>   create mode 100644 .gitlab-ci.d/containers.yml
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout

.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if 
something really has been changed. Could you use something similar here? 
E.g.:

rules:
  - changes:
    - .gitlab-ci.d/containers.yml
    - tests/docker/*
    - tests/docker/dockerfiles/*

?

  Thomas

On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> >   .gitlab-ci.yml              |   3 +
> >   2 files changed, 251 insertions(+)
> >   create mode 100644 .gitlab-ci.d/containers.yml
> > 
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > new file mode 100644
> > index 0000000000..ea1edbb196
> > --- /dev/null
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -0,0 +1,248 @@
> > +
> > +
> > +.container_job_template: &container_job_definition
> > +  image: docker:stable
> > +  stage: containers
> > +  services:
> > +    - docker:dind
> > +  before_script:
> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > +    - docker info
> > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> > +  script:
> > +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> > +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> > +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> > +    - docker push "$TAG"
> > +  after_script:
> > +    - docker logout
> 
> .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
> something really has been changed. Could you use something similar here?
> E.g.:
> 
> rules:
>  - changes:
>    - .gitlab-ci.d/containers.yml
>    - tests/docker/*
>    - tests/docker/dockerfiles/*
> 
> ?

If the OS distro base image changes, we'll never pick it up with that
kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
could configure a nightly/weekly/whatever job to force rebuild on a
periodic basis to pick up base image changes.  The downside of this
is that any users who fork qemu won't have that periodic job and so
will be testing their work against potentially outdated content.

Having said all that, I'm not 100% convinced I'm actually picking
up changed base images right now anyway, given our use of caching.

It is possible that I would need todo an explict "docker pull" of
the base image to force it to trigger a refresh othrewise I have
a feeling we're always cached.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 25/06/2020 12.24, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
>> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>    .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>>    .gitlab-ci.yml              |   3 +
>>>    2 files changed, 251 insertions(+)
>>>    create mode 100644 .gitlab-ci.d/containers.yml
>>>
>>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>>> new file mode 100644
>>> index 0000000000..ea1edbb196
>>> --- /dev/null
>>> +++ b/.gitlab-ci.d/containers.yml
>>> @@ -0,0 +1,248 @@
>>> +
>>> +
>>> +.container_job_template: &container_job_definition
>>> +  image: docker:stable
>>> +  stage: containers
>>> +  services:
>>> +    - docker:dind
>>> +  before_script:
>>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>>> +    - docker info
>>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>>> +  script:
>>> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
>>> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
>>> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
>>> +    - docker push "$TAG"
>>> +  after_script:
>>> +    - docker logout
>>
>> .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
>> something really has been changed. Could you use something similar here?
>> E.g.:
>>
>> rules:
>>   - changes:
>>     - .gitlab-ci.d/containers.yml
>>     - tests/docker/*
>>     - tests/docker/dockerfiles/*
>>
>> ?
> 
> If the OS distro base image changes, we'll never pick it up with that
> kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
> could configure a nightly/weekly/whatever job to force rebuild on a
> periodic basis to pick up base image changes.  The downside of this
> is that any users who fork qemu won't have that periodic job and so
> will be testing their work against potentially outdated content.
> 
> Having said all that, I'm not 100% convinced I'm actually picking
> up changed base images right now anyway, given our use of caching.
> 
> It is possible that I would need todo an explict "docker pull" of
> the base image to force it to trigger a refresh othrewise I have
> a feeling we're always cached.

But currently, each of the container stages currently takes > 2 minutes, 
even with the cached containers. I had a quick look, and it takes 7 
minutes 'till the "build" stage begins. So all the advantages of not 
having to do "yum/apt-get install" in the build containers anymore seem 
to be crushed by the time that the three container stages take now?

  Thomas

On Thu, Jun 25, 2020 at 03:25:19PM +0200, Thomas Huth wrote:
> On 25/06/2020 12.24, Daniel P. Berrangé wrote:
> > On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
> > > On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > > > We have a number of container images in tests/docker/dockerfiles
> > > > that are intended to provide well defined environments for doing
> > > > test builds. We want our CI system to use these containers too.
> > > > 
> > > > This introduces builds of all of them as the first stage in the
> > > > CI, so that the built containers are available for later build
> > > > jobs. The containers are setup to use the GitLab container
> > > > registry as the cache, so we only pay the penalty of the full
> > > > build when the dockerfiles change. The main qemu-project/qemu
> > > > repo is used as a second cache, so that users forking QEMU will
> > > > see a fast turnaround time on their CI jobs.
> > > > 
> > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > > ---
> > > >    .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> > > >    .gitlab-ci.yml              |   3 +
> > > >    2 files changed, 251 insertions(+)
> > > >    create mode 100644 .gitlab-ci.d/containers.yml
> > > > 
> > > > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > > > new file mode 100644
> > > > index 0000000000..ea1edbb196
> > > > --- /dev/null
> > > > +++ b/.gitlab-ci.d/containers.yml
> > > > @@ -0,0 +1,248 @@
> > > > +
> > > > +
> > > > +.container_job_template: &container_job_definition
> > > > +  image: docker:stable
> > > > +  stage: containers
> > > > +  services:
> > > > +    - docker:dind
> > > > +  before_script:
> > > > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > > > +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > > > +    - docker info
> > > > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> > > > +  script:
> > > > +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> > > > +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> > > > +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> > > > +    - docker push "$TAG"
> > > > +  after_script:
> > > > +    - docker logout
> > > 
> > > .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
> > > something really has been changed. Could you use something similar here?
> > > E.g.:
> > > 
> > > rules:
> > >   - changes:
> > >     - .gitlab-ci.d/containers.yml
> > >     - tests/docker/*
> > >     - tests/docker/dockerfiles/*
> > > 
> > > ?
> > 
> > If the OS distro base image changes, we'll never pick it up with that
> > kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
> > could configure a nightly/weekly/whatever job to force rebuild on a
> > periodic basis to pick up base image changes.  The downside of this
> > is that any users who fork qemu won't have that periodic job and so
> > will be testing their work against potentially outdated content.
> > 
> > Having said all that, I'm not 100% convinced I'm actually picking
> > up changed base images right now anyway, given our use of caching.
> > 
> > It is possible that I would need todo an explict "docker pull" of
> > the base image to force it to trigger a refresh othrewise I have
> > a feeling we're always cached.
> 
> But currently, each of the container stages currently takes > 2 minutes,
> even with the cached containers. I had a quick look, and it takes 7 minutes
> 'till the "build" stage begins. So all the advantages of not having to do
> "yum/apt-get install" in the build containers anymore seem to be crushed by
> the time that the three container stages take now?

That's still not an apples/apples comparison though. The containers have
many mre packages installed and so test more features. Add all those extra
packages into the original apt-get/dnf commands and then compare times.

I do agree though that I don't much like the 3 containers stages. I set
them up so they would parallelize. ie stage 2 can start without waiting
for entire of stage 1 to finish - only the parent container needs to
finish. The builds stage still waits for all the containers to complete.

We could do similar optimize for the actual build jobs, so they don't have
to wait for all containers to finish - they only need wait for their own
required container to finish.

Ultimately it might be a better tradeoff to not have inheritance between
the containers, and instead just duplicate the common packages in each.
This leads to bigger container images, but simpler builds. Libvirt went
this way for its cross compiler images, just duplicating the shared parts
in each.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
>
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml
>
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export
> COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"

It would be nice if we could keep the same form as they have in the
local registry which would make it easier to integrate with the rest of
the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Otherwise it looks good:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>


-- 
Alex Bennée

On Mon, Jun 22, 2020 at 07:26:39PM +0100, Alex Bennée wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> >
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> >  .gitlab-ci.yml              |   3 +
> >  2 files changed, 251 insertions(+)
> >  create mode 100644 .gitlab-ci.d/containers.yml
> >
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > new file mode 100644
> > index 0000000000..ea1edbb196
> > --- /dev/null
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -0,0 +1,248 @@
> > +
> > +
> > +.container_job_template: &container_job_definition
> > +  image: docker:stable
> > +  stage: containers
> > +  services:
> > +    - docker:dind
> > +  before_script:
> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > +    - export
> > COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> 
> It would be nice if we could keep the same form as they have in the
> local registry which would make it easier to integrate with the rest of
> the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Every time I re-discover it, I find the QEMU container naming really
surprising. It is not following the normal best practice for naming
containers. Expected container naming convention is that the image
name reflects the general content set, and the tag reflects a version
number. QEMU has shifted it along, so container name is just a fixed
string, and the tag reflects the contenet set, and there is no version.

QEMU's naming will cause problems with caching in GitLab. As GitLab
expects the normal container naming scheme, it has a job which expires
old versions of an image once there are more than 10 tags. So we have
to use the normal naming scheme. Ideally we would change rest of QEMU
to use the normal scheme too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jun 22, 2020 at 07:26:39PM +0100, Alex Bennée wrote:
>> 
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > We have a number of container images in tests/docker/dockerfiles
>> > that are intended to provide well defined environments for doing
>> > test builds. We want our CI system to use these containers too.
>> >
>> > This introduces builds of all of them as the first stage in the
>> > CI, so that the built containers are available for later build
>> > jobs. The containers are setup to use the GitLab container
>> > registry as the cache, so we only pay the penalty of the full
>> > build when the dockerfiles change. The main qemu-project/qemu
>> > repo is used as a second cache, so that users forking QEMU will
>> > see a fast turnaround time on their CI jobs.
>> >
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> >  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>> >  .gitlab-ci.yml              |   3 +
>> >  2 files changed, 251 insertions(+)
>> >  create mode 100644 .gitlab-ci.d/containers.yml
>> >
>> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>> > new file mode 100644
>> > index 0000000000..ea1edbb196
>> > --- /dev/null
>> > +++ b/.gitlab-ci.d/containers.yml
>> > @@ -0,0 +1,248 @@
>> > +
>> > +
>> > +.container_job_template: &container_job_definition
>> > +  image: docker:stable
>> > +  stage: containers
>> > +  services:
>> > +    - docker:dind
>> > +  before_script:
>> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>> > +    - export
>> > COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>> 
>> It would be nice if we could keep the same form as they have in the
>> local registry which would make it easier to integrate with the rest of
>> the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"
>
> Every time I re-discover it, I find the QEMU container naming really
> surprising. It is not following the normal best practice for naming
> containers. Expected container naming convention is that the image
> name reflects the general content set, and the tag reflects a version
> number. QEMU has shifted it along, so container name is just a fixed
> string, and the tag reflects the contenet set, and there is no version.
>
> QEMU's naming will cause problems with caching in GitLab. As GitLab
> expects the normal container naming scheme, it has a job which expires
> old versions of an image once there are more than 10 tags. So we have
> to use the normal naming scheme. Ideally we would change rest of QEMU
> to use the normal scheme too.

Fair enough.. I'll look into it.

>
> Regards,
> Daniel


-- 
Alex Bennée