1. Patchew
  2. QEMU
  3. View series

[PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug

Stefan Hajnoczi posted 1 patch 1 week, 6 days ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20260804165414.480435-1-stefanha@redhat.com
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
.gitlab-ci.d/container-template.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
[PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug
Posted by Stefan Hajnoczi 1 week, 6 days ago 
QEMU's CI pipeline involves building container images that will be used
to run builds and tests. A recent Docker change triggered the following
error:

  $ docker push "$TAG"
  ...
  error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9

https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372

This happens because Docker now pushes out-of-order and the GitLab
Container Registry rejects due to an unknown reference:
https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5

It is unclear at this point whether GitLab will modify the behavior of
Container Registry or whether Docker will ship a fix.

The current workaround is to disable the provenance attestation that is
involved in this issue. QEMU's CI pipeline container images are used
internally for testing and are not widely distributed. Provenance
attestation can be disabled as there are no external consumers of these
images. Expect to revert this commit in the future when GitLab or Docker
have released their own fixes.

Cc: Alex Bennée <alex.bennee@linaro.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Thomas Huth <thuth@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
Note that this is an urgent patch so that the CI will pass again and I
can tag v11.1.0-rc3 later today. Currently it is failing due to this
issue!

 .gitlab-ci.d/container-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 8c7311cca57..88317d650e2 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -14,7 +14,7 @@
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
     - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
-      --build-arg BUILDKIT_INLINE_CACHE=1
+      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
       -f "tests/docker/dockerfiles/$NAME.docker" "."
     - docker push "$TAG"
   after_script:
-- 
2.55.0
Re: [PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug
Posted by Stefan Hajnoczi 1 week, 6 days ago 
On Tue, Aug 04, 2026 at 12:54:14PM -0400, Stefan Hajnoczi wrote:
> QEMU's CI pipeline involves building container images that will be used
> to run builds and tests. A recent Docker change triggered the following
> error:
> 
>   $ docker push "$TAG"
>   ...
>   error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
> 
> This happens because Docker now pushes out-of-order and the GitLab
> Container Registry rejects due to an unknown reference:
> https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5
> 
> It is unclear at this point whether GitLab will modify the behavior of
> Container Registry or whether Docker will ship a fix.
> 
> The current workaround is to disable the provenance attestation that is
> involved in this issue. QEMU's CI pipeline container images are used
> internally for testing and are not widely distributed. Provenance
> attestation can be disabled as there are no external consumers of these
> images. Expect to revert this commit in the future when GitLab or Docker
> have released their own fixes.
> 
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> Note that this is an urgent patch so that the CI will pass again and I
> can tag v11.1.0-rc3 later today. Currently it is failing due to this
> issue!
> 
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Thanks, applied to my staging tree:
https://gitlab.com/stefanha/qemu/commits/staging

Stefan
Re: [PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug
Posted by Daniel P. Berrangé 1 week, 6 days ago 
On Tue, Aug 04, 2026 at 12:54:14PM -0400, Stefan Hajnoczi wrote:
> QEMU's CI pipeline involves building container images that will be used
> to run builds and tests. A recent Docker change triggered the following
> error:
> 
>   $ docker push "$TAG"
>   ...
>   error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
> 
> This happens because Docker now pushes out-of-order and the GitLab
> Container Registry rejects due to an unknown reference:
> https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5
> 
> It is unclear at this point whether GitLab will modify the behavior of
> Container Registry or whether Docker will ship a fix.

The forum post links to an issue where this comment:

  https://github.com/moby/buildkit/issues/7007#issuecomment-5178990414

suggests we can just set

  BUILDX_NO_DEFAULT_OCI_ARTIFACT=true

...except I don't think they've updated the docker docker image
with this fix yet. Once they do, that will be better as we can
set it in the GitLab env such that it applies to all branches,
instead of havnig to cherry-pick 

> 
> The current workaround is to disable the provenance attestation that is
> involved in this issue. QEMU's CI pipeline container images are used
> internally for testing and are not widely distributed. Provenance
> attestation can be disabled as there are no external consumers of these
> images. Expect to revert this commit in the future when GitLab or Docker
> have released their own fixes.
> 
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> Note that this is an urgent patch so that the CI will pass again and I
> can tag v11.1.0-rc3 later today. Currently it is failing due to this
> issue!
> 
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 8c7311cca57..88317d650e2 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -14,7 +14,7 @@
>      - echo "TAG:$TAG"
>      - echo "COMMON_TAG:$COMMON_TAG"
>      - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> -      --build-arg BUILDKIT_INLINE_CACHE=1
> +      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
>        -f "tests/docker/dockerfiles/$NAME.docker" "."
>      - docker push "$TAG"
>    after_script:

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Patchew 2.1-devel-ea86937

© 2016 - 2026 Red Hat, Inc.