gitlab: misc artifact / log handling fixes

[PATCH 4/4] gitlab: prevent duplicated meson log artifacts in test jobs

Posted by Daniel P. Berrangé 5 months ago

The build jobs will populate build/meson-logs/ with various files
that are added as artifacts.

The test jobs preserve the state of the build jobs, so we must
delete any pre-existing logs to prevent confusion from duplicate
artifacts.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest-template.yml | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 308490a35a..d866cb12bb 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -83,6 +83,10 @@
 
 .native_test_job_template:
   extends: .common_test_job_template
+  before_script:
+    # Prevent logs from the build job that run earlier
+    # from being duplicated in the test job artifacts
+    - rm -f build/meson-logs/*
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
     when: always
@@ -111,6 +115,9 @@
   before_script:
     - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
     - export QEMU_TEST_CACHE_DIR=${CI_PROJECT_DIR}/functional-cache
+    # Prevent logs from the build job that run earlier
+    # from being duplicated in the test job artifacts
+    - rm -f build/meson-logs/*
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/*-cache
-- 
2.50.1

Re: [PATCH 4/4] gitlab: prevent duplicated meson log artifacts in test jobs

Posted by Alex Bennée 5 months ago

Daniel P. Berrangé <berrange@redhat.com> writes:

> The build jobs will populate build/meson-logs/ with various files
> that are added as artifacts.
>
> The test jobs preserve the state of the build jobs, so we must
> delete any pre-existing logs to prevent confusion from duplicate
> artifacts.

Will this cause us to rebuild needlessly?

>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index 308490a35a..d866cb12bb 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -83,6 +83,10 @@
>  
>  .native_test_job_template:
>    extends: .common_test_job_template
> +  before_script:
> +    # Prevent logs from the build job that run earlier
> +    # from being duplicated in the test job artifacts
> +    - rm -f build/meson-logs/*
>    artifacts:
>      name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
>      when: always
> @@ -111,6 +115,9 @@
>    before_script:
>      - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
>      - export QEMU_TEST_CACHE_DIR=${CI_PROJECT_DIR}/functional-cache
> +    # Prevent logs from the build job that run earlier
> +    # from being duplicated in the test job artifacts
> +    - rm -f build/meson-logs/*
>    after_script:
>      - cd build
>      - du -chs ${CI_PROJECT_DIR}/*-cache

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 4/4] gitlab: prevent duplicated meson log artifacts in test jobs

Posted by Daniel P. Berrangé 5 months ago

On Mon, Sep 08, 2025 at 10:21:38PM +0100, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > The build jobs will populate build/meson-logs/ with various files
> > that are added as artifacts.
> >
> > The test jobs preserve the state of the build jobs, so we must
> > delete any pre-existing logs to prevent confusion from duplicate
> > artifacts.
> 
> Will this cause us to rebuild needlessly?

These files aren't involved in any rebuild / stale checks &
the test pipeline I ran show no difference in what was done.

> 
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/buildtest-template.yml | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> > index 308490a35a..d866cb12bb 100644
> > --- a/.gitlab-ci.d/buildtest-template.yml
> > +++ b/.gitlab-ci.d/buildtest-template.yml
> > @@ -83,6 +83,10 @@
> >  
> >  .native_test_job_template:
> >    extends: .common_test_job_template
> > +  before_script:
> > +    # Prevent logs from the build job that run earlier
> > +    # from being duplicated in the test job artifacts
> > +    - rm -f build/meson-logs/*
> >    artifacts:
> >      name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> >      when: always
> > @@ -111,6 +115,9 @@
> >    before_script:
> >      - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
> >      - export QEMU_TEST_CACHE_DIR=${CI_PROJECT_DIR}/functional-cache
> > +    # Prevent logs from the build job that run earlier
> > +    # from being duplicated in the test job artifacts
> > +    - rm -f build/meson-logs/*
> >    after_script:
> >      - cd build
> >      - du -chs ${CI_PROJECT_DIR}/*-cache
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 4/4] gitlab: prevent duplicated meson log artifacts in test jobs

Posted by Alex Bennée 5 months ago

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Sep 08, 2025 at 10:21:38PM +0100, Alex Bennée wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > The build jobs will populate build/meson-logs/ with various files
>> > that are added as artifacts.
>> >
>> > The test jobs preserve the state of the build jobs, so we must
>> > delete any pre-existing logs to prevent confusion from duplicate
>> > artifacts.
>> 
>> Will this cause us to rebuild needlessly?
>
> These files aren't involved in any rebuild / stale checks &
> the test pipeline I ran show no difference in what was done.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro