1. Patchew
  2. QEMU
  3. [RFC PATCH 0/2] ci: allow running Coverity Scan uploads via GitLab
  4. View patch

[RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity

Paolo Bonzini posted 2 patches 8 months, 3 weeks ago
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>, Peter Maydell <peter.maydell@linaro.org>
There is a newer version of this series
[RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity
Posted by Paolo Bonzini 8 months, 3 weeks ago 
Add a job that can be run, either manually or on a schedule, to upload
a build to Coverity Scan.  The job uses the run-coverity-scan script
in multiple phases of check, download tools and upload, in order to
avoid both wasting time (skip everything if you are above the upload
quota) and avoid filling the log with the progress of downloading
the tools.

The job is intended to run on a scheduled pipeline run, and scheduled
runs will not get any other job.  It requires two variables to be in
GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL.  Those are already set up
in qemu-project's configuration as protected and masked variables.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .gitlab-ci.d/base.yml      |  4 ++++
 .gitlab-ci.d/buildtest.yml | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index ef173a34e6..2dd8a9b57c 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -41,6 +41,10 @@ variables:
     - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
       when: never
 
+    # Scheduled runs on mainline don't get pipelines except for the special Coverity job
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
+      when: never
+
     # Cirrus jobs can't run unless the creds / target repo are set
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
       when: never
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index a1c030337b..378dee055b 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -729,3 +729,38 @@ pages:
       - public
   variables:
     QEMU_JOB_PUBLISH: 1
+
+coverity:
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
+  stage: build
+  allow_failure: true
+  timeout: 3h
+  needs:
+    - job: amd64-fedora-container
+      optional: true
+  before_script:
+    - dnf install -y curl wget
+  script:
+    # would be nice to cancel the job if over quota (https://gitlab.com/gitlab-org/gitlab/-/issues/256089)
+    - 'scripts/coverity-scan/run-coverity-scan --check-upload-only || (exitcode=$?; if test $exitcode = 1; then
+        exit 0;
+      else
+        exit $exitcode;
+      fi)'
+    - 'scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || cat update-tools.log'
+    - 'scripts/coverity-scan/run-coverity-scan --no-update-tools'
+  rules:
+    - if: '$COVERITY_TOKEN == null'
+      when: never
+    - if: '$COVERITY_EMAIL == null'
+      when: never
+    # Never included on upstream pipelines, except for schedules
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH'
+      when: on_success
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
+      when: never
+    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
+      when: never
+    # Always manual on forks even if $QEMU_CI == "2"
+    - when: manual
-- 
2.39.1
Re: [RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity
Posted by Daniel P. Berrangé 8 months, 3 weeks ago 
On Mon, Mar 04, 2024 at 05:06:31PM -0500, Paolo Bonzini wrote:
> Add a job that can be run, either manually or on a schedule, to upload
> a build to Coverity Scan.  The job uses the run-coverity-scan script
> in multiple phases of check, download tools and upload, in order to
> avoid both wasting time (skip everything if you are above the upload
> quota) and avoid filling the log with the progress of downloading
> the tools.
> 
> The job is intended to run on a scheduled pipeline run, and scheduled
> runs will not get any other job.  It requires two variables to be in
> GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL.  Those are already set up
> in qemu-project's configuration as protected and masked variables.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  .gitlab-ci.d/base.yml      |  4 ++++
>  .gitlab-ci.d/buildtest.yml | 36 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 40 insertions(+)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index ef173a34e6..2dd8a9b57c 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -41,6 +41,10 @@ variables:
>      - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
>        when: never
>  
> +    # Scheduled runs on mainline don't get pipelines except for the special Coverity job
> +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
> +      when: never
> +
>      # Cirrus jobs can't run unless the creds / target repo are set
>      - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
>        when: never
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index a1c030337b..378dee055b 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -729,3 +729,38 @@ pages:
>        - public
>    variables:
>      QEMU_JOB_PUBLISH: 1
> +
> +coverity:
> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
> +  stage: build
> +  allow_failure: true
> +  timeout: 3h
> +  needs:
> +    - job: amd64-fedora-container
> +      optional: true
> +  before_script:
> +    - dnf install -y curl wget
> +  script:
> +    # would be nice to cancel the job if over quota (https://gitlab.com/gitlab-org/gitlab/-/issues/256089)
> +    - 'scripts/coverity-scan/run-coverity-scan --check-upload-only || (exitcode=$?; if test $exitcode = 1; then
> +        exit 0;
> +      else
> +        exit $exitcode;
> +      fi)'
> +    - 'scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || cat update-tools.log'

Slightly shorter as:

  .....   2>&1 | tee update-tools.log

> +    - 'scripts/coverity-scan/run-coverity-scan --no-update-tools'
> +  rules:
> +    - if: '$COVERITY_TOKEN == null'
> +      when: never
> +    - if: '$COVERITY_EMAIL == null'
> +      when: never
> +    # Never included on upstream pipelines, except for schedules
> +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH'
> +      when: on_success
> +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
> +      when: never
> +    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
> +      when: never
> +    # Always manual on forks even if $QEMU_CI == "2"
> +    - when: manual
> -- 

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity
Posted by Paolo Bonzini 8 months, 3 weeks ago 
On Tue, Mar 5, 2024 at 9:52 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > +    - 'scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || cat update-tools.log'
>
> Slightly shorter as:
>
>   .....   2>&1 | tee update-tools.log

Note that stdout/stderr are only dumped if the download fails, so tee
cannot be used.

> > +    - 'scripts/coverity-scan/run-coverity-scan --no-update-tools'
> > +  rules:
> > +    - if: '$COVERITY_TOKEN == null'
> > +      when: never
> > +    - if: '$COVERITY_EMAIL == null'
> > +      when: never
> > +    # Never included on upstream pipelines, except for schedules
> > +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH'
> > +      when: on_success
> > +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
> > +      when: never
> > +    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
> > +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
> > +      when: never
> > +    # Always manual on forks even if $QEMU_CI == "2"
> > +    - when: manual
> > --
>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Thanks! There will be a few small changes after I actually tested this
on a schedule, but nothing major.

Paolo

Patchew 2.1-devel-b67e4c2

© 2016 - 2024 Red Hat, Inc.