gitlab-ci: Maintain a public runner, allow manual pipeline

[RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

Posted by Philippe Mathieu-Daudé 5 years, 1 month ago

It might be overkill (and frustrating) to rerun all the jobs when
iterating over a single code unit. As some users are interested
in only rerunning a specific set of jobs, allow them to use a
"manual" profile, when jobs are created in the pipeline but not
started. It is up to the user to manually start the jobs of
interest.

To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
in the project UI, following [1].

Once a repository is using the manual workflow, the user can still
automatically start the jobs by explicitly set the
QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.

Example while pushing a branch/tag from command line (see [2]):

  $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
      myrepo mybranch

[1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
[2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml |  4 ++++
 .gitlab-ci.yml               | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 864cad7cec5..56b053cbb52 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -18,6 +19,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $ACCEL_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -30,6 +32,7 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 .cross_user_build_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -40,6 +43,7 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 .cross_sanity_check_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 3m
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 98bff03b47b..26b85f08f35 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,7 +13,21 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
+.allow_skipping_job_template: &allow_skipping_job_definition
+  rules:
+    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
+    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
+    # variable.
+    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null
+      when: manual
+      allow_failure: true
+    - when: always
+
+.allow_skipping_job:
+  <<: *allow_skipping_job_definition
+
 .native_build_job_template: &native_build_job_definition
+  <<: *allow_skipping_job_definition
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -35,6 +49,7 @@ include:
       fi
 
 .native_test_job_template: &native_test_job_definition
+  <<: *allow_skipping_job_definition
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -43,6 +58,7 @@ include:
     - make $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
+  <<: *allow_skipping_job_definition
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -536,6 +552,7 @@ check-dco:
     GIT_DEPTH: 1000
 
 build-libvhost-user:
+  <<: *allow_skipping_job_definition
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   before_script:
@@ -547,6 +564,7 @@ build-libvhost-user:
     - ninja
 
 pages:
+  <<: *allow_skipping_job_definition
   image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
   stage: test
   needs:
-- 
2.26.2

Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

Posted by Thomas Huth 5 years, 1 month ago

On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
> It might be overkill (and frustrating) to rerun all the jobs when
> iterating over a single code unit. As some users are interested
> in only rerunning a specific set of jobs, allow them to use a
> "manual" profile, when jobs are created in the pipeline but not
> started. It is up to the user to manually start the jobs of
> interest.
> 
> To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
> in the project UI, following [1].
> 
> Once a repository is using the manual workflow, the user can still
> automatically start the jobs by explicitly set the
> QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.
> 
> Example while pushing a branch/tag from command line (see [2]):
> 
>   $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
>       myrepo mybranch
> 
> [1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
> [2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .gitlab-ci.d/crossbuilds.yml |  4 ++++
>  .gitlab-ci.yml               | 18 ++++++++++++++++++
>  2 files changed, 22 insertions(+)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 864cad7cec5..56b053cbb52 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -1,4 +1,5 @@
>  .cross_system_build_job:
> +  extends: .allow_skipping_job
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    timeout: 80m
> @@ -18,6 +19,7 @@
>  # KVM), and set extra options (such disabling other accelerators) via the
>  # $ACCEL_CONFIGURE_OPTS variable.
>  .cross_accel_build_job:
> +  extends: .allow_skipping_job
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    timeout: 30m
> @@ -30,6 +32,7 @@
>      - make -j$(expr $(nproc) + 1) all check-build
>  
>  .cross_user_build_job:
> +  extends: .allow_skipping_job
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    script:
> @@ -40,6 +43,7 @@
>      - make -j$(expr $(nproc) + 1) all check-build
>  
>  .cross_sanity_check_job:
> +  extends: .allow_skipping_job

So "extends" is possible ....

>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    timeout: 3m
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 98bff03b47b..26b85f08f35 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -13,7 +13,21 @@ include:
>    - local: '/.gitlab-ci.d/containers.yml'
>    - local: '/.gitlab-ci.d/crossbuilds.yml'
>  
> +.allow_skipping_job_template: &allow_skipping_job_definition
> +  rules:
> +    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
> +    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
> +    # variable.
> +    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null
> +      when: manual
> +      allow_failure: true
> +    - when: always
> +
> +.allow_skipping_job:
> +  <<: *allow_skipping_job_definition

... before the anchestor has been defined? That's nice!

I'm basically fine with this patch, but can you please also add some
sentence how to use this to docs/devel/testing.rst ?

 Thanks,
  Thomas

Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

Posted by Philippe Mathieu-Daudé 5 years, 1 month ago

On 12/14/20 2:32 PM, Thomas Huth wrote:
> On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
>> It might be overkill (and frustrating) to rerun all the jobs when
>> iterating over a single code unit. As some users are interested
>> in only rerunning a specific set of jobs, allow them to use a
>> "manual" profile, when jobs are created in the pipeline but not
>> started. It is up to the user to manually start the jobs of
>> interest.
>>
>> To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
>> in the project UI, following [1].
>>
>> Once a repository is using the manual workflow, the user can still
>> automatically start the jobs by explicitly set the
>> QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.
>>
>> Example while pushing a branch/tag from command line (see [2]):
>>
>>   $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
>>       myrepo mybranch
>>
>> [1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
>> [2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .gitlab-ci.d/crossbuilds.yml |  4 ++++
>>  .gitlab-ci.yml               | 18 ++++++++++++++++++
>>  2 files changed, 22 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
>> index 864cad7cec5..56b053cbb52 100644
>> --- a/.gitlab-ci.d/crossbuilds.yml
>> +++ b/.gitlab-ci.d/crossbuilds.yml
>> @@ -1,4 +1,5 @@
>>  .cross_system_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 80m
>> @@ -18,6 +19,7 @@
>>  # KVM), and set extra options (such disabling other accelerators) via the
>>  # $ACCEL_CONFIGURE_OPTS variable.
>>  .cross_accel_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 30m
>> @@ -30,6 +32,7 @@
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>>  .cross_user_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    script:
>> @@ -40,6 +43,7 @@
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>>  .cross_sanity_check_job:
>> +  extends: .allow_skipping_job
> 
> So "extends" is possible ....
> 
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 3m
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 98bff03b47b..26b85f08f35 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -13,7 +13,21 @@ include:
>>    - local: '/.gitlab-ci.d/containers.yml'
>>    - local: '/.gitlab-ci.d/crossbuilds.yml'
>>  
>> +.allow_skipping_job_template: &allow_skipping_job_definition
>> +  rules:
>> +    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
>> +    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
>> +    # variable.
>> +    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null
>> +      when: manual
>> +      allow_failure: true
>> +    - when: always
>> +
>> +.allow_skipping_job:
>> +  <<: *allow_skipping_job_definition
> 
> ... before the anchestor has been defined? That's nice!

'extends' is indeed more powerful and less hackish.

> I'm basically fine with this patch, but can you please also add some
> sentence how to use this to docs/devel/testing.rst ?

Thanks, I later discovered this isn't complete, because if a children
job is in automatic state and a parent (dependency) is in manual, there
is some "dead lock": the pipeline waits until manual job is started.

To be complete, I have to ensure if a job is in manual state, all its
dependencies also are.

Then we are back to 'extends' VS 'anchors', as this doesn't work with
anchors, I have to convert all jobs to 'extends'. Good news is I already
did it, I simply need to rebase this series:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html

Thanks for reviewing,

Phil.

[RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner
[RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template