1. Patchew
  2. Xen
  3. [PATCH v2 0/4] Few CI improvements
  4. View patch

[PATCH v2 3/4] automation: allow selecting individual jobs via CI variables

Marek Marczykowski-Górecki posted 4 patches 8 months, 3 weeks ago
There is a newer version of this series
[PATCH v2 3/4] automation: allow selecting individual jobs via CI variables
Posted by Marek Marczykowski-Górecki 8 months, 3 weeks ago 
Debugging sometimes involves running specific jobs on different
versions. It's useful to easily avoid running all of the not interesting
ones (for given case) to save both time and CI resources. Doing so used
to require changing the yaml files, usually in several places.
Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex.
Note that one needs to satisfy job dependencies on their own (for
example if a test job needs a build job, that specific build job
needs to be included too).

The variable can be specified via Gitlab web UI when scheduling a
pipeline, but it can be also set when doing git push directly:

    git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/"

More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html

The variable needs to include regex for selecting jobs, including
enclosing slashes.
A coma/space separated list of jobs to select would be friendlier UX,
but unfortunately that is not supported:
https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed
workaround doesn't work for job-level CI_JOB_NAME).
On the other hand, the regex is more flexible (one can select for
example all arm32 jobs).

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
This probably wants documenting beyond this commit message. I don't
think we have any CI-related docs anywhere, do we? Some new file in
docs/misc?

And also, it's possible to extend web ui for starting pipelines to
include pre-defined variables. I use it in qubes here if you want to
see:
https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new
Does it make sense to include SELECTED_JOBS_ONLY this way too?
Personally, I'll probably use it via cmdline push only anyway, but I
don't know what workflows other people have.
---
 automation/gitlab-ci/build.yaml |  6 ++++++
 automation/gitlab-ci/test.yaml  | 14 ++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 35e224366f62..f12de00a164a 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -12,6 +12,12 @@
       - '*/*.log'
     when: always
   needs: []
+  rules:
+  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
+    when: always
+  - if: $SELECTED_JOBS_ONLY
+    when: never
+  - when: on_success
 
 .gcc-tmpl:
   variables: &gcc
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index c21a37933881..93632f1f9204 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -1,6 +1,11 @@
 .test-jobs-common:
   stage: test
   image: ${XEN_REGISTRY}/${CONTAINER}
+  rules:
+  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
+  - if: $SELECTED_JOBS_ONLY
+    when: never
+  - when: on_success
 
 .arm64-test-needs: &arm64-test-needs
   - alpine-3.18-arm64-rootfs-export
@@ -99,6 +104,9 @@
       - '*.dtb'
     when: always
   rules:
+    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
+    - if: $SELECTED_JOBS_ONLY
+      when: never
     - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
   tags:
     - xilinx
@@ -117,6 +125,9 @@
       - '*.log'
     when: always
   rules:
+    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
+    - if: $SELECTED_JOBS_ONLY
+      when: never
     - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
   tags:
     - xilinx
@@ -137,6 +148,9 @@
       - '*.log'
     when: always
   rules:
+    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
+    - if: $SELECTED_JOBS_ONLY
+      when: never
     - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
   tags:
     - qubes-hw2
-- 
git-series 0.9.1
Re: [PATCH v2 3/4] automation: allow selecting individual jobs via CI variables
Posted by Stefano Stabellini 8 months, 2 weeks ago 
On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote:
> Debugging sometimes involves running specific jobs on different
> versions. It's useful to easily avoid running all of the not interesting
> ones (for given case) to save both time and CI resources. Doing so used
> to require changing the yaml files, usually in several places.
> Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex.
> Note that one needs to satisfy job dependencies on their own (for
> example if a test job needs a build job, that specific build job
> needs to be included too).
> 
> The variable can be specified via Gitlab web UI when scheduling a
> pipeline, but it can be also set when doing git push directly:
> 
>     git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/"
> 
> More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html
> 
> The variable needs to include regex for selecting jobs, including
> enclosing slashes.

Does it work with a single job like this?

git push -o ci.variable=SELECTED_JOBS_ONLY="job1"

If it does, is there any way we could use to automatically whitelist its
dependencies too? Because that would be so much easier to use...


> A coma/space separated list of jobs to select would be friendlier UX,
> but unfortunately that is not supported:
> https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed
> workaround doesn't work for job-level CI_JOB_NAME).
> On the other hand, the regex is more flexible (one can select for
> example all arm32 jobs).
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> This probably wants documenting beyond this commit message. I don't
> think we have any CI-related docs anywhere, do we? Some new file in
> docs/misc?

Please add documentation for it. Other than that, I think it could be
committed. I would prefer you also added documentation about alternative
methods such as removing all the jobs except the ones you care about.


> And also, it's possible to extend web ui for starting pipelines to
> include pre-defined variables. I use it in qubes here if you want to
> see:
> https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new
> Does it make sense to include SELECTED_JOBS_ONLY this way too?
> Personally, I'll probably use it via cmdline push only anyway, but I
> don't know what workflows other people have.
> ---
>  automation/gitlab-ci/build.yaml |  6 ++++++
>  automation/gitlab-ci/test.yaml  | 14 ++++++++++++++
>  2 files changed, 20 insertions(+)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 35e224366f62..f12de00a164a 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -12,6 +12,12 @@
>        - '*/*.log'
>      when: always
>    needs: []
> +  rules:
> +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> +    when: always
> +  - if: $SELECTED_JOBS_ONLY
> +    when: never
> +  - when: on_success
>  
>  .gcc-tmpl:
>    variables: &gcc
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index c21a37933881..93632f1f9204 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -1,6 +1,11 @@
>  .test-jobs-common:
>    stage: test
>    image: ${XEN_REGISTRY}/${CONTAINER}
> +  rules:
> +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> +  - if: $SELECTED_JOBS_ONLY
> +    when: never
> +  - when: on_success
>  
>  .arm64-test-needs: &arm64-test-needs
>    - alpine-3.18-arm64-rootfs-export
> @@ -99,6 +104,9 @@
>        - '*.dtb'
>      when: always
>    rules:
> +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> +    - if: $SELECTED_JOBS_ONLY
> +      when: never
>      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
>    tags:
>      - xilinx
> @@ -117,6 +125,9 @@
>        - '*.log'
>      when: always
>    rules:
> +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> +    - if: $SELECTED_JOBS_ONLY
> +      when: never
>      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
>    tags:
>      - xilinx
> @@ -137,6 +148,9 @@
>        - '*.log'
>      when: always
>    rules:
> +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> +    - if: $SELECTED_JOBS_ONLY
> +      when: never
>      - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
>    tags:
>      - qubes-hw2
> -- 
> git-series 0.9.1
>
Re: [PATCH v2 3/4] automation: allow selecting individual jobs via CI variables
Posted by Marek Marczykowski-Górecki 8 months, 2 weeks ago 
On Fri, Feb 14, 2025 at 04:29:17PM -0800, Stefano Stabellini wrote:
> On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote:
> > Debugging sometimes involves running specific jobs on different
> > versions. It's useful to easily avoid running all of the not interesting
> > ones (for given case) to save both time and CI resources. Doing so used
> > to require changing the yaml files, usually in several places.
> > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex.
> > Note that one needs to satisfy job dependencies on their own (for
> > example if a test job needs a build job, that specific build job
> > needs to be included too).
> > 
> > The variable can be specified via Gitlab web UI when scheduling a
> > pipeline, but it can be also set when doing git push directly:
> > 
> >     git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/"
> > 
> > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html
> > 
> > The variable needs to include regex for selecting jobs, including
> > enclosing slashes.
> 
> Does it work with a single job like this?
> 
> git push -o ci.variable=SELECTED_JOBS_ONLY="job1"

No, it works with:

    git push -o ci.variable=SELECTED_JOBS_ONLY="/job1/"

or rather:

    git push -o ci.variable=SELECTED_JOBS_ONLY="/^job1$/"

> If it does, is there any way we could use to automatically whitelist its
> dependencies too? Because that would be so much easier to use...

I guess it should be possible to add some extra condition for
dependencies, like extending rules for alpine-3.18-gcc-debug
specifically with

  - if: $SELECTED_JOBS_ONLY && "adl-smoke-x86-64-gcc-debug" =~ $SELECTED_JOBS_ONLY
    when: always

(and repeated for other tests depending on this build job)

But that means dependencies need to be kept in sync manually, in two
places. The absolute lack of any variables processing (even a simple
string concatenation...) at this stage of gitlab yaml processing makes
it challenging to propose any even semi-reasonable solution...

On the other hand, if you care about specific test, you can easily get
its dependencies by either looking at test.yaml, or clicking "show
depdendencies" in gitlab ui and hovering over the job you want.

> > A coma/space separated list of jobs to select would be friendlier UX,
> > but unfortunately that is not supported:
> > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed
> > workaround doesn't work for job-level CI_JOB_NAME).
> > On the other hand, the regex is more flexible (one can select for
> > example all arm32 jobs).
> > 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > This probably wants documenting beyond this commit message. I don't
> > think we have any CI-related docs anywhere, do we? Some new file in
> > docs/misc?
> 
> Please add documentation for it. Other than that, I think it could be
> committed. I would prefer you also added documentation about alternative
> methods such as removing all the jobs except the ones you care about.

Do we really want to recommend removing unnecessary jobs, given a better
option exists now?
I know https://xkcd.com/1171/, but still...

> > And also, it's possible to extend web ui for starting pipelines to
> > include pre-defined variables. I use it in qubes here if you want to
> > see:
> > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new
> > Does it make sense to include SELECTED_JOBS_ONLY this way too?
> > Personally, I'll probably use it via cmdline push only anyway, but I
> > don't know what workflows other people have.
> > ---
> >  automation/gitlab-ci/build.yaml |  6 ++++++
> >  automation/gitlab-ci/test.yaml  | 14 ++++++++++++++
> >  2 files changed, 20 insertions(+)
> > 
> > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> > index 35e224366f62..f12de00a164a 100644
> > --- a/automation/gitlab-ci/build.yaml
> > +++ b/automation/gitlab-ci/build.yaml
> > @@ -12,6 +12,12 @@
> >        - '*/*.log'
> >      when: always
> >    needs: []
> > +  rules:
> > +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    when: always
> > +  - if: $SELECTED_JOBS_ONLY
> > +    when: never
> > +  - when: on_success
> >  
> >  .gcc-tmpl:
> >    variables: &gcc
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index c21a37933881..93632f1f9204 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -1,6 +1,11 @@
> >  .test-jobs-common:
> >    stage: test
> >    image: ${XEN_REGISTRY}/${CONTAINER}
> > +  rules:
> > +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +  - if: $SELECTED_JOBS_ONLY
> > +    when: never
> > +  - when: on_success
> >  
> >  .arm64-test-needs: &arm64-test-needs
> >    - alpine-3.18-arm64-rootfs-export
> > @@ -99,6 +104,9 @@
> >        - '*.dtb'
> >      when: always
> >    rules:
> > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    - if: $SELECTED_JOBS_ONLY
> > +      when: never
> >      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> >    tags:
> >      - xilinx
> > @@ -117,6 +125,9 @@
> >        - '*.log'
> >      when: always
> >    rules:
> > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    - if: $SELECTED_JOBS_ONLY
> > +      when: never
> >      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> >    tags:
> >      - xilinx
> > @@ -137,6 +148,9 @@
> >        - '*.log'
> >      when: always
> >    rules:
> > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    - if: $SELECTED_JOBS_ONLY
> > +      when: never
> >      - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> >    tags:
> >      - qubes-hw2
> > -- 
> > git-series 0.9.1
> > 


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
Re: [PATCH v2 3/4] automation: allow selecting individual jobs via CI variables
Posted by Stefano Stabellini 8 months, 2 weeks ago 
On Sat, 15 Feb 2025, Marek Marczykowski-Górecki wrote:
> On Fri, Feb 14, 2025 at 04:29:17PM -0800, Stefano Stabellini wrote:
> > On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote:
> > > Debugging sometimes involves running specific jobs on different
> > > versions. It's useful to easily avoid running all of the not interesting
> > > ones (for given case) to save both time and CI resources. Doing so used
> > > to require changing the yaml files, usually in several places.
> > > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex.
> > > Note that one needs to satisfy job dependencies on their own (for
> > > example if a test job needs a build job, that specific build job
> > > needs to be included too).
> > > 
> > > The variable can be specified via Gitlab web UI when scheduling a
> > > pipeline, but it can be also set when doing git push directly:
> > > 
> > >     git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/"
> > > 
> > > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html
> > > 
> > > The variable needs to include regex for selecting jobs, including
> > > enclosing slashes.
> > 
> > Does it work with a single job like this?
> > 
> > git push -o ci.variable=SELECTED_JOBS_ONLY="job1"
> 
> No, it works with:
> 
>     git push -o ci.variable=SELECTED_JOBS_ONLY="/job1/"
> 
> or rather:
> 
>     git push -o ci.variable=SELECTED_JOBS_ONLY="/^job1$/"
> 
> > If it does, is there any way we could use to automatically whitelist its
> > dependencies too? Because that would be so much easier to use...
> 
> I guess it should be possible to add some extra condition for
> dependencies, like extending rules for alpine-3.18-gcc-debug
> specifically with
> 
>   - if: $SELECTED_JOBS_ONLY && "adl-smoke-x86-64-gcc-debug" =~ $SELECTED_JOBS_ONLY
>     when: always
> 
> (and repeated for other tests depending on this build job)
> 
> But that means dependencies need to be kept in sync manually, in two
> places. The absolute lack of any variables processing (even a simple
> string concatenation...) at this stage of gitlab yaml processing makes
> it challenging to propose any even semi-reasonable solution...

Yeah I agree with you


> On the other hand, if you care about specific test, you can easily get
> its dependencies by either looking at test.yaml, or clicking "show
> depdendencies" in gitlab ui and hovering over the job you want.

I am less sure about this. It becomes a matter of personal taste but if
I need to figure out the dependencies, I prefer to do it with vim, and
if I do that, I can very quickly edit the yaml file deleting all the
unneeded jobs. I can see someone else might instead prefer to write down
the list and setup the regex for SELECTED_JOBS_ONLY.

In an ideal world, we wouldn't need to do that because there would be a
way to do this automatically. Unfortunately it is not the case.

I am OK with this patch, I am sure someone will find it useful. It is
just that its usefulness is limited due to the restrictions we have with
Gitlab.



> > > A coma/space separated list of jobs to select would be friendlier UX,
> > > but unfortunately that is not supported:
> > > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed
> > > workaround doesn't work for job-level CI_JOB_NAME).
> > > On the other hand, the regex is more flexible (one can select for
> > > example all arm32 jobs).
> > > 
> > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > > ---
> > > This probably wants documenting beyond this commit message. I don't
> > > think we have any CI-related docs anywhere, do we? Some new file in
> > > docs/misc?
> > 
> > Please add documentation for it. Other than that, I think it could be
> > committed. I would prefer you also added documentation about alternative
> > methods such as removing all the jobs except the ones you care about.
> 
> Do we really want to recommend removing unnecessary jobs, given a better
> option exists now?
> I know https://xkcd.com/1171/, but still...

Not recommend! Only document as a way of explanation.


> > > And also, it's possible to extend web ui for starting pipelines to
> > > include pre-defined variables. I use it in qubes here if you want to
> > > see:
> > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new
> > > Does it make sense to include SELECTED_JOBS_ONLY this way too?
> > > Personally, I'll probably use it via cmdline push only anyway, but I
> > > don't know what workflows other people have.
> > > ---
> > >  automation/gitlab-ci/build.yaml |  6 ++++++
> > >  automation/gitlab-ci/test.yaml  | 14 ++++++++++++++
> > >  2 files changed, 20 insertions(+)
> > > 
> > > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> > > index 35e224366f62..f12de00a164a 100644
> > > --- a/automation/gitlab-ci/build.yaml
> > > +++ b/automation/gitlab-ci/build.yaml
> > > @@ -12,6 +12,12 @@
> > >        - '*/*.log'
> > >      when: always
> > >    needs: []
> > > +  rules:
> > > +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > > +    when: always
> > > +  - if: $SELECTED_JOBS_ONLY
> > > +    when: never
> > > +  - when: on_success
> > >  
> > >  .gcc-tmpl:
> > >    variables: &gcc
> > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > > index c21a37933881..93632f1f9204 100644
> > > --- a/automation/gitlab-ci/test.yaml
> > > +++ b/automation/gitlab-ci/test.yaml
> > > @@ -1,6 +1,11 @@
> > >  .test-jobs-common:
> > >    stage: test
> > >    image: ${XEN_REGISTRY}/${CONTAINER}
> > > +  rules:
> > > +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > > +  - if: $SELECTED_JOBS_ONLY
> > > +    when: never
> > > +  - when: on_success
> > >  
> > >  .arm64-test-needs: &arm64-test-needs
> > >    - alpine-3.18-arm64-rootfs-export
> > > @@ -99,6 +104,9 @@
> > >        - '*.dtb'
> > >      when: always
> > >    rules:
> > > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > > +    - if: $SELECTED_JOBS_ONLY
> > > +      when: never
> > >      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> > >    tags:
> > >      - xilinx
> > > @@ -117,6 +125,9 @@
> > >        - '*.log'
> > >      when: always
> > >    rules:
> > > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > > +    - if: $SELECTED_JOBS_ONLY
> > > +      when: never
> > >      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> > >    tags:
> > >      - xilinx
> > > @@ -137,6 +148,9 @@
> > >        - '*.log'
> > >      when: always
> > >    rules:
> > > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > > +    - if: $SELECTED_JOBS_ONLY
> > > +      when: never
> > >      - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> > >    tags:
> > >      - qubes-hw2
> > > -- 
> > > git-series 0.9.1
> > >

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.