1. Patchew
  2. Xen
  3. [PATCH v2 00/12] Changes for several CI improvements
  4. View patch

[PATCH v2 07/12] Support building arbitrary Linux branch/tag/commit

Marek Marczykowski-Górecki posted 12 patches 1 week, 1 day ago
Only 10 patches received!
[PATCH v2 07/12] Support building arbitrary Linux branch/tag/commit
Posted by Marek Marczykowski-Górecki 1 week, 1 day ago 
If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
init" + "git fetch" instead of "git clone" to support any of
branch/tag/commit.

This also defines optional linux-git-* jobs which will build the thing
if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
pipeline.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
The script variable and job variable need to have different names, so a
pipeline variable won't override it for all jobs. While LINUX_VERSION /
LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
LINUX_GIT_URL. Any better ideas?
---
 .gitlab-ci.yml         | 22 ++++++++++++++++++++++
 scripts/build-linux.sh | 18 +++++++++++++-----
 2 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 184d0b3..8d1deee 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,9 @@
 variables:
   REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
+  LINUX_GIT_VERSION:
+    description: "branch/tag/commit for the linux-git jobs"
+  LINUX_GIT_URL:
+    description: "git url for the linux-git jobs"
 
 stages:
   - build
@@ -53,6 +57,15 @@ linux-6.6.86-arm64:
   variables:
     LINUX_VERSION: 6.6.86
 
+linux-git-arm64:
+  extends: .arm64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: $LINUX_GIT_VERSION
+    LINUX_URL: $LINUX_GIT_URL
+  rules:
+  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
+
 #
 # x86_64 artifacts
 #
@@ -91,6 +104,15 @@ linux-6.12.60-x86_64:
   variables:
     LINUX_VERSION: 6.12.60
 
+linux-git-x86_64:
+  extends: .x86_64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: $LINUX_GIT_VERSION
+    LINUX_URL: $LINUX_GIT_URL
+  rules:
+  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
+
 microcode-x86:
   extends: .x86_64-artifacts
   script: ./scripts/x86-microcode.sh
diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
index cf0e744..1fc96d1 100755
--- a/scripts/build-linux.sh
+++ b/scripts/build-linux.sh
@@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
 UNAME=$(uname -m)
 
 # Build Linux
-MAJOR=${LINUX_VERSION%%.*}
-curl -fsSLO \
-    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
-tar xf linux-"${LINUX_VERSION}".tar.xz
-cd linux-"${LINUX_VERSION}"
+if [[ -n "${LINUX_URL}" ]]; then
+    mkdir linux
+    cd linux
+    git init
+    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
+    git checkout FETCH_HEAD
+else
+    MAJOR=${LINUX_VERSION%%.*}
+    curl -fsSLO \
+        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
+    tar xf linux-"${LINUX_VERSION}".tar.xz
+    cd linux-"${LINUX_VERSION}"
+fi
 
 make defconfig
 ./scripts/config --enable BRIDGE
-- 
git-series 0.9.1
Re: [PATCH v2 07/12] Support building arbitrary Linux branch/tag/commit
Posted by Stefano Stabellini 2 days, 19 hours ago 
On Thu, 4 Dec 2025, Marek Marczykowski-Górecki wrote:
> If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
> init" + "git fetch" instead of "git clone" to support any of
> branch/tag/commit.
> 
> This also defines optional linux-git-* jobs which will build the thing
> if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
> pipeline.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> The script variable and job variable need to have different names, so a
> pipeline variable won't override it for all jobs. While LINUX_VERSION /
> LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
> LINUX_GIT_URL. Any better ideas?

The problem is not LINUX_GIT_URL and LINUX_GIT_VERSION, those are good
names. The issue is ...

> ---
>  .gitlab-ci.yml         | 22 ++++++++++++++++++++++
>  scripts/build-linux.sh | 18 +++++++++++++-----
>  2 files changed, 35 insertions(+), 5 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 184d0b3..8d1deee 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,5 +1,9 @@
>  variables:
>    REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> +  LINUX_GIT_VERSION:
> +    description: "branch/tag/commit for the linux-git jobs"
> +  LINUX_GIT_URL:
> +    description: "git url for the linux-git jobs"
>  
>  stages:
>    - build
> @@ -53,6 +57,15 @@ linux-6.6.86-arm64:
>    variables:
>      LINUX_VERSION: 6.6.86
>  
> +linux-git-arm64:
> +  extends: .arm64-artifacts
> +  script: ./scripts/build-linux.sh
> +  variables:
> +    LINUX_VERSION: $LINUX_GIT_VERSION
> +    LINUX_URL: $LINUX_GIT_URL
> +  rules:
> +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> +
>  #
>  # x86_64 artifacts
>  #
> @@ -91,6 +104,15 @@ linux-6.12.60-x86_64:
>    variables:
>      LINUX_VERSION: 6.12.60
>  
> +linux-git-x86_64:
> +  extends: .x86_64-artifacts
> +  script: ./scripts/build-linux.sh
> +  variables:
> +    LINUX_VERSION: $LINUX_GIT_VERSION
> +    LINUX_URL: $LINUX_GIT_URL
> +  rules:
> +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> +
>  microcode-x86:
>    extends: .x86_64-artifacts
>    script: ./scripts/x86-microcode.sh
> diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
> index cf0e744..1fc96d1 100755
> --- a/scripts/build-linux.sh
> +++ b/scripts/build-linux.sh
> @@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
>  UNAME=$(uname -m)
>  
>  # Build Linux
> -MAJOR=${LINUX_VERSION%%.*}
> -curl -fsSLO \
> -    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> -tar xf linux-"${LINUX_VERSION}".tar.xz
> -cd linux-"${LINUX_VERSION}"
> +if [[ -n "${LINUX_URL}" ]]; then
> +    mkdir linux
> +    cd linux
> +    git init
> +    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
> +    git checkout FETCH_HEAD
> +else
> +    MAJOR=${LINUX_VERSION%%.*}
> +    curl -fsSLO \
> +        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> +    tar xf linux-"${LINUX_VERSION}".tar.xz
> +    cd linux-"${LINUX_VERSION}"
> +fi

... the issue is detecting to fetch via git or via curl based on the
presence of a variable called "LINUX_URL". Technically 

https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz

is a a valid URL as well.

So I think you should keep LINUX_GIT_URL and LINUX_GIT_VERSION named as
they are, expose them to scripts/build-linux.sh, and detect the fetch
program based on the presence of LINUX_GIT_URL.

Ideally, we should not have LINUX_GIT_VERSION. Instead we should have a
a common LINUX_VERSION used in both git and curl cases.


>  make defconfig
>  ./scripts/config --enable BRIDGE
> -- 
> git-series 0.9.1
>
Re: [PATCH v2 07/12] Support building arbitrary Linux branch/tag/commit
Posted by Marek Marczykowski-Górecki 2 days, 17 hours ago 
On Tue, Dec 09, 2025 at 04:29:02PM -0800, Stefano Stabellini wrote:
> On Thu, 4 Dec 2025, Marek Marczykowski-Górecki wrote:
> > If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
> > init" + "git fetch" instead of "git clone" to support any of
> > branch/tag/commit.
> > 
> > This also defines optional linux-git-* jobs which will build the thing
> > if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
> > pipeline.
> > 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > The script variable and job variable need to have different names, so a
> > pipeline variable won't override it for all jobs. While LINUX_VERSION /
> > LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
> > LINUX_GIT_URL. Any better ideas?
> 
> The problem is not LINUX_GIT_URL and LINUX_GIT_VERSION, those are good
> names. The issue is ...
> 
> > ---
> >  .gitlab-ci.yml         | 22 ++++++++++++++++++++++
> >  scripts/build-linux.sh | 18 +++++++++++++-----
> >  2 files changed, 35 insertions(+), 5 deletions(-)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 184d0b3..8d1deee 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -1,5 +1,9 @@
> >  variables:
> >    REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> > +  LINUX_GIT_VERSION:
> > +    description: "branch/tag/commit for the linux-git jobs"
> > +  LINUX_GIT_URL:
> > +    description: "git url for the linux-git jobs"
> >  
> >  stages:
> >    - build
> > @@ -53,6 +57,15 @@ linux-6.6.86-arm64:
> >    variables:
> >      LINUX_VERSION: 6.6.86
> >  
> > +linux-git-arm64:
> > +  extends: .arm64-artifacts
> > +  script: ./scripts/build-linux.sh
> > +  variables:
> > +    LINUX_VERSION: $LINUX_GIT_VERSION
> > +    LINUX_URL: $LINUX_GIT_URL
> > +  rules:
> > +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> > +
> >  #
> >  # x86_64 artifacts
> >  #
> > @@ -91,6 +104,15 @@ linux-6.12.60-x86_64:
> >    variables:
> >      LINUX_VERSION: 6.12.60
> >  
> > +linux-git-x86_64:
> > +  extends: .x86_64-artifacts
> > +  script: ./scripts/build-linux.sh
> > +  variables:
> > +    LINUX_VERSION: $LINUX_GIT_VERSION
> > +    LINUX_URL: $LINUX_GIT_URL
> > +  rules:
> > +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> > +
> >  microcode-x86:
> >    extends: .x86_64-artifacts
> >    script: ./scripts/x86-microcode.sh
> > diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
> > index cf0e744..1fc96d1 100755
> > --- a/scripts/build-linux.sh
> > +++ b/scripts/build-linux.sh
> > @@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
> >  UNAME=$(uname -m)
> >  
> >  # Build Linux
> > -MAJOR=${LINUX_VERSION%%.*}
> > -curl -fsSLO \
> > -    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > -tar xf linux-"${LINUX_VERSION}".tar.xz
> > -cd linux-"${LINUX_VERSION}"
> > +if [[ -n "${LINUX_URL}" ]]; then
> > +    mkdir linux
> > +    cd linux
> > +    git init
> > +    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
> > +    git checkout FETCH_HEAD
> > +else
> > +    MAJOR=${LINUX_VERSION%%.*}
> > +    curl -fsSLO \
> > +        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > +    tar xf linux-"${LINUX_VERSION}".tar.xz
> > +    cd linux-"${LINUX_VERSION}"
> > +fi
> 
> ... the issue is detecting to fetch via git or via curl based on the
> presence of a variable called "LINUX_URL". Technically 
> 
> https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> 
> is a a valid URL as well.
> 
> So I think you should keep LINUX_GIT_URL and LINUX_GIT_VERSION named as
> they are, expose them to scripts/build-linux.sh, and detect the fetch
> program based on the presence of LINUX_GIT_URL.
> 
> Ideally, we should not have LINUX_GIT_VERSION. Instead we should have a
> a common LINUX_VERSION used in both git and curl cases.

The problem here is conflicting variables for different jobs. If you
specify a variable when starting a pipeline (either manually, or via
schedule, or via settings), the variable will be set for all the jobs.
So, to be able to schedule a pipeline with both linux-6.12.60-x86_64 and
linux-git-x86_64 (for example based on linux-next, or maybe some rc),
the pipeline variable needs to be named differently than the one used by
this script. And IMO it's more important to have clear naming
(LINUX_GIT_VERSION+LINUX_GIT_URL) at the pipeline level.

I can change this script to use arguments instead of variables to avoid
this issue, but it will result in slightly more duplication (in
.gitlab-ci.yml file).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
Re: [PATCH v2 07/12] Support building arbitrary Linux branch/tag/commit
Posted by Stefano Stabellini 1 day, 23 hours ago 
On Wed, 10 Dec 2025, Marek Marczykowski-Górecki wrote:
> On Tue, Dec 09, 2025 at 04:29:02PM -0800, Stefano Stabellini wrote:
> > On Thu, 4 Dec 2025, Marek Marczykowski-Górecki wrote:
> > > If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
> > > init" + "git fetch" instead of "git clone" to support any of
> > > branch/tag/commit.
> > > 
> > > This also defines optional linux-git-* jobs which will build the thing
> > > if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
> > > pipeline.
> > > 
> > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > > ---
> > > The script variable and job variable need to have different names, so a
> > > pipeline variable won't override it for all jobs. While LINUX_VERSION /
> > > LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
> > > LINUX_GIT_URL. Any better ideas?
> > 
> > The problem is not LINUX_GIT_URL and LINUX_GIT_VERSION, those are good
> > names. The issue is ...
> > 
> > > ---
> > >  .gitlab-ci.yml         | 22 ++++++++++++++++++++++
> > >  scripts/build-linux.sh | 18 +++++++++++++-----
> > >  2 files changed, 35 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > index 184d0b3..8d1deee 100644
> > > --- a/.gitlab-ci.yml
> > > +++ b/.gitlab-ci.yml
> > > @@ -1,5 +1,9 @@
> > >  variables:
> > >    REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> > > +  LINUX_GIT_VERSION:
> > > +    description: "branch/tag/commit for the linux-git jobs"
> > > +  LINUX_GIT_URL:
> > > +    description: "git url for the linux-git jobs"
> > >  
> > >  stages:
> > >    - build
> > > @@ -53,6 +57,15 @@ linux-6.6.86-arm64:
> > >    variables:
> > >      LINUX_VERSION: 6.6.86
> > >  
> > > +linux-git-arm64:
> > > +  extends: .arm64-artifacts
> > > +  script: ./scripts/build-linux.sh
> > > +  variables:
> > > +    LINUX_VERSION: $LINUX_GIT_VERSION
> > > +    LINUX_URL: $LINUX_GIT_URL
> > > +  rules:
> > > +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> > > +
> > >  #
> > >  # x86_64 artifacts
> > >  #
> > > @@ -91,6 +104,15 @@ linux-6.12.60-x86_64:
> > >    variables:
> > >      LINUX_VERSION: 6.12.60
> > >  
> > > +linux-git-x86_64:
> > > +  extends: .x86_64-artifacts
> > > +  script: ./scripts/build-linux.sh
> > > +  variables:
> > > +    LINUX_VERSION: $LINUX_GIT_VERSION
> > > +    LINUX_URL: $LINUX_GIT_URL
> > > +  rules:
> > > +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> > > +
> > >  microcode-x86:
> > >    extends: .x86_64-artifacts
> > >    script: ./scripts/x86-microcode.sh
> > > diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
> > > index cf0e744..1fc96d1 100755
> > > --- a/scripts/build-linux.sh
> > > +++ b/scripts/build-linux.sh
> > > @@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
> > >  UNAME=$(uname -m)
> > >  
> > >  # Build Linux
> > > -MAJOR=${LINUX_VERSION%%.*}
> > > -curl -fsSLO \
> > > -    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > > -tar xf linux-"${LINUX_VERSION}".tar.xz
> > > -cd linux-"${LINUX_VERSION}"
> > > +if [[ -n "${LINUX_URL}" ]]; then
> > > +    mkdir linux
> > > +    cd linux
> > > +    git init
> > > +    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
> > > +    git checkout FETCH_HEAD
> > > +else
> > > +    MAJOR=${LINUX_VERSION%%.*}
> > > +    curl -fsSLO \
> > > +        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > > +    tar xf linux-"${LINUX_VERSION}".tar.xz
> > > +    cd linux-"${LINUX_VERSION}"
> > > +fi
> > 
> > ... the issue is detecting to fetch via git or via curl based on the
> > presence of a variable called "LINUX_URL". Technically 
> > 
> > https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > 
> > is a a valid URL as well.
> > 
> > So I think you should keep LINUX_GIT_URL and LINUX_GIT_VERSION named as
> > they are, expose them to scripts/build-linux.sh, and detect the fetch
> > program based on the presence of LINUX_GIT_URL.
> > 
> > Ideally, we should not have LINUX_GIT_VERSION. Instead we should have a
> > a common LINUX_VERSION used in both git and curl cases.
> 
> The problem here is conflicting variables for different jobs. If you
> specify a variable when starting a pipeline (either manually, or via
> schedule, or via settings), the variable will be set for all the jobs.
> So, to be able to schedule a pipeline with both linux-6.12.60-x86_64 and
> linux-git-x86_64 (for example based on linux-next, or maybe some rc),
> the pipeline variable needs to be named differently than the one used by
> this script. And IMO it's more important to have clear naming
> (LINUX_GIT_VERSION+LINUX_GIT_URL) at the pipeline level.
> 
> I can change this script to use arguments instead of variables to avoid
> this issue, but it will result in slightly more duplication (in
> .gitlab-ci.yml file).

It looks like it is a good idea to switch to arguments to avoid this
issue

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.