[v1] CI: Allow automatic rebuild of rolling release containers

[XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers

Posted by Anthony PERARD 5 days, 17 hours ago

Patch series available in this git branch:
https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v1

Hi,

This patch series would allow to setup a scheduled pipeline on GitLab which
would rebuild containers and run a test on them immediately (the container is
updated even if the test fail, but at least we can find out about it sooner
rather than later).

To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will
do.

I intend to add a scheduled pipeline to run once a month.

Thanks,

Anthony PERARD (3):
  CI: Remove deprecated "only:variables" in favor of "rules:if"
  CI: Define XEN_REGISTRY variable
  CI: New stage "containers" to rebuild some containers

 .gitlab-ci.yml                       |  5 +++
 automation/gitlab-ci/build.yaml      | 61 ++++++++++++++++++----------
 automation/gitlab-ci/containers.yaml | 29 +++++++++++++
 automation/gitlab-ci/test.yaml       | 30 +++++++++-----
 4 files changed, 93 insertions(+), 32 deletions(-)
 create mode 100644 automation/gitlab-ci/containers.yaml

-- 


Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers

Posted by Andrew Cooper 5 days, 17 hours ago

On 15/11/2024 5:07 pm, Anthony PERARD wrote:
> Patch series available in this git branch:
> https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v1
>
> Hi,
>
> This patch series would allow to setup a scheduled pipeline on GitLab which
> would rebuild containers and run a test on them immediately (the container is
> updated even if the test fail, but at least we can find out about it sooner
> rather than later).
>
> To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will
> do.
>
> I intend to add a scheduled pipeline to run once a month.

Oh excellent.  Thanks for looking into this.

One thing I was going to look into doing was to push the new containers
to a temporary repository (or a temp name in the main repository), kick
off a regular pipeline overriding image:, and on success doing a rename.

That would avoid pushing a container with problems into main CI.

~Andrew

Re: [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers

Posted by Anthony PERARD 2 days, 21 hours ago

On Fri, Nov 15, 2024 at 05:13:13PM +0000, Andrew Cooper wrote:
> On 15/11/2024 5:07 pm, Anthony PERARD wrote:
> > Patch series available in this git branch:
> > https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v1
> >
> > Hi,
> >
> > This patch series would allow to setup a scheduled pipeline on GitLab which
> > would rebuild containers and run a test on them immediately (the container is
> > updated even if the test fail, but at least we can find out about it sooner
> > rather than later).
> >
> > To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will
> > do.
> >
> > I intend to add a scheduled pipeline to run once a month.
> 
> Oh excellent.  Thanks for looking into this.
> 
> One thing I was going to look into doing was to push the new containers
> to a temporary repository (or a temp name in the main repository), kick
> off a regular pipeline overriding image:, and on success doing a rename.
> 
> That would avoid pushing a container with problems into main CI.

I tried something like that before, but I probably didn't approch the
problem from the right angle:

    [RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers
    https://lore.kernel.org/xen-devel/20230302175332.56052-1-anthony.perard@citrix.com/#r

I was trying to automatically rebuild container whose Dockerfile had
changed, while also having logic to rebuild containers based on
rolling-release distributions. But I did end-up duplicating
"build.yaml".

There's logic in that old series to test a container before "renaming
it" (which involve pulling then pushing the container with a new tag)

With this new series, I've introduced XEN_REGISTRY, so we could schedule
a pipeline by setting XEN_REGISTRY variable to a staging area, and
finish the pipeline by pushing to the main location.

But, the two containers been updated in this series are used in jobs
that are allowed to fail, so updating them don't have any consequence.
Also, there's no value in keeping to test with outdated container as
that mean we test with an environment that doesn't exist anymore.

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech