This is a follow up to:
https://listman.redhat.com/archives/libvir-list/2023-January/237201.html

The effort here is to unify the way builds/tests are executed in GitLab CI vs
local container executions and make another step forward in terms of
reproducibility of (specifically) GitLab environments.

Even though code to run all but one (coverity) jobs from GitLab via the
build.sh script is added with this series, local behavior remains the same as
before this series. The reason for that is that that will require more patches
ridding of the Makefile which is currently used and instead integrate usage of
lcitool with the ci/helper Python script which is currently the entry point for
local container executions.

Pipeline: https://gitlab.com/eskultety/libvirt/-/pipelines/768645158
    Ubuntu is having some repo connection issues today, so the one failed ^job
    can be ignored

Erik Skultety (20):
  gitlab-ci.yml: Replace all explicit calls to ninja with meson commands
  gitlab-ci.yml: potfile: Consolidate the meson compile calls
  gitlab-ci.yml: Use $HOME for rpmbuild's topdir instead of PWD
  ci: build.sh: Drop the commentary about CI_BUILD_SCRIPT
  ci: build.sh: Use 'meson setup' explicitly
  ci: build.sh: Always assume -Dsystem=true
  ci: build.sh: Drop the CI prefix from the CI_{MESON,NINJA}_ARGS vars
  ci: build.sh: Move off of ninja command to directly calling meson
  ci: build.sh: Join MESON_ARGS and MESON_OPTS
  ci: build.sh: Break the script functionality into helper functions
  ci: build.sh: Move the necessary env variables to build.sh
  ci: build.sh: Add support for individual GitLab jobs
  ci: build.sh: Wire up the individual job functions to the CLI
  ci: build.sh: Document CI_CONT_SRCDIR
  ci: build.sh: Make the build script fail ASAP with 'set -e'
  ci: build.sh: Update git index in local container environments on
    'dist'
  ci: build.sh: Make the script executable
  gitlab-ci.yml: Add 'after_script' stage to prep for artifact
    collection
  gitlab-ci.yml: Adopt job execution via a Bash script
  gitlab-ci.yml: Drop the usage of script variables reference

 .gitlab-ci.yml |  56 ++++++++++-------------
 ci/Makefile    |  16 ++++---
 ci/build.sh    | 121 +++++++++++++++++++++++++++++++++++++++++++------
 ci/helper      |  21 ++++++---
 4 files changed, 155 insertions(+), 59 deletions(-)
 mode change 100644 => 100755 ci/build.sh

-- 
2.39.1

On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> This is a follow up to:
> https://listman.redhat.com/archives/libvir-list/2023-January/237201.html
> 
> The effort here is to unify the way builds/tests are executed in GitLab CI vs
> local container executions and make another step forward in terms of
> reproducibility of (specifically) GitLab environments.
> 
> Even though code to run all but one (coverity) jobs from GitLab via the
> build.sh script is added with this series, local behavior remains the same as
> before this series. The reason for that is that that will require more patches
> ridding of the Makefile which is currently used and instead integrate usage of
> lcitool with the ci/helper Python script which is currently the entry point for
> local container executions.

snip

>  .gitlab-ci.yml |  56 ++++++++++-------------
>  ci/Makefile    |  16 ++++---
>  ci/build.sh    | 121 +++++++++++++++++++++++++++++++++++++++++++------
>  ci/helper      |  21 ++++++---
>  4 files changed, 155 insertions(+), 59 deletions(-)
>  mode change 100644 => 100755 ci/build.sh

I'm really super unenthusiastic about this change, and also the similar
change to add an ci/integration.sh. Shell scripting is something we
worked hard to eliminate from libvirt. It is an awful language
to do anything non-trivial with, error handling, lack of data
structures, variable handling, portability are all bug generators.

I know the .gitlab-ci.yml  'script' commands are technically shell
script, but they are pretty trivial bits and don't have to worry
about portability for dash vs bash etc, or complex control logic.
The majority of it is just a simple list of commands to invoke,
with the occasional conditional.

The build.sh script is by contrast significantly more complex. By
the very nature of taking "N" separate gitlab jobs and multiplexing
them onto the one shell script, we're now having todo command line
arg parsing in shell and de-multiplexing back to commands. The CI
logic is now split up across even more sources - the gitlab config,
the build.sh script and the meson.build files, which I think is
worse for maintaining this too. 

I appreciate the goal of being able to run CI commands locally, but
I'm not feeling this approach is a net win. I'd much rather just
having developers invoke the meson command directly, which is not
that hard.

If we do really want to provide something that 100% mirrors the
gitlab CI job commands though, I'm even more convinced now that
we should be using the .gitlab-ci.yml as the canonical source.

Since I last mentioned that idea, I found out that this should
be something we can achieve via the gitlab runner 'exec' command.

I've not quite figured out the right incantations though. I can
get it to work for a simple repo, but not for the lbivirt.git
yet as it seems to not find the included yml files.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote:
> On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> I appreciate the goal of being able to run CI commands locally, but
> I'm not feeling this approach is a net win. I'd much rather just
> having developers invoke the meson command directly, which is not
> that hard.
> 
> If we do really want to provide something that 100% mirrors the
> gitlab CI job commands though, I'm even more convinced now that
> we should be using the .gitlab-ci.yml as the canonical source.
> 
> Since I last mentioned that idea, I found out that this should
> be something we can achieve via the gitlab runner 'exec' command.

I wonder if we've been thinking about this from the wrong angle.

We've been looking at the problem of "we have a gitlab-ci.yml" and
we want to be able to locally run the jobs it defines. This creates
us the problem of interpreting the gitlab-ci.yml file, which is
very hard.

95% of what we define in the gitlab-ci.yml file comes in via the
includes of ci/gitlab.yml which is entirely auto-generated from
our ci/manifest.yml file. 

IOW, can we rephase the problem to "we have a ci/manifest.yml" and
we want to locally run the jobs it implies. We already have the
code for parsing the ci/manifest.yml file, and so (mostly) know
what jobs we expect to have.

What we're missing is the project specific build rules which
still live in the hand crafted .gitlab-ci.yml.

If we could get the project specific build rules to be expressed
in the ci/manifest.yml file,  then we have all the info we need
to be able to run jobs locally. We could then make even the root
.gitlab-ci.yml be auto-generated, which would be nice.

If we have the project build rules in ci/manifest.yml, we ahve
lots of flexibility. We can easily trigger the builds in containers
or VMs, or the local host, or whatever. 'lcitool' already has the
command for runing builds in VMs, but that relies on build rules
we defined separately in libvirt-ci.git and we've somewhat
neglected the VM build logic since adopting containers.

Essentially ci/manifest.yml would become a fully self-contained
canonical representation of any info needed to run builds, and be
independant of gitlab CI, or any other CI framework we might like
to plug into in the future. So 'lcitool' can be the entrypoint
for triggering builds  that 100% match what gets run in gitlab

Yes, this only addresses the core build/unittest side of things.

The integration test suite is still a separate task to address.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Feb 14, 2023 at 02:31:26PM +0000, Daniel P. Berrangé wrote:
> On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote:
> > On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> > I appreciate the goal of being able to run CI commands locally, but
> > I'm not feeling this approach is a net win. I'd much rather just
> > having developers invoke the meson command directly, which is not
> > that hard.
> > 
> > If we do really want to provide something that 100% mirrors the
> > gitlab CI job commands though, I'm even more convinced now that
> > we should be using the .gitlab-ci.yml as the canonical source.
> > 
> > Since I last mentioned that idea, I found out that this should
> > be something we can achieve via the gitlab runner 'exec' command.
> 
> I wonder if we've been thinking about this from the wrong angle.
> 
> We've been looking at the problem of "we have a gitlab-ci.yml" and
> we want to be able to locally run the jobs it defines. This creates
> us the problem of interpreting the gitlab-ci.yml file, which is
> very hard.
> 
> 95% of what we define in the gitlab-ci.yml file comes in via the
> includes of ci/gitlab.yml which is entirely auto-generated from
> our ci/manifest.yml file. 
> 
> IOW, can we rephase the problem to "we have a ci/manifest.yml" and
> we want to locally run the jobs it implies. We already have the
> code for parsing the ci/manifest.yml file, and so (mostly) know
> what jobs we expect to have.
> 
> What we're missing is the project specific build rules which
> still live in the hand crafted .gitlab-ci.yml.
> 
> If we could get the project specific build rules to be expressed
> in the ci/manifest.yml file,  then we have all the info we need
> to be able to run jobs locally. We could then make even the root
> .gitlab-ci.yml be auto-generated, which would be nice.

Honestly ^this to me sounds too meta. The way I look at this effort is
investing a significant amount of time into figuring out how build jobs which
are simply too specific for every project could be described in a generic
manner appropriately in YAML. I can't help it but it sounds like we'd be
inventing a problem for ourselves to solve. It might very well be relevant in
the future, I think we need faster results now. Like I mentioned in my previous
replies, I totally share your view on Bash from the language POV, but if we
make sure these scripts will avoid using any kind of CLI parsing and wannabe
data structure implementation, then I believe we can have a reasonable
compromise for the time being. After all it all boils down to how many people
are willing to invest how much of their time into improving CI.

> 
> If we have the project build rules in ci/manifest.yml, we ahve
> lots of flexibility. We can easily trigger the builds in containers
> or VMs, or the local host, or whatever. 'lcitool' already has the
> command for runing builds in VMs, but that relies on build rules
> we defined separately in libvirt-ci.git and we've somewhat
> neglected the VM build logic since adopting containers.
> 
> Essentially ci/manifest.yml would become a fully self-contained
> canonical representation of any info needed to run builds, and be
> independant of gitlab CI, or any other CI framework we might like
> to plug into in the future. So 'lcitool' can be the entrypoint
> for triggering builds  that 100% match what gets run in gitlab
> 
> Yes, this only addresses the core build/unittest side of things.

^This on its own would not be a problem as long as we have other tools ready to
be chained in the pipeline (or locally) appropriately. Still, I'd be wary of
trying to make a swiss-army knife from lcitool by generating everything
especially with tools like Tekton which has become the industry standard for
pipeline and job generation on K8S, something that might become relevant to
libvirt in coming years too, so I'm not completely stoked about this idea
proposal as I'm afraid we'll be forced to ditch some of it in the future
nonetheless.

Regards,
Erik

On Wed, Feb 15, 2023 at 09:06:46AM +0100, Erik Skultety wrote:
> On Tue, Feb 14, 2023 at 02:31:26PM +0000, Daniel P. Berrangé wrote:
> > On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote:
> > > On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> > > I appreciate the goal of being able to run CI commands locally, but
> > > I'm not feeling this approach is a net win. I'd much rather just
> > > having developers invoke the meson command directly, which is not
> > > that hard.
> > > 
> > > If we do really want to provide something that 100% mirrors the
> > > gitlab CI job commands though, I'm even more convinced now that
> > > we should be using the .gitlab-ci.yml as the canonical source.
> > > 
> > > Since I last mentioned that idea, I found out that this should
> > > be something we can achieve via the gitlab runner 'exec' command.
> > 
> > I wonder if we've been thinking about this from the wrong angle.
> > 
> > We've been looking at the problem of "we have a gitlab-ci.yml" and
> > we want to be able to locally run the jobs it defines. This creates
> > us the problem of interpreting the gitlab-ci.yml file, which is
> > very hard.
> > 
> > 95% of what we define in the gitlab-ci.yml file comes in via the
> > includes of ci/gitlab.yml which is entirely auto-generated from
> > our ci/manifest.yml file. 
> > 
> > IOW, can we rephase the problem to "we have a ci/manifest.yml" and
> > we want to locally run the jobs it implies. We already have the
> > code for parsing the ci/manifest.yml file, and so (mostly) know
> > what jobs we expect to have.
> > 
> > What we're missing is the project specific build rules which
> > still live in the hand crafted .gitlab-ci.yml.
> > 
> > If we could get the project specific build rules to be expressed
> > in the ci/manifest.yml file,  then we have all the info we need
> > to be able to run jobs locally. We could then make even the root
> > .gitlab-ci.yml be auto-generated, which would be nice.
> 
> Honestly ^this to me sounds too meta. The way I look at this effort is
> investing a significant amount of time into figuring out how build jobs which
> are simply too specific for every project could be described in a generic
> manner appropriately in YAML. I can't help it but it sounds like we'd be
> inventing a problem for ourselves to solve. It might very well be relevant in
> the future, I think we need faster results now. Like I mentioned in my previous
> replies, I totally share your view on Bash from the language POV, but if we
> make sure these scripts will avoid using any kind of CLI parsing and wannabe
> data structure implementation, then I believe we can have a reasonable
> compromise for the time being. After all it all boils down to how many people
> are willing to invest how much of their time into improving CI.
> 
> > 
> > If we have the project build rules in ci/manifest.yml, we ahve
> > lots of flexibility. We can easily trigger the builds in containers
> > or VMs, or the local host, or whatever. 'lcitool' already has the
> > command for runing builds in VMs, but that relies on build rules
> > we defined separately in libvirt-ci.git and we've somewhat
> > neglected the VM build logic since adopting containers.
> > 
> > Essentially ci/manifest.yml would become a fully self-contained
> > canonical representation of any info needed to run builds, and be
> > independant of gitlab CI, or any other CI framework we might like
> > to plug into in the future. So 'lcitool' can be the entrypoint
> > for triggering builds  that 100% match what gets run in gitlab
> > 
> > Yes, this only addresses the core build/unittest side of things.
> 
> ^This on its own would not be a problem as long as we have other tools ready to
> be chained in the pipeline (or locally) appropriately. Still, I'd be wary of
> trying to make a swiss-army knife from lcitool by generating everything
> especially with tools like Tekton which has become the industry standard for
> pipeline and job generation on K8S, something that might become relevant to
> libvirt in coming years too, so I'm not completely stoked about this idea
> proposal as I'm afraid we'll be forced to ditch some of it in the future
> nonetheless.

Ping. I'd still like some conclusion on this whether the proposed approach
(with changes like decomposition to scripts) is a complete no go and I need to
drop the whole series or if there's anything we can do about it right now.

Regards,
Erik

On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote:
> On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> > This is a follow up to:
> > https://listman.redhat.com/archives/libvir-list/2023-January/237201.html
> > 
> > The effort here is to unify the way builds/tests are executed in GitLab CI vs
> > local container executions and make another step forward in terms of
> > reproducibility of (specifically) GitLab environments.
> > 
> > Even though code to run all but one (coverity) jobs from GitLab via the
> > build.sh script is added with this series, local behavior remains the same as
> > before this series. The reason for that is that that will require more patches
> > ridding of the Makefile which is currently used and instead integrate usage of
> > lcitool with the ci/helper Python script which is currently the entry point for
> > local container executions.
> 
> snip
> 
> >  .gitlab-ci.yml |  56 ++++++++++-------------
> >  ci/Makefile    |  16 ++++---
> >  ci/build.sh    | 121 +++++++++++++++++++++++++++++++++++++++++++------
> >  ci/helper      |  21 ++++++---
> >  4 files changed, 155 insertions(+), 59 deletions(-)
> >  mode change 100644 => 100755 ci/build.sh
> 
> I'm really super unenthusiastic about this change, and also the similar
> change to add an ci/integration.sh. Shell scripting is something we
> worked hard to eliminate from libvirt. It is an awful language
> to do anything non-trivial with, error handling, lack of data
> structures, variable handling, portability are all bug generators.
> 
> I know the .gitlab-ci.yml  'script' commands are technically shell
> script, but they are pretty trivial bits and don't have to worry
> about portability for dash vs bash etc, or complex control logic.
> The majority of it is just a simple list of commands to invoke,
> with the occasional conditional.
> 
> The build.sh script is by contrast significantly more complex. By
> the very nature of taking "N" separate gitlab jobs and multiplexing
> them onto the one shell script, we're now having todo command line
> arg parsing in shell and de-multiplexing back to commands. The CI
> logic is now split up across even more sources - the gitlab config,
> the build.sh script and the meson.build files, which I think is
> worse for maintaining this too. 

True, except that even despite that, and I know what I'm talking about since I
wrote the integration jobs templates, GitLab is super picky; you can't debug
the syntax problems properly; certain common syntactic sugars don't work and
have to be replaced by less common counterparts; using YAML for bash formatting
leads to many table-flipping moments; you have to wait for Nx10 minutes before
it fails with some ridiculous error you would have caught early locally but you
couldn't because of the stage and artifact dependencies. So, once you have it
finally in place it's unarguably nice, but the road towards that isn't really
rainbow and unicorns either, it's still shell after all. Since I'm mainly
working on this part of libvirt, I myself would appreciate if I could easily
just run a single script instead of copy pasting commands one-by-one to a local
VM to catch stupid errors quickly rather than wait for GitLab to fail.

> 
> I appreciate the goal of being able to run CI commands locally, but
> I'm not feeling this approach is a net win. I'd much rather just
> having developers invoke the meson command directly, which is not
> that hard.

Well, the big picture here is about making running integration tests locally
less painful than it is now...plus increase the amount of automation involved.
That's the major driver here - libvirt needs to be built locally in a safe
environment before integration tests could be run against it without touching
the host. So, with this script, while I agree with everything you said about
Bash, was just one step towards getting the automation I mentioned in place. I
also considered Python (yeah..why), but that would be super akward.

TL;DR: if we don't do anything about how we currently maintain the build
recipes (note we're maintaining 2 already), everybody will continue ignoring the
integration test suite, unless we enable merge requests where the status quo
would be somewhat (but not completely) eliminated. With integration tests you
can't ignore the aspect of getting feedback early compared to waiting for
GitLab CI.

> 
> If we do really want to provide something that 100% mirrors the
> gitlab CI job commands though, I'm even more convinced now that
> we should be using the .gitlab-ci.yml as the canonical source.
> 
> Since I last mentioned that idea, I found out that this should
> be something we can achieve via the gitlab runner 'exec' command.

Haven't heard of ^this one, but I wonder how we can get something meaningful
out of it for the purposes I mentioned above.

Erik

> 
> I've not quite figured out the right incantations though. I can
> get it to work for a simple repo, but not for the lbivirt.git
> yet as it seems to not find the included yml files.
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

On Mon, Feb 06, 2023 at 05:19:52PM +0100, Erik Skultety wrote:
> On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote:
> > On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> > > This is a follow up to:
> > > https://listman.redhat.com/archives/libvir-list/2023-January/237201.html
> > > 
> > > The effort here is to unify the way builds/tests are executed in GitLab CI vs
> > > local container executions and make another step forward in terms of
> > > reproducibility of (specifically) GitLab environments.
> > > 
> > > Even though code to run all but one (coverity) jobs from GitLab via the
> > > build.sh script is added with this series, local behavior remains the same as
> > > before this series. The reason for that is that that will require more patches
> > > ridding of the Makefile which is currently used and instead integrate usage of
> > > lcitool with the ci/helper Python script which is currently the entry point for
> > > local container executions.
> > 
> > snip
> > 
> > >  .gitlab-ci.yml |  56 ++++++++++-------------
> > >  ci/Makefile    |  16 ++++---
> > >  ci/build.sh    | 121 +++++++++++++++++++++++++++++++++++++++++++------
> > >  ci/helper      |  21 ++++++---
> > >  4 files changed, 155 insertions(+), 59 deletions(-)
> > >  mode change 100644 => 100755 ci/build.sh
> > 
> > I'm really super unenthusiastic about this change, and also the similar
> > change to add an ci/integration.sh. Shell scripting is something we
> > worked hard to eliminate from libvirt. It is an awful language
> > to do anything non-trivial with, error handling, lack of data
> > structures, variable handling, portability are all bug generators.
> > 
> > I know the .gitlab-ci.yml  'script' commands are technically shell
> > script, but they are pretty trivial bits and don't have to worry
> > about portability for dash vs bash etc, or complex control logic.
> > The majority of it is just a simple list of commands to invoke,
> > with the occasional conditional.
> > 
> > The build.sh script is by contrast significantly more complex. By
> > the very nature of taking "N" separate gitlab jobs and multiplexing
> > them onto the one shell script, we're now having todo command line
> > arg parsing in shell and de-multiplexing back to commands. The CI
> > logic is now split up across even more sources - the gitlab config,
> > the build.sh script and the meson.build files, which I think is
> > worse for maintaining this too. 
> 
> True, except that even despite that, and I know what I'm talking about since I
> wrote the integration jobs templates, GitLab is super picky; you can't debug
> the syntax problems properly; certain common syntactic sugars don't work and
> have to be replaced by less common counterparts; using YAML for bash formatting
> leads to many table-flipping moments; you have to wait for Nx10 minutes before
> it fails with some ridiculous error you would have caught early locally but you
> couldn't because of the stage and artifact dependencies. So, once you have it
> finally in place it's unarguably nice, but the road towards that isn't really
> rainbow and unicorns either, it's still shell after all. Since I'm mainly
> working on this part of libvirt, I myself would appreciate if I could easily
> just run a single script instead of copy pasting commands one-by-one to a local
> VM to catch stupid errors quickly rather than wait for GitLab to fail.
> 
> > 
> > I appreciate the goal of being able to run CI commands locally, but
> > I'm not feeling this approach is a net win. I'd much rather just
> > having developers invoke the meson command directly, which is not
> > that hard.
> 
> Well, the big picture here is about making running integration tests locally
> less painful than it is now...plus increase the amount of automation involved.
> That's the major driver here - libvirt needs to be built locally in a safe
> environment before integration tests could be run against it without touching
> the host. So, with this script, while I agree with everything you said about
> Bash, was just one step towards getting the automation I mentioned in place. I
> also considered Python (yeah..why), but that would be super akward.
> 
> TL;DR: if we don't do anything about how we currently maintain the build
> recipes (note we're maintaining 2 already), everybody will continue ignoring the
> integration test suite, unless we enable merge requests where the status quo
> would be somewhat (but not completely) eliminated. With integration tests you
> can't ignore the aspect of getting feedback early compared to waiting for
> GitLab CI.
> 
> > 
> > If we do really want to provide something that 100% mirrors the
> > gitlab CI job commands though, I'm even more convinced now that
> > we should be using the .gitlab-ci.yml as the canonical source.
> > 
> > Since I last mentioned that idea, I found out that this should
> > be something we can achieve via the gitlab runner 'exec' command.
> 
> Haven't heard of ^this one, but I wonder how we can get something meaningful
> out of it for the purposes I mentioned above.
> 
> Erik

So, I had a brief look at gihttps://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797tlab-runner exec this morning. Long story short,
it's deprecated [1] and though it was scheduled for complete removal, that plan
was put on hold for now. Nevertheless, it's not getting any features that are
introduced to .gitlab-ci.yml. The gist is that it doesn't understand 'include'
nor 'extends' which we're using heavily across our gitlab configuration
hierarchy, so it's a no-go. It also doesn't support artifacts in any form, so
while it technically should be possible to save RPM builds in a volume (not
sure if the bind mount is cleared on job completion or not) we could not pass
them into a VM running the integration tests the same convenient way as we do
in the CI.

Another thing to consider is that this solution also requires to have the
gitlab-runner binary installed locally, which many in the community may not be
comfortable with, but that's up for a debate, but hey, you can always build it
yourself.

There's a plan to rework GitLab's local CI execution functionality heavily [2]
before the 'exec' command is sunset, but ultimately their plan is have
something working by Q3 FY24, which can be anywhere from March 2024 until
September 2024, that's more than a year ahead.

In your reply you mentioned the unappealing complexity of the script
potentially leading to bugs. At the same time though one currently can't consume
.gitlab-ci.yml recipes to run a local integration test in a VM.
So, how about I get rid of the multiplexing and CLI parsing by placing each job
recipe in a standalone script or even going one step further and extract the
commonalities to a separate file that would be source from within each job
script? Would you consider that meeting each other halfway?
As for POSIX compliance, I guess this would be a soft-requirement based on
whether shellcheck was run during review, does gitlab do something better? IIUC
there's no pre-check and you'll only know after a job was executed that there
was an incompatible statement.

[1] https://docs.gitlab.com/runner/commands/#gitlab-runner-exec-deprecated
[2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797

Regards,
Erik

On Tue, Feb 07, 2023 at 10:56:59AM +0100, Erik Skultety wrote:
> 
> So, I had a brief look at https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797tlab-runner exec this morning. Long story short,
> it's deprecated [1] and though it was scheduled for complete removal, that plan
> was put on hold for now. Nevertheless, it's not getting any features that are
> introduced to .gitlab-ci.yml. The gist is that it doesn't understand 'include'
> nor 'extends' which we're using heavily across our gitlab configuration
> hierarchy, so it's a no-go.

Urgh, I wasn't anticipating that they used completely different code to
interpret the config in the runner, than elsewhere in gitlab. No wonder
they want to kill it off :-(

>                            It also doesn't support artifacts in any form, so
> while it technically should be possible to save RPM builds in a volume (not
> sure if the bind mount is cleared on job completion or not) we could not pass
> them into a VM running the integration tests the same convenient way as we do
> in the CI.

Yep, I'm sure there are possible workarounds, but only worth it if the
rest of the thing is viable, which isn't the case with include/extends
support missing :-(

> In your reply you mentioned the unappealing complexity of the script
> potentially leading to bugs. At the same time though one currently can't consume
> .gitlab-ci.yml recipes to run a local integration test in a VM.

> So, how about I get rid of the multiplexing and CLI parsing by placing each job
> recipe in a standalone script or even going one step further and extract the
> commonalities to a separate file that would be source from within each job
> script? Would you consider that meeting each other halfway?

Lets consider the core CI (build + unit tests in .gitlab-ci.yml), separately
from the integration test CI (ci/integration-template.yml).

For the core CI, I'm just not convinced of the benefit of moving the commands
out into a shell script, as the set of commands is small and straightforward.

For the ingration CI though, I can see benefit because of all the command
logic related to fetching and building dependancies and setup tasks.

In retrospect this is a sign that we need to introduce a higher level frontend
for invoking the libvirt-tck tests. We already have avocado as a frontend to
replace the existing  'libvirt-tck' command line tool, but we never got rid of
the latter. We should probably get rid of the existing 'libvirt-tck' cli and
introduce a brand new pure python 'libvirt-tck' cli, with commands for running
avocado, optionally installing deps, and all the other interesting things devs
want help with.  The ci/integration-template.yml could then invoke this newly
re-invented 'libvirt-tck' command and eliminate most of the complexity in the
integration-template.yml file.

Essentially we need to look at the gitlab CI yml files as being a very thin
glue layer, and anything interesting should be in the end user facing tools
developers already use.

Introducing a new libvirt-tck python cli is likely not a quick fix though.
So if we split the ci/integration-template.yml commands out into a set of
bash scripts, that'll give us a good illustration of what commands we would
want in a new libvirt-tck cli impl. Then as the new libvirt-tck cli arrives,
we can eliminate the bash scripts from libvirt.git

I guess this would mean dropping much of this particular series, but keeping
much of the corresponding integration test series.

> As for POSIX compliance, I guess this would be a soft-requirement based on
> whether shellcheck was run during review, does gitlab do something better? IIUC
> there's no pre-check and you'll only know after a job was executed that there
> was an incompatible statement.

I think its ok if we use bash scripts as a short term solution, with a
plan to move to a more supportable python impl in future.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

...

> > So, how about I get rid of the multiplexing and CLI parsing by placing each job
> > recipe in a standalone script or even going one step further and extract the
> > commonalities to a separate file that would be source from within each job
> > script? Would you consider that meeting each other halfway?
> 
> Lets consider the core CI (build + unit tests in .gitlab-ci.yml), separately
> from the integration test CI (ci/integration-template.yml).
> 
> For the core CI, I'm just not convinced of the benefit of moving the commands
> out into a shell script, as the set of commands is small and straightforward.
> 
> For the ingration CI though, I can see benefit because of all the command
> logic related to fetching and building dependancies and setup tasks.
> 
> In retrospect this is a sign that we need to introduce a higher level frontend
> for invoking the libvirt-tck tests. We already have avocado as a frontend to
> replace the existing  'libvirt-tck' command line tool, but we never got rid of
> the latter. We should probably get rid of the existing 'libvirt-tck' cli and
> introduce a brand new pure python 'libvirt-tck' cli, with commands for running
> avocado, optionally installing deps, and all the other interesting things devs
> want help with.  The ci/integration-template.yml could then invoke this newly
> re-invented 'libvirt-tck' command and eliminate most of the complexity in the
> integration-template.yml file.

Well, that was the plan ever since June 2021 when the concept of lavocado was
proposed to the list, but it wasn't widely accepted as it appears we haven't
found the common ground back then. IIRC the idea of making lavocado the
new TCK re-incarnation was actually accepted, but the effort on its own didn't
get traction, particularly because IMO there's not that much value spending
time re-writing existing Perl tests to Python without investing time into
improving the environment preparation and hence eliminating the biggest
problem all these frameworks have which is that to some extent they're still
touching the system libvirt on the host, potentially leaving it in an
inconsitent state. And so I've been steering my attention towards making the
necessary changes to lcitool and hopefully make it the virt-stack's universal
environment preparation tool.
As for what the new TCK cli should look like, that's a good question, FWIW I
don't agree we should introduce any logic related to dependency installation
into the environment, nor anything that is environment related - we've done
that with Avocado-VT and we know how it ended; the pure Avocado framework also
provides some of these features which I don't necessarily agree with (Ansible
integration, container execution integration, etc.), but luckily these are not
mandatory. I am a strong advocate of dividing the CI problem into environment
preparation and test execution and not actually trying to combine these two,
it's not going to end up well IMO (we have a precedent) and instead the new
framework needs to do a single thing which is execute the tests and do it well
and do it agnostic to the environment AND to the language in which the tests
have been written - all of which Avocado is very good at.

Now, while I'm very happy that the idea of reworking TCK has come to the table
again, like you mention below, it's not an easy and quick task, so we need some
temporary bandaids before we get there and I'm totally fine dropping much of
the conversion efforts in the future, but I still think it can help us short
term.

> 
> Essentially we need to look at the gitlab CI yml files as being a very thin
> glue layer, and anything interesting should be in the end user facing tools
> developers already use.
> 
> Introducing a new libvirt-tck python cli is likely not a quick fix though.
> So if we split the ci/integration-template.yml commands out into a set of
> bash scripts, that'll give us a good illustration of what commands we would
> want in a new libvirt-tck cli impl. Then as the new libvirt-tck cli arrives,
> we can eliminate the bash scripts from libvirt.git

Yes, agreed.

> 
> I guess this would mean dropping much of this particular series, but keeping
> much of the corresponding integration test series.

How much of this series should be dropped in your opinion, given my vision I
tried to briefly outline above? I admit I got confused a bit by what your
actual expectation from this series is.

Erik

> 
> > As for POSIX compliance, I guess this would be a soft-requirement based on
> > whether shellcheck was run during review, does gitlab do something better? IIUC
> > there's no pre-check and you'll only know after a job was executed that there
> > was an incompatible statement.
> 
> I think its ok if we use bash scripts as a short term solution, with a
> plan to move to a more supportable python impl in future.
> 
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>