[libvirt-ci PATCH] guests: introduce libvirt-dist and libvirt-minimal projects

Daniel P. Berrangé posted 1 patch 3 years, 11 months ago
Failed in applying to current master (apply log)
guests/lcitool                           |  7 -------
guests/vars/mappings.yml                 |  5 +++++
guests/vars/projects/libvirt-dist.yml    |  3 +++
guests/vars/projects/libvirt-minimal.yml | 13 +++++++++++++
4 files changed, 21 insertions(+), 7 deletions(-)
create mode 100644 guests/vars/projects/libvirt-dist.yml
create mode 100644 guests/vars/projects/libvirt-minimal.yml
[libvirt-ci PATCH] guests: introduce libvirt-dist and libvirt-minimal projects
Posted by Daniel P. Berrangé 3 years, 11 months ago
The language bindings have traditionally built against a full libvirt,
however, this is serious overkill because all they should really need
most of the time is access to the main API, and the test driver if they
want to run functional tests. The full libvirt build with virt drivers
is only needed for integration testing which most bindings won't do.

Thus this introduces a new project "libvirt-minimal" which lists the
bare minimum dependencies required to build libvirt.

Taking libvirt-perl as an example, if creating a container using the
current "libvirt,libvirt-perl" project set, the result with 1.4 GB
in size for Fedora 31. With the "libvirt-minimal,libvirt-perl" set,
the size is just 777 MB.

Some projects also wish to have the ability to build against the distro
provided libvirt instead of the latest git master, and for this purpose
a "libvirt-dist" project is defined which pulls in the package needed
for building against the distro libvirt.

In the lcitool dockerfile command, there is a check that the requested
project is configured against the requested host. These new projects are
not listed against any host, because we don't want them installed by
default in the VMs. Thus the dockerfile check is no longer valid and is
removed. Whomever is invoking lcitool knows what combination they want
and whether it will work.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 guests/lcitool                           |  7 -------
 guests/vars/mappings.yml                 |  5 +++++
 guests/vars/projects/libvirt-dist.yml    |  3 +++
 guests/vars/projects/libvirt-minimal.yml | 13 +++++++++++++
 4 files changed, 21 insertions(+), 7 deletions(-)
 create mode 100644 guests/vars/projects/libvirt-dist.yml
 create mode 100644 guests/vars/projects/libvirt-minimal.yml

diff --git a/guests/lcitool b/guests/lcitool
index 0c89e13..abc87d2 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -1007,13 +1007,6 @@ class Application:
         for project in projects:
             if project.rfind("+mingw") >= 0:
                 raise Exception("Obsolete syntax, please use --cross-arch")
-            if project not in facts["projects"]:
-                raise Exception(
-                    "Host {} doesn't support project {}".format(
-                        host,
-                        project,
-                    )
-                )
 
         varmap = self._dockerfile_build_varmap(facts, mappings, pip_mappings, projects, cross_arch)
         self._dockerfile_format(facts, cross_arch, varmap)
diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml
index 753f0fe..a9ab3d5 100644
--- a/guests/vars/mappings.yml
+++ b/guests/vars/mappings.yml
@@ -450,6 +450,11 @@ mappings:
     rpm: libuuid-devel
     cross-policy-deb: foreign
 
+  libvirt-devel:
+    rpm: libvirt-devel
+    deb: libvirt-dev
+    pkg: libvirt
+
   libxml2:
     deb: libxml2-dev
     pkg: libxml2
diff --git a/guests/vars/projects/libvirt-dist.yml b/guests/vars/projects/libvirt-dist.yml
new file mode 100644
index 0000000..aa95f23
--- /dev/null
+++ b/guests/vars/projects/libvirt-dist.yml
@@ -0,0 +1,3 @@
+---
+packages:
+  - libvirt-devel
diff --git a/guests/vars/projects/libvirt-minimal.yml b/guests/vars/projects/libvirt-minimal.yml
new file mode 100644
index 0000000..9402c4f
--- /dev/null
+++ b/guests/vars/projects/libvirt-minimal.yml
@@ -0,0 +1,13 @@
+---
+packages:
+  - glib2
+  - gnutls
+  - libnl3
+  - libnlroute3
+  - libpcap
+  - libtirpc
+  - libxml2
+  - python3-docutils
+  - rpcgen
+  - xmllint
+  - xsltproc
-- 
2.25.4

Re: [libvirt-ci PATCH] guests: introduce libvirt-dist and libvirt-minimal projects
Posted by Andrea Bolognani 3 years, 11 months ago
On Wed, 2020-04-29 at 11:38 +0100, Daniel P. Berrangé wrote:
> Thus this introduces a new project "libvirt-minimal" which lists the
> bare minimum dependencies required to build libvirt.

I don't like the name libvirt-minimal because it appears to follows
the convention used for actual libvirt-based projects such as
libvirt-glib.

What about libvirt+minimal, following the project+variant convention
used for all MinGW builds[1] and in the past for libvirt+website?

> Some projects also wish to have the ability to build against the distro
> provided libvirt instead of the latest git master, and for this purpose
> a "libvirt-dist" project is defined which pulls in the package needed
> for building against the distro libvirt.

I don't like this name either :)

One reason is the same detailed above, but there's another one which
is tied to semantics: in lcitool, "$project" has always been used to
mean "the list of packages necessary to build $project from source".

This is the case for all the projects that exist today[2] and even
for libvirt+minimal, but this one is different: in this case,
libvirt-dist means "the list of packages necessary to build a project
that *depends* on libvirt from source". That's quite different.

I think a better approach would be to once again use variants: for
example, if you wanted to build libvirt-python against the version of
libvirt included in the OS, instead of having something like

  # libvirt-python.yml
  ---
  packages:
    - python3
    ...

  # libvirt-dist.yml
  ---
  packages:
    - libvirt

which is then used like

  $ lcitool dockerfile $OS libvirt-dist,libvirt-python

you'd have

  # libvirt-python+dist.yml
  ---
  packages:
    - libvirt
    - python-3
    ...

which is used like

  $ lcitool dockerfile $OS libvirt-python+dist

This would achieve the same result with less typing and without
subverting the existing semantics.

Another nice property of this approach is that it naturally extends
to cover projects that depend on more than just libvirt: for example,
we could have

  # libvirt-dbus+dist.yml
  ---
  packages:
    - libvirt
    - libvirt-glib
    ...

for use in libvirt-dbus CI.

What do you think?

[...]
> +++ b/guests/vars/mappings.yml
> +  libvirt-devel:
> +    rpm: libvirt-devel
> +    deb: libvirt-dev
> +    pkg: libvirt

The mapping should be called just "libvirt".


[1] Applying this convention to MinGW builds is increasingly
    inaccurate now that we have proper multi-arch support, and in
    fact at least at the Dockerfile generator level we've gotten
    away with this convention; eradicating it completely is something
    that I hope I'll be able to spend time on in the near future.
[2] Except for base, blacklist and jenkins, but those are internal
    projects which are never exposed to the user so they don't need
    to follow the rules as strictly :)
-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [libvirt-ci PATCH] guests: introduce libvirt-dist and libvirt-minimal projects
Posted by Daniel P. Berrangé 3 years, 11 months ago
On Wed, Apr 29, 2020 at 08:52:06PM +0200, Andrea Bolognani wrote:
> On Wed, 2020-04-29 at 11:38 +0100, Daniel P. Berrangé wrote:
> > Thus this introduces a new project "libvirt-minimal" which lists the
> > bare minimum dependencies required to build libvirt.
> 
> I don't like the name libvirt-minimal because it appears to follows
> the convention used for actual libvirt-based projects such as
> libvirt-glib.
> 
> What about libvirt+minimal, following the project+variant convention
> used for all MinGW builds[1] and in the past for libvirt+website?
> 
> > Some projects also wish to have the ability to build against the distro
> > provided libvirt instead of the latest git master, and for this purpose
> > a "libvirt-dist" project is defined which pulls in the package needed
> > for building against the distro libvirt.
> 
> I don't like this name either :)
> 
> One reason is the same detailed above, but there's another one which
> is tied to semantics: in lcitool, "$project" has always been used to
> mean "the list of packages necessary to build $project from source".
> 
> This is the case for all the projects that exist today[2] and even
> for libvirt+minimal, but this one is different: in this case,
> libvirt-dist means "the list of packages necessary to build a project
> that *depends* on libvirt from source". That's quite different.
> 
> I think a better approach would be to once again use variants: for
> example, if you wanted to build libvirt-python against the version of
> libvirt included in the OS, instead of having something like
> 
>   # libvirt-python.yml
>   ---
>   packages:
>     - python3
>     ...
> 
>   # libvirt-dist.yml
>   ---
>   packages:
>     - libvirt
> 
> which is then used like
> 
>   $ lcitool dockerfile $OS libvirt-dist,libvirt-python
> 
> you'd have
> 
>   # libvirt-python+dist.yml
>   ---
>   packages:
>     - libvirt
>     - python-3
>     ...
> 
> which is used like
> 
>   $ lcitool dockerfile $OS libvirt-python+dist
> 
> This would achieve the same result with less typing and without
> subverting the existing semantics.

This results in defining the combinatorial expansion of project sets
which just looks like unecessary duplication & work to me. It also
gives different syntax for configuring a container to build from git
vs dist.

There is only ever one project here "libvirt-project" and nothing
about it is is changing, except for which "libvirt" it is being
built against. It supports any libvirt, whether a full git build
or a minimal git build, or a distro build or some other build:

  $ lcitool dockerfile $OS libvirt,libvirt-python
  $ lcitool dockerfile $OS libvirt-dist,libvirt-python
  $ lcitool dockerfile $OS libvirt-minimal,libvirt-python

We could call it "libvirt+dist" instead "libvirt-dist" if we want

> Another nice property of this approach is that it naturally extends
> to cover projects that depend on more than just libvirt: for example,
> we could have
> 
>   # libvirt-dbus+dist.yml
>   ---
>   packages:
>     - libvirt
>     - libvirt-glib
>     ...
> 
> for use in libvirt-dbus CI.

I think that's a downside not a plus, because it makes the combinatorial
expansion even bigger.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [libvirt-ci PATCH] guests: introduce libvirt-dist and libvirt-minimal projects
Posted by Andrea Bolognani 3 years, 11 months ago
On Thu, 2020-04-30 at 12:17 +0100, Daniel P. Berrangé wrote:
> On Wed, Apr 29, 2020 at 08:52:06PM +0200, Andrea Bolognani wrote:
> >   # libvirt-python+dist.yml
> >   ---
> >   packages:
> >     - libvirt
> >     - python-3
> >     ...
> > 
> > which is used like
> > 
> >   $ lcitool dockerfile $OS libvirt-python+dist
> > 
> > This would achieve the same result with less typing and without
> > subverting the existing semantics.
> 
> This results in defining the combinatorial expansion of project sets
> which just looks like unecessary duplication & work to me. It also
> gives different syntax for configuring a container to build from git
> vs dist.
> 
> There is only ever one project here "libvirt-project" and nothing
> about it is is changing, except for which "libvirt" it is being
> built against. It supports any libvirt, whether a full git build
> or a minimal git build, or a distro build or some other build:
> 
>   $ lcitool dockerfile $OS libvirt,libvirt-python
>   $ lcitool dockerfile $OS libvirt-dist,libvirt-python
>   $ lcitool dockerfile $OS libvirt-minimal,libvirt-python

Okay, this is a pretty solid argument.

It also opens up the possibility of some external project, that
uses libvirt but is not known to lcitool, using

  $ lcitool dockerfile $OS libvirt+dist

to generate a base container for their CI use, which coud be useful.

The semantics of $project and $project+dist are still different
enough that we should document them properly in lcitool's README.

> We could call it "libvirt+dist" instead "libvirt-dist" if we want

Yes please.

-- 
Andrea Bolognani / Red Hat / Virtualization