[PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

Andrew Cooper posted 12 patches 2 months, 1 week ago
[PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Andrew Cooper 2 months ago
Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names
for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially.  Testing docs isn't very interesting and
saves a lot of space (pandoc pulls in all of Haskel).  Other savings come from
removing a huge pile of optional QEMU dependencies (QEMU just needs to build
the Xen parts to be useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies,
which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this
total overhaul.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

This includes both of Olaf's recent changes, python3-packaging (now required
for a rebuilt container), and findutils (not required yet; still gets pulled
in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine.
staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
hence it's safe to drop `which`.
---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile

diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
new file mode 100644
index 000000000000..8c5ff56cdc88
--- /dev/null
+++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
@@ -0,0 +1,76 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-devel
+        libzstd-devel
+        lzo-devel
+        xz-devel
+        zlib-devel
+        # libacpi
+        acpica
+        # libxl
+        libuuid-devel
+        libyajl-devel
+        # Header Check
+        gcc-c++
+        # xentop
+        ncurses-devel
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        glib2-devel
+        libpixman-1-0-devel
+        meson
+        ninja
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
deleted file mode 100644
index 077fed22260c..000000000000
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ /dev/null
@@ -1,74 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/tumbleweed
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper dup -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        libzstd-devel \
-        lzo-devel \
-        make \
-        meson \
-        nasm \
-        ncurses-devel \
-        ninja \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index b4139414bc8e..fd6ca1270e4e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -668,25 +668,25 @@ opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 4d5669c5ad01..c4f013702122 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -43,7 +43,7 @@ case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used
-- 
2.39.2


Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Olaf Hering 2 months ago
Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:

> use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Names of rpm packages do change over time, configure already searches for "something" where it is available, instead of some library or header.
If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:

libbz2-devel
    pkgconfig(bzip2) = 1.0.8
libzstd-devel
    pkgconfig(libzstd) = 1.5.6
lzo-devel
    pkgconfig(lzo2) = 2.10
xz-devel
    pkgconfig(liblzma) = 5.6.2
zlib-devel
    pkgconfig(zlib) = 1.3.1
libuuid-devel
    pkgconfig(uuid) = 2.40.1
libyajl-devel
    pkgconfig(yajl) = 2.1.0
ncurses-devel
    pkgconfig(form) = 6.5.20240629
    pkgconfig(formw) = 6.5.20240629
    pkgconfig(menu) = 6.5.20240629
    pkgconfig(menuw) = 6.5.20240629
    pkgconfig(ncurses) = 6.5.20240629
    pkgconfig(ncurses++) = 6.5.20240629
    pkgconfig(ncurses++w) = 6.5.20240629
    pkgconfig(ncursesw) = 6.5.20240629
    pkgconfig(panel) = 6.5.20240629
    pkgconfig(panelw) = 6.5.20240629
    pkgconfig(tic) = 6.5.20240629
    pkgconfig(tinfo) = 6.5.20240629
glib2-devel
    pkgconfig(gio-2.0) = 2.80.3
    pkgconfig(gio-unix-2.0) = 2.80.3
    pkgconfig(girepository-2.0) = 2.80.3
    pkgconfig(glib-2.0) = 2.80.3
    pkgconfig(gmodule-2.0) = 2.80.3
    pkgconfig(gmodule-export-2.0) = 2.80.3
    pkgconfig(gmodule-no-export-2.0) = 2.80.3
    pkgconfig(gobject-2.0) = 2.80.3
    pkgconfig(gthread-2.0) = 2.80.3
libpixman-1-0-devel
    pkgconfig(pixman-1) = 0.43.4

python3-devel is "special", can be used as is.

'git' could be just 'git-core'


Olaf
Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Andrew Cooper 2 months ago
On 11/07/2024 10:49 pm, Olaf Hering wrote:
> Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> use the zypper long names for the benefit of those wondering what was being referenced or duplicated.
> Names of rpm packages do change over time, configure already searches for "something" where it is available,

configure has to cope with distros taking different naming decisions,
and while I appreciate that packages may change over time even within a
distro ...

>  instead of some library or header.
> If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:
>
> libbz2-devel
>     pkgconfig(bzip2) = 1.0.8
> libzstd-devel
>     pkgconfig(libzstd) = 1.5.6
> lzo-devel
>     pkgconfig(lzo2) = 2.10
> xz-devel
>     pkgconfig(liblzma) = 5.6.2
> zlib-devel
>     pkgconfig(zlib) = 1.3.1
> libuuid-devel
>     pkgconfig(uuid) = 2.40.1
> libyajl-devel
>     pkgconfig(yajl) = 2.1.0
> ncurses-devel
>     pkgconfig(form) = 6.5.20240629
>     pkgconfig(formw) = 6.5.20240629
>     pkgconfig(menu) = 6.5.20240629
>     pkgconfig(menuw) = 6.5.20240629
>     pkgconfig(ncurses) = 6.5.20240629
>     pkgconfig(ncurses++) = 6.5.20240629
>     pkgconfig(ncurses++w) = 6.5.20240629
>     pkgconfig(ncursesw) = 6.5.20240629
>     pkgconfig(panel) = 6.5.20240629
>     pkgconfig(panelw) = 6.5.20240629
>     pkgconfig(tic) = 6.5.20240629
>     pkgconfig(tinfo) = 6.5.20240629
> glib2-devel
>     pkgconfig(gio-2.0) = 2.80.3
>     pkgconfig(gio-unix-2.0) = 2.80.3
>     pkgconfig(girepository-2.0) = 2.80.3
>     pkgconfig(glib-2.0) = 2.80.3
>     pkgconfig(gmodule-2.0) = 2.80.3
>     pkgconfig(gmodule-export-2.0) = 2.80.3
>     pkgconfig(gmodule-no-export-2.0) = 2.80.3
>     pkgconfig(gobject-2.0) = 2.80.3
>     pkgconfig(gthread-2.0) = 2.80.3
> libpixman-1-0-devel
>     pkgconfig(pixman-1) = 0.43.4

... this is just a differently-inconsistent list of names, with some
very dubious aliases such as `form` and `menu`.

What makes pkgconfig() any more stable than the main package name?

Why do we want to be treating libraries different to other packages?

> python3-devel is "special", can be used as is.

As an aside, I looked for a python3-minimal or equivalent, and couldn't
find one.  I take it there isn't one?

> 'git' could be just 'git-core'

Thanks.  Will fix locally, for Tumbleweed and Leap.

~Andrew

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Olaf Hering 2 months ago
Thu, 11 Jul 2024 23:39:42 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:

> configure has to cope with distros taking different naming decisions,
> and while I appreciate that packages may change over time even within a
> distro ...

configure does "pkgconfig --exists something" to probe for some API.
I'm sure noone will rename the pkgconfig files, "something.pc" in this case.

> ... this is just a differently-inconsistent list of names, with some
> very dubious aliases such as `form` and `menu`.

I should have truncated that list of names, 'pkgconfig(ncurses)' and
'pkgconfig(glib-2.0)' are relevant.

> What makes pkgconfig() any more stable than the main package name?

They are supposed to describe some API, and in some cases some functionality.
It does not matter much which package provides the API.

> Why do we want to be treating libraries different to other packages?

Because there is no other namespace to describe functionality.
It was implemented for pkg-config, it exists for some cmake and perl
modules.

> > python3-devel is "special", can be used as is.  
> As an aside, I looked for a python3-minimal or equivalent, and couldn't
> find one.  I take it there isn't one?

I think there are libraries compiled. There is indeed a 'pkgconfig(python3)',
my search just failed to find it. Otherwise, if there are really just .py
files in the tree, 'python3-base' is the string provided by 'python3N-base'.


Olaf
Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Andrew Cooper 2 months ago
On 11/07/2024 9:23 pm, Andrew Cooper wrote:
> Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
> consistency with all the other containers.
>
> Make it non-root, use heredocs for legibility, and use the zypper long names
> for the benefit of those wondering what was being referenced or duplicated.
>
> Trim the dependencies substantially.  Testing docs isn't very interesting and
> saves a lot of space (pandoc pulls in all of Haskel).  Other savings come from
> removing a huge pile of optional QEMU dependencies (QEMU just needs to build
> the Xen parts to be useful here, not have a full GUI environment).
>
> Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
> and nasm that I'm not aware of any reason to have had, even historically.
>
> This shaves 1G off the size of the container:
>
>   registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
>   registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB
>
> Furthermore, identify which components of the build use which dependencies,
> which will help managing them in the future.
>
> Thanks to Olaf Hering for dependency fixes that have been subsumed into this
> total overhaul.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Juergen Gross <jgross@suse.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Julien Grall <julien@xen.org>
> CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> CC: Shawn Anastasio <sanastasio@raptorengineering.com>
> CC: Olaf Hering <olaf@aepfle.de>
>
> This includes both of Olaf's recent changes, python3-packaging (now required
> for a rebuilt container), and findutils (not required yet; still gets pulled
> in transitively), but added for good measure.
>
> In terms of package delta, newly included are:
>
>   findutils            # As suggested
>   ocaml-findlib        # Replaces ocaml-findlib-devel
>   perl                 # Transitive previously
>   python3              # Transitive previously
>   python3-packaging    # As suggested
>
> while removed are:
>
>   bc                   # ?
>   cmake                # vtpm stubdom, optional
>   discount             # docs
>   ghostscript          # docs
>   glibc-devel          # Not really optional, given a toolchain
>   hostname             # Optional, not interesting
>   libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
>   libext2fs-devel      # Libfsimage, off by default, optional
>   libgnutls-devel      # Qemu ?
>   libjpeg62-devel      # Qemu ?
>   libnl3-devel         # Libxl COLO(?), option
>   libnuma-devel        # ?
>   libpng16-devel       # Qemu ?
>   libssh2-devel        # ?
>   libtasn1-devel       # ?
>   nasm                 # ?
>   ocaml-findlib-devel  # Swapped for non-devel
>   ocaml-ocamlbuild     # ?
>   ocaml-ocamldoc       # ?
>   pandoc               # docs
>   'pkgconfig(libpci)'  # Qemu
>   'pkgconfig(sdl)'     # Qemu
>   'pkgconfig(sdl2)'    # Qemu
>   systemd-devel        # Xen < 4.19, optional
>   transfig             # docs
>   valgrind-devel       # really only for developers
>   which                # Xen < 4.17
>
> I've tested this on staging-4.17 and staging, both of which build fine.
> staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
> hence it's safe to drop `which`.

For good measure, runs with this container:

  4.17: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322567620
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493465

and the expected failure:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499885

~Andrew

[PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Andrew Cooper 2 months ago
Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names
for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially.  Testing docs isn't very interesting and
saves a lot of space.  Other savings come from removing a huge pile of
optional QEMU dependencies (QEMU just needs to build the Xen parts to be
useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies,
which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this
total overhaul.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

This includes both of Olaf's recent changes, python3-packaging (now required
for a rebuilt container), and findutils (not required yet; still gets pulled
in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine.
staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
hence it's safe to drop `which`.

v2:
 * git -> git-core
 * Use 'pkgconfig()' references
---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile

diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
new file mode 100644
index 000000000000..61e840fc67da
--- /dev/null
+++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
@@ -0,0 +1,76 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        'pkgconfig(bzip2)'
+        'pkgconfig(libzstd)'
+        'pkgconfig(lzo2)'
+        'pkgconfig(liblzma)'
+        'pkgconfig(zlib)'
+        # libacpi
+        acpica
+        # libxl
+        'pkgconfig(uuid)'
+        'pkgconfig(yajl)'
+        # Header Check
+        gcc-c++
+        # xentop
+        'pkgconfig(ncurses)'
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        meson
+        ninja
+        'pkgconfig(glib-2.0)'
+        'pkgconfig(pixman-1)'
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
deleted file mode 100644
index 077fed22260c..000000000000
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ /dev/null
@@ -1,74 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/tumbleweed
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper dup -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        libzstd-devel \
-        lzo-devel \
-        make \
-        meson \
-        nasm \
-        ncurses-devel \
-        ninja \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index b4139414bc8e..fd6ca1270e4e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -668,25 +668,25 @@ opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 4d5669c5ad01..c4f013702122 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -43,7 +43,7 @@ case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used
-- 
2.39.2


Re: [PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Anthony PERARD 2 months ago
On Fri, Jul 12, 2024 at 12:14:26PM +0100, Andrew Cooper wrote:
> while removed are:
> 
>   bc                   # ?

I don't know why, but this one is also installed by osstest. The commit
adding it doesn't say why it was needed.

Also, looks like many QEMU containers are installing `bc`, but I can't
find a single use. Maybe historic.

>   libnl3-devel         # Libxl COLO(?), option

Yes.

>   libnuma-devel        # ?

There's some mention of it in QEMU.

>   libpng16-devel       # Qemu ?

It's added to a few QEMU containers, but I can't find any use.

>   libssh2-devel        # ?

It seems like it was used by QEMU at some point, but they switch to
'libssh'. Probably optional, like many QEMU deps.

>   libtasn1-devel       # ?

QEMU, and it seems to be only used in tests.

>   nasm                 # ?

'nasm' is used to build OVMF.


If we don't want to build OVMF in our GitLab CI, which I don't think
would be useful to do anyway (no dependency on xen.git), then:

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
Re: [PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container
Posted by Andrew Cooper 2 months ago
On 12/07/2024 1:33 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 12:14:26PM +0100, Andrew Cooper wrote:
>> while removed are:
>>
>>   bc                   # ?
> I don't know why, but this one is also installed by osstest. The commit
> adding it doesn't say why it was needed.
>
> Also, looks like many QEMU containers are installing `bc`, but I can't
> find a single use. Maybe historic.

Part of the reason why our containers are so massive is because
dependencies only ever accumulate...

But yes - this one is especially odd.
>>   nasm                 # ?
> 'nasm' is used to build OVMF.
>
>
> If we don't want to build OVMF in our GitLab CI, which I don't think
> would be useful to do anyway (no dependency on xen.git)

Oh, interesting.

I'd assumed (by virtue of the jobs succeeding) that we were building
OVMF by default, but we're clearly not.

./configure says:

  --enable-ovmf           Enable OVMF (default is DISABLED)

so it's really just a simple opt-in, which our build script does not.


In the longer term, we will want OVMF in Gitlab CI, but we'd want to
pull it in as a separate artefact and use
--with-system-ovmf=/path/to/artefact

So I agree we don't need to build OVMF inside this container.

> , then:
>
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

~Andrew