[PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

Andrew Cooper posted 12 patches 2 months, 1 week ago
[PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64
Posted by Andrew Cooper 2 months ago
Rework the container to be non-root, use heredocs for legibility, and use use
apt-get --no-install-recommends to keep the size down.

Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced
by Xen 4.16.  Therefore, drop the dependencies too.

This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    16.04-x86_64           700MB
  registry.gitlab.com/xen-project/xen/ubuntu    xenial                 1.21GB

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

Runs with the new container:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548805
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548805
---
 .../build/ubuntu/16.04-x86_64.dockerfile      | 65 +++++++++++++++++++
 automation/build/ubuntu/xenial.dockerfile     | 46 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 74 insertions(+), 55 deletions(-)
 create mode 100644 automation/build/ubuntu/16.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/xenial.dockerfile

diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
new file mode 100644
index 000000000000..11e878050b83
--- /dev/null
+++ b/automation/build/ubuntu/16.04-x86_64.dockerfile
@@ -0,0 +1,65 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:16.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/ubuntu/xenial.dockerfile b/automation/build/ubuntu/xenial.dockerfile
deleted file mode 100644
index aa8f560286b6..000000000000
--- a/automation/build/ubuntu/xenial.dockerfile
+++ /dev/null
@@ -1,46 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 ubuntu:16.04
-LABEL maintainer.name="The Xen Project " \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 2216392a8414..a2e6815f30b3 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -588,25 +588,25 @@ fedora-gcc-debug:
   variables:
     CONTAINER: fedora:29
 
-ubuntu-xenial-clang:
+ubuntu-16.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-clang-debug:
+ubuntu-16.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-gcc:
+ubuntu-16.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-gcc-debug:
+ubuntu-16.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
 ubuntu-bionic-clang:
   extends: .clang-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 0ff53b6fe4e1..5fc41c36a27c 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -52,9 +52,9 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
+    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
 
     *) guess_container ;;
 esac
-- 
2.39.2


Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64
Posted by Anthony PERARD 2 months ago
On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
> diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
> new file mode 100644
> +        python3-setuptools

Another thing,

I've just notice that "python3-setuptools" was originally only introduced
to focal container on purpose, by:
bbc72a7877d8 ("automation: Add python3's setuptools to some containers")

But now this patch series is introducing it to xenial and bionic. It
might be worth mentioning, or removing the deps.

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64
Posted by Andrew Cooper 2 months ago
On 12/07/2024 3:29 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
>> diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
>> new file mode 100644
>> +        python3-setuptools
> Another thing,
>
> I've just notice that "python3-setuptools" was originally only introduced
> to focal container on purpose, by:
> bbc72a7877d8 ("automation: Add python3's setuptools to some containers")
>
> But now this patch series is introducing it to xenial and bionic. It
> might be worth mentioning, or removing the deps.
>

Oh, good point.  I'll take this back out of Xenial/Bionic.

Stretch/Trusty have already been removed.

Thanks,

~Andrew

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64
Posted by Anthony PERARD 2 months ago
On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
> diff --git a/automation/scripts/containerize b/automation/scripts/containerize
> index 0ff53b6fe4e1..5fc41c36a27c 100755
> --- a/automation/scripts/containerize
> +++ b/automation/scripts/containerize
> @@ -52,9 +52,9 @@ case "_${CONTAINER}" in
>      _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
>      _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
>      _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
> -    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
>      _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
>      _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
> +    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
>  
>      *) guess_container ;;

I've been wondering why I couldn't apply the patch (and the next one),
and there's these 2 lines of extra context ;-)

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech
Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64
Posted by Andrew Cooper 2 months ago
On 12/07/2024 3:02 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
>> diff --git a/automation/scripts/containerize b/automation/scripts/containerize
>> index 0ff53b6fe4e1..5fc41c36a27c 100755
>> --- a/automation/scripts/containerize
>> +++ b/automation/scripts/containerize
>> @@ -52,9 +52,9 @@ case "_${CONTAINER}" in
>>      _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
>>      _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
>>      _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
>> -    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
>>      _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
>>      _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
>> +    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
>>  
>>      *) guess_container ;;
> I've been wondering why I couldn't apply the patch (and the next one),
> and there's these 2 lines of extra context ;-)

Yeah, sorry.  I found these when rebasing back into order.

https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen.git;a=shortlog;h=refs/heads/auto-4.19
(and its simlarly-named variants) are the branches I'm doing all of this
work on.

~Andrew

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64
Posted by Anthony PERARD 2 months ago
On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
> Rework the container to be non-root, use heredocs for legibility, and use use
> apt-get --no-install-recommends to keep the size down.
> 
> Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced
> by Xen 4.16.  Therefore, drop the dependencies too.
> 
> This saves ~500M:
> 
>   registry.gitlab.com/xen-project/xen/ubuntu    16.04-x86_64           700MB
>   registry.gitlab.com/xen-project/xen/ubuntu    xenial                 1.21GB
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech