1. Patchew
  2. QEMU
  3. View series

[RFC PATCH] tests/docker: merge debian-native with debian-amd64

Alex Bennée posted 1 patch 1 year ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20231108165602.3865524-1-alex.bennee@linaro.org
Maintainers: "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>
.gitlab-ci.d/buildtest.yml                    | 12 ++---
.gitlab-ci.d/containers.yml                   |  2 +-
tests/docker/Makefile.include                 |  3 --
tests/docker/dockerfiles/debian-native.docker | 54 -------------------
.../{debian-amd64.docker => debian.docker}    |  7 ++-
tests/lcitool/refresh                         |  9 ++--
6 files changed, 18 insertions(+), 69 deletions(-)
delete mode 100644 tests/docker/dockerfiles/debian-native.docker
rename tests/docker/dockerfiles/{debian-amd64.docker => debian.docker} (96%)
[RFC PATCH] tests/docker: merge debian-native with debian-amd64
Posted by Alex Bennée 1 year ago 
debian-native isn't really needed and suffers from the problem of
tracking a distros dependencies rather than the projects. With a
little surgery we can make the debian-amd64 container architecture
neutral and allow people to use it to build a native QEMU.

Rename it so it follows the same non-arch pattern of the other distro
containers.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Anders Roxell <anders.roxell@linaro.org>
---
 .gitlab-ci.d/buildtest.yml                    | 12 ++---
 .gitlab-ci.d/containers.yml                   |  2 +-
 tests/docker/Makefile.include                 |  3 --
 tests/docker/dockerfiles/debian-native.docker | 54 -------------------
 .../{debian-amd64.docker => debian.docker}    |  7 ++-
 tests/lcitool/refresh                         |  9 ++--
 6 files changed, 18 insertions(+), 69 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-native.docker
 rename tests/docker/dockerfiles/{debian-amd64.docker => debian.docker} (96%)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index da72f7c690..7f9af83b10 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -70,7 +70,7 @@ build-system-debian:
   needs:
     job: amd64-debian-container
   variables:
-    IMAGE: debian-amd64
+    IMAGE: debian
     CONFIGURE_ARGS: --with-coroutine=sigaltstack
     TARGETS: arm-softmmu i386-softmmu riscv64-softmmu sh4eb-softmmu
       sparc-softmmu xtensa-softmmu
@@ -82,7 +82,7 @@ check-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
-    IMAGE: debian-amd64
+    IMAGE: debian
     MAKE_CHECK_ARGS: check
 
 avocado-system-debian:
@@ -91,7 +91,7 @@ avocado-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
-    IMAGE: debian-amd64
+    IMAGE: debian
     MAKE_CHECK_ARGS: check-avocado
     AVOCADO_TAGS: arch:arm arch:i386 arch:riscv64 arch:sh4 arch:sparc arch:xtensa
 
@@ -101,7 +101,7 @@ crash-test-debian:
     - job: build-system-debian
       artifacts: true
   variables:
-    IMAGE: debian-amd64
+    IMAGE: debian
   script:
     - cd build
     - make NINJA=":" check-venv
@@ -589,7 +589,7 @@ build-tools-and-docs-debian:
     # when running on 'master' we use pre-existing container
     optional: true
   variables:
-    IMAGE: debian-amd64
+    IMAGE: debian
     MAKE_CHECK_ARGS: check-unit ctags TAGS cscope
     CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
     QEMU_JOB_PUBLISH: 1
@@ -609,7 +609,7 @@ build-tools-and-docs-debian:
 # of what topic branch they're currently using
 pages:
   extends: .base_job_template
-  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG
+  image: $CI_REGISTRY_IMAGE/qemu/debian:$QEMU_CI_CONTAINER_TAG
   stage: test
   needs:
     - job: build-tools-and-docs-debian
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 8637a13d86..ae79d4c58b 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -11,7 +11,7 @@ amd64-debian-container:
   extends: .container_job_template
   stage: containers
   variables:
-    NAME: debian-amd64
+    NAME: debian
 
 amd64-ubuntu2204-container:
   extends: .container_job_template
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index cd4688bf07..5ba5b50ab9 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -88,9 +88,6 @@ DOCKER_PARTIAL_IMAGES += debian-s390x-cross
 DOCKER_PARTIAL_IMAGES += fedora
 endif
 
-# The native build should never use the registry
-docker-image-debian-native: DOCKER_REGISTRY=
-
 # alpine has no adduser
 docker-image-alpine: NOUSER=1
 
diff --git a/tests/docker/dockerfiles/debian-native.docker b/tests/docker/dockerfiles/debian-native.docker
deleted file mode 100644
index abac7d7cd7..0000000000
--- a/tests/docker/dockerfiles/debian-native.docker
+++ /dev/null
@@ -1,54 +0,0 @@
-#
-# Docker Debian Native
-#
-# This is intended to build QEMU on native host systems. Debian is
-# chosen due to the broadest range on supported host systems for QEMU.
-#
-# This docker target is based on the docker.io Debian Bullseye base
-# image rather than QEMU's base because we would otherwise confuse the
-# build grabbing stuff from the registry built for other
-# architectures.
-#
-FROM docker.io/library/debian:bullseye-slim
-MAINTAINER Alex Bennée <alex.bennee@linaro.org>
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
-
-# Install common build utilities
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
-
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt build-dep -yy --arch-only qemu
-
-RUN apt update && \
-    DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends \
-        cscope \
-        genisoimage \
-        exuberant-ctags \
-        global \
-        libbz2-dev \
-        liblzo2-dev \
-        libgcrypt20-dev \
-        libfdt-dev \
-        librdmacm-dev \
-        libsasl2-dev \
-        libsnappy-dev \
-        libvte-dev \
-        netcat-openbsd \
-        ninja-build \
-        openssh-client \
-        python3-numpy \
-        python3-opencv \
-        python3-venv
-
-ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS
-ENV DEF_TARGET_LIST "none"
-# As a final step configure the user (if env is defined)
-ARG USER
-ARG UID
-RUN if [ "${USER}" ]; then \
-  id ${USER} 2>/dev/null || useradd -u ${UID} -U ${USER}; fi
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian.docker
similarity index 96%
rename from tests/docker/dockerfiles/debian-amd64.docker
rename to tests/docker/dockerfiles/debian.docker
index 9b50fb2f63..b5e642d5b6 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -155,10 +155,13 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
   apt install -y --no-install-recommends \
   cscope\
   global\
-  linux-headers-amd64
+  linux-headers-generic
 RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
 RUN cd /usr/src/netmap && git checkout v11.3
-RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
+RUN cd /usr/src/netmap/LINUX && \
+  ./configure --no-drivers --no-apps \
+  --kernel-dir=$(ls -d /usr/src/linux-headers-*-$(dpkg --print-architecture)) \
+  && make install
 ENV QEMU_CONFIGURE_OPTS --enable-netmap
 # As a final step configure the user (if env is defined)
 ARG USER
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 10c54a377c..13250f40c4 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -99,10 +99,13 @@ debian12_extras = [
     "  apt install -y --no-install-recommends \\\n",
     "  cscope\\\n",
     "  global\\\n",
-    "  linux-headers-amd64\n",
+    "  linux-headers-generic\n",
     "RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap\n",
     "RUN cd /usr/src/netmap && git checkout v11.3\n",
-    "RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install\n",
+    "RUN cd /usr/src/netmap/LINUX && \\\n",
+    "  ./configure --no-drivers --no-apps \\\n",
+    "  --kernel-dir=$(ls -d /usr/src/linux-headers-*-$(dpkg --print-architecture)) \\\n",
+    "  && make install\n",
     "ENV QEMU_CONFIGURE_OPTS --enable-netmap\n"
 ]
 
@@ -123,7 +126,7 @@ try:
     #
     generate_dockerfile("alpine", "alpine-318")
     generate_dockerfile("centos8", "centos-stream-8")
-    generate_dockerfile("debian-amd64", "debian-12",
+    generate_dockerfile("debian", "debian-12",
                         trailer="".join(debian12_extras))
     generate_dockerfile("fedora", "fedora-38")
     generate_dockerfile("opensuse-leap", "opensuse-leap-15")
-- 
2.39.2
Re: [RFC PATCH] tests/docker: merge debian-native with debian-amd64
Posted by Anders Roxell 1 year ago 
On Wed, 8 Nov 2023 at 17:56, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> debian-native isn't really needed and suffers from the problem of
> tracking a distros dependencies rather than the projects. With a
> little surgery we can make the debian-amd64 container architecture
> neutral and allow people to use it to build a native QEMU.
>
> Rename it so it follows the same non-arch pattern of the other distro
> containers.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Anders Roxell <anders.roxell@linaro.org>

I tried it on my arm machine and it worked as expected.
And this should work for me

Tested-by: Anders Roxell <anders.roxell@linaro.org>

Cheers,
Anders

> ---
>  .gitlab-ci.d/buildtest.yml                    | 12 ++---
>  .gitlab-ci.d/containers.yml                   |  2 +-
>  tests/docker/Makefile.include                 |  3 --
>  tests/docker/dockerfiles/debian-native.docker | 54 -------------------
>  .../{debian-amd64.docker => debian.docker}    |  7 ++-
>  tests/lcitool/refresh                         |  9 ++--
>  6 files changed, 18 insertions(+), 69 deletions(-)
>  delete mode 100644 tests/docker/dockerfiles/debian-native.docker
>  rename tests/docker/dockerfiles/{debian-amd64.docker => debian.docker} (96%)
>
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index da72f7c690..7f9af83b10 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -70,7 +70,7 @@ build-system-debian:
>    needs:
>      job: amd64-debian-container
>    variables:
> -    IMAGE: debian-amd64
> +    IMAGE: debian
>      CONFIGURE_ARGS: --with-coroutine=sigaltstack
>      TARGETS: arm-softmmu i386-softmmu riscv64-softmmu sh4eb-softmmu
>        sparc-softmmu xtensa-softmmu
> @@ -82,7 +82,7 @@ check-system-debian:
>      - job: build-system-debian
>        artifacts: true
>    variables:
> -    IMAGE: debian-amd64
> +    IMAGE: debian
>      MAKE_CHECK_ARGS: check
>
>  avocado-system-debian:
> @@ -91,7 +91,7 @@ avocado-system-debian:
>      - job: build-system-debian
>        artifacts: true
>    variables:
> -    IMAGE: debian-amd64
> +    IMAGE: debian
>      MAKE_CHECK_ARGS: check-avocado
>      AVOCADO_TAGS: arch:arm arch:i386 arch:riscv64 arch:sh4 arch:sparc arch:xtensa
>
> @@ -101,7 +101,7 @@ crash-test-debian:
>      - job: build-system-debian
>        artifacts: true
>    variables:
> -    IMAGE: debian-amd64
> +    IMAGE: debian
>    script:
>      - cd build
>      - make NINJA=":" check-venv
> @@ -589,7 +589,7 @@ build-tools-and-docs-debian:
>      # when running on 'master' we use pre-existing container
>      optional: true
>    variables:
> -    IMAGE: debian-amd64
> +    IMAGE: debian
>      MAKE_CHECK_ARGS: check-unit ctags TAGS cscope
>      CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
>      QEMU_JOB_PUBLISH: 1
> @@ -609,7 +609,7 @@ build-tools-and-docs-debian:
>  # of what topic branch they're currently using
>  pages:
>    extends: .base_job_template
> -  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG
> +  image: $CI_REGISTRY_IMAGE/qemu/debian:$QEMU_CI_CONTAINER_TAG
>    stage: test
>    needs:
>      - job: build-tools-and-docs-debian
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> index 8637a13d86..ae79d4c58b 100644
> --- a/.gitlab-ci.d/containers.yml
> +++ b/.gitlab-ci.d/containers.yml
> @@ -11,7 +11,7 @@ amd64-debian-container:
>    extends: .container_job_template
>    stage: containers
>    variables:
> -    NAME: debian-amd64
> +    NAME: debian
>
>  amd64-ubuntu2204-container:
>    extends: .container_job_template
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index cd4688bf07..5ba5b50ab9 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -88,9 +88,6 @@ DOCKER_PARTIAL_IMAGES += debian-s390x-cross
>  DOCKER_PARTIAL_IMAGES += fedora
>  endif
>
> -# The native build should never use the registry
> -docker-image-debian-native: DOCKER_REGISTRY=
> -
>  # alpine has no adduser
>  docker-image-alpine: NOUSER=1
>
> diff --git a/tests/docker/dockerfiles/debian-native.docker b/tests/docker/dockerfiles/debian-native.docker
> deleted file mode 100644
> index abac7d7cd7..0000000000
> --- a/tests/docker/dockerfiles/debian-native.docker
> +++ /dev/null
> @@ -1,54 +0,0 @@
> -#
> -# Docker Debian Native
> -#
> -# This is intended to build QEMU on native host systems. Debian is
> -# chosen due to the broadest range on supported host systems for QEMU.
> -#
> -# This docker target is based on the docker.io Debian Bullseye base
> -# image rather than QEMU's base because we would otherwise confuse the
> -# build grabbing stuff from the registry built for other
> -# architectures.
> -#
> -FROM docker.io/library/debian:bullseye-slim
> -MAINTAINER Alex Bennée <alex.bennee@linaro.org>
> -
> -# Duplicate deb line as deb-src
> -RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
> -
> -# Install common build utilities
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
> -
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive eatmydata \
> -    apt build-dep -yy --arch-only qemu
> -
> -RUN apt update && \
> -    DEBIAN_FRONTEND=noninteractive eatmydata \
> -    apt install -y --no-install-recommends \
> -        cscope \
> -        genisoimage \
> -        exuberant-ctags \
> -        global \
> -        libbz2-dev \
> -        liblzo2-dev \
> -        libgcrypt20-dev \
> -        libfdt-dev \
> -        librdmacm-dev \
> -        libsasl2-dev \
> -        libsnappy-dev \
> -        libvte-dev \
> -        netcat-openbsd \
> -        ninja-build \
> -        openssh-client \
> -        python3-numpy \
> -        python3-opencv \
> -        python3-venv
> -
> -ENV QEMU_CONFIGURE_OPTS $QEMU_CONFIGURE_OPTS
> -ENV DEF_TARGET_LIST "none"
> -# As a final step configure the user (if env is defined)
> -ARG USER
> -ARG UID
> -RUN if [ "${USER}" ]; then \
> -  id ${USER} 2>/dev/null || useradd -u ${UID} -U ${USER}; fi
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian.docker
> similarity index 96%
> rename from tests/docker/dockerfiles/debian-amd64.docker
> rename to tests/docker/dockerfiles/debian.docker
> index 9b50fb2f63..b5e642d5b6 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -155,10 +155,13 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>    apt install -y --no-install-recommends \
>    cscope\
>    global\
> -  linux-headers-amd64
> +  linux-headers-generic
>  RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
>  RUN cd /usr/src/netmap && git checkout v11.3
> -RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
> +RUN cd /usr/src/netmap/LINUX && \
> +  ./configure --no-drivers --no-apps \
> +  --kernel-dir=$(ls -d /usr/src/linux-headers-*-$(dpkg --print-architecture)) \
> +  && make install
>  ENV QEMU_CONFIGURE_OPTS --enable-netmap
>  # As a final step configure the user (if env is defined)
>  ARG USER
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index 10c54a377c..13250f40c4 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -99,10 +99,13 @@ debian12_extras = [
>      "  apt install -y --no-install-recommends \\\n",
>      "  cscope\\\n",
>      "  global\\\n",
> -    "  linux-headers-amd64\n",
> +    "  linux-headers-generic\n",
>      "RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap\n",
>      "RUN cd /usr/src/netmap && git checkout v11.3\n",
> -    "RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install\n",
> +    "RUN cd /usr/src/netmap/LINUX && \\\n",
> +    "  ./configure --no-drivers --no-apps \\\n",
> +    "  --kernel-dir=$(ls -d /usr/src/linux-headers-*-$(dpkg --print-architecture)) \\\n",
> +    "  && make install\n",
>      "ENV QEMU_CONFIGURE_OPTS --enable-netmap\n"
>  ]
>
> @@ -123,7 +126,7 @@ try:
>      #
>      generate_dockerfile("alpine", "alpine-318")
>      generate_dockerfile("centos8", "centos-stream-8")
> -    generate_dockerfile("debian-amd64", "debian-12",
> +    generate_dockerfile("debian", "debian-12",
>                          trailer="".join(debian12_extras))
>      generate_dockerfile("fedora", "fedora-38")
>      generate_dockerfile("opensuse-leap", "opensuse-leap-15")
> --
> 2.39.2
>
Re: [RFC PATCH] tests/docker: merge debian-native with debian-amd64
Posted by Philippe Mathieu-Daudé 1 year ago 
On 8/11/23 17:56, Alex Bennée wrote:
> debian-native isn't really needed and suffers from the problem of
> tracking a distros dependencies rather than the projects. With a
> little surgery we can make the debian-amd64 container architecture
> neutral and allow people to use it to build a native QEMU.
> 
> Rename it so it follows the same non-arch pattern of the other distro
> containers.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Anders Roxell <anders.roxell@linaro.org>
> ---
>   .gitlab-ci.d/buildtest.yml                    | 12 ++---
>   .gitlab-ci.d/containers.yml                   |  2 +-
>   tests/docker/Makefile.include                 |  3 --
>   tests/docker/dockerfiles/debian-native.docker | 54 -------------------
>   .../{debian-amd64.docker => debian.docker}    |  7 ++-
>   tests/lcitool/refresh                         |  9 ++--
>   6 files changed, 18 insertions(+), 69 deletions(-)
>   delete mode 100644 tests/docker/dockerfiles/debian-native.docker
>   rename tests/docker/dockerfiles/{debian-amd64.docker => debian.docker} (96%)

Good idea. Why RFC?

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Re: [RFC PATCH] tests/docker: merge debian-native with debian-amd64
Posted by Alex Bennée 1 year ago 
Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> On 8/11/23 17:56, Alex Bennée wrote:
>> debian-native isn't really needed and suffers from the problem of
>> tracking a distros dependencies rather than the projects. With a
>> little surgery we can make the debian-amd64 container architecture
>> neutral and allow people to use it to build a native QEMU.
>> Rename it so it follows the same non-arch pattern of the other
>> distro
>> containers.
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Anders Roxell <anders.roxell@linaro.org>
>> ---
>>   .gitlab-ci.d/buildtest.yml                    | 12 ++---
>>   .gitlab-ci.d/containers.yml                   |  2 +-
>>   tests/docker/Makefile.include                 |  3 --
>>   tests/docker/dockerfiles/debian-native.docker | 54 -------------------
>>   .../{debian-amd64.docker => debian.docker}    |  7 ++-
>>   tests/lcitool/refresh                         |  9 ++--
>>   6 files changed, 18 insertions(+), 69 deletions(-)
>>   delete mode 100644 tests/docker/dockerfiles/debian-native.docker
>>   rename tests/docker/dockerfiles/{debian-amd64.docker => debian.docker} (96%)
>
> Good idea. Why RFC?

Well mostly to see if it is usable for Ander's as a generic build env
for QEMU builds for tuxbuild images.

>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Patchew 2.1-devel-b67e4c2

© 2016 - 2024 Red Hat, Inc.