From nobody Mon May 6 22:17:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 154867718825275.48901258964145; Mon, 28 Jan 2019 04:06:28 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5F35AC05D3E4; Mon, 28 Jan 2019 12:06:25 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2074E5C22D; Mon, 28 Jan 2019 12:06:25 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id BA2503F609; Mon, 28 Jan 2019 12:06:24 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x0SC6Hv2009929 for ; Mon, 28 Jan 2019 07:06:17 -0500 Received: by smtp.corp.redhat.com (Postfix) id 51FFB611D4; Mon, 28 Jan 2019 12:06:17 +0000 (UTC) Received: from dhcp-16-175.lcy.redhat.com (unknown [10.42.16.175]) by smtp.corp.redhat.com (Postfix) with ESMTP id BC8F42E04A; Mon, 28 Jan 2019 12:06:16 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Mon, 28 Jan 2019 12:06:08 +0000 Message-Id: <20190128120610.24759-2-berrange@redhat.com> In-Reply-To: <20190128120610.24759-1-berrange@redhat.com> References: <20190128120610.24759-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 1/3] tests: add targets for building libvirt inside docker containers X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 28 Jan 2019 12:06:26 +0000 (UTC) The Travis CI system uses docker containers for its build environment. These are pre-built and hosted under quay.io/libvirt so that developers can use them for reproducing problems locally. Getting the right docker command syntax to use them, however, is not entirely easy. This patch addresses that usability issue by introducing some make targets make cibuild-fedora-28 will run a build of the local git repo using the buildenv-fedora-28 docker container, running the default make target in the container. make cibuild-fedora-28 will run "make check" while for more advanced debugging developers can get a direct shell and run any make command manually make cishell-fedora-28 In all cases the GIT source tree is cloned locally into a 'citree/src' sub-directory which is then exposed to the container at '/build'. It is setup to facilitate VPATH build so the initial working directory is '/build/vpath'. The make rules are kept in a standalone file that is included into the main Makefile.am, so that it is possible to run them without having to invoke autotools first. Signed-off-by: Daniel P. Berrang=C3=A9 --- .gitignore | 1 + Makefile.am | 2 + tests/Makefile.ci.inc | 123 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+) create mode 100644 tests/Makefile.ci.inc diff --git a/.gitignore b/.gitignore index df0ac8e3d4..a7499bbbb6 100644 --- a/.gitignore +++ b/.gitignore @@ -46,6 +46,7 @@ /autom4te.cache /build-aux/* /build/ +/citree/ /confdefs.h /config.cache /config.guess diff --git a/Makefile.am b/Makefile.am index 709064c6a6..0c8ab13ebc 100644 --- a/Makefile.am +++ b/Makefile.am @@ -123,3 +123,5 @@ gen-AUTHORS: mv -f $(distdir)/AUTHORS-tmp $(distdir)/AUTHORS && \ rm -f all.list maint.list contrib.list; \ fi + +include tests/Makefile.ci.inc diff --git a/tests/Makefile.ci.inc b/tests/Makefile.ci.inc new file mode 100644 index 0000000000..9deee91fd0 --- /dev/null +++ b/tests/Makefile.ci.inc @@ -0,0 +1,123 @@ +# -*- makefile -*- + +HERE =3D $(shell pwd) +TOP =3D $(shell git rev-parse --show-toplevel) + +# Default to using all possible CPUs +SMP =3D $(getconf _NPROCESSORS_ONLN) + +# Any extra arguments to pass to make +MAKE_ARGS =3D + +# Avoid pulling submodules over the network by locally +# cloning them +SUBMODULES =3D .gnulib src/keycodemapdb + +IMAGE_PREFIX =3D quay.io/libvirt/buildenv +IMAGE_TAG =3D :master + +# We delete the virtual root after completion, set +# to 0 if you need to keep it around for debugging +CLEAN =3D 1 + +# We'll always freshly clone the virtual root each +# time in case it was not cleaned up before. Set +# to 0 if you want to try restarting a previously +# preserved env +RECLONE =3D 1 + +SCRIPT =3D + +# We need the container process to run with current host IDs +# so that it can access the passed in build directory +UID =3D $(shell id -u) +GID =3D $(shell id -g) + +# Docker doesn't require the IDs you run as to exist in +# the container's /etc/passwd & /etc/group files, but +# if they do not, then libvirt's 'make check' will fail +# many tests +PWDB_MOUNTS =3D \ + --volume $(HERE)/citree/group:/etc/group:ro,z \ + --volume $(HERE)/citree/passwd:/etc/passwd:ro,z + +# Docker containers can have very large ulimits +# for nofiles - as much as 1048576. This makes +# libvirt very slow at exec'ing programs. +ULIMIT_FILES =3D 1024 + +DOCKER_ARGS =3D \ + --rm \ + --interactive \ + --tty \ + --user $(UID):$(GID) \ + $(PWDB_MOUNTS) \ + --volume $(HERE)/citree/src:/build:z \ + --workdir /build/vpath \ + --ulimit nofile=3D$(ULIMIT_FILES):$(ULIMIT_FILES) \ + $(NULL) + +preparetree: + @if test "$(RECLONE)" =3D "1" ; then \ + rm -rf citree ; \ + fi + @if ! test -d citree ; then \ + mkdir -p citree/src; \ + cp /etc/passwd citree; \ + cp /etc/group citree; \ + echo "Cloning $(TOP) to $(HERE)/citree/root"; \ + git clone -q --local $(TOP) $(HERE)/citree/src; \ + for mod in $(SUBMODULES) ; \ + do \ + echo "Cloning $(TOP)/$$mod to $(HERE)/citree/$$mod"; \ + git clone -q --local $(TOP)/$$mod $(HERE)/citree/s= rc/$$mod ; \ + done ; \ + mkdir -p citree/src/vpath ; \ + else \ + test "$(CLEAN)" =3D "1" && rm -rf citree || : ; \ + fi + +# $LIBVIRT_CONFIGURE_OPTS is a env that can optionally be +# set in the container, populated at build time from the +# Dockerfile. A typical use case would be to pass --host/--target +# args to trigger cross-compilation +cibuild-%: preparetree + docker run $(DOCKER_ARGS) $(IMAGE_PREFIX)-$*$(IMAGE_TAG) \ + /bin/bash -c '\ + echo "../autogen.sh $${LIBVIRT_CONFIGURE_OPTS}" && \ + ../autogen.sh $${LIBVIRT_CONFIGURE_OPTS} && make -j $(SMP)= $(MAKE_ARGS)' + @test "$(CLEAN)" =3D "1" && rm -rf citree || : + +cicheck-%: + $(MAKE) cibuild-$* MAKE_ARGS=3D"check gl_public_submodule_commit=3D" + +cishell-%: preparetree + docker run $(DOCKER_ARGS) $(IMAGE_PREFIX)-$*$(IMAGE_TAG) /bin/bash + @test "$(CLEAN)" =3D "1" && rm -rf citree || : + +cihelp: + @echo "Build libvirt inside docker containers used for CI" + @echo + @echo "Available targets:" + @echo + @echo " cibuild-\$$IMAGE - run a default 'make'" + @echo " cicheck-\$$IMAGE - run a 'make check'" + @echo " cishell-\$$IMAGE - run an interactive shell" + @echo + @echo "Available container images:" + @echo + @echo " centos-7" + @echo " debian-8" + @echo " debian-9" + @echo " debian-sid" + @echo " fedora-28" + @echo " fedora-29" + @echo " fedora-rawhide" + @echo " ubuntu-16" + @echo " ubuntu-18" + @echo + @echo "Available make variables:" + @echo + @echo " CLEAN=3D0 - do not delete '$(HERE)/citree' after completion" + @echo " RECLONE=3D0 - re-use existing '$(HERE)/citree' content" + @echo --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Mon May 6 22:17:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1548677187707428.3973161064472; Mon, 28 Jan 2019 04:06:27 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 05A2CC05D3E1; Mon, 28 Jan 2019 12:06:25 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B081B103BAAD; Mon, 28 Jan 2019 12:06:24 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 613EB3F607; Mon, 28 Jan 2019 12:06:24 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x0SC6ISL009940 for ; Mon, 28 Jan 2019 07:06:18 -0500 Received: by smtp.corp.redhat.com (Postfix) id 274C42E03B; Mon, 28 Jan 2019 12:06:18 +0000 (UTC) Received: from dhcp-16-175.lcy.redhat.com (unknown [10.42.16.175]) by smtp.corp.redhat.com (Postfix) with ESMTP id 91E85611D4; Mon, 28 Jan 2019 12:06:17 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Mon, 28 Jan 2019 12:06:09 +0000 Message-Id: <20190128120610.24759-3-berrange@redhat.com> In-Reply-To: <20190128120610.24759-1-berrange@redhat.com> References: <20190128120610.24759-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 2/3] tests: add cross compiler images to CI test help output X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 28 Jan 2019 12:06:25 +0000 (UTC) Signed-off-by: Daniel P. Berrang=C3=A9 --- tests/Makefile.ci.inc | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/Makefile.ci.inc b/tests/Makefile.ci.inc index 9deee91fd0..37f1455d03 100644 --- a/tests/Makefile.ci.inc +++ b/tests/Makefile.ci.inc @@ -104,7 +104,7 @@ cihelp: @echo " cicheck-\$$IMAGE - run a 'make check'" @echo " cishell-\$$IMAGE - run an interactive shell" @echo - @echo "Available container images:" + @echo "Available x86 native container images:" @echo @echo " centos-7" @echo " debian-8" @@ -116,6 +116,17 @@ cihelp: @echo " ubuntu-16" @echo " ubuntu-18" @echo + @echo "Available cross-compiler container images:" + @echo + @echo " cross-arm64" + @echo " cross-armel" + @echo " cross-armhf" + @echo " cross-mips64el" + @echo " cross-mips" + @echo " cross-mipsel" + @echo " cross-ppc64el" + @echo " cross-s390x" + @echo @echo "Available make variables:" @echo @echo " CLEAN=3D0 - do not delete '$(HERE)/citree' after completion" --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Mon May 6 22:17:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 154867718455775.01343773891858; Mon, 28 Jan 2019 04:06:24 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2279E8667A; Mon, 28 Jan 2019 12:06:22 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DF4A75D969; Mon, 28 Jan 2019 12:06:21 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 97ADE3F603; Mon, 28 Jan 2019 12:06:21 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x0SC6InU009952 for ; Mon, 28 Jan 2019 07:06:19 -0500 Received: by smtp.corp.redhat.com (Postfix) id ED5ED226FC; Mon, 28 Jan 2019 12:06:18 +0000 (UTC) Received: from dhcp-16-175.lcy.redhat.com (unknown [10.42.16.175]) by smtp.corp.redhat.com (Postfix) with ESMTP id 655B2648B6; Mon, 28 Jan 2019 12:06:18 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Mon, 28 Jan 2019 12:06:10 +0000 Message-Id: <20190128120610.24759-4-berrange@redhat.com> In-Reply-To: <20190128120610.24759-1-berrange@redhat.com> References: <20190128120610.24759-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 3/3] tests: perform cross compiler builds on GitLab CI X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 28 Jan 2019 12:06:22 +0000 (UTC) GitLab CI provides some shared build runners that use Docker containers. This resource can usefully run cross-compiled builds since all other CI build testing is currently x86 only, and Travis CI is already very busy testing native builds. Signed-off-by: Daniel P. Berrang=C3=A9 --- .gitlab-ci.yml | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000000..081d9d8f91 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,38 @@ +.job_template: &job_definition + script: + - mkdir vpath + - cd vpath + - ../autogen.sh $LIBVIRT_CONFIGURE_OPTS + - make -j $(getconf _NPROCESSORS_ONLN) + +crossarm64: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-arm64 + +crossarmel: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-armel + +crossarmhf: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-armhf + +crossmips64el: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-mips64el + +crossmips: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-mips + +crossmipsel: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-mipsel + +crossppc64el: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-ppc64el + +crosss390x: + <<: *job_definition + image: quay.io/libvirt/buildenv-cross-s390x --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list