From nobody Wed May 8 19:07:52 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1551864874087264.9732983659684; Wed, 6 Mar 2019 01:34:34 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7B91113AB0; Wed, 6 Mar 2019 09:34:32 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5141360E3D; Wed, 6 Mar 2019 09:34:32 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 0CE143FA49; Wed, 6 Mar 2019 09:34:32 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x269YUtL013994 for ; Wed, 6 Mar 2019 04:34:30 -0500 Received: by smtp.corp.redhat.com (Postfix) id 99E321001938; Wed, 6 Mar 2019 09:34:30 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-112-60.ams2.redhat.com [10.36.112.60]) by smtp.corp.redhat.com (Postfix) with ESMTP id 53D2A1001DD6; Wed, 6 Mar 2019 09:34:29 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Wed, 6 Mar 2019 09:34:23 +0000 Message-Id: <20190306093424.22469-2-berrange@redhat.com> In-Reply-To: <20190306093424.22469-1-berrange@redhat.com> References: <20190306093424.22469-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v2 1/2] tests: add targets for building libvirt inside docker containers X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 06 Mar 2019 09:34:33 +0000 (UTC) The Travis CI system uses docker containers for its build environment. These are pre-built and hosted under quay.io/libvirt so that developers can use them for reproducing problems locally. Getting the right docker command syntax to use them, however, is not entirely easy. This patch addresses that usability issue by introducing some make targets. To run a simple build (aka 'make all') using the Fedora 28 containr: make cibuild-fedora-28 To also run unit tests make cicheck-fedora-28 This is just syntax sugar for calling the previous command with a custom make target make cibuild-fedora-28 MAKE_ARGS=3D"check" To do a purely interactive build it is possible to request a shell make cishell-fedora-28 To do a mingw build, it is currently possible to use the fedora-rawhide and request a different configure script make cibuild-fedora-rawhide CONFIGURE=3Dmingw32-configure In all cases the GIT source tree is cloned locally into a 'citree/src' sub-directory which is then exposed to the container at '/build'. It is setup to facilitate VPATH build so the initial working directory is '/build/vpath'. An in source tree build can be requested instead by passing VPATHDIR=3D SRCDIR=3D. args to make. The make rules are kept in a standalone file that is included into the main Makefile.am, so that it is possible to run them without having to invoke autotools first. Signed-off-by: Daniel P. Berrang=C3=A9 --- .gitignore | 1 + Makefile.am | 2 + tests/Makefile.ci.inc | 174 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 177 insertions(+) create mode 100644 tests/Makefile.ci.inc diff --git a/.gitignore b/.gitignore index 3303eed411..a30882d72b 100644 --- a/.gitignore +++ b/.gitignore @@ -46,6 +46,7 @@ /autom4te.cache /build-aux/* /build/ +/citree/ /confdefs.h /config.cache /config.guess diff --git a/Makefile.am b/Makefile.am index 709064c6a6..0c8ab13ebc 100644 --- a/Makefile.am +++ b/Makefile.am @@ -123,3 +123,5 @@ gen-AUTHORS: mv -f $(distdir)/AUTHORS-tmp $(distdir)/AUTHORS && \ rm -f all.list maint.list contrib.list; \ fi + +include tests/Makefile.ci.inc diff --git a/tests/Makefile.ci.inc b/tests/Makefile.ci.inc new file mode 100644 index 0000000000..7d521326cc --- /dev/null +++ b/tests/Makefile.ci.inc @@ -0,0 +1,174 @@ +# -*- makefile -*- + +HERE =3D $(shell pwd) +TOP =3D $(shell git rev-parse --show-toplevel) + +# Run in a separate build directory. Set to empty +# for a in-source tree build, but note SRCDIR must +# also be set to a corresponding relative path +VPATHDIR =3D vpath +SRCDIR =3D .. + +# Can be overridden with mingw{32,64}-configure if desired +CONFIGURE =3D $(SRCDIR)/configure + +# Default to using all possible CPUs +SMP =3D $(shell getconf _NPROCESSORS_ONLN) + +# Any extra arguments to pass to make +MAKE_ARGS =3D + +# Any extra arguments to pass to configure +CONFIGURE_ARGS =3D + +# Avoid pulling submodules over the network by locally +# cloning them +SUBMODULES =3D .gnulib src/keycodemapdb + +IMAGE_PREFIX =3D quay.io/libvirt/buildenv +IMAGE_TAG =3D :master + +# We delete the virtual root after completion, set +# to 0 if you need to keep it around for debugging +CLEAN =3D 1 + +# We'll always freshly clone the virtual root each +# time in case it was not cleaned up before. Set +# to 0 if you want to try restarting a previously +# preserved env +RECLONE =3D 1 + +# We need the container process to run with current host IDs +# so that it can access the passed in build directory +UID =3D $(shell id -u) +GID =3D $(shell id -g) + +# Docker doesn't require the IDs you run as to exist in +# the container's /etc/passwd & /etc/group files, but +# if they do not, then libvirt's 'make check' will fail +# many tests +PWDB_MOUNTS =3D \ + --volume $(HERE)/citree/group:/etc/group:ro,z \ + --volume $(HERE)/citree/passwd:/etc/passwd:ro,z + +# Docker containers can have very large ulimits +# for nofiles - as much as 1048576. This makes +# libvirt very slow at exec'ing programs. +ULIMIT_FILES =3D 1024 + +# Args to use when cloning a git repo +GIT_ARGS =3D \ + -c advice.detachedHead=3Dfalse \ + -q \ + --local \ + $(NULL) + +# Args to use when running the docker env +# --rm stop inactive containers getting left behind +# --user we execute as the same user & group account +# as dev so that file ownership matches host +# instead of root:root +# --volume to pass in the cloned git repo & config +# --workdir to set cwd to vpath build location +# --ulimit lower files limit for performance reasons +# --interactive +# --tty Ensure we have ability to Ctrl-C the build +DOCKER_ARGS =3D \ + --rm \ + --user $(UID):$(GID) \ + --interactive \ + --tty \ + $(PWDB_MOUNTS) \ + --volume $(HERE)/citree/src:/build:z \ + --workdir /build/$(VPATHDIR) \ + --ulimit nofile=3D$(ULIMIT_FILES):$(ULIMIT_FILES) \ + $(NULL) + +checkdocker: + @echo -n "Checking if docker is available and running..." && \ + docker version 1>/dev/null && echo "yes" + +preparetree: + @if test "$(RECLONE)" =3D "1" ; then \ + rm -rf citree ; \ + fi + @if ! test -d citree ; then \ + mkdir -p citree/src; \ + cp /etc/passwd citree; \ + cp /etc/group citree; \ + echo "Cloning $(TOP) to $(HERE)/citree/root"; \ + git clone $(GIT_ARGS) $(TOP) $(HERE)/citree/src || exit 1; \ + for mod in $(SUBMODULES) ; \ + do \ + if test -d $(TOP)/$$mod ; \ + then \ + echo "Cloning $(TOP)/$$mod to $(HERE)/citree/$$mod"; \ + git clone $(GIT_ARGS) $(TOP)/$$mod $(HERE)/citree/src/$$mod || exit 1;= \ + fi ; \ + done ; \ + mkdir -p citree/src/$(VPATHDIR) ; \ + else \ + test "$(CLEAN)" =3D "1" && rm -rf citree || : ; \ + fi + +# $CONFIGURE_OPTS is a env that can optionally be set in the container, +# populated at build time from the Dockerfile. A typical use case would +# be to pass --host/--target args to trigger cross-compilation +# +# This can be augmented by make local args in $(CONFIGURE_ARGS) +cibuild-%: checkdocker preparetree + docker run $(DOCKER_ARGS) $(IMAGE_PREFIX)-$*$(IMAGE_TAG) \ + /bin/bash -c '\ + NOCONFIGURE=3D1 $(SRCDIR)/autogen.sh || exit 1 ; \ + $(CONFIGURE) $${CONFIGURE_OPTS}; $(CONFIGURE_ARGS) \ + if test $$? !=3D 0 ; \ + then \ + test -f config.log && cat config.log ; \ + exit 1 ; \ + fi; \ + find -name test-suite.log -delete ; \ + make -j $(SMP) $(MAKE_ARGS) ; \ + if test $$? !=3D 0 ; then \ + LOGS=3D`find -name test-suite.log` ; \ + if test "$${LOGS}" !=3D "" ; then \ + echo "=3D=3D=3D LOG FILE(S) START =3D=3D=3D" ; \ + cat $${LOGS} ; \ + echo "=3D=3D=3D LOG FILE(S) END =3D=3D=3D" ; \ + fi ; \ + exit 1 ;\ + fi' + @test "$(CLEAN)" =3D "1" && rm -rf citree || : + +cicheck-%: + $(MAKE) cibuild-$* MAKE_ARGS=3D"check gl_public_submodule_commit=3D" + +cishell-%: preparetree + docker run $(DOCKER_ARGS) $(IMAGE_PREFIX)-$*$(IMAGE_TAG) /bin/bash + @test "$(CLEAN)" =3D "1" && rm -rf citree || : + +cihelp: + @echo "Build libvirt inside docker containers used for CI" + @echo + @echo "Available targets:" + @echo + @echo " cibuild-\$$IMAGE - run a default 'make'" + @echo " cicheck-\$$IMAGE - run a 'make check'" + @echo " cishell-\$$IMAGE - run an interactive shell" + @echo + @echo "Available x86 container images:" + @echo + @echo " centos-7" + @echo " debian-8" + @echo " debian-9" + @echo " debian-sid" + @echo " fedora-28" + @echo " fedora-29" + @echo " fedora-rawhide" + @echo " ubuntu-16" + @echo " ubuntu-18" + @echo + @echo "Available make variables:" + @echo + @echo " CLEAN=3D0 - do not delete '$(HERE)/citree' after completion" + @echo " RECLONE=3D0 - re-use existing '$(HERE)/citree' content" + @echo --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Wed May 8 19:07:52 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1551864876115878.7986884185506; Wed, 6 Mar 2019 01:34:36 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 197B388305; Wed, 6 Mar 2019 09:34:34 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E773217B40; Wed, 6 Mar 2019 09:34:33 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id AA53E1819AFB; Wed, 6 Mar 2019 09:34:33 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x269YW8n014002 for ; Wed, 6 Mar 2019 04:34:32 -0500 Received: by smtp.corp.redhat.com (Postfix) id 52D261001DD6; Wed, 6 Mar 2019 09:34:32 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-112-60.ams2.redhat.com [10.36.112.60]) by smtp.corp.redhat.com (Postfix) with ESMTP id 29AA81001938; Wed, 6 Mar 2019 09:34:30 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Wed, 6 Mar 2019 09:34:24 +0000 Message-Id: <20190306093424.22469-3-berrange@redhat.com> In-Reply-To: <20190306093424.22469-1-berrange@redhat.com> References: <20190306093424.22469-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v2 2/2] travis: convert ubuntu, centos & mingw builds to use new make rules X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Wed, 06 Mar 2019 09:34:34 +0000 (UTC) Change the travis configuration to invoke the new cibuild-$IMAGE target instead of directly running docker. This guarantees that when a developer runs cibuild-$IMAGE locally, the container build setup is identical to that used in Travis, with exception of the host kernel and docker version. Signed-off-by: Daniel P. Berrang=C3=A9 --- .travis.yml | 48 ++++++++++-------------------------------------- 1 file changed, 10 insertions(+), 38 deletions(-) diff --git a/.travis.yml b/.travis.yml index 55ba340a34..c093dbf550 100644 --- a/.travis.yml +++ b/.travis.yml @@ -11,26 +11,30 @@ matrix: - docker env: - IMAGE=3D"ubuntu-18" - - DISTCHECK_CONFIGURE_FLAGS=3D"--with-init-script=3Dsystemd" - - DOCKER_CMD=3D"$LINUX_CMD" + - MAKE_ARGS=3D"syntax-check distcheck DISTCHECK_CONFIGURE_FLAGS=3D= --with-init-script-systemd" + script: + - make -f tests/Makefile.ci.inc cibuild-$IMAGE MAKE_ARGS=3D"$MAKE= _ARGS" - services: - docker env: - IMAGE=3D"centos-7" - - DISTCHECK_CONFIGURE_FLAGS=3D"--with-init-script=3Dupstart" - - DOCKER_CMD=3D"$LINUX_CMD" + - MAKE_ARGS=3D"syntax-check distcheck DISTCHECK_CONFIGURE_FLAGS=3D= --with-init-script-systemd" + script: + - make -f tests/Makefile.ci.inc cibuild-$IMAGE MAKE_ARGS=3D"$MAKE= _ARGS" - services: - docker env: - IMAGE=3D"fedora-rawhide" - MINGW=3D"mingw32" - - DOCKER_CMD=3D"$MINGW_CMD" + script: + - make -f tests/Makefile.ci.inc cibuild-$IMAGE CONFIGURE=3D$MINGW-= configure MAKE_ARGS=3D"$MAKE_ARGS" - services: - docker env: - IMAGE=3D"fedora-rawhide" - MINGW=3D"mingw64" - - DOCKER_CMD=3D"$MINGW_CMD" + script: + - make -f tests/Makefile.ci.inc cibuild-$IMAGE CONFIGURE=3D$MINGW-= configure MAKE_ARGS=3D"$MAKE_ARGS" - compiler: clang language: c os: osx @@ -39,44 +43,12 @@ matrix: script: /bin/sh -xc "$MACOS_CMD" =20 -script: - - docker run - -v $(pwd):/build - -w /build - -e VIR_TEST_DEBUG=3D"$VIR_TEST_DEBUG" - -e MINGW=3D"$MINGW" - -e DISTCHECK_CONFIGURE_FLAGS=3D"$DISTCHECK_CONFIGURE_FLAGS" - "quay.io/libvirt/buildenv-$IMAGE:master" - /bin/sh -xc "$DOCKER_CMD" - git: submodules: true =20 env: global: - VIR_TEST_DEBUG=3D1 - - LINUX_CMD=3D" - ./autogen.sh && - make -j3 syntax-check && - make -j3 distcheck DISTCHECK_CONFIGURE_FLAGS=3D\"\$DISTCHECK_CONFI= GURE_FLAGS\" || - ( - echo '=3D=3D=3D LOG FILE(S) START =3D=3D=3D'; - find -name test-suite.log | xargs cat; - echo '=3D=3D=3D LOG FILE(S) END =3D=3D=3D'; - exit 1 - ) - " - - MINGW_CMD=3D" - NOCONFIGURE=3D1 ./autogen.sh && - \$MINGW-configure && - make -j3 || - ( - echo '=3D=3D=3D LOG FILE(S) START =3D=3D=3D'; - find -name test-suite.log | xargs cat; - echo '=3D=3D=3D LOG FILE(S) END =3D=3D=3D'; - exit 1 - ) - " # We can't run 'distcheck' or 'syntax-check' because they fail on # macOS, but doing 'install' and 'dist' gives us some useful coverage - MACOS_CMD=3D" --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list