From nobody Wed Nov 27 08:44:42 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 155370665080385.36677898180164; Wed, 27 Mar 2019 10:10:50 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6FD2E72665; Wed, 27 Mar 2019 17:10:49 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4371E62660; Wed, 27 Mar 2019 17:10:49 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id E74AC3D387; Wed, 27 Mar 2019 17:10:48 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x2RHAjcX013344 for ; Wed, 27 Mar 2019 13:10:45 -0400 Received: by smtp.corp.redhat.com (Postfix) id E7C211001DD4; Wed, 27 Mar 2019 17:10:45 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-112-25.ams2.redhat.com [10.36.112.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id B02661001E84; Wed, 27 Mar 2019 17:10:44 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Wed, 27 Mar 2019 17:10:34 +0000 Message-Id: <20190327171038.24095-3-berrange@redhat.com> In-Reply-To: <20190327171038.24095-1-berrange@redhat.com> References: <20190327171038.24095-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v3 2/6] tests: add targets for building libvirt inside docker containers X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 27 Mar 2019 17:10:50 +0000 (UTC) The Travis CI system uses docker containers for its build environment. These are pre-built and hosted under quay.io/libvirt so that developers can use them for reproducing problems locally. Getting the right docker command syntax to use them, however, is not entirely easy. This patch addresses that usability issue by introducing some make targets. To run a simple build (aka 'make all') using the Fedora 28 container: make ci-build@fedora-28 To also run unit tests make ci-check@fedora-28 This is just syntax sugar for calling the previous command with a custom make target make ci-build@fedora-28 MAKE_ARGS=3D"check" To do a purely interactive build it is possible to request a shell make ci-shell@fedora-28 To do a mingw build, it is currently possible to use the fedora-rawhide and request a different configure script make ci-build@fedora-rawhide CONFIGURE=3Dmingw32-configure It is also possible to do cross compiled builds via the Debian containers make ci-build@debian-9-cross-s390x In all cases the GIT source tree is cloned locally into a 'ci-tree/src' sub-directory which is then exposed to the container at '/src'. It is setup to facilitate VPATH build so the initial working directory is '/src/build'. An in source tree build can be requested instead by passing an empty string VPATH=3D arg to make. The make rules are kept in a standalone file that is included into the main Makefile.am, so that it is possible to run them without having to invoke autotools first. It is neccessary to disable the gnulib submodule commit check because this fails due to the way we have manually cloned submodule repos as primary git repos with their own .git directory, instead of letting git treat them as submodules in the top level .git directory. make[1]: Entering directory '/src/build' fatal: Not a valid object name origin fatal: run_command returned non-zero status for .gnulib . maint.mk: found non-public submodule commit make: *** [/src/maint.mk:1448: public-submodule-commit] Error 1 Signed-off-by: Daniel P. Berrang=C3=A9 --- .gitignore | 1 + Makefile.am | 2 + Makefile.ci | 206 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 209 insertions(+) create mode 100644 Makefile.ci diff --git a/.gitignore b/.gitignore index 3223cf1b76..a67a842294 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,7 @@ /autom4te.cache /build-aux/* /build/ +/ci-tree/ /confdefs.h /config.cache /config.guess diff --git a/Makefile.am b/Makefile.am index 3c06e2619a..8add2f5d61 100644 --- a/Makefile.am +++ b/Makefile.am @@ -109,3 +109,5 @@ gen-AUTHORS: mv -f $(distdir)/AUTHORS-tmp $(distdir)/AUTHORS && \ rm -f all.list maint.list contrib.list; \ fi + +include Makefile.ci diff --git a/Makefile.ci b/Makefile.ci new file mode 100644 index 0000000000..5448fedd97 --- /dev/null +++ b/Makefile.ci @@ -0,0 +1,206 @@ +# -*- makefile -*- +# vim: filetype=3Dmake + +HERE =3D $(shell pwd) + +# Figure out name and path to this file. This isn't +# portable but we only care for modern GNU make +THIS_FILE =3D $(abspath $(lastword $(MAKEFILE_LIST))) + +# The directory holding content on the host that we will +# expose to the container. +SCRATCHDIR =3D $(HERE)/ci-tree + +TOP =3D $(shell git rev-parse --show-toplevel) + +# The directory holding the clo%%%%ne of the git repo that +# we will expose to the container +HOST_SRCDIR =3D $(SCRATCHDIR)/src + +# The directory holding the source inside the +# container. ie where we told docker to expose +# the $GIT/ci-tree directory from the host +CONT_SRCDIR =3D /src + +# Relative directory to perform the build in. This +# defaults to using a separate build dir, but can be +# set to empty string for an in-source tree build. +CONT_VPATH =3D build + +# The directory holding the build output inside the +# container. +CONT_BUILDDIR =3D $(CONT_SRCDIR)/$(CONT_VPATH) + +# Can be overridden with mingw{32,64}-configure if desired +CONFIGURE =3D $(CONT_SRCDIR)/configure + +# Default to using all possible CPUs +SMP =3D $(shell getconf _NPROCESSORS_ONLN) + +# Any extra arguments to pass to make +MAKE_ARGS =3D + +# Any extra arguments to pass to configure +CONFIGURE_ARGS =3D + +# Avoid pulling submodules over the network by locally +# cloning them +SUBMODULES =3D .gnulib src/keycodemapdb + +IMAGE_PREFIX =3D quay.io/libvirt/buildenv- +IMAGE_TAG =3D :master + +# We delete the virtual root after completion, set +# to 0 if you need to keep it around for debugging +CLEAN =3D 1 + +# We'll always freshly clone the virtual root each +# time in case it was not cleaned up before. Set +# to 1 if you want to try restarting a previously +# preserved env +REUSE =3D 0 + +# We need the container process to run with current host IDs +# so that it can access the passed in build directory +UID =3D $(shell id -u) +GID =3D $(shell id -g) + +# Docker doesn't require the IDs you run as to exist in +# the container's /etc/passwd & /etc/group files, but +# if they do not, then libvirt's 'make check' will fail +# many tests. +# +# We do not directly mount /etc/{passwd,group} as Docker +# is liable to mess with SELinux labelling which will +# then prevent the host accessing them. Copying them +# first is safer. +PWDB_MOUNTS =3D \ + --volume $(SCRATCHDIR)/group:/etc/group:ro,z \ + --volume $(SCRATCHDIR)/passwd:/etc/passwd:ro,z \ + $(NULL) + +# Docker containers can have very large ulimits +# for nofiles - as much as 1048576. This makes +# libvirt very slow at exec'ing programs. +ULIMIT_FILES =3D 1024 + +# Args to use when cloning a git repo. +# -c stop it complaining about checking out a random hash +# -q stop it displaying progress info for local clone +# --local ensure we don't actually copy files +GIT_ARGS =3D \ + -c advice.detachedHead=3Dfalse \ + -q \ + --local \ + $(NULL) + +# Args to use when running the docker env +# --rm stop inactive containers getting left behind +# --user we execute as the same user & group account +# as dev so that file ownership matches host +# instead of root:root +# --volume to pass in the cloned git repo & config +# --workdir to set cwd to vpath build location +# --ulimit lower files limit for performance reasons +# --interactive +# --tty Ensure we have ability to Ctrl-C the build +DOCKER_ARGS =3D \ + --rm \ + --user $(UID):$(GID) \ + --interactive \ + --tty \ + $(PWDB_MOUNTS) \ + --volume $(HOST_SRCDIR):$(CONT_SRCDIR):z \ + --workdir $(CONT_SRCDIR) \ + --ulimit nofile=3D$(ULIMIT_FILES):$(ULIMIT_FILES) \ + $(NULL) + +check-docker: + @echo -n "Checking if Docker is available and running..." && \ + docker version 1>/dev/null && echo "yes" + +prepare-tree: + @if test "$(REUSE)" !=3D "1" ; then \ + rm -rf ci-tree ; \ + fi + @if ! test -d ci-tree ; then \ + mkdir -p ci-tree/src; \ + cp /etc/passwd ci-tree; \ + cp /etc/group ci-tree; \ + echo "Cloning $(TOP) to $(HOST_SRCDIR)"; \ + git clone $(GIT_ARGS) $(TOP) $(HOST_SRCDIR) || exit 1; \ + for mod in $(SUBMODULES) ; \ + do \ + if test -d $(TOP)/$$mod ; \ + then \ + echo "Cloning $(TOP)/$$mod to $(HOST_SRCDIR)/$$mod"; \ + git clone $(GIT_ARGS) $(TOP)/$$mod $(HOST_SRCDIR)/$$mod || exit 1; \ + fi ; \ + done ; \ + else \ + test "$(CLEAN)" =3D "1" && rm -rf ci-tree || : ; \ + fi + +# $CONFIGURE_OPTS is a env that can optionally be set in the container, +# populated at build time from the Dockerfile. A typical use case would +# be to pass --host/--target args to trigger cross-compilation +# +# This can be augmented by make local args in $(CONFIGURE_ARGS) +ci-build@%: check-docker prepare-tree + docker run $(DOCKER_ARGS) $(IMAGE_PREFIX)$*$(IMAGE_TAG) \ + /bin/bash -c '\ + mkdir -p $(CONT_BUILDDIR) || exit 1 ; \ + cd $(CONT_BUILDDIR) ; \ + NOCONFIGURE=3D1 $(CONT_SRCDIR)/autogen.sh || exit 1 ; \ + $(CONFIGURE) $${CONFIGURE_OPTS} $(CONFIGURE_ARGS) ; \ + if test $$? !=3D 0 ; \ + then \ + test -f config.log && cat config.log ; \ + exit 1 ; \ + fi; \ + find -name test-suite.log -delete ; \ + make -j$(SMP) $(MAKE_ARGS) ; \ + if test $$? !=3D 0 ; then \ + LOGS=3D`find -name test-suite.log` ; \ + if test "$${LOGS}" !=3D "" ; then \ + echo "=3D=3D=3D LOG FILE(S) START =3D=3D=3D" ; \ + cat $${LOGS} ; \ + echo "=3D=3D=3D LOG FILE(S) END =3D=3D=3D" ; \ + fi ; \ + exit 1 ;\ + fi' + @test "$(CLEAN)" =3D "1" && rm -rf ci-tree || : + +ci-check@%: + $(MAKE) -f $(THIS_FILE) ci-build@$* MAKE_ARGS=3D"check gl_public_submodul= e_commit=3D" + +ci-shell@%: prepare-tree + docker run $(DOCKER_ARGS) $(IMAGE_PREFIX)$*$(IMAGE_TAG) /bin/bash + @test "$(CLEAN)" =3D "1" && rm -rf ci-tree || : + +ci-help: + @echo "Build libvirt inside docker containers used for CI" + @echo + @echo "Available targets:" + @echo + @echo " ci-build@\$$IMAGE - run a default 'make'" + @echo " ci-check@\$$IMAGE - run a 'make check'" + @echo " ci-shell@\$$IMAGE - run an interactive shell" + @echo + @echo "Available x86 container images:" + @echo + @echo " centos-7" + @echo " debian-8" + @echo " debian-9" + @echo " debian-sid" + @echo " fedora-28" + @echo " fedora-29" + @echo " fedora-rawhide" + @echo " ubuntu-16" + @echo " ubuntu-18" + @echo + @echo "Available make variables:" + @echo + @echo " CLEAN=3D0 - do not delete '$(SCRATCHDIR)' after completion" + @echo " REUSE=3D1 - re-use existing '$(SCRATCHDIR)' content" + @echo --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list