From nobody Wed Apr 24 00:22:35 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1557481986; cv=none; d=zoho.com; s=zohoarc; b=GGqEOOmxGHKDC9pMJmZ+Q7V7I9UL6igLw7xMFN3/C42X7UUuY+tSeFpGq9jF8OzBHtUmNUQRvPDpGL/P6+6b5bqvlYpMKGItmw8QJghzFCxx2s2VBLhsjgxAVh6VPlg/LKpwi9VkXUjMJAplzwrXwx1d4BGRqcRYWzGvXYqTJ5o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557481986; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To:ARC-Authentication-Results; bh=2C5V7EL6pZl1vMxgpFG8O2cjhRNZGoUgGsTGoucPdJA=; b=Ttw4RFMr4y4PfHNip+7eoX5P0Edv7MIn85vrMYd8quIdmoNfo0/B3gRvdDe+WH7AUUUckX3V0eHpqcSK2sFA/IlUBUamzMwR+2Kf6VLWF8bDqJoNNW8mIy5HqmOimpH6efD06FlgSIwGKtL7M5oRIGxATA30qHzs5EcS0OISJZY= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1557481986600313.1061112793327; Fri, 10 May 2019 02:53:06 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6DFE981F19; Fri, 10 May 2019 09:53:04 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1499D6660C; Fri, 10 May 2019 09:53:04 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 9263341F58; Fri, 10 May 2019 09:53:03 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x4A9r2aF024325 for ; Fri, 10 May 2019 05:53:02 -0400 Received: by smtp.corp.redhat.com (Postfix) id 2F77B5C29A; Fri, 10 May 2019 09:53:02 +0000 (UTC) Received: from caroline (unknown [10.43.2.67]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D1F046660C for ; Fri, 10 May 2019 09:52:59 +0000 (UTC) Received: from caroline.brq.redhat.com (caroline.usersys.redhat.com [127.0.0.1]) by caroline (Postfix) with ESMTP id 14DE31206CB for ; Fri, 10 May 2019 11:52:58 +0200 (CEST) From: Martin Kletzander To: libvir-list@redhat.com Date: Fri, 10 May 2019 11:52:54 +0200 Message-Id: <81bba3c91548b3678b4565d85522f4dc814793cb.1557481974.git.mkletzan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v2] Add support for podman in Makefile.ci X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Fri, 10 May 2019 09:53:05 +0000 (UTC) Content-Type: text/plain; charset="utf-8" This way more users can run our CI builds locally. Signed-off-by: Martin Kletzander Reviewed-by: Daniel P. Berrang=C3=A9 --- Makefile.ci | 91 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 70 insertions(+), 21 deletions(-) diff --git a/Makefile.ci b/Makefile.ci index 12a62167cc67..241c58d2d4e9 100644 --- a/Makefile.ci +++ b/Makefile.ci @@ -17,7 +17,7 @@ CI_GIT_ROOT =3D $(shell git rev-parse --show-toplevel) CI_HOST_SRCDIR =3D $(CI_SCRATCHDIR)/src =20 # The directory holding the source inside the -# container. ie where we told Docker to expose +# container, i.e. where we want to expose # the $(CI_HOST_SRCDIR) directory from the host CI_CONT_SRCDIR =3D /src =20 @@ -46,14 +46,13 @@ CI_CONFIGURE_ARGS =3D # cloning them CI_SUBMODULES =3D $(shell git submodule | awk '{ print $$2 }') =20 -# Location of the Docker images we're going to pull +# Location of the container images we're going to pull # Can be useful to overridde to use a locally built # image instead CI_IMAGE_PREFIX =3D quay.io/libvirt/buildenv- =20 -# Docker defaults to pulling the ':latest' tag but -# if the Docker repo above uses different conventions -# this can override it +# The default tag is ':latest' but if the container +# repo above uses different conventions this can override it CI_IMAGE_TAG =3D :master =20 # We delete the virtual root after completion, set @@ -71,15 +70,23 @@ CI_REUSE =3D 0 CI_UID =3D $(shell id -u) CI_GID =3D $(shell id -g) =20 -# Docker doesn't require the IDs you run as to exist in +CI_ENGINE =3D auto +# Container engine we are going to use, can be overridden per make +# invocation, if it is not we try podman and then default to docker. +ifeq ($(CI_ENGINE),auto) + override CI_ENGINE =3D $(shell podman version >/dev/null 2>&1 && echo pod= man || echo docker) +endif + +# IDs you run as do not need to exist in # the container's /etc/passwd & /etc/group files, but -# if they do not, then libvirt's 'make check' will fail +# if they do not, then libvirt's 'make check' will fail # many tests. -# + # We do not directly mount /etc/{passwd,group} as Docker # is liable to mess with SELinux labelling which will -# then prevent the host accessing them. Copying them -# first is safer. +# then prevent the host accessing them. And podman cannot +# relabel the files due to it running rootless. So +# copying them first is safer and error-prone. CI_PWDB_MOUNTS =3D \ --volume $(CI_SCRATCHDIR)/group:/etc/group:ro,z \ --volume $(CI_SCRATCHDIR)/passwd:/etc/passwd:ro,z \ @@ -90,6 +97,46 @@ CI_PWDB_MOUNTS =3D \ # libvirt very slow at exec'ing programs. CI_ULIMIT_FILES =3D 1024 =20 +ifeq ($(CI_ENGINE),podman) + # Podman cannot reuse host namespace when running non-root containers. U= ntil + # support for --keep-uid is added we can just create another mapping that= will + # do that for us. Beware, that in {uid,git}map=3Dcontainer_id:host_id:ra= nge, + # the host_id does actually refer to the uid in the first mapping where 0 + # (root) is mapped to the current user and rest is offset. + + # In order to set up this mapping, we need to keep all the user IDs to pr= event + # possible errors as some images might expect UIDs up to 90000 (looking a= t you + # fedora), so we don't want the overflowuid to be used for them. For map= ping + # all the other users properly ther eneeds to be some math done. Don't w= orry, + # it's just addition and subtraction. + + # 65536 ought to be enough (tm), but for really rare cases the maximums m= ight + # need to be higher, but that only happens when your /etc/sub{u,g}id allow + # users to have more IDs. Unless --keep-uid is supported, let's do this = in a + # way that should work for everyone. + CI_MAX_UID =3D $(shell sed -n "s/^$USER:[^:]\+://p" /etc/subuid) + CI_MAX_GID =3D $(shell sed -n "s/^$USER:[^:]\+://p" /etc/subgid) + ifeq ($(CI_MAX_UID),) + CI_MAX_UID =3D 65536 + endif + ifeq ($(CI_MAX_GID),) + CI_MAX_GID =3D 65536 + endif + CI_UID_OTHER =3D $(shell echo $$(($(CI_UID)+1))) + CI_GID_OTHER =3D $(shell echo $$(($(CI_GID)+1))) + CI_UID_OTHER_RANGE =3D $(shell echo $$(($(CI_MAX_UID)-$(CI_UID)))) + CI_GID_OTHER_RANGE =3D $(shell echo $$(($(CI_MAX_GID)-$(CI_GID)))) + + CI_PODMAN_ARGS =3D \ + --uidmap 0:1:$(CI_UID) \ + --uidmap $(CI_UID):0:1 \ + --uidmap $(CI_UID_OTHER):$(CI_UID_OTHER):$(CI_UID_OTHER_RANGE) \ + --gidmap 0:1:$(CI_GID) \ + --gidmap $(CI_GID):0:1 \ + --gidmap $(CI_GID_OTHER):$(CI_GID_OTHER):$(CI_GID_OTHER_RANGE) \ + $(NULL) +endif + # Args to use when cloning a git repo. # -c stop it complaining about checking out a random hash # -q stop it displaying progress info for local clone @@ -100,7 +147,7 @@ CI_GIT_ARGS =3D \ --local \ $(NULL) =20 -# Args to use when running the Docker env +# Args to use when running the container # --rm stop inactive containers getting left behind # --user we execute as the same user & group account # as dev so that file ownership matches host @@ -110,22 +157,23 @@ CI_GIT_ARGS =3D \ # --ulimit lower files limit for performance reasons # --interactive # --tty Ensure we have ability to Ctrl-C the build -CI_DOCKER_ARGS =3D \ +CI_ENGINE_ARGS =3D \ --rm \ --user $(CI_UID):$(CI_GID) \ --interactive \ --tty \ + $(CI_PODMAN_ARGS) \ $(CI_PWDB_MOUNTS) \ --volume $(CI_HOST_SRCDIR):$(CI_CONT_SRCDIR):z \ --workdir $(CI_CONT_SRCDIR) \ --ulimit nofile=3D$(CI_ULIMIT_FILES):$(CI_ULIMIT_FILES) \ $(NULL) =20 -ci-check-docker: - @echo -n "Checking if Docker is available and running..." && \ - docker version 1>/dev/null && echo "yes" +ci-check-engine: + @echo -n "Checking if $(CI_ENGINE) is available..." && \ + $(CI_ENGINE) version 1>/dev/null && echo "yes" =20 -ci-prepare-tree: ci-check-docker +ci-prepare-tree: ci-check-engine @test "$(CI_REUSE)" !=3D "1" && rm -rf $(CI_SCRATCHDIR) || : @if ! test -d $(CI_SCRATCHDIR) ; then \ mkdir -p $(CI_SCRATCHDIR); \ @@ -150,7 +198,7 @@ ci-prepare-tree: ci-check-docker # gl_public_submodule_commit=3D to disable gnulib's submodule check # which breaks due to way we clone the submodules ci-build@%: ci-prepare-tree - docker run $(CI_DOCKER_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) \ + $(CI_ENGINE) run $(CI_ENGINE_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) \ /bin/bash -c '\ mkdir -p $(CI_CONT_BUILDDIR) || exit 1 ; \ cd $(CI_CONT_BUILDDIR) ; \ @@ -179,11 +227,11 @@ ci-check@%: $(MAKE) -f $(CI_MAKEFILE) ci-build@$* CI_MAKE_ARGS=3D"check" =20 ci-shell@%: ci-prepare-tree - docker run $(CI_DOCKER_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) /bin/bash + $(CI_ENGINE) run $(CI_ENGINE_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) /b= in/bash @test "$(CI_CLEAN)" =3D "1" && rm -rf $(CI_SCRATCHDIR) || : =20 ci-help: - @echo "Build libvirt inside Docker containers used for CI" + @echo "Build libvirt inside containers used for CI" @echo @echo "Available targets:" @echo @@ -215,6 +263,7 @@ ci-help: @echo @echo "Available make variables:" @echo - @echo " CI_CLEAN=3D0 - do not delete '$(CI_SCRATCHDIR)' after completi= on" - @echo " CI_REUSE=3D1 - re-use existing '$(CI_SCRATCHDIR)' content" + @echo " CI_CLEAN=3D0 - do not delete '$(CI_SCRATCHDIR)' after comp= letion" + @echo " CI_REUSE=3D1 - re-use existing '$(CI_SCRATCHDIR)' content" + @echo " CI_ENGINE=3Dauto - container engine to use (podman, docker)" @echo --=20 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list