From nobody Thu Apr 25 12:21:41 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1557243976; cv=none; d=zoho.com; s=zohoarc; b=kDZtlJwszsVBrPvsKfjnThubJHXUZuqIN5XHYKM0xW1Sk0cWPx99Tq/RPE5tLuCeihz0Rlc3T/wKpu8shFmhwp4C+05qJU5/0esbWJe1r61/gdLsPrBkBDvavyJIq9cBdU/oSsW4Ueu8gVJ45O6yoV0FI73jwOoZVKDy/4Zs+lw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557243976; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=ga1jqByIgTotc08Blgom6We8WC/khqRtxQPdTG4hl1g=; b=Jv6VRnuGPd9RSAXfuQaNlIH/+31rVS0acpnzrtasY74g80wbP1sLmOkkVCLFeRm64v2HYBqN3r31pKrc4jCtgt4x5OWx3BV6sp+KCjXjilLdut9P8zw5cQez2hACfcSj7cVx9w6jdI4ia7tDRRtCzu3oH/vcnSoKM3GGstlYadw= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1557243976473581.8967671612152; Tue, 7 May 2019 08:46:16 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0880E3004139; Tue, 7 May 2019 15:46:08 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D99D23DA5; Tue, 7 May 2019 15:45:59 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 29B394EE05; Tue, 7 May 2019 15:45:46 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x47FjhGD028353 for ; Tue, 7 May 2019 11:45:43 -0400 Received: by smtp.corp.redhat.com (Postfix) id C8D05377F; Tue, 7 May 2019 15:45:43 +0000 (UTC) Received: from caroline (unknown [10.43.2.67]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9CE7C627D8 for ; Tue, 7 May 2019 15:45:39 +0000 (UTC) Received: from caroline.brq.redhat.com (caroline.usersys.redhat.com [127.0.0.1]) by caroline (Postfix) with ESMTP id CF373120744 for ; Tue, 7 May 2019 17:45:37 +0200 (CEST) From: Martin Kletzander To: libvir-list@redhat.com Date: Tue, 7 May 2019 17:45:30 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 1/2] Don't include Makefile.ci in Makefile.am X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Tue, 07 May 2019 15:46:09 +0000 (UTC) Content-Type: text/plain; charset="utf-8" The way it works now the Makefile needs to be both make valid and automake valid. That is fine for now, but if we want to use anything more advanced,= like conditionals, we cannot have it like that any more. So instead forward all ci-* rules to that file. Signed-off-by: Martin Kletzander Reviewed-by: Daniel P. Berrang=C3=A9 --- Makefile.am | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Makefile.am b/Makefile.am index 0d8bb733e6d2..442bae511828 100644 --- a/Makefile.am +++ b/Makefile.am @@ -35,6 +35,7 @@ EXTRA_DIST =3D \ libvirt-qemu.pc.in \ libvirt-lxc.pc.in \ libvirt-admin.pc.in \ + Makefile.ci \ Makefile.nonreentrant \ autogen.sh \ cfg.mk \ @@ -107,4 +108,5 @@ gen-AUTHORS: rm -f all.list maint.list contrib.list; \ fi =20 -include Makefile.ci +ci-%: + $(MAKE) -f Makefile.ci $@ --=20 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Thu Apr 25 12:21:41 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1557243980; cv=none; d=zoho.com; s=zohoarc; b=BEdLXq439wO1M1DDRg5+TZHedwjC7JniC/R1c6efjpFMKK7a4qM6uoS7YyyqkgRSCIcRrQGZmZCgxTc7pqfpJD+OCqqcCvD5k0gmMM6I4qkZGtbxFGSSfeCbCW+NIDMDoJp97pxMei113gDQEGh50jiei8VqhdqPzsjgXeTuvH8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557243980; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=EXlk9Zq4q/qjkcP1VcNDEPiegheaE+JWPuG3UqiFC6w=; b=N6QB4ugVkj54SsDOXsquJlnroqaBxV44SeF5CD83RvFEg/qOWUTnYkPJikMOvId4fscX6ISN5DCBkrOeEyj2jciz9+UTpSu/to/8OEawhbvhzvEKcWmdW3TuNrewc8pU4wNNRuCNWiM3L+jbgqaFZdyFz+SpyXPRF286oKM2phw= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 155724398012031.866173700055015; Tue, 7 May 2019 08:46:20 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6ADBDC0740EA; Tue, 7 May 2019 15:46:09 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 74BCB5C221; Tue, 7 May 2019 15:46:03 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id D5BD64EE06; Tue, 7 May 2019 15:45:46 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x47FjfFw028337 for ; Tue, 7 May 2019 11:45:41 -0400 Received: by smtp.corp.redhat.com (Postfix) id B193F5B0B7; Tue, 7 May 2019 15:45:41 +0000 (UTC) Received: from caroline (unknown [10.43.2.67]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5E7146031F for ; Tue, 7 May 2019 15:45:39 +0000 (UTC) Received: from caroline.brq.redhat.com (caroline.usersys.redhat.com [127.0.0.1]) by caroline (Postfix) with ESMTP id E3C1D120748 for ; Tue, 7 May 2019 17:45:37 +0200 (CEST) From: Martin Kletzander To: libvir-list@redhat.com Date: Tue, 7 May 2019 17:45:31 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 2/2] Add support for podman in Makefile.ci X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 07 May 2019 15:46:11 +0000 (UTC) Content-Type: text/plain; charset="utf-8" This way more users can run our CI builds locally. Signed-off-by: Martin Kletzander --- Makefile.ci | 125 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 93 insertions(+), 32 deletions(-) diff --git a/Makefile.ci b/Makefile.ci index 12a62167cc67..e2989ada313c 100644 --- a/Makefile.ci +++ b/Makefile.ci @@ -17,7 +17,7 @@ CI_GIT_ROOT =3D $(shell git rev-parse --show-toplevel) CI_HOST_SRCDIR =3D $(CI_SCRATCHDIR)/src =20 # The directory holding the source inside the -# container. ie where we told Docker to expose +# container. ie where we want to expose # the $(CI_HOST_SRCDIR) directory from the host CI_CONT_SRCDIR =3D /src =20 @@ -46,14 +46,13 @@ CI_CONFIGURE_ARGS =3D # cloning them CI_SUBMODULES =3D $(shell git submodule | awk '{ print $$2 }') =20 -# Location of the Docker images we're going to pull +# Location of the container images we're going to pull # Can be useful to overridde to use a locally built # image instead CI_IMAGE_PREFIX =3D quay.io/libvirt/buildenv- =20 -# Docker defaults to pulling the ':latest' tag but -# if the Docker repo above uses different conventions -# this can override it +# The default tag is ':latest' but if the container +# repo above uses different conventions this can override it CI_IMAGE_TAG =3D :master =20 # We delete the virtual root after completion, set @@ -71,24 +70,82 @@ CI_REUSE =3D 0 CI_UID =3D $(shell id -u) CI_GID =3D $(shell id -g) =20 -# Docker doesn't require the IDs you run as to exist in +# Container engine runtime we are going to use, can be overridden per make +# invocation, if it is not, we try podman and then default to docker. +ifeq ($(CI_CENGINE),) + CI_CENGINE =3D $(shell podman version >/dev/null && echo podman || echo d= ocker) +endif + +# IDs you run as do not need to exist in # the container's /etc/passwd & /etc/group files, but -# if they do not, then libvirt's 'make check' will fail +# if they do not, then libvirt's 'make check' will fail # many tests. -# -# We do not directly mount /etc/{passwd,group} as Docker -# is liable to mess with SELinux labelling which will -# then prevent the host accessing them. Copying them -# first is safer. -CI_PWDB_MOUNTS =3D \ - --volume $(CI_SCRATCHDIR)/group:/etc/group:ro,z \ - --volume $(CI_SCRATCHDIR)/passwd:/etc/passwd:ro,z \ - $(NULL) +ifeq ($(CI_CENGINE),podman) + CI_PWDB_MOUNTS =3D \ + --volume /etc/group:/etc/group:ro,z \ + --volume /etc/passwd:/etc/passwd:ro,z \ + $(NULL) +else + # We do not directly mount /etc/{passwd,group} as Docker + # is liable to mess with SELinux labelling which will + # then prevent the host accessing them. Copying them + # first is safer. + CI_PWDB_MOUNTS =3D \ + --volume $(CI_SCRATCHDIR)/group:/etc/group:ro,z \ + --volume $(CI_SCRATCHDIR)/passwd:/etc/passwd:ro,z \ + $(NULL) +endif + +ifeq ($(CI_CENGINE),docker) + # Docker containers can have very large ulimits + # for nofiles - as much as 1048576. This makes + # libvirt very slow at exec'ing programs. + CI_ULIMIT_FILES =3D 1024 +endif =20 -# Docker containers can have very large ulimits -# for nofiles - as much as 1048576. This makes -# libvirt very slow at exec'ing programs. -CI_ULIMIT_FILES =3D 1024 +ifeq ($(CI_CENGINE),podman) + # Podman cannot reuse host namespace when running non-root containers. U= ntil + # support for --keep-uid is added we can just create another mapping that= will + # do that for us. Beware, that in {uid,git}map=3Dcontainer_id:host_id:ra= nge, + # the host_id does actually refer to the uid in the first mapping where 0 + # (root) is mapped to the current user and rest is offset. + + # In order to set up this mapping, we need to keep all the user IDs to pr= event + # possible errors as some images might expect UIDs up to 90000 (looking a= t you + # fedora), so we don't want the overflowuid to be used for them. For map= ping + # all the other users properly ther eneeds to be some math done. Don't w= orry, + # it's just addition and subtraction. + + # 65536 ought to be enough (tm), but for really rare cases the maximums m= ight + # need to be higher, but that only happens when your /etc/sub{u,g}id allow + # users to have more IDs. Unless --keep-uid is supported, let's do this = in a + # way that should work for everyone. + CI_MAX_UID =3D $(shell sed -n "s/^$USER:[^:]\+://p" /etc/subuid) + CI_MAX_GID =3D $(shell sed -n "s/^$USER:[^:]\+://p" /etc/subgid) + ifeq ($(CI_MAX_UID),) + CI_MAX_UID =3D 65536 + endif + ifeq ($(CI_MAX_GID),) + CI_MAX_GID =3D 65536 + endif + CI_UID_OTHER =3D $(shell echo $$(($(CI_UID)+1))) + CI_GID_OTHER =3D $(shell echo $$(($(CI_GID)+1))) + CI_UID_OTHER_RANGE =3D $(shell echo $$(($(CI_MAX_UID)-$(CI_UID)))) + CI_GID_OTHER_RANGE =3D $(shell echo $$(($(CI_MAX_GID)-$(CI_GID)))) + + CI_PODMAN_ARGS =3D \ + --uidmap 0:1:$(CI_UID) \ + --uidmap $(CI_UID):0:1 \ + --uidmap $(CI_UID_OTHER):$(CI_UID_OTHER):$(CI_UID_OTHER_RANGE) \ + --gidmap 0:1:$(CI_GID) \ + --gidmap $(CI_GID):0:1 \ + --gidmap $(CI_GID_OTHER):$(CI_GID_OTHER):$(CI_GID_OTHER_RANGE) \ + $(NULL) +else + CI_DOCKER_ARGS =3D \ + --ulimit nofile=3D$(CI_ULIMIT_FILES):$(CI_ULIMIT_FILES) \ + $(NULL) +endif =20 # Args to use when cloning a git repo. # -c stop it complaining about checking out a random hash @@ -100,7 +157,7 @@ CI_GIT_ARGS =3D \ --local \ $(NULL) =20 -# Args to use when running the Docker env +# Args to use when running the container # --rm stop inactive containers getting left behind # --user we execute as the same user & group account # as dev so that file ownership matches host @@ -110,27 +167,30 @@ CI_GIT_ARGS =3D \ # --ulimit lower files limit for performance reasons # --interactive # --tty Ensure we have ability to Ctrl-C the build -CI_DOCKER_ARGS =3D \ +CI_CENGINE_ARGS =3D \ --rm \ --user $(CI_UID):$(CI_GID) \ --interactive \ --tty \ + $(CI_PODMAN_ARGS) \ + $(CI_DOCKER_ARGS) \ $(CI_PWDB_MOUNTS) \ --volume $(CI_HOST_SRCDIR):$(CI_CONT_SRCDIR):z \ --workdir $(CI_CONT_SRCDIR) \ - --ulimit nofile=3D$(CI_ULIMIT_FILES):$(CI_ULIMIT_FILES) \ $(NULL) =20 -ci-check-docker: - @echo -n "Checking if Docker is available and running..." && \ - docker version 1>/dev/null && echo "yes" +ci-check-cengine: + @echo -n "Checking if $(CI_CENGINE) is available..." && \ + $(CI_CENGINE) version 1>/dev/null && echo "yes" =20 -ci-prepare-tree: ci-check-docker +ci-prepare-tree: ci-check-cengine @test "$(CI_REUSE)" !=3D "1" && rm -rf $(CI_SCRATCHDIR) || : @if ! test -d $(CI_SCRATCHDIR) ; then \ mkdir -p $(CI_SCRATCHDIR); \ - cp /etc/passwd $(CI_SCRATCHDIR); \ - cp /etc/group $(CI_SCRATCHDIR); \ + if test "$(CI_CENGINE)" !=3D "podman"; then \ + cp /etc/passwd $(CI_SCRATCHDIR); \ + cp /etc/group $(CI_SCRATCHDIR); \ + fi; \ echo "Cloning $(CI_GIT_ROOT) to $(CI_HOST_SRCDIR)"; \ git clone $(CI_GIT_ARGS) $(CI_GIT_ROOT) $(CI_HOST_SRCDIR) || exit 1; \ for mod in $(CI_SUBMODULES) ; \ @@ -150,7 +210,7 @@ ci-prepare-tree: ci-check-docker # gl_public_submodule_commit=3D to disable gnulib's submodule check # which breaks due to way we clone the submodules ci-build@%: ci-prepare-tree - docker run $(CI_DOCKER_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) \ + $(CI_CENGINE) run $(CI_CENGINE_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) \ /bin/bash -c '\ mkdir -p $(CI_CONT_BUILDDIR) || exit 1 ; \ cd $(CI_CONT_BUILDDIR) ; \ @@ -179,11 +239,11 @@ ci-check@%: $(MAKE) -f $(CI_MAKEFILE) ci-build@$* CI_MAKE_ARGS=3D"check" =20 ci-shell@%: ci-prepare-tree - docker run $(CI_DOCKER_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) /bin/bash + $(CI_CENGINE) run $(CI_CENGINE_ARGS) $(CI_IMAGE_PREFIX)$*$(CI_IMAGE_TAG) = /bin/bash @test "$(CI_CLEAN)" =3D "1" && rm -rf $(CI_SCRATCHDIR) || : =20 ci-help: - @echo "Build libvirt inside Docker containers used for CI" + @echo "Build libvirt inside containers used for CI" @echo @echo "Available targets:" @echo @@ -215,6 +275,7 @@ ci-help: @echo @echo "Available make variables:" @echo + @echo " CI_CENGINE=3Dengine - container engine to use (podman (default= ) or docker)" @echo " CI_CLEAN=3D0 - do not delete '$(CI_SCRATCHDIR)' after completi= on" @echo " CI_REUSE=3D1 - re-use existing '$(CI_SCRATCHDIR)' content" @echo --=20 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list