From nobody Tue May 7 21:20:03 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) client-ip=207.211.31.81; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1585664925; cv=none; d=zohomail.com; s=zohoarc; b=Z+Vy8vb0UlE2INenjHsMeY+nN4v7F6KgCBYvI4NpP4ci1s7Yz3H+HEARkBAF5/P2PvFJ4pLr7BirtO1L2mJEyXqiDMjXgHAgzjM3pSyUn6ypiroBgmrcIplIBnIM6xumk3Blq4TwONeWxgGVlYbrts6S1PFHkHs/9gNhoEMWlvc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1585664925; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=NfA6MOyMYP+Wb2VuP5SOu6F5jfPDyEngapPxUsoidE0=; b=ahBMNutFRdsWqGMbLeiSuEnUSE2zvWTAA2mWGnz5+FYQMUYTXylMo+JEbLIPqQxjW3DfVNtlbGnsFV93yJyZBI1q1lahiANllx0qESO21MCS4r+Hx/YcM1TjJhrNXdCIX6R3KGu2fx0GW/VIDb+GLC/dp+wEZc09PLkVAujKGW8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by mx.zohomail.com with SMTPS id 1585664925123811.2621729056968; Tue, 31 Mar 2020 07:28:45 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-197-lzXQqp4TOYWbZtP-8jU48Q-1; Tue, 31 Mar 2020 10:28:41 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 97AEADB22; Tue, 31 Mar 2020 14:28:33 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A403A1001DC2; Tue, 31 Mar 2020 14:28:32 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 1E23586BF0; Tue, 31 Mar 2020 14:28:27 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 02VESPQr013412 for ; Tue, 31 Mar 2020 10:28:26 -0400 Received: by smtp.corp.redhat.com (Postfix) id EE0D51001DD8; Tue, 31 Mar 2020 14:28:25 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.36.110.48]) by smtp.corp.redhat.com (Postfix) with ESMTP id 11E941001B2D; Tue, 31 Mar 2020 14:28:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1585664923; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=NfA6MOyMYP+Wb2VuP5SOu6F5jfPDyEngapPxUsoidE0=; b=BcqwxaOXpD6oEbB1tu4+PNxVqj2D1My2wLlJRZN9J4GD+nuPGaJW3ohgTkAkdmvyoV+Cp6 pWVTDlv0sML8CkiDUU1lB7FAz4aU0bFXJdMUBfC1NjgZQqK19abuAo71fu4Y4RJceo0NGs 1/R9P/iPSLSjZ5D3oAgdDgRhBnPR7Hk= X-MC-Unique: lzXQqp4TOYWbZtP-8jU48Q-1 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Subject: [libvirt-jenkins-ci PATCH] guests: allow for container image inheritance Date: Tue, 31 Mar 2020 15:28:20 +0100 Message-Id: <20200331142820.377510-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Currently when creating a Dockerfile for a container, we include the full set of base packages, along with the packages for the project itself. If building a Perl binding, this would require us to install the base package, libvirt packages and Perl packages. With the use of the "--inherit libvirt-fedora-30" arg, it is possible to have a dockerfile that only adds the Perl packages, getting everything else from the parent image. For example, a full Dockerfile for libvirt-go would thus be: FROM libvirt-centos-7:latest RUN yum install -y \ golang && \ yum autoremove -y && \ yum clean all -y Note there is no need to set ENV either, as that's inherited from the parent container. Signed-off-by: Daniel P. Berrang=C3=A9 --- guests/lcitool | 108 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 98 insertions(+), 10 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 689a8cf..b3afb6a 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -396,6 +396,12 @@ class Application: help=3D"target architecture for cross compiler", ) =20 + def add_inherit_arg(parser): + parser.add_argument( + "-i", "--inherit", + help=3D"inherit from an intermediate container image", + ) + def add_wait_arg(parser): parser.add_argument( "-w", "--wait", @@ -442,6 +448,7 @@ class Application: add_hosts_arg(dockerfileparser) add_projects_arg(dockerfileparser) add_cross_arch_arg(dockerfileparser) + add_inherit_arg(dockerfileparser) =20 def _execute_playbook(self, playbook, hosts, projects, git_revision): base =3D Util.get_base() @@ -644,11 +651,11 @@ class Application: with open(keyfile, "r") as r: return r.read().rstrip() =20 - def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, proj= ects, cross_arch): + def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, proj= ects, cross_arch, needbase): if facts["package_format"] =3D=3D "deb": - varmap =3D self._dockerfile_build_varmap_deb(facts, mappings, = pip_mappings, projects, cross_arch) + varmap =3D self._dockerfile_build_varmap_deb(facts, mappings, = pip_mappings, projects, cross_arch, needbase) if facts["package_format"] =3D=3D "rpm": - varmap =3D self._dockerfile_build_varmap_rpm(facts, mappings, = pip_mappings, projects, cross_arch) + varmap =3D self._dockerfile_build_varmap_rpm(facts, mappings, = pip_mappings, projects, cross_arch, needbase) =20 varmap["package_manager"] =3D facts["package_manager"] varmap["cc"] =3D facts["cc"] @@ -662,7 +669,7 @@ class Application: =20 return varmap =20 - def _dockerfile_build_varmap_deb(self, facts, mappings, pip_mappings, = projects, cross_arch): + def _dockerfile_build_varmap_deb(self, facts, mappings, pip_mappings, = projects, cross_arch, needbase): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] os_name =3D facts["os_name"] @@ -682,7 +689,10 @@ class Application: =20 # We need to add the base project manually here: the standard # machinery hides it because it's an implementation detail - for project in projects + ["base"]: + pkgprojects =3D projects + if needbase: + pkgprojects =3D projects + ["base"] + for project in pkgprojects: for package in self._projects.get_packages(project): cross_policy =3D "native" for key in cross_keys: @@ -730,7 +740,7 @@ class Application: =20 return varmap =20 - def _dockerfile_build_varmap_rpm(self, facts, mappings, pip_mappings, = projects, cross_arch): + def _dockerfile_build_varmap_rpm(self, facts, mappings, pip_mappings, = projects, cross_arch, needbase): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] os_name =3D facts["os_name"] @@ -744,7 +754,10 @@ class Application: =20 # We need to add the base project manually here: the standard # machinery hides it because it's an implementation detail - for project in projects + ["base"]: + pkgprojects =3D projects + if needbase: + pkgprojects =3D projects + ["base"] + for project in pkgprojects: for package in self._projects.get_packages(project): for key in keys: if key in mappings[package]: @@ -791,7 +804,7 @@ class Application: =20 return varmap =20 - def _dockerfile_format(self, facts, cross_arch, varmap): + def _dockerfile_base_format(self, facts, cross_arch, varmap): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] os_name =3D facts["os_name"] @@ -931,6 +944,74 @@ class Application: ENV CONFIGURE_OPTS "--host=3D{cross_abi}" """).format(**varmap)) =20 + def _dockerfile_child_format(self, facts, cross_arch, inherit, varmap): + package_format =3D facts["package_format"] + package_manager =3D facts["package_manager"] + os_name =3D facts["os_name"] + os_version =3D facts["os_version"] + + print("FROM {}".format(inherit)) + + commands =3D [] + + if package_format =3D=3D "deb": + commands.extend([ + "export DEBIAN_FRONTEND=3Dnoninteractive", + "{package_manager} update", + "{package_manager} install --no-install-recommends -y {pkg= s}", + "{package_manager} autoremove -y", + "{package_manager} autoclean -y", + ]) + elif package_format =3D=3D "rpm": + commands.extend([ + "{package_manager} install -y {pkgs}", + ]) + + # openSUSE doesn't seem to have a convenient way to remove all + # unnecessary packages, but CentOS and Fedora do + if os_name =3D=3D "OpenSUSE": + commands.extend([ + "{package_manager} clean --all", + ]) + else: + commands.extend([ + "{package_manager} autoremove -y", + "{package_manager} clean all -y", + ]) + + + script =3D "\nRUN " + (" && \\\n ".join(commands)) + "\n" + sys.stdout.write(script.format(**varmap)) + + if cross_arch: + cross_commands =3D [] + + # Intentionally a separate RUN command from the above + # so that the common packages of all cross-built images + # share a Docker image layer. + if package_format =3D=3D "deb": + cross_commands.extend([ + "export DEBIAN_FRONTEND=3Dnoninteractive", + "{package_manager} update", + "{package_manager} install --no-install-recommends -y = {cross_pkgs}", + "{package_manager} autoremove -y", + "{package_manager} autoclean -y", + ]) + elif package_format =3D=3D "rpm": + cross_commands.extend([ + "{package_manager} install -y {cross_pkgs}", + "{package_manager} clean all -y", + ]) + + cross_script =3D "\nRUN " + (" && \\\n ".join(cross_command= s)) + "\n" + sys.stdout.write(cross_script.format(**varmap)) + + if "pip_pkgs" in varmap: + sys.stdout.write(textwrap.dedent(""" + RUN pip3 install {pip_pkgs} + """).format(**varmap)) + + def _action_dockerfile(self, args): mappings =3D self._projects.get_mappings() pip_mappings =3D self._projects.get_pip_mappings() @@ -947,6 +1028,7 @@ class Application: os_version =3D facts["os_version"] os_full =3D os_name + os_version cross_arch =3D args.cross_arch + inherit =3D args.inherit =20 if package_format not in ["deb", "rpm"]: raise Exception("Host {} doesn't support Dockerfiles".format(h= ost)) @@ -983,8 +1065,14 @@ class Application: ) ) =20 - varmap =3D self._dockerfile_build_varmap(facts, mappings, pip_mapp= ings, projects, cross_arch) - self._dockerfile_format(facts, cross_arch, varmap) + needbase =3D True + if inherit is not None: + needbase =3D False + varmap =3D self._dockerfile_build_varmap(facts, mappings, pip_mapp= ings, projects, cross_arch, needbase) + if inherit is None: + self._dockerfile_base_format(facts, cross_arch, varmap) + else: + self._dockerfile_child_format(facts, cross_arch, inherit, varm= ap) =20 def run(self): args =3D self._parser.parse_args() --=20 2.24.1