From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) client-ip=207.211.31.81; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by mx.zohomail.com with SMTPS id 1581355144631828.5638082432762; Mon, 10 Feb 2020 09:19:04 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-416--mR0CgmvPeCjYmRaZmzYyQ-1; Mon, 10 Feb 2020 12:18:34 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8D50C800D41; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1F5EB60BF3; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id E921B18089CE; Mon, 10 Feb 2020 17:18:24 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHINJD028000 for ; Mon, 10 Feb 2020 12:18:23 -0500 Received: by smtp.corp.redhat.com (Postfix) id BF1735C21B; Mon, 10 Feb 2020 17:18:23 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 478C15C1D6 for ; Mon, 10 Feb 2020 17:18:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355143; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=di2wZyPz0Qsd0aD+inmSRjmP6rBdSulPIR0vkohdD8I=; b=RtDsyYnJZQOeWoRcbwA5RkuEohWt095sHj/ziKRmNBP+t9x2mKQ5t43ZWrO1Cgn7YeJOnt Ze/koA26gKKTg85C2nTlLa+0IyttlhHKQrq+tzTwU2q8NDDiQ9BfBrnFJdYWd2O7QE/68p UmY185uFaxvv36/zqS6qT3PpUm0piLQ= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 1/8] lcitool: Introduce cross_arch local variable Date: Mon, 10 Feb 2020 18:18:10 +0100 Message-Id: <20200210171817.53126-2-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-MC-Unique: -mR0CgmvPeCjYmRaZmzYyQ-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 8a681e4..6624887 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -657,15 +657,16 @@ class Application: os_name =3D facts["os_name"] os_version =3D facts["os_version"] os_full =3D os_name + os_version + cross_arch =3D args.cross_arch =20 if package_format not in ["deb", "rpm"]: raise Exception("Host {} doesn't support Dockerfiles".format(h= ost)) - if args.cross_arch: + if cross_arch: if os_name !=3D "Debian": raise Exception("Cannot cross compile on {}".format(os_nam= e)) - if args.cross_arch =3D=3D self._native_arch: + if cross_arch =3D=3D self._native_arch: raise Exception("Cross arch {} should differ from native {= }". - format(args.cross_arch, self._native_arch)) + format(cross_arch, self._native_arch)) =20 projects =3D self._projects.expand_pattern(args.projects) for project in projects: @@ -682,8 +683,8 @@ class Application: pip_pkgs =3D {} base_keys =3D ["default", package_format, os_name, os_full] cross_keys =3D [] - if args.cross_arch: - keys =3D base_keys + [args.cross_arch + "-" + k for k in base_= keys] + if cross_arch: + keys =3D base_keys + [cross_arch + "-" + k for k in base_keys] cross_keys =3D ["cross-policy-" + k for k in base_keys] else: keys =3D base_keys + [self._native_arch + "-" + k for k in bas= e_keys] @@ -727,10 +728,10 @@ class Application: varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(sort= ed(set(pip_pkgs.values()))) =20 if package_format =3D=3D "deb": - if args.cross_arch: - deb_arch =3D Util.native_arch_to_deb_arch(args.cross_arch) - abi =3D Util.native_arch_to_abi(args.cross_arch) - lib =3D Util.native_arch_to_lib(args.cross_arch) + if cross_arch: + deb_arch =3D Util.native_arch_to_deb_arch(cross_arch) + abi =3D Util.native_arch_to_abi(cross_arch) + lib =3D Util.native_arch_to_lib(cross_arch) gcc =3D "gcc-" + abi varmap["cross_arch"] =3D deb_arch pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.valu= es()] @@ -749,7 +750,7 @@ class Application: sed -Ei 's,^# (en_US\\.UTF-8 .*)$,\\1,' /etc/locale.ge= n && \\ dpkg-reconfigure locales """).format(**varmap)) - if args.cross_arch: + if cross_arch: # Intentionally a separate RUN command from the above # so that the common packages of all cross-built images # share a Docker image layer. @@ -831,7 +832,7 @@ class Application: ENV LANG "en_US.UTF-8" """).format(**varmap)) =20 - if args.cross_arch: + if cross_arch: sys.stdout.write(textwrap.dedent(""" ENV ABI "{cross_abi}" ENV CONFIGURE_OPTS "--host=3D{cross_abi} \\ --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) client-ip=205.139.110.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by mx.zohomail.com with SMTPS id 1581355122899976.2221521568018; Mon, 10 Feb 2020 09:18:42 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-98-eoAKvSFuMNuv0W_6DmIScg-1; Mon, 10 Feb 2020 12:18:33 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 269A4800D48; Mon, 10 Feb 2020 17:18:28 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F204F84D8A; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 9003E18089CD; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHIOo3028010 for ; Mon, 10 Feb 2020 12:18:24 -0500 Received: by smtp.corp.redhat.com (Postfix) id 902EB5C28F; Mon, 10 Feb 2020 17:18:24 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 18CA95C21B for ; Mon, 10 Feb 2020 17:18:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355121; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=6HgCGMetBNkeCuM9DQ32LYBlaUjfk6b6DjJ6ISfLDf4=; b=EaN6PU4MM1hujx2klmpRog4gExYr23DZRV/o28K8gbjpN4nmbHayhxR0TMLf45aiOEi0+c 12kZhzV8LlK96/l7sDBvfDnVjVJEJUEoTDEHKj0BOGmpKTqz7SO3T2pYU3y6oZwHkC4DBa N04NxlmAizeQcmzniWbiIMSAMq84gbs= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 2/8] lcitool: Change check for pip_pkgs formatting Date: Mon, 10 Feb 2020 18:18:11 +0100 Message-Id: <20200210171817.53126-3-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: eoAKvSFuMNuv0W_6DmIScg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 6624887..80a0d6a 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -725,7 +725,9 @@ class Application: varmap =3D {} varmap["package_manager"] =3D package_manager varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) - varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(sort= ed(set(pip_pkgs.values()))) + + if pip_pkgs: + varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) =20 if package_format =3D=3D "deb": if cross_arch: @@ -823,7 +825,7 @@ class Application: =20 sys.stdout.write(script.format(**varmap)) =20 - if pip_pkgs: + if "pip_pkgs" in varmap: sys.stdout.write(textwrap.dedent(""" RUN pip3 install {pip_pkgs} """).format(**varmap)) --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) client-ip=205.139.110.61; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) by mx.zohomail.com with SMTPS id 1581355135878350.1254858131849; Mon, 10 Feb 2020 09:18:55 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-58-16kzQ360N7mDhVB1uC_dYA-1; Mon, 10 Feb 2020 12:18:49 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6F6E1800D54; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 45C9D5DA82; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id EF06C866C1; Mon, 10 Feb 2020 17:18:43 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHIPFG028020 for ; Mon, 10 Feb 2020 12:18:25 -0500 Received: by smtp.corp.redhat.com (Postfix) id 6152D5C28F; Mon, 10 Feb 2020 17:18:25 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DE1AF5C1D6 for ; Mon, 10 Feb 2020 17:18:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355134; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=0PU70jcP7kwhbnwuwoaDup8NjPaJypb0Tt9E3Qu0Bzs=; b=DrllrOfgJQsKqDVIU5/+HffuC1oMAaAIeczZAiiZzIj1EXaLwaGOvujKqjpCCFencSbflC NDWY+ZZSitwD7wvZjCikqOnwoZGb3EQiMjOqkBl+FkY7d9g2b2dCTx9sY/GVv7ixdNzwP0 pkHRMxP8WjdqQmMS7lodqnUl+KpRk5U= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 3/8] lcitool: Separate computation and formatting Date: Mon, 10 Feb 2020 18:18:12 +0100 Message-Id: <20200210171817.53126-4-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-MC-Unique: 16kzQ360N7mDhVB1uC_dYA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 80a0d6a..6da2206 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -720,8 +720,6 @@ class Application: pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) =20 - print("FROM {}".format(facts["docker_base"])) - varmap =3D {} varmap["package_manager"] =3D package_manager varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) @@ -729,19 +727,21 @@ class Application: if pip_pkgs: varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) =20 - if package_format =3D=3D "deb": - if cross_arch: - deb_arch =3D Util.native_arch_to_deb_arch(cross_arch) - abi =3D Util.native_arch_to_abi(cross_arch) - lib =3D Util.native_arch_to_lib(cross_arch) - gcc =3D "gcc-" + abi - varmap["cross_arch"] =3D deb_arch - pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.valu= es()] - pkg_names.append(gcc) - varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(so= rted(set(pkg_names))) - varmap["cross_abi"] =3D abi - varmap["cross_lib"] =3D lib + if package_format =3D=3D "deb" and args.cross_arch: + deb_arch =3D Util.native_arch_to_deb_arch(args.cross_arch) + abi =3D Util.native_arch_to_abi(args.cross_arch) + lib =3D Util.native_arch_to_lib(args.cross_arch) + gcc =3D "gcc-" + abi + varmap["cross_arch"] =3D deb_arch + pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.values()] + pkg_names.append(gcc) + varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted= (set(pkg_names))) + varmap["cross_abi"] =3D abi + varmap["cross_lib"] =3D lib =20 + print("FROM {}".format(facts["docker_base"])) + + if package_format =3D=3D "deb": sys.stdout.write(textwrap.dedent(""" RUN export DEBIAN_FRONTEND=3Dnoninteractive && \\ {package_manager} update && \\ --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) client-ip=205.139.110.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by mx.zohomail.com with SMTPS id 1581355133352911.9568272767623; Mon, 10 Feb 2020 09:18:53 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-44-h9CvaGZAPwau6cJkh3pfKg-1; Mon, 10 Feb 2020 12:18:49 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 53ECD800D55; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 27CFA26FCF; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id D5CD78175A; Mon, 10 Feb 2020 17:18:43 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHIQ2s028029 for ; Mon, 10 Feb 2020 12:18:26 -0500 Received: by smtp.corp.redhat.com (Postfix) id 336605C28F; Mon, 10 Feb 2020 17:18:26 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AF2295C1D6 for ; Mon, 10 Feb 2020 17:18:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355131; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=/2/Y2EmaOgSMdk8Ci8mrUGuwhcVaVg/r3sxXlZ7LoV8=; b=CRe98CzEQ9757iYcXuvT1A/7yWXDWOpWATBdn/pLSMwLEj2y0OsQavzubsKo5ckzdpypaY YPDiXqzppJpDdCR+cDH6y1QKADpK5056pnOBGjvK9UoL97F8wj8aJ/UJmnuoUyWSq/oCfQ 2P2s2RQ7qjhlhWH9KKTQj70lN0hygVM= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 4/8] lcitool: Introduce _dockerfile_format() Date: Mon, 10 Feb 2020 18:18:13 +0100 Message-Id: <20200210171817.53126-5-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-MC-Unique: h9CvaGZAPwau6cJkh3pfKg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This function takes care of approximately half of the job that was up until now handled by _action_dockerfile(), namely taking a populated dictionary and turning it into a Dockerfile. Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 192 +++++++++++++++++++++++++------------------------ 1 file changed, 100 insertions(+), 92 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 6da2206..f216910 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -642,102 +642,11 @@ class Application: with open(keyfile, "r") as r: return r.read().rstrip() =20 - def _action_dockerfile(self, args): - mappings =3D self._projects.get_mappings() - pip_mappings =3D self._projects.get_pip_mappings() - - hosts =3D self._inventory.expand_pattern(args.hosts) - if len(hosts) > 1: - raise Exception("Can't generate Dockerfile for multiple hosts") - host =3D hosts[0] - - facts =3D self._inventory.get_facts(host) + def _dockerfile_format(self, facts, cross_arch, varmap): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] os_name =3D facts["os_name"] os_version =3D facts["os_version"] - os_full =3D os_name + os_version - cross_arch =3D args.cross_arch - - if package_format not in ["deb", "rpm"]: - raise Exception("Host {} doesn't support Dockerfiles".format(h= ost)) - if cross_arch: - if os_name !=3D "Debian": - raise Exception("Cannot cross compile on {}".format(os_nam= e)) - if cross_arch =3D=3D self._native_arch: - raise Exception("Cross arch {} should differ from native {= }". - format(cross_arch, self._native_arch)) - - projects =3D self._projects.expand_pattern(args.projects) - for project in projects: - if project not in facts["projects"]: - raise Exception( - "Host {} doesn't support project {}".format( - host, - project, - ) - ) - - pkgs =3D {} - cross_pkgs =3D {} - pip_pkgs =3D {} - base_keys =3D ["default", package_format, os_name, os_full] - cross_keys =3D [] - if cross_arch: - keys =3D base_keys + [cross_arch + "-" + k for k in base_keys] - cross_keys =3D ["cross-policy-" + k for k in base_keys] - else: - keys =3D base_keys + [self._native_arch + "-" + k for k in bas= e_keys] - - # We need to add the base project manually here: the standard - # machinery hides it because it's an implementation detail - for project in projects + ["base"]: - for package in self._projects.get_packages(project): - cross_policy =3D "native" - for key in cross_keys: - if key in mappings[package]: - cross_policy =3D mappings[package][key] - if cross_policy not in ["native", "foreign", "skip"]: - raise Exception( - "Unexpected cross arch policy {} for {}".format - (cross_policy, package)) - - for key in keys: - if key in mappings[package]: - pkgs[package] =3D mappings[package][key] - if package in pip_mappings and key in pip_mappings[pac= kage]: - pip_pkgs[package] =3D pip_mappings[package][key] - - if package not in pkgs: - continue - if package in pip_pkgs and pkgs[package] is not None: - del pip_pkgs[package] - if cross_policy =3D=3D "foreign" and pkgs[package] is not = None: - cross_pkgs[package] =3D pkgs[package] - if pkgs[package] is None or cross_policy in ["skip", "fore= ign"]: - del pkgs[package] - - pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) - pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) - - varmap =3D {} - varmap["package_manager"] =3D package_manager - varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) - - if pip_pkgs: - varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) - - if package_format =3D=3D "deb" and args.cross_arch: - deb_arch =3D Util.native_arch_to_deb_arch(args.cross_arch) - abi =3D Util.native_arch_to_abi(args.cross_arch) - lib =3D Util.native_arch_to_lib(args.cross_arch) - gcc =3D "gcc-" + abi - varmap["cross_arch"] =3D deb_arch - pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.values()] - pkg_names.append(gcc) - varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted= (set(pkg_names))) - varmap["cross_abi"] =3D abi - varmap["cross_lib"] =3D lib =20 print("FROM {}".format(facts["docker_base"])) =20 @@ -842,6 +751,105 @@ class Application: ENV PKG_CONFIG_LIBDIR "/usr/lib/{cross_lib}/pkgconfig" """).format(**varmap)) =20 + def _action_dockerfile(self, args): + mappings =3D self._projects.get_mappings() + pip_mappings =3D self._projects.get_pip_mappings() + + hosts =3D self._inventory.expand_pattern(args.hosts) + if len(hosts) > 1: + raise Exception("Can't generate Dockerfile for multiple hosts") + host =3D hosts[0] + + facts =3D self._inventory.get_facts(host) + package_format =3D facts["package_format"] + package_manager =3D facts["package_manager"] + os_name =3D facts["os_name"] + os_version =3D facts["os_version"] + os_full =3D os_name + os_version + cross_arch =3D args.cross_arch + + if package_format not in ["deb", "rpm"]: + raise Exception("Host {} doesn't support Dockerfiles".format(h= ost)) + if cross_arch: + if os_name !=3D "Debian": + raise Exception("Cannot cross compile on {}".format(os_nam= e)) + if cross_arch =3D=3D self._native_arch: + raise Exception("Cross arch {} should differ from native {= }". + format(cross_arch, self._native_arch)) + + projects =3D self._projects.expand_pattern(args.projects) + for project in projects: + if project not in facts["projects"]: + raise Exception( + "Host {} doesn't support project {}".format( + host, + project, + ) + ) + + pkgs =3D {} + cross_pkgs =3D {} + pip_pkgs =3D {} + base_keys =3D ["default", package_format, os_name, os_full] + cross_keys =3D [] + if cross_arch: + keys =3D base_keys + [cross_arch + "-" + k for k in base_keys] + cross_keys =3D ["cross-policy-" + k for k in base_keys] + else: + keys =3D base_keys + [self._native_arch + "-" + k for k in bas= e_keys] + + # We need to add the base project manually here: the standard + # machinery hides it because it's an implementation detail + for project in projects + ["base"]: + for package in self._projects.get_packages(project): + cross_policy =3D "native" + for key in cross_keys: + if key in mappings[package]: + cross_policy =3D mappings[package][key] + if cross_policy not in ["native", "foreign", "skip"]: + raise Exception( + "Unexpected cross arch policy {} for {}".format + (cross_policy, package)) + + for key in keys: + if key in mappings[package]: + pkgs[package] =3D mappings[package][key] + if package in pip_mappings and key in pip_mappings[pac= kage]: + pip_pkgs[package] =3D pip_mappings[package][key] + + if package not in pkgs: + continue + if package in pip_pkgs and pkgs[package] is not None: + del pip_pkgs[package] + if cross_policy =3D=3D "foreign" and pkgs[package] is not = None: + cross_pkgs[package] =3D pkgs[package] + if pkgs[package] is None or cross_policy in ["skip", "fore= ign"]: + del pkgs[package] + + pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) + pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) + + varmap =3D {} + varmap["package_manager"] =3D package_manager + varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) + + if package_format =3D=3D "deb" and cross_arch: + deb_arch =3D Util.native_arch_to_deb_arch(cross_arch) + abi =3D Util.native_arch_to_abi(cross_arch) + lib =3D Util.native_arch_to_lib(cross_arch) + gcc =3D "gcc-" + abi + varmap["cross_arch"] =3D deb_arch + pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.values()] + pkg_names.append(gcc) + varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted= (set(pkg_names))) + varmap["cross_abi"] =3D abi + varmap["cross_lib"] =3D lib + + if pip_pkgs: + varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) + + self._dockerfile_format(facts, cross_arch, varmap) + def run(self): args =3D self._parser.parse_args() if args.debug: --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) client-ip=207.211.31.81; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by mx.zohomail.com with SMTPS id 1581355151790582.540397422154; Mon, 10 Feb 2020 09:19:11 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-366-xrc0ofPzN022ZJ2U6t7Mvg-1; Mon, 10 Feb 2020 12:18:53 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DA6A9108BD10; Mon, 10 Feb 2020 17:18:47 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B79955D9CA; Mon, 10 Feb 2020 17:18:47 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 70DA018089D6; Mon, 10 Feb 2020 17:18:47 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHIREv028034 for ; Mon, 10 Feb 2020 12:18:27 -0500 Received: by smtp.corp.redhat.com (Postfix) id 040345C28F; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 80C805C1D6 for ; Mon, 10 Feb 2020 17:18:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355150; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=oKwTRJpK01PxVllSOn5jB7t4f8+HsgV4muM4obqjUus=; b=NlpyGbyO+iZTJRHTMsuryHnXgEsn54OOwPTxZt45nb+j4G+66s9akvuot9NYsLFEPYqMFC sDADb5Qk+DIkR285mNWAzegF1YSTr4D2i45h7PvUMgd0YZLCcoqU4yITAynsSUz6dazou2 Z1IQXLsFp86jWgye09/yrHY2rsRJbdc= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 5/8] lcitool: Introduce _dockerfile_build_varmap() Date: Mon, 10 Feb 2020 18:18:14 +0100 Message-Id: <20200210171817.53126-6-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-MC-Unique: xrc0ofPzN022ZJ2U6t7Mvg-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This function takes care of most of what's still handled by _action_dockerfile(), namely populating the dictionary that _dockerfile_format() will later consume. Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 132 ++++++++++++++++++++++++++----------------------- 1 file changed, 71 insertions(+), 61 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index f216910..82e98c5 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -642,6 +642,76 @@ class Application: with open(keyfile, "r") as r: return r.read().rstrip() =20 + def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, proj= ects, cross_arch): + package_format =3D facts["package_format"] + package_manager =3D facts["package_manager"] + os_name =3D facts["os_name"] + os_version =3D facts["os_version"] + os_full =3D os_name + os_version + + pkgs =3D {} + cross_pkgs =3D {} + pip_pkgs =3D {} + base_keys =3D ["default", package_format, os_name, os_full] + cross_keys =3D [] + if cross_arch: + keys =3D base_keys + [cross_arch + "-" + k for k in base_keys] + cross_keys =3D ["cross-policy-" + k for k in base_keys] + else: + keys =3D base_keys + [self._native_arch + "-" + k for k in bas= e_keys] + + # We need to add the base project manually here: the standard + # machinery hides it because it's an implementation detail + for project in projects + ["base"]: + for package in self._projects.get_packages(project): + cross_policy =3D "native" + for key in cross_keys: + if key in mappings[package]: + cross_policy =3D mappings[package][key] + if cross_policy not in ["native", "foreign", "skip"]: + raise Exception( + "Unexpected cross arch policy {} for {}".format + (cross_policy, package)) + + for key in keys: + if key in mappings[package]: + pkgs[package] =3D mappings[package][key] + if package in pip_mappings and key in pip_mappings[pac= kage]: + pip_pkgs[package] =3D pip_mappings[package][key] + + if package not in pkgs: + continue + if package in pip_pkgs and pkgs[package] is not None: + del pip_pkgs[package] + if cross_policy =3D=3D "foreign" and pkgs[package] is not = None: + cross_pkgs[package] =3D pkgs[package] + if pkgs[package] is None or cross_policy in ["skip", "fore= ign"]: + del pkgs[package] + + pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) + pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) + + varmap =3D {} + varmap["package_manager"] =3D package_manager + varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) + + if package_format =3D=3D "deb" and cross_arch: + deb_arch =3D Util.native_arch_to_deb_arch(cross_arch) + abi =3D Util.native_arch_to_abi(cross_arch) + lib =3D Util.native_arch_to_lib(cross_arch) + gcc =3D "gcc-" + abi + varmap["cross_arch"] =3D deb_arch + pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.values()] + pkg_names.append(gcc) + varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted= (set(pkg_names))) + varmap["cross_abi"] =3D abi + varmap["cross_lib"] =3D lib + + if pip_pkgs: + varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) + + return varmap + def _dockerfile_format(self, facts, cross_arch, varmap): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] @@ -787,67 +857,7 @@ class Application: ) ) =20 - pkgs =3D {} - cross_pkgs =3D {} - pip_pkgs =3D {} - base_keys =3D ["default", package_format, os_name, os_full] - cross_keys =3D [] - if cross_arch: - keys =3D base_keys + [cross_arch + "-" + k for k in base_keys] - cross_keys =3D ["cross-policy-" + k for k in base_keys] - else: - keys =3D base_keys + [self._native_arch + "-" + k for k in bas= e_keys] - - # We need to add the base project manually here: the standard - # machinery hides it because it's an implementation detail - for project in projects + ["base"]: - for package in self._projects.get_packages(project): - cross_policy =3D "native" - for key in cross_keys: - if key in mappings[package]: - cross_policy =3D mappings[package][key] - if cross_policy not in ["native", "foreign", "skip"]: - raise Exception( - "Unexpected cross arch policy {} for {}".format - (cross_policy, package)) - - for key in keys: - if key in mappings[package]: - pkgs[package] =3D mappings[package][key] - if package in pip_mappings and key in pip_mappings[pac= kage]: - pip_pkgs[package] =3D pip_mappings[package][key] - - if package not in pkgs: - continue - if package in pip_pkgs and pkgs[package] is not None: - del pip_pkgs[package] - if cross_policy =3D=3D "foreign" and pkgs[package] is not = None: - cross_pkgs[package] =3D pkgs[package] - if pkgs[package] is None or cross_policy in ["skip", "fore= ign"]: - del pkgs[package] - - pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) - pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) - - varmap =3D {} - varmap["package_manager"] =3D package_manager - varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) - - if package_format =3D=3D "deb" and cross_arch: - deb_arch =3D Util.native_arch_to_deb_arch(cross_arch) - abi =3D Util.native_arch_to_abi(cross_arch) - lib =3D Util.native_arch_to_lib(cross_arch) - gcc =3D "gcc-" + abi - varmap["cross_arch"] =3D deb_arch - pkg_names =3D [p + ":" + deb_arch for p in cross_pkgs.values()] - pkg_names.append(gcc) - varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted= (set(pkg_names))) - varmap["cross_abi"] =3D abi - varmap["cross_lib"] =3D lib - - if pip_pkgs: - varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) - + varmap =3D self._dockerfile_build_varmap(facts, mappings, pip_mapp= ings, projects, cross_arch) self._dockerfile_format(facts, cross_arch, varmap) =20 def run(self): --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) client-ip=205.139.110.61; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) by mx.zohomail.com with SMTPS id 1581355147560274.9886759946405; Mon, 10 Feb 2020 09:19:07 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-407-OoVnHePIPUOc1ieuaHGD_A-1; Mon, 10 Feb 2020 12:18:57 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 780F7DB2C; Mon, 10 Feb 2020 17:18:51 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 54E9A5DA84; Mon, 10 Feb 2020 17:18:51 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 17D99180880D; Mon, 10 Feb 2020 17:18:51 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHIRhM028039 for ; Mon, 10 Feb 2020 12:18:27 -0500 Received: by smtp.corp.redhat.com (Postfix) id C819B5C28F; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 515EE5C1D6 for ; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355146; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=H+szaMIERroUwZyKA17Km26tAt9INz4S4N/O9Kl31uc=; b=JvfCusaMpcydJL9qRBzPbfmGiLBW6lI3v54FR/TOqB2L1aVMqrn9VEX0qJ8BJv3uQ+MMPn mzegHXOgkS9hWJMGTPhRLmsbd37eP8b5Nuu957kCZ3tEwAqdCbNIuSMvJFCIbg96zMriRh ygZCRl3Fu5OTOlJW/IBcnYbGz2IBPAw= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 6/8] lcitool: Add RPM-specific _dockerfile_build_varmap() variant Date: Mon, 10 Feb 2020 18:18:15 +0100 Message-Id: <20200210171817.53126-7-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-MC-Unique: OoVnHePIPUOc1ieuaHGD_A-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" A lot of the code in _dockerfile_build_varmap() only applies to Debian, so we can have a simpler version for use with Fedora. This split will come in handy later on, when we start introducing features that work in Fedora and not in Debian. Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 48 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/guests/lcitool b/guests/lcitool index 82e98c5..37a0253 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -643,6 +643,12 @@ class Application: return r.read().rstrip() =20 def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, proj= ects, cross_arch): + if facts["package_format"] =3D=3D "deb": + return self._dockerfile_build_varmap_deb(facts, mappings, pip_= mappings, projects, cross_arch) + if facts["package_format"] =3D=3D "rpm": + return self._dockerfile_build_varmap_rpm(facts, mappings, pip_= mappings, projects, cross_arch) + + def _dockerfile_build_varmap_deb(self, facts, mappings, pip_mappings, = projects, cross_arch): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] os_name =3D facts["os_name"] @@ -695,7 +701,7 @@ class Application: varmap["package_manager"] =3D package_manager varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) =20 - if package_format =3D=3D "deb" and cross_arch: + if cross_arch: deb_arch =3D Util.native_arch_to_deb_arch(cross_arch) abi =3D Util.native_arch_to_abi(cross_arch) lib =3D Util.native_arch_to_lib(cross_arch) @@ -712,6 +718,46 @@ class Application: =20 return varmap =20 + def _dockerfile_build_varmap_rpm(self, facts, mappings, pip_mappings, = projects, cross_arch): + package_format =3D facts["package_format"] + package_manager =3D facts["package_manager"] + os_name =3D facts["os_name"] + os_version =3D facts["os_version"] + os_full =3D os_name + os_version + + pkgs =3D {} + pip_pkgs =3D {} + keys =3D ["default", package_format, os_name, os_full] + + # We need to add the base project manually here: the standard + # machinery hides it because it's an implementation detail + for project in projects + ["base"]: + for package in self._projects.get_packages(project): + for key in keys: + if key in mappings[package]: + pkgs[package] =3D mappings[package][key] + if package in pip_mappings and key in pip_mappings[pac= kage]: + pip_pkgs[package] =3D pip_mappings[package][key] + + if package not in pkgs: + continue + if package in pip_pkgs and pkgs[package] is not None: + del pip_pkgs[package] + if pkgs[package] is None: + del pkgs[package] + + pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) + pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) + + varmap =3D {} + varmap["package_manager"] =3D package_manager + varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) + + if pip_pkgs: + varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) + + return varmap + def _dockerfile_format(self, facts, cross_arch, varmap): package_format =3D facts["package_format"] package_manager =3D facts["package_manager"] --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) client-ip=205.139.110.61; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.61 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) by mx.zohomail.com with SMTPS id 1581355134865321.7658965368918; Mon, 10 Feb 2020 09:18:54 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-66-0GGKufGaOhueZlfBnmhROQ-1; Mon, 10 Feb 2020 12:18:49 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 796FA18C8C0E; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4FDEF8ED05; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 026C4866D3; Mon, 10 Feb 2020 17:18:44 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHIS04028057 for ; Mon, 10 Feb 2020 12:18:28 -0500 Received: by smtp.corp.redhat.com (Postfix) id E12C95C28F; Mon, 10 Feb 2020 17:18:28 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 214E55C1D6 for ; Mon, 10 Feb 2020 17:18:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355133; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=9vifU1tpyAYGZiE/Ml6do4aopj5lAdS5d4X8nZ0Ho78=; b=Y0/M+EVZ5DPUHwU+e+7e+mNDx8EiomhXDv4jWSncTSIbic2cBgeUIuyriGT8xwDqQ9HdmL QXKCv41d/0BYPk8bYYw+ZXkoirEqN/YTlWTiLQF6vfqCoKlkq3xmQocXsyeSNjsNfgoIOo GQJT4bZYk50OUUMwN0jDP4sj1/8DAKk= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 7/8] lcitool: Support MinGW cross-build Dockerfiles on Fedora Date: Mon, 10 Feb 2020 18:18:16 +0100 Message-Id: <20200210171817.53126-8-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-MC-Unique: 0GGKufGaOhueZlfBnmhROQ-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Up until this point, MinGW has been treated as a very special case, even having its own sub-projects. While this was perfectly reasonable back when lcitool was introduced, now that we support cross-building as a first-class concept in the Dockerfile generator it makes little sense for MinGW builds not to make use of it. Note that, as of this commit, MinGW is still considered somewhat special: in particular, only the Dockerfile generator can treat it like any other architecture, and everything else still needs to use the sub-projects. In the longer run we'll want to expand the first-class cross-building support to the rest of lcitool; this is merely another step in that direction. Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 46 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 37a0253..abc803b 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -93,6 +93,8 @@ class Util: "armv6l": "arm-linux-gnueabi", "armv7l": "arm-linux-gnueabihf", "i686": "i686-linux-gnu", + "mingw32": "i686-w64-mingw32", + "mingw64": "x86_64-w64-mingw32", "mips": "mips-linux-gnu", "mipsel": "mipsel-linux-gnu", "mips64el": "mips64el-linux-gnuabi64", @@ -726,6 +728,7 @@ class Application: os_full =3D os_name + os_version =20 pkgs =3D {} + cross_pkgs =3D {} pip_pkgs =3D {} keys =3D ["default", package_format, os_name, os_full] =20 @@ -746,6 +749,24 @@ class Application: if pkgs[package] is None: del pkgs[package] =20 + if cross_arch: + cross_projects =3D [] + for project in projects: + cross_project =3D project + "+" + cross_arch + if cross_project in facts["projects"]: + cross_projects.extend([cross_project]) + + for project in cross_projects: + for package in self._projects.get_packages(project): + for key in keys: + if key in mappings[package]: + cross_pkgs[package] =3D mappings[package][key] + + if package not in cross_pkgs: + continue + if cross_pkgs[package] is None: + del cross_pkgs[package] + pkg_align =3D " \\\n" + (" " * len("RUN " + package_manager + " ")) pip_pkg_align =3D " \\\n" + (" " * len("RUN pip3 ")) =20 @@ -753,6 +774,10 @@ class Application: varmap["package_manager"] =3D package_manager varmap["pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted(set(pkgs.= values()))) =20 + if cross_arch: + varmap["cross_abi"] =3D Util.native_arch_to_abi(cross_arch) + varmap["cross_pkgs"] =3D pkg_align[1:] + pkg_align.join(sorted= (set(cross_pkgs.values()))) + if pip_pkgs: varmap["pip_pkgs"] =3D pip_pkg_align[1:] + pip_pkg_align.join(= sorted(set(pip_pkgs.values()))) =20 @@ -850,6 +875,15 @@ class Application: =20 sys.stdout.write(script.format(**varmap)) =20 + if cross_arch: + # Intentionally a separate RUN command from the above + # so that the common packages of all cross-built images + # share a Docker image layer. + sys.stdout.write(textwrap.dedent(""" + RUN {package_manager} install -y {cross_pkgs} && \\ + {package_manager} clean all -y + """).format(**varmap)) + if "pip_pkgs" in varmap: sys.stdout.write(textwrap.dedent(""" RUN pip3 install {pip_pkgs} @@ -859,7 +893,7 @@ class Application: ENV LANG "en_US.UTF-8" """).format(**varmap)) =20 - if cross_arch: + if cross_arch and package_format =3D=3D "deb": sys.stdout.write(textwrap.dedent(""" ENV ABI "{cross_abi}" ENV CONFIGURE_OPTS "--host=3D{cross_abi} \\ @@ -867,6 +901,14 @@ class Application: ENV PKG_CONFIG_LIBDIR "/usr/lib/{cross_lib}/pkgconfig" """).format(**varmap)) =20 + if cross_arch and package_format =3D=3D "rpm": + sys.stdout.write(textwrap.dedent(""" + ENV ABI "{cross_abi}" + ENV CONFIGURE_OPTS "--host=3D{cross_abi} \\ + --target=3D{cross_abi}" + ENV PKG_CONFIG_LIBDIR "/usr/{cross_abi}/sys-root/mingw/lib= /pkgconfig:/usr/{cross_abi}/sys-root/mingw/share/pkgconfig" + """).format(**varmap)) + def _action_dockerfile(self, args): mappings =3D self._projects.get_mappings() pip_mappings =3D self._projects.get_pip_mappings() @@ -887,7 +929,7 @@ class Application: if package_format not in ["deb", "rpm"]: raise Exception("Host {} doesn't support Dockerfiles".format(h= ost)) if cross_arch: - if os_name !=3D "Debian": + if os_name not in ["Debian", "Fedora"]: raise Exception("Cannot cross compile on {}".format(os_nam= e)) if cross_arch =3D=3D self._native_arch: raise Exception("Cross arch {} should differ from native {= }". --=20 2.24.1 From nobody Fri Apr 19 10:32:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) client-ip=205.139.110.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 205.139.110.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by mx.zohomail.com with SMTPS id 15813551413741007.4014434965652; Mon, 10 Feb 2020 09:19:01 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-164-yNuQkh2RP1mFvNXPZF0JvA-1; Mon, 10 Feb 2020 12:18:53 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EA9A91005516; Mon, 10 Feb 2020 17:18:47 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C172C26FDD; Mon, 10 Feb 2020 17:18:47 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 80A7918089D0; Mon, 10 Feb 2020 17:18:47 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 01AHITiA028064 for ; Mon, 10 Feb 2020 12:18:29 -0500 Received: by smtp.corp.redhat.com (Postfix) id B29D25C28F; Mon, 10 Feb 2020 17:18:29 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.43.2.246]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3A9895C1D6 for ; Mon, 10 Feb 2020 17:18:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581355139; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=NILzrXSAuoFRMa8PwRjbvnWnJJGCBfSYDmTInJ6XWXI=; b=bNscGiEHeCNM3NLqXkh0sbvupSEfE86Wpjco8YfK84ALlBZNEkUANW6kznADcSSSgVnsk5 UsdCn0Av3Vjvcspqh7Puq4mvCTJWHEr7dxLp2IenIewSM8BEwdIYCEBabu6MSRk6D/EsPb 4/EBIgWlW/HwKKyb3L1SMwoS4ybh08s= From: Andrea Bolognani To: libvir-list@redhat.com Subject: [jenkins-ci PATCH 8/8] lcitool: Add more checks to _action_dockerfile() Date: Mon, 10 Feb 2020 18:18:17 +0100 Message-Id: <20200210171817.53126-9-abologna@redhat.com> In-Reply-To: <20200210171817.53126-1-abologna@redhat.com> References: <20200210171817.53126-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-MC-Unique: yNuQkh2RP1mFvNXPZF0JvA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" There are a few problematic scenarios that we can catch early. Signed-off-by: Andrea Bolognani Reviewed-by: J=C3=A1n Tomko --- guests/lcitool | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/guests/lcitool b/guests/lcitool index abc803b..771402e 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -931,12 +931,28 @@ class Application: if cross_arch: if os_name not in ["Debian", "Fedora"]: raise Exception("Cannot cross compile on {}".format(os_nam= e)) + if os_name =3D=3D "Debian" and cross_arch.startswith("mingw"): + raise Exception( + "Cannot cross compile for {} on {}".format( + cross_arch, + os_name, + ) + ) + if os_name =3D=3D "Fedora" and not cross_arch.startswith("ming= w"): + raise Exception( + "Cannot cross compile for {} on {}".format( + cross_arch, + os_name, + ) + ) if cross_arch =3D=3D self._native_arch: raise Exception("Cross arch {} should differ from native {= }". format(cross_arch, self._native_arch)) =20 projects =3D self._projects.expand_pattern(args.projects) for project in projects: + if project.rfind("+mingw") >=3D 0: + raise Exception("Obsolete syntax, please use --cross-arch") if project not in facts["projects"]: raise Exception( "Host {} doesn't support project {}".format( --=20 2.24.1