From nobody Sun Feb 8 05:41:02 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1550846020182976.851863921303; Fri, 22 Feb 2019 06:33:40 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6B4B630A81A2; Fri, 22 Feb 2019 14:33:38 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 39E6C60BF3; Fri, 22 Feb 2019 14:33:38 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id E979B247E2; Fri, 22 Feb 2019 14:33:37 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x1LGYCgp015373 for ; Thu, 21 Feb 2019 11:34:12 -0500 Received: by smtp.corp.redhat.com (Postfix) id 6FC3860BFB; Thu, 21 Feb 2019 16:34:12 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-112-65.ams2.redhat.com [10.36.112.65]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B4B160BF2; Thu, 21 Feb 2019 16:34:10 +0000 (UTC) From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Thu, 21 Feb 2019 16:33:54 +0000 Message-Id: <20190221163354.23482-6-berrange@redhat.com> In-Reply-To: <20190221163354.23482-1-berrange@redhat.com> References: <20190221163354.23482-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: libvir-list@redhat.com Subject: [libvirt] [jenkins-ci PATCH v4 5/5] lcitool: support generating cross compiler dockerfiles X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Fri, 22 Feb 2019 14:33:39 +0000 (UTC) Debian's filesystem layout has a nice advantage over Fedora which is that it can install non-native RPMs in the main root filesystem. It is thus possible to prepare an x86_64 filesystem containing -dev packages for a foreign architecture, along with a GCC cross compiler. QEMU has used this technique to facilitate developer build testing of non-x86 architectures, since few people have access to physical hardware for most of these architectures. For the same reason it would be helpful to libvirt developers. This patch extends the 'dockerfile' command to 'lcitool' so that it accepts a '-x $ARCH' argument. $ lcitool dockerfile -x s390x libvirt-debian-9 libvirt This is only valid when using a 'deb' based distro. Signed-off-by: Daniel P. Berrang=C3=A9 --- guests/host_vars/libvirt-debian-9/main.yml | 44 +++++++++++++ guests/host_vars/libvirt-debian-sid/main.yml | 45 ++++++++++++++ guests/lcitool | 65 +++++++++++++++++++- guests/vars/mappings.yml | 59 ++++++++++++++++++ 4 files changed, 211 insertions(+), 2 deletions(-) diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/= libvirt-debian-9/main.yml index ec7e6b4..3bf4ae1 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -21,3 +21,47 @@ os_name: 'Debian' os_version: '9' =20 ansible_python_interpreter: /usr/bin/python3 + +arches: + aarch64: + package_arch: arm64 + abi: aarch64-linux-gnu + cross_gcc: gcc-aarch64-linux-gnu + armv6l: + package_arch: armel + abi: arm-linux-gnueabi + cross_gcc: gcc-arm-linux-gnueabi + armv7l: + package_arch: armhf + abi: arm-linux-gnueabihf + cross_gcc: gcc-arm-linux-gnueabihf + i686: + package_arch: i386 + abi: i386-linux-gnu + mips: + package_arch: mips + abi: mips-linux-gnu + cross_gcc: gcc-mips-linux-gnu + mipsel: + package_arch: mipsel + abi: mipsel-linux-gnu + cross_gcc: gcc-mipsel-linux-gnu + mips64: + package_arch: mips64 + abi: mips64-linux-gnu + cross_gcc: gcc-mips64-linux-gnu + mips64el: + package_arch: mips64el + abi: mips64el-linux-gnu + cross_gcc: gcc-mips64el-linux-gnu + ppc64el: + package_arch: ppc64el + abi: ppc64el-linux-gnu + cross_gcc: gcc-ppc64el-linux-gnu + s390x: + package_arch: s390x + abi: s390x-linux-gnu + cross_gcc: gcc-s390x-linux-gnu + x86_64: + package_arch: amd64 + abi: x86_64-linux-gnu diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_var= s/libvirt-debian-sid/main.yml index 1c7a29b..b14a564 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -21,3 +21,48 @@ os_name: 'Debian' os_version: 'Sid' =20 ansible_python_interpreter: /usr/bin/python3 + +arches: + aarch64: + package_arch: arm64 + abi: aarch64-linux-gnu + cross_gcc: gcc-aarch64-linux-gnu + armv6l: + package_arch: armel + abi: arm-linux-gnueabi + cross_gcc: gcc-arm-linux-gnueabi + armv7l: + package_arch: armhf + abi: arm-linux-gnueabihf + cross_gcc: gcc-arm-linux-gnueabihf + i686: + package_arch: i386 + abi: i686-linux-gnu + cross_gcc: gcc-i686-linux-gnu + mips: + package_arch: mips + abi: mips-linux-gnu + cross_gcc: gcc-mips-linux-gnu + mipsel: + package_arch: mipsel + abi: mipsel-linux-gnu + cross_gcc: gcc-mipsel-linux-gnu + mips64: + package_arch: mips64 + abi: mips64-linux-gnu + cross_gcc: gcc-mips64-linux-gnu + mips64el: + package_arch: mips64el + abi: mips64el-linux-gnu + cross_gcc: gcc-mips64el-linux-gnu + ppc64el: + package_arch: ppc64el + abi: ppc64el-linux-gnu + cross_gcc: gcc-ppc64el-linux-gnu + s390x: + package_arch: s390x + abi: s390x-linux-gnu + cross_gcc: gcc-s390x-linux-gnu + x86_64: + package_arch: amd64 + abi: x86_64-linux-gnu diff --git a/guests/lcitool b/guests/lcitool index ae7e4ee..b431b93 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -376,6 +376,10 @@ class Application: =20 add_hosts_arg(dockerfileparser) add_projects_arg(dockerfileparser) + dockerfileparser.add_argument( + "-x", "--cross-arch", + help=3D"cross compiler architecture", + ) =20 def _execute_playbook(self, playbook, hosts, projects, git_revision): base =3D Util.get_base() @@ -546,10 +550,23 @@ class Application: os_name =3D facts["os_name"] os_version =3D facts["os_version"] os_full =3D os_name + os_version + cross_facts =3D None =20 if package_format not in ["deb", "rpm"]: raise Error("Host {} doesn't support Dockerfiles".format(host)) =20 + if args.cross_arch is not None: + if "arches" not in facts: + raise Error("Non x86_64 arches not supported for this host= ") + if args.cross_arch not in facts["arches"]: + raise Error("Arch {} not supported for this host".format( + args.cross_arch)) + if "cross_gcc" not in facts["arches"][args.cross_arch]: + raise Error("Arch {} cross compiler not supported for this= host".format + (args.cross_arch)) + + cross_build_facts =3D facts["arches"][args.cross_arch] + projects =3D self._projects.expand_pattern(args.projects) for project in projects: if project not in facts["projects"]: @@ -561,25 +578,50 @@ class Application: ) =20 pkgs =3D {} + cross_pkgs =3D {} base_keys =3D ["default", package_format, os_name, os_full] - keys =3D base_keys + [k + "-" + self._native_arch for k in base_ke= ys] + cross_keys =3D [] + if args.cross_arch: + keys =3D base_keys + [k + "-" + args.cross_arch for k in base_= keys] + cross_keys =3D [k + "-cross-arch" for k in base_keys] + else: + keys =3D base_keys + [k + "-" + self._native_arch for k in bas= e_keys] + # We need to add the base project manually here: the standard # machinery hides it because it's an implementation detail for project in projects + ["base"]: for package in self._projects.get_packages(project): + policy =3D "native" + for key in cross_keys: + if key in mappings[package]: + policy =3D mappings[package][key] + if policy not in ["native", "foreign", "skip"]: + raise Error("Unexpected policy {} for {}", + policy, package) + for key in keys: if key in mappings[package]: pkgs[package] =3D mappings[package][key] =20 if package not in pkgs: continue - if pkgs[package] is None: + if policy =3D=3D "foreign" and pkgs[package] is not None: + cross_pkgs[package] =3D pkgs[package] + if pkgs[package] is None or policy in ["skip", "foreign"]: del pkgs[package] =20 print("FROM {}".format(facts["docker_base"])) =20 varmap =3D {} varmap["pkgs"] =3D " \\\n ".join(sorted(set(pkgs.values= ()))) + if args.cross_arch: + if package_format !=3D "deb": + raise Error("Cannot install foreign {} packages".format(pa= ckage_format)) + varmap["cross_arch"] =3D cross_build_facts["package_arch"] + pkg_names =3D [p + ":" + cross_build_facts["package_arch"] for= p in cross_pkgs.values()] + pkg_names.append(cross_build_facts["cross_gcc"]) + varmap["cross_pkgs"] =3D " \\\n ".join(sorted(set(p= kg_names))) + varmap["cross_target"] =3D cross_build_facts["abi"] if package_format =3D=3D "deb": sys.stdout.write(textwrap.dedent(""" RUN export DEBIAN_FRONTEND=3Dnoninteractive && \\ @@ -590,6 +632,25 @@ class Application: apt-get autoremove -y && \\ apt-get autoclean -y """).format(**varmap)) + if args.cross_arch: + # Intentionally a separate RUN command from the above + # so that the common packages of all cross-built images + # share a docker image layer. + sys.stdout.write(textwrap.dedent(""" + RUN export DEBIAN_FRONTEND=3Dnoninteractive && \\ + dpkg --add-architecture {cross_arch} && \\ + apt-get update && \\ + apt-get dist-upgrade -y && \\ + apt-get install --no-install-recommends -y \\ + {cross_pkgs} && \\ + apt-get autoremove -y && \\ + apt-get autoclean -y + + ENV TARGET "{cross_target}" + ENV CONFIGURE_OPTS "--host=3D{cross_target} \\ + --target=3D{cross_target}" + ENV PKG_CONFIG_LIBDIR "/usr/lib/{cross_target}/pkgconf= ig" + """).format(**varmap)) elif package_format =3D=3D "rpm": if os_name =3D=3D "Fedora" and os_version =3D=3D "Rawhide": sys.stdout.write(textwrap.dedent(""" diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index 4ca8fae..a70a6a5 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -57,11 +57,15 @@ # deb-x86_64: libxen-dev # Fedora-x86_64: xen-devel # +# In parallel with this 'cross-arch: native|foreign|skip' entries can +# used to indicate the policy when setting up a cross-architecture +# build environment. If omitted 'native' is assumed =20 mappings: =20 apparmor: deb: libapparmor-dev + deb-cross-arch: foreign =20 augeas: default: augeas @@ -80,6 +84,7 @@ mappings: =20 avahi: deb: libavahi-client-dev + deb-cross-arch: foreign pkg: avahi rpm: avahi-devel =20 @@ -106,6 +111,7 @@ mappings: =20 cyrus-sasl: deb: libsasl2-dev + deb-cross-arch: foreign pkg: cyrus-sasl rpm: cyrus-sasl-devel =20 @@ -116,6 +122,7 @@ mappings: =20 device-mapper: deb: libdevmapper-dev + deb-cross-arch: foreign rpm: device-mapper-devel =20 dnsmasq: @@ -124,6 +131,7 @@ mappings: =20 dtrace: deb: systemtap-sdt-dev + deb-cross-arch: skip rpm: systemtap-sdt-devel =20 dwarves: @@ -144,6 +152,7 @@ mappings: =20 fuse: deb: libfuse-dev + deb-cross-arch: foreign pkg: fusefs-libs rpm: fuse-devel =20 @@ -159,19 +168,23 @@ mappings: =20 glib2: deb: libglib2.0-dev + deb-cross-arch: foreign pkg: glib rpm: glib2-devel =20 glibc: deb: libc6-dev + deb-cross-arch: foreign rpm: glibc-devel =20 glibc-static: deb: libc6-dev + deb-cross-arch: foreign rpm: glibc-static =20 glusterfs: deb: libglusterfs-dev + deb-cross-arch: foreign rpm: glusterfs-api-devel Debian8: glusterfs-common Debian9: glusterfs-common @@ -183,6 +196,7 @@ mappings: =20 gnutls: deb: libgnutls28-dev + deb-cross-arch: foreign pkg: gnutls rpm: gnutls-devel =20 @@ -192,11 +206,13 @@ mappings: =20 gobject-introspection: deb: libgirepository1.0-dev + deb-cross-arch: foreign pkg: gobject-introspection rpm: gobject-introspection-devel =20 gtk3: deb: libgtk-3-dev + deb-cross-arch: foreign pkg: gtk3 rpm: gtk3-devel =20 @@ -211,6 +227,7 @@ mappings: =20 gtk-vnc2: deb: libgtk-vnc-2.0-dev + deb-cross-arch: foreign pkg: gtk-vnc rpm: gtk-vnc2-devel =20 @@ -243,32 +260,39 @@ mappings: =20 json-glib: deb: libjson-glib-dev + deb-cross-arch: foreign pkg: json-glib rpm: json-glib-devel =20 libacl: deb: libacl1-dev + deb-cross-arch: foreign rpm: libacl-devel =20 libarchive: deb: libarchive-dev + deb-cross-arch: foreign pkg: libarchive rpm: libarchive-devel =20 libattr: deb: libattr1-dev + deb-cross-arch: foreign rpm: libattr-devel =20 libaudit: deb: libaudit-dev + deb-cross-arch: foreign rpm: audit-libs-devel =20 libblkid: deb: libblkid-dev + deb-cross-arch: foreign rpm: libblkid-devel =20 libcap-ng: deb: libcap-ng-dev + deb-cross-arch: foreign rpm: libcap-ng-devel =20 libcmpiutil: @@ -276,67 +300,81 @@ mappings: =20 libconfig: deb: libconfig-dev + deb-cross-arch: foreign pkg: libconfig rpm: libconfig-devel =20 libcurl: deb: libcurl4-gnutls-dev + deb-cross-arch: foreign pkg: curl rpm: libcurl-devel =20 libdbus: deb: libdbus-1-dev + deb-cross-arch: foreign pkg: dbus rpm: dbus-devel =20 libgovirt: rpm: libgovirt-devel Debian: libgovirt-dev + Debian-cross-arch: foreign Debian8: =20 libiscsi: deb: libiscsi-dev + deb-cross-arch: foreign rpm: libiscsi-devel =20 libnl3: deb: libnl-3-dev + deb-cross-arch: foreign rpm: libnl3-devel =20 libnlroute3: deb: libnl-route-3-dev + deb-cross-arch: foreign rpm: libnl3-devel =20 libnuma: deb: libnuma-dev + deb-cross-arch: foreign deb-armv6l: deb-armv7l: rpm: numactl-devel =20 libparted: deb: libparted-dev + deb-cross-arch: foreign rpm: parted-devel =20 libpcap: deb: libpcap0.8-dev + deb-cross-arch: foreign pkg: libpcap rpm: libpcap-devel =20 libpciaccess: deb: libpciaccess-dev + deb-cross-arch: foreign pkg: libpciaccess rpm: libpciaccess-devel =20 librbd: deb: librbd-dev + deb-cross-arch: foreign Fedora: librbd-devel CentOS7: librbd1-devel =20 libselinux: deb: libselinux1-dev + deb-cross-arch: foreign rpm: libselinux-devel =20 libsoup: deb: libsoup2.4-dev + deb-cross-arch: foreign pkg: libsoup rpm: libsoup-devel =20 @@ -344,15 +382,18 @@ mappings: pkg: libssh rpm: libssh-devel Debian: libssh-gcrypt-dev + Debian-cross-arch: foreign Ubuntu: libssh-dev =20 libssh2: deb: libssh2-1-dev + deb-cross-arch: foreign pkg: libssh2 rpm: libssh2-devel =20 libtirpc: deb: libtirpc-dev + deb-cross-arch: foreign rpm: libtirpc-devel =20 libtool: @@ -364,20 +405,24 @@ mappings: =20 libudev: deb: libudev-dev + deb-cross-arch: foreign rpm: libudev-devel =20 libuuid: deb: uuid-dev + deb-cross-arch: foreign pkg: e2fsprogs-libuuid rpm: libuuid-devel =20 libxml2: deb: libxml2-dev + deb-cross-arch: foreign pkg: libxml2 rpm: libxml2-devel =20 libxslt: deb: libxslt1-dev + deb-cross-arch: foreign pkg: libxslt rpm: libxslt-devel =20 @@ -550,6 +595,7 @@ mappings: =20 netcf: deb: libnetcf-dev + deb-cross-arch: skip rpm: netcf-devel =20 numad: @@ -710,6 +756,7 @@ mappings: =20 python2-devel: deb: python-dev + deb-cross-arch: foreign pkg: python2 rpm: python2-devel =20 @@ -734,6 +781,7 @@ mappings: =20 python3-devel: deb: python3-dev + deb-cross-arch: foreign pkg: python3 Fedora: python3-devel =20 @@ -779,6 +827,7 @@ mappings: =20 readline: deb: libreadline-dev + deb-cross-arch: foreign pkg: readline rpm: readline-devel =20 @@ -793,6 +842,7 @@ mappings: =20 sanlock: deb: libsanlock-dev + deb-cross-arch: foreign rpm: sanlock-devel =20 screen: @@ -814,6 +864,7 @@ mappings: =20 spice-gtk3: deb: libspice-client-gtk-3.0-dev + deb-cross-arch: foreign pkg: spice-gtk rpm: spice-gtk3-devel =20 @@ -845,10 +896,12 @@ mappings: =20 wireshark: deb: wireshark-dev + deb-cross-arch: skip Fedora: wireshark-devel Debian8: =20 xen: + deb-cross-arch: foreign deb-x86_64: libxen-dev deb-armv7l: libxen-dev deb-aarch64: libxen-dev @@ -856,6 +909,7 @@ mappings: =20 xfsprogs: deb: xfslibs-dev + deb-cross-arch: foreign rpm: xfsprogs-devel =20 xmllint: @@ -868,14 +922,17 @@ mappings: =20 xz: deb: liblzma-dev + deb-cross-arch: foreign rpm: xz-devel =20 xz-static: deb: liblzma-dev + deb-cross-arch: foreign Fedora: xz-static =20 yajl: deb: libyajl-dev + deb-cross-arch: foreign pkg: yajl rpm: yajl-devel =20 @@ -886,8 +943,10 @@ mappings: =20 zlib: deb: zlib1g-dev + deb-cross-arch: foreign rpm: zlib-devel =20 zlib-static: deb: zlib1g-dev + deb-cross-arch: foreign rpm: zlib-static --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list