Debian's filesystem layout has a nice advantage over Fedora which is
that it can install non-native RPMs in the main root filesystem. It is
thus possible to prepare an x86_64 filesystem containing -dev packages
for a foreign architecture, along with a GCC cross compiler.

QEMU has used this technique to facilitate developer build testing of
non-x86 architectures, since few people have access to physical
hardware for most of these architectures. For the same reason it would
be helpful to libvirt developers.

This docker file allows creation of containers  buildenv-cross-$ARCH
for arm64, armel, armhf, mips, mipsel, mips64el, ppc64el, s390x, based
off Debian 9, in which libvirt can be cross-compiled. Debian can support
further architectures, but only by using the unstable 'sid' repos. This
does not enable use of 'sid' but it could be added fairly easily at a
later date if desired. Most of the other architectures are fairly
old & obscure, the main interesting exception being riscv64.

If qemu-user emulators are registered with the kernel binfmt, then it
is theoretically possible to run "make check" tool, though this has not
been tested.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitignore                           |  1 +
 cross-build/arch-config.csv          |  8 +++
 cross-build/buildenv-cross.docker.in | 96 ++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+)
 create mode 100644 cross-build/arch-config.csv
 create mode 100644 cross-build/buildenv-cross.docker.in

diff --git a/.gitignore b/.gitignore
index b25c15b..4884352 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,2 @@
 *~
+cross-build/*docker
diff --git a/cross-build/arch-config.csv b/cross-build/arch-config.csv
new file mode 100644
index 0000000..36fcb5f
--- /dev/null
+++ b/cross-build/arch-config.csv
@@ -0,0 +1,8 @@
+arm64,crossbuild-essential-arm64,aarch64-linux-gnu
+armel,crossbuild-essential-armel,arm-linux-gnueabi
+armhf,crossbuild-essential-armhf,arm-linux-gnueabihf
+mips64el,gcc-mips64el-linux-gnuabi64,mips64el-linux-gnuabi64
+mips,gcc-mips-linux-gnu,mips-linux-gnu
+mipsel,gcc-mipsel-linux-gnu,mipsel-linux-gnu
+ppc64el,crossbuild-essential-ppc64el,powerpc64le-linux-gnu
+s390x,gcc-multilib-s390x-linux-gnu,s390x-linux-gnu
diff --git a/cross-build/buildenv-cross.docker.in b/cross-build/buildenv-cross.docker.in
new file mode 100644
index 0000000..97a9e69
--- /dev/null
+++ b/cross-build/buildenv-cross.docker.in
@@ -0,0 +1,96 @@
+#
+# ::ARCH:: cross-compiler target
+#
+FROM debian:stretch-slim
+
+# Grab the updated list of packages
+RUN apt update && apt dist-upgrade -yy
+
+# First some common native binaries. Any program that the
+# build will execute needs to have a native binary package
+# installed. We don't want to rely on qemu-user emulation
+# for executing non-native binaries as that is slow & flaky
+RUN DEBIAN_FRONTEND=noninteractive  \
+    apt-get install -y --no-install-recommends \
+        git \
+        build-essential \
+        ca-certificates \
+        libtool\
+        autoconf \
+        automake \
+        python3 \
+        xsltproc \
+        parted \
+        radvd \
+        qemu-system-common \
+        dwarves \
+        libxml2-utils \
+        dnsmasq-base \
+        openssh-client \
+        netcat-openbsd \
+        ebtables \
+        iptables \
+        sheepdog \
+        qemu-utils \
+        qemu-system-x86 \
+        pkg-config
+
+
+# Add the ::ARCH:: architecture
+RUN dpkg --add-architecture ::ARCH::
+
+
+# Grab the freshly updated list of packages
+RUN apt update
+
+
+# Pull in the native package for the cross-compiler
+RUN DEBIAN_FRONTEND=noninteractive  \
+    apt-get install -y --no-install-recommends \
+        ::COMPILER::
+
+# Finally pull in the foreign architecture library
+# packages that we're going to link against. Everything
+# here should be a -dev package. Binaries belong earlier
+RUN DEBIAN_FRONTEND=noninteractive  \
+    apt-get install -y --no-install-recommends \
+        libxml2-dev:::ARCH:: \
+        libncurses5-dev:::ARCH:: \
+        libreadline-dev:::ARCH:: \
+        zlib1g-dev:::ARCH:: \
+        libgcrypt20-dev:::ARCH:: \
+        libgnutls28-dev:::ARCH:: \
+        libavahi-client-dev:::ARCH:: \
+        libsasl2-dev:::ARCH:: \
+        libparted-dev:::ARCH:: \
+        libdevmapper-dev:::ARCH:: \
+        uuid-dev:::ARCH:: \
+        libudev-dev:::ARCH:: \
+        libpciaccess-dev:::ARCH:: \
+        libpolkit-gobject-1-dev:::ARCH:: \
+        libcap-ng-dev:::ARCH:: \
+        libnl-3-dev:::ARCH:: \
+        libnl-route-3-dev:::ARCH:: \
+        libyajl-dev:::ARCH:: \
+        libpcap0.8-dev:::ARCH:: \
+        libsanlock-dev:::ARCH:: \
+        libaudit-dev:::ARCH:: \
+        libselinux1-dev:::ARCH:: \
+        libapparmor-dev:::ARCH:: \
+        libdbus-1-dev:::ARCH:: \
+        librbd-dev:::ARCH:: \
+        librados-dev:::ARCH:: \
+        libfuse-dev:::ARCH:: \
+        libssh2-1-dev:::ARCH:: \
+        libcurl4-gnutls-dev:::ARCH:: \
+        libacl1-dev:::ARCH:: \
+        libiscsi-dev:::ARCH:: \
+        xfslibs-dev:::ARCH:: \
+        libssh-dev:::ARCH:: \
+        libgnutls28-dev:::ARCH::
+
+# Tell libvirt which architecture we want to target
+ENV LIBVIRT_CONFIGURE_OPTS "--host=::ABI:: --build=x86_64-linux-gnu --target=::ABI::"
+
+# Tell pkg-config where to find the non-native .pc files
+ENV PKG_CONFIG_LIBDIR=/usr/lib/::ABI::/pkgconfig
-- 
2.20.1

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Mon, 2019-01-28 at 12:03 +0000, Daniel P. Berrangé wrote:
[...]
>  cross-build/arch-config.csv          |  8 +++
>  cross-build/buildenv-cross.docker.in | 96 ++++++++++++++++++++++++++++
>  3 files changed, 105 insertions(+)
>  create mode 100644 cross-build/arch-config.csv
>  create mode 100644 cross-build/buildenv-cross.docker.in

You're providing a Dockerfile template and a CSV file containing
the data that should be substituted into it, and it's fairly obvious
how one would go about it, but I would expect you to provide a
script and possibly a Makefile along with this... I assume you just
forgot to commit them? :)

[...]
> +++ b/cross-build/arch-config.csv
> @@ -0,0 +1,8 @@
> +arm64,crossbuild-essential-arm64,aarch64-linux-gnu
> +armel,crossbuild-essential-armel,arm-linux-gnueabi
> +armhf,crossbuild-essential-armhf,arm-linux-gnueabihf
> +mips64el,gcc-mips64el-linux-gnuabi64,mips64el-linux-gnuabi64
> +mips,gcc-mips-linux-gnu,mips-linux-gnu
> +mipsel,gcc-mipsel-linux-gnu,mipsel-linux-gnu
> +ppc64el,crossbuild-essential-ppc64el,powerpc64le-linux-gnu
> +s390x,gcc-multilib-s390x-linux-gnu,s390x-linux-gnu

The file format is not documented, and it really should be.

As for the architecture selection, we should drop armel (it's
unlikely to be part of the next Debian release[1]) and possibly
all the MIPS architectures too? I'm fairly sure nobody is actually
running and testing libvirt on those architectures, so making them
official targets for cross-compilation would be a bit dishonest in
my opinion. Same possibly goes for armhf, but it's less clear-cut
given how many Raspberry Pis are out there.

We should also use the virArch names instead of the Debian names,
even if that means having additional logic to map between them.

[...]
> +++ b/cross-build/buildenv-cross.docker.in
> @@ -0,0 +1,96 @@
> +#
> +# ::ARCH:: cross-compiler target
> +#
> +FROM debian:stretch-slim

How much smaller is the -slim variant than the regular stretch
image? Because according to the documentation

  These tags are an experiment in providing a slimmer base (removing
  some extra files that are normally not necessary within containers,
  such as man pages and documentation), and are definitely subject
  to change.

so unless the savings are very significant, I'd much rather
stick with the regular images.

[...]
> +# First some common native binaries. Any program that the
> +# build will execute needs to have a native binary package
> +# installed. We don't want to rely on qemu-user emulation
> +# for executing non-native binaries as that is slow & flaky
> +RUN DEBIAN_FRONTEND=noninteractive  \
> +    apt-get install -y --no-install-recommends \

I never considered it until now, but we might want to use
DEBIAN_FRONTEND=noninteractive and --no-install-recommends in our
existing Debian/Ubuntu Dockerfiles as well :)

[...]
> +        qemu-system-x86 \

Based on what you wrote just a few lines above, QEMU shouldn't be
necessary, so why are you dragging it in?

[...]
> +# Finally pull in the foreign architecture library
> +# packages that we're going to link against. Everything
> +# here should be a -dev package. Binaries belong earlier
> +RUN DEBIAN_FRONTEND=noninteractive  \
> +    apt-get install -y --no-install-recommends \
> +        libxml2-dev:::ARCH:: \

Given that the format is package:architecture, I'd say ::VAR:: is
a pretty poor choice when it comes to marking stuff for
substitution... I suggest using good old @VAR@ instead, so the
above would look like

  libxml2-dev:@ARCH@

which is much clearer.

[...]
> +        xfslibs-dev:::ARCH:: \
> +        libssh-dev:::ARCH:: \
> +        libgnutls28-dev:::ARCH::

How did you come up with this list of packages, anyway? It doesn't
match the one in buildenv-debian-9.Dockerfile from the
libvirt-dockerfiles repository.

On a more general note, we shouldn't have such a list at all: the
source of truth for build dependencies are the YAML files in
guests/vars/, and anything that needs it should obtain it from
there.

Which leads me to an even more general point: this should be part
of lcitool instead of its own thing. We already have (admittedly
not great) code to generate Dockerfiles in there; cleaning it up
and extending it so that it can support this additional variant of
the same idea shouldn't be too difficult.

> +# Tell libvirt which architecture we want to target
> +ENV LIBVIRT_CONFIGURE_OPTS "--host=::ABI:: --build=x86_64-linux-gnu --target=::ABI::"
> +
> +# Tell pkg-config where to find the non-native .pc files
> +ENV PKG_CONFIG_LIBDIR=/usr/lib/::ABI::/pkgconfig

I really don't like injecting this much environment into the
container. Can we limit it to a single $ABI variable, or perhaps
even store it on disk as, I don't know, /root/abi or something?
Either way, we could pick up the information and prepare the above
environment as part of the build process, eg. on the libvirt side.


[1] https://release.debian.org/buster/arch_qualify.html
-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Mon, Jan 28, 2019 at 04:13:07PM +0100, Andrea Bolognani wrote:
> On Mon, 2019-01-28 at 12:03 +0000, Daniel P. Berrangé wrote:
> [...]
> >  cross-build/arch-config.csv          |  8 +++
> >  cross-build/buildenv-cross.docker.in | 96 ++++++++++++++++++++++++++++
> >  3 files changed, 105 insertions(+)
> >  create mode 100644 cross-build/arch-config.csv
> >  create mode 100644 cross-build/buildenv-cross.docker.in
> 
> You're providing a Dockerfile template and a CSV file containing
> the data that should be substituted into it, and it's fairly obvious
> how one would go about it, but I would expect you to provide a
> script and possibly a Makefile along with this... I assume you just
> forgot to commit them? :)

Sigh, yes I forgot the Makefile.

> [...]
> > +++ b/cross-build/arch-config.csv
> > @@ -0,0 +1,8 @@
> > +arm64,crossbuild-essential-arm64,aarch64-linux-gnu
> > +armel,crossbuild-essential-armel,arm-linux-gnueabi
> > +armhf,crossbuild-essential-armhf,arm-linux-gnueabihf
> > +mips64el,gcc-mips64el-linux-gnuabi64,mips64el-linux-gnuabi64
> > +mips,gcc-mips-linux-gnu,mips-linux-gnu
> > +mipsel,gcc-mipsel-linux-gnu,mipsel-linux-gnu
> > +ppc64el,crossbuild-essential-ppc64el,powerpc64le-linux-gnu
> > +s390x,gcc-multilib-s390x-linux-gnu,s390x-linux-gnu
> 
> The file format is not documented, and it really should be.
> 
> As for the architecture selection, we should drop armel (it's
> unlikely to be part of the next Debian release[1]) and possibly
> all the MIPS architectures too? I'm fairly sure nobody is actually
> running and testing libvirt on those architectures, so making them
> official targets for cross-compilation would be a bit dishonest in
> my opinion. Same possibly goes for armhf, but it's less clear-cut
> given how many Raspberry Pis are out there.

I don't want to take a stance on popularity of the architectures.
The goal was to provide containers for every architecture that
Debian currently builds libvirt, to ensure that libvirt does not
regress in its ability to build on anything that Debian targets.

> We should also use the virArch names instead of the Debian names,
> even if that means having additional logic to map between them.

I don't think virArch names offer any benefit, only the pain of
having to remap them to what we actually want to work with.

> > +++ b/cross-build/buildenv-cross.docker.in
> > @@ -0,0 +1,96 @@
> > +#
> > +# ::ARCH:: cross-compiler target
> > +#
> > +FROM debian:stretch-slim
> 
> How much smaller is the -slim variant than the regular stretch
> image? Because according to the documentation
> 
>   These tags are an experiment in providing a slimmer base (removing
>   some extra files that are normally not necessary within containers,
>   such as man pages and documentation), and are definitely subject
>   to change.
> 
> so unless the savings are very significant, I'd much rather
> stick with the regular images.

It depends what you want to measure. The debian:9-slim  is ~55 MB
while debian:9 is ~100MB (varies per arch).  IOW, the -slim variant
is half the size.

Once you add in the libvirt build deps though this difference is
significantly reduced, as the overall libvirt image is around
1000 MB for debian:9 vs  950 MB for debian:9-slim

> > +# First some common native binaries. Any program that the
> > +# build will execute needs to have a native binary package
> > +# installed. We don't want to rely on qemu-user emulation
> > +# for executing non-native binaries as that is slow & flaky
> > +RUN DEBIAN_FRONTEND=noninteractive  \
> > +    apt-get install -y --no-install-recommends \
> 
> I never considered it until now, but we might want to use
> DEBIAN_FRONTEND=noninteractive and --no-install-recommends in our
> existing Debian/Ubuntu Dockerfiles as well :)

Yes, we should.

> [...]
> > +        qemu-system-x86 \
> 
> Based on what you wrote just a few lines above, QEMU shouldn't be
> necessary, so why are you dragging it in?

The comments above were about the qemu userspace emulators,
while this is the system emulator. configure.ac checks for
existance of qemu-system-x86, though this is merely to
warn you if you try to build a QEMU version that needs QMP
while not having yajl installed. So that's not really
required. We should in fact update that check in libvirt.git
really since we mandate yajl for QEMU now, thre's no need
to check QEMU versions.

> > +        xfslibs-dev:::ARCH:: \
> > +        libssh-dev:::ARCH:: \
> > +        libgnutls28-dev:::ARCH::
> 
> How did you come up with this list of packages, anyway? It doesn't
> match the one in buildenv-debian-9.Dockerfile from the
> libvirt-dockerfiles repository.

It comes from the list of build requires listed for the
debian source package

  https://packages.debian.org/source/sid/libvirt

> On a more general note, we shouldn't have such a list at all: the
> source of truth for build dependencies are the YAML files in
> guests/vars/, and anything that needs it should obtain it from
> there.
> 
> Which leads me to an even more general point: this should be part
> of lcitool instead of its own thing. We already have (admittedly
> not great) code to generate Dockerfiles in there; cleaning it up
> and extending it so that it can support this additional variant of
> the same idea shouldn't be too difficult.

It was not entirely straightforward, as it turns out we had
some bugs in our existing package set that caused conflicts
with cross-arch installs, but I managed to resolve this.

> 
> > +# Tell libvirt which architecture we want to target
> > +ENV LIBVIRT_CONFIGURE_OPTS "--host=::ABI:: --build=x86_64-linux-gnu --target=::ABI::"
> > +
> > +# Tell pkg-config where to find the non-native .pc files
> > +ENV PKG_CONFIG_LIBDIR=/usr/lib/::ABI::/pkgconfig
> 
> I really don't like injecting this much environment into the
> container. Can we limit it to a single $ABI variable, or perhaps
> even store it on disk as, I don't know, /root/abi or something?
> Either way, we could pick up the information and prepare the above
> environment as part of the build process, eg. on the libvirt side.

I don't really see any downside to using these env vars as is.
This is pretty much a standard use for use of docker's ENV feature
and makes it easy to use these containers in the correct manner.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list