[Qemu-devel] [PULL for 3.0 00/17] docker fixes and tcg test tweak

Alex Bennée posted 17 patches 7 years, 3 months ago
Only 2 patches received!
tests/.gitignore                                   |   1 +
tests/docker/Makefile.include                      |  23 ++-
tests/docker/common.rc                             |  24 ++-
tests/docker/docker.py                             |  37 +++-
tests/docker/dockerfiles/debian-bootstrap.docker   |   1 +
tests/docker/dockerfiles/debian-bootstrap.pre      |  13 +-
.../docker/dockerfiles/debian-tricore-cross.docker |   6 +-
tests/docker/test-clang                            |   2 +-
tests/docker/test-debug                            |   2 +-
tests/docker/test-full                             |   2 +-
tests/docker/test-quick                            |   2 +-
tests/docker/test-unit                             |  21 +++
tests/tcg/i386/Makefile.target                     |   5 -
tests/tcg/i386/README                              |   3 -
tests/tcg/i386/pi_10.com                           | Bin 54 -> 0 bytes
tests/tcg/i386/runcom.c                            | 192 ---------------------
16 files changed, 111 insertions(+), 223 deletions(-)
create mode 100755 tests/docker/test-unit
delete mode 100644 tests/tcg/i386/pi_10.com
delete mode 100644 tests/tcg/i386/runcom.c
[Qemu-devel] [PULL for 3.0 00/17] docker fixes and tcg test tweak
Posted by Alex Bennée 7 years, 3 months ago
The following changes since commit 768cef2974fb1fa30dd188b043ea737e13fea477:

  Merge remote-tracking branch 'remotes/ehabkost/tags/x86-next-pull-request' into staging (2018-07-24 10:37:52 +0100)

are available in the Git repository at:

  https://github.com/stsquad/qemu.git tags/pull-docker-fixes-for-3.0-240718-1

for you to fetch changes up to 1a3bdc6111482fbb41eacacf850002e4589a81ef:

  tests/tcg: remove runcom test (2018-07-24 11:45:25 +0100)

----------------------------------------------------------------
docker fixes & tcg test tweak

  - graceful handling of testing under cross-compile
  - fixes for debootstrap handling
  - more helpful errors (binfmt_misc/EXECUTABLE missing)
  - drop runcom TCG test

----------------------------------------------------------------
Alex Bennée (16):
      tests/.gitignore: don't ignore docker tests
      docker: base debian-tricore on qemu:debian9
      docker: par down QEMU_CONFIGURE_OPTS in debian-tricore-cross
      docker: fail more gracefully on docker.py check
      docker: split configure_qemu from build_qemu
      docker: move make check into check_qemu helper
      docker: gracefully skip check_qemu
      docker: Makefile.include don't include partial images
      docker: add test-unit runner
      docker: add expansion for docker-test-FOO to Makefile.include
      docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set
      docker: report hint when docker.py check fails
      docker: add commentary to debian-bootstrap.docker
      docker: ignore distro versioning of debootstrap
      docker: perform basic binfmt_misc validation in docker.py
      tests/tcg: remove runcom test

Philippe Mathieu-Daudé (1):
      docker: Update debootstrap script after Debian migration from Alioth to Salsa

 tests/.gitignore                                   |   1 +
 tests/docker/Makefile.include                      |  23 ++-
 tests/docker/common.rc                             |  24 ++-
 tests/docker/docker.py                             |  37 +++-
 tests/docker/dockerfiles/debian-bootstrap.docker   |   1 +
 tests/docker/dockerfiles/debian-bootstrap.pre      |  13 +-
 .../docker/dockerfiles/debian-tricore-cross.docker |   6 +-
 tests/docker/test-clang                            |   2 +-
 tests/docker/test-debug                            |   2 +-
 tests/docker/test-full                             |   2 +-
 tests/docker/test-quick                            |   2 +-
 tests/docker/test-unit                             |  21 +++
 tests/tcg/i386/Makefile.target                     |   5 -
 tests/tcg/i386/README                              |   3 -
 tests/tcg/i386/pi_10.com                           | Bin 54 -> 0 bytes
 tests/tcg/i386/runcom.c                            | 192 ---------------------
 16 files changed, 111 insertions(+), 223 deletions(-)
 create mode 100755 tests/docker/test-unit
 delete mode 100644 tests/tcg/i386/pi_10.com
 delete mode 100644 tests/tcg/i386/runcom.c

-- 
2.17.1


Re: [Qemu-devel] [PULL for 3.0 00/17] docker fixes and tcg test tweak
Posted by Peter Maydell 7 years, 3 months ago
On 24 July 2018 at 11:50, Alex Bennée <alex.bennee@linaro.org> wrote:
> The following changes since commit 768cef2974fb1fa30dd188b043ea737e13fea477:
>
>   Merge remote-tracking branch 'remotes/ehabkost/tags/x86-next-pull-request' into staging (2018-07-24 10:37:52 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/stsquad/qemu.git tags/pull-docker-fixes-for-3.0-240718-1
>
> for you to fetch changes up to 1a3bdc6111482fbb41eacacf850002e4589a81ef:
>
>   tests/tcg: remove runcom test (2018-07-24 11:45:25 +0100)
>
> ----------------------------------------------------------------
> docker fixes & tcg test tweak
>
>   - graceful handling of testing under cross-compile
>   - fixes for debootstrap handling
>   - more helpful errors (binfmt_misc/EXECUTABLE missing)
>   - drop runcom TCG test
>

Applied, thanks.

-- PMM

[Qemu-devel] [PULL 01/17] tests/.gitignore: don't ignore docker tests
Posted by Alex Bennée 7 years, 3 months ago
The .gitignore was being a little over enthusiastic hiding files.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/.gitignore b/tests/.gitignore
index 08e2df1ce1..72c18aaab0 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -9,6 +9,7 @@ qht-bench
 rcutorture
 test-*
 !test-*.c
+!docker/test-*
 test-qapi-commands.[ch]
 test-qapi-events.[ch]
 test-qapi-types.[ch]
-- 
2.17.1


[Qemu-devel] [PULL 02/17] docker: base debian-tricore on qemu:debian9
Posted by Alex Bennée 7 years, 3 months ago
We need both git and a working compiler to build the tools. Although
the qemu:debian9 image also has a bunch of extra dependencies it would
be fairly unusual for a user not to already have this layer available
for one of our many other docker images so lets not complicate things.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 898b8dd511..180ca646c8 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -7,7 +7,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-FROM debian:9
+FROM qemu:debian9
 
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
-- 
2.17.1


[Qemu-devel] [PULL 03/17] docker: par down QEMU_CONFIGURE_OPTS in debian-tricore-cross
Posted by Alex Bennée 7 years, 3 months ago
This image isn't going to build anything significant as it is just
intended for building test cases. In case it does end up getting
inadvertently included in a build lets aim for the minimal possible
product.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 180ca646c8..4a0f7706a3 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -19,5 +19,5 @@ RUN git clone --single-branch \
     make && make install && \
     rm -rf /usr/src/binutils
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV QEMU_CONFIGURE_OPTS --cross-prefix=tricore-
+# This image isn't designed for building QEMU but building tests
+ENV QEMU_CONFIGURE_OPTS --disable-system --disable-user
-- 
2.17.1


[Qemu-devel] [PULL 04/17] docker: fail more gracefully on docker.py check
Posted by Alex Bennée 7 years, 3 months ago
As this is called directly from the Makefile while determining
dependencies and it is possible the user was configured in one window
but not have credentials in the other. Let's catch the Exceptions and
deal with it quietly.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 69e7130db7..2f81c6b13b 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -479,7 +479,12 @@ class CheckCommand(SubCommand):
     def run(self, args, argv):
         tag = args.tag
 
-        dkr = Docker()
+        try:
+            dkr = Docker()
+        except:
+            print("Docker not set up")
+            return 1
+
         info = dkr.inspect_tag(tag)
         if info is None:
             print("Image does not exist")
-- 
2.17.1


[Qemu-devel] [PULL 05/17] docker: split configure_qemu from build_qemu
Posted by Alex Bennée 7 years, 3 months ago
This allows some tests that just want to configure QEMU's source tree
to do so.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 046f8a5921..ba1f942328 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -21,7 +21,7 @@ requires()
     done
 }
 
-build_qemu()
+configure_qemu()
 {
     config_opts="--enable-werror \
                  ${TARGET_LIST:+--target-list=${TARGET_LIST}} \
@@ -32,6 +32,11 @@ build_qemu()
     echo $config_opts
     $QEMU_SRC/configure $config_opts || \
         { cat config.log && test_fail "Failed to run 'configure'"; }
+}
+
+build_qemu()
+{
+    configure_qemu $@
     make $MAKEFLAGS
 }
 
-- 
2.17.1


[Qemu-devel] [PULL 07/17] docker: gracefully skip check_qemu
Posted by Alex Bennée 7 years, 3 months ago
Not all our images are able to run the tests. Rather than use features
we can just check for the existence and run-ability of gtester. If the
image has been setup for binfmt_misc it will be able to run anyway.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 4ff5974016..4011561587 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -48,7 +48,13 @@ check_qemu()
     else
         INVOCATION="$@"
     fi
-    make $MAKEFLAGS $INVOCATION
+
+    if command -v gtester > /dev/null 2>&1 && \
+           gtester --version > /dev/null 2>&1; then
+        make $MAKEFLAGS $INVOCATION
+    else
+        echo "No working gtester, skipping make $INVOCATION"
+    fi
 }
 
 test_fail()
-- 
2.17.1


[Qemu-devel] [PULL 08/17] docker: Makefile.include don't include partial images
Posted by Alex Bennée 7 years, 3 months ago
Rename DOCKER_INTERMEDIATE_IMAGES to DOCKER_PARTIAL_IMAGES and add the
incomplete cross compiler images that can build tests but can't build
QEMU itself. We also add debian, debian-bootstrap and the tricode
images to the list.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index b2a7e761cc..09fb7db7fa 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -6,7 +6,7 @@ DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 DOCKER_DEPRECATED_IMAGES := debian
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports debian-sid
+DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian8-mxe debian-ports debian-sid debian-bootstrap
 DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
@@ -121,6 +121,11 @@ docker-image-travis: NOUSER=1
 # Specialist build images, sometimes very limited tools
 docker-image-tricore-cross: docker-image-debian9
 
+# These images may be good enough for building tests but not for test builds
+DOCKER_PARTIAL_IMAGES += debian-alpha-cross debian-hppa-cross debian-m68k-cross debian-sh4-cross
+DOCKER_PARTIAL_IMAGES += debian-sparc64-cross debian-mips64-cross debian-riscv64-cross
+DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-cross
+
 # Rules for building linux-user powered images
 #
 # These are slower than using native cross compiler setups but can
@@ -137,7 +142,7 @@ docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
+$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
-- 
2.17.1


[Qemu-devel] [PULL 09/17] docker: add test-unit runner
Posted by Alex Bennée 7 years, 3 months ago
This test doesn't even build QEMU, it just builds and runs all the
unit tests. Intended to make checking unit tests on all docker images
easier.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/test-unit b/tests/docker/test-unit
new file mode 100755
index 0000000000..8905d01150
--- /dev/null
+++ b/tests/docker/test-unit
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+#
+# Build and run the unit tests
+#
+# Copyright (c) 2018 Linaro Ltd.
+#
+# Authors:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+cd "$BUILD_DIR"
+
+# although we are not building QEMU itself we still need a configured
+# build for the unit tests to be built and run
+configure_qemu
+check_qemu check-unit
-- 
2.17.1


[Qemu-devel] [PULL 10/17] docker: add expansion for docker-test-FOO to Makefile.include
Posted by Alex Bennée 7 years, 3 months ago
This allows us to run a particular test on all docker images. For
example:

  make docker-test-unit

Will run the unit tests on every supported image. At the same time
rename docker-test to docker-all-tests to be clearer.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 09fb7db7fa..8fbb076396 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -148,7 +148,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
 	) \
 	$(foreach t,$(DOCKER_TESTS), \
-		$(eval docker-test: docker-$t@$i) \
+		$(eval docker-all-tests: docker-$t@$i) \
+		$(eval docker-$t: docker-$t@$i) \
 	) \
 )
 
@@ -158,7 +159,8 @@ docker:
 	@echo 'Available targets:'
 	@echo
 	@echo '    docker:              Print this help.'
-	@echo '    docker-test:         Run all image/test combinations.'
+	@echo '    docker-all-tests:    Run all image/test combinations.'
+	@echo '    docker-TEST:         Run TEST on all image combinations.'
 	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
 	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
 	@echo '                         Note: "TEST" is one of the listed test name,'
-- 
2.17.1


[Qemu-devel] [PULL 11/17] docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set
Posted by Alex Bennée 7 years, 3 months ago
The addition of QEMU_TARGET was intended to ensure we fall back to
checking for the existence of an image if the build system was not
currently configured to build it. However this breaks the direct use
of the rule for building custom binfmt_misc images. We already check
for EXECUTABLE so let us just use that as a proxy for deciding if we
are just going to check the image exits.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 8fbb076396..05afeb64a7 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -58,13 +58,11 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 	$(if $(EXECUTABLE),,\
 		$(error EXECUTABLE not set, debootstrap of debian-$* would fail))
-	$(if $(wildcard $(EXECUTABLE)),,\
-		$(error Please build $(EXECUTABLE) first))
 	$(if $(DEB_ARCH),,\
 		$(error DEB_ARCH not set, debootstrap of debian-$* would fail))
 	$(if $(DEB_TYPE),,\
 		$(error DEB_TYPE not set, debootstrap of debian-$* would fail))
-	$(if $(filter $(QEMU_TARGET),$(TARGET_DIRS)),				\
+	$(if $(wildcard $(EXECUTABLE)),						\
 		$(call quiet-command,						\
 			DEB_ARCH=$(DEB_ARCH)					\
 			DEB_TYPE=$(DEB_TYPE)					\
@@ -136,7 +134,6 @@ DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-c
 # broken so we need a qemu-linux-user for this target
 docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
 docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
-docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
 docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
 docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
-- 
2.17.1


[Qemu-devel] [PULL 12/17] docker: report hint when docker.py check fails
Posted by Alex Bennée 7 years, 3 months ago
When a check fails we currently just report why we failed. This is not
totally helpful to people who want to boot-strap a new image. Report a
hint as to why it failed.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Suggested-by: Fam Zheng <famz@redhat.com>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 05afeb64a7..1aaa795743 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
 			"BUILD","binfmt debian-$* (debootstrapped)"),		\
 		$(call quiet-command,						\
-			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,	\
+			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $< || 	\
+			{ echo "You will need to build $(EXECUTABLE)"; exit 1;},\
 			"CHECK", "debian-$* exists"))
 
 endif
-- 
2.17.1


[Qemu-devel] [PULL 13/17] docker: Update debootstrap script after Debian migration from Alioth to Salsa
Posted by Alex Bennée 7 years, 3 months ago
From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This silents the following warning:

  Cloning into './debootstrap.git'...
  warning: redirecting to https://salsa.debian.org/installer-team/debootstrap.git/

See https://lists.debian.org/debian-devel-announce/2018/01/msg00004.html

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 56e1aa7a21..ea324d6e4a 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -62,7 +62,7 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
         NEED_DEBOOTSTRAP=true
     fi
     if $NEED_DEBOOTSTRAP; then
-        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
+        DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
         git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
         export DEBOOTSTRAP_DIR=./debootstrap.git
         DEBOOTSTRAP=./debootstrap.git/debootstrap
-- 
2.17.1


[Qemu-devel] [PULL 14/17] docker: add commentary to debian-bootstrap.docker
Posted by Alex Bennée 7 years, 3 months ago
This is just a note that later versions of debootstrap don't
technically need this hack.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
index 14212b9cf4..e13c26a7ed 100644
--- a/tests/docker/dockerfiles/debian-bootstrap.docker
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -9,6 +9,7 @@ FROM scratch
 ADD . /
 
 # Patch all mounts as docker already has stuff set up
+# (this is not needed for later debootstraps but is harmless atm)
 RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
 
 # Run stage 2
-- 
2.17.1


[Qemu-devel] [PULL 15/17] docker: ignore distro versioning of debootstrap
Posted by Alex Bennée 7 years, 3 months ago
We do a minimum version check for the debootstrap but if the distro
has added their own minor version tick it would fail and fall-back to
the SCM version. This is sub-optimal as the latest/greatest version
may be broken at any one particular time. We fix that with a little
sed magic on the version string before passing to our ugly shell
versioning check.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index ea324d6e4a..3b0ef95374 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -56,10 +56,13 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
     if [ -z $DEBOOTSTRAP ]; then
         echo "No debootstrap installed, attempting to install from SCM"
         NEED_DEBOOTSTRAP=true
-    elif ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; "${DEBOOTSTRAP}" --version \
-            | cut -d ' ' -f 2) | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -c &>/dev/null; then
-        echo "debootstrap too old, attempting to install from SCM"
-        NEED_DEBOOTSTRAP=true
+    else
+        INSTALLED_VERSION=$(${DEBOOTSTRAP} --version | sed 's/debootstrap \([0-9\.]*\)[^0-9\.]*.*/\1/')
+        if ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; echo "${INSTALLED_VERSION}") \
+                | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -C ; then
+            echo "debootstrap too old, attempting to install from SCM"
+            NEED_DEBOOTSTRAP=true
+        fi
     fi
     if $NEED_DEBOOTSTRAP; then
         DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
-- 
2.17.1


[Qemu-devel] [PULL 16/17] docker: perform basic binfmt_misc validation in docker.py
Posted by Alex Bennée 7 years, 3 months ago
Setting up binfmt_misc is outside of the scope of the docker.py script
but we can at least validate it with any given executable so we have a
more useful error message than the sed line of deboostrap failing
cryptically.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Richard Henderson <richard.henderson@linaro.org>

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 2f81c6b13b..d3006d4dae 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -112,6 +112,31 @@ def _copy_binary_with_libs(src, dest_dir):
             so_path = os.path.dirname(l)
             _copy_with_mkdir(l , dest_dir, so_path)
 
+
+def _check_binfmt_misc(executable):
+    """Check binfmt_misc has entry for executable in the right place.
+
+    The details of setting up binfmt_misc are outside the scope of
+    this script but we should at least fail early with a useful
+    message if it won't work."""
+
+    binary = os.path.basename(executable)
+    binfmt_entry = "/proc/sys/fs/binfmt_misc/%s" % (binary)
+
+    if not os.path.exists(binfmt_entry):
+        print ("No binfmt_misc entry for %s" % (binary))
+        return False
+
+    with open(binfmt_entry) as x: entry = x.read()
+
+    qpath = "/usr/bin/%s" % (binary)
+    if not re.search("interpreter %s\n" % (qpath), entry):
+        print ("binfmt_misc for %s does not point to %s" % (binary, qpath))
+        return False
+
+    return True
+
+
 def _read_qemu_dockerfile(img_name):
     # special case for Debian linux-user images
     if img_name.startswith("debian") and img_name.endswith("user"):
@@ -315,6 +340,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Validate binfmt_misc will work
+            if args.include_executable:
+                if not _check_binfmt_misc(args.include_executable):
+                    return 1
+
             # Is there a .pre file to run in the build context?
             docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
             if os.path.exists(docker_pre):
-- 
2.17.1