[Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets

Philippe Mathieu-Daudé posted 21 patches 6 years, 10 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170508221759.15616-1-f4bug@amsat.org
Test checkpatch passed
Test docker passed
Test s390x passed
There is a newer version of this series
.shippable.yml                                     | 23 +++++++++--
MAINTAINERS                                        |  2 +
tests/docker/Makefile.include                      |  8 +++-
tests/docker/docker.py                             |  7 +++-
tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
tests/docker/dockerfiles/debian-arm64-cross.docker | 16 ++++++--
tests/docker/dockerfiles/debian-armhf-cross.docker | 16 ++++++--
.../docker/dockerfiles/debian-mipsel-cross.docker  | 31 +++++++++++++++
.../docker/dockerfiles/debian-powerpc-cross.docker | 31 +++++++++++++++
tests/docker/dockerfiles/debian-s390x-cross.docker |  6 +++
tests/docker/dockerfiles/debian.docker             | 26 ++++++------
11 files changed, 186 insertions(+), 26 deletions(-)
create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
[Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets
Posted by Philippe Mathieu-Daudé 6 years, 10 months ago
This patchset add 2 more architectures to the cross-build farm.

There is still some issue trying to cross-build mips64el-softmmu, it seems the
cross tools use the system outdated libfdt instead of the one checkouted in the
dtc submodule. I disabled this target for now.

The branch https://github.com/philmd/qemu/tree/docker_shippable_v2 can be
checked at Shippable:
  https://app.shippable.com/github/philmd/qemu/status/dashboard

With the free open source projects service, each arch builds in around ~8min,
the 5 jobs take ~38min in total.

v2:
- Addressed review feedback from Alex, added his R-B
- Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
  installing the ca-certificates package.
- Squashed/split some commits
- use 'apt-get clean' instead of brutal 'rm -rf'
- checked how many cores are available on Shippable and use them fully
  (reduced total time from 44min to 38min)
- build armeb-linux-user target

v1:
- Reorganize Dockerfiles to use less layers, resulting in smaller images.
  This also reduce time of image transfer, for example while using:
  `docker save qemu:debian | ssh remote docker load`
- Install more debian packages so the configure script enable more features and
  more code can be compiled/covered.
- There are still some incorrect multiarch packages on debian/jessie used in
  the docker images, add a script to generate fake packages and avoid
  dependencies issues.
- Modify the docker.py script to include an extra file while building images.

Regards,

Phil.

Philippe Mathieu-Daudé (21):
  docker: let _copy_with_mkdir() sub_path argument be optional
  docker: add --include-file argument to 'build' command
  docker: add 'apt-fake' script which generate fake debian packages
  docker: install ca-certificates package in base image
  docker: compact debian base
  docker: compact debian armhf
  docker: compact debian arm64
  docker: add extra libs to armhf target to extend codebase coverage
  docker: add extra libs to arm64 target to extend codebase coverage
  docker: add extra libs to s390x target to extend codebase coverage
  docker: add mipsel build target
  docker: add powerpc build target
  shippable: use C locale to simplify console output
  shippable: build using all available cpus
  shippable: do not initialize submodules automatically
  shippable: be verbose while building docker images
  shippable: add armeb-linux-user target
  shippable: add powerpc target
  shippable: add mipsel target
  MAINTAINERS: add Shippable automation platform URL
  MAINTAINERS: self-appoint me as reviewer in build/test automation

 .shippable.yml                                     | 23 +++++++++--
 MAINTAINERS                                        |  2 +
 tests/docker/Makefile.include                      |  8 +++-
 tests/docker/docker.py                             |  7 +++-
 tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
 tests/docker/dockerfiles/debian-arm64-cross.docker | 16 ++++++--
 tests/docker/dockerfiles/debian-armhf-cross.docker | 16 ++++++--
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 31 +++++++++++++++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 31 +++++++++++++++
 tests/docker/dockerfiles/debian-s390x-cross.docker |  6 +++
 tests/docker/dockerfiles/debian.docker             | 26 ++++++------
 11 files changed, 186 insertions(+), 26 deletions(-)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

-- 
2.11.0


Re: [Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets
Posted by Alex Bennée 6 years, 10 months ago
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> This patchset add 2 more architectures to the cross-build farm.
>
> There is still some issue trying to cross-build mips64el-softmmu, it seems the
> cross tools use the system outdated libfdt instead of the one checkouted in the
> dtc submodule. I disabled this target for now.
>
> The branch https://github.com/philmd/qemu/tree/docker_shippable_v2 can be
> checked at Shippable:
>   https://app.shippable.com/github/philmd/qemu/status/dashboard
>
> With the free open source projects service, each arch builds in around ~8min,
> the 5 jobs take ~38min in total.
>
> v2:
> - Addressed review feedback from Alex, added his R-B
> - Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
>   installing the ca-certificates package.
> - Squashed/split some commits
> - use 'apt-get clean' instead of brutal 'rm -rf'
> - checked how many cores are available on Shippable and use them fully
>   (reduced total time from 44min to 38min)
> - build armeb-linux-user target

Ping - is there a v3 coming with Fam's comments addressed?

<snip>

--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets
Posted by Philippe Mathieu-Daudé 6 years, 10 months ago
Hi Alex,

On 05/17/2017 11:28 AM, Alex Bennée wrote:
>> v2:
>> - Addressed review feedback from Alex, added his R-B
>> - Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
>>   installing the ca-certificates package.
>> - Squashed/split some commits
>> - use 'apt-get clean' instead of brutal 'rm -rf'
>> - checked how many cores are available on Shippable and use them fully
>>   (reduced total time from 44min to 38min)
>> - build armeb-linux-user target
>
> Ping - is there a v3 coming with Fam's comments addressed?

I did address Fam's comment, I'm not sure about how Fam wanted to 
document "where" extra files are copied to in the container's filesystem 
but tried to improve what was here.

There is an issue when you add files with the ADD directive, if any file 
is changed the script generates the same SHA1 hash and avoid to rebuild 
the image, I added some lines to also consider per-file checksum, but 
not for the shlibs case (I let a FIXME entry for that).

I then got stuck trying to improve `--include-executable`, I think the 
correct way would be to use ldd to get linked libraries from the host, 
then try to install them in the guest from the distrib packages, and 
finally copy what can't be find in the guest.
But better let it for another serie.

Regards,

Phil.