This patch series is a first attempt to check if we could use Yocto in
gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is making sure build-yocto.sh is not catched by
gitignore.

The second patch is creating a container with all elements required to
build Yocto, a checkout of the yocto layers required and an helper
script to build and run xen on qemu with yocto.

The third patch is creating containers with a first build of yocto done
so that susbsequent build with those containers would only rebuild what
was changed and take the rest from the cache.

The fourth patch is adding a way to easily clean locally created
containers.

This is is mainly for discussion and sharing as there are still some
issues/problem to solve:
- building the qemu* containers can take several hours depending on the
  network bandwith and computing power of the machine where those are
  created
- produced containers containing the cache have a size between 8 and
  12GB depending on the architecture. We might need to store the build
  cache somewhere else to reduce the size. If we choose to have one
  single image, the needed size is around 20GB and we need up to 40GB
  during the build, which is why I splitted them.
- during the build and run, we use a bit more then 20GB of disk which is
  over the allowed size in gitlab

Once all problems passed, this can be used to build and run dom0 on qemu
with a modified Xen on the 3 archs in less than 10 minutes.

This has been tested on a x86 host machine and on an arm host machine
(with mk_dsdt.c fix).

Bertrand Marquis (4):
  automation: Only filter build-*/ in gitignore
  automation: Add elements for Yocto test and run
  automation: Add yocto containers with cache
  automation: Add a clean rule for containers

 .gitignore                                    |   2 +-
 automation/build/Makefile                     |  19 +-
 automation/build/yocto/build-yocto.sh         | 328 ++++++++++++++++++
 .../build/yocto/kirkstone-qemuarm.dockerfile  |  28 ++
 .../yocto/kirkstone-qemuarm64.dockerfile      |  28 ++
 .../yocto/kirkstone-qemux86-64.dockerfile     |  28 ++
 automation/build/yocto/kirkstone.dockerfile   | 100 ++++++
 7 files changed, 531 insertions(+), 2 deletions(-)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/kirkstone-qemuarm.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemuarm64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone-qemux86-64.dockerfile
 create mode 100644 automation/build/yocto/kirkstone.dockerfile

-- 
2.25.1

On Wed, 24 Aug 2022, Bertrand Marquis wrote:
> This patch series is a first attempt to check if we could use Yocto in
> gitlab ci to build and run xen on qemu for arm, arm64 and x86.
> 
> The first patch is making sure build-yocto.sh is not catched by
> gitignore.
> 
> The second patch is creating a container with all elements required to
> build Yocto, a checkout of the yocto layers required and an helper
> script to build and run xen on qemu with yocto.
> 
> The third patch is creating containers with a first build of yocto done
> so that susbsequent build with those containers would only rebuild what
> was changed and take the rest from the cache.
> 
> The fourth patch is adding a way to easily clean locally created
> containers.
> 
> This is is mainly for discussion and sharing as there are still some
> issues/problem to solve:
> - building the qemu* containers can take several hours depending on the
>   network bandwith and computing power of the machine where those are
>   created
> - produced containers containing the cache have a size between 8 and
>   12GB depending on the architecture. We might need to store the build
>   cache somewhere else to reduce the size. If we choose to have one
>   single image, the needed size is around 20GB and we need up to 40GB
>   during the build, which is why I splitted them.
> - during the build and run, we use a bit more then 20GB of disk which is
>   over the allowed size in gitlab
> 

So I tried to build one of the build containers on my x86 workstation
with the following:

  make yocto/kirkstone-qemuarm64

but I get an error from the build:

  21:30:20 build qemuarm64: Error
  22:00:38 run qemuarm64: Error
  22:00:41 Build Complete (2 errors)
  The command '/bin/sh -c /home/$USER_NAME/bin/build-yocto.sh $target' returned a non-zero code: 2

Anyone else is having a better luck than me?


I don't think it is a problem if it takes a long time to build the build
containers because they are not built often and they are not built as
part of the gitlab-ci runs.

The issue could be the resulting container size. I wasn't aware of a
limit in gitlab -- I would like to try if there is a way around the
limit (either by changing a setting, or potentially switching to a
premium account.) However I need to be able to complete a container
build first :-)

How did you find out about the 20 GB limit? I couldn't find it in the
docs. The only info I could find states that there is no hard limit on
registry.gitlab.com.

Cheers,

Stefano

Hi Stefano,

> On 8 Oct 2022, at 00:36, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Wed, 24 Aug 2022, Bertrand Marquis wrote:
>> This patch series is a first attempt to check if we could use Yocto in
>> gitlab ci to build and run xen on qemu for arm, arm64 and x86.
>> 
>> The first patch is making sure build-yocto.sh is not catched by
>> gitignore.
>> 
>> The second patch is creating a container with all elements required to
>> build Yocto, a checkout of the yocto layers required and an helper
>> script to build and run xen on qemu with yocto.
>> 
>> The third patch is creating containers with a first build of yocto done
>> so that susbsequent build with those containers would only rebuild what
>> was changed and take the rest from the cache.
>> 
>> The fourth patch is adding a way to easily clean locally created
>> containers.
>> 
>> This is is mainly for discussion and sharing as there are still some
>> issues/problem to solve:
>> - building the qemu* containers can take several hours depending on the
>>  network bandwith and computing power of the machine where those are
>>  created
>> - produced containers containing the cache have a size between 8 and
>>  12GB depending on the architecture. We might need to store the build
>>  cache somewhere else to reduce the size. If we choose to have one
>>  single image, the needed size is around 20GB and we need up to 40GB
>>  during the build, which is why I splitted them.
>> - during the build and run, we use a bit more then 20GB of disk which is
>>  over the allowed size in gitlab
>> 
> 
> So I tried to build one of the build containers on my x86 workstation
> with the following:
> 
>  make yocto/kirkstone-qemuarm64
> 
> but I get an error from the build:
> 
>  21:30:20 build qemuarm64: Error
>  22:00:38 run qemuarm64: Error
>  22:00:41 Build Complete (2 errors)
>  The command '/bin/sh -c /home/$USER_NAME/bin/build-yocto.sh $target' returned a non-zero code: 2
> 
> Anyone else is having a better luck than me?
> 

I did a new run and everything went ok on my side.
I will push a v2 of the serie to dump more logs when an error is happening.

Cheers
Bertrand

> 
> I don't think it is a problem if it takes a long time to build the build
> containers because they are not built often and they are not built as
> part of the gitlab-ci runs.
> 
> The issue could be the resulting container size. I wasn't aware of a
> limit in gitlab -- I would like to try if there is a way around the
> limit (either by changing a setting, or potentially switching to a
> premium account.) However I need to be able to complete a container
> build first :-)
> 
> How did you find out about the 20 GB limit? I couldn't find it in the
> docs. The only info I could find states that there is no hard limit on
> registry.gitlab.com.
> 
> Cheers,
> 
> Stefano

HI Stefano,

> On 8 Oct 2022, at 00:36, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Wed, 24 Aug 2022, Bertrand Marquis wrote:
>> This patch series is a first attempt to check if we could use Yocto in
>> gitlab ci to build and run xen on qemu for arm, arm64 and x86.
>> 
>> The first patch is making sure build-yocto.sh is not catched by
>> gitignore.
>> 
>> The second patch is creating a container with all elements required to
>> build Yocto, a checkout of the yocto layers required and an helper
>> script to build and run xen on qemu with yocto.
>> 
>> The third patch is creating containers with a first build of yocto done
>> so that susbsequent build with those containers would only rebuild what
>> was changed and take the rest from the cache.
>> 
>> The fourth patch is adding a way to easily clean locally created
>> containers.
>> 
>> This is is mainly for discussion and sharing as there are still some
>> issues/problem to solve:
>> - building the qemu* containers can take several hours depending on the
>>  network bandwith and computing power of the machine where those are
>>  created
>> - produced containers containing the cache have a size between 8 and
>>  12GB depending on the architecture. We might need to store the build
>>  cache somewhere else to reduce the size. If we choose to have one
>>  single image, the needed size is around 20GB and we need up to 40GB
>>  during the build, which is why I splitted them.
>> - during the build and run, we use a bit more then 20GB of disk which is
>>  over the allowed size in gitlab
>> 
> 
> So I tried to build one of the build containers on my x86 workstation
> with the following:
> 
>  make yocto/kirkstone-qemuarm64
> 
> but I get an error from the build:
> 
>  21:30:20 build qemuarm64: Error
>  22:00:38 run qemuarm64: Error
>  22:00:41 Build Complete (2 errors)
>  The command '/bin/sh -c /home/$USER_NAME/bin/build-yocto.sh $target' returned a non-zero code: 2
> 
> Anyone else is having a better luck than me?

I will relaunch an image creation test locally and come back to you.
It could be that Yocto has been updated since I last tested that.

> 
> 
> I don't think it is a problem if it takes a long time to build the build
> containers because they are not built often and they are not built as
> part of the gitlab-ci runs.
> 
> The issue could be the resulting container size. I wasn't aware of a
> limit in gitlab -- I would like to try if there is a way around the
> limit (either by changing a setting, or potentially switching to a
> premium account.) However I need to be able to complete a container
> build first :-)
> 
> How did you find out about the 20 GB limit? I couldn't find it in the
> docs. The only info I could find states that there is no hard limit on
> registry.gitlab.com.

This came after discussions internally with our internal gitlab and I found
some forum discussions saying something like that.
But it could be that Xen Project does not have this limitation hence the
need to test.

Cheers
Bertrand

> 
> Cheers,
> 
> Stefano