Here are some patches to enable ccache in the gitlab-CI. Using the
compiler cache can speed up some of the jobs in the CI by avoiding
to recompile the same sources again and again. With this patch series,
some jobs indeed get quite a bit faster. For example "build-system-fedora"
goes down from 29 minutes (https://gitlab.com/thuth/qemu/-/jobs/1168149051)
to 12 minutes (https://gitlab.com/thuth/qemu/-/jobs/1175479324) in the
best case.
However, and that's why I've marked this series still as RFC, there
are some jobs which refuse to improve much, for example the *-cfi-* jobs
are still running for a long time, see e.g.:

 https://gitlab.com/thuth/qemu/-/jobs/1168149079 (50 minutes on master)
 https://gitlab.com/thuth/qemu/-/jobs/1175479342 (43 minutes with ccache)

Thus the overall run time did not improve very much yet.

Additionally, the jobs are sometimes running even slower, e.g. if the
cache has not been populated yet or if there are a lot of cache misses,
and I've seen some jobs hitting the 60 min time limit because of this.
And I've even seen two or three jobs failing intermittendly with a
crashing test, e.g.:

 https://gitlab.com/thuth/qemu/-/jobs/1175479336

Not sure whether that's related to some bad caching of ccache, or
whether it's something that also occurs on the master branch recently?
Has anybody else experienced such a problem?

Thus this series is RFC only - in case somebody else wants to play
with it and help to improve it, you're very welcome!

Thomas Huth (5):
  tests/docker/dockerfiles: Add ccache to containers where it was
    missing
  tests/docker/dockerfiles/ubuntu2004: Add missing symlink for cc
  gitlab-ci.d/crossbuilds: Limit the amount of targets in the
    cross-win64 build
  gitlab-ci.yml: Compile with ccache enabled
  gitlab-ci.d/crossbuilds.yml: Enable compilation with ccache

 .gitlab-ci.d/crossbuilds.yml                  | 34 +++++++++++++++++++
 .gitlab-ci.yml                                | 20 +++++++++++
 tests/docker/dockerfiles/alpine.docker        |  1 +
 .../dockerfiles/fedora-i386-cross.docker      |  1 +
 .../dockerfiles/fedora-win32-cross.docker     |  1 +
 .../dockerfiles/fedora-win64-cross.docker     |  1 +
 tests/docker/dockerfiles/opensuse-leap.docker |  1 +
 tests/docker/dockerfiles/ubuntu2004.docker    |  3 ++
 8 files changed, 62 insertions(+)

-- 
2.27.0

On Wed, Apr 14, 2021 at 10:19:02AM +0200, Thomas Huth wrote:
> Here are some patches to enable ccache in the gitlab-CI. Using the
> compiler cache can speed up some of the jobs in the CI by avoiding
> to recompile the same sources again and again. With this patch series,
> some jobs indeed get quite a bit faster. For example "build-system-fedora"
> goes down from 29 minutes (https://gitlab.com/thuth/qemu/-/jobs/1168149051)
> to 12 minutes (https://gitlab.com/thuth/qemu/-/jobs/1175479324) in the
> best case.
> However, and that's why I've marked this series still as RFC, there
> are some jobs which refuse to improve much, for example the *-cfi-* jobs
> are still running for a long time, see e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/1168149079 (50 minutes on master)
>  https://gitlab.com/thuth/qemu/-/jobs/1175479342 (43 minutes with ccache)
> 
> Thus the overall run time did not improve very much yet.
> 
> Additionally, the jobs are sometimes running even slower, e.g. if the
> cache has not been populated yet or if there are a lot of cache misses,
> and I've seen some jobs hitting the 60 min time limit because of this.
> And I've even seen two or three jobs failing intermittendly with a
> crashing test, e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/1175479336
> 
> Not sure whether that's related to some bad caching of ccache, or
> whether it's something that also occurs on the master branch recently?
> Has anybody else experienced such a problem?

The GitLab CI cache implementation does not appear to be designed for
ccache because the files are saved in a cache.zip file and then
extracted again each time a job runs.

That is very inefficient for something like a 1 GiB ccache directory.

More details:
https://docs.gitlab.com/ee/ci/caching/#availability-of-the-cache

The GitLab CI cache feature in its current form just doesn't look
suitable for ccache.

I wonder if we can implement ccache differently on dedicated runners.
The ccache directory would really be shared (mounted within job
containers) without the additional cache.zip step to avoid wasting CPU
cycles compressing/decompressing and RAM with duplicated page cache.

Stefan