[v1] gitlab: aggressively avoid extra GIT data

[RFC PATCH] gitlab: aggressively avoid extra GIT data

Posted by Alex Bennée 1 month, 2 weeks ago

This avoids fetching blobs and tree references for branches we are not
going to worry about. Also skip tag references which are similarly not
useful and keep the default --prune. This keeps the .git data to
around 100M rather than the ~400M even a shallow clone takes.

So we can check the savings we also run a quick du while setting up
the build.

We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the
Windows build (git too old?) and the migration legacy test where we
build an older QEMU alongside the main one.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/base.yml               | 4 ++++
 .gitlab-ci.d/buildtest-template.yml | 1 +
 .gitlab-ci.d/buildtest.yml          | 2 ++
 .gitlab-ci.d/windows.yml            | 2 ++
 4 files changed, 9 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index ef173a34e6..7cb800fc3b 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -24,6 +24,10 @@ variables:
     # Each script line from will be in a collapsible section in the job output
     # and show the duration of each line.
     FF_SCRIPT_SECTIONS: 1
+    # The project has a fairly fat GIT repo so we try and avoid bringing in things
+    # we don't need. The --filter options avoid blobs and tree references we aren't going to use
+    # and we also avoid fetching tags.
+    GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune
 
   interruptible: true
 
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 4fbfeb6667..22045add80 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -14,6 +14,7 @@
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
     - export CCACHE_MAXSIZE="500M"
     - export PATH="$CCACHE_WRAPPERSDIR:$PATH"
+    - du -sh .git
     - mkdir build
     - cd build
     - ccache --zero-stats
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index a1c030337b..def469a9dc 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -187,6 +187,8 @@ build-previous-qemu:
   variables:
     IMAGE: opensuse-leap
     TARGETS: x86_64-softmmu aarch64-softmmu
+    # Override the default flags as we need more to grab the old version
+    GIT_FETCH_EXTRA_FLAGS: --prune
   before_script:
     - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)"
     - git remote add upstream https://gitlab.com/qemu-project/qemu
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index f116b8012d..f9da745042 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -28,6 +28,8 @@ msys2-64bit:
     # qTests don't run successfully with "--without-default-devices",
     # so let's exclude the qtests from CI for now.
     TEST_ARGS: --no-suite qtest
+    # The Windows git is a bit older so override the default
+    GIT_FETCH_EXTRA_FLAGS: --no-tags --prune
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
     expire_in: 7 days
-- 
2.39.2

Re: [RFC PATCH] gitlab: aggressively avoid extra GIT data

Posted by Thomas Huth 1 month, 2 weeks ago

On 08/03/2024 16.07, Alex Bennée wrote:
> This avoids fetching blobs and tree references for branches we are not
> going to worry about. Also skip tag references which are similarly not
> useful and keep the default --prune. This keeps the .git data to
> around 100M rather than the ~400M even a shallow clone takes.
> 
> So we can check the savings we also run a quick du while setting up
> the build.
> 
> We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the
> Windows build (git too old?) and the migration legacy test where we
> build an older QEMU alongside the main one.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/base.yml               | 4 ++++
>   .gitlab-ci.d/buildtest-template.yml | 1 +
>   .gitlab-ci.d/buildtest.yml          | 2 ++
>   .gitlab-ci.d/windows.yml            | 2 ++
>   4 files changed, 9 insertions(+)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index ef173a34e6..7cb800fc3b 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -24,6 +24,10 @@ variables:
>       # Each script line from will be in a collapsible section in the job output
>       # and show the duration of each line.
>       FF_SCRIPT_SECTIONS: 1
> +    # The project has a fairly fat GIT repo so we try and avoid bringing in things
> +    # we don't need. The --filter options avoid blobs and tree references we aren't going to use
> +    # and we also avoid fetching tags.
> +    GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune

The original settings included --quiet according to the docs:

 
https://docs.gitlab.com/ee/ci/runners/configure_runners.html#git-fetch-extra-flags

Should we include --quite here, too?

  Thomas