1. Patchew
  2. QEMU
  3. View series

[PATCH] gitlab: purge msys pacman cache

Daniel P. Berrangé posted 1 patch 1 month ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20251010160545.144760-1-berrange@redhat.com
Maintainers: Yonggang Luo <luoyonggang@gmail.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>
.gitlab-ci.d/windows.yml | 1 +
1 file changed, 1 insertion(+)
[PATCH] gitlab: purge msys pacman cache
Posted by Daniel P. Berrangé 1 month ago 
For the Windows msys2 CI job we install many packages using pacman
and use the GitLab cache to preserve the pacman cache across CI
runs. While metadata still needs downloading, this avoids pacman
re-downloading packages from msys2 if they have not changed.

The problem is that pacman never automatically purges anything
from its package cache. Thus the GitLab cache is growing without
bound and packing/unpacking the cache is consuming an increasing
amount of time in the CI job.

If we run 'pacman -Sc' /after/ installing our desired package set,
it will purge any cached downloaded packages that are not matching
any installed package.

This will (currently) cap the pacman download cache at approx
256 MB.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 1 +
 1 file changed, 1 insertion(+)

See a test job with this change, plus a find across the msys
pacman cache, showing the cleanup effects....

Before cleanup:

  https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34

After cleanup:

  https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 1e6a01bd9a..6e1135d8b8 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -87,6 +87,7 @@ msys2-64bit:
       mingw-w64-x86_64-pkgconf
       mingw-w64-x86_64-python
       mingw-w64-x86_64-zstd"
+  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
   - Write-Output "Running build at $(Get-Date -Format u)"
   - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
-- 
2.50.1
Re: [PATCH] gitlab: purge msys pacman cache
Posted by 罗勇刚(Yonggang Luo) 1 month ago 
On Sat, Oct 11, 2025 at 12:05 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> For the Windows msys2 CI job we install many packages using pacman
> and use the GitLab cache to preserve the pacman cache across CI
> runs. While metadata still needs downloading, this avoids pacman
> re-downloading packages from msys2 if they have not changed.
>
> The problem is that pacman never automatically purges anything
> from its package cache. Thus the GitLab cache is growing without
> bound and packing/unpacking the cache is consuming an increasing
> amount of time in the CI job.
>
> If we run 'pacman -Sc' /after/ installing our desired package set,
> it will purge any cached downloaded packages that are not matching
> any installed package.
>
> This will (currently) cap the pacman download cache at approx
> 256 MB.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/windows.yml | 1 +
>  1 file changed, 1 insertion(+)
>
> See a test job with this change, plus a find across the msys
> pacman cache, showing the cleanup effects....
>
> Before cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
>
> After cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
>
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 1e6a01bd9a..6e1135d8b8 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -87,6 +87,7 @@ msys2-64bit:
>        mingw-w64-x86_64-pkgconf
>        mingw-w64-x86_64-python
>        mingw-w64-x86_64-zstd"
> +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
>    - Write-Output "Running build at $(Get-Date -Format u)"
>    - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
> --
> 2.50.1
>

Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>

-- 
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo
Re: [PATCH] gitlab: purge msys pacman cache
Posted by Thomas Huth 1 month ago 
On 10/10/2025 18.05, Daniel P. Berrangé wrote:
> For the Windows msys2 CI job we install many packages using pacman
> and use the GitLab cache to preserve the pacman cache across CI
> runs. While metadata still needs downloading, this avoids pacman
> re-downloading packages from msys2 if they have not changed.
> 
> The problem is that pacman never automatically purges anything
> from its package cache. Thus the GitLab cache is growing without
> bound and packing/unpacking the cache is consuming an increasing
> amount of time in the CI job.
> 
> If we run 'pacman -Sc' /after/ installing our desired package set,
> it will purge any cached downloaded packages that are not matching
> any installed package.
> 
> This will (currently) cap the pacman download cache at approx
> 256 MB.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 1 +
>   1 file changed, 1 insertion(+)
> 
> See a test job with this change, plus a find across the msys
> pacman cache, showing the cleanup effects....
> 
> Before cleanup:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
> 
> After cleanup:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 1e6a01bd9a..6e1135d8b8 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -87,6 +87,7 @@ msys2-64bit:
>         mingw-w64-x86_64-pkgconf
>         mingw-w64-x86_64-python
>         mingw-w64-x86_64-zstd"
> +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
>     - Write-Output "Running build at $(Get-Date -Format u)"
>     - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>     - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory

Reviewed-by: Thomas Huth <thuth@redhat.com>

I just tested this with some additional "ls -lR /var/cache/pacman" added, 
and it seems to work as expected:

  https://gitlab.com/thuth/qemu/-/jobs/11680562487#L293

This just removed an old "mingw-w64-x86_64-python-3.12.11-4-any.pkg.tar.zst" 
package, but kept the newer 3.12.12 package (and all other currently 
installed packages) around. Thus also:

Tested-by: Thomas Huth <thuth@redhat.com>
Re: [PATCH] gitlab: purge msys pacman cache
Posted by Peter Maydell 1 month ago 
On Fri, 10 Oct 2025 at 17:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> For the Windows msys2 CI job we install many packages using pacman
> and use the GitLab cache to preserve the pacman cache across CI
> runs. While metadata still needs downloading, this avoids pacman
> re-downloading packages from msys2 if they have not changed.
>
> The problem is that pacman never automatically purges anything
> from its package cache. Thus the GitLab cache is growing without
> bound and packing/unpacking the cache is consuming an increasing
> amount of time in the CI job.
>
> If we run 'pacman -Sc' /after/ installing our desired package set,
> it will purge any cached downloaded packages that are not matching
> any installed package.
>
> This will (currently) cap the pacman download cache at approx
> 256 MB.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/windows.yml | 1 +
>  1 file changed, 1 insertion(+)
>
> See a test job with this change, plus a find across the msys
> pacman cache, showing the cleanup effects....
>
> Before cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
>
> After cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
>
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 1e6a01bd9a..6e1135d8b8 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -87,6 +87,7 @@ msys2-64bit:
>        mingw-w64-x86_64-pkgconf
>        mingw-w64-x86_64-python
>        mingw-w64-x86_64-zstd"
> +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
>    - Write-Output "Running build at $(Get-Date -Format u)"
>    - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
> --

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

Hopefully this will make the CI job take less time (for the
record, currently in the main CI run it takes about 70 minutes,
of which 10 minutes is "unpack the cache" at the start and
20 minutes is "repack the cache" at the end).

thanks
-- PMM
Re: [PATCH] gitlab: purge msys pacman cache
Posted by Daniel P. Berrangé 1 month ago 
On Fri, Oct 10, 2025 at 06:23:22PM +0100, Peter Maydell wrote:
> On Fri, 10 Oct 2025 at 17:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > For the Windows msys2 CI job we install many packages using pacman
> > and use the GitLab cache to preserve the pacman cache across CI
> > runs. While metadata still needs downloading, this avoids pacman
> > re-downloading packages from msys2 if they have not changed.
> >
> > The problem is that pacman never automatically purges anything
> > from its package cache. Thus the GitLab cache is growing without
> > bound and packing/unpacking the cache is consuming an increasing
> > amount of time in the CI job.
> >
> > If we run 'pacman -Sc' /after/ installing our desired package set,
> > it will purge any cached downloaded packages that are not matching
> > any installed package.
> >
> > This will (currently) cap the pacman download cache at approx
> > 256 MB.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/windows.yml | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > See a test job with this change, plus a find across the msys
> > pacman cache, showing the cleanup effects....
> >
> > Before cleanup:
> >
> >   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
> >
> > After cleanup:
> >
> >   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
> >
> > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > index 1e6a01bd9a..6e1135d8b8 100644
> > --- a/.gitlab-ci.d/windows.yml
> > +++ b/.gitlab-ci.d/windows.yml
> > @@ -87,6 +87,7 @@ msys2-64bit:
> >        mingw-w64-x86_64-pkgconf
> >        mingw-w64-x86_64-python
> >        mingw-w64-x86_64-zstd"
> > +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
> >    - Write-Output "Running build at $(Get-Date -Format u)"
> >    - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
> >    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
> > --
> 
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
> 
> Hopefully this will make the CI job take less time (for the
> record, currently in the main CI run it takes about 70 minutes,
> of which 10 minutes is "unpack the cache" at the start and
> 20 minutes is "repack the cache" at the end).

Based on how long it takes to pack/unpack the cache in my job, I'd
be expecting approx a 25 minute win.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.