1. Patchew
  2. QEMU
  3. [PATCH 0/8] gitlab: speed up msys windows jobs with GCC
  4. View patch

[PATCH 3/8] gitlab: always use updated msys installer

Daniel P. Berrangé posted 8 patches 2 years, 6 months ago
Maintainers: Yonggang Luo <luoyonggang@gmail.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Beraldo Leal <bleal@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>
[PATCH 3/8] gitlab: always use updated msys installer
Posted by Daniel P. Berrangé 2 years, 6 months ago 
We current reference an msys installer binary from mid-2022, which means
after installation, it immediately has to re-download a bunch of newer
content. This wastes precious CI time.

The msys project publishes an installer binary with a fixed URL that
always references the latest content. We cache the downloads in gitlab
though and so once downloaded we would never re-fetch the installer
leading back to the same problem.

To deal with this we also fetch the pgp signature for the installer
on every run, and compare that to the previously cached signature. If
the signature changes, we re-download the full installer.

This ensures we always have the latest installer for msys, while also
maximising use of the gitlab cache.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 831b080d12..0bc04ad068 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -23,10 +23,34 @@
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
     }
-  - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+  - Invoke-WebRequest
+    "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe.sig"
+    -outfile "msys2.exe.sig"
+  - if ( Test-Path -Path msys64\var\cache\msys2.exe.sig ) {
+      Write-Output "Cached installer sig" ;
+      if ( ((Get-FileHash msys2.exe.sig).Hash -ne (Get-FileHash msys64\var\cache\msys2.exe.sig).Hash) ) {
+        Write-Output "Mis-matched installer sig, new installer download required" ;
+        Remove-Item -Path msys64\var\cache\msys2.exe.sig ;
+        if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+          Remove-Item -Path msys64\var\cache\msys2.exe
+        }
+      } else {
+        Write-Output "Matched installer sig, cached installer still valid"
+      }
+    } else {
+      Write-Output "No cached installer sig, new installer download required" ;
+      if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+        Remove-Item -Path msys64\var\cache\msys2.exe
+      }
+    }
+  - if ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+      Write-Output "Fetching latest installer" ;
       Invoke-WebRequest
-      "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
-      -outfile "msys64\var\cache\msys2.exe"
+      "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe"
+      -outfile "msys64\var\cache\msys2.exe" ;
+      Copy-Item -Path msys2.exe.sig -Destination msys64\var\cache\msys2.exe.sig
+    } else {
+      Write-Output "Using cached installer"
     }
   - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
   - msys64\var\cache\msys2.exe -y
-- 
2.41.0
Re: [PATCH 3/8] gitlab: always use updated msys installer
Posted by Thomas Huth 2 years, 6 months ago 
On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> We current reference an msys installer binary from mid-2022, which means
> after installation, it immediately has to re-download a bunch of newer
> content. This wastes precious CI time.
> 
> The msys project publishes an installer binary with a fixed URL that
> always references the latest content. We cache the downloads in gitlab
> though and so once downloaded we would never re-fetch the installer
> leading back to the same problem.
> 
> To deal with this we also fetch the pgp signature for the installer
> on every run, and compare that to the previously cached signature. If
> the signature changes, we re-download the full installer.
> 
> This ensures we always have the latest installer for msys, while also
> maximising use of the gitlab cache.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 30 +++++++++++++++++++++++++++---
>   1 file changed, 27 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 831b080d12..0bc04ad068 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -23,10 +23,34 @@
>     - If ( !(Test-Path -Path msys64\var\cache ) ) {
>         mkdir msys64\var\cache
>       }
> -  - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
> +  - Invoke-WebRequest
> +    "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe.sig"
> +    -outfile "msys2.exe.sig"
> +  - if ( Test-Path -Path msys64\var\cache\msys2.exe.sig ) {
> +      Write-Output "Cached installer sig" ;
> +      if ( ((Get-FileHash msys2.exe.sig).Hash -ne (Get-FileHash msys64\var\cache\msys2.exe.sig).Hash) ) {
> +        Write-Output "Mis-matched installer sig, new installer download required" ;
> +        Remove-Item -Path msys64\var\cache\msys2.exe.sig ;
> +        if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
> +          Remove-Item -Path msys64\var\cache\msys2.exe
> +        }
> +      } else {
> +        Write-Output "Matched installer sig, cached installer still valid"
> +      }
> +    } else {
> +      Write-Output "No cached installer sig, new installer download required" ;
> +      if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
> +        Remove-Item -Path msys64\var\cache\msys2.exe
> +      }
> +    }
> +  - if ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
> +      Write-Output "Fetching latest installer" ;
>         Invoke-WebRequest
> -      "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
> -      -outfile "msys64\var\cache\msys2.exe"
> +      "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe"
> +      -outfile "msys64\var\cache\msys2.exe" ;
> +      Copy-Item -Path msys2.exe.sig -Destination msys64\var\cache\msys2.exe.sig
> +    } else {
> +      Write-Output "Using cached installer"
>       }
>     - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
>     - msys64\var\cache\msys2.exe -y

Reviewed-by: Thomas Huth <thuth@redhat.com>

... I think the original idea was to use a "tagged" version to avoid that we 
have some kind of "rolling release" here, but since the latest content is 
fetched anyway during the following update, that idea was likely not working 
as expected...
Re: [PATCH 3/8] gitlab: always use updated msys installer
Posted by Daniel P. Berrangé 2 years, 6 months ago 
On Wed, Aug 02, 2023 at 05:49:42PM +0200, Thomas Huth wrote:
> On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> > We current reference an msys installer binary from mid-2022, which means
> > after installation, it immediately has to re-download a bunch of newer
> > content. This wastes precious CI time.
> > 
> > The msys project publishes an installer binary with a fixed URL that
> > always references the latest content. We cache the downloads in gitlab
> > though and so once downloaded we would never re-fetch the installer
> > leading back to the same problem.
> > 
> > To deal with this we also fetch the pgp signature for the installer
> > on every run, and compare that to the previously cached signature. If
> > the signature changes, we re-download the full installer.
> > 
> > This ensures we always have the latest installer for msys, while also
> > maximising use of the gitlab cache.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>


> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 
> ... I think the original idea was to use a "tagged" version to avoid that we
> have some kind of "rolling release" here, but since the latest content is
> fetched anyway during the following update, that idea was likely not working
> as expected...

I think using the "latest" installer is also more in keeping with what
we do for other distros, where we'll always pick up the latest content
when the containers get rebuilt, or macOS/FreeBSD where we pick the
latest from Ports/HomeBrew.  

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.