gitlab-ci: Build MSYS2 job using multiple CPUs

[PATCH] gitlab-ci: Build MSYS2 job using multiple CPUs

Posted by Philippe Mathieu-Daudé 1 year, 5 months ago

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
I don't know how to use Powershell do use nproc+1 jobs
to optimize jobs waiting on I/O.
---
 .gitlab-ci.d/windows.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index a83f23a786..35ccb74fee 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -110,6 +110,7 @@ msys2-64bit:
       mingw-w64-x86_64-usbredir
       mingw-w64-x86_64-zstd"
   - Write-Output "Running build at $(Get-Date -Format u)"
+  - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
@@ -121,7 +122,7 @@ msys2-64bit:
   - cd build
   - ..\msys64\usr\bin\bash -lc "ccache --zero-stats"
   - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
-  - ..\msys64\usr\bin\bash -lc "make"
+  - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
   - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
   - ..\msys64\usr\bin\bash -lc "ccache --show-stats"
   - Write-Output "Finished build at $(Get-Date -Format u)"
-- 
2.45.2

Re: [PATCH] gitlab-ci: Build MSYS2 job using multiple CPUs

Posted by Thomas Huth 1 year, 5 months ago

According to 
https://docs.gitlab.com/ee/ci/runners/hosted_runners/windows.html the 
Windows shared runner should have 2 vCPUs nowadays, indeed! Maybe worth to 
mention it in the patch description?

Also, how much faster does the job now run for you?

On 19/08/2024 13.21, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
> I don't know how to use Powershell do use nproc+1 jobs
> to optimize jobs waiting on I/O.

Well, we're calling into bash there, so you could do it as part of the bash 
statement?

  Thomas

> ---
>   .gitlab-ci.d/windows.yml | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index a83f23a786..35ccb74fee 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -110,6 +110,7 @@ msys2-64bit:
>         mingw-w64-x86_64-usbredir
>         mingw-w64-x86_64-zstd"
>     - Write-Output "Running build at $(Get-Date -Format u)"
> +  - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>     - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
>     - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
>     - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
> @@ -121,7 +122,7 @@ msys2-64bit:
>     - cd build
>     - ..\msys64\usr\bin\bash -lc "ccache --zero-stats"
>     - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
> -  - ..\msys64\usr\bin\bash -lc "make"
> +  - ..\msys64\usr\bin\bash -lc "make -j$env:JOBS"
>     - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
>     - ..\msys64\usr\bin\bash -lc "ccache --show-stats"
>     - Write-Output "Finished build at $(Get-Date -Format u)"