Currently, make parallelism at build time is defined as #cpus+1.
Some build jobs may need (or benefit from) a different number.

An example is builds with LTO where, because of the huge demand
of memory at link time, gitlab runners fails if two linkers are
run concurrently

This patch retains the default value of #cpus+1 but allows setting
the "JOBS" variable to a different number when applying the template

Signed-off-by: Daniele Buono <dbuono@linux.vnet.ibm.com>
---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8b6d495288..5c198f05d4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -17,7 +17,7 @@ include:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
-    - JOBS=$(expr $(nproc) + 1)
+    - JOBS=${JOBS:-$(expr $(nproc) + 1)}
   script:
     - mkdir build
     - cd build
-- 
2.30.0

On 23/02/21 00:01, Daniele Buono wrote:
> Currently, make parallelism at build time is defined as #cpus+1. Some 
> build jobs may need (or benefit from) a different number. An example is 
> builds with LTO where, because of the huge demand of memory at link 
> time, gitlab runners fails if two linkers are run concurrently This 
> patch retains the default value of #cpus+1 but allows setting the "JOBS" 
> variable to a different number when applying the template

As I just found out, you can add -Dbackend_max_links=1 to the meson 
command line instead if LTO is enabled.

Paolo

This works, but setting this value to 1 for everybody seems a bit too
restrictive. While the gitlab ci runners don't have enough memory for
this, that's not necessarily true for every build platform, and linking
multiple targets in parallel with LTO can result in a big save in time,
so I'd prefer a customizable way.

How about adding a flag `--max-ld-procs` to configure to manually set
backend_max_links?

This would also allow setting it up to any specific number above 1,
which looking at the Makefile seems to not be possible now: because of
how the -j flag is passed from make to ninja, a compilation is either
sequential or parallel based on #cpus

On 2/23/2021 3:12 AM, Paolo Bonzini wrote:
> On 23/02/21 00:01, Daniele Buono wrote:
>> Currently, make parallelism at build time is defined as #cpus+1. Some 
>> build jobs may need (or benefit from) a different number. An example 
>> is builds with LTO where, because of the huge demand of memory at link 
>> time, gitlab runners fails if two linkers are run concurrently This 
>> patch retains the default value of #cpus+1 but allows setting the 
>> "JOBS" variable to a different number when applying the template
> 
> As I just found out, you can add -Dbackend_max_links=1 to the meson 
> command line instead if LTO is enabled.
> 
> Paolo
> 

On 23/02/21 20:34, Daniele Buono wrote:
> This works, but setting this value to 1 for everybody seems a bit too
> restrictive. While the gitlab ci runners don't have enough memory for
> this, that's not necessarily true for every build platform, and linking
> multiple targets in parallel with LTO can result in a big save in time,
> so I'd prefer a customizable way.
> 
> How about adding a flag `--max-ld-procs` to configure to manually set
> backend_max_links?

Another possibility is to invoke "meson configure build 
-Dbackend_max_links=1" after configure.

Paolo

> This would also allow setting it up to any specific number above 1,
> which looking at the Makefile seems to not be possible now: because of
> how the -j flag is passed from make to ninja, a compilation is either
> sequential or parallel based on #cpus

On 2/24/2021 2:44 AM, Paolo Bonzini wrote:
> On 23/02/21 20:34, Daniele Buono wrote:
>> This works, but setting this value to 1 for everybody seems a bit too
>> restrictive. While the gitlab ci runners don't have enough memory for
>> this, that's not necessarily true for every build platform, and linking
>> multiple targets in parallel with LTO can result in a big save in time,
>> so I'd prefer a customizable way.
>>
>> How about adding a flag `--max-ld-procs` to configure to manually set
>> backend_max_links?
> 
> Another possibility is to invoke "meson configure build 
> -Dbackend_max_links=1" after configure.

I like this, I'll send a v2 soon where I replace this patch with one
just for linking.

Daniele

On 2/22/21 8:01 PM, Daniele Buono wrote:
> Currently, make parallelism at build time is defined as #cpus+1.
> Some build jobs may need (or benefit from) a different number.
>
> An example is builds with LTO where, because of the huge demand
> of memory at link time, gitlab runners fails if two linkers are
> run concurrently
>
> This patch retains the default value of #cpus+1 but allows setting
> the "JOBS" variable to a different number when applying the template
>
> Signed-off-by: Daniele Buono <dbuono@linux.vnet.ibm.com>
> ---
>   .gitlab-ci.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 8b6d495288..5c198f05d4 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -17,7 +17,7 @@ include:
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     before_script:
> -    - JOBS=$(expr $(nproc) + 1)
> +    - JOBS=${JOBS:-$(expr $(nproc) + 1)}
>     script:
>       - mkdir build
>       - cd build