1. Patchew
  2. QEMU
  3. View series

[PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

Thomas Huth posted 1 patch 4 years, 9 months ago
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20210126065757.403853-1-thuth@redhat.com
Maintainers: "Philippe Mathieu-Daudé" <philmd@redhat.com>, Wainer dos Santos Moschetta <wainersm@redhat.com>, Thomas Huth <thuth@redhat.com>, "Alex Bennée" <alex.bennee@linaro.org>
.gitlab-ci.yml | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
[PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs
Posted by Thomas Huth 4 years, 9 months ago 
Currently, our check-system-* jobs are recompiling the whole sources
again. This happens due to the fact that the jobs are checking out
the whole source tree and required submodules again, and only try
to use the "build" directory with the binaries and object files as an
artifact from the previous stage - which simply does not work right
anymore (with the current version of meson). Due to some changed
time stamps, meson/ninja are always trying to rebuild the whole tree.

In the long run, we could likely use "meson test --no-rebuild", but
there is still some work going on in that area to improve the user
experience. So until this has been done, simply avoid recompiling the
sources with a trick: pass NINJA=":" to the make process in the test
jobs. Also check out the submodules manually before updating the
timestamps in the build folder, so that the binaries are definitely
newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index de3a3d25b5..16fea25ba9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -38,9 +38,11 @@ include:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
+    - scripts/git-submodule.sh update
+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
     - cd build
     - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
   cache:
-- 
2.27.0
Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs
Posted by Philippe Mathieu-Daudé 4 years, 9 months ago 
On 1/26/21 7:57 AM, Thomas Huth wrote:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files as an
> artifact from the previous stage - which simply does not work right
> anymore (with the current version of meson). Due to some changed
> time stamps, meson/ninja are always trying to rebuild the whole tree.
> 
> In the long run, we could likely use "meson test --no-rebuild", but
> there is still some work going on in that area to improve the user
> experience. So until this has been done, simply avoid recompiling the
> sources with a trick: pass NINJA=":" to the make process in the test
> jobs. Also check out the submodules manually before updating the
> timestamps in the build folder, so that the binaries are definitely
> newer that all the source files.
> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
> 
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.yml | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index de3a3d25b5..16fea25ba9 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -38,9 +38,11 @@ include:
>    stage: test
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    script:
> +    - scripts/git-submodule.sh update
> +        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
>      - cd build
>      - find . -type f -exec touch {} +
> -    - make $MAKE_CHECK_ARGS
> +    - make NINJA=":" $MAKE_CHECK_ARGS

This ninja trick deserves a comment in the YAML file.

Otherwise:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

>  
>  .acceptance_template: &acceptance_definition
>    cache:
>
Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs
Posted by Thomas Huth 4 years, 9 months ago 
On 26/01/2021 08.36, Philippe Mathieu-Daudé wrote:
> On 1/26/21 7:57 AM, Thomas Huth wrote:
>> Currently, our check-system-* jobs are recompiling the whole sources
>> again. This happens due to the fact that the jobs are checking out
>> the whole source tree and required submodules again, and only try
>> to use the "build" directory with the binaries and object files as an
>> artifact from the previous stage - which simply does not work right
>> anymore (with the current version of meson). Due to some changed
>> time stamps, meson/ninja are always trying to rebuild the whole tree.
>>
>> In the long run, we could likely use "meson test --no-rebuild", but
>> there is still some work going on in that area to improve the user
>> experience. So until this has been done, simply avoid recompiling the
>> sources with a trick: pass NINJA=":" to the make process in the test
>> jobs. Also check out the submodules manually before updating the
>> timestamps in the build folder, so that the binaries are definitely
>> newer that all the source files.
>> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
>>
>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .gitlab-ci.yml | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index de3a3d25b5..16fea25ba9 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -38,9 +38,11 @@ include:
>>     stage: test
>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>     script:
>> +    - scripts/git-submodule.sh update
>> +        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
>>       - cd build
>>       - find . -type f -exec touch {} +
>> -    - make $MAKE_CHECK_ARGS
>> +    - make NINJA=":" $MAKE_CHECK_ARGS
> 
> This ninja trick deserves a comment in the YAML file.

I'll add:

     # Avoid recompiling by hiding ninja with NINJA=":"

Ok?

  Thomas
Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs
Posted by Philippe Mathieu-Daudé 4 years, 9 months ago 
On 1/26/21 10:08 AM, Thomas Huth wrote:
> On 26/01/2021 08.36, Philippe Mathieu-Daudé wrote:
>> On 1/26/21 7:57 AM, Thomas Huth wrote:
>>> Currently, our check-system-* jobs are recompiling the whole sources
>>> again. This happens due to the fact that the jobs are checking out
>>> the whole source tree and required submodules again, and only try
>>> to use the "build" directory with the binaries and object files as an
>>> artifact from the previous stage - which simply does not work right
>>> anymore (with the current version of meson). Due to some changed
>>> time stamps, meson/ninja are always trying to rebuild the whole tree.
>>>
>>> In the long run, we could likely use "meson test --no-rebuild", but
>>> there is still some work going on in that area to improve the user
>>> experience. So until this has been done, simply avoid recompiling the
>>> sources with a trick: pass NINJA=":" to the make process in the test
>>> jobs. Also check out the submodules manually before updating the
>>> timestamps in the build folder, so that the binaries are definitely
>>> newer that all the source files.
>>> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
>>>
>>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>   .gitlab-ci.yml | 4 +++-
>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index de3a3d25b5..16fea25ba9 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -38,9 +38,11 @@ include:
>>>     stage: test
>>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>>     script:
>>> +    - scripts/git-submodule.sh update
>>> +        $(grep GIT_SUBMODULES build/config-host.mak | sed
>>> 's/GIT_SUBMODULES=//')
>>>       - cd build
>>>       - find . -type f -exec touch {} +
>>> -    - make $MAKE_CHECK_ARGS
>>> +    - make NINJA=":" $MAKE_CHECK_ARGS
>>
>> This ninja trick deserves a comment in the YAML file.
> 
> I'll add:
> 
>     # Avoid recompiling by hiding ninja with NINJA=":"
> 
> Ok?

Perfect, thanks!

Phil.
Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs
Posted by Wainer dos Santos Moschetta 4 years, 9 months ago 
On 1/26/21 3:57 AM, Thomas Huth wrote:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files as an
> artifact from the previous stage - which simply does not work right
> anymore (with the current version of meson). Due to some changed
> time stamps, meson/ninja are always trying to rebuild the whole tree.
>
> In the long run, we could likely use "meson test --no-rebuild", but
> there is still some work going on in that area to improve the user
> experience. So until this has been done, simply avoid recompiling the
> sources with a trick: pass NINJA=":" to the make process in the test
> jobs. Also check out the submodules manually before updating the
> timestamps in the build folder, so that the binaries are definitely
> newer that all the source files.
> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
Thanks for working on this!
>
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci.yml | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index de3a3d25b5..16fea25ba9 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -38,9 +38,11 @@ include:
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     script:
> +    - scripts/git-submodule.sh update
> +        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
>       - cd build
>       - find . -type f -exec touch {} +
> -    - make $MAKE_CHECK_ARGS
> +    - make NINJA=":" $MAKE_CHECK_ARGS
>   
>   .acceptance_template: &acceptance_definition
>     cache:

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.