[Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms

Philippe Mathieu-Daudé posted 5 patches 6 years, 8 months ago
[Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Philippe Mathieu-Daudé 6 years, 8 months ago
Add a job to build and install the EDK2 platform firmware binaries.

This job is only triggered if the last commit matches the EDK2
name (case insensitive), or when tag are created (such releases
or release candidates).

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index e942175dd3..628cc52c99 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -258,3 +258,24 @@ matrix:
     - env:
         - CONFIG="--disable-system"
         - TEST_CMD="make -j3 check-tcg V=1"
+
+
+    # EDK2 roms builds
+    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
+      env:
+        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
+        - EDK2_BUILD_OPTIONS="--quiet --silent"
+      script:
+        - git submodule update --init roms/edk2
+        - make -j3
+        - make -C roms efi -j2
+        - make install
+      addons:
+        apt:
+          packages:
+            - dos2unix
+            - gcc-aarch64-linux-gnu
+            - gcc-arm-linux-gnueabihf
+            - iasl
+            - nasm
+            - uuid-dev
-- 
2.20.1


Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Laszlo Ersek 6 years, 8 months ago
On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
> Add a job to build and install the EDK2 platform firmware binaries.
> 
> This job is only triggered if the last commit matches the EDK2
> name (case insensitive), or when tag are created (such releases
> or release candidates).
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .travis.yml | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/.travis.yml b/.travis.yml
> index e942175dd3..628cc52c99 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -258,3 +258,24 @@ matrix:
>      - env:
>          - CONFIG="--disable-system"
>          - TEST_CMD="make -j3 check-tcg V=1"
> +
> +
> +    # EDK2 roms builds
> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
> +      env:
> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
> +      script:
> +        - git submodule update --init roms/edk2
> +        - make -j3
> +        - make -C roms efi -j2
> +        - make install
> +      addons:
> +        apt:
> +          packages:
> +            - dos2unix
> +            - gcc-aarch64-linux-gnu
> +            - gcc-arm-linux-gnueabihf
> +            - iasl
> +            - nasm
> +            - uuid-dev
> 

Regardless of what problem we're trying to address with "--quiet
--silent", those options are wrong. You certainly want detailed build
logs for the case a CI job fails (at build time or at runtime).

The reason why I only include DEBUG firmware builds in the edk2 bundling
series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
and as such they are 100% unsupportable in my book. Bugs in software are
the norm, not the exception, so we should allow (even force) the user
(and remote systems) to provide as much information as they can.

Thanks
Laszlo

Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Philippe Mathieu-Daudé 6 years, 4 months ago
Hi Laszlo,

On 3/12/19 5:29 PM, Laszlo Ersek wrote:
> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>> Add a job to build and install the EDK2 platform firmware binaries.
>>
>> This job is only triggered if the last commit matches the EDK2
>> name (case insensitive), or when tag are created (such releases
>> or release candidates).
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .travis.yml | 21 +++++++++++++++++++++
>>  1 file changed, 21 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index e942175dd3..628cc52c99 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -258,3 +258,24 @@ matrix:
>>      - env:
>>          - CONFIG="--disable-system"
>>          - TEST_CMD="make -j3 check-tcg V=1"
>> +
>> +
>> +    # EDK2 roms builds
>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>> +      env:
>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>> +      script:
>> +        - git submodule update --init roms/edk2
>> +        - make -j3
>> +        - make -C roms efi -j2
>> +        - make install
>> +      addons:
>> +        apt:
>> +          packages:
>> +            - dos2unix
>> +            - gcc-aarch64-linux-gnu
>> +            - gcc-arm-linux-gnueabihf
>> +            - iasl
>> +            - nasm
>> +            - uuid-dev
>>
> 
> Regardless of what problem we're trying to address with "--quiet
> --silent", those options are wrong. You certainly want detailed build
> logs for the case a CI job fails (at build time or at runtime).

On Travis we get:

The job exceeded the maximum log length, and has been terminated.
(https://travis-ci.org/philmd/qemu/jobs/545329905)

So I moved to GitLab, but we still get:

"Job's log exceeded limit of 4194304 bytes."
(https://gitlab.com/philmd/qemu/-/jobs/230772314)

Regarding the options to pass to edk2-build.sh,

$ build --help
  -j LOGFILE, --log=LOGFILE
                   Put log in specified file as well as on console.
  -s, --silent     Make use of silent mode of (n)make.
  -q, --quiet      Disable all messages except FATAL ERRORS.
  -v, --verbose    Turn on verbose output with informational messages
                   printed, including library instances selected, final
                   dependency expression, and warning messages, etc.

'--log' duplicate the output, and I don't want to reduce the log
details, so I understand I should use:

  ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log

Is that correct? But then I'd need to modify Makefile.edk2 to redirect
stdout.

> The reason why I only include DEBUG firmware builds in the edk2 bundling
> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
> and as such they are 100% unsupportable in my book. Bugs in software are
> the norm, not the exception, so we should allow (even force) the user
> (and remote systems) to provide as much information as they can.

Sure, we have the same book here ;)

Regards,

Phil.

Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Laszlo Ersek 6 years, 4 months ago
On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
> Hi Laszlo,
> 
> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>
>>> This job is only triggered if the last commit matches the EDK2
>>> name (case insensitive), or when tag are created (such releases
>>> or release candidates).
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> ---
>>>  .travis.yml | 21 +++++++++++++++++++++
>>>  1 file changed, 21 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index e942175dd3..628cc52c99 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -258,3 +258,24 @@ matrix:
>>>      - env:
>>>          - CONFIG="--disable-system"
>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>> +
>>> +
>>> +    # EDK2 roms builds
>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>> +      env:
>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>> +      script:
>>> +        - git submodule update --init roms/edk2
>>> +        - make -j3
>>> +        - make -C roms efi -j2
>>> +        - make install
>>> +      addons:
>>> +        apt:
>>> +          packages:
>>> +            - dos2unix
>>> +            - gcc-aarch64-linux-gnu
>>> +            - gcc-arm-linux-gnueabihf
>>> +            - iasl
>>> +            - nasm
>>> +            - uuid-dev
>>>
>>
>> Regardless of what problem we're trying to address with "--quiet
>> --silent", those options are wrong. You certainly want detailed build
>> logs for the case a CI job fails (at build time or at runtime).
> 
> On Travis we get:
> 
> The job exceeded the maximum log length, and has been terminated.
> (https://travis-ci.org/philmd/qemu/jobs/545329905)
> 
> So I moved to GitLab, but we still get:
> 
> "Job's log exceeded limit of 4194304 bytes."
> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
> 
> Regarding the options to pass to edk2-build.sh,
> 
> $ build --help
>   -j LOGFILE, --log=LOGFILE
>                    Put log in specified file as well as on console.
>   -s, --silent     Make use of silent mode of (n)make.
>   -q, --quiet      Disable all messages except FATAL ERRORS.
>   -v, --verbose    Turn on verbose output with informational messages
>                    printed, including library instances selected, final
>                    dependency expression, and warning messages, etc.
> 
> '--log' duplicate the output, and I don't want to reduce the log
> details, so I understand I should use:
> 
>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
> 
> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
> stdout.

Would it be possible to invoke the outermost make like this?

  make -C roms -j2 efi >make.roms.efi.log 2>&1 \
  || ( tail -c 2M make.roms.efi.log; false )

Thanks
Laszlo

> 
>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>> and as such they are 100% unsupportable in my book. Bugs in software are
>> the norm, not the exception, so we should allow (even force) the user
>> (and remote systems) to provide as much information as they can.
> 
> Sure, we have the same book here ;)
> 
> Regards,
> 
> Phil.
> 


Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Laszlo Ersek 6 years, 4 months ago
On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
> Hi Laszlo,
> 
> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>
>>> This job is only triggered if the last commit matches the EDK2
>>> name (case insensitive), or when tag are created (such releases
>>> or release candidates).
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> ---
>>>  .travis.yml | 21 +++++++++++++++++++++
>>>  1 file changed, 21 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index e942175dd3..628cc52c99 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -258,3 +258,24 @@ matrix:
>>>      - env:
>>>          - CONFIG="--disable-system"
>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>> +
>>> +
>>> +    # EDK2 roms builds
>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>> +      env:
>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>> +      script:
>>> +        - git submodule update --init roms/edk2
>>> +        - make -j3
>>> +        - make -C roms efi -j2
>>> +        - make install
>>> +      addons:
>>> +        apt:
>>> +          packages:
>>> +            - dos2unix
>>> +            - gcc-aarch64-linux-gnu
>>> +            - gcc-arm-linux-gnueabihf
>>> +            - iasl
>>> +            - nasm
>>> +            - uuid-dev
>>>
>>
>> Regardless of what problem we're trying to address with "--quiet
>> --silent", those options are wrong. You certainly want detailed build
>> logs for the case a CI job fails (at build time or at runtime).
> 
> On Travis we get:
> 
> The job exceeded the maximum log length, and has been terminated.
> (https://travis-ci.org/philmd/qemu/jobs/545329905)
> 
> So I moved to GitLab, but we still get:
> 
> "Job's log exceeded limit of 4194304 bytes."
> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
> 
> Regarding the options to pass to edk2-build.sh,
> 
> $ build --help
>   -j LOGFILE, --log=LOGFILE
>                    Put log in specified file as well as on console.
>   -s, --silent     Make use of silent mode of (n)make.
>   -q, --quiet      Disable all messages except FATAL ERRORS.
>   -v, --verbose    Turn on verbose output with informational messages
>                    printed, including library instances selected, final
>                    dependency expression, and warning messages, etc.
> 
> '--log' duplicate the output, and I don't want to reduce the log
> details, so I understand I should use:
> 
>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
> 
> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
> stdout.

Would it be possible to invoke the outermost make like this?

  make -C roms -j2 efi >make.roms.efi.log 2>&1 \
  || ( tail -c 2M make.roms.efi.log; false )

Thanks
Laszlo

> 
>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>> and as such they are 100% unsupportable in my book. Bugs in software are
>> the norm, not the exception, so we should allow (even force) the user
>> (and remote systems) to provide as much information as they can.
> 
> Sure, we have the same book here ;)
> 
> Regards,
> 
> Phil.
> 


Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Alex Bennée 6 years, 4 months ago
Laszlo Ersek <lersek@redhat.com> writes:

> On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
>> Hi Laszlo,
>>
>> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>>
>>>> This job is only triggered if the last commit matches the EDK2
>>>> name (case insensitive), or when tag are created (such releases
>>>> or release candidates).
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>> ---
>>>>  .travis.yml | 21 +++++++++++++++++++++
>>>>  1 file changed, 21 insertions(+)
>>>>
>>>> diff --git a/.travis.yml b/.travis.yml
>>>> index e942175dd3..628cc52c99 100644
>>>> --- a/.travis.yml
>>>> +++ b/.travis.yml
>>>> @@ -258,3 +258,24 @@ matrix:
>>>>      - env:
>>>>          - CONFIG="--disable-system"
>>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>>> +
>>>> +
>>>> +    # EDK2 roms builds
>>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>>> +      env:
>>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>>> +      script:
>>>> +        - git submodule update --init roms/edk2
>>>> +        - make -j3
>>>> +        - make -C roms efi -j2
>>>> +        - make install
>>>> +      addons:
>>>> +        apt:
>>>> +          packages:
>>>> +            - dos2unix
>>>> +            - gcc-aarch64-linux-gnu
>>>> +            - gcc-arm-linux-gnueabihf
>>>> +            - iasl
>>>> +            - nasm
>>>> +            - uuid-dev
>>>>
>>>
>>> Regardless of what problem we're trying to address with "--quiet
>>> --silent", those options are wrong. You certainly want detailed build
>>> logs for the case a CI job fails (at build time or at runtime).
>>
>> On Travis we get:
>>
>> The job exceeded the maximum log length, and has been terminated.
>> (https://travis-ci.org/philmd/qemu/jobs/545329905)
>>
>> So I moved to GitLab, but we still get:
>>
>> "Job's log exceeded limit of 4194304 bytes."
>> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
>>
>> Regarding the options to pass to edk2-build.sh,
>>
>> $ build --help
>>   -j LOGFILE, --log=LOGFILE
>>                    Put log in specified file as well as on console.
>>   -s, --silent     Make use of silent mode of (n)make.
>>   -q, --quiet      Disable all messages except FATAL ERRORS.
>>   -v, --verbose    Turn on verbose output with informational messages
>>                    printed, including library instances selected, final
>>                    dependency expression, and warning messages, etc.
>>
>> '--log' duplicate the output, and I don't want to reduce the log
>> details, so I understand I should use:
>>
>>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
>>
>> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
>> stdout.
>
> Would it be possible to invoke the outermost make like this?
>
>   make -C roms -j2 efi >make.roms.efi.log 2>&1 \
>   || ( tail -c 2M make.roms.efi.log; false )

Or something like:

     after_failure:
       - tail -c 2M make.roms.efi.log

as Cleber suggested for his acceptance tests. Generally we want to try
and make the builds less noisy and only echo relevant details when it
fails. However we've tended to turn up the noise so we can debug stuff
and that is just painful to browser on the Travis website.


>
> Thanks
> Laszlo
>
>>
>>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>>> and as such they are 100% unsupportable in my book. Bugs in software are
>>> the norm, not the exception, so we should allow (even force) the user
>>> (and remote systems) to provide as much information as they can.
>>
>> Sure, we have the same book here ;)
>>
>> Regards,
>>
>> Phil.
>>


--
Alex Bennée

Re: [PATCH 5/5] .travis.yml: Build and install EDK2 roms
Posted by Philippe Mathieu-Daudé 6 years, 1 month ago
On 6/14/19 8:44 PM, Alex Bennée wrote:
> Laszlo Ersek <lersek@redhat.com> writes:
>> On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
>>> Hi Laszlo,
>>>
>>> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>>>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>>>
>>>>> This job is only triggered if the last commit matches the EDK2
>>>>> name (case insensitive), or when tag are created (such releases
>>>>> or release candidates).
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>>> ---
>>>>>  .travis.yml | 21 +++++++++++++++++++++
>>>>>  1 file changed, 21 insertions(+)
>>>>>
>>>>> diff --git a/.travis.yml b/.travis.yml
>>>>> index e942175dd3..628cc52c99 100644
>>>>> --- a/.travis.yml
>>>>> +++ b/.travis.yml
>>>>> @@ -258,3 +258,24 @@ matrix:
>>>>>      - env:
>>>>>          - CONFIG="--disable-system"
>>>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>>>> +
>>>>> +
>>>>> +    # EDK2 roms builds
>>>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>>>> +      env:
>>>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>>>> +      script:
>>>>> +        - git submodule update --init roms/edk2
>>>>> +        - make -j3
>>>>> +        - make -C roms efi -j2
>>>>> +        - make install
>>>>> +      addons:
>>>>> +        apt:
>>>>> +          packages:
>>>>> +            - dos2unix
>>>>> +            - gcc-aarch64-linux-gnu
>>>>> +            - gcc-arm-linux-gnueabihf
>>>>> +            - iasl
>>>>> +            - nasm
>>>>> +            - uuid-dev
>>>>>
>>>>
>>>> Regardless of what problem we're trying to address with "--quiet
>>>> --silent", those options are wrong. You certainly want detailed build
>>>> logs for the case a CI job fails (at build time or at runtime).
>>>
>>> On Travis we get:
>>>
>>> The job exceeded the maximum log length, and has been terminated.
>>> (https://travis-ci.org/philmd/qemu/jobs/545329905)
>>>
>>> So I moved to GitLab, but we still get:
>>>
>>> "Job's log exceeded limit of 4194304 bytes."
>>> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
>>>
>>> Regarding the options to pass to edk2-build.sh,
>>>
>>> $ build --help
>>>   -j LOGFILE, --log=LOGFILE
>>>                    Put log in specified file as well as on console.
>>>   -s, --silent     Make use of silent mode of (n)make.
>>>   -q, --quiet      Disable all messages except FATAL ERRORS.
>>>   -v, --verbose    Turn on verbose output with informational messages
>>>                    printed, including library instances selected, final
>>>                    dependency expression, and warning messages, etc.
>>>
>>> '--log' duplicate the output, and I don't want to reduce the log
>>> details, so I understand I should use:
>>>
>>>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
>>>
>>> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
>>> stdout.
>>
>> Would it be possible to invoke the outermost make like this?
>>
>>   make -C roms -j2 efi >make.roms.efi.log 2>&1 \
>>   || ( tail -c 2M make.roms.efi.log; false )

The build gets killed:

  No output has been received in the last 10m0s, this potentially
  indicates a stalled build or something wrong with the build itself.
  Check the details on how to adjust your build configuration on:

https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

  The build has been terminated

> Or something like:
> 
>      after_failure:
>        - tail -c 2M make.roms.efi.log
> 
> as Cleber suggested for his acceptance tests. Generally we want to try
> and make the builds less noisy and only echo relevant details when it
> fails. However we've tended to turn up the noise so we can debug stuff
> and that is just painful to browser on the Travis website.

I'll try that.

>>>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>>>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>>>> and as such they are 100% unsupportable in my book. Bugs in software are
>>>> the norm, not the exception, so we should allow (even force) the user
>>>> (and remote systems) to provide as much information as they can.
>>>
>>> Sure, we have the same book here ;)
>>>
>>> Regards,
>>>
>>> Phil.
>>>
> --
> Alex Bennée
>