[v3] tests/avocado: update sbsa-ref firmware to latest

[PATCH v3 0/4] tests/avocado: update sbsa-ref firmware to latest

Posted by Marcin Juszkiewicz 1 month, 1 week ago

Updating sbsa-ref firmware for QEMU CI was manual task. Now it is
replaced by CI job run on CodeLinaro Gitlab instance.

This patchset updates to current state:

- Trusted Firmware v2.10.2 (latest LTS)
- Tianocore EDK2 stable202402 (latest release)

And Tianocore EDK2-platforms commit 085c2fb (edk2-platforms does not
have releases).

Firmware images were built using Debian 'bookworm' cross gcc 12.2.0
compiler.

And while I am in that file I dropped use of 'virtio-rng-pci' device as
sbsa-ref is supposed to emulate physical hardware.

Added 'max' tests with 'pauth=off' and 'pauth-impdef=on' variants.

(01/11) test_sbsaref_edk2_firmware: PASS (2.51 s)
(02/11) test_sbsaref_alpine_linux_cortex_a57: PASS (23.72 s)
(03/11) test_sbsaref_alpine_linux_neoverse_n1: PASS (23.70 s)
(04/11) test_sbsaref_alpine_linux_max_pauth_off: PASS (23.00 s)
(05/11) test_sbsaref_alpine_linux_max_pauth_impdef: PASS (29.03 s)
(06/11) test_sbsaref_alpine_linux_max: PASS (80.69 s)
(07/11) test_sbsaref_openbsd73_cortex_a57: PASS (16.05 s)
(08/11) test_sbsaref_openbsd73_neoverse_n1: PASS (15.97 s)
(09/11) test_sbsaref_openbsd73_max_pauth_off: PASS (16.22 s)
(10/11) test_sbsaref_openbsd73_max_pauth_impdef: PASS (16.11 s)
(11/11) test_sbsaref_openbsd73_max: PASS (16.08 s)

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
Changes in v3:
- left OpenBSD at 7.3 (7.4+ is known to not boot)
  https://gitlab.com/qemu-project/qemu/-/issues/2224
  https://marc.info/?l=openbsd-arm&m=171050428327850&w=2
- added pauth variants of 'max' to OpenBSD tests
- Link to v2: https://lore.kernel.org/r/20240314-sbsa-ref-firmware-update-v2-0-b557c56559cd@linaro.org

Changes in v2:
- disabled 'max' tests on OpenBSD
- moved tags to 'one tag per line'
- added 'os:linux' tags to Alpine ones
- Link to v1: https://lore.kernel.org/r/20240313-sbsa-ref-firmware-update-v1-0-e166703c5424@linaro.org

---
Marcin Juszkiewicz (4):
      tests/avocado: update sbsa-ref firmware
      tests/avocado: drop virtio-rng from sbsa-ref tests
      tests/avocado: sbsa-ref: add Alpine tests for misc 'max' setup
      tests/avocado: sbsa-ref: add OpenBSD tests for misc 'max' setup

 tests/avocado/machine_aarch64_sbsaref.py | 86 +++++++++++++++++++++-----------
 1 file changed, 58 insertions(+), 28 deletions(-)
---
base-commit: ba49d760eb04630e7b15f423ebecf6c871b8f77b
change-id: 20240313-sbsa-ref-firmware-update-7579d9f6d59b

Best regards,
-- 
Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

Re: [PATCH v3 0/4] tests/avocado: update sbsa-ref firmware to latest

Posted by Alex Bennée 1 month, 1 week ago

Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:

> Updating sbsa-ref firmware for QEMU CI was manual task. Now it is
> replaced by CI job run on CodeLinaro Gitlab instance.
>
> This patchset updates to current state:
>
> - Trusted Firmware v2.10.2 (latest LTS)
> - Tianocore EDK2 stable202402 (latest release)
>
> And Tianocore EDK2-platforms commit 085c2fb (edk2-platforms does not
> have releases).
>
> Firmware images were built using Debian 'bookworm' cross gcc 12.2.0
> compiler.
>
> And while I am in that file I dropped use of 'virtio-rng-pci' device as
> sbsa-ref is supposed to emulate physical hardware.
>
> Added 'max' tests with 'pauth=off' and 'pauth-impdef=on' variants.

Queued to for-9.0/misc-fixes, thanks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v3 0/4] tests/avocado: update sbsa-ref firmware to latest

Posted by Philippe Mathieu-Daudé 1 month, 1 week ago

Hi Marcin,

On 18/3/24 15:08, Marcin Juszkiewicz wrote:
> Updating sbsa-ref firmware for QEMU CI was manual task. Now it is
> replaced by CI job run on CodeLinaro Gitlab instance.
> 
> This patchset updates to current state:
> 
> - Trusted Firmware v2.10.2 (latest LTS)
> - Tianocore EDK2 stable202402 (latest release)
> 
> And Tianocore EDK2-platforms commit 085c2fb (edk2-platforms does not
> have releases).
> 
> Firmware images were built using Debian 'bookworm' cross gcc 12.2.0
> compiler.
> 
> And while I am in that file I dropped use of 'virtio-rng-pci' device as
> sbsa-ref is supposed to emulate physical hardware.
> 
> Added 'max' tests with 'pauth=off' and 'pauth-impdef=on' variants.
> 
> (01/11) test_sbsaref_edk2_firmware: PASS (2.51 s)
> (02/11) test_sbsaref_alpine_linux_cortex_a57: PASS (23.72 s)
> (03/11) test_sbsaref_alpine_linux_neoverse_n1: PASS (23.70 s)
> (04/11) test_sbsaref_alpine_linux_max_pauth_off: PASS (23.00 s)
> (05/11) test_sbsaref_alpine_linux_max_pauth_impdef: PASS (29.03 s)
> (06/11) test_sbsaref_alpine_linux_max: PASS (80.69 s)

This one is timeouting for me, should we skip it with
AVOCADO_TIMEOUT_EXPECTED? (See below)

> (07/11) test_sbsaref_openbsd73_cortex_a57: PASS (16.05 s)
> (08/11) test_sbsaref_openbsd73_neoverse_n1: PASS (15.97 s)
> (09/11) test_sbsaref_openbsd73_max_pauth_off: PASS (16.22 s)
> (10/11) test_sbsaref_openbsd73_max_pauth_impdef: PASS (16.11 s)
> (11/11) test_sbsaref_openbsd73_max: PASS (16.08 s)
> 
> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> ---
> Changes in v3:
> - left OpenBSD at 7.3 (7.4+ is known to not boot)
>    https://gitlab.com/qemu-project/qemu/-/issues/2224
>    https://marc.info/?l=openbsd-arm&m=171050428327850&w=2
> - added pauth variants of 'max' to OpenBSD tests
> - Link to v2: https://lore.kernel.org/r/20240314-sbsa-ref-firmware-update-v2-0-b557c56559cd@linaro.org
> 
> Changes in v2:
> - disabled 'max' tests on OpenBSD
> - moved tags to 'one tag per line'
> - added 'os:linux' tags to Alpine ones
> - Link to v1: https://lore.kernel.org/r/20240313-sbsa-ref-firmware-update-v1-0-e166703c5424@linaro.org
> 
> ---
> Marcin Juszkiewicz (4):
>        tests/avocado: update sbsa-ref firmware
>        tests/avocado: drop virtio-rng from sbsa-ref tests
>        tests/avocado: sbsa-ref: add Alpine tests for misc 'max' setup
>        tests/avocado: sbsa-ref: add OpenBSD tests for misc 'max' setup

$ make check-avocado AVOCADO_TAGS='machine:sbsa-ref'
ninja: no work to do.
JOB ID     : 76d5dc90c6f70f0801c5269ff1c1db6c0d2cb27b
JOB LOG    : 
build/system_arm/tests/results/job-2024-03-18T15.54-76d5dc9/job.log
  (1/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_edk2_firmware: 
PASS (4.96 s)
  (2/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_cortex_a57: 
PASS (52.67 s)
  (3/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_neoverse_n1: 
PASS (51.01 s)
  (4/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max: 
INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout 
reached\nOriginal status: ERROR\n{'name': 
'4-tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max', 
'logdir': 'build/system_... (180.50 s)
  (5/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_openbsd73_cortex_a57: 
PASS (21.15 s)
  (6/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_openbsd73_neoverse_n1: 
PASS (20.88 s)
  (7/7) 
tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_openbsd73_max: 
PASS (20.70 s)
RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 1 | 
CANCEL 0
JOB TIME   : 355.21 s
make: *** [tests/Makefile.include:139: check-avocado] Error 8

Looking at debug.log:

15:56:17 DEBUG| Opening console file
15:56:17 DEBUG| Opening console socket
15:56:17 DEBUG| NOTICE:  Booting Trusted Firmware
15:56:17 DEBUG| NOTICE:  BL1: v2.9(release):v2.9.0-764-g7c3ff62d2
15:56:17 DEBUG| NOTICE:  BL1: Built : 12:32:06, Sep 27 2023
15:56:17 DEBUG| NOTICE:  BL1: Booting BL2
15:56:17 DEBUG| NOTICE:  BL2: v2.9(release):v2.9.0-764-g7c3ff62d2
15:56:17 DEBUG| NOTICE:  BL2: Built : 12:32:06, Sep 27 2023
15:56:17 DEBUG| NOTICE:  BL1: Booting BL31
15:56:17 DEBUG| NOTICE:  BL31: v2.9(release):v2.9.0-764-g7c3ff62d2
15:56:17 DEBUG| NOTICE:  BL31: Built : 12:32:06, Sep 27 2023
15:56:17 DEBUG| UEFI firmware (version 1.0 built at 15:45:23 on Sep 20 2023)
...
15:57:48 DEBUG| * Mounting security filesystem ... [ ok ]
15:57:49 DEBUG| * Mounting debug filesystem ... [ ok ]
15:57:49 DEBUG| * Mounting persistent storage (pstore) filesystem ... [ ok ]
15:57:50 DEBUG| * Mounting efivarfs filesystem ... [ ok ]
15:57:51 DEBUG| * Starting busybox mdev ... [ ok ]
15:58:05 DEBUG| * Scanning hardware for mdev ... [ ok ]
15:59:05 DEBUG| * Loading hardware drivers ... [ ok ]
15:59:05 DEBUG| * WARNING: clock skew detected!
15:59:07 DEBUG| * Setting the local clock based on last shutdown time 
... [ ok ]
15:59:11 DEBUG| * Loading modules ... [ ok ]
15:59:13 ERROR|
15:59:13 ERROR| RuntimeError: Test interrupted by SIGTERM

Re: [PATCH v3 0/4] tests/avocado: update sbsa-ref firmware to latest

Posted by Philippe Mathieu-Daudé 1 month, 1 week ago

On 18/3/24 17:30, Philippe Mathieu-Daudé wrote:
> Hi Marcin,
> 
> On 18/3/24 15:08, Marcin Juszkiewicz wrote:
>> Updating sbsa-ref firmware for QEMU CI was manual task. Now it is
>> replaced by CI job run on CodeLinaro Gitlab instance.
>>
>> This patchset updates to current state:
>>
>> - Trusted Firmware v2.10.2 (latest LTS)
>> - Tianocore EDK2 stable202402 (latest release)
>>
>> And Tianocore EDK2-platforms commit 085c2fb (edk2-platforms does not
>> have releases).
>>
>> Firmware images were built using Debian 'bookworm' cross gcc 12.2.0
>> compiler.
>>
>> And while I am in that file I dropped use of 'virtio-rng-pci' device as
>> sbsa-ref is supposed to emulate physical hardware.
>>
>> Added 'max' tests with 'pauth=off' and 'pauth-impdef=on' variants.
>>
>> (01/11) test_sbsaref_edk2_firmware: PASS (2.51 s)
>> (02/11) test_sbsaref_alpine_linux_cortex_a57: PASS (23.72 s)
>> (03/11) test_sbsaref_alpine_linux_neoverse_n1: PASS (23.70 s)
>> (04/11) test_sbsaref_alpine_linux_max_pauth_off: PASS (23.00 s)
>> (05/11) test_sbsaref_alpine_linux_max_pauth_impdef: PASS (29.03 s)
>> (06/11) test_sbsaref_alpine_linux_max: PASS (80.69 s)
> 
> This one is timeouting for me, should we skip it with
> AVOCADO_TIMEOUT_EXPECTED? (See below)

Well AVOCADO_TIMEOUT_EXPECTED is present but apparently ignored :/

>> (07/11) test_sbsaref_openbsd73_cortex_a57: PASS (16.05 s)
>> (08/11) test_sbsaref_openbsd73_neoverse_n1: PASS (15.97 s)
>> (09/11) test_sbsaref_openbsd73_max_pauth_off: PASS (16.22 s)
>> (10/11) test_sbsaref_openbsd73_max_pauth_impdef: PASS (16.11 s)
>> (11/11) test_sbsaref_openbsd73_max: PASS (16.08 s)
>>
>> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
>> ---
>> Changes in v3:
>> - left OpenBSD at 7.3 (7.4+ is known to not boot)
>>    https://gitlab.com/qemu-project/qemu/-/issues/2224
>>    https://marc.info/?l=openbsd-arm&m=171050428327850&w=2
>> - added pauth variants of 'max' to OpenBSD tests
>> - Link to v2: 
>> https://lore.kernel.org/r/20240314-sbsa-ref-firmware-update-v2-0-b557c56559cd@linaro.org
>>
>> Changes in v2:
>> - disabled 'max' tests on OpenBSD
>> - moved tags to 'one tag per line'
>> - added 'os:linux' tags to Alpine ones
>> - Link to v1: 
>> https://lore.kernel.org/r/20240313-sbsa-ref-firmware-update-v1-0-e166703c5424@linaro.org
>>
>> ---
>> Marcin Juszkiewicz (4):
>>        tests/avocado: update sbsa-ref firmware
>>        tests/avocado: drop virtio-rng from sbsa-ref tests
>>        tests/avocado: sbsa-ref: add Alpine tests for misc 'max' setup
>>        tests/avocado: sbsa-ref: add OpenBSD tests for misc 'max' setup
> 
> $ make check-avocado AVOCADO_TAGS='machine:sbsa-ref'
> ninja: no work to do.
> JOB ID     : 76d5dc90c6f70f0801c5269ff1c1db6c0d2cb27b
> JOB LOG    : 
> build/system_arm/tests/results/job-2024-03-18T15.54-76d5dc9/job.log
>   (1/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_edk2_firmware: PASS (4.96 s)
>   (2/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_cortex_a57: PASS (52.67 s)
>   (3/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_neoverse_n1: PASS (51.01 s)
>   (4/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{'name': '4-tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max', 'logdir': 'build/system_... (180.50 s)
>   (5/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_openbsd73_cortex_a57: PASS (21.15 s)
>   (6/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_openbsd73_neoverse_n1: PASS (20.88 s)
>   (7/7) 
> tests/avocado/machine_aarch64_sbsaref.py:Aarch64SbsarefMachine.test_sbsaref_openbsd73_max: PASS (20.70 s)
> RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 1 | 
> CANCEL 0
> JOB TIME   : 355.21 s
> make: *** [tests/Makefile.include:139: check-avocado] Error 8
> 
> Looking at debug.log:
> 
> 15:56:17 DEBUG| Opening console file
> 15:56:17 DEBUG| Opening console socket
> 15:56:17 DEBUG| NOTICE:  Booting Trusted Firmware
> 15:56:17 DEBUG| NOTICE:  BL1: v2.9(release):v2.9.0-764-g7c3ff62d2
> 15:56:17 DEBUG| NOTICE:  BL1: Built : 12:32:06, Sep 27 2023
> 15:56:17 DEBUG| NOTICE:  BL1: Booting BL2
> 15:56:17 DEBUG| NOTICE:  BL2: v2.9(release):v2.9.0-764-g7c3ff62d2
> 15:56:17 DEBUG| NOTICE:  BL2: Built : 12:32:06, Sep 27 2023
> 15:56:17 DEBUG| NOTICE:  BL1: Booting BL31
> 15:56:17 DEBUG| NOTICE:  BL31: v2.9(release):v2.9.0-764-g7c3ff62d2
> 15:56:17 DEBUG| NOTICE:  BL31: Built : 12:32:06, Sep 27 2023
> 15:56:17 DEBUG| UEFI firmware (version 1.0 built at 15:45:23 on Sep 20 
> 2023)
> ...
> 15:57:48 DEBUG| * Mounting security filesystem ... [ ok ]
> 15:57:49 DEBUG| * Mounting debug filesystem ... [ ok ]
> 15:57:49 DEBUG| * Mounting persistent storage (pstore) filesystem ... [ 
> ok ]
> 15:57:50 DEBUG| * Mounting efivarfs filesystem ... [ ok ]
> 15:57:51 DEBUG| * Starting busybox mdev ... [ ok ]
> 15:58:05 DEBUG| * Scanning hardware for mdev ... [ ok ]
> 15:59:05 DEBUG| * Loading hardware drivers ... [ ok ]
> 15:59:05 DEBUG| * WARNING: clock skew detected!
> 15:59:07 DEBUG| * Setting the local clock based on last shutdown time 
> ... [ ok ]
> 15:59:11 DEBUG| * Loading modules ... [ ok ]
> 15:59:13 ERROR|
> 15:59:13 ERROR| RuntimeError: Test interrupted by SIGTERM
>