Hi,

This RFC series add simple acceptance tests which boot SeaBIOS and EDK2 on
Q35 and virt/aarch64.

It is more of a proof of concept (to motivate the Avocado team ;) ).

Regards,

Phil.

Philippe Mathieu-Daudé (3):
  acceptance tests: Add SeaBIOS boot and debug console checking test
  acceptance tests: Add EDK2 OVMF boot and debug console checking test
  acceptance tests: Add EDK2 AAVMF boot and console checking test

 tests/acceptance/boot_firmware.py | 167 ++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
 create mode 100644 tests/acceptance/boot_firmware.py

-- 
2.17.1

Hi Phil,

(+Daniel, +Kashyap)

On 09/28/18 02:30, Philippe Mathieu-Daudé wrote:
> Hi,
>
> This RFC series add simple acceptance tests which boot SeaBIOS and
> EDK2 on Q35 and virt/aarch64.
>
> It is more of a proof of concept (to motivate the Avocado team ;) ).
>
> Regards,
>
> Phil.
>
> Philippe Mathieu-Daudé (3):
>   acceptance tests: Add SeaBIOS boot and debug console checking test
>   acceptance tests: Add EDK2 OVMF boot and debug console checking test
>   acceptance tests: Add EDK2 AAVMF boot and console checking test
>
>  tests/acceptance/boot_firmware.py | 167 ++++++++++++++++++++++++++++++
>  1 file changed, 167 insertions(+)
>  create mode 100644 tests/acceptance/boot_firmware.py
>

I'm not experienced with Avocado, so I'm basically reading the patches
as a "story". My comments are made at that level too. :)

* In the blurb, you say "Q35". But the first two patches have

  vm.set_machine('pc')

* Please don't call the edk2 ArmVirtQemu platform AAVMF in upstream
  patches :) Call it ArmVirtQemu pls.

* Finding the right way to launch  OVMF and/or ArmVirtQemu firmware
  images is complicated. (The right way is definitely not "-bios"!)

  The general idea is that you need three files (and two pflash chips);
  (a) a firmware executable mapped read-only, and (b) a variable store
  file, mapped read-write, that was first copied from (c) a read-only
  variable store *template* that is never itself mapped. And, this is
  not the whole story.

  Figuring out the options is complicated enough (for management tools
  as well) that Daniel made us define a metadata schema for describing
  firmware packages. Please see:

  docs/interop/firmware.json

  I'm not necessarily suggesting that Avocado be able to parse the
  firmware descriptor metafiles that conform to this schema. I'm just
  pointing out that the QEMU command line will depend on the exact build
  of the firmware image under test. The pathname
  "/usr/share/OVMF/OVMF_CODE.fd" and the URL
  "https://snapshots.linaro.org/.../QEMU_EFI.img.gz" don't give us
  enough information.

  Therefore, if we want to keep the test case simple (= hard-wire the
  command lines), then we'll have to refer to OVMF and ArmVirtQemu
  images with precisely known build configs.

* Looking for debug console messages as "vital signs" is brittle. For
  example, the line "DetectSmbiosVersion: SMBIOS version from QEMU:
  0x0208" will change if QEMU changes the SMBIOS version number in the
  SMBIOS anchor that it generates. It's likely better to make the
  firmware "do" something.

  The simplest I can imagine is: prepare a virtual disk with a
  "startup.nsh" UEFI shell script on it. The script can print a known
  fixed string, and then power down the VM. (See the UEFI Shell
  Specification for commands; <http://uefi.org/specifications>.)

  I'm not sure if Avocado provides disk image preparation utilities, but
  perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
  preformat a small image, and track it as a binary file in git.

Thank you for working on this!
Laszlo

On 9/28/18 6:51 AM, Laszlo Ersek wrote:
> Hi Phil,
> 
> (+Daniel, +Kashyap)
> 
> On 09/28/18 02:30, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> This RFC series add simple acceptance tests which boot SeaBIOS and
>> EDK2 on Q35 and virt/aarch64.
>>
>> It is more of a proof of concept (to motivate the Avocado team ;) ).
>>
>> Regards,
>>
>> Phil.
>>
>> Philippe Mathieu-Daudé (3):
>>   acceptance tests: Add SeaBIOS boot and debug console checking test
>>   acceptance tests: Add EDK2 OVMF boot and debug console checking test
>>   acceptance tests: Add EDK2 AAVMF boot and console checking test
>>
>>  tests/acceptance/boot_firmware.py | 167 ++++++++++++++++++++++++++++++
>>  1 file changed, 167 insertions(+)
>>  create mode 100644 tests/acceptance/boot_firmware.py
>>
> 
> I'm not experienced with Avocado, so I'm basically reading the patches
> as a "story". My comments are made at that level too. :)
> 
> * In the blurb, you say "Q35". But the first two patches have
> 
>   vm.set_machine('pc')
> 
> * Please don't call the edk2 ArmVirtQemu platform AAVMF in upstream
>   patches :) Call it ArmVirtQemu pls.
> 
> * Finding the right way to launch  OVMF and/or ArmVirtQemu firmware
>   images is complicated. (The right way is definitely not "-bios"!)
> 
>   The general idea is that you need three files (and two pflash chips);
>   (a) a firmware executable mapped read-only, and (b) a variable store
>   file, mapped read-write, that was first copied from (c) a read-only
>   variable store *template* that is never itself mapped. And, this is
>   not the whole story.
> 
>   Figuring out the options is complicated enough (for management tools
>   as well) that Daniel made us define a metadata schema for describing
>   firmware packages. Please see:
> 
>   docs/interop/firmware.json
> 
>   I'm not necessarily suggesting that Avocado be able to parse the
>   firmware descriptor metafiles that conform to this schema. I'm just
>   pointing out that the QEMU command line will depend on the exact build
>   of the firmware image under test. The pathname
>   "/usr/share/OVMF/OVMF_CODE.fd" and the URL
>   "https://snapshots.linaro.org/.../QEMU_EFI.img.gz" don't give us
>   enough information.
> 
>   Therefore, if we want to keep the test case simple (= hard-wire the
>   command lines), then we'll have to refer to OVMF and ArmVirtQemu
>   images with precisely known build configs.
> 
> * Looking for debug console messages as "vital signs" is brittle. For
>   example, the line "DetectSmbiosVersion: SMBIOS version from QEMU:
>   0x0208" will change if QEMU changes the SMBIOS version number in the
>   SMBIOS anchor that it generates. It's likely better to make the
>   firmware "do" something.
> 
>   The simplest I can imagine is: prepare a virtual disk with a
>   "startup.nsh" UEFI shell script on it. The script can print a known
>   fixed string, and then power down the VM. (See the UEFI Shell
>   Specification for commands; <http://uefi.org/specifications>.)
> 
>   I'm not sure if Avocado provides disk image preparation utilities, but
>   perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
>   preformat a small image, and track it as a binary file in git.
> 

So far we've added support for generating ISO images (with pure Python).
 I'm not sure if that's useful here.  We can think about trying to add
the same thing for vvfat.

- Cleber.

On 10/03/18 02:23, Cleber Rosa wrote:
> On 9/28/18 6:51 AM, Laszlo Ersek wrote:

>>   I'm not sure if Avocado provides disk image preparation utilities, but
>>   perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
>>   preformat a small image, and track it as a binary file in git.
>>
> 
> So far we've added support for generating ISO images (with pure Python).
> I'm not sure if that's useful here.  We can think about trying to add
> the same thing for vvfat.

The ability to generate ISO images (natively at that!) seems useful.
UEFI-readable ISO images need an extension on top: the ISO9660
filesystem has to get the ElTorito extension, and the ElTorito boot
image should be a FAT filesystem. Under UEFI, what's visible isn't the
ISO9660 filesystem itself, but the contents of the embedded ElTorito
boot image.

In terms of shell utilities, this usually involves:

- creating and populating the FAT filesystem image (with guestfish, or
  with mkdosfs+mtools),

- invoking genisoimage with "-efi-boot fat.img -no-emul-boot -- fat.img"
  (basically, place the FAT image into the ISO9660 filesystem like any
  other file, but pass additional options so that genisoimage know it's
  meant as the ElTorito boot image for UEFI).

If you can add this feature generically, such as:
- input: a hierarchy of files,
- output: a temporary ISO image file,

then IMO it would help with UEFI testing. Any given test case could then
generate a "startup.nsh" UEFI shell script, invoking UEFI shell
builtins, and possibly custom UEFI applications (also to be placed in
the ISO image). This could cover a good amount of batch scenarios (where
no interaction is needed).

Regarding interaction with the UEFI shell over serial, the
<https://github.com/puiterwijk/qemu-ovmf-secureboot> project for example
has successfully used "subprocess.Popen()". But, I guess I only want to
highlight the idea here ("talk to the UEFI shell via serial"), not the
exact implementation. I assume Avocado already has polished,
"expect"-like tools, for talking to the guest over the serial port.

Thanks!
Laszlo

On 10/3/18 3:13 AM, Laszlo Ersek wrote:
> On 10/03/18 02:23, Cleber Rosa wrote:
>> On 9/28/18 6:51 AM, Laszlo Ersek wrote:
> 
>>>   I'm not sure if Avocado provides disk image preparation utilities, but
>>>   perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
>>>   preformat a small image, and track it as a binary file in git.
>>>
>>
>> So far we've added support for generating ISO images (with pure Python).
>> I'm not sure if that's useful here.  We can think about trying to add
>> the same thing for vvfat.
> 
> The ability to generate ISO images (natively at that!) seems useful.
> UEFI-readable ISO images need an extension on top: the ISO9660
> filesystem has to get the ElTorito extension, and the ElTorito boot
> image should be a FAT filesystem. Under UEFI, what's visible isn't the
> ISO9660 filesystem itself, but the contents of the embedded ElTorito
> boot image.
> 
> In terms of shell utilities, this usually involves:
> 
> - creating and populating the FAT filesystem image (with guestfish, or
>   with mkdosfs+mtools),
> 

Is FAT12 an option here?  The reason I ask is that there may be code
FAT12 capable code ready to be incorporated:

https://github.com/clalancette/pyfat

> - invoking genisoimage with "-efi-boot fat.img -no-emul-boot -- fat.img"
>   (basically, place the FAT image into the ISO9660 filesystem like any
>   other file, but pass additional options so that genisoimage know it's
>   meant as the ElTorito boot image for UEFI).
> 

ISO with ElTorito should be easy to do with what we have right now, I'll
check what "-efi-boot" and "-no-emul-boot" do to make sure we can mimic it.

> If you can add this feature generically, such as:
> - input: a hierarchy of files,
> - output: a temporary ISO image file,
> 

Yes, this sounds like the interface we need here.

> then IMO it would help with UEFI testing. Any given test case could then
> generate a "startup.nsh" UEFI shell script, invoking UEFI shell
> builtins, and possibly custom UEFI applications (also to be placed in
> the ISO image). This could cover a good amount of batch scenarios (where
> no interaction is needed).
> 
> Regarding interaction with the UEFI shell over serial, the
> <https://github.com/puiterwijk/qemu-ovmf-secureboot> project for example
> has successfully used "subprocess.Popen()". But, I guess I only want to
> highlight the idea here ("talk to the UEFI shell via serial"), not the
> exact implementation. I assume Avocado already has polished,
> "expect"-like tools, for talking to the guest over the serial port.
> 

Avocado has a subproject called aexpect:

https://github.com/avocado-framework/aexpect

It can be used to communicate with (among others) serial consoles.  This
is how it's done on Avocado-VT:

https://github.com/avocado-framework/avocado-vt/blob/master/virttest/qemu_vm.py#L2376

Thanks!
- Cleber.

> Thanks!
> Laszlo
> 

On 10/03/18 17:20, Cleber Rosa wrote:
> On 10/3/18 3:13 AM, Laszlo Ersek wrote:
>> On 10/03/18 02:23, Cleber Rosa wrote:
>>> On 9/28/18 6:51 AM, Laszlo Ersek wrote:
>>
>>>>   I'm not sure if Avocado provides disk image preparation utilities, but
>>>>   perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
>>>>   preformat a small image, and track it as a binary file in git.
>>>>
>>>
>>> So far we've added support for generating ISO images (with pure Python).
>>> I'm not sure if that's useful here.  We can think about trying to add
>>> the same thing for vvfat.
>>
>> The ability to generate ISO images (natively at that!) seems useful.
>> UEFI-readable ISO images need an extension on top: the ISO9660
>> filesystem has to get the ElTorito extension, and the ElTorito boot
>> image should be a FAT filesystem. Under UEFI, what's visible isn't the
>> ISO9660 filesystem itself, but the contents of the embedded ElTorito
>> boot image.
>>
>> In terms of shell utilities, this usually involves:
>>
>> - creating and populating the FAT filesystem image (with guestfish, or
>>   with mkdosfs+mtools),
>>
> 
> Is FAT12 an option here?  The reason I ask is that there may be code
> FAT12 capable code ready to be incorporated:
> 
> https://github.com/clalancette/pyfat

Theoretically, I should answer "yes". For two reasons:

(1) In "13.3 File System Format", the UEFI-2.7 spec writes,

"[...] EFI encompasses the use of FAT32 for a system partition, and
FAT12 or FAT16 for removable media. [...]"

(2) When invoking mkdosfs without the "-F" option, mkdosfs chooses the
smallest FAT filesystem variant that can accommodate the requested size.
We successfully format UEFI-readable ISO images that don't exceed e.g.
3MB in final size. This implies (and I have now actually checked, with
"dosfsck -v") that their embedded ElTorito image is FAT12. edk2 has no
trouble reading that.


However... the maximum volume size for FAT12 appears to be 32 MB,
according to wikipedia:

  https://en.wikipedia.org/wiki/File_Allocation_Table#FAT12

It doesn't look good for the long term. For example, I can imagine a
test case where you place a kernel executable (containing a UEFI stub)
and an initial ramdisk on the UEFI-readable ISO. Using the RHEL-7.5
kernel and the matching initrd from my laptop as an example: that's
already 6.2MB + 25MB.

So, technically, FAT12 should be fine; in practice, it could prove limiting.

[...]

Thanks!
Laszlo

On 10/3/18 11:59 AM, Laszlo Ersek wrote:
> On 10/03/18 17:20, Cleber Rosa wrote:
>> On 10/3/18 3:13 AM, Laszlo Ersek wrote:
>>> On 10/03/18 02:23, Cleber Rosa wrote:
>>>> On 9/28/18 6:51 AM, Laszlo Ersek wrote:
>>>
>>>>>   I'm not sure if Avocado provides disk image preparation utilities, but
>>>>>   perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
>>>>>   preformat a small image, and track it as a binary file in git.
>>>>>
>>>>
>>>> So far we've added support for generating ISO images (with pure Python).
>>>> I'm not sure if that's useful here.  We can think about trying to add
>>>> the same thing for vvfat.
>>>
>>> The ability to generate ISO images (natively at that!) seems useful.
>>> UEFI-readable ISO images need an extension on top: the ISO9660
>>> filesystem has to get the ElTorito extension, and the ElTorito boot
>>> image should be a FAT filesystem. Under UEFI, what's visible isn't the
>>> ISO9660 filesystem itself, but the contents of the embedded ElTorito
>>> boot image.
>>>
>>> In terms of shell utilities, this usually involves:
>>>
>>> - creating and populating the FAT filesystem image (with guestfish, or
>>>   with mkdosfs+mtools),
>>>
>>
>> Is FAT12 an option here?  The reason I ask is that there may be code
>> FAT12 capable code ready to be incorporated:
>>
>> https://github.com/clalancette/pyfat
> 
> Theoretically, I should answer "yes". For two reasons:
> 
> (1) In "13.3 File System Format", the UEFI-2.7 spec writes,
> 
> "[...] EFI encompasses the use of FAT32 for a system partition, and
> FAT12 or FAT16 for removable media. [...]"
> 
> (2) When invoking mkdosfs without the "-F" option, mkdosfs chooses the
> smallest FAT filesystem variant that can accommodate the requested size.
> We successfully format UEFI-readable ISO images that don't exceed e.g.
> 3MB in final size. This implies (and I have now actually checked, with
> "dosfsck -v") that their embedded ElTorito image is FAT12. edk2 has no
> trouble reading that.
> 
> 
> However... the maximum volume size for FAT12 appears to be 32 MB,
> according to wikipedia:
> 
>   https://en.wikipedia.org/wiki/File_Allocation_Table#FAT12
> 
> It doesn't look good for the long term. For example, I can imagine a
> test case where you place a kernel executable (containing a UEFI stub)
> and an initial ramdisk on the UEFI-readable ISO. Using the RHEL-7.5
> kernel and the matching initrd from my laptop as an example: that's
> already 6.2MB + 25MB.
> 
> So, technically, FAT12 should be fine; in practice, it could prove limiting.
> 

Nice, thanks for the detailed info.  I think it's safe to start with
that, and improve the original project with FAT16/32 support.

Regards,
- Cleber.

> [...]
> 
> Thanks!
> Laszlo
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Hi Laszlo,

On 28/09/2018 12:51, Laszlo Ersek wrote:
> Hi Phil,
> 
> (+Daniel, +Kashyap)
> 
> On 09/28/18 02:30, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> This RFC series add simple acceptance tests which boot SeaBIOS and
>> EDK2 on Q35 and virt/aarch64.
>>
>> It is more of a proof of concept (to motivate the Avocado team ;) ).
>>
>> Regards,
>>
>> Phil.
>>
>> Philippe Mathieu-Daudé (3):
>>   acceptance tests: Add SeaBIOS boot and debug console checking test
>>   acceptance tests: Add EDK2 OVMF boot and debug console checking test
>>   acceptance tests: Add EDK2 AAVMF boot and console checking test
>>
>>  tests/acceptance/boot_firmware.py | 167 ++++++++++++++++++++++++++++++
>>  1 file changed, 167 insertions(+)
>>  create mode 100644 tests/acceptance/boot_firmware.py
>>
> 
> I'm not experienced with Avocado, so I'm basically reading the patches
> as a "story". My comments are made at that level too. :)

You gave the kind of useful comment I was expecting :) This is what RFC
are for after all.

The idea of this series is to provide users the ability to hack QEMU and
tests they didn't break the boot process (interaction with firmwares).

> 
> * In the blurb, you say "Q35". But the first two patches have
> 
>   vm.set_machine('pc')

Oops, mistake. I suppose we want to have both covered.

> 
> * Please don't call the edk2 ArmVirtQemu platform AAVMF in upstream
>   patches :) Call it ArmVirtQemu pls.

OK.

> 
> * Finding the right way to launch  OVMF and/or ArmVirtQemu firmware
>   images is complicated. (The right way is definitely not "-bios"!)>
>   The general idea is that you need three files (and two pflash chips);
>   (a) a firmware executable mapped read-only, and (b) a variable store
>   file, mapped read-write, that was first copied from (c) a read-only
>   variable store *template* that is never itself mapped. And, this is
>   not the whole story.

We definitively need the variable store to run useful tests.

> 
>   Figuring out the options is complicated enough (for management tools
>   as well) that Daniel made us define a metadata schema for describing
>   firmware packages. Please see:
> 
>   docs/interop/firmware.json
> 
>   I'm not necessarily suggesting that Avocado be able to parse the
>   firmware descriptor metafiles that conform to this schema. I'm just
>   pointing out that the QEMU command line will depend on the exact build
>   of the firmware image under test. The pathname
>   "/usr/share/OVMF/OVMF_CODE.fd" and the URL
>   "https://snapshots.linaro.org/.../QEMU_EFI.img.gz" don't give us
>   enough information.
> 
>   Therefore, if we want to keep the test case simple (= hard-wire the
>   command lines), then we'll have to refer to OVMF and ArmVirtQemu
>   images with precisely known build configs.

To focus on QEMU here, I'd like to not have to build a know EDK2 config
but rather use releases (or snapshots).

scripts/qemu.py should provide utilities to pack a flash image for
further testing.

IMO tests using a stable QEMU to test different EDK2 configs have to go
in another repository.
But this is true there are also linked and usually to test a new feature
in one component, you need to update/build the other.

> * Looking for debug console messages as "vital signs" is brittle. For
>   example, the line "DetectSmbiosVersion: SMBIOS version from QEMU:
>   0x0208" will change if QEMU changes the SMBIOS version number in the
>   SMBIOS anchor that it generates. It's likely better to make the
>   firmware "do" something.
> 
>   The simplest I can imagine is: prepare a virtual disk with a
>   "startup.nsh" UEFI shell script on it. The script can print a known
>   fixed string, and then power down the VM. (See the UEFI Shell
>   Specification for commands; <http://uefi.org/specifications>.)

I didn't want to interact with the console but your suggestion is simple
enough for a PoC.

> 
>   I'm not sure if Avocado provides disk image preparation utilities, but
>   perhaps (a) we could use the vvfat driver (*shudder*) or (b) we could
>   preformat a small image, and track it as a binary file in git.

I guess we have to go with the utilities (for reproducibility).

Thanks for your comments,

Phil.

On 09/28/18 19:07, Philippe Mathieu-Daudé wrote:
> On 28/09/2018 12:51, Laszlo Ersek wrote:

>>   Figuring out the options is complicated enough (for management tools
>>   as well) that Daniel made us define a metadata schema for describing
>>   firmware packages. Please see:
>>
>>   docs/interop/firmware.json
>>
>>   I'm not necessarily suggesting that Avocado be able to parse the
>>   firmware descriptor metafiles that conform to this schema. I'm just
>>   pointing out that the QEMU command line will depend on the exact build
>>   of the firmware image under test. The pathname
>>   "/usr/share/OVMF/OVMF_CODE.fd" and the URL
>>   "https://snapshots.linaro.org/.../QEMU_EFI.img.gz" don't give us
>>   enough information.
>>
>>   Therefore, if we want to keep the test case simple (= hard-wire the
>>   command lines), then we'll have to refer to OVMF and ArmVirtQemu
>>   images with precisely known build configs.
> 
> To focus on QEMU here, I'd like to not have to build a know EDK2 config
> but rather use releases (or snapshots).

If you mean specific binary releases (released by some well-known
specific party), that should work; but they should be available to everyone.

If you mean source releases / snapshots, those don't provide enough
information. You can still build OVMF with and without -D SMM_REQUIRE,
for example, and the QEMU command line options will differ.

This is not much different from using a Linux guest kernel image for
testing. For example, you can build Linux without virtio drivers, and
then you can't use the resultant image for checking QEMU's virtio
devices. I.e., the guest build config is part of the equation.

> scripts/qemu.py should provide utilities to pack a flash image for
> further testing.

AIUI, upstream QEMU wouldn't like to bundle more binaries / depend on
more external projects.

Also, if you build OVMF with a feature that requires OpenSSL, then the
resultant binary (covered by the 2-clause BSD license and the OpenSSL
license) might not be suitable for bundling with QEMU.

Thanks,
Laszlo