Repo:   https://github.com/lersek/qemu.git
Branch: edk2_build_v3

Version 2, that is:

  [Qemu-devel] [PATCH v2 00/12] bundle edk2 platform firmware with QEMU

was posted at:

  https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04670.html
  http://mid.mail-archive.com/20190313210057.32584-1-lersek@redhat.com

Updates in v3 are noted on each patch individually, in the Notes
section. In summary,

- I've picked up feedback tags from the v2 thread (see above),

- I've replaced the xz compression with bz2 compression according to the
  subthread

    Re: [Qemu-devel] [PULL 00/12] EDK2 Firmware roms

  at:

    https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg06016.html
    http://mid.mail-archive.com/CAFEAcA-UcX6Brv3O28vcCw31bt=QpvbUwnPBBjhETm=Vfciuvw@mail.gmail.com

I'm pasting a cumulative diffstat, and diff, between v2 and v3 below.
Note that a normal diffstat for this v3 series is at the bottom of the
cover letter, as usual.

>  Makefile                               |   6 +++---
>  configure                              |   2 +-
>  pc-bios/README                         |   2 +-
>  pc-bios/edk2-aarch64-code.fd.bz2       | Bin 0 -> 1177603 bytes
>  pc-bios/edk2-aarch64-code.fd.xz        | Bin 1146804 -> 0 bytes
>  pc-bios/edk2-arm-code.fd.bz2           | Bin 0 -> 1173662 bytes
>  pc-bios/edk2-arm-code.fd.xz            | Bin 1147852 -> 0 bytes
>  pc-bios/edk2-arm-vars.fd.bz2           | Bin 0 -> 263 bytes
>  pc-bios/edk2-arm-vars.fd.xz            | Bin 10008 -> 0 bytes
>  pc-bios/edk2-i386-code.fd.bz2          | Bin 0 -> 1688659 bytes
>  pc-bios/edk2-i386-code.fd.xz           | Bin 1674764 -> 0 bytes
>  pc-bios/edk2-i386-secure-code.fd.bz2   | Bin 0 -> 1881979 bytes
>  pc-bios/edk2-i386-secure-code.fd.xz    | Bin 1870024 -> 0 bytes
>  pc-bios/edk2-i386-vars.fd.bz2          | Bin 0 -> 190 bytes
>  pc-bios/edk2-i386-vars.fd.xz           | Bin 320 -> 0 bytes
>  pc-bios/edk2-x86_64-code.fd.bz2        | Bin 0 -> 1669280 bytes
>  pc-bios/edk2-x86_64-code.fd.xz         | Bin 1655276 -> 0 bytes
>  pc-bios/edk2-x86_64-secure-code.fd.bz2 | Bin 0 -> 1901210 bytes
>  pc-bios/edk2-x86_64-secure-code.fd.xz  | Bin 1889024 -> 0 bytes
>  roms/Makefile.edk2                     |   6 +++---
>  20 files changed, 8 insertions(+), 8 deletions(-)

> diff --git a/configure b/configure
> index 5061f68653a3..df6e01ada833 100755
> --- a/configure
> +++ b/configure
> @@ -7865,7 +7865,7 @@ for bios_file in \
>      $source_path/pc-bios/*.img \
>      $source_path/pc-bios/openbios-* \
>      $source_path/pc-bios/u-boot.* \
> -    $source_path/pc-bios/edk2-*.fd.xz \
> +    $source_path/pc-bios/edk2-*.fd.bz2 \
>      $source_path/pc-bios/palcode-*
>  do
>      LINKS="$LINKS pc-bios/$(basename $bios_file)"
> diff --git a/Makefile b/Makefile
> index 6c5c1a1cad6e..f4562092e8dc 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -296,9 +296,9 @@ ui/input-keymap-%.c: $(KEYCODEMAP_GEN) $(KEYCODEMAP_CSV) $(SRC_PATH)/ui/Makefile
>  $(KEYCODEMAP_GEN): .git-submodule-status
>  $(KEYCODEMAP_CSV): .git-submodule-status
>
> -edk2-decompressed = $(basename $(wildcard pc-bios/edk2-*.fd.xz))
> -pc-bios/edk2-%.fd: pc-bios/edk2-%.fd.xz
> -	$(call quiet-command,xz -d -c $< > $@,"UNXZ",$<)
> +edk2-decompressed = $(basename $(wildcard pc-bios/edk2-*.fd.bz2))
> +pc-bios/edk2-%.fd: pc-bios/edk2-%.fd.bz2
> +	$(call quiet-command,bzip2 -d -c $< > $@,"BUNZIP2",$<)
>
>  # Don't try to regenerate Makefile or configure
>  # We don't generate any of them
> diff --git a/roms/Makefile.edk2 b/roms/Makefile.edk2
> index e5c3036949c8..822c547fec64 100644
> --- a/roms/Makefile.edk2
> +++ b/roms/Makefile.edk2
> @@ -34,11 +34,11 @@ flashdevs := \
>  	arm-vars \
>  	i386-vars
>
> -all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.xz) \
> +all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
>  	../pc-bios/edk2-licenses.txt
>
> -../pc-bios/edk2-%.fd.xz: ../pc-bios/edk2-%.fd
> -	xz -9 -e -c $< > $@
> +../pc-bios/edk2-%.fd.bz2: ../pc-bios/edk2-%.fd
> +	bzip2 -9 -c $< > $@
>
>  # When the build completes, we need not keep the uncompressed flash device
>  # files.
> diff --git a/pc-bios/README b/pc-bios/README
> index f6acd3209610..807d95dd5418 100644
> --- a/pc-bios/README
> +++ b/pc-bios/README
> @@ -51,7 +51,7 @@
>  - QemuMacDrivers (https://github.com/ozbenh/QemuMacDrivers) is a project to
>    provide virtualised drivers for PPC MacOS guests.
>
> -- The "edk2-*.fd.xz" images are platform firmware binaries and matching UEFI
> +- The "edk2-*.fd.bz2" images are platform firmware binaries and matching UEFI
>    variable store templates built from the TianoCore community's EFI Development
>    Kit II project
>    <https://github.com/tianocore/tianocore.github.io/wiki/EDK-II>. The images
> diff --git a/pc-bios/edk2-aarch64-code.fd.bz2 b/pc-bios/edk2-aarch64-code.fd.bz2
> new file mode 100644
> index 000000000000..b213334d9553
> Binary files /dev/null and b/pc-bios/edk2-aarch64-code.fd.bz2 differ
> diff --git a/pc-bios/edk2-aarch64-code.fd.xz b/pc-bios/edk2-aarch64-code.fd.xz
> deleted file mode 100644
> index 8ca34c56b51b..000000000000
> Binary files a/pc-bios/edk2-aarch64-code.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-arm-code.fd.bz2 b/pc-bios/edk2-arm-code.fd.bz2
> new file mode 100644
> index 000000000000..a494a8391eed
> Binary files /dev/null and b/pc-bios/edk2-arm-code.fd.bz2 differ
> diff --git a/pc-bios/edk2-arm-code.fd.xz b/pc-bios/edk2-arm-code.fd.xz
> deleted file mode 100644
> index db2c20b71e29..000000000000
> Binary files a/pc-bios/edk2-arm-code.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-arm-vars.fd.bz2 b/pc-bios/edk2-arm-vars.fd.bz2
> new file mode 100644
> index 000000000000..2a5a9af31a6c
> Binary files /dev/null and b/pc-bios/edk2-arm-vars.fd.bz2 differ
> diff --git a/pc-bios/edk2-arm-vars.fd.xz b/pc-bios/edk2-arm-vars.fd.xz
> deleted file mode 100644
> index 8c3a4b5c8bdc..000000000000
> Binary files a/pc-bios/edk2-arm-vars.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-i386-code.fd.bz2 b/pc-bios/edk2-i386-code.fd.bz2
> new file mode 100644
> index 000000000000..5440ca856db6
> Binary files /dev/null and b/pc-bios/edk2-i386-code.fd.bz2 differ
> diff --git a/pc-bios/edk2-i386-code.fd.xz b/pc-bios/edk2-i386-code.fd.xz
> deleted file mode 100644
> index f25dbec37e72..000000000000
> Binary files a/pc-bios/edk2-i386-code.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-i386-secure-code.fd.bz2 b/pc-bios/edk2-i386-secure-code.fd.bz2
> new file mode 100644
> index 000000000000..24ae88ac3e40
> Binary files /dev/null and b/pc-bios/edk2-i386-secure-code.fd.bz2 differ
> diff --git a/pc-bios/edk2-i386-secure-code.fd.xz b/pc-bios/edk2-i386-secure-code.fd.xz
> deleted file mode 100644
> index 11d499b90d85..000000000000
> Binary files a/pc-bios/edk2-i386-secure-code.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-i386-vars.fd.bz2 b/pc-bios/edk2-i386-vars.fd.bz2
> new file mode 100644
> index 000000000000..4a1bc4523880
> Binary files /dev/null and b/pc-bios/edk2-i386-vars.fd.bz2 differ
> diff --git a/pc-bios/edk2-i386-vars.fd.xz b/pc-bios/edk2-i386-vars.fd.xz
> deleted file mode 100644
> index 1656532daa6e..000000000000
> Binary files a/pc-bios/edk2-i386-vars.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-x86_64-code.fd.bz2 b/pc-bios/edk2-x86_64-code.fd.bz2
> new file mode 100644
> index 000000000000..6a0b1f983199
> Binary files /dev/null and b/pc-bios/edk2-x86_64-code.fd.bz2 differ
> diff --git a/pc-bios/edk2-x86_64-code.fd.xz b/pc-bios/edk2-x86_64-code.fd.xz
> deleted file mode 100644
> index 92398935a482..000000000000
> Binary files a/pc-bios/edk2-x86_64-code.fd.xz and /dev/null differ
> diff --git a/pc-bios/edk2-x86_64-secure-code.fd.bz2 b/pc-bios/edk2-x86_64-secure-code.fd.bz2
> new file mode 100644
> index 000000000000..3ef6452c0375
> Binary files /dev/null and b/pc-bios/edk2-x86_64-secure-code.fd.bz2 differ
> diff --git a/pc-bios/edk2-x86_64-secure-code.fd.xz b/pc-bios/edk2-x86_64-secure-code.fd.xz
> deleted file mode 100644
> index a95891dcf134..000000000000
> Binary files a/pc-bios/edk2-x86_64-secure-code.fd.xz and /dev/null differ

Thanks,
Laszlo

Laszlo Ersek (12):
  roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
  roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
  tests/uefi-test-tools/build.sh: work around TianoCore#1607
  roms/edk2: advance to tag edk2-stable201903
  roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
  roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
  roms: build edk2 firmware binaries and variable store templates
  pc-bios: add edk2 firmware binaries and variable store templates
  pc-bios: document the edk2 firmware images; add firmware descriptors
  tests: add missing dependency to build QTEST_QEMU_BINARY, round 2
  Makefile: install the edk2 firmware images and their descriptors
  MAINTAINERS: add the "EDK2 Firmware" subsystem

 .gitignore                                     |   1 +
 MAINTAINERS                                    |  12 +
 Makefile                                       |  29 ++-
 configure                                      |   1 +
 pc-bios/README                                 |  11 +
 pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
 pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
 pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
 pc-bios/descriptors/60-edk2-arm.json           |  31 +++
 pc-bios/descriptors/60-edk2-i386.json          |  33 +++
 pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
 pc-bios/edk2-aarch64-code.fd.bz2               | Bin 0 -> 1177603 bytes
 pc-bios/edk2-arm-code.fd.bz2                   | Bin 0 -> 1173662 bytes
 pc-bios/edk2-arm-vars.fd.bz2                   | Bin 0 -> 263 bytes
 pc-bios/edk2-i386-code.fd.bz2                  | Bin 0 -> 1688659 bytes
 pc-bios/edk2-i386-secure-code.fd.bz2           | Bin 0 -> 1881979 bytes
 pc-bios/edk2-i386-vars.fd.bz2                  | Bin 0 -> 190 bytes
 pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
 pc-bios/edk2-x86_64-code.fd.bz2                | Bin 0 -> 1669280 bytes
 pc-bios/edk2-x86_64-secure-code.fd.bz2         | Bin 0 -> 1901210 bytes
 roms/Makefile                                  |   9 +-
 roms/Makefile.edk2                             | 148 ++++++++++++
 roms/edk2                                      |   2 +-
 roms/edk2-build.sh                             |  55 +++++
 roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
 tests/Makefile.include                         |   2 +-
 tests/uefi-test-tools/build.sh                 | 100 +-------
 27 files changed, 934 insertions(+), 96 deletions(-)
 create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
 create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
 create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
 create mode 100644 pc-bios/descriptors/60-edk2-arm.json
 create mode 100644 pc-bios/descriptors/60-edk2-i386.json
 create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
 create mode 100644 pc-bios/edk2-aarch64-code.fd.bz2
 create mode 100644 pc-bios/edk2-arm-code.fd.bz2
 create mode 100644 pc-bios/edk2-arm-vars.fd.bz2
 create mode 100644 pc-bios/edk2-i386-code.fd.bz2
 create mode 100644 pc-bios/edk2-i386-secure-code.fd.bz2
 create mode 100644 pc-bios/edk2-i386-vars.fd.bz2
 create mode 100644 pc-bios/edk2-licenses.txt
 create mode 100644 pc-bios/edk2-x86_64-code.fd.bz2
 create mode 100644 pc-bios/edk2-x86_64-secure-code.fd.bz2
 create mode 100644 roms/Makefile.edk2
 create mode 100755 roms/edk2-build.sh
 create mode 100644 roms/edk2-funcs.sh

-- 
2.19.1.3.g30247aa5d201

On Thu, 21 Mar 2019 at 11:34, Laszlo Ersek <lersek@redhat.com> wrote:
>
> Repo:   https://github.com/lersek/qemu.git
> Branch: edk2_build_v3
>
> Version 2, that is:
>
>   [Qemu-devel] [PATCH v2 00/12] bundle edk2 platform firmware with QEMU
>
> was posted at:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04670.html
>   http://mid.mail-archive.com/20190313210057.32584-1-lersek@redhat.com
>
> Updates in v3 are noted on each patch individually, in the Notes
> section. In summary,
>
> - I've picked up feedback tags from the v2 thread (see above),
>
> - I've replaced the xz compression with bz2 compression according to the
>   subthread
>
>     Re: [Qemu-devel] [PULL 00/12] EDK2 Firmware roms

Thanks. Could you check that this works in the OpenBSD VM test?
It should just be a matter of
  make -C build vm-build-openbsd

(with 'build' replaced with whatever your build directory's name is):
it will download the VM image and run it automatically. You'll
want to be doing this as a user that can use KVM so that we
do the build in a KVM VM rather than a slow emulated one :-))

thanks
-- PMM

On 03/21/19 12:52, Peter Maydell wrote:
> On Thu, 21 Mar 2019 at 11:34, Laszlo Ersek <lersek@redhat.com> wrote:
>>
>> Repo:   https://github.com/lersek/qemu.git
>> Branch: edk2_build_v3
>>
>> Version 2, that is:
>>
>>   [Qemu-devel] [PATCH v2 00/12] bundle edk2 platform firmware with QEMU
>>
>> was posted at:
>>
>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04670.html
>>   http://mid.mail-archive.com/20190313210057.32584-1-lersek@redhat.com
>>
>> Updates in v3 are noted on each patch individually, in the Notes
>> section. In summary,
>>
>> - I've picked up feedback tags from the v2 thread (see above),
>>
>> - I've replaced the xz compression with bz2 compression according to the
>>   subthread
>>
>>     Re: [Qemu-devel] [PULL 00/12] EDK2 Firmware roms
> 
> Thanks. Could you check that this works in the OpenBSD VM test?
> It should just be a matter of
>   make -C build vm-build-openbsd

Actually I tried to do that before posting v3, following the
instructions in "docs/devel/testing.rst":

  cd $QEMU_SRC/tests/vm
  ./openbsd --build-image --image /mnt/data/tmp/openbsd.img
  ./openbsd --debug --image /mnt/data/tmp/openbsd.img \
    --build-qemu $QEMU_SRC

The first "openbsd" invocation succeeded, but the second failed, with a
complaint about some (cross?)compiler missing.

Anyway I'll now retry with the make command you specify above.

> (with 'build' replaced with whatever your build directory's name is):

Aha! So first I guess I should run at least configure, for setting up an
out-of-tree build, and then try the "make" command.

> it will download the VM image and run it automatically. You'll
> want to be doing this as a user that can use KVM so that we
> do the build in a KVM VM rather than a slow emulated one :-))

Thanks, I'll report back.

Laszlo

On Thu, 21 Mar 2019 at 16:55, Laszlo Ersek <lersek@redhat.com> wrote:
> On 03/21/19 12:52, Peter Maydell wrote:
> > Thanks. Could you check that this works in the OpenBSD VM test?
> > It should just be a matter of
> >   make -C build vm-build-openbsd
>
> Actually I tried to do that before posting v3, following the
> instructions in "docs/devel/testing.rst":
>
>   cd $QEMU_SRC/tests/vm
>   ./openbsd --build-image --image /mnt/data/tmp/openbsd.img
>   ./openbsd --debug --image /mnt/data/tmp/openbsd.img \
>     --build-qemu $QEMU_SRC

This is in the "manual invocation" subsection of the docs.
You want the "quickstart" subsection which has the same
'make vm-build-whatever' command I suggest.

> Aha! So first I guess I should run at least configure, for setting up an
> out-of-tree build, and then try the "make" command.

You need to configure, yes. I think it should work on an in-tree
build, but I've never tried it as I never use in-tree builds.
(We're probably going to get rid of in-tree build support in 4.1.)

thanks
-- PMM

On 3/21/19 6:03 PM, Peter Maydell wrote:
> (We're probably going to get rid of in-tree build support in 4.1.)

Sorry for hijicking the thread, but can you point in me the discussion 
on the list where was this discussed please? I'm unable to find it. As a 
person who does in tree builds only I'm interested to learn more.

Thanks,
Michal

On Mon, 25 Mar 2019 at 08:29, Michal Privoznik <mprivozn@redhat.com> wrote:
>
> On 3/21/19 6:03 PM, Peter Maydell wrote:
> > (We're probably going to get rid of in-tree build support in 4.1.)
>
> Sorry for hijicking the thread, but can you point in me the discussion
> on the list where was this discussed please? I'm unable to find it. As a
> person who does in tree builds only I'm interested to learn more.

https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg01949.html
and subthread.

The short reasoning is that there's nothing you can do with
an in-tree build that you can't do with an out-of-tree build
(whereas the reverse is not true), and maintaining both
mechanisms leads to lots of bugs where we accidentally break
one of them because the developers are only using the other
kind. So in-tree builds are more pain to support than they're
worth.

thanks
-- PMM

On 03/21/19 17:55, Laszlo Ersek wrote:
> On 03/21/19 12:52, Peter Maydell wrote:
>> On Thu, 21 Mar 2019 at 11:34, Laszlo Ersek <lersek@redhat.com> wrote:
>>>
>>> Repo:   https://github.com/lersek/qemu.git
>>> Branch: edk2_build_v3
>>>
>>> Version 2, that is:
>>>
>>>   [Qemu-devel] [PATCH v2 00/12] bundle edk2 platform firmware with QEMU
>>>
>>> was posted at:
>>>
>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04670.html
>>>   http://mid.mail-archive.com/20190313210057.32584-1-lersek@redhat.com
>>>
>>> Updates in v3 are noted on each patch individually, in the Notes
>>> section. In summary,
>>>
>>> - I've picked up feedback tags from the v2 thread (see above),
>>>
>>> - I've replaced the xz compression with bz2 compression according to the
>>>   subthread
>>>
>>>     Re: [Qemu-devel] [PULL 00/12] EDK2 Firmware roms
>>
>> Thanks. Could you check that this works in the OpenBSD VM test?
>> It should just be a matter of
>>   make -C build vm-build-openbsd
>
> Actually I tried to do that before posting v3, following the
> instructions in "docs/devel/testing.rst":
>
>   cd $QEMU_SRC/tests/vm
>   ./openbsd --build-image --image /mnt/data/tmp/openbsd.img
>   ./openbsd --debug --image /mnt/data/tmp/openbsd.img \
>     --build-qemu $QEMU_SRC
>
> The first "openbsd" invocation succeeded, but the second failed, with a
> complaint about some (cross?)compiler missing.
>
> Anyway I'll now retry with the make command you specify above.

Unfortunately, I got the exact same setup error:

> make: Entering directory `.../tmp/qemu-build-opt.d'
>     VM-IMAGE openbsd
> --2019-03-21 18:14:46--  http://download.patchew.org/openbsd-6.1-amd64.img.xz
> Resolving download.patchew.org (download.patchew.org)... 140.211.15.4
> Connecting to download.patchew.org (download.patchew.org)|140.211.15.4|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 675094624 (644M) [application/octet-stream]
> Saving to: '.../.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd.download'
>
> 100%[=====================================================================================================================================================================================================>] 675,094,624 12.0MB/s   in 53s
>
> 2019-03-21 18:15:40 (12.1 MB/s) - '.../.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd.download' saved [675094624/675094624]
>
> Extracting the image...
> .../.cache/qemu-vm/images/openbsd.img.tmp.xz (1/1)
>   100 %        643.8 MiB / 32.0 GiB = 0.020   253 MiB/s       2:09
>     VM-BUILD openbsd
> Cloning into '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot'...
> done.
> Checking out files: 100% (6748/6748), done.
> Submodule 'dtc' (https://git.qemu.org/git/dtc.git) registered for path 'dtc'
> Cloning into '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/dtc'...
> Submodule 'ui/keycodemapdb' (https://git.qemu.org/git/keycodemapdb.git) registered for path 'ui/keycodemapdb'
> Cloning into '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/ui/keycodemapdb'...
> Submodule 'tests/fp/berkeley-softfloat-3' (https://github.com/cota/berkeley-softfloat-3) registered for path 'tests/fp/berkeley-softfloat-3'
> Cloning into '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/tests/fp/berkeley-softfloat-3'...
> Submodule 'tests/fp/berkeley-testfloat-3' (https://github.com/cota/berkeley-testfloat-3) registered for path 'tests/fp/berkeley-testfloat-3'
> Cloning into '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/tests/fp/berkeley-testfloat-3'...
>
> ERROR: "x86_64-unknown-openbsd6.1-gcc-4.9.4" either does not exist or does not work
>
> make: *** [vm-build-openbsd] Error 3
> make: Leaving directory `.../tmp/qemu-build-opt.d'

Thanks,
Laszlo

Le jeu. 21 mars 2019 18:22, Laszlo Ersek <lersek@redhat.com> a écrit :

> On 03/21/19 17:55, Laszlo Ersek wrote:
> > On 03/21/19 12:52, Peter Maydell wrote:
> >> On Thu, 21 Mar 2019 at 11:34, Laszlo Ersek <lersek@redhat.com> wrote:
> >>>
> >>> Repo:   https://github.com/lersek/qemu.git
> >>> Branch: edk2_build_v3
> >>>
> >>> Version 2, that is:
> >>>
> >>>   [Qemu-devel] [PATCH v2 00/12] bundle edk2 platform firmware with QEMU
> >>>
> >>> was posted at:
> >>>
> >>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04670.html
> >>>   http://mid.mail-archive.com/20190313210057.32584-1-lersek@redhat.com
> >>>
> >>> Updates in v3 are noted on each patch individually, in the Notes
> >>> section. In summary,
> >>>
> >>> - I've picked up feedback tags from the v2 thread (see above),
> >>>
> >>> - I've replaced the xz compression with bz2 compression according to
> the
> >>>   subthread
> >>>
> >>>     Re: [Qemu-devel] [PULL 00/12] EDK2 Firmware roms
> >>
> >> Thanks. Could you check that this works in the OpenBSD VM test?
> >> It should just be a matter of
> >>   make -C build vm-build-openbsd
> >
> > Actually I tried to do that before posting v3, following the
> > instructions in "docs/devel/testing.rst":
> >
> >   cd $QEMU_SRC/tests/vm
> >   ./openbsd --build-image --image /mnt/data/tmp/openbsd.img
> >   ./openbsd --debug --image /mnt/data/tmp/openbsd.img \
> >     --build-qemu $QEMU_SRC
>

These isn't the line I remember using with vmtest openbsd. I remember
specifying host-cc and python.
I guess you can find it in openbsd.py under tests/vm/.

>
> > The first "openbsd" invocation succeeded, but the second failed, with a
> > complaint about some (cross?)compiler missing.
> >
> > Anyway I'll now retry with the make command you specify above.
>
> Unfortunately, I got the exact same setup error:
>
> > make: Entering directory `.../tmp/qemu-build-opt.d'
> >     VM-IMAGE openbsd
> > --2019-03-21 18:14:46--
> http://download.patchew.org/openbsd-6.1-amd64.img.xz
> > Resolving download.patchew.org (download.patchew.org)... 140.211.15.4
> > Connecting to download.patchew.org (download.patchew.org)|140.211.15.4|:80...
> connected.
> > HTTP request sent, awaiting response... 200 OK
> > Length: 675094624 (644M) [application/octet-stream]
> > Saving to:
> '.../.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd.download'
> >
> >
> 100%[=====================================================================================================================================================================================================>]
> 675,094,624 12.0MB/s   in 53s
> >
> > 2019-03-21 18:15:40 (12.1 MB/s) -
> '.../.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd.download'
> saved [675094624/675094624]
> >
> > Extracting the image...
> > .../.cache/qemu-vm/images/openbsd.img.tmp.xz (1/1)
> >   100 %        643.8 MiB / 32.0 GiB = 0.020   253 MiB/s       2:09
> >     VM-BUILD openbsd
> > Cloning into
> '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot'...
> > done.
> > Checking out files: 100% (6748/6748), done.
> > Submodule 'dtc' (https://git.qemu.org/git/dtc.git) registered for path
> 'dtc'
> > Cloning into
> '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/dtc'...
> > Submodule 'ui/keycodemapdb' (https://git.qemu.org/git/keycodemapdb.git)
> registered for path 'ui/keycodemapdb'
> > Cloning into
> '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/ui/keycodemapdb'...
> > Submodule 'tests/fp/berkeley-softfloat-3' (
> https://github.com/cota/berkeley-softfloat-3) registered for path
> 'tests/fp/berkeley-softfloat-3'
> > Cloning into
> '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/tests/fp/berkeley-softfloat-3'...
> > Submodule 'tests/fp/berkeley-testfloat-3' (
> https://github.com/cota/berkeley-testfloat-3) registered for path
> 'tests/fp/berkeley-testfloat-3'
> > Cloning into
> '.../tmp/qemu-build-opt.d/vm-test-N8zknt.tmp/data-35ed0.tar.vroot/tests/fp/berkeley-testfloat-3'...
> >
> > ERROR: "x86_64-unknown-openbsd6.1-gcc-4.9.4" either does not exist or
> does not work
> >
> > make: *** [vm-build-openbsd] Error 3
> > make: Leaving directory `.../tmp/qemu-build-opt.d'
>

To reach shell even on failure you have to use smth like:

$ VERBOSE=1 make
Or DEBUG=1. If default ./configure && make fails you get shell and can run
again with nodefault options. You can check basevm.py in tests/.


> Thanks,
> Laszlo
>

On 03/21/19 21:58, Philippe Mathieu-Daudé wrote:
> Le jeu. 21 mars 2019 18:22, Laszlo Ersek <lersek@redhat.com> a écrit :

>> Unfortunately, I got the exact same setup error:

>>> ERROR: "x86_64-unknown-openbsd6.1-gcc-4.9.4" either does not exist
>>> or does not work

> $ VERBOSE=1 make
> Or DEBUG=1. If default ./configure && make fails you get shell and can
> run again with nodefault options. You can check basevm.py in tests/.

I used the "interactive shell" commands from "docs/devel/testing.rst",
to look around inside the image.

The preinstalled gcc is this one:

> qemu-openbsd# gcc --version
> gcc (GCC) 4.2.1 20070719

not what the "openbsd" test case refers to.

Let's see what I could install myself:

> qemu-openbsd# cat /etc/installurl
> https://fastly.cdn.openbsd.org/pub/OpenBSD

> qemu-openbsd# pkg_info -Q gcc
> [...]
> gcc-4.9.4p12
> [...]

> qemu-openbsd# pkg_add gcc-4.9.4p12

And then this gives me "x86_64-unknown-openbsd6.4-gcc-4.9.4". Again,
*not* what the "openbsd" script refers to.

Then I look at the "openbsd" script itself, and realize it downloads:

>         cimg = self._download_with_cache("http://download.patchew.org/openbsd-6.1-amd64.img.xz",
>                 sha256sum='8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf')

Hmmm. Can I perhaps *browse* the download directory on patchew, in
Firefox? I sure can, directory listing is enabled:

> README                                             25-Aug-2017 12:14                   0
> freebsd-11.1-amd64.img.xz                          03-Aug-2017 09:21           545422116
> netbsd-7.1-amd64.img.xz                            03-Aug-2017 09:32           224195948
> openbsd-6.1-amd64.img.xz                           27-Feb-2019 11:48           675094624
> openbsd-6.1-amd64.img.xz.old                       16-Aug-2017 03:23           622579972
> patchew-db-small.tar.xz                            15-Mar-2018 10:17            10451488

There it is. "openbsd-6.1-amd64.img.xz.old" has an mtime of "16-Aug-2017
03:23", which slightly precedes commit fdfaa33291eb ("tests: Add OpenBSD
image", 2017-09-22).

But someone must have added that ".old" suffix to the filename
*recently*, and uploaded "openbsd-6.1-amd64.img.xz" in its place -- see
the mtime on "openbsd-6.1-amd64.img.xz": "27-Feb-2019 11:48".

I guess the "openbsd" script should have been updated too, in sync. :/

Bonus question: why doesn't the download abort with an sha256sum
mismatch? Here's what I get on my end, after manually fetching both
images:

> 0a836e5013fd502c2c1e14f3f29b25bb1f1ee946abb27d3ca3596d678a728476  openbsd-6.1-amd64.img.xz
> 8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf  openbsd-6.1-amd64.img.xz.old

The "old" image matches the checksum (added in original commit
fdfaa33291eb, see above), but the new one doesn't.

... Aaargh, "basevm.py" only uses the checksum for local caching (i.e.
for identifying stale vs. up-to-date local copies), not for
verification!

Cool, so let me try this. I'm going to download the xz.old file
manually. Rename it to just xz. It will then match the built-in
checksum, and will be used as a cached copy. Then I will try building my
series in *that* ("old") VM.

Thanks
Laszlo

On 03/21/19 23:32, Laszlo Ersek wrote:

> Cool, so let me try this. I'm going to download the xz.old file
> manually. Rename it to just xz. It will then match the built-in
> checksum, and will be used as a cached copy. Then I will try building my
> series in *that* ("old") VM.

Summary:

(1) The image file at
<http://download.patchew.org/openbsd-6.1-amd64.img.xz> has been recently
uploaded ("Last-Modified: Wed, 27 Feb 2019 11:48:18 GMT") by someone
unknown to me, and its sha256sum doesn't match the sha256sum in the
"tests/vm/openbsd" test script.

This is why my earlier attempts at the OpenBSD build test have failed.
And in fact I don't understand how it could work for anyone else -- the
compiler that the "tests/vm/openbsd" script specifies is neither
installed, nor available with "pkg_add", in this image.


(2) Against the "old" image
<http://download.patchew.org/openbsd-6.1-amd64.img.xz.old>, which indeed
has the expected
sha256sum=8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf,
the build test *does* succeed.

(

In order to make use of the old image, it has to be downloaded manually,
then moved/renamed to:

  $HOME/.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd

because the last filename component must be the sha1sum of the URL
itself, for the caching mechanism to recognize the compressed image:

> $ echo -n 'http://download.patchew.org/openbsd-6.1-amd64.img.xz' \
>   | sha1sum
> bc4733f6c6e76931702528a515a1bf70eb8baecd  -

)

I'm attaching the log of the successful OpenBSD build test, which I
captured with "screen" (see the "BUNZIP2" lines in it, in particular).

Thanks,
Laszlo

Le ven. 22 mars 2019 00:33, Laszlo Ersek <lersek@redhat.com> a écrit :

> On 03/21/19 23:32, Laszlo Ersek wrote:
>
> > Cool, so let me try this. I'm going to download the xz.old file
> > manually. Rename it to just xz. It will then match the built-in
> > checksum, and will be used as a cached copy. Then I will try building my
> > series in *that* ("old") VM.
>
> Summary:
>
> (1) The image file at
> <http://download.patchew.org/openbsd-6.1-amd64.img.xz> has been recently
> uploaded ("Last-Modified: Wed, 27 Feb 2019 11:48:18 GMT") by someone
> unknown to me, and its sha256sum doesn't match the sha256sum in the
> "tests/vm/openbsd" test script.
>
> This is why my earlier attempts at the OpenBSD build test have failed.
>

Can someone include Fam/Paolo/Brad in this thread please? (I don't have
their emails in my cellphone). Thanks.

And in fact I don't understand how it could work for anyone else -- the
> compiler that the "tests/vm/openbsd" script specifies is neither
> installed, nor available with "pkg_add", in this image.
>
>
> (2) Against the "old" image
> <http://download.patchew.org/openbsd-6.1-amd64.img.xz.old>, which indeed
> has the expected
> sha256sum=8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf,
> the build test *does* succeed.
>
> (
>
> In order to make use of the old image, it has to be downloaded manually,
> then moved/renamed to:
>
>   $HOME/.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd
>
> because the last filename component must be the sha1sum of the URL
> itself, for the caching mechanism to recognize the compressed image:
>
> > $ echo -n 'http://download.patchew.org/openbsd-6.1-amd64.img.xz' \
> >   | sha1sum
> > bc4733f6c6e76931702528a515a1bf70eb8baecd  -
>
> )
>
> I'm attaching the log of the successful OpenBSD build test, which I
> captured with "screen" (see the "BUNZIP2" lines in it, in particular).
>
> Thanks,
> Laszlo
>

On 03/22/19 08:02, Philippe Mathieu-Daudé wrote:
> Le ven. 22 mars 2019 00:33, Laszlo Ersek <lersek@redhat.com> a écrit :
> 
>> On 03/21/19 23:32, Laszlo Ersek wrote:
>>
>>> Cool, so let me try this. I'm going to download the xz.old file
>>> manually. Rename it to just xz. It will then match the built-in
>>> checksum, and will be used as a cached copy. Then I will try building my
>>> series in *that* ("old") VM.
>>
>> Summary:
>>
>> (1) The image file at
>> <http://download.patchew.org/openbsd-6.1-amd64.img.xz> has been recently
>> uploaded ("Last-Modified: Wed, 27 Feb 2019 11:48:18 GMT") by someone
>> unknown to me, and its sha256sum doesn't match the sha256sum in the
>> "tests/vm/openbsd" test script.
>>
>> This is why my earlier attempts at the OpenBSD build test have failed.
>>
> 
> Can someone include Fam/Paolo/Brad in this thread please? (I don't have
> their emails in my cellphone). Thanks.

Done.

Thanks
Laszlo

>> And in fact I don't understand how it could work for anyone else -- the
>> compiler that the "tests/vm/openbsd" script specifies is neither
>> installed, nor available with "pkg_add", in this image.
>>
>>
>> (2) Against the "old" image
>> <http://download.patchew.org/openbsd-6.1-amd64.img.xz.old>, which indeed
>> has the expected
>> sha256sum=8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf,
>> the build test *does* succeed.
>>
>> (
>>
>> In order to make use of the old image, it has to be downloaded manually,
>> then moved/renamed to:
>>
>>   $HOME/.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd
>>
>> because the last filename component must be the sha1sum of the URL
>> itself, for the caching mechanism to recognize the compressed image:
>>
>>> $ echo -n 'http://download.patchew.org/openbsd-6.1-amd64.img.xz' \
>>>   | sha1sum
>>> bc4733f6c6e76931702528a515a1bf70eb8baecd  -
>>
>> )
>>
>> I'm attaching the log of the successful OpenBSD build test, which I
>> captured with "screen" (see the "BUNZIP2" lines in it, in particular).
>>
>> Thanks,
>> Laszlo
>>
> 

Hi Peter,

can you please comment:

On 03/22/19 10:17, Laszlo Ersek wrote:
> On 03/22/19 08:02, Philippe Mathieu-Daudé wrote:
>> Le ven. 22 mars 2019 00:33, Laszlo Ersek <lersek@redhat.com> a écrit :
>>
>>> On 03/21/19 23:32, Laszlo Ersek wrote:
>>>
>>>> Cool, so let me try this. I'm going to download the xz.old file
>>>> manually. Rename it to just xz. It will then match the built-in
>>>> checksum, and will be used as a cached copy. Then I will try building my
>>>> series in *that* ("old") VM.
>>>
>>> Summary:
>>>
>>> (1) The image file at
>>> <http://download.patchew.org/openbsd-6.1-amd64.img.xz> has been recently
>>> uploaded ("Last-Modified: Wed, 27 Feb 2019 11:48:18 GMT") by someone
>>> unknown to me, and its sha256sum doesn't match the sha256sum in the
>>> "tests/vm/openbsd" test script.
>>>
>>> This is why my earlier attempts at the OpenBSD build test have failed.
>>>
>>
>> Can someone include Fam/Paolo/Brad in this thread please? (I don't have
>> their emails in my cellphone). Thanks.
> 
> Done.

- do we have any idea what happened on download.patchew.org (i.e. why
the image matching the script was replaced with an image not matching
the script)?

>>> And in fact I don't understand how it could work for anyone else -- the
>>> compiler that the "tests/vm/openbsd" script specifies is neither
>>> installed, nor available with "pkg_add", in this image.
>>>
>>>
>>> (2) Against the "old" image
>>> <http://download.patchew.org/openbsd-6.1-amd64.img.xz.old>, which indeed
>>> has the expected
>>> sha256sum=8c6cedc483e602cfee5e04f0406c64eb99138495e8ca580bc0293bcf0640c1bf,
>>> the build test *does* succeed.
>>>
>>> (
>>>
>>> In order to make use of the old image, it has to be downloaded manually,
>>> then moved/renamed to:
>>>
>>>   $HOME/.cache/qemu-vm/download/bc4733f6c6e76931702528a515a1bf70eb8baecd
>>>
>>> because the last filename component must be the sha1sum of the URL
>>> itself, for the caching mechanism to recognize the compressed image:
>>>
>>>> $ echo -n 'http://download.patchew.org/openbsd-6.1-amd64.img.xz' \
>>>>   | sha1sum
>>>> bc4733f6c6e76931702528a515a1bf70eb8baecd  -
>>>
>>> )
>>>
>>> I'm attaching the log of the successful OpenBSD build test, which I
>>> captured with "screen" (see the "BUNZIP2" lines in it, in particular).

- are you satisfied with the OpenBSD build testing I did (using the
"old" image, matching the script)?

Thanks
Laszlo

On 25/03/19 11:40, Laszlo Ersek wrote:
>>>>
>>>> (1) The image file at
>>>> <http://download.patchew.org/openbsd-6.1-amd64.img.xz> has been recently
>>>> uploaded ("Last-Modified: Wed, 27 Feb 2019 11:48:18 GMT") by someone
>>>> unknown to me, and its sha256sum doesn't match the sha256sum in the
>>>> "tests/vm/openbsd" test script.
>>>>
>>>> This is why my earlier attempts at the OpenBSD build test have failed.
>>>>
>>> Can someone include Fam/Paolo/Brad in this thread please? (I don't have
>>> their emails in my cellphone). Thanks.
>> Done.
> - do we have any idea what happened on download.patchew.org (i.e. why
> the image matching the script was replaced with an image not matching
> the script)?

The update was requested by Daniel in this thread:
https://patchew.org/QEMU/20181031025712.18602-1-famz@redhat.com/

Fam did the update, but forgot to update the sha256sum.

Paolo

(Keeping Fam on the address list, adding Dan & Markus)

On 03/27/19 14:18, Paolo Bonzini wrote:
> On 25/03/19 11:40, Laszlo Ersek wrote:
>>>>>
>>>>> (1) The image file at
>>>>> <http://download.patchew.org/openbsd-6.1-amd64.img.xz> has been
>>>>> recently uploaded ("Last-Modified: Wed, 27 Feb 2019 11:48:18 GMT")
>>>>> by someone unknown to me, and its sha256sum doesn't match the
>>>>> sha256sum in the "tests/vm/openbsd" test script.
>>>>>
>>>>> This is why my earlier attempts at the OpenBSD build test have
>>>>> failed.
>>>>>
>>>> Can someone include Fam/Paolo/Brad in this thread please? (I don't
>>>> have their emails in my cellphone). Thanks.
>>> Done.
>> - do we have any idea what happened on download.patchew.org (i.e. why
>> the image matching the script was replaced with an image not matching
>> the script)?
>
> The update was requested by Daniel in this thread:
> https://patchew.org/QEMU/20181031025712.18602-1-famz@redhat.com/
>
> Fam did the update, but forgot to update the sha256sum.

Thank you for tracking this down!

But, it's not just the sha256sum that needs an update in
"tests/vm/openbsd". In addition to that, the script refers to the
compiler under the name

  x86_64-unknown-openbsd6.1-gcc-4.9.4

This compiler is not available in the updated OpenBSD disk image, either
pre-installed, or from the network. Like I described here:

  https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg06277.html

the preinstalled compiler ("gcc") is version 4.2.1, and the one that's
available from the network, with "pkg_add", is called differently:

  x86_64-unknown-openbsd6.4-gcc-4.9.4

So, my take is that *both* the script is out-of-sync with the new
OpenBSD image, *and* the OpenBSD image is unusable (for this particular
test) and has never been verified.

... Which in turn raises the question: *before* Peter reported the "xz"
failure first, against my series, at

  https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg05452.html

what image had Peter been using? Because, as far as I see it, at
that point there was *no* way for the OpenBSD build test to succeed,
*regardless* of my series. The compiler must not have been available.

In fact, what OpenBSD image is Peter using *right now*, for the
build-QEMU-on-OpenBSD test? Again, I see no way how that could
succeed right now.

... Honestly, given that the OpenBSD image was broken & out-of-sync to
begin with, I feel it was questionable to block my series (= Phil's
PULL) from 4.0, just because the series placed an *additional*
requirement (namely "xz") on the OpenBSD image, which had *already* been
in sore need of a functional C compiler. :/

---*---

OK, let me be constructive here. Right now, the *current*
"tests/vm/openbsd" script is unusable for *any* testing. (My testing did
succeed agains the previous image, which is still available under a
different name.) Who's going to fix the current image, and who's going
to fix the script? And when?

Regarding my own series, after all this churn, I'll stick with using bz2
in it; the xz->bz2 storage "loss" is minimal, about 1%.

Thanks,
Laszlo

On 27/03/19 15:31, Laszlo Ersek wrote:
> OK, let me be constructive here. Right now, the *current*
> "tests/vm/openbsd" script is unusable for *any* testing. (My testing did
> succeed agains the previous image, which is still available under a
> different name.) Who's going to fix the current image, and who's going
> to fix the script? And when?

For now I'll just revert the image to the old one.

Paolo