I've attempted to get Xen to boot Arch Linux as a unified EFI binary.
Using https://xenbits.xen.org/docs/unstable/misc/efi.html as my source
of information, I've been able to build a unified binary. When trying to
boot the kernel Xen complains that the stream is corrupt
("ZSTD-compressed data is corrupt"). I've been able to reproduce the
issue locally in user-mode, and confirmed that the issue is also present
in the latest ZSTD version.

Using streaming decompression the kernel gets unpacked properly and the
output is the same as if doing `cat kernel.zst | unzstd > bzImage`.

A problem I ran into was that adding book keeping to decompress.c would
result in either a .data section being added or a .bss.* section. The
linker would complain about this. And since I am not familiar with this
code, and why it is this way, I opted to add a user-pointer to the
internal decompression API.

Rafaël Kooi (2):
  xen/decompress: Add a user pointer for book keeping in the callbacks
  x86/Dom0: Use streaming decompression for ZSTD compressed kernels

 xen/common/bunzip2.c         | 23 ++++++++++++----------
 xen/common/decompress.c      | 37 ++++++++++++++++++++++++++++++------
 xen/common/unlz4.c           | 15 ++++++++-------
 xen/common/unlzma.c          | 30 +++++++++++++++++------------
 xen/common/unlzo.c           | 13 +++++++------
 xen/common/unxz.c            | 11 ++++++-----
 xen/common/unzstd.c          | 13 +++++++------
 xen/include/xen/decompress.h | 10 +++++++---
 8 files changed, 97 insertions(+), 55 deletions(-)

--
2.40.0

On 10.05.2023 02:18, Rafaël Kooi wrote:
> A problem I ran into was that adding book keeping to decompress.c would
> result in either a .data section being added or a .bss.* section. The
> linker would complain about this. And since I am not familiar with this
> code, and why it is this way, I opted to add a user-pointer to the
> internal decompression API.

At least gunzip uses global variables to track state as well. Without you
being explicit, I guess you mean "Error: size of <file>:<section> is <size>",
which is emitted by our build system, not the linker? That's easy to overcome
without touching all decompressors - use __initdata. See common/gunzip.c.

Jan

On 10/05/2023 02:18, Rafaël Kooi wrote:> I've attempted to get Xen to boot Arch Linux as a unified EFI binary.
> Using https://xenbits.xen.org/docs/unstable/misc/efi.html as my source
> of information, I've been able to build a unified binary. When trying to
> boot the kernel Xen complains that the stream is corrupt
> ("ZSTD-compressed data is corrupt"). I've been able to reproduce the
> issue locally in user-mode, and confirmed that the issue is also present
> in the latest ZSTD version.
> 
> Using streaming decompression the kernel gets unpacked properly and the
> output is the same as if doing `cat kernel.zst | unzstd > bzImage`.
> 
> A problem I ran into was that adding book keeping to decompress.c would
> result in either a .data section being added or a .bss.* section. The
> linker would complain about this. And since I am not familiar with this
> code, and why it is this way, I opted to add a user-pointer to the
> internal decompression API.
> 
> Rafaël Kooi (2):
>    xen/decompress: Add a user pointer for book keeping in the callbacks
>    x86/Dom0: Use streaming decompression for ZSTD compressed kernels
> 
>   xen/common/bunzip2.c         | 23 ++++++++++++----------
>   xen/common/decompress.c      | 37 ++++++++++++++++++++++++++++++------
>   xen/common/unlz4.c           | 15 ++++++++-------
>   xen/common/unlzma.c          | 30 +++++++++++++++++------------
>   xen/common/unlzo.c           | 13 +++++++------
>   xen/common/unxz.c            | 11 ++++++-----
>   xen/common/unzstd.c          | 13 +++++++------
>   xen/include/xen/decompress.h | 10 +++++++---
>   8 files changed, 97 insertions(+), 55 deletions(-)
> 
> --
> 2.40.0
> 

This patch can be dropped in its entirety. The issue that this code
fixes is a symptom of another issue entirely. What ended up being the
issue was that the SSD of my laptop is dead, and that I messed up the
alignment of the sections inserted into xen.efi.

I plan to add some sanity checks to the EFI boot loader code to inform
the user if the sections are misaligned. That's a much more user friendly
error than whatever the decompressor will report.

To Jan, sorry for wasting your time, and thanks again for being patient
with me.

Rafaël