xen/arch/x86/bzimage.c | 4 +- xen/arch/x86/hvm/dom0_build.c | 113 ++++++++++++++++++++++++++++- xen/arch/x86/include/asm/bzimage.h | 3 +- xen/arch/x86/pv/dom0_build.c | 2 +- xen/include/public/features.h | 9 ++- 5 files changed, 124 insertions(+), 7 deletions(-)
Xen tries to load a PVH dom0 kernel at the fixed guest physical address from the elf headers. For Linux, this defaults to 0x1000000 (16MB), but it can be configured. Unfortunately there exist firmwares that have reserved regions at this address, so Xen fails to load the dom0 kernel since it's not RAM. The other issue is that the Linux PVH entry point is not position-independent. It expects to run at the compiled CONFIG_PHYSICAL_ADDRESS. This patch set expands the PVH dom0 builder to try to relocate the kernel if needed and possible. XENFEAT_pvh_relocatable is added for kernels to indicate they are relocatable. However, we may want to switch to an additional ELF note with the kernel alignment. Linux specifies a kernel alignment in the bzImage boot_params.setup_header, but that is not present the ELF vmlinux file. The first patch fixes some whitespace in features.h The second patch enhances bzimage_parse to pull the kernel_alignment into an optional align pointer. The third patch expands the pvh dom0 kernel placement code. I'll post an additional patch showing the Linux changes to make PVH relocatable. Jason Andryuk (3): features.h: Replace hard tabs xen/x86: bzImage parse kernel_alignment x86/PVH: Support relocatable dom0 kernels xen/arch/x86/bzimage.c | 4 +- xen/arch/x86/hvm/dom0_build.c | 113 ++++++++++++++++++++++++++++- xen/arch/x86/include/asm/bzimage.h | 3 +- xen/arch/x86/pv/dom0_build.c | 2 +- xen/include/public/features.h | 9 ++- 5 files changed, 124 insertions(+), 7 deletions(-) -- 2.44.0
On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote: > Xen tries to load a PVH dom0 kernel at the fixed guest physical address > from the elf headers. For Linux, this defaults to 0x1000000 (16MB), but > it can be configured. > > Unfortunately there exist firmwares that have reserved regions at this > address, so Xen fails to load the dom0 kernel since it's not RAM. > > The other issue is that the Linux PVH entry point is not > position-independent. It expects to run at the compiled > CONFIG_PHYSICAL_ADDRESS. > > This patch set expands the PVH dom0 builder to try to relocate the > kernel if needed and possible. XENFEAT_pvh_relocatable is added for > kernels to indicate they are relocatable. However, we may want to > switch to an additional ELF note with the kernel alignment. Linux > specifies a kernel alignment in the bzImage boot_params.setup_header, > but that is not present the ELF vmlinux file. I wonder whether we need a pair of notes, to signal the min/max addresses the kernel supports being relocated to. Thanks, Roger.
On 07.03.2024 11:00, Roger Pau Monné wrote: > On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote: >> Xen tries to load a PVH dom0 kernel at the fixed guest physical address >> from the elf headers. For Linux, this defaults to 0x1000000 (16MB), but >> it can be configured. >> >> Unfortunately there exist firmwares that have reserved regions at this >> address, so Xen fails to load the dom0 kernel since it's not RAM. >> >> The other issue is that the Linux PVH entry point is not >> position-independent. It expects to run at the compiled >> CONFIG_PHYSICAL_ADDRESS. >> >> This patch set expands the PVH dom0 builder to try to relocate the >> kernel if needed and possible. XENFEAT_pvh_relocatable is added for >> kernels to indicate they are relocatable. However, we may want to >> switch to an additional ELF note with the kernel alignment. Linux >> specifies a kernel alignment in the bzImage boot_params.setup_header, >> but that is not present the ELF vmlinux file. > > I wonder whether we need a pair of notes, to signal the min/max > addresses the kernel supports being relocated to. Plus, as per your other reply, a 3rd one to specify alignment needs. Then again - a single note can have multiple values. So perhaps it doesn't need to be more than one new notes (except if dealing with multi-value ones is deemed too complicated). Jan
On Thu, Mar 07, 2024 at 11:08:37AM +0100, Jan Beulich wrote: > On 07.03.2024 11:00, Roger Pau Monné wrote: > > On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote: > >> Xen tries to load a PVH dom0 kernel at the fixed guest physical address > >> from the elf headers. For Linux, this defaults to 0x1000000 (16MB), but > >> it can be configured. > >> > >> Unfortunately there exist firmwares that have reserved regions at this > >> address, so Xen fails to load the dom0 kernel since it's not RAM. > >> > >> The other issue is that the Linux PVH entry point is not > >> position-independent. It expects to run at the compiled > >> CONFIG_PHYSICAL_ADDRESS. > >> > >> This patch set expands the PVH dom0 builder to try to relocate the > >> kernel if needed and possible. XENFEAT_pvh_relocatable is added for > >> kernels to indicate they are relocatable. However, we may want to > >> switch to an additional ELF note with the kernel alignment. Linux > >> specifies a kernel alignment in the bzImage boot_params.setup_header, > >> but that is not present the ELF vmlinux file. > > > > I wonder whether we need a pair of notes, to signal the min/max > > addresses the kernel supports being relocated to. > > Plus, as per your other reply, a 3rd one to specify alignment needs. Alignment we could in theory get from the ELF program header, if OSes fill those reliably. FreeBSD seems to do so, haven't checked Linux. > Then again - a single note can have multiple values. So perhaps it > doesn't need to be more than one new notes (except if dealing with > multi-value ones is deemed too complicated). I've never dealt with a multi-value note, if that's not overly complicated I would be fine with it, otherwise multiple notes are OK IMO. Thanks, Roger.
On 2024-03-07 05:20, Roger Pau Monné wrote: > On Thu, Mar 07, 2024 at 11:08:37AM +0100, Jan Beulich wrote: >> On 07.03.2024 11:00, Roger Pau Monné wrote: >>> On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote: >>>> Xen tries to load a PVH dom0 kernel at the fixed guest physical address >>>> from the elf headers. For Linux, this defaults to 0x1000000 (16MB), but >>>> it can be configured. >>>> >>>> Unfortunately there exist firmwares that have reserved regions at this >>>> address, so Xen fails to load the dom0 kernel since it's not RAM. >>>> >>>> The other issue is that the Linux PVH entry point is not >>>> position-independent. It expects to run at the compiled >>>> CONFIG_PHYSICAL_ADDRESS. >>>> >>>> This patch set expands the PVH dom0 builder to try to relocate the >>>> kernel if needed and possible. XENFEAT_pvh_relocatable is added for >>>> kernels to indicate they are relocatable. However, we may want to >>>> switch to an additional ELF note with the kernel alignment. Linux >>>> specifies a kernel alignment in the bzImage boot_params.setup_header, >>>> but that is not present the ELF vmlinux file. >>> >>> I wonder whether we need a pair of notes, to signal the min/max >>> addresses the kernel supports being relocated to. >> >> Plus, as per your other reply, a 3rd one to specify alignment needs. > > Alignment we could in theory get from the ELF program header, if OSes > fill those reliably. FreeBSD seems to do so, haven't checked Linux. I will look into this more, but at first glance, I don't see a connection between Linux's CONFIG_PHYSICAL_ALIGN and the values in the elf headers. As a quick test, I set CONFIG_PHYSICAL_ALIGN=0x800000, but the elf align values are still 0x200000. The elf header values may be a suitable fallback though? >> Then again - a single note can have multiple values. So perhaps it >> doesn't need to be more than one new notes (except if dealing with >> multi-value ones is deemed too complicated). > > I've never dealt with a multi-value note, if that's not overly > complicated I would be fine with it, otherwise multiple notes are OK > IMO. It looks like a single note can be followed by arbitrary data, so putting three values in should be fine. PVH is defined with a 32bit entry point, so are 32bit min, max & align values acceptable, or should 64bit be specified? Linux uses _ASM_PTR for PHYS32_ENTRY so its size matches the bitness of the target (32 on 32/ 64 on 64). I guess I'll follow that unless I hear a preference otherwise. Regards, Jason
On 07.03.2024 18:33, Jason Andryuk wrote: > On 2024-03-07 05:20, Roger Pau Monné wrote: >> On Thu, Mar 07, 2024 at 11:08:37AM +0100, Jan Beulich wrote: >>> On 07.03.2024 11:00, Roger Pau Monné wrote: >>>> On Wed, Mar 06, 2024 at 01:50:29PM -0500, Jason Andryuk wrote: >>>>> Xen tries to load a PVH dom0 kernel at the fixed guest physical address >>>>> from the elf headers. For Linux, this defaults to 0x1000000 (16MB), but >>>>> it can be configured. >>>>> >>>>> Unfortunately there exist firmwares that have reserved regions at this >>>>> address, so Xen fails to load the dom0 kernel since it's not RAM. >>>>> >>>>> The other issue is that the Linux PVH entry point is not >>>>> position-independent. It expects to run at the compiled >>>>> CONFIG_PHYSICAL_ADDRESS. >>>>> >>>>> This patch set expands the PVH dom0 builder to try to relocate the >>>>> kernel if needed and possible. XENFEAT_pvh_relocatable is added for >>>>> kernels to indicate they are relocatable. However, we may want to >>>>> switch to an additional ELF note with the kernel alignment. Linux >>>>> specifies a kernel alignment in the bzImage boot_params.setup_header, >>>>> but that is not present the ELF vmlinux file. >>>> >>>> I wonder whether we need a pair of notes, to signal the min/max >>>> addresses the kernel supports being relocated to. >>> >>> Plus, as per your other reply, a 3rd one to specify alignment needs. >> >> Alignment we could in theory get from the ELF program header, if OSes >> fill those reliably. FreeBSD seems to do so, haven't checked Linux. > > I will look into this more, but at first glance, I don't see a > connection between Linux's CONFIG_PHYSICAL_ALIGN and the values in the > elf headers. As a quick test, I set CONFIG_PHYSICAL_ALIGN=0x800000, but > the elf align values are still 0x200000. > > The elf header values may be a suitable fallback though? Imo, given the above, explicit values should be required. Better not load a kernel than doing so and then getting hard to debug crashes. Jan
© 2016 - 2024 Red Hat, Inc.