On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> 64-bit pci io window, failing during boot with errors like:
Well. Why people would use *that* ubuntu version is not clear to me.
It's *loooooong* out of support. Even the LTS version from that year
(16.04) is not supported any more. But it is at least available for
download still, so I gave it a spin.
Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
44 phys-bits). Test patch below.
take care,
Gerd
diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
index bb44dc296047..a43876a931c9 100644
--- a/src/fw/pciinit.c
+++ b/src/fw/pciinit.c
@@ -1189,11 +1189,16 @@ pci_setup(void)
if (CPUPhysBits) {
pci_mem64_top = 1LL << CPUPhysBits;
- if (CPUPhysBits > 46) {
- // Old linux kernels have trouble dealing with more than 46
- // phys-bits, so avoid that for now. Seems to be a bug in the
- // virtio-pci driver. Reported: centos-7, ubuntu-18.04
- pci_mem64_top = 1LL << 46;
+ if (CPUPhysBits > 44) {
+ // Old linux kernels have trouble dealing with more than 44/46
+ // phys-bits. Seems to be a bug in the virtio-pci driver.
+ // 46: centos-7, ubuntu-18.04
+ // 44: ubuntu-16.04
+ // Limit the used address space to mitigate the bug, except we are
+ // running in a guest with more than 1TB of memory installed.
+ if (RamSizeOver4G < (1LL << 40)) {
+ pci_mem64_top = 1LL << 44;
+ }
}
}
_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org
Hi, Am 21.06.24 um 14:05 schrieb Gerd Hoffmann: > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: >> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the >> 64-bit pci io window, failing during boot with errors like: > > Well. Why people would use *that* ubuntu version is not clear to me. > It's *loooooong* out of support. Even the LTS version from that year > (16.04) is not supported any more. But it is at least available for > download still, so I gave it a spin. > > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka > 44 phys-bits). Test patch below. > > take care, > Gerd > > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c > index bb44dc296047..a43876a931c9 100644 > --- a/src/fw/pciinit.c > +++ b/src/fw/pciinit.c > @@ -1189,11 +1189,16 @@ pci_setup(void) > > if (CPUPhysBits) { > pci_mem64_top = 1LL << CPUPhysBits; > - if (CPUPhysBits > 46) { > - // Old linux kernels have trouble dealing with more than 46 > - // phys-bits, so avoid that for now. Seems to be a bug in the > - // virtio-pci driver. Reported: centos-7, ubuntu-18.04 > - pci_mem64_top = 1LL << 46; > + if (CPUPhysBits > 44) { > + // Old linux kernels have trouble dealing with more than 44/46 > + // phys-bits. Seems to be a bug in the virtio-pci driver. > + // 46: centos-7, ubuntu-18.04 > + // 44: ubuntu-16.04 > + // Limit the used address space to mitigate the bug, except we are > + // running in a guest with more than 1TB of memory installed. > + if (RamSizeOver4G < (1LL << 40)) { > + pci_mem64_top = 1LL << 44; > + } > } > } > we've also had two reports about issues with 32-bit guests now[0][1] (both had '-m 4096' in the QEMU commandline), which I was able to reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The QEMU commandline is below[2]. Unfortunately, it still fails to boot, even with the "limit address space used for pci devices, part two" patch applied on top of rel-1.16.3 (and using current QEMU master). It boots fine with '-m 3583', but not anymore with '-m 3584'. So is bumping the limit for the check necessary after all? Best Regards, Fiona [0]: https://forum.proxmox.com/threads/150217/ [1]: https://forum.proxmox.com/threads/149772/post-683562 [2]: > ./qemu-system-x86_64 \ > -accel 'kvm' \ > -cpu 'host' \ > -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \ > -mon 'chardev=qmp,mode=control' \ > -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \ > -mon 'chardev=qmp-event,mode=control' \ > -pidfile /var/run/qemu-server/121.pid \ > -smp '4,sockets=1,cores=4,maxcpus=4' \ > -nodefaults \ > -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \ > -m 4096 \ > -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \ > -device 'VGA,id=vga,bus=pci.0,addr=0x2' \ > -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \ > -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \ > -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \ > -machine 'type=pc' _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
Hi, > we've also had two reports about issues with 32-bit guests now[0][1] > (both had '-m 4096' in the QEMU commandline), which I was able to > reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The > QEMU commandline is below[2]. > > Unfortunately, it still fails to boot, even with the "limit address > space used for pci devices, part two" patch applied on top of rel-1.16.3 > (and using current QEMU master). Reproduces here. Seems there is something seriously wrong in that debian kernel. Even when going for 36 phys bits (-cpu host,host-phys-bits-limit=36), which is the physical address space of PAE mode when it was introduced by the pentium pro, the guest fails to boot. That *really* should not happen. I guess what you are seeing here is i386 support in the linux kernel starting to bitrot due to everybody moving to 64-bit ... take care, Gerd _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, 12 Jul 2024 14:24:40 +0200 Fiona Ebner <f.ebner@proxmox.com> wrote: > Hi, > > Am 21.06.24 um 14:05 schrieb Gerd Hoffmann: > > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: > >> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the > >> 64-bit pci io window, failing during boot with errors like: > > > > Well. Why people would use *that* ubuntu version is not clear to me. > > It's *loooooong* out of support. Even the LTS version from that year > > (16.04) is not supported any more. But it is at least available for > > download still, so I gave it a spin. > > > > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka > > 44 phys-bits). Test patch below. > > > > take care, > > Gerd > > > > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c > > index bb44dc296047..a43876a931c9 100644 > > --- a/src/fw/pciinit.c > > +++ b/src/fw/pciinit.c > > @@ -1189,11 +1189,16 @@ pci_setup(void) > > > > if (CPUPhysBits) { > > pci_mem64_top = 1LL << CPUPhysBits; > > - if (CPUPhysBits > 46) { > > - // Old linux kernels have trouble dealing with more than 46 > > - // phys-bits, so avoid that for now. Seems to be a bug in the > > - // virtio-pci driver. Reported: centos-7, ubuntu-18.04 > > - pci_mem64_top = 1LL << 46; > > + if (CPUPhysBits > 44) { > > + // Old linux kernels have trouble dealing with more than 44/46 > > + // phys-bits. Seems to be a bug in the virtio-pci driver. > > + // 46: centos-7, ubuntu-18.04 > > + // 44: ubuntu-16.04 > > + // Limit the used address space to mitigate the bug, except we are > > + // running in a guest with more than 1TB of memory installed. > > + if (RamSizeOver4G < (1LL << 40)) { > > + pci_mem64_top = 1LL << 44; > > + } > > } > > } > > > > we've also had two reports about issues with 32-bit guests now[0][1] > (both had '-m 4096' in the QEMU commandline), which I was able to > reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The > QEMU commandline is below[2]. is it also reproducible with upstream kernel? if yes, it would be better to fix that on guest kernel side, rather than in SeaBIOS which has no idea what guest OS is going to be running after it. > Unfortunately, it still fails to boot, even with the "limit address > space used for pci devices, part two" patch applied on top of rel-1.16.3 > (and using current QEMU master). > > It boots fine with '-m 3583', but not anymore with '-m 3584'. So is > bumping the limit for the check necessary after all? > > Best Regards, > Fiona > > [0]: https://forum.proxmox.com/threads/150217/ > [1]: https://forum.proxmox.com/threads/149772/post-683562 well, with current QEMU master branch (I assume it haven't even got topic patch yet) ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600 following CLI boots just fine an 32-bit XP on Haswell host. Also using 'host' with ancient OS, is basically asking for trouble. If it works for user with old 'XP contemporary' CPU model, user should use that instead or workarounds (aka it's management task to configure CLI in compatible with OS manner). From suspicions config options in that post I see 'viommu' and 'vmgenid', are you sure XP even knows what to do with that, perhaps it triggers BSOD. > [2]: > > > ./qemu-system-x86_64 \ > > -accel 'kvm' \ > > -cpu 'host' \ > > -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \ > > -mon 'chardev=qmp,mode=control' \ > > -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \ > > -mon 'chardev=qmp-event,mode=control' \ > > -pidfile /var/run/qemu-server/121.pid \ > > -smp '4,sockets=1,cores=4,maxcpus=4' \ > > -nodefaults \ > > -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \ > > -m 4096 \ > > -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \ > > -device 'VGA,id=vga,bus=pci.0,addr=0x2' \ > > -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \ > > -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \ > > -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \ > > -machine 'type=pc' can you try to boot with q35 + intel iommu enabled and/or force virtio into legacy mode. > > _______________________________________________ > SeaBIOS mailing list -- seabios@seabios.org > To unsubscribe send an email to seabios-leave@seabios.org > _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
Am 12.07.24 um 15:26 schrieb Igor Mammedov: > On Fri, 12 Jul 2024 14:24:40 +0200 > Fiona Ebner <f.ebner@proxmox.com> wrote: > >> Hi, >> >> Am 21.06.24 um 14:05 schrieb Gerd Hoffmann: >>> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: >>>> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the >>>> 64-bit pci io window, failing during boot with errors like: >>> >>> Well. Why people would use *that* ubuntu version is not clear to me. >>> It's *loooooong* out of support. Even the LTS version from that year >>> (16.04) is not supported any more. But it is at least available for >>> download still, so I gave it a spin. >>> >>> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka >>> 44 phys-bits). Test patch below. >>> >>> take care, >>> Gerd >>> >>> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c >>> index bb44dc296047..a43876a931c9 100644 >>> --- a/src/fw/pciinit.c >>> +++ b/src/fw/pciinit.c >>> @@ -1189,11 +1189,16 @@ pci_setup(void) >>> >>> if (CPUPhysBits) { >>> pci_mem64_top = 1LL << CPUPhysBits; >>> - if (CPUPhysBits > 46) { >>> - // Old linux kernels have trouble dealing with more than 46 >>> - // phys-bits, so avoid that for now. Seems to be a bug in the >>> - // virtio-pci driver. Reported: centos-7, ubuntu-18.04 >>> - pci_mem64_top = 1LL << 46; >>> + if (CPUPhysBits > 44) { >>> + // Old linux kernels have trouble dealing with more than 44/46 >>> + // phys-bits. Seems to be a bug in the virtio-pci driver. >>> + // 46: centos-7, ubuntu-18.04 >>> + // 44: ubuntu-16.04 >>> + // Limit the used address space to mitigate the bug, except we are >>> + // running in a guest with more than 1TB of memory installed. >>> + if (RamSizeOver4G < (1LL << 40)) { >>> + pci_mem64_top = 1LL << 44; >>> + } >>> } >>> } >>> >> >> we've also had two reports about issues with 32-bit guests now[0][1] >> (both had '-m 4096' in the QEMU commandline), which I was able to >> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The >> QEMU commandline is below[2]. > > is it also reproducible with upstream kernel? > if yes, it would be better to fix that on guest kernel side, > rather than in SeaBIOS which has no idea what guest OS is going to be > running after it. > Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE boots fine (but the PAE was the one installed by default). I built a 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't. >> Unfortunately, it still fails to boot, even with the "limit address >> space used for pci devices, part two" patch applied on top of rel-1.16.3 >> (and using current QEMU master). >> >> It boots fine with '-m 3583', but not anymore with '-m 3584'. So is >> bumping the limit for the check necessary after all? >> >> Best Regards, >> Fiona >> >> [0]: https://forum.proxmox.com/threads/150217/ > >> [1]: https://forum.proxmox.com/threads/149772/post-683562 > well, with current QEMU master branch (I assume it haven't even got topic patch yet) > ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600 > following CLI boots just fine an 32-bit XP on Haswell host. > > Also using 'host' with ancient OS, is basically asking for trouble. > If it works for user with old 'XP contemporary' CPU model, user should use that > instead or workarounds (aka it's management task to configure CLI in > compatible with OS manner). > > From suspicions config options in that post I see 'viommu' and 'vmgenid', > are you sure XP even knows what to do with that, perhaps it triggers BSOD. > No idea why the user enabled viommu for XP. vmgenid is added by our management stack by default and I don't remember it having ever caused problems. The user said the VM booted fine after adding lm=off to the CPU options. I'm not super interested in the Windows case to be honest O:) >> [2]: >> >>> ./qemu-system-x86_64 \ >>> -accel 'kvm' \ >>> -cpu 'host' \ >>> -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \ >>> -mon 'chardev=qmp,mode=control' \ >>> -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \ >>> -mon 'chardev=qmp-event,mode=control' \ >>> -pidfile /var/run/qemu-server/121.pid \ >>> -smp '4,sockets=1,cores=4,maxcpus=4' \ >>> -nodefaults \ >>> -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \ >>> -m 4096 \ >>> -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \ >>> -device 'VGA,id=vga,bus=pci.0,addr=0x2' \ >>> -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \ >>> -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \ >>> -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \ >>> -machine 'type=pc' > > can you try to boot with q35 + intel iommu enabled > and/or force virtio into legacy mode. > I tried > -device 'intel-iommu' \ > -machine 'type=q35' and intel_iommu=on in the guest's kernel cmdline, but it still failed with kernels with PAE. Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci line made it work however (also with pc machine) :) Best Regards, Fiona _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Tue, 16 Jul 2024 13:56:08 +0200 Fiona Ebner <f.ebner@proxmox.com> wrote: > Am 12.07.24 um 15:26 schrieb Igor Mammedov: > > On Fri, 12 Jul 2024 14:24:40 +0200 > > Fiona Ebner <f.ebner@proxmox.com> wrote: > > > >> Hi, > >> > >> Am 21.06.24 um 14:05 schrieb Gerd Hoffmann: > >>> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: > >>>> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the > >>>> 64-bit pci io window, failing during boot with errors like: > >>> > >>> Well. Why people would use *that* ubuntu version is not clear to me. > >>> It's *loooooong* out of support. Even the LTS version from that year > >>> (16.04) is not supported any more. But it is at least available for > >>> download still, so I gave it a spin. > >>> > >>> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka > >>> 44 phys-bits). Test patch below. > >>> > >>> take care, > >>> Gerd > >>> > >>> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c > >>> index bb44dc296047..a43876a931c9 100644 > >>> --- a/src/fw/pciinit.c > >>> +++ b/src/fw/pciinit.c > >>> @@ -1189,11 +1189,16 @@ pci_setup(void) > >>> > >>> if (CPUPhysBits) { > >>> pci_mem64_top = 1LL << CPUPhysBits; > >>> - if (CPUPhysBits > 46) { > >>> - // Old linux kernels have trouble dealing with more than 46 > >>> - // phys-bits, so avoid that for now. Seems to be a bug in the > >>> - // virtio-pci driver. Reported: centos-7, ubuntu-18.04 > >>> - pci_mem64_top = 1LL << 46; > >>> + if (CPUPhysBits > 44) { > >>> + // Old linux kernels have trouble dealing with more than 44/46 > >>> + // phys-bits. Seems to be a bug in the virtio-pci driver. > >>> + // 46: centos-7, ubuntu-18.04 > >>> + // 44: ubuntu-16.04 > >>> + // Limit the used address space to mitigate the bug, except we are > >>> + // running in a guest with more than 1TB of memory installed. > >>> + if (RamSizeOver4G < (1LL << 40)) { > >>> + pci_mem64_top = 1LL << 44; > >>> + } > >>> } > >>> } > >>> > >> > >> we've also had two reports about issues with 32-bit guests now[0][1] > >> (both had '-m 4096' in the QEMU commandline), which I was able to > >> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The > >> QEMU commandline is below[2]. > > > > is it also reproducible with upstream kernel? > > if yes, it would be better to fix that on guest kernel side, > > rather than in SeaBIOS which has no idea what guest OS is going to be > > running after it. > > > > Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE > boots fine (but the PAE was the one installed by default). I built a > 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't. > > >> Unfortunately, it still fails to boot, even with the "limit address > >> space used for pci devices, part two" patch applied on top of rel-1.16.3 > >> (and using current QEMU master). > >> > >> It boots fine with '-m 3583', but not anymore with '-m 3584'. So is > >> bumping the limit for the check necessary after all? > >> > >> Best Regards, > >> Fiona > >> > >> [0]: https://forum.proxmox.com/threads/150217/ > > > >> [1]: https://forum.proxmox.com/threads/149772/post-683562 > > well, with current QEMU master branch (I assume it haven't even got topic patch yet) > > ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600 > > following CLI boots just fine an 32-bit XP on Haswell host. > > > > Also using 'host' with ancient OS, is basically asking for trouble. > > If it works for user with old 'XP contemporary' CPU model, user should use that > > instead or workarounds (aka it's management task to configure CLI in > > compatible with OS manner). > > > > From suspicions config options in that post I see 'viommu' and 'vmgenid', > > are you sure XP even knows what to do with that, perhaps it triggers BSOD. > > > > No idea why the user enabled viommu for XP. vmgenid is added by our > management stack by default and I don't remember it having ever caused > problems. The user said the VM booted fine after adding lm=off to the > CPU options. I'm not super interested in the Windows case to be honest O:) > >> [2]: > >> > >>> ./qemu-system-x86_64 \ > >>> -accel 'kvm' \ > >>> -cpu 'host' \ > >>> -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \ > >>> -mon 'chardev=qmp,mode=control' \ > >>> -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \ > >>> -mon 'chardev=qmp-event,mode=control' \ > >>> -pidfile /var/run/qemu-server/121.pid \ > >>> -smp '4,sockets=1,cores=4,maxcpus=4' \ > >>> -nodefaults \ > >>> -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \ > >>> -m 4096 \ > >>> -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \ > >>> -device 'VGA,id=vga,bus=pci.0,addr=0x2' \ > >>> -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \ > >>> -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \ > >>> -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \ > >>> -machine 'type=pc' > > > > can you try to boot with q35 + intel iommu enabled > > and/or force virtio into legacy mode. > > > > I tried > > > -device 'intel-iommu' \ > > -machine 'type=q35' > > and intel_iommu=on in the guest's kernel cmdline, but it still failed > with kernels with PAE. > > Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci > line made it work however (also with pc machine) :) does it also help in Windows case? Perhaps it's a better workaround (compared to lm=off or going back to older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side, as it directly targets bug in guest's virtio-driver. Can we put this config tweak into libosinfo somehow, so that provisioning tools/mgmt could all reuse that to properly configure virtio for PAE kernels? > Best Regards, > Fiona > _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
Am 16.07.24 um 14:48 schrieb Igor Mammedov: > On Tue, 16 Jul 2024 13:56:08 +0200 > Fiona Ebner <f.ebner@proxmox.com> wrote: > >> Am 12.07.24 um 15:26 schrieb Igor Mammedov: >>> On Fri, 12 Jul 2024 14:24:40 +0200 >>> Fiona Ebner <f.ebner@proxmox.com> wrote: >>>> we've also had two reports about issues with 32-bit guests now[0][1] >>>> (both had '-m 4096' in the QEMU commandline), which I was able to >>>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The >>>> QEMU commandline is below[2]. >>> >>> is it also reproducible with upstream kernel? >>> if yes, it would be better to fix that on guest kernel side, >>> rather than in SeaBIOS which has no idea what guest OS is going to be >>> running after it. >>> >> >> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE >> boots fine (but the PAE was the one installed by default). I built a >> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't. >> >> >> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci >> line made it work however (also with pc machine) :) > does it also help in Windows case? > Sorry, I haven't set it up right now. I only reproduced the issue with Debian. > Perhaps it's a better workaround (compared to lm=off or going back to > older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side, > as it directly targets bug in guest's virtio-driver. > > Can we put this config tweak into libosinfo somehow, so that provisioning > tools/mgmt could all reuse that to properly configure virtio for PAE kernels? How would you detect whether the guest kernel is PAE or not before starting QEMU? Best Regards, Fiona _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Tue, 16 Jul 2024 15:58:35 +0200 Fiona Ebner <f.ebner@proxmox.com> wrote: > Am 16.07.24 um 14:48 schrieb Igor Mammedov: > > On Tue, 16 Jul 2024 13:56:08 +0200 > > Fiona Ebner <f.ebner@proxmox.com> wrote: > > > >> Am 12.07.24 um 15:26 schrieb Igor Mammedov: > >>> On Fri, 12 Jul 2024 14:24:40 +0200 > >>> Fiona Ebner <f.ebner@proxmox.com> wrote: > >>>> we've also had two reports about issues with 32-bit guests now[0][1] > >>>> (both had '-m 4096' in the QEMU commandline), which I was able to > >>>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The > >>>> QEMU commandline is below[2]. > >>> > >>> is it also reproducible with upstream kernel? > >>> if yes, it would be better to fix that on guest kernel side, > >>> rather than in SeaBIOS which has no idea what guest OS is going to be > >>> running after it. > >>> > >> > >> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE > >> boots fine (but the PAE was the one installed by default). I built a > >> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't. > >> > >> > >> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci > >> line made it work however (also with pc machine) :) > > does it also help in Windows case? > > > > Sorry, I haven't set it up right now. I only reproduced the issue with > Debian. > > > Perhaps it's a better workaround (compared to lm=off or going back to > > older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side, > > as it directly targets bug in guest's virtio-driver. > > > > Can we put this config tweak into libosinfo somehow, so that provisioning > > tools/mgmt could all reuse that to properly configure virtio for PAE kernels? > > How would you detect whether the guest kernel is PAE or not before > starting QEMU? that's what mgmt can do by poking in install media/disk image or asking user only mgmt can potentially know what target OS would be and properly configure QEMU (it's not something qemu or firmware can reasonably deal with on their own). (libosinfo can detect OS on install media, perhaps it also could be taught to probe what kernel would be used) As Gerd have said, all we can do at firmware level is heuristics, and in this case there is no good solution, we either hurt PAE guests with buggy driver by increasing size or hurt 64-bit guests by reducing it. In both cases we would get bug reports, and I'd expect a shift in numbers from PAE reports towards large device hotplug as time goes on. > Best Regards, > Fiona > _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
Am 16.07.24 um 17:15 schrieb Igor Mammedov: > On Tue, 16 Jul 2024 15:58:35 +0200 > Fiona Ebner <f.ebner@proxmox.com> wrote: > >> Am 16.07.24 um 14:48 schrieb Igor Mammedov: >>> On Tue, 16 Jul 2024 13:56:08 +0200 >>> Fiona Ebner <f.ebner@proxmox.com> wrote: >>> >>>> Am 12.07.24 um 15:26 schrieb Igor Mammedov: >>>>> On Fri, 12 Jul 2024 14:24:40 +0200 >>>>> Fiona Ebner <f.ebner@proxmox.com> wrote: >>>>>> we've also had two reports about issues with 32-bit guests now[0][1] >>>>>> (both had '-m 4096' in the QEMU commandline), which I was able to >>>>>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The >>>>>> QEMU commandline is below[2]. >>>>> >>>>> is it also reproducible with upstream kernel? >>>>> if yes, it would be better to fix that on guest kernel side, >>>>> rather than in SeaBIOS which has no idea what guest OS is going to be >>>>> running after it. >>>>> >>>> >>>> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE >>>> boots fine (but the PAE was the one installed by default). I built a >>>> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't. >>>> >>>> >>>> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci >>>> line made it work however (also with pc machine) :) >>> does it also help in Windows case? >>> >> >> Sorry, I haven't set it up right now. I only reproduced the issue with >> Debian. >> >>> Perhaps it's a better workaround (compared to lm=off or going back to >>> older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side, >>> as it directly targets bug in guest's virtio-driver. >>> >>> Can we put this config tweak into libosinfo somehow, so that provisioning >>> tools/mgmt could all reuse that to properly configure virtio for PAE kernels? >> >> How would you detect whether the guest kernel is PAE or not before >> starting QEMU? > > that's what mgmt can do by poking in install media/disk image or asking user > only mgmt can potentially know what target OS would be and > properly configure QEMU (it's not something qemu or firmware can reasonably > deal with on their own). (libosinfo can detect OS on install media, perhaps > it also could be taught to probe what kernel would be used) > > As Gerd have said, all we can do at firmware level is heuristics, > and in this case there is no good solution, we either hurt PAE guests > with buggy driver by increasing size or hurt 64-bit guests by reducing it. > In both cases we would get bug reports, and I'd expect a shift in numbers > from PAE reports towards large device hotplug as time goes on. > I see, thank you very much for the explanations! Best Regards, Fiona _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, 21 Jun 2024 14:05:17 +0200 Gerd Hoffmann <kraxel@redhat.com> wrote: > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the > > 64-bit pci io window, failing during boot with errors like: > > Well. Why people would use *that* ubuntu version is not clear to me. > It's *loooooong* out of support. Even the LTS version from that year > (16.04) is not supported any more. But it is at least available for > download still, so I gave it a spin. > > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka > 44 phys-bits). Test patch below. > > take care, > Gerd > > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c > index bb44dc296047..a43876a931c9 100644 > --- a/src/fw/pciinit.c > +++ b/src/fw/pciinit.c > @@ -1189,11 +1189,16 @@ pci_setup(void) > > if (CPUPhysBits) { > pci_mem64_top = 1LL << CPUPhysBits; > - if (CPUPhysBits > 46) { > - // Old linux kernels have trouble dealing with more than 46 > - // phys-bits, so avoid that for now. Seems to be a bug in the > - // virtio-pci driver. Reported: centos-7, ubuntu-18.04 > - pci_mem64_top = 1LL << 46; > + if (CPUPhysBits > 44) { > + // Old linux kernels have trouble dealing with more than 44/46 > + // phys-bits. Seems to be a bug in the virtio-pci driver. > + // 46: centos-7, ubuntu-18.04 > + // 44: ubuntu-16.04 > + // Limit the used address space to mitigate the bug, except we are > + // running in a guest with more than 1TB of memory installed. Is it possible to fix those broken drivers (centos-7 for example) and ditch this heuristic altogether? The rest of downstream can pick it up from there if they care about their customers. > + if (RamSizeOver4G < (1LL << 40)) { > + pci_mem64_top = 1LL << 44; > + } > } > } > > > _______________________________________________ > SeaBIOS mailing list -- seabios@seabios.org > To unsubscribe send an email to seabios-leave@seabios.org > _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, Jun 21, 2024 at 03:20:19PM GMT, Igor Mammedov wrote: > > > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c > > index bb44dc296047..a43876a931c9 100644 > > --- a/src/fw/pciinit.c > > +++ b/src/fw/pciinit.c > > @@ -1189,11 +1189,16 @@ pci_setup(void) > > > > if (CPUPhysBits) { > > pci_mem64_top = 1LL << CPUPhysBits; > > - if (CPUPhysBits > 46) { > > - // Old linux kernels have trouble dealing with more than 46 > > - // phys-bits, so avoid that for now. Seems to be a bug in the > > - // virtio-pci driver. Reported: centos-7, ubuntu-18.04 > > - pci_mem64_top = 1LL << 46; > > + if (CPUPhysBits > 44) { > > + // Old linux kernels have trouble dealing with more than 44/46 > > + // phys-bits. Seems to be a bug in the virtio-pci driver. > > + // 46: centos-7, ubuntu-18.04 > > + // 44: ubuntu-16.04 > > + // Limit the used address space to mitigate the bug, except we are > > + // running in a guest with more than 1TB of memory installed. > > Is it possible to fix those broken drivers (centos-7 for example) > and ditch this heuristic altogether? > The rest of downstream can pick it up from there if they care about > their customers. Some further testing showed that this is not version-specific but arch specific. Old 32-bit kernels fail >44, old 64-bit kernels fail >46. Note that 44 = 32 + 12, i.e. this could be pfn (page frame number) hitting MAX_UINT32. Should that be the case the fix is probably not easy (didn't check the kernel source though). Also note that releasing a kernel fix is not enough, you also have to respin install media. Distros which are *that* old typically don't get regular install media updates any more ... In short: The idea to fix distros and drop the heuristic is IMHO not realistic. take care, Gerd _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
Hi, Dne 21. 06. 24 v 15:20 Igor Mammedov napsal(a): >> + // Old linux kernels have trouble dealing with more than 44/46 >> + // phys-bits. Seems to be a bug in the virtio-pci driver. >> + // 46: centos-7, ubuntu-18.04 >> + // 44: ubuntu-16.04 >> + // Limit the used address space to mitigate the bug, except we are >> + // running in a guest with more than 1TB of memory installed. > > Is it possible to fix those broken drivers (centos-7 for example) > and ditch this heuristic altogether? Does this code ever runs in some baremetal use cases as well? Or at least, meybe this could be applied only when virtio devices are present (maybe even just when transitional devices are present) ? Thanks, Rudolf _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, Jun 21, 2024 at 10:29:45PM +0200, Rudolf Marek wrote: > Hi, > > Dne 21. 06. 24 v 15:20 Igor Mammedov napsal(a): > > > + // Old linux kernels have trouble dealing with more than 44/46 > > > + // phys-bits. Seems to be a bug in the virtio-pci driver. > > > + // 46: centos-7, ubuntu-18.04 > > > + // 44: ubuntu-16.04 > > > + // Limit the used address space to mitigate the bug, except we are > > > + // running in a guest with more than 1TB of memory installed. > > > > Is it possible to fix those broken drivers (centos-7 for example) > > and ditch this heuristic altogether? > > Does this code ever runs in some baremetal use cases as well? > > Or at least, meybe this could be applied only when virtio devices are present (maybe even just when transitional devices are present) ? This code is only used on qemu (and derivatives) - specifically CONFIG_QEMU must be true. When running on coreboot, SeaBIOS expects coreboot to map all the PCI devices. Cheers, -Kevin _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, Jun 21, 2024 at 02:05:17PM +0200, Gerd Hoffmann wrote: > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the > > 64-bit pci io window, failing during boot with errors like: > > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka > 44 phys-bits). Test patch below. Thanks for the patch, I can confirm this also works with Ubuntu 14.04 (oldest we had to hand) as well as a couple of 32-bit Windows VMs. This is a much better fix! > Even the LTS version from that year (16.04) is not supported any more. Even 14.04 is not yet end of life. If you're prepared to pay, they'll still support you. https://wiki.ubuntu.com/Releases > Well. Why people would use *that* ubuntu version is not clear to me. > It's *loooooong* out of support. You're in the IT dept of a large corporation. You have some critical application running on some old Dell server with Ubuntu 16.04. A move to virtualization has been mandated across the org, so you need to decommission that server. The application was built by some contractor - before your time - and the source code was long lost, due to a misadventure with a misconfigured array - again before your time. You've tried to use a newer version, but the application depends on lots of libraries that didn't take compatibility seriously (like, say, GNOME), so it simply can't run on newer versions. You've tried for some time to work around this by building and installing dependencies but you're not an expert on dynamic linkers, and could never get that last C++ symbol to resolve. There's no funding to build a new replacement for the app. You're aware that the OS is out of full support, so you do your best to lock down any network access and mitigate the relevant CVEs. Now you try to upgrade your virtualization cluster, and your VM doesn't boot any more. This kind of situation is very common. It's Long tail is long :( regards john _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, Jun 21, 2024 at 01:37:24PM GMT, John Levon wrote: > On Fri, Jun 21, 2024 at 02:05:17PM +0200, Gerd Hoffmann wrote: > > > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: > > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the > > > 64-bit pci io window, failing during boot with errors like: > > > Well. Why people would use *that* ubuntu version is not clear to me. > > It's *loooooong* out of support. > > You're in the IT dept of a large corporation. You have some critical > application running on some old Dell server with Ubuntu 16.04. I was specifically referring to the "16.10" listed above, i.e. running critical stuff on non-LTS distros in the first place. 16.04-LTS is a much better choice, and that there are plenty of things which can happen in real life which can delay or even prevent moving applications to a newer LTS version is pretty clear. take care, Gerd _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
On Fri, Jun 21, 2024 at 01:37:24PM +0100, John Levon wrote: > On Fri, Jun 21, 2024 at 02:05:17PM +0200, Gerd Hoffmann wrote: > > > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote: > > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the > > > 64-bit pci io window, failing during boot with errors like: > > > > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka > > 44 phys-bits). Test patch below. > > Thanks for the patch, I can confirm this also works with Ubuntu 14.04 (oldest we > had to hand) as well as a couple of 32-bit Windows VMs. This is a much better > fix! Thanks for tracking this down and testing. I agree this looks like an improved fix. -Kevin _______________________________________________ SeaBIOS mailing list -- seabios@seabios.org To unsubscribe send an email to seabios-leave@seabios.org
© 2016 - 2024 Red Hat, Inc.