On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> 64-bit pci io window, failing during boot with errors like:

Well.  Why people would use *that* ubuntu version is not clear to me.
It's *loooooong* out of support.  Even the LTS version from that year
(16.04) is not supported any more.  But it is at least available for
download still,  so I gave it a spin.

Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
44 phys-bits).  Test patch below.

take care,
  Gerd

diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
index bb44dc296047..a43876a931c9 100644
--- a/src/fw/pciinit.c
+++ b/src/fw/pciinit.c
@@ -1189,11 +1189,16 @@ pci_setup(void)
 
     if (CPUPhysBits) {
         pci_mem64_top = 1LL << CPUPhysBits;
-        if (CPUPhysBits > 46) {
-            // Old linux kernels have trouble dealing with more than 46
-            // phys-bits, so avoid that for now.  Seems to be a bug in the
-            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
-            pci_mem64_top = 1LL << 46;
+        if (CPUPhysBits > 44) {
+            // Old linux kernels have trouble dealing with more than 44/46
+            // phys-bits. Seems to be a bug in the virtio-pci driver.
+            //   46:  centos-7, ubuntu-18.04
+            //   44:  ubuntu-16.04
+            // Limit the used address space to mitigate the bug, except we are
+            // running in a guest with more than 1TB of memory installed.
+            if (RamSizeOver4G < (1LL << 40)) {
+                pci_mem64_top = 1LL << 44;
+            }
         }
     }
 

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

Hi,

Am 21.06.24 um 14:05 schrieb Gerd Hoffmann:
> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
>> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
>> 64-bit pci io window, failing during boot with errors like:
> 
> Well.  Why people would use *that* ubuntu version is not clear to me.
> It's *loooooong* out of support.  Even the LTS version from that year
> (16.04) is not supported any more.  But it is at least available for
> download still,  so I gave it a spin.
> 
> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> 44 phys-bits).  Test patch below.
> 
> take care,
>   Gerd
> 
> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> index bb44dc296047..a43876a931c9 100644
> --- a/src/fw/pciinit.c
> +++ b/src/fw/pciinit.c
> @@ -1189,11 +1189,16 @@ pci_setup(void)
>  
>      if (CPUPhysBits) {
>          pci_mem64_top = 1LL << CPUPhysBits;
> -        if (CPUPhysBits > 46) {
> -            // Old linux kernels have trouble dealing with more than 46
> -            // phys-bits, so avoid that for now.  Seems to be a bug in the
> -            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
> -            pci_mem64_top = 1LL << 46;
> +        if (CPUPhysBits > 44) {
> +            // Old linux kernels have trouble dealing with more than 44/46
> +            // phys-bits. Seems to be a bug in the virtio-pci driver.
> +            //   46:  centos-7, ubuntu-18.04
> +            //   44:  ubuntu-16.04
> +            // Limit the used address space to mitigate the bug, except we are
> +            // running in a guest with more than 1TB of memory installed.
> +            if (RamSizeOver4G < (1LL << 40)) {
> +                pci_mem64_top = 1LL << 44;
> +            }
>          }
>      }
>  

we've also had two reports about issues with 32-bit guests now[0][1]
(both had '-m 4096' in the QEMU commandline), which I was able to
reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
QEMU commandline is below[2].

Unfortunately, it still fails to boot, even with the "limit address
space used for pci devices, part two" patch applied on top of rel-1.16.3
(and using current QEMU master).

It boots fine with '-m 3583', but not anymore with '-m 3584'. So is
bumping the limit for the check necessary after all?

Best Regards,
Fiona

[0]: https://forum.proxmox.com/threads/150217/
[1]: https://forum.proxmox.com/threads/149772/post-683562
[2]:

> ./qemu-system-x86_64 \
>   -accel 'kvm' \
>   -cpu 'host' \
>   -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \
>   -mon 'chardev=qmp,mode=control' \
>   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
>   -mon 'chardev=qmp-event,mode=control' \
>   -pidfile /var/run/qemu-server/121.pid \
>   -smp '4,sockets=1,cores=4,maxcpus=4' \
>   -nodefaults \
>   -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \
>   -m 4096 \
>   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
>   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
>   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
>   -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
>   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
>   -machine 'type=pc'

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

  Hi,

> we've also had two reports about issues with 32-bit guests now[0][1]
> (both had '-m 4096' in the QEMU commandline), which I was able to
> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
> QEMU commandline is below[2].
> 
> Unfortunately, it still fails to boot, even with the "limit address
> space used for pci devices, part two" patch applied on top of rel-1.16.3
> (and using current QEMU master).

Reproduces here.  Seems there is something seriously wrong in that
debian kernel.

Even when going for 36 phys bits (-cpu host,host-phys-bits-limit=36),
which is the physical address space of PAE mode when it was introduced
by the pentium pro, the guest fails to boot.  That *really* should not
happen.

I guess what you are seeing here is i386 support in the linux kernel
starting to bitrot due to everybody moving to 64-bit ...

take care,
  Gerd

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, 12 Jul 2024 14:24:40 +0200
Fiona Ebner <f.ebner@proxmox.com> wrote:

> Hi,
> 
> Am 21.06.24 um 14:05 schrieb Gerd Hoffmann:
> > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:  
> >> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> >> 64-bit pci io window, failing during boot with errors like:  
> > 
> > Well.  Why people would use *that* ubuntu version is not clear to me.
> > It's *loooooong* out of support.  Even the LTS version from that year
> > (16.04) is not supported any more.  But it is at least available for
> > download still,  so I gave it a spin.
> > 
> > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> > 44 phys-bits).  Test patch below.
> > 
> > take care,
> >   Gerd
> > 
> > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> > index bb44dc296047..a43876a931c9 100644
> > --- a/src/fw/pciinit.c
> > +++ b/src/fw/pciinit.c
> > @@ -1189,11 +1189,16 @@ pci_setup(void)
> >  
> >      if (CPUPhysBits) {
> >          pci_mem64_top = 1LL << CPUPhysBits;
> > -        if (CPUPhysBits > 46) {
> > -            // Old linux kernels have trouble dealing with more than 46
> > -            // phys-bits, so avoid that for now.  Seems to be a bug in the
> > -            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
> > -            pci_mem64_top = 1LL << 46;
> > +        if (CPUPhysBits > 44) {
> > +            // Old linux kernels have trouble dealing with more than 44/46
> > +            // phys-bits. Seems to be a bug in the virtio-pci driver.
> > +            //   46:  centos-7, ubuntu-18.04
> > +            //   44:  ubuntu-16.04
> > +            // Limit the used address space to mitigate the bug, except we are
> > +            // running in a guest with more than 1TB of memory installed.
> > +            if (RamSizeOver4G < (1LL << 40)) {
> > +                pci_mem64_top = 1LL << 44;
> > +            }
> >          }
> >      }
> >    
> 
> we've also had two reports about issues with 32-bit guests now[0][1]
> (both had '-m 4096' in the QEMU commandline), which I was able to
> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
> QEMU commandline is below[2].

is it also reproducible with upstream kernel?
if yes, it would be better to fix that on guest kernel side,
rather than in SeaBIOS which has no idea what guest OS is going to be
running after it.

> Unfortunately, it still fails to boot, even with the "limit address
> space used for pci devices, part two" patch applied on top of rel-1.16.3
> (and using current QEMU master).
> 
> It boots fine with '-m 3583', but not anymore with '-m 3584'. So is
> bumping the limit for the check necessary after all?
> 
> Best Regards,
> Fiona
> 
> [0]: https://forum.proxmox.com/threads/150217/

> [1]: https://forum.proxmox.com/threads/149772/post-683562
well, with current QEMU master branch (I assume it haven't even got topic patch yet)
  ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600
following CLI boots just fine an 32-bit XP on Haswell host.

Also using 'host' with ancient OS, is basically asking for trouble.
If it works for user with old 'XP contemporary' CPU model, user should use that
instead or workarounds (aka it's management task to configure CLI in
compatible with OS manner).

From suspicions config options in that post I see 'viommu' and 'vmgenid',
are you sure XP even knows what to do with that, perhaps it triggers BSOD.

> [2]:
> 
> > ./qemu-system-x86_64 \
> >   -accel 'kvm' \
> >   -cpu 'host' \
> >   -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \
> >   -mon 'chardev=qmp,mode=control' \
> >   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
> >   -mon 'chardev=qmp-event,mode=control' \
> >   -pidfile /var/run/qemu-server/121.pid \
> >   -smp '4,sockets=1,cores=4,maxcpus=4' \
> >   -nodefaults \
> >   -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \
> >   -m 4096 \
> >   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
> >   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
> >   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
> >   -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
> >   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
> >   -machine 'type=pc'  

can you try to boot with q35 + intel iommu enabled
and/or force virtio into legacy mode.

> 
> _______________________________________________
> SeaBIOS mailing list -- seabios@seabios.org
> To unsubscribe send an email to seabios-leave@seabios.org
> 

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

Am 12.07.24 um 15:26 schrieb Igor Mammedov:
> On Fri, 12 Jul 2024 14:24:40 +0200
> Fiona Ebner <f.ebner@proxmox.com> wrote:
> 
>> Hi,
>>
>> Am 21.06.24 um 14:05 schrieb Gerd Hoffmann:
>>> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:  
>>>> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
>>>> 64-bit pci io window, failing during boot with errors like:  
>>>
>>> Well.  Why people would use *that* ubuntu version is not clear to me.
>>> It's *loooooong* out of support.  Even the LTS version from that year
>>> (16.04) is not supported any more.  But it is at least available for
>>> download still,  so I gave it a spin.
>>>
>>> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
>>> 44 phys-bits).  Test patch below.
>>>
>>> take care,
>>>   Gerd
>>>
>>> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
>>> index bb44dc296047..a43876a931c9 100644
>>> --- a/src/fw/pciinit.c
>>> +++ b/src/fw/pciinit.c
>>> @@ -1189,11 +1189,16 @@ pci_setup(void)
>>>  
>>>      if (CPUPhysBits) {
>>>          pci_mem64_top = 1LL << CPUPhysBits;
>>> -        if (CPUPhysBits > 46) {
>>> -            // Old linux kernels have trouble dealing with more than 46
>>> -            // phys-bits, so avoid that for now.  Seems to be a bug in the
>>> -            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
>>> -            pci_mem64_top = 1LL << 46;
>>> +        if (CPUPhysBits > 44) {
>>> +            // Old linux kernels have trouble dealing with more than 44/46
>>> +            // phys-bits. Seems to be a bug in the virtio-pci driver.
>>> +            //   46:  centos-7, ubuntu-18.04
>>> +            //   44:  ubuntu-16.04
>>> +            // Limit the used address space to mitigate the bug, except we are
>>> +            // running in a guest with more than 1TB of memory installed.
>>> +            if (RamSizeOver4G < (1LL << 40)) {
>>> +                pci_mem64_top = 1LL << 44;
>>> +            }
>>>          }
>>>      }
>>>    
>>
>> we've also had two reports about issues with 32-bit guests now[0][1]
>> (both had '-m 4096' in the QEMU commandline), which I was able to
>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
>> QEMU commandline is below[2].
> 
> is it also reproducible with upstream kernel?
> if yes, it would be better to fix that on guest kernel side,
> rather than in SeaBIOS which has no idea what guest OS is going to be
> running after it.
> 

Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE
boots fine (but the PAE was the one installed by default). I built a
6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't.

>> Unfortunately, it still fails to boot, even with the "limit address
>> space used for pci devices, part two" patch applied on top of rel-1.16.3
>> (and using current QEMU master).
>>
>> It boots fine with '-m 3583', but not anymore with '-m 3584'. So is
>> bumping the limit for the check necessary after all?
>>
>> Best Regards,
>> Fiona
>>
>> [0]: https://forum.proxmox.com/threads/150217/
> 
>> [1]: https://forum.proxmox.com/threads/149772/post-683562
> well, with current QEMU master branch (I assume it haven't even got topic patch yet)
>   ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600
> following CLI boots just fine an 32-bit XP on Haswell host.
> 
> Also using 'host' with ancient OS, is basically asking for trouble.
> If it works for user with old 'XP contemporary' CPU model, user should use that
> instead or workarounds (aka it's management task to configure CLI in
> compatible with OS manner).
> 
> From suspicions config options in that post I see 'viommu' and 'vmgenid',
> are you sure XP even knows what to do with that, perhaps it triggers BSOD.
> 

No idea why the user enabled viommu for XP. vmgenid is added by our
management stack by default and I don't remember it having ever caused
problems. The user said the VM booted fine after adding lm=off to the
CPU options. I'm not super interested in the Windows case to be honest O:)

>> [2]:
>>
>>> ./qemu-system-x86_64 \
>>>   -accel 'kvm' \
>>>   -cpu 'host' \
>>>   -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \
>>>   -mon 'chardev=qmp,mode=control' \
>>>   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
>>>   -mon 'chardev=qmp-event,mode=control' \
>>>   -pidfile /var/run/qemu-server/121.pid \
>>>   -smp '4,sockets=1,cores=4,maxcpus=4' \
>>>   -nodefaults \
>>>   -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \
>>>   -m 4096 \
>>>   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
>>>   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
>>>   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
>>>   -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
>>>   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
>>>   -machine 'type=pc'  
> 
> can you try to boot with q35 + intel iommu enabled
> and/or force virtio into legacy mode.
> 

I tried

>   -device 'intel-iommu' \
>   -machine 'type=q35'

and intel_iommu=on in the guest's kernel cmdline, but it still failed
with kernels with PAE.

Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci
line made it work however (also with pc machine) :)

Best Regards,
Fiona

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Tue, 16 Jul 2024 13:56:08 +0200
Fiona Ebner <f.ebner@proxmox.com> wrote:

> Am 12.07.24 um 15:26 schrieb Igor Mammedov:
> > On Fri, 12 Jul 2024 14:24:40 +0200
> > Fiona Ebner <f.ebner@proxmox.com> wrote:
> >   
> >> Hi,
> >>
> >> Am 21.06.24 um 14:05 schrieb Gerd Hoffmann:  
> >>> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:    
> >>>> Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> >>>> 64-bit pci io window, failing during boot with errors like:    
> >>>
> >>> Well.  Why people would use *that* ubuntu version is not clear to me.
> >>> It's *loooooong* out of support.  Even the LTS version from that year
> >>> (16.04) is not supported any more.  But it is at least available for
> >>> download still,  so I gave it a spin.
> >>>
> >>> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> >>> 44 phys-bits).  Test patch below.
> >>>
> >>> take care,
> >>>   Gerd
> >>>
> >>> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> >>> index bb44dc296047..a43876a931c9 100644
> >>> --- a/src/fw/pciinit.c
> >>> +++ b/src/fw/pciinit.c
> >>> @@ -1189,11 +1189,16 @@ pci_setup(void)
> >>>  
> >>>      if (CPUPhysBits) {
> >>>          pci_mem64_top = 1LL << CPUPhysBits;
> >>> -        if (CPUPhysBits > 46) {
> >>> -            // Old linux kernels have trouble dealing with more than 46
> >>> -            // phys-bits, so avoid that for now.  Seems to be a bug in the
> >>> -            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
> >>> -            pci_mem64_top = 1LL << 46;
> >>> +        if (CPUPhysBits > 44) {
> >>> +            // Old linux kernels have trouble dealing with more than 44/46
> >>> +            // phys-bits. Seems to be a bug in the virtio-pci driver.
> >>> +            //   46:  centos-7, ubuntu-18.04
> >>> +            //   44:  ubuntu-16.04
> >>> +            // Limit the used address space to mitigate the bug, except we are
> >>> +            // running in a guest with more than 1TB of memory installed.
> >>> +            if (RamSizeOver4G < (1LL << 40)) {
> >>> +                pci_mem64_top = 1LL << 44;
> >>> +            }
> >>>          }
> >>>      }
> >>>      
> >>
> >> we've also had two reports about issues with 32-bit guests now[0][1]
> >> (both had '-m 4096' in the QEMU commandline), which I was able to
> >> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
> >> QEMU commandline is below[2].  
> > 
> > is it also reproducible with upstream kernel?
> > if yes, it would be better to fix that on guest kernel side,
> > rather than in SeaBIOS which has no idea what guest OS is going to be
> > running after it.
> >   
> 
> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE
> boots fine (but the PAE was the one installed by default). I built a
> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't.
> 
> >> Unfortunately, it still fails to boot, even with the "limit address
> >> space used for pci devices, part two" patch applied on top of rel-1.16.3
> >> (and using current QEMU master).
> >>
> >> It boots fine with '-m 3583', but not anymore with '-m 3584'. So is
> >> bumping the limit for the check necessary after all?
> >>
> >> Best Regards,
> >> Fiona
> >>
> >> [0]: https://forum.proxmox.com/threads/150217/  
> >   
> >> [1]: https://forum.proxmox.com/threads/149772/post-683562  
> > well, with current QEMU master branch (I assume it haven't even got topic patch yet)
> >   ./qemu-system-x86_64 --enable-kvm -cpu host -smp 4 -snapshot -m 4096 -M pc-i440fx-6.1 winxp_x86_build2600
> > following CLI boots just fine an 32-bit XP on Haswell host.
> > 
> > Also using 'host' with ancient OS, is basically asking for trouble.
> > If it works for user with old 'XP contemporary' CPU model, user should use that
> > instead or workarounds (aka it's management task to configure CLI in
> > compatible with OS manner).
> > 
> > From suspicions config options in that post I see 'viommu' and 'vmgenid',
> > are you sure XP even knows what to do with that, perhaps it triggers BSOD.
> >   
> 
> No idea why the user enabled viommu for XP. vmgenid is added by our
> management stack by default and I don't remember it having ever caused
> problems. The user said the VM booted fine after adding lm=off to the
> CPU options. I'm not super interested in the Windows case to be honest O:)
> >> [2]:
> >>  
> >>> ./qemu-system-x86_64 \
> >>>   -accel 'kvm' \
> >>>   -cpu 'host' \
> >>>   -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server=on,wait=off' \
> >>>   -mon 'chardev=qmp,mode=control' \
> >>>   -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
> >>>   -mon 'chardev=qmp-event,mode=control' \
> >>>   -pidfile /var/run/qemu-server/121.pid \
> >>>   -smp '4,sockets=1,cores=4,maxcpus=4' \
> >>>   -nodefaults \
> >>>   -vnc 'unix:/var/run/qemu-server/121.vnc,password=on' \
> >>>   -m 4096 \
> >>>   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
> >>>   -device 'VGA,id=vga,bus=pci.0,addr=0x2' \
> >>>   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
> >>>   -drive 'file=/dev/lvmthinbig/vm-121-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' \
> >>>   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' \
> >>>   -machine 'type=pc'    
> > 
> > can you try to boot with q35 + intel iommu enabled
> > and/or force virtio into legacy mode.
> >   
> 
> I tried
> 
> >   -device 'intel-iommu' \
> >   -machine 'type=q35'  
> 
> and intel_iommu=on in the guest's kernel cmdline, but it still failed
> with kernels with PAE.
> 
> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci
> line made it work however (also with pc machine) :)
does it also help in Windows case?

Perhaps it's a better workaround (compared to lm=off or going back to
older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side,
as it directly targets bug in guest's virtio-driver.
 
Can we put this config tweak into libosinfo somehow, so that provisioning
tools/mgmt could all reuse that to properly configure virtio for PAE kernels?


> Best Regards,
> Fiona
> 

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

Am 16.07.24 um 14:48 schrieb Igor Mammedov:
> On Tue, 16 Jul 2024 13:56:08 +0200
> Fiona Ebner <f.ebner@proxmox.com> wrote:
> 
>> Am 12.07.24 um 15:26 schrieb Igor Mammedov:
>>> On Fri, 12 Jul 2024 14:24:40 +0200
>>> Fiona Ebner <f.ebner@proxmox.com> wrote:
>>>> we've also had two reports about issues with 32-bit guests now[0][1]
>>>> (both had '-m 4096' in the QEMU commandline), which I was able to
>>>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
>>>> QEMU commandline is below[2].  
>>>
>>> is it also reproducible with upstream kernel?
>>> if yes, it would be better to fix that on guest kernel side,
>>> rather than in SeaBIOS which has no idea what guest OS is going to be
>>> running after it.
>>>   
>>
>> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE
>> boots fine (but the PAE was the one installed by default). I built a
>> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't.
>>
>>
>> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci
>> line made it work however (also with pc machine) :)
> does it also help in Windows case?
> 

Sorry, I haven't set it up right now. I only reproduced the issue with
Debian.

> Perhaps it's a better workaround (compared to lm=off or going back to
> older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side,
> as it directly targets bug in guest's virtio-driver.
>  
> Can we put this config tweak into libosinfo somehow, so that provisioning
> tools/mgmt could all reuse that to properly configure virtio for PAE kernels?

How would you detect whether the guest kernel is PAE or not before
starting QEMU?

Best Regards,
Fiona

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Tue, 16 Jul 2024 15:58:35 +0200
Fiona Ebner <f.ebner@proxmox.com> wrote:

> Am 16.07.24 um 14:48 schrieb Igor Mammedov:
> > On Tue, 16 Jul 2024 13:56:08 +0200
> > Fiona Ebner <f.ebner@proxmox.com> wrote:
> >   
> >> Am 12.07.24 um 15:26 schrieb Igor Mammedov:  
> >>> On Fri, 12 Jul 2024 14:24:40 +0200
> >>> Fiona Ebner <f.ebner@proxmox.com> wrote:  
> >>>> we've also had two reports about issues with 32-bit guests now[0][1]
> >>>> (both had '-m 4096' in the QEMU commandline), which I was able to
> >>>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
> >>>> QEMU commandline is below[2].    
> >>>
> >>> is it also reproducible with upstream kernel?
> >>> if yes, it would be better to fix that on guest kernel side,
> >>> rather than in SeaBIOS which has no idea what guest OS is going to be
> >>> running after it.
> >>>     
> >>
> >> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE
> >> boots fine (but the PAE was the one installed by default). I built a
> >> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't.
> >>
> >>
> >> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci
> >> line made it work however (also with pc machine) :)  
> > does it also help in Windows case?
> >   
> 
> Sorry, I haven't set it up right now. I only reproduced the issue with
> Debian.
> 
> > Perhaps it's a better workaround (compared to lm=off or going back to
> > older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side,
> > as it directly targets bug in guest's virtio-driver.
> >  
> > Can we put this config tweak into libosinfo somehow, so that provisioning
> > tools/mgmt could all reuse that to properly configure virtio for PAE kernels?  
> 
> How would you detect whether the guest kernel is PAE or not before
> starting QEMU?

that's what mgmt can do by poking in install media/disk image or asking user
only mgmt can potentially know what target OS would be and
properly configure QEMU (it's not something qemu or firmware can  reasonably
deal with on their own). (libosinfo can detect OS on install media, perhaps
it also could be taught to probe what kernel would be used)

As Gerd have said, all we can do at firmware level is heuristics,
and in this case there is no good solution, we either hurt PAE guests
with buggy driver by increasing size or hurt 64-bit guests by reducing it.
In both cases we would get bug reports, and I'd expect a shift in numbers
from PAE reports towards large device hotplug as time goes on.

> Best Regards,
> Fiona
> 

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

Am 16.07.24 um 17:15 schrieb Igor Mammedov:
> On Tue, 16 Jul 2024 15:58:35 +0200
> Fiona Ebner <f.ebner@proxmox.com> wrote:
> 
>> Am 16.07.24 um 14:48 schrieb Igor Mammedov:
>>> On Tue, 16 Jul 2024 13:56:08 +0200
>>> Fiona Ebner <f.ebner@proxmox.com> wrote:
>>>   
>>>> Am 12.07.24 um 15:26 schrieb Igor Mammedov:  
>>>>> On Fri, 12 Jul 2024 14:24:40 +0200
>>>>> Fiona Ebner <f.ebner@proxmox.com> wrote:  
>>>>>> we've also had two reports about issues with 32-bit guests now[0][1]
>>>>>> (both had '-m 4096' in the QEMU commandline), which I was able to
>>>>>> reproduce with a 32-bit Debian 12.6 install, so nothing ancient ;) The
>>>>>> QEMU commandline is below[2].    
>>>>>
>>>>> is it also reproducible with upstream kernel?
>>>>> if yes, it would be better to fix that on guest kernel side,
>>>>> rather than in SeaBIOS which has no idea what guest OS is going to be
>>>>> running after it.
>>>>>     
>>>>
>>>> Turns out it's only kernels with PAE. The 6.1 Debian kernel without PAE
>>>> boots fine (but the PAE was the one installed by default). I built a
>>>> 6.10 kernel and it also boots fine, a 6.10 build with PAE doesn't.
>>>>
>>>>
>>>> Appending 'disable-modern=on,disable-legacy=off' to the virtio-scsi-pci
>>>> line made it work however (also with pc machine) :)  
>>> does it also help in Windows case?
>>>   
>>
>> Sorry, I haven't set it up right now. I only reproduced the issue with
>> Debian.
>>
>>> Perhaps it's a better workaround (compared to lm=off or going back to
>>> older bios or dumb-ing down bios to suit PAE guests) for use on mgmt side,
>>> as it directly targets bug in guest's virtio-driver.
>>>  
>>> Can we put this config tweak into libosinfo somehow, so that provisioning
>>> tools/mgmt could all reuse that to properly configure virtio for PAE kernels?  
>>
>> How would you detect whether the guest kernel is PAE or not before
>> starting QEMU?
> 
> that's what mgmt can do by poking in install media/disk image or asking user
> only mgmt can potentially know what target OS would be and
> properly configure QEMU (it's not something qemu or firmware can  reasonably
> deal with on their own). (libosinfo can detect OS on install media, perhaps
> it also could be taught to probe what kernel would be used)
> 
> As Gerd have said, all we can do at firmware level is heuristics,
> and in this case there is no good solution, we either hurt PAE guests
> with buggy driver by increasing size or hurt 64-bit guests by reducing it.
> In both cases we would get bug reports, and I'd expect a shift in numbers
> from PAE reports towards large device hotplug as time goes on.
> 

I see, thank you very much for the explanations!

Best Regards,
Fiona

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, 21 Jun 2024 14:05:17 +0200
Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> > 64-bit pci io window, failing during boot with errors like:  
> 
> Well.  Why people would use *that* ubuntu version is not clear to me.
> It's *loooooong* out of support.  Even the LTS version from that year
> (16.04) is not supported any more.  But it is at least available for
> download still,  so I gave it a spin.
> 
> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> 44 phys-bits).  Test patch below.
> 
> take care,
>   Gerd
> 
> diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> index bb44dc296047..a43876a931c9 100644
> --- a/src/fw/pciinit.c
> +++ b/src/fw/pciinit.c
> @@ -1189,11 +1189,16 @@ pci_setup(void)
>  
>      if (CPUPhysBits) {
>          pci_mem64_top = 1LL << CPUPhysBits;
> -        if (CPUPhysBits > 46) {
> -            // Old linux kernels have trouble dealing with more than 46
> -            // phys-bits, so avoid that for now.  Seems to be a bug in the
> -            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
> -            pci_mem64_top = 1LL << 46;
> +        if (CPUPhysBits > 44) {
> +            // Old linux kernels have trouble dealing with more than 44/46
> +            // phys-bits. Seems to be a bug in the virtio-pci driver.
> +            //   46:  centos-7, ubuntu-18.04
> +            //   44:  ubuntu-16.04
> +            // Limit the used address space to mitigate the bug, except we are
> +            // running in a guest with more than 1TB of memory installed.

Is it possible to fix those broken drivers (centos-7 for example)
and ditch this heuristic altogether?
The rest of downstream can pick it up from there if they care about
their customers.

> +            if (RamSizeOver4G < (1LL << 40)) {
> +                pci_mem64_top = 1LL << 44;
> +            }
>          }
>      }
>  
> 
> _______________________________________________
> SeaBIOS mailing list -- seabios@seabios.org
> To unsubscribe send an email to seabios-leave@seabios.org
> 

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, Jun 21, 2024 at 03:20:19PM GMT, Igor Mammedov wrote:
> 
> > diff --git a/src/fw/pciinit.c b/src/fw/pciinit.c
> > index bb44dc296047..a43876a931c9 100644
> > --- a/src/fw/pciinit.c
> > +++ b/src/fw/pciinit.c
> > @@ -1189,11 +1189,16 @@ pci_setup(void)
> >  
> >      if (CPUPhysBits) {
> >          pci_mem64_top = 1LL << CPUPhysBits;
> > -        if (CPUPhysBits > 46) {
> > -            // Old linux kernels have trouble dealing with more than 46
> > -            // phys-bits, so avoid that for now.  Seems to be a bug in the
> > -            // virtio-pci driver.  Reported: centos-7, ubuntu-18.04
> > -            pci_mem64_top = 1LL << 46;
> > +        if (CPUPhysBits > 44) {
> > +            // Old linux kernels have trouble dealing with more than 44/46
> > +            // phys-bits. Seems to be a bug in the virtio-pci driver.
> > +            //   46:  centos-7, ubuntu-18.04
> > +            //   44:  ubuntu-16.04
> > +            // Limit the used address space to mitigate the bug, except we are
> > +            // running in a guest with more than 1TB of memory installed.
> 
> Is it possible to fix those broken drivers (centos-7 for example)
> and ditch this heuristic altogether?
> The rest of downstream can pick it up from there if they care about
> their customers.

Some further testing showed that this is not version-specific but arch
specific.  Old 32-bit kernels fail >44, old 64-bit kernels fail >46.

Note that 44 = 32 + 12, i.e. this could be pfn (page frame number)
hitting MAX_UINT32.  Should that be the case the fix is probably not
easy (didn't check the kernel source though).

Also note that releasing a kernel fix is not enough, you also have to
respin install media.  Distros which are *that* old typically don't get
regular install media updates any more ...

In short:  The idea to fix distros and drop the heuristic is IMHO not
realistic.

take care,
  Gerd

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

Hi,

Dne 21. 06. 24 v 15:20 Igor Mammedov napsal(a):
>> +            // Old linux kernels have trouble dealing with more than 44/46
>> +            // phys-bits. Seems to be a bug in the virtio-pci driver.
>> +            //   46:  centos-7, ubuntu-18.04
>> +            //   44:  ubuntu-16.04
>> +            // Limit the used address space to mitigate the bug, except we are
>> +            // running in a guest with more than 1TB of memory installed.
> 
> Is it possible to fix those broken drivers (centos-7 for example)
> and ditch this heuristic altogether?

Does this code ever runs in some baremetal use cases as well?

Or at least, meybe this could be applied only when virtio devices are present (maybe even just when transitional devices are present) ?

Thanks,
Rudolf

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, Jun 21, 2024 at 10:29:45PM +0200, Rudolf Marek wrote:
> Hi,
> 
> Dne 21. 06. 24 v 15:20 Igor Mammedov napsal(a):
> > > +            // Old linux kernels have trouble dealing with more than 44/46
> > > +            // phys-bits. Seems to be a bug in the virtio-pci driver.
> > > +            //   46:  centos-7, ubuntu-18.04
> > > +            //   44:  ubuntu-16.04
> > > +            // Limit the used address space to mitigate the bug, except we are
> > > +            // running in a guest with more than 1TB of memory installed.
> > 
> > Is it possible to fix those broken drivers (centos-7 for example)
> > and ditch this heuristic altogether?
> 
> Does this code ever runs in some baremetal use cases as well?
> 
> Or at least, meybe this could be applied only when virtio devices are present (maybe even just when transitional devices are present) ?

This code is only used on qemu (and derivatives) - specifically
CONFIG_QEMU must be true.  When running on coreboot, SeaBIOS expects
coreboot to map all the PCI devices.

Cheers,
-Kevin
_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, Jun 21, 2024 at 02:05:17PM +0200, Gerd Hoffmann wrote:

> On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> > 64-bit pci io window, failing during boot with errors like:
> 
> Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> 44 phys-bits).  Test patch below.

Thanks for the patch, I can confirm this also works with Ubuntu 14.04 (oldest we
had to hand) as well as a couple of 32-bit Windows VMs. This is a much better
fix!

> Even the LTS version from that year (16.04) is not supported any more.

Even 14.04 is not yet end of life. If you're prepared to pay, they'll still
support you. https://wiki.ubuntu.com/Releases

> Well.  Why people would use *that* ubuntu version is not clear to me.
> It's *loooooong* out of support.

You're in the IT dept of a large corporation. You have some critical
application running on some old Dell server with Ubuntu 16.04. A move to
virtualization has been mandated across the org, so you need to decommission
that server. The application was built by some contractor - before your time -
and the source code was long lost, due to a misadventure with a misconfigured
array - again before your time.

You've tried to use a newer version, but the application depends on lots of
libraries that didn't take compatibility seriously (like, say, GNOME), so it
simply can't run on newer versions. You've tried for some time to work around
this by building and installing dependencies but you're not an expert on dynamic
linkers, and could never get that last C++ symbol to resolve.

There's no funding to build a new replacement for the app. You're aware that the
OS is out of full support, so you do your best to lock down any network access
and mitigate the relevant CVEs.

Now you try to upgrade your virtualization cluster, and your VM doesn't boot any
more.

This kind of situation is very common. It's Long tail is long :(

regards
john
_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, Jun 21, 2024 at 01:37:24PM GMT, John Levon wrote:
> On Fri, Jun 21, 2024 at 02:05:17PM +0200, Gerd Hoffmann wrote:
> 
> > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> > > 64-bit pci io window, failing during boot with errors like:
> 
> > Well.  Why people would use *that* ubuntu version is not clear to me.
> > It's *loooooong* out of support.
> 
> You're in the IT dept of a large corporation. You have some critical
> application running on some old Dell server with Ubuntu 16.04.

I was specifically referring to the "16.10" listed above, i.e. running
critical stuff on non-LTS distros in the first place.

16.04-LTS is a much better choice, and that there are plenty of things
which can happen in real life which can delay or even prevent moving
applications to a newer LTS version is pretty clear.

take care,
  Gerd

_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org

On Fri, Jun 21, 2024 at 01:37:24PM +0100, John Levon wrote:
> On Fri, Jun 21, 2024 at 02:05:17PM +0200, Gerd Hoffmann wrote:
> 
> > On Wed, Jun 19, 2024 at 11:21:14AM GMT, John Levon wrote:
> > > Older 32-bit Linux VMs (including Ubuntu 16.10) have issues with the
> > > 64-bit pci io window, failing during boot with errors like:
> > 
> > Turns out it apparently can't deal with PCI bars mapped above 16TB (aka
> > 44 phys-bits).  Test patch below.
> 
> Thanks for the patch, I can confirm this also works with Ubuntu 14.04 (oldest we
> had to hand) as well as a couple of 32-bit Windows VMs. This is a much better
> fix!

Thanks for tracking this down and testing.  I agree this looks like an
improved fix.

-Kevin
_______________________________________________
SeaBIOS mailing list -- seabios@seabios.org
To unsubscribe send an email to seabios-leave@seabios.org