[SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init

Aleksandr Bezzubikov posted 2 patches 6 years, 8 months ago
Failed in applying to current master (apply log)
There is a newer version of this series
src/fw/pciinit.c   | 12 ++++++++++--
src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
src/hw/pcidevice.h |  1 +
3 files changed, 35 insertions(+), 2 deletions(-)
[SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Aleksandr Bezzubikov 6 years, 8 months ago
Now PCI bridges (and PCIE root port too) get a bus range number in system init,
basing on currently plugged devices. That's why when one wants to hotplug another bridge,
it needs his child bus, which the parent is unable to provide.
The suggested workaround is to have vendor-specific capability in RedHat generic pcie-root-port
that contains number of additional bus to reserve on BIOS PCI init.

Aleksandr Bezzubikov (2):
  pci: add support for direct usage of bdf for capability lookup
  pci: enable RedHat pci bridges to reserve more buses

 src/fw/pciinit.c   | 12 ++++++++++--
 src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
 src/hw/pcidevice.h |  1 +
 3 files changed, 35 insertions(+), 2 deletions(-)

-- 
2.7.4


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [Qemu-devel] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Konrad Rzeszutek Wilk 6 years, 8 months ago
On Wed, Jul 19, 2017 at 04:20:12PM +0300, Aleksandr Bezzubikov wrote:
> Now PCI bridges (and PCIE root port too) get a bus range number in system init,
> basing on currently plugged devices. That's why when one wants to hotplug another bridge,
> it needs his child bus, which the parent is unable to provide.

Could you explain how you trigger this?

> The suggested workaround is to have vendor-specific capability in RedHat generic pcie-root-port
> that contains number of additional bus to reserve on BIOS PCI init.

But wouldn't the proper fix be for the PCI bridge to have the subordinate
value be extended to fit more bus ranges?

> 
> Aleksandr Bezzubikov (2):
>   pci: add support for direct usage of bdf for capability lookup
>   pci: enable RedHat pci bridges to reserve more buses
> 
>  src/fw/pciinit.c   | 12 ++++++++++--
>  src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
>  src/hw/pcidevice.h |  1 +
>  3 files changed, 35 insertions(+), 2 deletions(-)
> 
> -- 
> 2.7.4
> 
> 

_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [Qemu-devel] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Alexander Bezzubikov 6 years, 8 months ago
ср, 19 июля 2017 г. в 16:57, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>:

> On Wed, Jul 19, 2017 at 04:20:12PM +0300, Aleksandr Bezzubikov wrote:
> > Now PCI bridges (and PCIE root port too) get a bus range number in
> system init,
> > basing on currently plugged devices. That's why when one wants to
> hotplug another bridge,
> > it needs his child bus, which the parent is unable to provide.
>
> Could you explain how you trigger this?


I'm trying to hot plug pcie-pci bridge into pcie root port, and Linux says
'cannot allocate bus number for device bla-bla'. This obviously does not
allow me to use the bridge at all.

>
>
> > The suggested workaround is to have vendor-specific capability in RedHat
> generic pcie-root-port
> > that contains number of additional bus to reserve on BIOS PCI init.
>
> But wouldn't the proper fix be for the PCI bridge to have the subordinate
> value be extended to fit more bus ranges?


What do you mean? This is what I'm trying to do. Do you suppose to get rid
of vendor-specific cap and use original register value instead of it?

>
> >
> > Aleksandr Bezzubikov (2):
> >   pci: add support for direct usage of bdf for capability lookup
> >   pci: enable RedHat pci bridges to reserve more buses
> >
> >  src/fw/pciinit.c   | 12 ++++++++++--
> >  src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
> >  src/hw/pcidevice.h |  1 +
> >  3 files changed, 35 insertions(+), 2 deletions(-)
> >
> > --
> > 2.7.4
> >
> >
>
-- 
Alexander Bezzubikov
_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [Qemu-devel] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Konrad Rzeszutek Wilk 6 years, 8 months ago
On Wed, Jul 19, 2017 at 05:14:41PM +0000, Alexander Bezzubikov wrote:
> ср, 19 июля 2017 г. в 16:57, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>:
> 
> > On Wed, Jul 19, 2017 at 04:20:12PM +0300, Aleksandr Bezzubikov wrote:
> > > Now PCI bridges (and PCIE root port too) get a bus range number in
> > system init,
> > > basing on currently plugged devices. That's why when one wants to
> > hotplug another bridge,
> > > it needs his child bus, which the parent is unable to provide.
> >
> > Could you explain how you trigger this?
> 
> 
> I'm trying to hot plug pcie-pci bridge into pcie root port, and Linux says
> 'cannot allocate bus number for device bla-bla'. This obviously does not
> allow me to use the bridge at all.
> 
> >
> >
> > > The suggested workaround is to have vendor-specific capability in RedHat
> > generic pcie-root-port
> > > that contains number of additional bus to reserve on BIOS PCI init.
> >
> > But wouldn't the proper fix be for the PCI bridge to have the subordinate
> > value be extended to fit more bus ranges?
> 
> 
> What do you mean? This is what I'm trying to do. Do you suppose to get rid
> of vendor-specific cap and use original register value instead of it?

I would suggest a simple fix - each bridge has a a number of bus devices
it can use. You have up to 255 - so you split the number of northbridge
numbers by the amount of NUMA nodes (if that is used) - so for example
if you have 4 NUMA nodes, each bridge would cover 63 bus numbers.

Meaning the root bridge would cover 0->63 bus, 64->128, and so on.
That gives you enough space to plug in your plugged in devices
(up to 63).

And if you need sub-briges then carve out a specific range.


> 
> >
> > >
> > > Aleksandr Bezzubikov (2):
> > >   pci: add support for direct usage of bdf for capability lookup
> > >   pci: enable RedHat pci bridges to reserve more buses
> > >
> > >  src/fw/pciinit.c   | 12 ++++++++++--
> > >  src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
> > >  src/hw/pcidevice.h |  1 +
> > >  3 files changed, 35 insertions(+), 2 deletions(-)
> > >
> > > --
> > > 2.7.4
> > >
> > >
> >
> -- 
> Alexander Bezzubikov

_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Gerd Hoffmann 6 years, 8 months ago
On Wed, 2017-07-19 at 16:20 +0300, Aleksandr Bezzubikov wrote:
> Now PCI bridges (and PCIE root port too) get a bus range number in
> system init,
> basing on currently plugged devices. That's why when one wants to
> hotplug another bridge,
> it needs his child bus, which the parent is unable to provide.
> The suggested workaround is to have vendor-specific capability in
> RedHat generic pcie-root-port
> that contains number of additional bus to reserve on BIOS PCI init.

Where is the qemu patch for this?

What about window sizes?  IIRC there was a plan to provide allocation
hints for them too ...

cheers,
  Gerd


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Marcel Apfelbaum 6 years, 8 months ago
On 20/07/2017 9:52, Gerd Hoffmann wrote:
> On Wed, 2017-07-19 at 16:20 +0300, Aleksandr Bezzubikov wrote:
>> Now PCI bridges (and PCIE root port too) get a bus range number in
>> system init,
>> basing on currently plugged devices. That's why when one wants to
>> hotplug another bridge,
>> it needs his child bus, which the parent is unable to provide.
>> The suggested workaround is to have vendor-specific capability in
>> RedHat generic pcie-root-port
>> that contains number of additional bus to reserve on BIOS PCI init.
> 

Hi Gerd,
Thanks for looking into this.

> Where is the qemu patch for this?
> 

On the way, Aleksandr is working on it, should
be ready soon.

> What about window sizes?  IIRC there was a plan to provide allocation
> hints for them too ...

Yes, is in my TODO list, however not as part of Aleksandr's series
which aims to provide PCIe-PCI bridge hotplug support.
I should mention that for this case both Windows and Linux guests
(modern OSes) trigger correctly IO/MEM resource re-balancing
process and succeeds to complete the hotplug operation.

Thanks,
Marcel

> 
> cheers,
>    Gerd
> 


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [Qemu-devel] [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Gerd Hoffmann 6 years, 8 months ago
  Hi,

> > What about window sizes?  IIRC there was a plan to provide
> > allocation
> > hints for them too ...
> 
> Yes, is in my TODO list, however not as part of Aleksandr's series
> which aims to provide PCIe-PCI bridge hotplug support.

I'd prefer to have a single vendor capability for all resource
allocation hints provided by qemu.

cheers,
  Gerd


Re: [Qemu-devel] [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Marcel Apfelbaum 6 years, 8 months ago
On 21/07/2017 13:04, Gerd Hoffmann wrote:
>    Hi,
> 
>>> What about window sizes?  IIRC there was a plan to provide
>>> allocation
>>> hints for them too ...
>>
>> Yes, is in my TODO list, however not as part of Aleksandr's series
>> which aims to provide PCIe-PCI bridge hotplug support.
> 
> I'd prefer to have a single vendor capability for all resource
> allocation hints provided by qemu.
> 

[Adding Laszlo]


Sure, the capability looking something like:

    [flags: reserve-buses|reserve-IO|reserve-MEM|...]
    [extra-buses][IO-size][MEM-size]

    if reserve-buses -> use 'extra-buses' value and so on

would be OK?

Thanks,
Marcel


> cheers,
>    Gerd
> 


Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Gerd Hoffmann 6 years, 8 months ago
On Fri, 2017-07-21 at 15:15 +0300, Marcel Apfelbaum wrote:
> On 21/07/2017 13:04, Gerd Hoffmann wrote:
> >    Hi,
> > 
> > > > What about window sizes?  IIRC there was a plan to provide
> > > > allocation
> > > > hints for them too ...
> > > 
> > > Yes, is in my TODO list, however not as part of Aleksandr's
> > > series
> > > which aims to provide PCIe-PCI bridge hotplug support.
> > 
> > I'd prefer to have a single vendor capability for all resource
> > allocation hints provided by qemu.
> > 
> 
> [Adding Laszlo]
> 
> 
> Sure, the capability looking something like:
> 
>     [flags: reserve-buses|reserve-IO|reserve-MEM|...]
>     [extra-buses][IO-size][MEM-size]
> 
>     if reserve-buses -> use 'extra-buses' value and so on

Do we need the flags?  We can use "value == 0 -> no hint for you".
Also what about prefetchable vs. non-prefetchable memory?  I guess we
want a size hint for both memory windows?

cheers,
  Gerd


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Marcel Apfelbaum 6 years, 8 months ago
On 21/07/2017 15:42, Gerd Hoffmann wrote:
> On Fri, 2017-07-21 at 15:15 +0300, Marcel Apfelbaum wrote:
>> On 21/07/2017 13:04, Gerd Hoffmann wrote:
>>>     Hi,
>>>
>>>>> What about window sizes?  IIRC there was a plan to provide
>>>>> allocation
>>>>> hints for them too ...
>>>>
>>>> Yes, is in my TODO list, however not as part of Aleksandr's
>>>> series
>>>> which aims to provide PCIe-PCI bridge hotplug support.
>>>
>>> I'd prefer to have a single vendor capability for all resource
>>> allocation hints provided by qemu.
>>>
>>
>> [Adding Laszlo]
>>
>>
>> Sure, the capability looking something like:
>>
>>      [flags: reserve-buses|reserve-IO|reserve-MEM|...]
>>      [extra-buses][IO-size][MEM-size]
>>
>>      if reserve-buses -> use 'extra-buses' value and so on
> 
> Do we need the flags?  We can use "value == 0 -> no hint for you".

Maybe we don't want to allocate IO at all,
So value 0 would say do not allocate.

But we can disable IO by other means.

> Also what about prefetchable vs. non-prefetchable memory?  I guess we
> want a size hint for both memory windows?


Good point, thanks!
Marcel

> 
> cheers,
>    Gerd
> 


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [Qemu-devel] [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Gerd Hoffmann 6 years, 8 months ago
  Hi,

> Maybe we don't want to allocate IO at all,
> So value 0 would say do not allocate.
> 
> But we can disable IO by other means.

Indeed.  As IO is optional for pcie we don't need a qemu-specific
extension for that.

cheers,
  Gerd


Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Kevin O'Connor 6 years, 8 months ago
On Fri, Jul 21, 2017 at 03:15:46PM +0300, Marcel Apfelbaum wrote:
> On 21/07/2017 13:04, Gerd Hoffmann wrote:
> > I'd prefer to have a single vendor capability for all resource
> > allocation hints provided by qemu.
> Sure, the capability looking something like:
> 
>    [flags: reserve-buses|reserve-IO|reserve-MEM|...]
>    [extra-buses][IO-size][MEM-size]
> 
>    if reserve-buses -> use 'extra-buses' value and so on

I don't have any objection to using a PCI capability, but I do wonder
if fw_cfg would be a better fit.  This information is purely qemu ->
firmware, right?

-Kevin

_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Marcel Apfelbaum 6 years, 8 months ago
On 21/07/2017 20:28, Kevin O'Connor wrote:
> On Fri, Jul 21, 2017 at 03:15:46PM +0300, Marcel Apfelbaum wrote:
>> On 21/07/2017 13:04, Gerd Hoffmann wrote:
>>> I'd prefer to have a single vendor capability for all resource
>>> allocation hints provided by qemu.
>> Sure, the capability looking something like:
>>
>>     [flags: reserve-buses|reserve-IO|reserve-MEM|...]
>>     [extra-buses][IO-size][MEM-size]
>>
>>     if reserve-buses -> use 'extra-buses' value and so on
> 
> I don't have any objection to using a PCI capability, but I do wonder
> if fw_cfg would be a better fit.  This information is purely qemu ->
> firmware, right?
> 

Hi Kevin,

Right, but theoretically speaking a guest OS driver could also
get the hint on hotplug, or simply because the OS chooses to re-assign
resources on its own.

A while ago we discussed the fw_cfg option, but Gerd preferred
the vendor capability, and since the capability looked cleaner
Aleksandr opted for it.

He will send V2 soon together with the QEMU counterpart feature.

Thanks,
Marcel

> -Kevin
> 


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios
Re: [SeaBIOS] [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init
Posted by Gerd Hoffmann 6 years, 8 months ago
On Fri, 2017-07-21 at 13:28 -0400, Kevin O'Connor wrote:
> On Fri, Jul 21, 2017 at 03:15:46PM +0300, Marcel Apfelbaum wrote:
> > On 21/07/2017 13:04, Gerd Hoffmann wrote:
> > > I'd prefer to have a single vendor capability for all resource
> > > allocation hints provided by qemu.
> > 
> > Sure, the capability looking something like:
> > 
> >    [flags: reserve-buses|reserve-IO|reserve-MEM|...]
> >    [extra-buses][IO-size][MEM-size]
> > 
> >    if reserve-buses -> use 'extra-buses' value and so on
> 
> I don't have any objection to using a PCI capability, but I do wonder
> if fw_cfg would be a better fit.  This information is purely qemu ->
> firmware, right?

fw_cfg quickly becomes messy if you want allow different hints per
bridge.

cheers,
  Gerd


_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
https://mail.coreboot.org/mailman/listinfo/seabios