From: Laine Stump <laine@redhat.com>

A guest was created with PCIe bus and 2 PCIe devices attached to two
separate pcie-root-ports that were themselves attached to a
pcie-expander-bus. This guest was unable to boot from the 2nd device
on the pcie-expander-bus. It turns out this failure was because not
enough "bus numbers" were reserved for the pcie-expander-bus in the
configuration, and this led to a failure of the system firmware to
initialize those devices properly.

Background: Each pci[e]-expander-bus has a busNr attribute that can be
optionally specified in the <target> subelement; it defines the bus
number of the expander bus itself. Any pci controller attached to that
expander bus is then assigned a bus number between busNr+1 and the
busNr of the next higher expander bus (or 255, if there are no
expander buses with a higher busNr. The pci-root (or pcie-root) bus is
bus number 0, and any pci controller attached to that bus (or its
subordinate buses *except for pci[e]-expander-buses*) will be
automatically numbered starting at 1, and continuing up until the
lowest numbered expander bus.

When libvirt originally added support for the pci-expander-bus, each
pci-expander-bus' busNr was initialized by default to be the busNr of
the current lowest expander bus - 2 (reserving one for the
pci-expander-bus itself, and one for the implicit pci-bridge that is
always added (by qemu) along with the pci-expander-bus). When
pcie-expander-bus support was added, this logic was left untouched,
but that is incorrect - a hotplug-supporting PCIe topology requires
that each device have its own pcie-root-port, and so adding multiple
hotpluggable devices to a pcie-expander-bus uses up one bus number for
each device.

So, the solution to the original problem is to just lower the busNr of
each pcie-expander-bus by 16 (rather than 2) from the previous. This
allows connection of up to 15 pcie-root-port controllers (and thus 15
hotpluggable endpoint devices) to each pcie-expander bus with the
default setting of busNr.

Resolves: https://redhat.atlassian.net/browse/RHEL-144343
Signed-off-by: Laine Stump <laine@redhat.com>

--

Note that I could have chosen 32 or 8 or any other number, and no
matter which I chose it would be likely to not work for
*somebody*. Anyway, if this scheme ever doesn't work out, manual
intervention is always possible, either by manually setting busNr, or
by putting multiple endpoint devices on different functions of the
same pcie-root-port (which is completely unproblematic if the device
will never be unplugged at runtime).

Signed-off-by: Laine Stump <laine@redhat.com>
---
 src/qemu/qemu_domain_address.c                | 85 ++++++++++++-------
 ...e-expander-bus-aarch64.aarch64-latest.args |  2 +-
 ...ie-expander-bus-aarch64.aarch64-latest.xml |  2 +-
 3 files changed, 57 insertions(+), 32 deletions(-)

diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c
index a8292a9782..81cf34fe86 100644
--- a/src/qemu/qemu_domain_address.c
+++ b/src/qemu/qemu_domain_address.c
@@ -2567,43 +2567,63 @@ qemuDomainAddressFindNewTargetIndex(virDomainDef *def)
 
 
 static int
-qemuDomainAddressFindNewBusNr(virDomainDef *def)
+qemuDomainAddressFindNewBusNr(virDomainDef *def,
+                              virDomainControllerModelPCI model)
 {
     /* Try to find a nice default for busNr for a new pci-expander-bus.
      * This is a bit tricky, since you need to satisfy the following:
      *
-     * 1) There need to be enough unused bus numbers between busNr of this
-     *    bus and busNr of the next highest bus for the guest to assign a
-     *    unique bus number to each PCI bus that is a child of this
-     *    bus. Each PCI controller. On top of this, the pxb device (which
-     *    implements the pci-expander-bus) includes a pci-bridge within
-     *    it, and that bridge also uses one bus number (so each pxb device
-     *    requires at least 2 bus numbers).
+     * 1) There need to be enough unused bus numbers between busNr of
+     *    this bus and busNr of the next highest bus for the guest to
+     *    assign a unique bus number to each PCI bus that is a child
+     *    of this bus.
      *
      * 2) There need to be enough bus numbers *below* this for all the
-     *    child controllers of the pci-expander-bus with the next lower
+     *    child controllers of the pci[e]-expander-bus with the next lower
      *    busNr (or the pci-root bus if there are no lower
-     *    pci-expander-buses).
+     *    pci[e]-expander-buses).
      *
      * 3) If at all possible, we want to avoid needing to change the busNr
      *    of a bus in the future, as that changes the guest's device ABI,
      *    which could potentially lead to issues with a guest OS that is
      *    picky about such things.
      *
-     *  Due to the impossibility of predicting what might be added to the
-     *  config in the future, we can't make a foolproof choice, but since
-     *  a pci-expander-bus (pxb) has slots for 32 devices, and the only
-     *  practical use for it is to assign real devices on a particular
-     *  NUMA node in the host, it's reasonably safe to assume it should
-     *  never need any additional child buses (probably only a few of the
-     *  32 will ever be used). So for pci-expander-bus we find the lowest
-     *  existing busNr, and set this one to the current lowest - 2 (one
-     *  for the pxb, one for the integrated pci-bridge), thus leaving the
-     *  maximum possible bus numbers available for other buses plugged
-     *  into pci-root (i.e. pci-bridges and other
-     *  pci-expander-buses). Anyone who needs more than 32 devices
-     *  descended from one pci-expander-bus should set the busNr manually
-     *  in the config.
+     *  Due to the impossibility of predicting what might be added to
+     *  the config in the future, we can't make a foolproof choice,
+     *  and the parameters are different for a pci-expander-bus and a
+     *  pcie-expander-bus.
+     *
+     *  pci-expander-bus: the pxb device (which implements the
+     *  pci-expander-bus) includes a pci-bridge within it, and that
+     *  bridge uuses one bus number, so each pxb device requires at
+     *  least 2 bus numbers. That bridge has slots for 32 devices, and
+     *  its only practical use is to assign real physical devices on a
+     *  particular NUMA node in the host. It's reasonably safe to
+     *  assume no guest should never have more than 32 physical
+     *  devices on a single NUMA node that need to be assigned to a
+     *  guest (probably only a few of the 32 will ever be used), and
+     *  so we shouldn't need more than 2 bus numbers for each
+     *  pci-expander-bus. So we find the lowest existing busNr, and
+     *  set busNr of the new pci-expander-bus to that value -
+     *  2. Anyone who needs more than 32 devices descended from one
+     *  pci-expander-bus should set the busNr manually in the config.
+     *
+     *  pcie-expander-bus: this is (as you'd expect) a more
+     *  complicated story - if you want to support hotplugging on a
+     *  pcie based machinetype, then each endpoint device needs its
+     *  own pcie-root-port, and each root port uses up one bus number
+     *  in the range available to its parent pcie-expander-bus. Taking
+     *  a very generous example of a guest that has 8 GPUs, 8 NICs,
+     *  and 8 NVME drives split between two NUMA nodes, giving 16 bus
+     *  numbers to each pcie-expander-bus gives plenty of bus number
+     *  space for all the controllers needed for each device in this
+     *  extreme config, while still leaving plenty of overhead for 1)
+     *  lots of devices on NUMA node 0 and 2) more than 2 NUMA nodes
+     *  in the unlikely case that someone was forced to create a guest
+     *  with more than 2 NUMA nodes. Again, if someone has a config
+     *  more extreme than this, they should just set busNr manually
+     *  (after all, the pci addresses of all these devices are already
+     *  all being set manually, so that shouldn't be a huge burden).
      *
      *  There is room for more error checking here - in particular we
      *  can/should determine the ultimate parent (root-bus) of each PCI
@@ -2614,6 +2634,10 @@ qemuDomainAddressFindNewBusNr(virDomainDef *def)
 
     size_t i;
     int lowestBusNr = 256;
+    int maxChildControllers = 1; /* default for pci-expander-bus */
+
+    if (model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS)
+        maxChildControllers = 15;
 
     for (i = 0; i < def->ncontrollers; i++) {
         virDomainControllerDef *cont = def->controllers[i];
@@ -2626,14 +2650,15 @@ qemuDomainAddressFindNewBusNr(virDomainDef *def)
         }
     }
 
-    /* If we already have a busNR = 1, then we can't auto-assign (0 is
-     * the pci[e]-root, and the others may have been assigned
-     * purposefully).
+    /* If we already have a controller with busNR <=
+     * maxChildControllers+1, then we can't auto-assign busNr to this
+     * new expander bus (0 is the pci[e]-root, and the others may have
+     * been assigned purposefully).
      */
-    if (lowestBusNr <= 2)
+    if (lowestBusNr <= (maxChildControllers + 1))
         return -1;
 
-    return lowestBusNr - 2;
+    return lowestBusNr - (maxChildControllers + 1);
 }
 
 
@@ -2877,7 +2902,7 @@ qemuDomainAssignPCIAddresses(virDomainDef *def,
             case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS:
             case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS:
                 if (options->busNr == -1)
-                    options->busNr = qemuDomainAddressFindNewBusNr(def);
+                    options->busNr = qemuDomainAddressFindNewBusNr(def, cont->model);
                 if (options->busNr == -1) {
                     virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
                                    _("No free busNr lower than current lowest busNr is available to auto-assign to bus %1$d. Must be manually assigned"),
diff --git a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args
index e76510dbbb..52011865a0 100644
--- a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args
+++ b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args
@@ -25,7 +25,7 @@ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-pcie-expander-bus-te/.config \
 -rtc base=utc \
 -no-shutdown \
 -boot strict=on \
--device '{"driver":"pxb-pcie","bus_nr":254,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \
+-device '{"driver":"pxb-pcie","bus_nr":240,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \
 -audiodev '{"id":"audio1","driver":"none"}' \
 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
 -msg timestamp=on
diff --git a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml
index ce88da574f..0d0cac7653 100644
--- a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml
+++ b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml
@@ -20,7 +20,7 @@
     <controller type='pci' index='0' model='pcie-root'/>
     <controller type='pci' index='1' model='pcie-expander-bus'>
       <model name='pxb-pcie'/>
-      <target busNr='254'/>
+      <target busNr='240'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
     </controller>
     <audio id='1' type='none'/>
-- 
2.55.0

On 05/10/2026 05:18, Laine Stump via Devel wrote:
>
>        * 3) If at all possible, we want to avoid needing to change the busNr
>        *    of a bus in the future, as that changes the guest's device ABI,
>        *    which could potentially lead to issues with a guest OS that is
>        *    picky about such things.

This change violates this constraint.

As a non-backward compatible change to the guest ABI I don't think you 
can change the default bus allocation method.

You may find it more useful for your management plane to evenly 
distribute bus numbers for the (presumably) per-vNUMA node pcie-expander 
buses and explicitly specify the busNr for each.

David

On 10/5/26 5:40 AM, David Vrabel wrote:
> 
> 
> On 05/10/2026 05:18, Laine Stump via Devel wrote:
>>
>>        * 3) If at all possible, we want to avoid needing to change the 
>> busNr
>>        *    of a bus in the future, as that changes the guest's device 
>> ABI,
>>        *    which could potentially lead to issues with a guest OS 
>> that is
>>        *    picky about such things.

(Side note: I was the author of the above comment, made 10 years ago, 
and I was mainly concerned about runtime changes in ABI, for example 
during a migration from one host to another where the guest OS doesn't 
reboot. That would never happen in this case, since the busNr is stored 
with other attributes of the controller device whether it was set 
explicitly by the user or auto-set to a default value by libvirt when 
the domain was defined (or when the controller was added, if that 
happened later). A change in the default busNr would only have an effect 
on a newly defined domain (or newly added pcie-expander-bus).

> This change violates this constraint.

When a new domain is created and no busNr is given, the default value 
decided on by libvirt at that time is written into the XML, so 
subsequent starts of that particular domain using that definition will 
use the same busNr - only newly created domains will get a different 
default busNr than they used to. So in that case, this change in default 
setting has 0 effect.

As for migration (and domain save/restore) the change here will never 
have any effect on runtime ABI at all, since whatever the setting of 
busNr (whether it was the default value set by libvirt, or explicitly 
set by the user in the XML config, when the domain was defined) that 
will be written in the XML that is sent along with the migration data 
and so the destination end of the migration won't ever be using just 
"the current default busNr setting"; it will always be using "the exact 
same busNr that was used when the guest was started".

The place where this change *would* have any effect is when the 
management software doesn't re-use the defined domain for subsequent 
restarts, but instead just redefines the domain from its original 
internal format each time it is started. However, note that even in that 
case the change in busNr will only happen when the guest is completely 
stopped, shutdown, and booted from scratch (new QEMU process and all). 
So there will be no case of a running OS having the hardware "change out 
from under it", there will only be cases where the OS is shut down, and 
then when it is booted the next time the busNr will be different. Also 
note that a similar "change in ABI" can happen if, for example, a new 
device is added to the "raw" XML config of the management software 
(which is missing PCI addresses), and then that XML is fed back into 
libvirt - since libvirt internally determines the ordering of the 
devices (and thus which device is assigned to which PCI address) you can 
easily end up with, e.g. a network card suddenly appearing at a 
different PCI address after adding a disk device to the config. If that 
(changing the PCI address of an endpoint device between OS boots) is 
acceptable behavior, then I'd say changing the bus number of a PCI 
controller between guest OS boots is also acceptable.

(another example of guest ABI quietly changing between reboots is when a 
domain is defined using a machinetype of simply "q35". That generic name 
will be canonicalized into, e.g. pc-q35-10.1 and the guest OS will be 
booted with that exact machinetype and any migration will convey that 
exact machinetype to the destination of the migration. But when the 
guest is stopped, if the management software re-defines the  domain from 
scratch (rather than continuing to use the same domain definition) still 
using "q35", then the next time it is booted it might come up with a 
machinetype of pc-q35-11.2 (if the host's QEMU package has been updated).)


> 
> As a non-backward compatible change to the guest ABI I don't think you 
> can change the default bus allocation method.

While I agree that the busNr shouldn't be changed during the lifetime of 
a guest OS (e.g. during migration), this change in default at domain 
definition time doesn't lead to a busNr changing during migration. If 
you are aware of any guest OS that cares whether such a change occurs 
between reboots, then that's very interesting and is something we need 
to consider.

> You may find it more useful for your management plane to evenly 
> distribute bus numbers for the (presumably) per-vNUMA node pcie-expander 
> buses and explicitly specify the busNr for each.

If someone wants to explicitly specify the busNr they're free to do that 
(as suggested in the comments) just as they are free to manually specify 
the PCI address of each device. This is just setting a default value at 
domain definition time that is more likely to be usable in the general 
case than the original default; the current default has limited value, 
since it doesn't work if you have more than a single hotpluggable 
endpoint device on one pcie-expander-bus anyway. In any case libvirt has 
always saved the auto-assigned default value of busNr (and other obscure 
things normally auto-set by libvirt, like chassis number) in the domain 
config so that they would be maintained during migration, or at a 
subsequent start of the domain using the same domain definition.

(Again, if you're aware of an example of busNr changing from one boot to 
the next of a guest OS, then that would require more caution. Or maybe 
I'm being too laissez faire about it, but my current opinion is that I'm 
not :-)).

On 05/10/2026 20:19, Laine Stump wrote:
> 
> When a new domain is created and no busNr is given, the default value 
> decided on by libvirt at that time is written into the XML, so 
> subsequent starts of that particular domain using that definition will 
> use the same busNr - only newly created domains will get a different 
> default busNr than they used to. So in that case, this change in default 
> setting has 0 effect.

There are management stacks that re-create the XML for every power on, 
so every power cycle these VMs would see different SBDFs across the 
libvirt upgrade and some VMs (e.g., Windows) get upset if SBDFs change 
(of which I'm sure you're aware).

If you think management stacks that do that should also take care of the 
busNr allocations then I personally find that acceptable (as the 
management stack I care about already does this), but I'm not sure I'd 
want to make assumptions about the behaviour of every one.

So whilst I agree that the current bus number allocation algorithm isn't 
very useful, I do think a new algorithm needs to be explicitly requested 
by the user or management stack.

You could have something like:

<controller type='pci' index='1' model='pcie-expander-bus'>
   <model name='pxb-pcie'/>
   <target busNr='+16'/>
   ...

Where the +16 means reserve 16 "spare" bus numbers for downstream 
devices/ports, without having to care about exactly which bus numbers 
are allocated. The default value is the current behaviour of "+2".

If you wish to continue with the patch as-is, I think the description 
could benefit from a clearer description of the current allocation 
algorithm (e.g., clarify that numbering starts from the highest bus and 
goes down).

David

tl;dr - you are correct and I don't know where my head was. :-)

On 10/6/26 12:27 PM, David Vrabel wrote:
> On 05/10/2026 20:19, Laine Stump wrote:
>>
>> When a new domain is created and no busNr is given, the default value 
>> decided on by libvirt at that time is written into the XML, so 
>> subsequent starts of that particular domain using that definition will 
>> use the same busNr - only newly created domains will get a different 
>> default busNr than they used to. So in that case, this change in 
>> default setting has 0 effect.
> 
> There are management stacks that re-create the XML for every power on, 

Yeah, I hadn't forgotten that, and was even considering those systems, 
but...

> so every power cycle these VMs would see different SBDFs across the 
> libvirt upgrade

.. my mind was somehow blocking the fact that the attribute we call 
busNr directly determines the *actual* bus number of the controller and 
subordinate devices[*]. I mean, it completely makes sense that's what it 
would be used for (it's right there in the name!), but for some reason 
as I was looking at it, my brain was thinking of it more like the 
"chassis" attribute, which (as far as I've been told by lower level 
guys) is visible to the guest in a register somewhere, but not used to 
enumerate (and sometimes name) devices the way that SBDF is (it was you 
saying "SBDF" that finally jogged my memory).

[*] as opposed to the "bus" attribute of the <address type='pci'/> 
element in libvirt, which does get passed directly to QEMU, but is 
really just an index in a table somewhere in QEMU (I'm assuming), and 
not necessarily related to the bus number that is visible to the guest 
OS - we've spent a lot of time trying to explain that over the years to 
users who expect that when they set "bus='5'" in the XML, the device 
will always show up in the guest on bus 5, even if they have no bus 4 
(for example))

> and some VMs (e.g., Windows) get upset if SBDFs change 
> (of which I'm sure you're aware).

Painfully! I don't know if it's still the case, but way back when 
libvirt began explicitly assigning pci addresses to devices and saving 
them in the domain XML, I think one of the reasons was that Windows 
could invalidate the guest OS's license key activation if certain/too 
many devices' pci addresses changed, and also would pop up a window 
about a "new network connection" if the pci address of a network device 
changed (and if you think *that's* annoying, it also pops up that dialog 
if the MAC address of the next hop default route changes :-/).

And you can't even point the finger only at Windows - in modern Linux 
with "predictable" network device naming, the name of a PCI network 
device will change when its PCI address changes, thus invalidating any 
config based on the device name (which I should have noticed when 
testing to verify this patch fixed the originally reported problem, but 
was so happy it booted Fedora from the pcie-expander-bus connected disk 
image that I only looked to see if the pcie-expander-bus connected NIC 
had gotten a an IP from DHCP and was able to communicate).

> If you think management stacks that do that should also take care of the 
> busNr allocations then I personally find that acceptable (as the 
> management stack I care about already does this),

It's a bit of a digression, but really I think the management layer 
shouldn't need to worry about details like PCI addresses at all - they 
should be able to just say "make this device hotpluggable, put this 
device on NUMA node 1 and not hotpluggable" etc, and [someone else] 
would figure out the details of PCI topology for them. That's a bit 
outside libvirt's scope though (even though libvirt does automatically 
assign PCI addresses for all PCI devices, that was originally done as a 
defense mechanism to assure that PCI addresses remained stable as 
devices were added and removed, and just encoded what addresses had been 
previously auto-assigned by QEMU on startup, and grew organically from 
there).

> but I'm not sure I'd 
> want to make assumptions about the behaviour of every one.
> 
> So whilst I agree that the current bus number allocation algorithm isn't 
> very useful, I do think a new algorithm needs to be explicitly requested 
> by the user or management stack.

Yeah, after you've jogged my memory about what a change in busNr really 
leads to, I agree, and am dropping this patch.

Thanks for bringing me back to reality :-)