From nobody Sat Oct 24 20:21:02 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) client-ip=38.145.34.151; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass(p=reject dis=none) header.from=lists.libvirt.org ARC-Seal: i=1; a=rsa-sha256; t=1791174058; cv=none; d=zohomail.com; s=zohoarc; b=RHaCvi4s27abDTv/mkUVZb3V5Hh9aiFkkQlizkXxBaflk6UIf656PpW3f/HFGNzbi6qLfs0hJtiXl9W7/aetc9yWtOdVr5h1zHX/k8QykoKyY7UgFRitJAMnm/lhW9FMQKkLET84emBC04bhs8/xcJ7AB1CiCTdioosaPW2K2TQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1791174058; h=Content-Type:Content-Transfer-Encoding:Date:Date:From:From:List-Subscribe:List-Post:List-Owner:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:Subject:Subject:To:To:Message-Id:Cc; bh=wyU/HD3AO6OlrqT8kCBqI0IsnQ7XTdVvTcE0QOAsfbA=; b=DJmePfN8VWSxtj+DDG4qAyRRQB18yVaF8iK1QWQjpCSH8hDFohJUd0t9WKdwoJlH41jE3+TgYMoJVentVcDh6Qpryu05YqHM8QE9SOHkeQl9CNlOhCOhdUQpDMtSw5cCZq3u+QYKspKw3DQRUIPGlKhNG833aLDrUEgf0No08rc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [38.145.34.151]) by mx.zohomail.com with SMTPS id 1791174058477310.81223680008213; Sun, 4 Oct 2026 21:20:58 -0700 (PDT) Received: by lists.libvirt.org (Postfix, from userid 993) id BC02A417FC; Mon, 5 Oct 2026 00:20:56 -0400 (EDT) Received: from [172.19.199.16] (unknown [10.16.107.18]) by lists.libvirt.org (Postfix) with ESMTP id 263B5419E7; Mon, 5 Oct 2026 00:18:59 -0400 (EDT) Received: by lists.libvirt.org (Postfix, from userid 993) id A35DE3F30F; Mon, 5 Oct 2026 00:18:51 -0400 (EDT) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (3072 bits) server-digest SHA256) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id A97E43F30F for ; Mon, 5 Oct 2026 00:18:49 -0400 (EDT) Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-437-sMAnGJ44PaGYSIVW9NODew-1; Mon, 05 Oct 2026 00:18:47 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 10C21195604E for ; Mon, 5 Oct 2026 04:18:47 +0000 (UTC) Received: from vhost3.router.laine.org (unknown [10.22.80.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9839830000E5 for ; Mon, 5 Oct 2026 04:18:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=0.6 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, RCVD_IN_SBL_CSS,SPF_HELO_PASS autolearn=no autolearn_force=no version=4.0.1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1791173929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wyU/HD3AO6OlrqT8kCBqI0IsnQ7XTdVvTcE0QOAsfbA=; b=WlziA/ekVgfLGHHjZQvayJ2Z4VbjBZn/M/DDK+7MfMhwMubDrCPtewVYaEw7iP4e2QFT7Q 1XTxdzXtLegs/OXxqZjbxfMXF7exPmQnwwiNKHHPG0E9vX3pzQqTPdLcP/hutvzs5rPZbZ EuBD5yRoE049+fFaqy8A9NM8KWL3brw= X-MC-Unique: sMAnGJ44PaGYSIVW9NODew-1 X-Mimecast-MFC-AGG-ID: sMAnGJ44PaGYSIVW9NODew_1791173927 To: devel@lists.libvirt.org Subject: [PATCH] qemu: allow for 15 subordinate pci controllers under each pcie-expander-bus Date: Mon, 5 Oct 2026 00:18:45 -0400 Message-ID: <20261005041845.3259207-1-laine@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: J3TY4l1nYGPdrYIwYFMpKb95AHzjf8HkEZWN0iGfnQk_1791173927 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Message-ID-Hash: Z7CVDDKI5HG37H7GMZ3B2FKHJ2BI4XS4 X-Message-ID-Hash: Z7CVDDKI5HG37H7GMZ3B2FKHJ2BI4XS4 X-MailFrom: laine@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; header-match-devel.lists.libvirt.org-0; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Laine Stump via Devel Reply-To: Laine Stump X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1791174059986158500 Content-Type: text/plain; charset="utf-8"; x-default="true" From: Laine Stump A guest was created with PCIe bus and 2 PCIe devices attached to two separate pcie-root-ports that were themselves attached to a pcie-expander-bus. This guest was unable to boot from the 2nd device on the pcie-expander-bus. It turns out this failure was because not enough "bus numbers" were reserved for the pcie-expander-bus in the configuration, and this led to a failure of the system firmware to initialize those devices properly. Background: Each pci[e]-expander-bus has a busNr attribute that can be optionally specified in the subelement; it defines the bus number of the expander bus itself. Any pci controller attached to that expander bus is then assigned a bus number between busNr+1 and the busNr of the next higher expander bus (or 255, if there are no expander buses with a higher busNr. The pci-root (or pcie-root) bus is bus number 0, and any pci controller attached to that bus (or its subordinate buses *except for pci[e]-expander-buses*) will be automatically numbered starting at 1, and continuing up until the lowest numbered expander bus. When libvirt originally added support for the pci-expander-bus, each pci-expander-bus' busNr was initialized by default to be the busNr of the current lowest expander bus - 2 (reserving one for the pci-expander-bus itself, and one for the implicit pci-bridge that is always added (by qemu) along with the pci-expander-bus). When pcie-expander-bus support was added, this logic was left untouched, but that is incorrect - a hotplug-supporting PCIe topology requires that each device have its own pcie-root-port, and so adding multiple hotpluggable devices to a pcie-expander-bus uses up one bus number for each device. So, the solution to the original problem is to just lower the busNr of each pcie-expander-bus by 16 (rather than 2) from the previous. This allows connection of up to 15 pcie-root-port controllers (and thus 15 hotpluggable endpoint devices) to each pcie-expander bus with the default setting of busNr. Resolves: https://redhat.atlassian.net/browse/RHEL-144343 Signed-off-by: Laine Stump -- Note that I could have chosen 32 or 8 or any other number, and no matter which I chose it would be likely to not work for *somebody*. Anyway, if this scheme ever doesn't work out, manual intervention is always possible, either by manually setting busNr, or by putting multiple endpoint devices on different functions of the same pcie-root-port (which is completely unproblematic if the device will never be unplugged at runtime). Signed-off-by: Laine Stump --- src/qemu/qemu_domain_address.c | 85 ++++++++++++------- ...e-expander-bus-aarch64.aarch64-latest.args | 2 +- ...ie-expander-bus-aarch64.aarch64-latest.xml | 2 +- 3 files changed, 57 insertions(+), 32 deletions(-) diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index a8292a9782..81cf34fe86 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -2567,43 +2567,63 @@ qemuDomainAddressFindNewTargetIndex(virDomainDef *d= ef) =20 =20 static int -qemuDomainAddressFindNewBusNr(virDomainDef *def) +qemuDomainAddressFindNewBusNr(virDomainDef *def, + virDomainControllerModelPCI model) { /* Try to find a nice default for busNr for a new pci-expander-bus. * This is a bit tricky, since you need to satisfy the following: * - * 1) There need to be enough unused bus numbers between busNr of this - * bus and busNr of the next highest bus for the guest to assign a - * unique bus number to each PCI bus that is a child of this - * bus. Each PCI controller. On top of this, the pxb device (which - * implements the pci-expander-bus) includes a pci-bridge within - * it, and that bridge also uses one bus number (so each pxb device - * requires at least 2 bus numbers). + * 1) There need to be enough unused bus numbers between busNr of + * this bus and busNr of the next highest bus for the guest to + * assign a unique bus number to each PCI bus that is a child + * of this bus. * * 2) There need to be enough bus numbers *below* this for all the - * child controllers of the pci-expander-bus with the next lower + * child controllers of the pci[e]-expander-bus with the next lower * busNr (or the pci-root bus if there are no lower - * pci-expander-buses). + * pci[e]-expander-buses). * * 3) If at all possible, we want to avoid needing to change the busNr * of a bus in the future, as that changes the guest's device ABI, * which could potentially lead to issues with a guest OS that is * picky about such things. * - * Due to the impossibility of predicting what might be added to the - * config in the future, we can't make a foolproof choice, but since - * a pci-expander-bus (pxb) has slots for 32 devices, and the only - * practical use for it is to assign real devices on a particular - * NUMA node in the host, it's reasonably safe to assume it should - * never need any additional child buses (probably only a few of the - * 32 will ever be used). So for pci-expander-bus we find the lowest - * existing busNr, and set this one to the current lowest - 2 (one - * for the pxb, one for the integrated pci-bridge), thus leaving the - * maximum possible bus numbers available for other buses plugged - * into pci-root (i.e. pci-bridges and other - * pci-expander-buses). Anyone who needs more than 32 devices - * descended from one pci-expander-bus should set the busNr manually - * in the config. + * Due to the impossibility of predicting what might be added to + * the config in the future, we can't make a foolproof choice, + * and the parameters are different for a pci-expander-bus and a + * pcie-expander-bus. + * + * pci-expander-bus: the pxb device (which implements the + * pci-expander-bus) includes a pci-bridge within it, and that + * bridge uuses one bus number, so each pxb device requires at + * least 2 bus numbers. That bridge has slots for 32 devices, and + * its only practical use is to assign real physical devices on a + * particular NUMA node in the host. It's reasonably safe to + * assume no guest should never have more than 32 physical + * devices on a single NUMA node that need to be assigned to a + * guest (probably only a few of the 32 will ever be used), and + * so we shouldn't need more than 2 bus numbers for each + * pci-expander-bus. So we find the lowest existing busNr, and + * set busNr of the new pci-expander-bus to that value - + * 2. Anyone who needs more than 32 devices descended from one + * pci-expander-bus should set the busNr manually in the config. + * + * pcie-expander-bus: this is (as you'd expect) a more + * complicated story - if you want to support hotplugging on a + * pcie based machinetype, then each endpoint device needs its + * own pcie-root-port, and each root port uses up one bus number + * in the range available to its parent pcie-expander-bus. Taking + * a very generous example of a guest that has 8 GPUs, 8 NICs, + * and 8 NVME drives split between two NUMA nodes, giving 16 bus + * numbers to each pcie-expander-bus gives plenty of bus number + * space for all the controllers needed for each device in this + * extreme config, while still leaving plenty of overhead for 1) + * lots of devices on NUMA node 0 and 2) more than 2 NUMA nodes + * in the unlikely case that someone was forced to create a guest + * with more than 2 NUMA nodes. Again, if someone has a config + * more extreme than this, they should just set busNr manually + * (after all, the pci addresses of all these devices are already + * all being set manually, so that shouldn't be a huge burden). * * There is room for more error checking here - in particular we * can/should determine the ultimate parent (root-bus) of each PCI @@ -2614,6 +2634,10 @@ qemuDomainAddressFindNewBusNr(virDomainDef *def) =20 size_t i; int lowestBusNr =3D 256; + int maxChildControllers =3D 1; /* default for pci-expander-bus */ + + if (model =3D=3D VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS) + maxChildControllers =3D 15; =20 for (i =3D 0; i < def->ncontrollers; i++) { virDomainControllerDef *cont =3D def->controllers[i]; @@ -2626,14 +2650,15 @@ qemuDomainAddressFindNewBusNr(virDomainDef *def) } } =20 - /* If we already have a busNR =3D 1, then we can't auto-assign (0 is - * the pci[e]-root, and the others may have been assigned - * purposefully). + /* If we already have a controller with busNR <=3D + * maxChildControllers+1, then we can't auto-assign busNr to this + * new expander bus (0 is the pci[e]-root, and the others may have + * been assigned purposefully). */ - if (lowestBusNr <=3D 2) + if (lowestBusNr <=3D (maxChildControllers + 1)) return -1; =20 - return lowestBusNr - 2; + return lowestBusNr - (maxChildControllers + 1); } =20 =20 @@ -2877,7 +2902,7 @@ qemuDomainAssignPCIAddresses(virDomainDef *def, case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS: case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: if (options->busNr =3D=3D -1) - options->busNr =3D qemuDomainAddressFindNewBusNr(def); + options->busNr =3D qemuDomainAddressFindNewBusNr(def, = cont->model); if (options->busNr =3D=3D -1) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("No free busNr lower than current low= est busNr is available to auto-assign to bus %1$d. Must be manually assigne= d"), diff --git a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest= .args b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args index e76510dbbb..52011865a0 100644 --- a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args +++ b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args @@ -25,7 +25,7 @@ XDG_CONFIG_HOME=3D/var/lib/libvirt/qemu/domain--1-pcie-ex= pander-bus-te/.config \ -rtc base=3Dutc \ -no-shutdown \ -boot strict=3Don \ --device '{"driver":"pxb-pcie","bus_nr":254,"id":"pci.1","bus":"pcie.0","ad= dr":"0x1"}' \ +-device '{"driver":"pxb-pcie","bus_nr":240,"id":"pci.1","bus":"pcie.0","ad= dr":"0x1"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -sandbox on,obsolete=3Ddeny,elevateprivileges=3Ddeny,spawn=3Ddeny,resource= control=3Ddeny \ -msg timestamp=3Don diff --git a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest= .xml b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml index ce88da574f..0d0cac7653 100644 --- a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml +++ b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml @@ -20,7 +20,7 @@ - +