From nobody Sun May 5 14:54:28 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) client-ip=207.211.31.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586886806; cv=none; d=zohomail.com; s=zohoarc; b=YOubUoHqVS171A73JVv4MW4z/+xAnfo7Ji3DvMJ8hlyJ9O+RISAq4pOoEmoqWL0TFgBHBuYRni9bLuaFax0RKKuTLcvRFdRaU8LlVISDqt0Ty05lyCANVyz4hibUeAbQPT/IACr7aLXNWlEOYRsvc0Ee9/Wqtzn2YkL94MxjKPU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586886806; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=gGk4vdU1bVVYTTsMFq6wJELJbKgNceqWiLzCCNgJjEc=; b=QAD76TI32fPpwj9F6msmejJrh10SWhRAEZoZPXBrHnPXE5v8FC4zZpC4H4n7gXxQI74fkn3AvNchJK2SxjC0hp0wX2FMw2riucm0v/rY0WeKDQ2i13+MqBATOBu7Zo4zUyulJ71FUdtqlqfQRbUTTKPzgejUhL+I+LLDcbTQPXs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) by mx.zohomail.com with SMTPS id 158688680666845.13270840015616; Tue, 14 Apr 2020 10:53:26 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-137-UM_XQAyPOz6S0oHxvfcxLw-1; Tue, 14 Apr 2020 13:53:23 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BADE71137844; Tue, 14 Apr 2020 17:53:17 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1FF905C1D6; Tue, 14 Apr 2020 17:53:17 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 8179918089C8; Tue, 14 Apr 2020 17:53:14 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 03EHrDTC032346 for ; Tue, 14 Apr 2020 13:53:13 -0400 Received: by smtp.corp.redhat.com (Postfix) id 2EC2C9F989; Tue, 14 Apr 2020 17:53:13 +0000 (UTC) Received: from kinshicho.usersys.redhat.com (unknown [10.40.194.64]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DBA999DD93 for ; Tue, 14 Apr 2020 17:53:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586886805; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=gGk4vdU1bVVYTTsMFq6wJELJbKgNceqWiLzCCNgJjEc=; b=WNmpB3v5fiUTReLln8lcUFJnO5McnlMI3shf9ero9oQonJobUGY+TLmP+X+w+fRhiVSPFq JIKYAO7Vt6cPUsjFAzjRCJUJyQ5d57Kt5PiklRvpiqViJjuqjH1pA7iucyy6vnwQfFL5Ag z/Db5TcOYuxhx925qzbaRd0/hu8gbmc= X-MC-Unique: UM_XQAyPOz6S0oHxvfcxLw-1 From: Andrea Bolognani To: libvir-list@redhat.com Subject: [libvirt PATCH] docs: Add pci-addresses.rst Date: Tue, 14 Apr 2020 19:53:05 +0200 Message-Id: <20200414175305.348601-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This document describes the relationship between PCI addresses as seen in the domain XML and by the guest OS, which is a topic that people get confused by time and time again. Signed-off-by: Andrea Bolognani Reviewed-by: Laine Stump --- docs/formatdomain.html.in | 6 +- docs/pci-addresses.rst | 184 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 189 insertions(+), 1 deletion(-) create mode 100644 docs/pci-addresses.rst diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 6f43976815..0077666862 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -4286,7 +4286,11 @@ element with no other attributes as an explicit request to assign a PCI address for the device rather than some other type of address that may also be appropriate for that same - device (e.g. virtio-mmio). + device (e.g. virtio-mmio).
+ The relationship between the PCI addresses configured in the domain + XML and those seen by the guest OS can sometime seem confusing: a + separate document describes how PCI + addresses work in more detail.
drive
Drive addresses have the following additional diff --git a/docs/pci-addresses.rst b/docs/pci-addresses.rst new file mode 100644 index 0000000000..96c6466899 --- /dev/null +++ b/docs/pci-addresses.rst @@ -0,0 +1,184 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +PCI addresses in domain XML and guest OS +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. contents:: + +When discussing PCI addresses, it's important to understand the the +relationship between the addresses that can be seen in the domain XML +and those that are visible inside the guest OS. + + +Simple cases +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +When the PCI topology of the VM is very simple, the PCI addresses +will usually match. + +For example, the domain XML snippet + +:: + + + + + +
+ + + + +
+ + +will result in the PCI topology + +:: + + 0000:00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DR= AM Controller + 0000:00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port + 0000:01:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (r= ev 01) + +showing up in the guest OS. + +The PCI address of the ``virtio-net`` adapter, ``0000:01:00.0``, is +the same in both cases, so there's no confusion. + + +More complex cases +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +In more complex cases, the PCI address visible in the domain XML will +correlate to the one seen by the guest OS in a less obvious way. + +pcie-expander-bus +----------------- + +This fairly uncommon device, which can be used with ``x86_64/q35`` +guests, will help illustrate one such scenario. + +For example, the domain XML snippet + +:: + + + + + +
+ + + + +
+ + + + +
+ + +will result in the PCI topology + +:: + + 0000:00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DR= AM Controller + 0000:00:01.0 Host bridge: Red Hat, Inc. QEMU PCIe Expander bridge + 0000:fe:00.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port + 0000:ff:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (r= ev 01) + +showing up in the guest OS. + +This time the addresses don't match: this is because the ``busNr`` +property for the ``pcie-expander-bus`` controller causes it to show +up as bus 254 (``0xfe`` in hexadecimal) instead of bus 1 as one might +expect based on its ``index`` property. + +How can the domain XML shown above work at all, then? Surely the +``pcie-root-port`` controller and the ``virtio-net`` adapter should +use ``bus=3D0xfe`` and ``bus=3D0xff`` respectively for the configuration +to be accepted by libvirt? + +As it turns out, that's not the case. The reason for this is that +QEMU, and consequently libvirt, uses the ``bus`` property of a +device's PCI address only to match it with the PCI controller that +has the same ``index`` property, and not to set the actual PCI +address, which is decided by the guest OS. + +So, by looking at the XML snippet above, we can see that the +``virtio-net`` adapter plugs into the ``pcie-root-port`` controller, +which plugs into the ``pcie-expander-bus`` controller, which plugs +into ``pcie-root``: the guest OS sees the same topology, but assigns +different PCI addresses to some of its component. + +The takeaway is that the *relationship* between controllers are the +very same whether you look at the domain XML or at the guest OS, but +the *actual PCI addresses* are not guaranteed to match and in fact, +except for the very simplest cases, they usually will not. + +spapr-pci-host-bridge +--------------------- + +This device, which is unique to ``ppc64/pseries`` guests, will help +illustrate another scenario. + +For example, the domain XML snippet + +:: + + + + + + + + + + + + +
+ + +will result in the PCI topology + +:: + + 0001:00:01.0 Ethernet controller: Red Hat, Inc. Virtio network device + +showing up in the guest OS. Note that the two +``spapr-pci-host-bridge`` controllers are not listed. + +This time, in addition to the bus not matching just like in the +previous example, the interesting part is that the domain doesn't +match either: this is because each ``spapr-pci-host-bridge`` +controller creates a separate PCI domain. + +Once again, while the PCI addresses seen in the domain XML and those +seen by the guest OS do not match, the relationships between the +various devices are preserved. + + +Device assignment +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +When using VFIO to assign host devices to a guest, an additional +caveat to keep in mind that the guest OS will base its decisions upon +the *target address* rather than the *source address*. + +For example, the domain XML snippet + +:: + + + + +
+ +
+ + +will result in the device showing up as ``0000:00:01.0`` in the +guest OS rather than as ``0001:08:00.1``. + +Of course, all the rules and behaviors described above still apply. --=20 2.25.2