From nobody Mon Apr 29 11:40:28 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1517902863066493.3360654806238; Mon, 5 Feb 2018 23:41:03 -0800 (PST) Received: from localhost ([::1]:35464 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eixsA-0005CZ-9y for importer@patchew.org; Tue, 06 Feb 2018 02:40:58 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56746) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eixrB-0004gT-2q for qemu-devel@nongnu.org; Tue, 06 Feb 2018 02:39:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eixr7-000346-Uz for qemu-devel@nongnu.org; Tue, 06 Feb 2018 02:39:57 -0500 Received: from mx1.redhat.com ([209.132.183.28]:52732) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eixr7-00033c-Nc for qemu-devel@nongnu.org; Tue, 06 Feb 2018 02:39:53 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3CC1128221 for ; Tue, 6 Feb 2018 07:39:52 +0000 (UTC) Received: from xz-mi.nay.redhat.com (dhcp-14-120.nay.redhat.com [10.66.14.120]) by smtp.corp.redhat.com (Postfix) with ESMTP id B0A805C885; Tue, 6 Feb 2018 07:39:34 +0000 (UTC) From: Peter Xu To: qemu-devel@nongnu.org Date: Tue, 6 Feb 2018 15:39:33 +0800 Message-Id: <20180206073933.4071-1-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Tue, 06 Feb 2018 07:39:52 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v2] pci/bus: let it has higher migration priority X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Laurent Vivier , Juan Quintela , "Michael S . Tsirkin" , "Dr . David Alan Gilbert" , peterx@redhat.com, Alex Williamson , Marcel Apfelbaum Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" In the past, we prioritized IOMMU migration so that we have such a priority order: IOMMU > PCI Devices When migrating a guest with both vIOMMU and a pcie-root-port, we'll always migrate vIOMMU first, since pci buses will be seen to have the same priority of general PCI devices. That's problematic. The thing is that PCI bus number information is stored in the root port, and that is needed by vIOMMU during post_load(), e.g., to figure out context entry for a device. If we don't have correct bus numbers for devices, we won't be able to recover device state of the DMAR memory regions, and things will be messed up. So let's boost the PCIe root ports to be even with higher priority: PCIe Root Port > IOMMU > PCI Devices A smoke test shows that this patch fixes bug 1538953. Also, apply this rule to all the PCI bus/bridge devices: ioh3420, xio3130_downstream, xio3130_upstream, pcie_pci_bridge, pci-pci bridge, i82801b11. I noted that we set pcie_pci_bridge_dev_vmstate twice. Clean that up together. CC: Alex Williamson CC: Marcel Apfelbaum CC: Michael S. Tsirkin CC: Dr. David Alan Gilbert CC: Juan Quintela CC: Laurent Vivier Bug: https://bugzilla.redhat.com/show_bug.cgi?id=3D1538953 Reported-by: Maxime Coquelin Signed-off-by: Peter Xu Reviewed-by: Marcel Apfelbaum --- v2: - add more devices that Marcel mentioned - rename to MIG_PRI_PCI_BUS - remove one useless line in existing code --- hw/pci-bridge/gen_pcie_root_port.c | 1 + hw/pci-bridge/i82801b11.c | 1 + hw/pci-bridge/ioh3420.c | 1 + hw/pci-bridge/pci_bridge_dev.c | 1 + hw/pci-bridge/pcie_pci_bridge.c | 2 +- hw/pci-bridge/xio3130_downstream.c | 1 + hw/pci-bridge/xio3130_upstream.c | 1 + include/migration/vmstate.h | 1 + 8 files changed, 8 insertions(+), 1 deletion(-) diff --git a/hw/pci-bridge/gen_pcie_root_port.c b/hw/pci-bridge/gen_pcie_ro= ot_port.c index 0e2f2e8bf1..435fbaa60e 100644 --- a/hw/pci-bridge/gen_pcie_root_port.c +++ b/hw/pci-bridge/gen_pcie_root_port.c @@ -101,6 +101,7 @@ static void gen_rp_realize(DeviceState *dev, Error **er= rp) =20 static const VMStateDescription vmstate_rp_dev =3D { .name =3D "pcie-root-port", + .priority =3D MIG_PRI_PCI_BUS, .version_id =3D 1, .minimum_version_id =3D 1, .post_load =3D pcie_cap_slot_post_load, diff --git a/hw/pci-bridge/i82801b11.c b/hw/pci-bridge/i82801b11.c index cb522bf30c..60df9b2c96 100644 --- a/hw/pci-bridge/i82801b11.c +++ b/hw/pci-bridge/i82801b11.c @@ -80,6 +80,7 @@ err_bridge: =20 static const VMStateDescription i82801b11_bridge_dev_vmstate =3D { .name =3D "i82801b11_bridge", + .priority =3D MIG_PRI_PCI_BUS, .fields =3D (VMStateField[]) { VMSTATE_PCI_DEVICE(parent_obj, PCIBridge), VMSTATE_END_OF_LIST() diff --git a/hw/pci-bridge/ioh3420.c b/hw/pci-bridge/ioh3420.c index 5f56a2feb6..a7bfbdd238 100644 --- a/hw/pci-bridge/ioh3420.c +++ b/hw/pci-bridge/ioh3420.c @@ -83,6 +83,7 @@ static void ioh3420_interrupts_uninit(PCIDevice *d) =20 static const VMStateDescription vmstate_ioh3420 =3D { .name =3D "ioh-3240-express-root-port", + .priority =3D MIG_PRI_PCI_BUS, .version_id =3D 1, .minimum_version_id =3D 1, .post_load =3D pcie_cap_slot_post_load, diff --git a/hw/pci-bridge/pci_bridge_dev.c b/hw/pci-bridge/pci_bridge_dev.c index d56f6638c2..b2d861d216 100644 --- a/hw/pci-bridge/pci_bridge_dev.c +++ b/hw/pci-bridge/pci_bridge_dev.c @@ -174,6 +174,7 @@ static bool pci_device_shpc_present(void *opaque, int v= ersion_id) =20 static const VMStateDescription pci_bridge_dev_vmstate =3D { .name =3D "pci_bridge", + .priority =3D MIG_PRI_PCI_BUS, .fields =3D (VMStateField[]) { VMSTATE_PCI_DEVICE(parent_obj, PCIBridge), SHPC_VMSTATE(shpc, PCIDevice, pci_device_shpc_present), diff --git a/hw/pci-bridge/pcie_pci_bridge.c b/hw/pci-bridge/pcie_pci_bridg= e.c index a4d827c99d..e5ac7974cf 100644 --- a/hw/pci-bridge/pcie_pci_bridge.c +++ b/hw/pci-bridge/pcie_pci_bridge.c @@ -129,6 +129,7 @@ static Property pcie_pci_bridge_dev_properties[] =3D { =20 static const VMStateDescription pcie_pci_bridge_dev_vmstate =3D { .name =3D TYPE_PCIE_PCI_BRIDGE_DEV, + .priority =3D MIG_PRI_PCI_BUS, .fields =3D (VMStateField[]) { VMSTATE_PCI_DEVICE(parent_obj, PCIBridge), SHPC_VMSTATE(shpc, PCIDevice, NULL), @@ -178,7 +179,6 @@ static void pcie_pci_bridge_class_init(ObjectClass *kla= ss, void *data) k->config_write =3D pcie_pci_bridge_write_config; dc->vmsd =3D &pcie_pci_bridge_dev_vmstate; dc->props =3D pcie_pci_bridge_dev_properties; - dc->vmsd =3D &pcie_pci_bridge_dev_vmstate; dc->reset =3D &pcie_pci_bridge_reset; set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories); hc->plug =3D pcie_pci_bridge_hotplug_cb; diff --git a/hw/pci-bridge/xio3130_downstream.c b/hw/pci-bridge/xio3130_dow= nstream.c index 1e09d2afb7..4dd2e65118 100644 --- a/hw/pci-bridge/xio3130_downstream.c +++ b/hw/pci-bridge/xio3130_downstream.c @@ -161,6 +161,7 @@ static Property xio3130_downstream_props[] =3D { =20 static const VMStateDescription vmstate_xio3130_downstream =3D { .name =3D "xio3130-express-downstream-port", + .priority =3D MIG_PRI_PCI_BUS, .version_id =3D 1, .minimum_version_id =3D 1, .post_load =3D pcie_cap_slot_post_load, diff --git a/hw/pci-bridge/xio3130_upstream.c b/hw/pci-bridge/xio3130_upstr= eam.c index 227997ce46..c5f02a6ee8 100644 --- a/hw/pci-bridge/xio3130_upstream.c +++ b/hw/pci-bridge/xio3130_upstream.c @@ -133,6 +133,7 @@ PCIEPort *xio3130_upstream_init(PCIBus *bus, int devfn,= bool multifunction, =20 static const VMStateDescription vmstate_xio3130_upstream =3D { .name =3D "xio3130-express-upstream-port", + .priority =3D MIG_PRI_PCI_BUS, .version_id =3D 1, .minimum_version_id =3D 1, .fields =3D (VMStateField[]) { diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h index 8c3889433c..df463fd33d 100644 --- a/include/migration/vmstate.h +++ b/include/migration/vmstate.h @@ -148,6 +148,7 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT =3D 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_PCI_BUS, /* Must happen before IOMMU */ MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX, --=20 2.14.3