From nobody Mon Apr 29 06:20:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542746681681191.17195100856213; Tue, 20 Nov 2018 12:44:41 -0800 (PST) Received: from localhost ([::1]:35825 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCst-0008HL-I4 for importer@patchew.org; Tue, 20 Nov 2018 15:44:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52452) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqE-0005r1-RX for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCq9-0003L0-3o for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:50 -0500 Received: from hqemgate16.nvidia.com ([216.228.121.65]:10891) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCq7-0003ID-8O for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:45 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:49 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:40 -0800 Received: from HQMAIL102.nvidia.com (172.18.146.10) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:39 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL102.nvidia.com (172.18.146.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:39 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:32 +0000 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 20 Nov 2018 12:41:40 -0800 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:39 +0530 Message-ID: <1542746383-18288-2-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746509; bh=+760c3xMZ5cmBUggliF6j0hmN+CTRFRqNQY7cEt2zaE=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=Bmv/ULA8Ou5mBSKJmWzqIYZHTo45aNbwoZyVlWMrW1WZYmI9dVIxNX1SxdydUtEbb e9O2gLsMQbHBe4GX1K/JkEVaQIQKwSlIiVPcbdgrwgMTIPmJ1PxEz//Lf48CV9h8ci PnAYEATF/QsRubBkm3iT3Jy1YD8aZwsmNbZHxElkmo7N7TAMwYWAVLgILb3KJgsrqA qO+0BzvgR8g5eEN2XosEKdM8hCkjGmT1Gf15sl/J4BEYJnCLZ0j+bFe912UsBSR7MS tvEI2Z1Ht5vUCqfbzFLerfctZZ5YRmUYPAvO6hkZuO79F4CBS7lRdijdCAEUiT0iJy yf2G0Rf4JopZw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH 1/5] VFIO KABI for migration interface X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" - Defined MIGRATION region type and sub-type. - Defined VFIO device states during migration process. - Defined vfio_device_migration_info structure which will be placed at 0th offset of migration region to get/set VFIO device related information. Defined actions and members of structure usage for each action: * To convey VFIO device state to be transitioned to. * To get pending bytes yet to be migrated for VFIO device * To ask driver to write data to migration region and return number of = bytes written in the region * In migration resume path, user space app writes to migration region a= nd communicates it to vendor driver. * Get bitmap of dirty pages from vendor driver from given start address Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- linux-headers/linux/vfio.h | 130 +++++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 130 insertions(+) diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index 3615a269d378..a6e45cb2cae2 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -301,6 +301,10 @@ struct vfio_region_info_cap_type { #define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2) #define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) =20 +/* Migration region type and sub-type */ +#define VFIO_REGION_TYPE_MIGRATION (1 << 30) +#define VFIO_REGION_SUBTYPE_MIGRATION (1) + /* * The MSIX mappable capability informs that MSIX data of a BAR can be mma= pped * which allows direct access to non-MSIX registers which happened to be w= ithin @@ -602,6 +606,132 @@ struct vfio_device_ioeventfd { =20 #define VFIO_DEVICE_IOEVENTFD _IO(VFIO_TYPE, VFIO_BASE + 16) =20 +/** + * VFIO device states : + * VFIO User space application should set the device state to indicate ven= dor + * driver in which state the VFIO device should transitioned. + * - VFIO_DEVICE_STATE_NONE: + * State when VFIO device is initialized but not yet running. + * - VFIO_DEVICE_STATE_RUNNING: + * Transition VFIO device in running state, that is, user space applicat= ion or + * VM is active. + * - VFIO_DEVICE_STATE_MIGRATION_SETUP: + * Transition VFIO device in migration setup state. This is used to prep= are + * VFIO device for migration while application or VM and vCPUs are still= in + * running state. + * - VFIO_DEVICE_STATE_MIGRATION_PRECOPY: + * When VFIO user space application or VM is active and vCPUs are runnin= g, + * transition VFIO device in pre-copy state. + * - VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY: + * When VFIO user space application or VM is stopped and vCPUs are halte= d, + * transition VFIO device in stop-and-copy state. + * - VFIO_DEVICE_STATE_MIGRATION_SAVE_COMPLETED: + * When VFIO user space application has copied data provided by vendor d= river. + * This state is used by vendor driver to clean up all software state th= at was + * setup during MIGRATION_SETUP state. + * - VFIO_DEVICE_STATE_MIGRATION_RESUME: + * Transition VFIO device to resume state, that is, start resuming VFIO = device + * when user space application or VM is not running and vCPUs are halted. + * - VFIO_DEVICE_STATE_MIGRATION_RESUME_COMPLETED: + * When user space application completes iterations of providing device = state + * data, transition device in resume completed state. + * - VFIO_DEVICE_STATE_MIGRATION_FAILED: + * Migration process failed due to some reason, transition device to fai= led + * state. If migration process fails while saving at source, resume devi= ce at + * source. If migration process fails while resuming application or VM at + * destination, stop restoration at destination and resume at source. + * - VFIO_DEVICE_STATE_MIGRATION_CANCELLED: + * User space application has cancelled migration process either for some + * known reason or due to user's intervention. Transition device to Canc= elled + * state, that is, resume device state as it was during running state at + * source. + */ + +enum { + VFIO_DEVICE_STATE_NONE, + VFIO_DEVICE_STATE_RUNNING, + VFIO_DEVICE_STATE_MIGRATION_SETUP, + VFIO_DEVICE_STATE_MIGRATION_PRECOPY, + VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY, + VFIO_DEVICE_STATE_MIGRATION_SAVE_COMPLETED, + VFIO_DEVICE_STATE_MIGRATION_RESUME, + VFIO_DEVICE_STATE_MIGRATION_RESUME_COMPLETED, + VFIO_DEVICE_STATE_MIGRATION_FAILED, + VFIO_DEVICE_STATE_MIGRATION_CANCELLED, +}; + +/** + * Structure vfio_device_migration_info is placed at 0th offset of + * VFIO_REGION_SUBTYPE_MIGRATION region to get/set VFIO device related mig= ration + * information. + * + * Action Set state: + * To tell vendor driver the state VFIO device should be transitioned= to. + * device_state [input] : User space app sends device state to vendor + * driver on state change, the state to which VFIO device should= be + * transitioned to. + * + * Action Get pending bytes: + * To get pending bytes yet to be migrated from vendor driver + * pending.threshold_size [Input] : threshold of buffer in User space= app. + * pending.precopy_only [output] : pending data which must be migrate= d in + * precopy phase or in stopped state, in other words - before tar= get + * user space application or VM start. In case of migration, this + * indicates pending bytes to be transfered while application or = VM or + * vCPUs are active and running. + * pending.compatible [output] : pending data which may be migrated a= ny + * time , either when application or VM is active and vCPUs are a= ctive + * or when application or VM is halted and vCPUs are halted. + * pending.postcopy_only [output] : pending data which must be migrat= ed in + * postcopy phase or in stopped state, in other words - after so= urce + * application or VM stopped and vCPUs are halted. + * Sum of pending.precopy_only, pending.compatible and + * pending.postcopy_only is the whole amount of pending data. + * + * Action Get buffer: + * On this action, vendor driver should write data to migration regio= n and + * return number of bytes written in the region. + * data.offset [output] : offset in the region from where data is wri= tten. + * data.size [output] : number of bytes written in migration buffer by + * vendor driver. + * + * Action Set buffer: + * In migration resume path, user space app writes to migration regio= n and + * communicates it to vendor driver with this action. + * data.offset [Input] : offset in the region from where data is writ= ten. + * data.size [Input] : number of bytes written in migration buffer by + * user space app. + * + * Action Get dirty pages bitmap: + * Get bitmap of dirty pages from vendor driver from given start addr= ess. + * dirty_pfns.start_addr [Input] : start address + * dirty_pfns.total [Input] : Total pfn count from start_addr for whi= ch + * dirty bitmap is requested + * dirty_pfns.copied [Output] : pfn count for which dirty bitmap is c= opied + * to migration region. + * Vendor driver should copy the bitmap with bits set only for pages = to be + * marked dirty in migration region. + */ + +struct vfio_device_migration_info { + __u32 device_state; /* VFIO device state */ + struct { + __u64 precopy_only; + __u64 compatible; + __u64 postcopy_only; + __u64 threshold_size; + } pending; + struct { + __u64 offset; /* offset */ + __u64 size; /* size */ + } data; + struct { + __u64 start_addr; + __u64 total; + __u64 copied; + } dirty_pfns; +} __attribute__((packed)); + /* -------- API for Type1 VFIO IOMMU -------- */ =20 /** --=20 2.7.0 From nobody Mon Apr 29 06:20:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542746642177364.6572059707282; Tue, 20 Nov 2018 12:44:02 -0800 (PST) Received: from localhost ([::1]:35820 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCsI-0007lt-P2 for importer@patchew.org; Tue, 20 Nov 2018 15:43:58 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52501) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqJ-0005tb-6b for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqE-0003SW-TJ for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:55 -0500 Received: from hqemgate16.nvidia.com ([216.228.121.65]:10903) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqD-0003Q7-3q for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:50 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:56 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:47 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL106.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:47 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:39 +0000 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 20 Nov 2018 12:41:47 -0800 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:40 +0530 Message-ID: <1542746383-18288-3-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746516; bh=Fsa9hp8Frjv38+rK7idpKFnTlEMNyVBZ02YCObJA0sc=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=q9mnt2cQh1vLb7R8ZxRmvA2ltvh+rI3q6LyumG7vwPLb0U6ioRvgDME3W9s6eGFFK 3CARMmefyQO+0iFqSNNO5U3XbOsbD5RAq8316AZGmLlqfcs5HQp88Y1m7DZIVFB4ta MEvkAv8ejviB1b9IYkEsDDV0z1U1qZhdwfJHWT5JOkhdANhoYFyWb1+nGUcjh8+5xD P/gGRVwuJy5Ww5fkgZCiD3jRl5bBl6YdEyjN4efgzohDlKxj9wRQ7hLVnaiNnb+tFL uqqrIo59o4DZLvsFNCYPzwaGrgKhQ+jGnexwTzdqvOR/BS8w9PJkojBfP7IHDt6QDn ZADlDZB9HdygQ== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH 2/5] Add save and load functions for VFIO PCI devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Save and restore with MSIX type is not tested. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/pci.h | 29 ++++++++++++++++++ 2 files changed, 124 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 6cbb8fa0549d..72daf1a358a0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1234,6 +1234,101 @@ void vfio_pci_write_config(PCIDevice *pdev, } } =20 +void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + PCIDevice *pdev =3D &vdev->pdev; + int i; + + for (i =3D 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar; + + bar =3D pci_default_read_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, = 4); + qemu_put_be32(f, bar); + } + + qemu_put_be32(f, vdev->interrupt); + if (vdev->interrupt =3D=3D VFIO_INT_MSI) { + uint32_t msi_flags, msi_addr_lo, msi_addr_hi =3D 0, msi_data; + bool msi_64bit; + + msi_flags =3D pci_default_read_config(pdev, pdev->msi_cap + PCI_MS= I_FLAGS, + 2); + msi_64bit =3D (msi_flags & PCI_MSI_FLAGS_64BIT); + + msi_addr_lo =3D pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_L= O, 4); + qemu_put_be32(f, msi_addr_lo); + + if (msi_64bit) { + msi_addr_hi =3D pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRE= SS_HI, + 4); + } + qemu_put_be32(f, msi_addr_hi); + + msi_data =3D pci_default_read_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DAT= A_32), + 2); + qemu_put_be32(f, msi_data); + } else if (vdev->interrupt =3D=3D VFIO_INT_MSIX) { + msix_save(pdev, f); + } +} + +void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + PCIDevice *pdev =3D &vdev->pdev; + uint32_t pci_cmd, interrupt_type; + uint32_t msi_flags, msi_addr_lo, msi_addr_hi =3D 0, msi_data; + bool msi_64bit; + int i; + + /* retore pci bar configuration */ + pci_cmd =3D pci_default_read_config(pdev, PCI_COMMAND, 2); + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd & (!(PCI_COMMAND_IO | PCI_COMMAND_MEMORY))= , 2); + for (i =3D 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar =3D qemu_get_be32(f); + + vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar, 4); + } + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd | PCI_COMMAND_IO | PCI_COMMAND_MEMORY, 2= ); + + interrupt_type =3D qemu_get_be32(f); + + if (interrupt_type =3D=3D VFIO_INT_MSI) { + /* restore msi configuration */ + msi_flags =3D pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_FLAGS,= 2); + msi_64bit =3D (msi_flags & PCI_MSI_FLAGS_64BIT); + + vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags & (!PCI_MSI_FLAGS_ENABLE), 2); + + msi_addr_lo =3D qemu_get_be32(f); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, + msi_addr_lo, 4); + + msi_addr_hi =3D qemu_get_be32(f); + if (msi_64bit) { + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI, + msi_addr_hi, 4); + } + msi_data =3D qemu_get_be32(f); + vfio_pci_write_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DAT= A_32), + msi_data, 2); + + vfio_pci_write_config(&vdev->pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags | PCI_MSI_FLAGS_ENABLE, 2); + } else if (vdev->interrupt =3D=3D VFIO_INT_MSIX) { + msix_load(pdev, f); + } +} + /* * Interrupt setup */ diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 52b065421a68..890d77d66a6b 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -20,6 +20,7 @@ #include "qemu/queue.h" #include "qemu/timer.h" =20 +#ifdef CONFIG_LINUX #define PCI_ANY_ID (~0) =20 struct VFIOPCIDevice; @@ -198,4 +199,32 @@ void vfio_display_reset(VFIOPCIDevice *vdev); int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp); void vfio_display_finalize(VFIOPCIDevice *vdev); =20 +void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f); +void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f); + +static inline Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + + return OBJECT(vdev); +} + +#else +static inline void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + g_assert(false); +} + +static inline void vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + g_assert(false); +} + +static inline Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + return NULL; +} + +#endif + #endif /* HW_VFIO_VFIO_PCI_H */ --=20 2.7.0 From nobody Mon Apr 29 06:20:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542746665764903.1451869286196; Tue, 20 Nov 2018 12:44:25 -0800 (PST) Received: from localhost ([::1]:35823 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCsY-0007xa-Nb for importer@patchew.org; Tue, 20 Nov 2018 15:44:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52638) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqQ-0005zK-Mw for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqM-0003c7-9I for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:02 -0500 Received: from hqemgate14.nvidia.com ([216.228.121.143]:16421) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqL-0003ZV-Oh for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:58 -0500 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:56 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:56 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:55 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:47 +0000 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Tue, 20 Nov 2018 12:41:56 -0800 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:41 +0530 Message-ID: <1542746383-18288-4-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746516; bh=7wbLhfIH2Wr/3Kis5uhHsHAcHZiDPSdTau7uvr9wiyc=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=f2tXFCBmmf72y9FybBteES0aoZn32rZbnR7c717xU7PACK9A4/WoAaX0lflxWvKli N8MiGZDmOdGBw3dxc16FN9N5sS9PBTQ0p13JwiEHU9KouBO5hF069h+WQ6BuKkm5MR ULBw9QsCk72aobdMLYexqUU9vIK49qo3zGVxjLXrba4U6lnXbLX3hCukhlWiXZFqw9 WRi1ah0Iotnx3WpgZ/o8af+vJ5Fiaztsx3iL3/0FI0LfSMgLvk79vjcn8zZWsF+6oL 1WVmk0nivHJa89vDdDoj1vocHHbzOYsOLSYpqEgE0Peu6yedzEJHALqdeorRq9i/+j uA9Brd2jbEHYA== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH 3/5] Add migration functions for VFIO devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" - Migration function are implemented for VFIO_DEVICE_TYPE_PCI device. - Added SaveVMHandlers and implemented all basic functions required for live migration. - Added VM state change handler to know running or stopped state of VM. - Added migration state change notifier to get notification on migration st= ate change. This state is translated to VFIO device state and conveyed to ven= dor driver. - VFIO device supportd migration or not is decided based of migration region query. If migration region query is successful then migration is supported else migration is blocked. - Structure vfio_device_migration_info is mapped at 0th offset of migration region and should always trapped by VFIO device's driver. Added both type= of access support, trapped or mmapped, for data section of the region. - To save device state, read data offset and size using structure vfio_device_migration_info.data, accordingly copy data from the region. - To restore device state, write data offset and size in the structure and = write data in the region. - To get dirty page bitmap, write start address and pfn count then read cou= nt of pfns copied and accordingly read those from the rest of the region or mma= ped part of the region. This copy is iterated till page bitmap for all reques= ted pfns are copied. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/Makefile.objs | 2 +- hw/vfio/migration.c | 729 ++++++++++++++++++++++++++++++++++++++= ++++ include/hw/vfio/vfio-common.h | 23 ++ 3 files changed, 753 insertions(+), 1 deletion(-) create mode 100644 hw/vfio/migration.c diff --git a/hw/vfio/Makefile.objs b/hw/vfio/Makefile.objs index a2e7a0a7cf02..2cf2ba1440f2 100644 --- a/hw/vfio/Makefile.objs +++ b/hw/vfio/Makefile.objs @@ -1,5 +1,5 @@ ifeq ($(CONFIG_LINUX), y) -obj-$(CONFIG_SOFTMMU) +=3D common.o +obj-$(CONFIG_SOFTMMU) +=3D common.o migration.o obj-$(CONFIG_PCI) +=3D pci.o pci-quirks.o display.o obj-$(CONFIG_VFIO_CCW) +=3D ccw.o obj-$(CONFIG_SOFTMMU) +=3D platform.o diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c new file mode 100644 index 000000000000..717fb63e4f43 --- /dev/null +++ b/hw/vfio/migration.c @@ -0,0 +1,729 @@ +/* + * Migration support for VFIO devices + * + * Copyright NVIDIA, Inc. 2018 + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include + +#include "hw/vfio/vfio-common.h" +#include "cpu.h" +#include "migration/migration.h" +#include "migration/qemu-file.h" +#include "migration/register.h" +#include "migration/blocker.h" +#include "migration/misc.h" +#include "qapi/error.h" +#include "exec/ramlist.h" +#include "exec/ram_addr.h" +#include "pci.h" + +/* + * Flags used as delimiter: + * 0xffffffff =3D> MSB 32-bit all 1s + * 0xef10 =3D> emulated (virtual) function IO + * 0x0000 =3D> 16-bits reserved for flags + */ +#define VFIO_MIG_FLAG_END_OF_STATE (0xffffffffef100001ULL) +#define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) +#define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) + +static void vfio_migration_region_exit(VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + + if (!migration) { + return; + } + + if (migration->region.buffer.size) { + vfio_region_exit(&migration->region.buffer); + vfio_region_finalize(&migration->region.buffer); + } +} + +static int vfio_migration_region_init(VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + Object *obj =3D NULL; + int ret =3D -EINVAL; + + if (!migration) { + return ret; + } + + /* Migration support added for PCI device only */ + if (vbasedev->type =3D=3D VFIO_DEVICE_TYPE_PCI) { + obj =3D vfio_pci_get_object(vbasedev); + } + + if (!obj) { + return ret; + } + + ret =3D vfio_region_setup(obj, vbasedev, &migration->region.buffer, + migration->region.index, "migration"); + if (ret) { + error_report("Failed to setup VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + + if (!migration->region.buffer.size) { + ret =3D -EINVAL; + error_report("Invalid region size of VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + + if (migration->region.buffer.mmaps) { + ret =3D vfio_region_mmap(&migration->region.buffer); + if (ret) { + error_report("Failed to mmap VFIO migration region %d: %s", + migration->region.index, strerror(-ret)); + goto err; + } + } + + return 0; + +err: + vfio_migration_region_exit(vbasedev); + return ret; +} + +static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t state) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region.buffer; + int ret =3D 0; + + if (vbasedev->device_state =3D=3D state) { + return ret; + } + + ret =3D pwrite(vbasedev->fd, &state, sizeof(state), + region->fd_offset + offsetof(struct vfio_device_migration= _info, + device_state)); + if (ret < 0) { + error_report("Failed to set migration state %d %s", + ret, strerror(errno)); + return ret; + } + + vbasedev->device_state =3D state; + return ret; +} + +void vfio_get_dirty_page_list(VFIODevice *vbasedev, + uint64_t start_addr, + uint64_t pfn_count) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region.buffer; + struct vfio_device_migration_info migration_info; + uint64_t count =3D 0; + int ret; + + migration_info.dirty_pfns.start_addr =3D start_addr; + migration_info.dirty_pfns.total =3D pfn_count; + + ret =3D pwrite(vbasedev->fd, &migration_info.dirty_pfns, + sizeof(migration_info.dirty_pfns), + region->fd_offset + offsetof(struct vfio_device_migration= _info, + dirty_pfns)); + if (ret < 0) { + error_report("Failed to set dirty pages start address %d %s", + ret, strerror(errno)); + return; + } + + do { + /* Read dirty_pfns.copied */ + ret =3D pread(vbasedev->fd, &migration_info.dirty_pfns, + sizeof(migration_info.dirty_pfns), + region->fd_offset + offsetof(struct vfio_device_migration_= info, + dirty_pfns)); + if (ret < 0) { + error_report("Failed to get dirty pages bitmap count %d %s", + ret, strerror(errno)); + return; + } + + if (migration_info.dirty_pfns.copied) { + uint64_t bitmap_size; + void *buf =3D NULL; + bool buffer_mmaped =3D false; + + bitmap_size =3D (BITS_TO_LONGS(migration_info.dirty_pfns.copie= d) + 1) + * sizeof(unsigned long); + + if (region->mmaps) { + int i; + + for (i =3D 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].size >=3D bitmap_size) { + buf =3D region->mmaps[i].mmap; + buffer_mmaped =3D true; + break; + } + } + } + + if (!buffer_mmaped) { + buf =3D g_malloc0(bitmap_size); + + ret =3D pread(vbasedev->fd, buf, bitmap_size, + region->fd_offset + sizeof(migration_info) + 1= ); + if (ret !=3D bitmap_size) { + error_report("Failed to get migration data %d", ret); + g_free(buf); + return; + } + } + + cpu_physical_memory_set_dirty_lebitmap((unsigned long *)buf, + start_addr + (count * TARGET_PAGE_= SIZE), + migration_info.dirty_pfns.copied); + count +=3D migration_info.dirty_pfns.copied; + + if (!buffer_mmaped) { + g_free(buf); + } + } + } while (count < migration_info.dirty_pfns.total); +} + +static int vfio_save_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_STATE); + + if (vbasedev->type =3D=3D VFIO_DEVICE_TYPE_PCI) { + vfio_pci_save_config(vbasedev, f); + } + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + return qemu_file_get_error(f); +} + +static int vfio_load_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + + if (vbasedev->type =3D=3D VFIO_DEVICE_TYPE_PCI) { + vfio_pci_load_config(vbasedev, f); + } + + if (qemu_get_be64(f) !=3D VFIO_MIG_FLAG_END_OF_STATE) { + error_report("Wrong end of block while loading device config space= "); + return -EINVAL; + } + + return qemu_file_get_error(f); +} + +/* ---------------------------------------------------------------------- = */ + +static bool vfio_is_active_iterate(void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + + if (vbasedev->vm_running && vbasedev->migration && + (vbasedev->migration->pending_precopy_only !=3D 0)) + return true; + + if (!vbasedev->vm_running && vbasedev->migration && + (vbasedev->migration->pending_postcopy !=3D 0)) + return true; + + return false; +} + +static int vfio_save_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + int ret; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + + qemu_mutex_lock_iothread(); + ret =3D vfio_migration_region_init(vbasedev); + qemu_mutex_unlock_iothread(); + if (ret) { + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + return 0; +} + +static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region.buffer; + struct vfio_device_migration_info migration_info; + int ret; + + ret =3D pread(vbasedev->fd, &migration_info.data, + sizeof(migration_info.data), + region->fd_offset + offsetof(struct vfio_device_migration_= info, + data)); + if (ret !=3D sizeof(migration_info.data)) { + error_report("Failed to get migration buffer information %d", + ret); + return -EINVAL; + } + + if (migration_info.data.size) { + void *buf =3D NULL; + bool buffer_mmaped =3D false; + + if (region->mmaps) { + int i; + + for (i =3D 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].offset =3D=3D migration_info.data.off= set) { + buf =3D region->mmaps[i].mmap; + buffer_mmaped =3D true; + break; + } + } + } + + if (!buffer_mmaped) { + buf =3D g_malloc0(migration_info.data.size); + ret =3D pread(vbasedev->fd, buf, migration_info.data.size, + region->fd_offset + migration_info.data.offset); + if (ret !=3D migration_info.data.size) { + error_report("Failed to get migration data %d", ret); + return -EINVAL; + } + } + + qemu_put_be64(f, migration_info.data.size); + qemu_put_buffer(f, buf, migration_info.data.size); + + if (!buffer_mmaped) { + g_free(buf); + } + + } else { + qemu_put_be64(f, migration_info.data.size); + } + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + return migration_info.data.size; +} + +static int vfio_save_iterate(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + int ret; + + ret =3D vfio_save_buffer(f, vbasedev); + if (ret < 0) { + error_report("vfio_save_buffer failed %s", + strerror(errno)); + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + return ret; +} + +static void vfio_update_pending(VFIODevice *vbasedev, uint64_t threshold_s= ize) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region.buffer; + struct vfio_device_migration_info migration_info; + int ret; + + ret =3D pwrite(vbasedev->fd, &threshold_size, sizeof(threshold_size), + region->fd_offset + offsetof(struct vfio_device_migration= _info, + pending.threshold_size)); + if (ret < 0) { + error_report("Failed to set threshold size %d %s", + ret, strerror(errno)); + return; + } + + ret =3D pread(vbasedev->fd, &migration_info.pending, + sizeof(migration_info.pending), + region->fd_offset + offsetof(struct vfio_device_migration_= info, + pending)); + if (ret !=3D sizeof(migration_info.pending)) { + error_report("Failed to get pending bytes %d", ret); + return; + } + + migration->pending_precopy_only =3D migration_info.pending.precopy_onl= y; + migration->pending_compatible =3D migration_info.pending.compatible; + migration->pending_postcopy =3D migration_info.pending.postcopy_only; + + return; +} + +static void vfio_save_pending(QEMUFile *f, void *opaque, + uint64_t threshold_size, + uint64_t *res_precopy_only, + uint64_t *res_compatible, + uint64_t *res_postcopy_only) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + + vfio_update_pending(vbasedev, threshold_size); + + *res_precopy_only +=3D migration->pending_precopy_only; + *res_compatible +=3D migration->pending_compatible; + *res_postcopy_only +=3D migration->pending_postcopy; +} + +static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + MigrationState *ms =3D migrate_get_current(); + int ret; + + if (vbasedev->vm_running) { + vbasedev->vm_running =3D 0; + } + + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY); + if (ret) { + error_report("Failed to set state STOPNCOPY_ACTIVE"); + return ret; + } + + ret =3D vfio_save_device_config_state(f, opaque); + if (ret) { + return ret; + } + + do { + vfio_update_pending(vbasedev, ms->threshold_size); + + if (vfio_is_active_iterate(opaque)) { + ret =3D vfio_save_buffer(f, vbasedev); + if (ret < 0) { + error_report("Failed to save buffer"); + break; + } else if (ret =3D=3D 0) { + break; + } + } + } while ((migration->pending_compatible + migration->pending_postcopy)= > 0); + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_SAVE_COMPLE= TED); + if (ret) { + error_report("Failed to set state SAVE_COMPLETED"); + return ret; + } + return ret; +} + +static void vfio_save_cleanup(void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + + vfio_migration_region_exit(vbasedev); +} + +static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) +{ + VFIODevice *vbasedev =3D opaque; + int ret; + uint64_t data; + + data =3D qemu_get_be64(f); + while (data !=3D VFIO_MIG_FLAG_END_OF_STATE) { + if (data =3D=3D VFIO_MIG_FLAG_DEV_CONFIG_STATE) { + ret =3D vfio_load_device_config_state(f, opaque); + if (ret) { + return ret; + } + } else if (data =3D=3D VFIO_MIG_FLAG_DEV_SETUP_STATE) { + data =3D qemu_get_be64(f); + if (data =3D=3D VFIO_MIG_FLAG_END_OF_STATE) { + return 0; + } else { + error_report("SETUP STATE: EOS not found 0x%lx", data); + return -EINVAL; + } + } else if (data !=3D 0) { + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region.buffer; + struct vfio_device_migration_info migration_info; + void *buf =3D NULL; + bool buffer_mmaped =3D false; + + migration_info.data.size =3D data; + + if (region->mmaps) { + int i; + + for (i =3D 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].mmap && + (region->mmaps[i].size >=3D data)) { + buf =3D region->mmaps[i].mmap; + migration_info.data.offset =3D region->mmaps[i].of= fset; + buffer_mmaped =3D true; + break; + } + } + } + + if (!buffer_mmaped) { + buf =3D g_malloc0(migration_info.data.size); + migration_info.data.offset =3D sizeof(migration_info) + 1; + } + + qemu_get_buffer(f, buf, data); + + ret =3D pwrite(vbasedev->fd, &migration_info.data, + sizeof(migration_info.data), + region->fd_offset + + offsetof(struct vfio_device_migration_info, data)= ); + if (ret !=3D sizeof(migration_info.data)) { + error_report("Failed to set migration buffer information %= d", + ret); + return -EINVAL; + } + + if (!buffer_mmaped) { + ret =3D pwrite(vbasedev->fd, buf, migration_info.data.size, + region->fd_offset + migration_info.data.offse= t); + g_free(buf); + + if (ret !=3D migration_info.data.size) { + error_report("Failed to set migration buffer %d", ret); + return -EINVAL; + } + } + } + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + data =3D qemu_get_be64(f); + } + + return 0; +} + +static int vfio_load_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + int ret; + + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_RESUME); + if (ret) { + error_report("Failed to set state RESUME"); + } + + ret =3D vfio_migration_region_init(vbasedev); + if (ret) { + error_report("Failed to initialise migration region"); + return ret; + } + + return 0; +} + +static int vfio_load_cleanup(void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + int ret =3D 0; + + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_RESUME_COMPLE= TED); + if (ret) { + error_report("Failed to set state RESUME_COMPLETED"); + } + + vfio_migration_region_exit(vbasedev); + return ret; +} + +static SaveVMHandlers savevm_vfio_handlers =3D { + .save_setup =3D vfio_save_setup, + .save_live_iterate =3D vfio_save_iterate, + .save_live_complete_precopy =3D vfio_save_complete_precopy, + .save_live_pending =3D vfio_save_pending, + .save_cleanup =3D vfio_save_cleanup, + .load_state =3D vfio_load_state, + .load_setup =3D vfio_load_setup, + .load_cleanup =3D vfio_load_cleanup, + .is_active_iterate =3D vfio_is_active_iterate, +}; + +static void vfio_vmstate_change(void *opaque, int running, RunState state) +{ + VFIODevice *vbasedev =3D opaque; + + if ((vbasedev->vm_running !=3D running) && running) { + int ret; + + ret =3D vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RUNNI= NG); + if (ret) { + error_report("Failed to set state RUNNING"); + } + } + + vbasedev->vm_running =3D running; +} + +static void vfio_migration_state_notifier(Notifier *notifier, void *data) +{ + MigrationState *s =3D data; + VFIODevice *vbasedev =3D container_of(notifier, VFIODevice, migration_= state); + int ret; + + switch (s->state) { + case MIGRATION_STATUS_SETUP: + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_SETUP); + if (ret) { + error_report("Failed to set state SETUP"); + } + return; + + case MIGRATION_STATUS_ACTIVE: + if (vbasedev->device_state =3D=3D VFIO_DEVICE_STATE_MIGRATION_SETU= P) { + if (vbasedev->vm_running) { + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_PREC= OPY); + if (ret) { + error_report("Failed to set state PRECOPY_ACTIVE"); + } + } else { + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_STOPNC= OPY); + if (ret) { + error_report("Failed to set state STOPNCOPY_ACTIVE"); + } + } + } else { + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_RES= UME); + if (ret) { + error_report("Failed to set state RESUME"); + } + } + return; + + case MIGRATION_STATUS_CANCELLING: + case MIGRATION_STATUS_CANCELLED: + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_CANCELL= ED); + if (ret) { + error_report("Failed to set state CANCELLED"); + } + return; + + case MIGRATION_STATUS_FAILED: + ret =3D vfio_migration_set_state(vbasedev, + VFIO_DEVICE_STATE_MIGRATION_FAILED); + if (ret) { + error_report("Failed to set state FAILED"); + } + return; + } +} + +static int vfio_migration_init(VFIODevice *vbasedev, + struct vfio_region_info *info) +{ + vbasedev->migration =3D g_new0(VFIOMigration, 1); + vbasedev->migration->region.index =3D info->index; + + register_savevm_live(NULL, "vfio", -1, 1, &savevm_vfio_handlers, vbase= dev); + vbasedev->vm_state =3D qemu_add_vm_change_state_handler(vfio_vmstate_c= hange, + vbasedev); + + vbasedev->migration_state.notify =3D vfio_migration_state_notifier; + add_migration_state_change_notifier(&vbasedev->migration_state); + + return 0; +} + + +/* ---------------------------------------------------------------------- = */ + +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) +{ + struct vfio_region_info *info; + int ret; + + ret =3D vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION, + VFIO_REGION_SUBTYPE_MIGRATION, &info); + if (ret) { + Error *local_err =3D NULL; + + error_setg(&vbasedev->migration_blocker, + "VFIO device doesn't support migration"); + ret =3D migrate_add_blocker(vbasedev->migration_blocker, &local_er= r); + if (local_err) { + error_propagate(errp, local_err); + error_free(vbasedev->migration_blocker); + return ret; + } + } else { + return vfio_migration_init(vbasedev, info); + } + + return 0; +} + +void vfio_migration_finalize(VFIODevice *vbasedev) +{ + if (!vbasedev->migration) { + return; + } + + if (vbasedev->vm_state) { + qemu_del_vm_change_state_handler(vbasedev->vm_state); + remove_migration_state_change_notifier(&vbasedev->migration_state); + } + + if (vbasedev->migration_blocker) { + migrate_del_blocker(vbasedev->migration_blocker); + error_free(vbasedev->migration_blocker); + } + + g_free(vbasedev->migration); +} diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index a9036929b220..ab8217c9e249 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -30,6 +30,8 @@ #include #endif =20 +#include "sysemu/sysemu.h" + #define ERR_PREFIX "vfio error: %s: " #define WARN_PREFIX "vfio warning: %s: " =20 @@ -57,6 +59,16 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; =20 +typedef struct VFIOMigration { + struct { + VFIORegion buffer; + uint32_t index; + } region; + uint64_t pending_precopy_only; + uint64_t pending_compatible; + uint64_t pending_postcopy; +} VFIOMigration; + typedef struct VFIOAddressSpace { AddressSpace *as; QLIST_HEAD(, VFIOContainer) containers; @@ -116,6 +128,12 @@ typedef struct VFIODevice { unsigned int num_irqs; unsigned int num_regions; unsigned int flags; + uint32_t device_state; + VMChangeStateEntry *vm_state; + int vm_running; + Notifier migration_state; + VFIOMigration *migration; + Error *migration_blocker; } VFIODevice; =20 struct VFIODeviceOps { @@ -193,4 +211,9 @@ int vfio_spapr_create_window(VFIOContainer *container, int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); =20 +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); +void vfio_migration_finalize(VFIODevice *vbasedev); +void vfio_get_dirty_page_list(VFIODevice *vbasedev, uint64_t start_addr, + uint64_t pfn_count); + #endif /* HW_VFIO_VFIO_COMMON_H */ --=20 2.7.0 From nobody Mon Apr 29 06:20:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542746811658550.7679909638007; Tue, 20 Nov 2018 12:46:51 -0800 (PST) Received: from localhost ([::1]:35843 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCus-0001XB-Hk for importer@patchew.org; Tue, 20 Nov 2018 15:46:38 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52807) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqX-000653-KF for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqT-0003pY-4M for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:09 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:4274) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqS-0003my-A6 for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:04 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:49 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:42:02 -0800 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:42:02 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:56 +0000 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 20 Nov 2018 12:42:02 -0800 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:42 +0530 Message-ID: <1542746383-18288-5-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746510; bh=gyn4bhWH/PoYVA0P8KcuSgdTA0KWVgAOZrqTd5xTVyk=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=FmHq3aHCvoChx1s8UT30gBmL96gg58C6RifrKyIF9Hhicg7+oyjbWSQP8aZW6hU7M jwSdxNyorezGInDIo96l3WqPDdGJTBjWS4tBhSiE+FsaXbOh9Cv755YK3bPy+EZ7P0 q2PBBAx/48QDoOdYt/XycG/bbdrNGFqQmMuWDHwaw4hLZLQ2lSIlg+NyYwO1DVdowr W+EnjuEznKGUAmiNZ3RuzSR9xb58uEW+i6oPdhpXu9rDL9BW4wN8JIxPx149NgDCrt JSqf+/DT7iZr1DkWv1lbZyfw4eCv03oPs/UdJGo0+HKhRlQenHpF0lZObwg3ZVSK0o CuwCY5iBdqfkw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH 4/5] Add vfio_listerner_log_sync to mark dirty pages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vfio_listerner_log_sync gets list of dirty pages from vendor driver and mark those pages dirty. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/common.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index fb396cf00ac4..338aad7426f0 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -697,9 +697,41 @@ static void vfio_listener_region_del(MemoryListener *l= istener, } } =20 +static void vfio_listerner_log_sync(MemoryListener *listener, + MemoryRegionSection *section) +{ + uint64_t start_addr, size, pfn_count; + VFIOGroup *group; + VFIODevice *vbasedev; + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + switch (vbasedev->device_state) { + case VFIO_DEVICE_STATE_MIGRATION_PRECOPY: + case VFIO_DEVICE_STATE_MIGRATION_STOPNCOPY: + continue; + + default: + return; + } + } + } + + start_addr =3D TARGET_PAGE_ALIGN(section->offset_within_address_space); + size =3D int128_get64(section->size); + pfn_count =3D size >> TARGET_PAGE_BITS; + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + vfio_get_dirty_page_list(vbasedev, start_addr, pfn_count); + } + } +} + static const MemoryListener vfio_memory_listener =3D { .region_add =3D vfio_listener_region_add, .region_del =3D vfio_listener_region_del, + .log_sync =3D vfio_listerner_log_sync, }; =20 static void vfio_listener_release(VFIOContainer *container) --=20 2.7.0 From nobody Mon Apr 29 06:20:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542746842173354.12667418421756; Tue, 20 Nov 2018 12:47:22 -0800 (PST) Received: from localhost ([::1]:35848 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCvU-0001vU-4d for importer@patchew.org; Tue, 20 Nov 2018 15:47:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52908) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCqq-0006Lh-Ux for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:33 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCqc-00044a-Rn for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:23 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:4291) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCqZ-0003yY-2D for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:42:12 -0500 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:56 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:42:09 -0800 Received: from HQMAIL102.nvidia.com (172.18.146.10) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:42:09 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL102.nvidia.com (172.18.146.10) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:42:08 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:42:02 +0000 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Tue, 20 Nov 2018 12:42:09 -0800 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:43 +0530 Message-ID: <1542746383-18288-6-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> References: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746516; bh=uJUAloTLG8+CnoIbbmdia4XyNi6ws0pOtAQSOcwm+AI=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=mADwhIuOci+YtlNmmsSsYKlPzObhqUWxiY8ut5aYQrrZvK+3sRpgraZOnzUqLYgkB GjhKkEO7pK+0OCwu9t1p5Pku06z9lLUllz9NZbSVLVDjh2tZTp/3EZKMouIdGGTOCT X3Ar3gck0Z9+F3x4kJOMZUYuvUlK/LY8RQgVGyd7LtGidT3oXO0YqXN6BkBpovJW/T S/VYfUcAbcwprXqUbdZ3J4zuCNMJZgVagtSHzvb4lxi62oyAhhJC7mAiQCeOdffYHR OJs0I28iOhbeHvgrdy2xu/p1G9dzerJFbmteWYDvO73Da1MDdtVm/2NkEqEma15HPR KMvJkKKalaWXw== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH 5/5] Make vfio-pci device migration capable. X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Call vfio_migration_probe() and vfio_migration_finalize() functions for vfio-pci device to enable migration for vfio PCI device. Removed vfio_pci_vmstate structure. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 72daf1a358a0..0f9d06981b1b 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2930,6 +2930,7 @@ static void vfio_realize(PCIDevice *pdev, Error **err= p) vdev->vbasedev.ops =3D &vfio_pci_ops; vdev->vbasedev.type =3D VFIO_DEVICE_TYPE_PCI; vdev->vbasedev.dev =3D &vdev->pdev.qdev; + vdev->vbasedev.device_state =3D VFIO_DEVICE_STATE_NONE; =20 tmp =3D g_strdup_printf("%s/iommu_group", vdev->vbasedev.sysfsdev); len =3D readlink(tmp, group_path, sizeof(group_path)); @@ -3141,10 +3142,11 @@ static void vfio_realize(PCIDevice *pdev, Error **e= rrp) } } =20 + ret =3D vfio_migration_probe(&vdev->vbasedev, errp); + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); - return; =20 out_teardown: @@ -3180,6 +3182,8 @@ static void vfio_exitfn(PCIDevice *pdev) { VFIOPCIDevice *vdev =3D DO_UPCAST(VFIOPCIDevice, pdev, pdev); =20 + vdev->vbasedev.device_state =3D VFIO_DEVICE_STATE_NONE; + vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); @@ -3189,6 +3193,7 @@ static void vfio_exitfn(PCIDevice *pdev) } vfio_teardown_msi(vdev); vfio_bars_exit(vdev); + vfio_migration_finalize(&vdev->vbasedev); } =20 static void vfio_pci_reset(DeviceState *dev) @@ -3294,11 +3299,6 @@ static Property vfio_pci_dev_properties[] =3D { DEFINE_PROP_END_OF_LIST(), }; =20 -static const VMStateDescription vfio_pci_vmstate =3D { - .name =3D "vfio-pci", - .unmigratable =3D 1, -}; - static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc =3D DEVICE_CLASS(klass); @@ -3306,7 +3306,6 @@ static void vfio_pci_dev_class_init(ObjectClass *klas= s, void *data) =20 dc->reset =3D vfio_pci_reset; dc->props =3D vfio_pci_dev_properties; - dc->vmsd =3D &vfio_pci_vmstate; dc->desc =3D "VFIO-based PCI device assignment"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); pdc->realize =3D vfio_realize; --=20 2.7.0