From nobody Fri May 3 04:43:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1624030679547443.9275365400332; Fri, 18 Jun 2021 08:37:59 -0700 (PDT) Received: from localhost ([::1]:53918 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1luGZC-0003H1-Fx for importer@patchew.org; Fri, 18 Jun 2021 11:37:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41164) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1luGWO-0005jx-BI for qemu-devel@nongnu.org; Fri, 18 Jun 2021 11:35:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:25896) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1luGWL-0004Mu-Sl for qemu-devel@nongnu.org; Fri, 18 Jun 2021 11:35:04 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-268-PwmIIHj_N9OxrII7Nvq3XQ-1; Fri, 18 Jun 2021 11:34:55 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B8FAC804143; Fri, 18 Jun 2021 15:34:54 +0000 (UTC) Received: from [172.30.41.16] (ovpn-112-106.phx2.redhat.com [10.3.112.106]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7BA9C5C225; Fri, 18 Jun 2021 15:34:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624030501; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ejRfTn8OHHxfyK+9KULs55XRj5XtjP/OHRapM/1fQ6U=; b=V0sfLEO2hjfDtPWaWpw1naQAyprBjNtes9YyWs1oXjyR5WCiwMNgTs3bTElU5ZrFMLdyZE oGRY2cxI4889E1+IHwZTHknW3jRs7TE2ccwv7iSkJEmn1c23AEpJvNxBS7vqaiP4XzHs1U 6BS4t1STLYhqWyNHsatIv1f14LdVYm8= X-MC-Unique: PwmIIHj_N9OxrII7Nvq3XQ-1 Subject: [PULL 1/3] docs/devel: Add VFIO device migration documentation From: Alex Williamson To: qemu-devel@nongnu.org Date: Fri, 18 Jun 2021 09:34:54 -0600 Message-ID: <162403049431.1793226.2362883307115162874.stgit@omen> In-Reply-To: <162403041204.1793226.16018359908572247606.stgit@omen> References: <162403041204.1793226.16018359908572247606.stgit@omen> User-Agent: StGit/1.0-8-g6af9-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.194, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Cornelia Huck , alex.williamson@redhat.com, Tarun Gupta Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: Tarun Gupta Document interfaces used for VFIO device migration. Added flow of state changes during live migration with VFIO device. Reviewed-by: Cornelia Huck Co-developed-by: Kirti Wankhede Signed-off-by: Kirti Wankhede Signed-off-by: Tarun Gupta Message-Id: <20210418122251.88809-1-targupta@nvidia.com> Signed-off-by: Alex Williamson --- MAINTAINERS | 1=20 docs/devel/index.rst | 1=20 docs/devel/vfio-migration.rst | 150 +++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 152 insertions(+) create mode 100644 docs/devel/vfio-migration.rst diff --git a/MAINTAINERS b/MAINTAINERS index 4842cc26e5ce..f7e12ea2488d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1817,6 +1817,7 @@ S: Supported F: hw/vfio/* F: include/hw/vfio/ F: docs/igd-assign.txt +F: docs/devel/vfio-migration.rst =20 vfio-ccw M: Cornelia Huck diff --git a/docs/devel/index.rst b/docs/devel/index.rst index 791925dcda54..977c3893bdaf 100644 --- a/docs/devel/index.rst +++ b/docs/devel/index.rst @@ -44,3 +44,4 @@ Contents: block-coroutine-wrapper multi-process ebpf_rss + vfio-migration diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst new file mode 100644 index 000000000000..9ff6163c8811 --- /dev/null +++ b/docs/devel/vfio-migration.rst @@ -0,0 +1,150 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +VFIO device Migration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Migration of virtual machine involves saving the state for each device that +the guest is running on source host and restoring this saved state on the +destination host. This document details how saving and restoring of VFIO +devices is done in QEMU. + +Migration of VFIO devices consists of two phases: the optional pre-copy ph= ase, +and the stop-and-copy phase. The pre-copy phase is iterative and allows to +accommodate VFIO devices that have a large amount of data that needs to be +transferred. The iterative pre-copy phase of migration allows for the gues= t to +continue whilst the VFIO device state is transferred to the destination, t= his +helps to reduce the total downtime of the VM. VFIO devices can choose to s= kip +the pre-copy phase of migration by returning pending_bytes as zero during = the +pre-copy phase. + +A detailed description of the UAPI for VFIO device migration can be found = in +the comment for the ``vfio_device_migration_info`` structure in the header +file linux-headers/linux/vfio.h. + +VFIO implements the device hooks for the iterative approach as follows: + +* A ``save_setup`` function that sets up the migration region and sets _SA= VING + flag in the VFIO device state. + +* A ``load_setup`` function that sets up the migration region on the + destination and sets _RESUMING flag in the VFIO device state. + +* A ``save_live_pending`` function that reads pending_bytes from the vendor + driver, which indicates the amount of data that the vendor driver has ye= t to + save for the VFIO device. + +* A ``save_live_iterate`` function that reads the VFIO device's data from = the + vendor driver through the migration region during iterative phase. + +* A ``save_state`` function to save the device config space if it is prese= nt. + +* A ``save_live_complete_precopy`` function that resets _RUNNING flag from= the + VFIO device state and iteratively copies the remaining data for the VFIO + device until the vendor driver indicates that no data remains (pending b= ytes + is zero). + +* A ``load_state`` function that loads the config section and the data + sections that are generated by the save functions above + +* ``cleanup`` functions for both save and load that perform any migration + related cleanup, including unmapping the migration region + + +The VFIO migration code uses a VM state change handler to change the VFIO +device state when the VM state changes from running to not-running, and +vice versa. + +Similarly, a migration state change handler is used to trigger a transitio= n of +the VFIO device state when certain changes of the migration state occur. F= or +example, the VFIO device state is transitioned back to _RUNNING in case a +migration failed or was canceled. + +System memory dirty pages tracking +---------------------------------- + +A ``log_global_start`` and ``log_global_stop`` memory listener callback in= forms +the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync`` +memory listener callback marks those system memory pages as dirty which are +used for DMA by the VFIO device. The dirty pages bitmap is queried per +container. All pages pinned by the vendor driver through external APIs hav= e to +be marked as dirty during migration. When there are CPU writes, CPU dirty = page +tracking can identify dirtied pages, but any page pinned by the vendor dri= ver +can also be written by the device. There is currently no device or IOMMU +support for dirty page tracking in hardware. + +By default, dirty pages are tracked when the device is in pre-copy as well= as +stop-and-copy phase. So, a page pinned by the vendor driver will be copied= to +the destination in both phases. Copying dirty pages in pre-copy phase helps +QEMU to predict if it can achieve its downtime tolerances. If QEMU during +pre-copy phase keeps finding dirty pages continuously, then it understands +that even in stop-and-copy phase, it is likely to find dirty pages and can +predict the downtime accordingly. + +QEMU also provides a per device opt-out option ``pre-copy-dirty-page-track= ing`` +which disables querying the dirty bitmap during pre-copy phase. If it is s= et to +off, all dirty pages will be copied to the destination in stop-and-copy ph= ase +only. + +System memory dirty pages tracking when vIOMMU is enabled +--------------------------------------------------------- + +With vIOMMU, an IO virtual address range can get unmapped while in pre-copy +phase of migration. In that case, the unmap ioctl returns any dirty pages = in +that range and QEMU reports corresponding guest physical pages dirty. Duri= ng +stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped +pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for t= hose +mapped ranges. + +Flow of state changes during Live migration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Below is the flow of state change during live migration. +The values in the brackets represent the VM state, the migration state, and +the VFIO device state, respectively. + +Live migration save path +------------------------ + +:: + + QEMU normal running state + (RUNNING, _NONE, _RUNNING) + | + migrate_init spawns migration_thread + Migration thread then calls each device's .save_setup() + (RUNNING, _SETUP, _RUNNING|_SAVING) + | + (RUNNING, _ACTIVE, _RUNNING|_SAVING) + If device is active, get pending_bytes by .save_live_pending() + If total pending_bytes >=3D threshold_size, call .save_live_iter= ate() + Data of VFIO device for pre-copy phase is copied + Iterate till total pending bytes converge and are less than thresh= old + | + On migration completion, vCPU stops and calls .save_live_complete_precop= y for + each active device. The VFIO device is then transitioned into _SAVING s= tate + (FINISH_MIGRATE, _DEVICE, _SAVING) + | + For the VFIO device, iterate in .save_live_complete_precopy until + pending data is 0 + (FINISH_MIGRATE, _DEVICE, _STOPPED) + | + (FINISH_MIGRATE, _COMPLETED, _STOPPED) + Migraton thread schedules cleanup bottom half and exits + +Live migration resume path +-------------------------- + +:: + + Incoming migration calls .load_setup for each device + (RESTORE_VM, _ACTIVE, _STOPPED) + | + For each device, .load_state is called for that device section data + (RESTORE_VM, _ACTIVE, _RESUMING) + | + At the end, .load_cleanup is called for each device and vCPUs are star= ted + (RUNNING, _NONE, _RUNNING) + +Postcopy +=3D=3D=3D=3D=3D=3D=3D=3D + +Postcopy migration is currently not supported for VFIO devices. From nobody Fri May 3 04:43:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1624030603606956.7216003545835; Fri, 18 Jun 2021 08:36:43 -0700 (PDT) Received: from localhost ([::1]:48768 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1luGXw-0008Hz-MO for importer@patchew.org; Fri, 18 Jun 2021 11:36:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41166) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1luGWQ-0005po-2t for qemu-devel@nongnu.org; Fri, 18 Jun 2021 11:35:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:53806) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1luGWN-0004Ni-BI for qemu-devel@nongnu.org; Fri, 18 Jun 2021 11:35:05 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-153-c0tdYkoVONSPWl-oT95VIA-1; Fri, 18 Jun 2021 11:35:01 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 285DC19057A0; Fri, 18 Jun 2021 15:35:00 +0000 (UTC) Received: from [172.30.41.16] (ovpn-112-106.phx2.redhat.com [10.3.112.106]) by smtp.corp.redhat.com (Postfix) with ESMTP id ED3C55C225; Fri, 18 Jun 2021 15:34:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624030502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4uqYtYkQcuUFVLSkepnppzeqiteFwmP3l09VgQcC/94=; b=aTT4S3cnNW6SU1tWSUfh7XwXe+fukb/Cyvc8jpc6rxynNS0idPYSNSaSVqeS3gnQYRtF8W 4bn5lGg25v0P6UPa+ebDropHUVLArof+wNCjgNmae5iyTdu9E3cxDLXN4Fn2HEuOpr85ke bOFZA6bfEpiA195SCwn+JELcisESQeM= X-MC-Unique: c0tdYkoVONSPWl-oT95VIA-1 Subject: [PULL 2/3] vfio: Fix unregister SaveVMHandler in vfio_migration_finalize From: Alex Williamson To: qemu-devel@nongnu.org Date: Fri, 18 Jun 2021 09:34:59 -0600 Message-ID: <162403049979.1793226.9917806596521410205.stgit@omen> In-Reply-To: <162403041204.1793226.16018359908572247606.stgit@omen> References: <162403041204.1793226.16018359908572247606.stgit@omen> User-Agent: StGit/1.0-8-g6af9-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.194, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alex.williamson@redhat.com, Kunkun Jiang , Qixin Gan Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: Kunkun Jiang In the vfio_migration_init(), the SaveVMHandler is registered for VFIO device. But it lacks the operation of 'unregister'. It will lead to 'Segmentation fault (core dumped)' in qemu_savevm_state_setup(), if performing live migration after a VFIO device is hot deleted. Fixes: 7c2f5f75f94 (vfio: Register SaveVMHandlers for VFIO device) Reported-by: Qixin Gan Signed-off-by: Kunkun Jiang Message-Id: <20210527123101.289-1-jiangkunkun@huawei.com> Reviewed by: Kirti Wankhede Signed-off-by: Alex Williamson --- hw/vfio/migration.c | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 201642d75e6b..ef397ebe6c09 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -892,6 +892,7 @@ void vfio_migration_finalize(VFIODevice *vbasedev) =20 remove_migration_state_change_notifier(&migration->migration_state= ); qemu_del_vm_change_state_handler(migration->vm_state); + unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); vfio_migration_exit(vbasedev); } =20 From nobody Fri May 3 04:43:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1624030668; cv=none; d=zohomail.com; s=zohoarc; b=QzxnbEAqhD9dPn1VVt7ul2OpwHMssvla39PzWAZO8FKlPl1airZ3UNRhq/4x3lrTBOyT3k6ZvfoI1MTuGUZ7pgp1RhZc8YMkKTCsKOJr13owQc2kpBsyhcYzg0W2gB0esFaPy0CJJoHLtS3LusQ1jEM+Q7bTrifB1B30LzT7fxY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1624030668; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=b497Plty6INS5DVpmKNuonj9C7Ksw1Na97HVdIQJWVo=; b=DmCNmhJqv9XEpvfr5jlYhWu9xveSBmMcH0P/qMQ9SeJUkpIlopjQQT9IvltTIeOZeBncIX/HCWN9qe3fqScV9t7wNR44t5bVvXS4hnP3yrdUXCe9orw1C6DLqIJnSg2NoziVJ6ieH27ICxYXdwTdirjD1u2swoJI2vpQ6DMT3Tk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1624030668680779.7995355807257; Fri, 18 Jun 2021 08:37:48 -0700 (PDT) Received: from localhost ([::1]:53210 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1luGZ1-0002no-HY for importer@patchew.org; Fri, 18 Jun 2021 11:37:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41190) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1luGWb-0006Hh-JD for qemu-devel@nongnu.org; Fri, 18 Jun 2021 11:35:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:32100) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1luGWZ-0004UC-AU for qemu-devel@nongnu.org; Fri, 18 Jun 2021 11:35:17 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-527-nJnlRGl1ONSpPZVvCWCngA-1; Fri, 18 Jun 2021 11:35:13 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1057A9F92A; Fri, 18 Jun 2021 15:35:12 +0000 (UTC) Received: from [172.30.41.16] (ovpn-112-106.phx2.redhat.com [10.3.112.106]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5EBA41001281; Fri, 18 Jun 2021 15:35:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624030514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b497Plty6INS5DVpmKNuonj9C7Ksw1Na97HVdIQJWVo=; b=aDMmxllWYQflErGjs8fgbm6+kdGQ6Y6KwigyNeGmJTyBoYpmgvgn4LsGjzUXle0kFfizNJ I8FWfJcmytgJzVPvsinPUwfBGgAIDIbhyOHHadU+mHs3oYv0UHAvb/O+30MWZArtbCu6LG +fwErPYRY4Fl3wcBzFSf5hE23tqMXHA= X-MC-Unique: nJnlRGl1ONSpPZVvCWCngA-1 Subject: [PULL 3/3] vfio/migration: Correct device state from vmstate change for savevm case From: Alex Williamson To: qemu-devel@nongnu.org Date: Fri, 18 Jun 2021 09:35:05 -0600 Message-ID: <162403050520.1793226.14555349399059942330.stgit@omen> In-Reply-To: <162403041204.1793226.16018359908572247606.stgit@omen> References: <162403041204.1793226.16018359908572247606.stgit@omen> User-Agent: StGit/1.0-8-g6af9-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.194, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , alex.williamson@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) From: Kirti Wankhede Set _SAVING flag for device state from vmstate change handler when it gets called from savevm. Currently State transition savevm/suspend is seen as: _RUNNING -> _STOP -> Stop-and-copy -> _STOP State transition savevm/suspend should be: _RUNNING -> Stop-and-copy -> _STOP State transition from _RUNNING to _STOP occurs from vfio_vmstate_change() where when vmstate changes from running to !running, _RUNNING flag is reset but at the same time when vfio_vmstate_change() is called for RUN_STATE_SAVE_VM, _SAVING bit should be set. Reported by: Yishai Hadas Signed-off-by: Kirti Wankhede Message-Id: <1623177441-27496-1-git-send-email-kwankhede@nvidia.com> Signed-off-by: Alex Williamson --- hw/vfio/migration.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index ef397ebe6c09..82f654afb64a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -724,7 +724,16 @@ static void vfio_vmstate_change(void *opaque, bool run= ning, RunState state) * _RUNNING bit */ mask =3D ~VFIO_DEVICE_STATE_RUNNING; - value =3D 0; + + /* + * When VM state transition to stop for savevm command, device sho= uld + * start saving data. + */ + if (state =3D=3D RUN_STATE_SAVE_VM) { + value =3D VFIO_DEVICE_STATE_SAVING; + } else { + value =3D 0; + } } =20 ret =3D vfio_migration_set_state(vbasedev, mask, value);