From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819416; cv=none; d=zohomail.com; s=zohoarc; b=aWHq6VJVa27N5kpMJ7eAtTBJrMSzuEtEgxLkQXcxmE/cKw1wctL+ij+3G3PlWewu18N9KiCYndjBPCsz48UC2OOazQvpPHJGSXDQWe/IvDY5of+9wjebLMSYHe2cH7gsmOIBmGDQcHr6WwRAtn8V+wQXOuVgKRVgWoh/Ya8gaRM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819416; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=xtSQlqXVxqY0Ii7mEm8QI0dDKK/m5fS9hZu49+0IBfE=; b=Y67QQlvyW7fuMbi+qnAW9HHOX1iTgQdM6DqHNxjcAaF5wuH7iECWNavbwKG4I5QPHwNwjpfFNYcqMkajf2M3IlY+R/yHWKt8A40KtNQQFTkD6wlopD5NRGSSPBhQOG9BO5HEg5wmGnoG/Q73lx39Abyjaskbjv+st4L+EbbmVwE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819416031141.79398867154657; Tue, 22 Sep 2020 17:03:36 -0700 (PDT) Received: from localhost ([::1]:46136 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsFy-0003XW-Pe for importer@patchew.org; Tue, 22 Sep 2020 20:03:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46782) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsEk-0002Ku-OU for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:18 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3716) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsEi-000176-Ut for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:18 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:56:25 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:03 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:56:55 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 01/17] vfio: Add function to unmap VFIO region Date: Wed, 23 Sep 2020 04:54:03 +0530 Message-ID: <1600817059-26721-2-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600818985; bh=xtSQlqXVxqY0Ii7mEm8QI0dDKK/m5fS9hZu49+0IBfE=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=YISvtgBgQoaXWd2XLE80YTCBb3/umSiy8YnhoHJKcQuuf0+O3xIbEVfm1iyHAIKfZ 2fYt5CjZivygfJOUrMTY9jDtS82x+B7KhUjOCx6OG5quukpW0KKYampGaiV7fhoo9+ QQrma8eim6Yx9R40WwnWspl4zGFaasfT3AdSa8qKBACHkJoptEx8KT18JXIEX7oNB/ 2MtXymvrrkd2pMiW+OirU7+Hi7Uga9HZNYFSUudvksnhV244on18/q3e7DNKdoTnya 7dfP/bUrFjXxfjuFm5lPNsxjF0YQda35lPd3GTA6+AroCzZ7yiO+xDhY0aluLh566+ CMPHOiLKSVIeA== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.64; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate25.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:12 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This function will be used for migration region. Migration region is mmaped when migration starts and will be unmapped when migration is complete. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Reviewed-by: Cornelia Huck --- hw/vfio/common.c | 32 ++++++++++++++++++++++++++++---- hw/vfio/trace-events | 1 + include/hw/vfio/vfio-common.h | 1 + 3 files changed, 30 insertions(+), 4 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 13471ae29436..c6e98b8d61be 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -924,6 +924,18 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasede= v, VFIORegion *region, return 0; } =20 +static void vfio_subregion_unmap(VFIORegion *region, int index) +{ + trace_vfio_region_unmap(memory_region_name(®ion->mmaps[index].mem), + region->mmaps[index].offset, + region->mmaps[index].offset + + region->mmaps[index].size - 1); + memory_region_del_subregion(region->mem, ®ion->mmaps[index].mem); + munmap(region->mmaps[index].mmap, region->mmaps[index].size); + object_unparent(OBJECT(®ion->mmaps[index].mem)); + region->mmaps[index].mmap =3D NULL; +} + int vfio_region_mmap(VFIORegion *region) { int i, prot =3D 0; @@ -954,10 +966,7 @@ int vfio_region_mmap(VFIORegion *region) region->mmaps[i].mmap =3D NULL; =20 for (i--; i >=3D 0; i--) { - memory_region_del_subregion(region->mem, ®ion->mmaps[i]= .mem); - munmap(region->mmaps[i].mmap, region->mmaps[i].size); - object_unparent(OBJECT(®ion->mmaps[i].mem)); - region->mmaps[i].mmap =3D NULL; + vfio_subregion_unmap(region, i); } =20 return ret; @@ -982,6 +991,21 @@ int vfio_region_mmap(VFIORegion *region) return 0; } =20 +void vfio_region_unmap(VFIORegion *region) +{ + int i; + + if (!region->mem) { + return; + } + + for (i =3D 0; i < region->nr_mmaps; i++) { + if (region->mmaps[i].mmap) { + vfio_subregion_unmap(region, i); + } + } +} + void vfio_region_exit(VFIORegion *region) { int i; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 93a0bc2522f8..a0c7b49a2ebc 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -113,6 +113,7 @@ vfio_region_mmap(const char *name, unsigned long offset= , unsigned long end) "Reg vfio_region_exit(const char *name, int index) "Device %s, region %d" vfio_region_finalize(const char *name, int index) "Device %s, region %d" vfio_region_mmaps_set_enabled(const char *name, bool enabled) "Region %s m= maps enabled: %d" +vfio_region_unmap(const char *name, unsigned long offset, unsigned long en= d) "Region %s unmap [0x%lx - 0x%lx]" vfio_region_sparse_mmap_header(const char *name, int index, int nr_areas) = "Device %s region %d: %d sparse mmap entries" vfio_region_sparse_mmap_entry(int i, unsigned long start, unsigned long en= d) "sparse entry %d [0x%lx - 0x%lx]" vfio_get_dev_region(const char *name, int index, uint32_t type, uint32_t s= ubtype) "%s index %d, %08x/%0x8" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index c78f3ff5593c..dc95f527b583 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -171,6 +171,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev= , VFIORegion *region, int index, const char *name); int vfio_region_mmap(VFIORegion *region); void vfio_region_mmaps_set_enabled(VFIORegion *region, bool enabled); +void vfio_region_unmap(VFIORegion *region); void vfio_region_exit(VFIORegion *region); void vfio_region_finalize(VFIORegion *region); void vfio_reset_handler(void *opaque); --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819575; cv=none; d=zohomail.com; s=zohoarc; b=AwQ7mBj1lUBDAImLRqtxBCtTCydbVIu3cORjWeUqlMB8aIFuSdQWLXGiH2OIH6wQX4rwcxm8X6TmorQuJEHCwQqVeP8pRV5OST9KO8rcS5dW/zGDvVUQ8bCTeUrHbbqnaM9c+Oc6QF0Y39FWT/LCTHJa04pAiWI2BEUWDuam8Fk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819575; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=mU8d8K+D3qa+ZRgdEnJg1v/fWKHItlS0NCXLsW8dowA=; b=O880H8ECz8OMgcQ+rFriigHR8bAB5dR2+T39T9cR0V7vL7HyAF8IN05CnX6ALkBWJf+mYq7/xaEMXPl6v4C45yOxRF+HjIm2RibyAn/UY+8J8GB1MNkQ61aPyuts/RZdroy6E+TFUCnQ0Lz1+gCDjgLP/twQUgA8gRxqKH/9BhQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819575061780.0466133354956; Tue, 22 Sep 2020 17:06:15 -0700 (PDT) Received: from localhost ([::1]:54524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsIX-000774-Qv for importer@patchew.org; Tue, 22 Sep 2020 20:06:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46798) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsEm-0002Mx-Mt for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:20 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14414) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsEk-00017I-VL for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:20 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:57:03 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:10 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:03 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 02/17] vfio: Add vfio_get_object callback to VFIODeviceOps Date: Wed, 23 Sep 2020 04:54:04 +0530 Message-ID: <1600817059-26721-3-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819023; bh=mU8d8K+D3qa+ZRgdEnJg1v/fWKHItlS0NCXLsW8dowA=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=JueQ5JIGHPESw32Mc4iuz6ocB1qKod3ge88xERlPeG1bAxVPHqeR9z9TfJd8tDbPn TV/CgqwGcCgaYDulZbYFjahxdEkpTshyUV4MYs1ubjySx8W4dGxWCJAx1/uFju8dkg NpyNSNqtAXf6zXT0vQ5iAugfV5dkPBSJBuoUnLQwBwfQtdVpTRVaTZQbAchq9ADtJr PCYD8zZyggFC/HXgtpid3GSKWd+ram7FEwtJtECN6LCl7737T97g0AGTFYeuXgUOqA Hm/z60+7ahoKcYEzM8R7iDOS87eYN4CYmO38+Yld+z9OJBsTYz/h3HWw26gAzR+kTl DVJN+YWGty66w== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hook vfio_get_object callback for PCI devices. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Suggested-by: Cornelia Huck Reviewed-by: Cornelia Huck --- hw/vfio/pci.c | 8 ++++++++ include/hw/vfio/vfio-common.h | 1 + 2 files changed, 9 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 0d83eb0e47bb..bffd5bfe3b78 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2394,10 +2394,18 @@ static void vfio_pci_compute_needs_reset(VFIODevice= *vbasedev) } } =20 +static Object *vfio_pci_get_object(VFIODevice *vbasedev) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + + return OBJECT(vdev); +} + static VFIODeviceOps vfio_pci_ops =3D { .vfio_compute_needs_reset =3D vfio_pci_compute_needs_reset, .vfio_hot_reset_multi =3D vfio_pci_hot_reset_multi, .vfio_eoi =3D vfio_intx_eoi, + .vfio_get_object =3D vfio_pci_get_object, }; =20 int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index dc95f527b583..fe99c36a693a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -119,6 +119,7 @@ struct VFIODeviceOps { void (*vfio_compute_needs_reset)(VFIODevice *vdev); int (*vfio_hot_reset_multi)(VFIODevice *vdev); void (*vfio_eoi)(VFIODevice *vdev); + Object *(*vfio_get_object)(VFIODevice *vdev); }; =20 typedef struct VFIOGroup { --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819441; cv=none; d=zohomail.com; s=zohoarc; b=QwlSELxlekWcvbo3/0hHhKrcjSin4OXrXeeUB5f3LfFJH9sxMj1gWBZAx0saayp6X46TY2WhH5YtrID94+sFe4QXVujTm+S26PLEUQ5ErOCDBHDhW82RAHZxZsDUKFtBr6eb68EpWL05TFZJx8bRyrpiWpQ37GAmB5/Ouyj23q0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819441; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=fVVKupjBp5nVINBHcjYaq+u4sJhtvT0UMkUJdqNaspU=; b=fCCIz1J9Jz+jXozP0nzp9K9ULJdsQOBAC06Lav9wtijc9mVURX8RlsaeEkFqs/gHcbfY9ijeBc9ObO28vAR74sImZLf7fcWgrMNvtaLAPXJkXP3IEdZCn4v+2ZIrrKyeWSy7IR7t12zbaDslwN21aJfGnkCYn9xwh9C3d/IWW9s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819441138531.8848485039704; Tue, 22 Sep 2020 17:04:01 -0700 (PDT) Received: from localhost ([::1]:47250 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsGN-0003zI-QV for importer@patchew.org; Tue, 22 Sep 2020 20:03:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46836) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsEx-0002Xb-7B for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:31 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3726) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsEv-00018A-4E for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:30 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:56:39 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:19 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:11 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 03/17] vfio: Add save and load functions for VFIO PCI devices Date: Wed, 23 Sep 2020 04:54:05 +0530 Message-ID: <1600817059-26721-4-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600818999; bh=fVVKupjBp5nVINBHcjYaq+u4sJhtvT0UMkUJdqNaspU=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=GFPoHMFGTVrS2FasKVCdR8inmPaJGGsf2jpyq3QXt8J5Hg1ZLqu+8ZWQWN4tHGTsJ 1cRavVdf2Di7fDRITLtPpjq1w4DnWaW/2KzYGlJx7LMVZYG0coEMg5Vxx1iAKzntkC TCbDL/yDPKhE3fIIuJ7NaL+W7/UtQ+c+d7cllrxQsvOmn+8OQcBvE4n99xl4U0F1+S fEuBgG+qr40xVtv/rcb66C4Vl40lahTmw9MpnjQhBY4Z9qoR9ZYjgov+KyT6rZkOLD 6qmENx3BtufapbGo81+t1dK9pTcaj783KE9PsY+R9DV7mK9nFN10gB3dBmJ3VhGndC fWCSddUGkE3nw== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.64; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate25.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:12 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" These functions save and restore PCI device specific data - config space of PCI device. Used VMStateDescription to save and restore interrupt state. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/pci.c | 134 ++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 2 + 3 files changed, 137 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bffd5bfe3b78..9968cc553391 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -41,6 +41,7 @@ #include "trace.h" #include "qapi/error.h" #include "migration/blocker.h" +#include "migration/qemu-file.h" =20 #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" =20 @@ -2401,11 +2402,142 @@ static Object *vfio_pci_get_object(VFIODevice *vba= sedev) return OBJECT(vdev); } =20 +static int vfio_get_pci_irq_state(QEMUFile *f, void *pv, size_t size, + const VMStateField *field) +{ + VFIOPCIDevice *vdev =3D container_of(pv, VFIOPCIDevice, vbasedev); + PCIDevice *pdev =3D &vdev->pdev; + uint32_t interrupt_type; + + interrupt_type =3D qemu_get_be32(f); + + if (interrupt_type =3D=3D VFIO_INT_MSI) { + uint32_t msi_flags, msi_addr_lo, msi_addr_hi =3D 0, msi_data; + bool msi_64bit; + + /* restore msi configuration */ + msi_flags =3D pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_FLAGS,= 2); + msi_64bit =3D (msi_flags & PCI_MSI_FLAGS_64BIT); + + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags & ~PCI_MSI_FLAGS_ENABLE, 2); + + msi_addr_lo =3D pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_LO= , 4); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_LO, + msi_addr_lo, 4); + + if (msi_64bit) { + msi_addr_hi =3D pci_default_read_config(pdev, + pdev->msi_cap + PCI_MSI_ADDRESS_HI= , 4); + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_ADDRESS_HI, + msi_addr_hi, 4); + } + + msi_data =3D pci_default_read_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DAT= A_32), + 2); + + vfio_pci_write_config(pdev, + pdev->msi_cap + (msi_64bit ? PCI_MSI_DATA_64 : PCI_MSI_DAT= A_32), + msi_data, 2); + + vfio_pci_write_config(pdev, pdev->msi_cap + PCI_MSI_FLAGS, + msi_flags | PCI_MSI_FLAGS_ENABLE, 2); + } else if (interrupt_type =3D=3D VFIO_INT_MSIX) { + uint16_t offset; + + msix_load(pdev, f); + offset =3D pci_default_read_config(pdev, + pdev->msix_cap + PCI_MSIX_FLAGS + 1= , 2); + /* load enable bit and maskall bit */ + vfio_pci_write_config(pdev, pdev->msix_cap + PCI_MSIX_FLAGS + 1, + offset, 2); + } + return 0; +} + +static int vfio_put_pci_irq_state(QEMUFile *f, void *pv, size_t size, + const VMStateField *field, QJSON *vmdesc) +{ + VFIOPCIDevice *vdev =3D container_of(pv, VFIOPCIDevice, vbasedev); + PCIDevice *pdev =3D &vdev->pdev; + + qemu_put_be32(f, vdev->interrupt); + if (vdev->interrupt =3D=3D VFIO_INT_MSIX) { + msix_save(pdev, f); + } + + return 0; +} + +static const VMStateInfo vmstate_info_vfio_pci_irq_state =3D { + .name =3D "VFIO PCI irq state", + .get =3D vfio_get_pci_irq_state, + .put =3D vfio_put_pci_irq_state, +}; + +const VMStateDescription vmstate_vfio_pci_config =3D { + .name =3D "VFIOPCIDevice", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (VMStateField[]) { + VMSTATE_INT32_POSITIVE_LE(version_id, VFIOPCIDevice), + VMSTATE_BUFFER_UNSAFE_INFO(interrupt, VFIOPCIDevice, 1, + vmstate_info_vfio_pci_irq_state, + sizeof(int32_t)), + VMSTATE_END_OF_LIST() + } +}; + +static void vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + PCIDevice *pdev =3D &vdev->pdev; + + + pci_device_save(pdev, f); + vmstate_save_state(f, &vmstate_vfio_pci_config, vbasedev, NULL); +} + +static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + PCIDevice *pdev =3D &vdev->pdev; + uint16_t pci_cmd; + int ret, i; + + ret =3D pci_device_load(pdev, f); + if (ret) { + return ret; + } + + /* retore pci bar configuration */ + pci_cmd =3D pci_default_read_config(pdev, PCI_COMMAND, 2); + vfio_pci_write_config(pdev, PCI_COMMAND, + pci_cmd & ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY), = 2); + for (i =3D 0; i < PCI_ROM_SLOT; i++) { + uint32_t bar =3D pci_default_read_config(pdev, + PCI_BASE_ADDRESS_0 + i * 4,= 4); + + vfio_pci_write_config(pdev, PCI_BASE_ADDRESS_0 + i * 4, bar, 4); + } + + ret =3D vmstate_load_state(f, &vmstate_vfio_pci_config, vbasedev, + vdev->version_id); + + vfio_pci_write_config(pdev, PCI_COMMAND, pci_cmd, 2); + return ret; +} + static VFIODeviceOps vfio_pci_ops =3D { .vfio_compute_needs_reset =3D vfio_pci_compute_needs_reset, .vfio_hot_reset_multi =3D vfio_pci_hot_reset_multi, .vfio_eoi =3D vfio_intx_eoi, .vfio_get_object =3D vfio_pci_get_object, + .vfio_save_config =3D vfio_pci_save_config, + .vfio_load_config =3D vfio_pci_load_config, }; =20 int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) @@ -2755,6 +2887,8 @@ static void vfio_realize(PCIDevice *pdev, Error **err= p) vdev->vbasedev.ops =3D &vfio_pci_ops; vdev->vbasedev.type =3D VFIO_DEVICE_TYPE_PCI; vdev->vbasedev.dev =3D DEVICE(vdev); + vdev->vbasedev.device_state =3D 0; + vdev->version_id =3D 1; =20 tmp =3D g_strdup_printf("%s/iommu_group", vdev->vbasedev.sysfsdev); len =3D readlink(tmp, group_path, sizeof(group_path)); diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index bce71a9ac93f..9f46af7e153f 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -156,6 +156,7 @@ struct VFIOPCIDevice { uint32_t display_yres; int32_t bootindex; uint32_t igd_gms; + int32_t version_id; /* Version id needed for VMState */ OffAutoPCIBAR msix_relo; uint8_t pm_cap; uint8_t nv_gpudirect_clique; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index fe99c36a693a..ba6169cd926e 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -120,6 +120,8 @@ struct VFIODeviceOps { int (*vfio_hot_reset_multi)(VFIODevice *vdev); void (*vfio_eoi)(VFIODevice *vdev); Object *(*vfio_get_object)(VFIODevice *vdev); + void (*vfio_save_config)(VFIODevice *vdev, QEMUFile *f); + int (*vfio_load_config)(VFIODevice *vdev, QEMUFile *f); }; =20 typedef struct VFIOGroup { --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819585; cv=none; d=zohomail.com; s=zohoarc; b=j2x/jJuP8VN5SguW0tIUqvNrSeL/93X4dMCwBdiXLmTJysGCUEoBWwMsscGbQRIkOvpea3XoRRNpK8r52gWkpjt04DNuFOhnnsj/gIPi55FVldXaPoTO5/ttp09IEvBTYJAProYLMqVAT+71ddbPA1ooaQdaxWV5+L8N5g6K+aA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819585; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=QOkO4BMHMe8fPAicgB4DmNBsdIcbCRTqDtygCgaf2vI=; b=maTZB4SSHCuxQGWShq9xlSetgPY7WK9fL9SUNzgbI868MOKfChvenfh1VY0G45voRvnotIczDU9tHyfrLIw5w7vNGhp56zqvMSL8Lyz5XxGpVyC/ThB2m2yBvo6CCGBXi6mv7AIpN+4jGHftC3nJDXgb6Y7wvBDJ2T+15BCrZWo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 160081958557272.29862783066028; Tue, 22 Sep 2020 17:06:25 -0700 (PDT) Received: from localhost ([::1]:55658 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsIi-0007YT-9l for importer@patchew.org; Tue, 22 Sep 2020 20:06:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46872) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFD-0002tq-RE for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:47 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14433) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFB-00018e-Hp for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:47 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:57:21 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:27 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:19 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 04/17] vfio: Add migration region initialization and finalize function Date: Wed, 23 Sep 2020 04:54:06 +0530 Message-ID: <1600817059-26721-5-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819041; bh=QOkO4BMHMe8fPAicgB4DmNBsdIcbCRTqDtygCgaf2vI=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=UJ6bWxpXt7vYBuhcREqsmOt4et+kwngybr8hB6V14Xh6wswaLfc9rgQKhBug/L6Qh zGnuam/dzs856+Xn2kmDWH9p5P6XzuqNsONqPxfdt+OlTAiI8CPi9/Icu2VekMqTPn VzpzmF5UOgrrUgWa7JyqMNPunX+y3DJKXr5rV0gQBTuT+dcMHjABCLwzXz3H/7tqBq WgxHBoByJ9Vv+RmDtunMrR/E14DG/HERSNOoex53G38NQLEvyN2XZ4V8MYriP97qqG Y1j5IhmBT0z+TuTw0V2E4Mgl9OCkpQ/RlHhqiL5MaBIY7Af1eZJlEPVQkR8/SczWeZ g0idBbS5m+zJQ== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Whether the VFIO device supports migration or not is decided based of migration region query. If migration region query is successful and migrati= on region initialization is successful then migration is supported else migration is blocked. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Acked-by: Dr. David Alan Gilbert --- hw/vfio/meson.build | 1 + hw/vfio/migration.c | 142 ++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/trace-events | 5 ++ include/hw/vfio/vfio-common.h | 9 +++ 4 files changed, 157 insertions(+) create mode 100644 hw/vfio/migration.c diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index 37efa74018bc..da9af297a0c5 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -2,6 +2,7 @@ vfio_ss =3D ss.source_set() vfio_ss.add(files( 'common.c', 'spapr.c', + 'migration.c', )) vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c new file mode 100644 index 000000000000..2f760f1f9c47 --- /dev/null +++ b/hw/vfio/migration.c @@ -0,0 +1,142 @@ +/* + * Migration support for VFIO devices + * + * Copyright NVIDIA, Inc. 2020 + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include + +#include "hw/vfio/vfio-common.h" +#include "cpu.h" +#include "migration/migration.h" +#include "migration/qemu-file.h" +#include "migration/register.h" +#include "migration/blocker.h" +#include "migration/misc.h" +#include "qapi/error.h" +#include "exec/ramlist.h" +#include "exec/ram_addr.h" +#include "pci.h" +#include "trace.h" + +static void vfio_migration_region_exit(VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + + if (!migration) { + return; + } + + if (migration->region.size) { + vfio_region_exit(&migration->region); + vfio_region_finalize(&migration->region); + } +} + +static int vfio_migration_region_init(VFIODevice *vbasedev, int index) +{ + VFIOMigration *migration =3D vbasedev->migration; + Object *obj =3D NULL; + int ret =3D -EINVAL; + + obj =3D vbasedev->ops->vfio_get_object(vbasedev); + if (!obj) { + return ret; + } + + ret =3D vfio_region_setup(obj, vbasedev, &migration->region, index, + "migration"); + if (ret) { + error_report("%s: Failed to setup VFIO migration region %d: %s", + vbasedev->name, index, strerror(-ret)); + goto err; + } + + if (!migration->region.size) { + ret =3D -EINVAL; + error_report("%s: Invalid region size of VFIO migration region %d:= %s", + vbasedev->name, index, strerror(-ret)); + goto err; + } + + return 0; + +err: + vfio_migration_region_exit(vbasedev); + return ret; +} + +static int vfio_migration_init(VFIODevice *vbasedev, + struct vfio_region_info *info) +{ + int ret =3D -EINVAL; + + if (!vbasedev->ops->vfio_get_object) { + return ret; + } + + vbasedev->migration =3D g_new0(VFIOMigration, 1); + + ret =3D vfio_migration_region_init(vbasedev, info->index); + if (ret) { + error_report("%s: Failed to initialise migration region", + vbasedev->name); + g_free(vbasedev->migration); + vbasedev->migration =3D NULL; + } + + return ret; +} + +/* ---------------------------------------------------------------------- = */ + +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) +{ + struct vfio_region_info *info =3D NULL; + Error *local_err =3D NULL; + int ret; + + ret =3D vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION, + VFIO_REGION_SUBTYPE_MIGRATION, &info); + if (ret) { + goto add_blocker; + } + + ret =3D vfio_migration_init(vbasedev, info); + if (ret) { + goto add_blocker; + } + + g_free(info); + trace_vfio_migration_probe(vbasedev->name, info->index); + return 0; + +add_blocker: + error_setg(&vbasedev->migration_blocker, + "VFIO device doesn't support migration"); + g_free(info); + + ret =3D migrate_add_blocker(vbasedev->migration_blocker, &local_err); + if (local_err) { + error_propagate(errp, local_err); + error_free(vbasedev->migration_blocker); + vbasedev->migration_blocker =3D NULL; + } + return ret; +} + +void vfio_migration_finalize(VFIODevice *vbasedev) +{ + if (vbasedev->migration_blocker) { + migrate_del_blocker(vbasedev->migration_blocker); + error_free(vbasedev->migration_blocker); + vbasedev->migration_blocker =3D NULL; + } + + vfio_migration_region_exit(vbasedev); + g_free(vbasedev->migration); +} diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index a0c7b49a2ebc..8fe913175d85 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -145,3 +145,8 @@ vfio_display_edid_link_up(void) "" vfio_display_edid_link_down(void) "" vfio_display_edid_update(uint32_t prefx, uint32_t prefy) "%ux%u" vfio_display_edid_write_error(void) "" + + +# migration.c +vfio_migration_probe(const char *name, uint32_t index) " (%s) Region %d" + diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ba6169cd926e..8275c4c68f45 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -57,6 +57,10 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; =20 +typedef struct VFIOMigration { + VFIORegion region; +} VFIOMigration; + typedef struct VFIOAddressSpace { AddressSpace *as; QLIST_HEAD(, VFIOContainer) containers; @@ -113,6 +117,8 @@ typedef struct VFIODevice { unsigned int num_irqs; unsigned int num_regions; unsigned int flags; + VFIOMigration *migration; + Error *migration_blocker; } VFIODevice; =20 struct VFIODeviceOps { @@ -204,4 +210,7 @@ int vfio_spapr_create_window(VFIOContainer *container, int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); =20 +int vfio_migration_probe(VFIODevice *vbasedev, Error **errp); +void vfio_migration_finalize(VFIODevice *vbasedev); + #endif /* HW_VFIO_VFIO_COMMON_H */ --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819569; cv=none; d=zohomail.com; s=zohoarc; b=LZ+ZtdCaPr5UzCBcBck2fkv0zmJvcsbmKuwA2vOVKUadmrpSlhxcAClseQl6KnFwG8C21IuSABQPHTX1sjRmn9SzRwGP12TZtYKdFLfenl2iVkp3dQS/pBEB3wXocvRUKtgaHvNW+YGGSxSc1Wr+oPCNZ8cNDtO8aeXpWND7N4w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819569; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pS2lzl3wGvfvvw14J04f3Hh1oA7ekgCbp5LTVJUVOns=; b=A/H22AZNKbGKdYGsOqTHaeOJTb8hRaSsYjC9TM1Qb0QMfZ6JYA9h5l5NBw7RVGA4SWZygBJKu3MqqirL7j21IjhA+1WNgLQU+lwb4ZFHBjkuPsridCNqaltPOUL58T5M2sJh9K9PQA05BY/zafdSzpPI1A5FItiE60D1paznrZI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819569928230.16230947142458; Tue, 22 Sep 2020 17:06:09 -0700 (PDT) Received: from localhost ([::1]:54156 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsIS-0006wK-Gd for importer@patchew.org; Tue, 22 Sep 2020 20:06:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46888) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFH-00030X-07 for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:51 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14435) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFE-0001A9-Q1 for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:02:50 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:57:32 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:35 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:27 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 05/17] vfio: Add VM state change handler to know state of VM Date: Wed, 23 Sep 2020 04:54:07 +0530 Message-ID: <1600817059-26721-6-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819052; bh=pS2lzl3wGvfvvw14J04f3Hh1oA7ekgCbp5LTVJUVOns=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=D2xyqNxpo09f+FpiLPfK2Tq2yEB/fy9Q1eZXeipXrLV1xXYuxhtxspyhqfRt3Xijx Ai9aRPEvJGZyNQ7IG27zw1TzPjXx0IQE45PS6EiEj8p4DrVFMPEhObKUy9X9/i/bXN XUx6kUw1v0XdNhTAQH/MamJ/dMo07L7h3R5/UbQA4CsCGmK3dy9QB9lihCtVGN5v8s As2xp5o7/IGNdHzpQo3degYYVqXUB7Bdt9r7evos22OOF6qbmlSbeA6vxuRJ0LCn9G +v+lRYxFtIV5pNxuLMJr0cqd6BllUO5CVV3YR7apyOfz5IfalfdvT1FIv3XVPu0KDD gAOPr4JVshqIg== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" VM state change handler gets called on change in VM's state. This is used t= o set VFIO device state to _RUNNING. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Reviewed-by: Dr. David Alan Gilbert --- hw/vfio/migration.c | 136 ++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/trace-events | 3 +- include/hw/vfio/vfio-common.h | 4 ++ 3 files changed, 142 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 2f760f1f9c47..a30d628ba963 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -10,6 +10,7 @@ #include "qemu/osdep.h" #include =20 +#include "sysemu/runstate.h" #include "hw/vfio/vfio-common.h" #include "cpu.h" #include "migration/migration.h" @@ -22,6 +23,58 @@ #include "exec/ram_addr.h" #include "pci.h" #include "trace.h" +#include "hw/hw.h" + +static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int cou= nt, + off_t off, bool iswrite) +{ + int ret; + + ret =3D iswrite ? pwrite(vbasedev->fd, val, count, off) : + pread(vbasedev->fd, val, count, off); + if (ret < count) { + error_report("vfio_mig_%s%d %s: failed at offset 0x%lx, err: %s", + iswrite ? "write" : "read", count * 8, + vbasedev->name, off, strerror(errno)); + return (ret < 0) ? ret : -EINVAL; + } + return 0; +} + +static int vfio_mig_rw(VFIODevice *vbasedev, __u8 *buf, size_t count, + off_t off, bool iswrite) +{ + int ret, done =3D 0; + __u8 *tbuf =3D buf; + + while (count) { + int bytes =3D 0; + + if (count >=3D 8 && !(off % 8)) { + bytes =3D 8; + } else if (count >=3D 4 && !(off % 4)) { + bytes =3D 4; + } else if (count >=3D 2 && !(off % 2)) { + bytes =3D 2; + } else { + bytes =3D 1; + } + + ret =3D vfio_mig_access(vbasedev, tbuf, bytes, off, iswrite); + if (ret) { + return ret; + } + + count -=3D bytes; + done +=3D bytes; + off +=3D bytes; + tbuf +=3D bytes; + } + return done; +} + +#define vfio_mig_read(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, fa= lse) +#define vfio_mig_write(f, v, c, o) vfio_mig_rw(f, (__u8 *)v, c, o, tr= ue) =20 static void vfio_migration_region_exit(VFIODevice *vbasedev) { @@ -70,6 +123,82 @@ err: return ret; } =20 +static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, + uint32_t value) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region; + off_t dev_state_off =3D region->fd_offset + + offsetof(struct vfio_device_migration_info, device_s= tate); + uint32_t device_state; + int ret; + + ret =3D vfio_mig_read(vbasedev, &device_state, sizeof(device_state), + dev_state_off); + if (ret < 0) { + return ret; + } + + device_state =3D (device_state & mask) | value; + + if (!VFIO_DEVICE_STATE_VALID(device_state)) { + return -EINVAL; + } + + ret =3D vfio_mig_write(vbasedev, &device_state, sizeof(device_state), + dev_state_off); + if (ret < 0) { + ret =3D vfio_mig_read(vbasedev, &device_state, sizeof(device_state= ), + dev_state_off); + if (ret < 0) { + return ret; + } + + if (VFIO_DEVICE_STATE_IS_ERROR(device_state)) { + hw_error("%s: Device is in error state 0x%x", + vbasedev->name, device_state); + return -EFAULT; + } + } + + vbasedev->device_state =3D device_state; + trace_vfio_migration_set_state(vbasedev->name, device_state); + return 0; +} + +static void vfio_vmstate_change(void *opaque, int running, RunState state) +{ + VFIODevice *vbasedev =3D opaque; + + if ((vbasedev->vm_running !=3D running)) { + int ret; + uint32_t value =3D 0, mask =3D 0; + + if (running) { + value =3D VFIO_DEVICE_STATE_RUNNING; + if (vbasedev->device_state & VFIO_DEVICE_STATE_RESUMING) { + mask =3D ~VFIO_DEVICE_STATE_RESUMING; + } + } else { + mask =3D ~VFIO_DEVICE_STATE_RUNNING; + } + + ret =3D vfio_migration_set_state(vbasedev, mask, value); + if (ret) { + /* + * vm_state_notify() doesn't support reporting failure. If such + * error reporting support added in furure, migration should be + * aborted. + */ + error_report("%s: Failed to set device state 0x%x", + vbasedev->name, value & mask); + } + vbasedev->vm_running =3D running; + trace_vfio_vmstate_change(vbasedev->name, running, RunState_str(st= ate), + value & mask); + } +} + static int vfio_migration_init(VFIODevice *vbasedev, struct vfio_region_info *info) { @@ -87,8 +216,11 @@ static int vfio_migration_init(VFIODevice *vbasedev, vbasedev->name); g_free(vbasedev->migration); vbasedev->migration =3D NULL; + return ret; } =20 + vbasedev->vm_state =3D qemu_add_vm_change_state_handler(vfio_vmstate_c= hange, + vbasedev); return ret; } =20 @@ -131,6 +263,10 @@ add_blocker: =20 void vfio_migration_finalize(VFIODevice *vbasedev) { + if (vbasedev->vm_state) { + qemu_del_vm_change_state_handler(vbasedev->vm_state); + } + if (vbasedev->migration_blocker) { migrate_del_blocker(vbasedev->migration_blocker); error_free(vbasedev->migration_blocker); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 8fe913175d85..6524734bf7b4 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -149,4 +149,5 @@ vfio_display_edid_write_error(void) "" =20 # migration.c vfio_migration_probe(const char *name, uint32_t index) " (%s) Region %d" - +vfio_migration_set_state(char *name, uint32_t state) " (%s) state %d" +vfio_vmstate_change(char *name, int running, const char *reason, uint32_t = dev_state) " (%s) running %d reason %s device state %d" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8275c4c68f45..25e3b1a3b90a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -29,6 +29,7 @@ #ifdef CONFIG_LINUX #include #endif +#include "sysemu/sysemu.h" =20 #define VFIO_MSG_PREFIX "vfio %s: " =20 @@ -119,6 +120,9 @@ typedef struct VFIODevice { unsigned int flags; VFIOMigration *migration; Error *migration_blocker; + VMChangeStateEntry *vm_state; + uint32_t device_state; + int vm_running; } VFIODevice; =20 struct VFIODeviceOps { --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819674; cv=none; d=zohomail.com; s=zohoarc; b=KhmmkVIYHKbJSS+F/JL9IYA+GqM+LK1kvvtMMhdDin5e1SuBpbUbP4r5QK1HGuosQZiJ4nkLNU7HZk5WhOZJzHc8wL1s8PbjZidAg449xKXLmmrw81wB7N/tYC1ILqj56IwpIhn6t2Gl1lZP0CS7c998PFyQIyLO4AAXd6yCVE4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819674; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=t25FEilbNETTC91wdWv6zWhCI1IxPHxhtB91G+0YP+8=; b=noAuSZzZlvzpUHCamvCSN+LZbwLw+aUIrOUH6UEtCU9z/o2IiLowXTHkWhO04Dh5/Ldz+w1x3Z5LdDcU204ruYF/2R2c/pornzLb+4cTY9oRsnBQ30Km6vSnpyZNaQmbFFIMZ52rXtshDpAS1Ho/7Yg3Cznd+Hqh5IVXTs/ZbE0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819674050190.85887693986786; Tue, 22 Sep 2020 17:07:54 -0700 (PDT) Received: from localhost ([::1]:34266 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsK8-0001ys-Po for importer@patchew.org; Tue, 22 Sep 2020 20:07:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46934) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFS-0003GN-51 for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:02 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:16927) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFQ-0001AY-9s for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:01 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:56:19 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:44 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:36 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 06/17] vfio: Add migration state change notifier Date: Wed, 23 Sep 2020 04:54:08 +0530 Message-ID: <1600817059-26721-7-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600818979; bh=t25FEilbNETTC91wdWv6zWhCI1IxPHxhtB91G+0YP+8=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=ieyKPfB/XlnmvIiesURO7pfOzy+Xy6L+UnTcswVAPQV4NINZG+l8n59ZGSbMtEyjP rkkUEAF947ybcd7KR/FaLpP7lg+A43AzN2yBK0qArK2MGGPPosJE4KyJGDbGj7i76y OHd5JNu83TfeG7pv01ETAbOIKAd5ebffvZFNCbGFcBOYqXjB6vTnyfU19UPyxZ11rs aZmj78ZKuGIK0r+h65uONb4B7U0khiMHuijB1xT0lkyIW25e/1GweYNOe/dd3tJ61a 6oV4iPbs4qm5eWPEEJbyF0JxKfdRLFmOS/3RD4skNGWN7MIsiUSMICmJoIoG4ea7fw eJjX7YKoPR/9A== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.143; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate24.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:06 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added migration state change notifier to get notification on migration state change. These states are translated to VFIO device state and conveyed to ve= ndor driver. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Reviewed-by: Dr. David Alan Gilbert --- hw/vfio/migration.c | 29 +++++++++++++++++++++++++++++ hw/vfio/trace-events | 5 +++-- include/hw/vfio/vfio-common.h | 1 + 3 files changed, 33 insertions(+), 2 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index a30d628ba963..f650fe9fc3c8 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -199,6 +199,28 @@ static void vfio_vmstate_change(void *opaque, int runn= ing, RunState state) } } =20 +static void vfio_migration_state_notifier(Notifier *notifier, void *data) +{ + MigrationState *s =3D data; + VFIODevice *vbasedev =3D container_of(notifier, VFIODevice, migration_= state); + int ret; + + trace_vfio_migration_state_notifier(vbasedev->name, + MigrationStatus_str(s->state)); + + switch (s->state) { + case MIGRATION_STATUS_CANCELLING: + case MIGRATION_STATUS_CANCELLED: + case MIGRATION_STATUS_FAILED: + ret =3D vfio_migration_set_state(vbasedev, + ~(VFIO_DEVICE_STATE_SAVING | VFIO_DEVICE_STATE_RESUM= ING), + VFIO_DEVICE_STATE_RUNNING); + if (ret) { + error_report("%s: Failed to set state RUNNING", vbasedev->name= ); + } + } +} + static int vfio_migration_init(VFIODevice *vbasedev, struct vfio_region_info *info) { @@ -221,6 +243,8 @@ static int vfio_migration_init(VFIODevice *vbasedev, =20 vbasedev->vm_state =3D qemu_add_vm_change_state_handler(vfio_vmstate_c= hange, vbasedev); + vbasedev->migration_state.notify =3D vfio_migration_state_notifier; + add_migration_state_change_notifier(&vbasedev->migration_state); return ret; } =20 @@ -263,6 +287,11 @@ add_blocker: =20 void vfio_migration_finalize(VFIODevice *vbasedev) { + + if (vbasedev->migration_state.notify) { + remove_migration_state_change_notifier(&vbasedev->migration_state); + } + if (vbasedev->vm_state) { qemu_del_vm_change_state_handler(vbasedev->vm_state); } diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 6524734bf7b4..bcb3fa7314d7 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -149,5 +149,6 @@ vfio_display_edid_write_error(void) "" =20 # migration.c vfio_migration_probe(const char *name, uint32_t index) " (%s) Region %d" -vfio_migration_set_state(char *name, uint32_t state) " (%s) state %d" -vfio_vmstate_change(char *name, int running, const char *reason, uint32_t = dev_state) " (%s) running %d reason %s device state %d" +vfio_migration_set_state(const char *name, uint32_t state) " (%s) state %d" +vfio_vmstate_change(const char *name, int running, const char *reason, uin= t32_t dev_state) " (%s) running %d reason %s device state %d" +vfio_migration_state_notifier(const char *name, const char *state) " (%s) = state %s" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 25e3b1a3b90a..49c7c7a0e29a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -123,6 +123,7 @@ typedef struct VFIODevice { VMChangeStateEntry *vm_state; uint32_t device_state; int vm_running; + Notifier migration_state; } VFIODevice; =20 struct VFIODeviceOps { --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819503; cv=none; d=zohomail.com; s=zohoarc; b=iplwQB6HXs+IAQ3dZgRPR0DAtN82IfL/fuGwpNyudAXAHuM+WkzSijHqdpvN9hsLtFrro/kEy/VKswvOai+8bqStGWj7KVLuuqQYG8/Tb/rBDA04Yt37R1EquvtHrhYTXnC/2uMejq4Y0E/wdWmQ7p8hNONUE4GtBewbl0h0l6g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819503; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=HbXeaMzsQUExhDrDtnMu8DlfL4do0L9CUykQLh71TKU=; b=ZdNC2jMCo3PnYQHEXPIgHtQ+hZjNfMKoBU6LVwpe4b2160SiJ0om76eFanizZwStXXU3mHvZ6lDXM12mOGuWKOboJEjqDqCkV3iVfU9z5+4dP5BYVmChLOCmas8L4h3uqY5uw8SEX2wtLqmuZAjAZPgGikrAcMvuBIFSuNf6WrY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819503273699.6075664247751; Tue, 22 Sep 2020 17:05:03 -0700 (PDT) Received: from localhost ([::1]:50012 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsHO-0005Al-1X for importer@patchew.org; Tue, 22 Sep 2020 20:05:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46956) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFW-0003Q3-Lo for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:06 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14446) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFU-0001B7-Ht for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:06 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:57:49 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:57:52 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:44 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 07/17] vfio: Register SaveVMHandlers for VFIO device Date: Wed, 23 Sep 2020 04:54:09 +0530 Message-ID: <1600817059-26721-8-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819069; bh=HbXeaMzsQUExhDrDtnMu8DlfL4do0L9CUykQLh71TKU=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=NGpRToTpov1EhDUX1EtAmOtfcl0vV6yUTXPbrkXbkC5rpTPlMylyDEXS040hwXq6B jF7dS8sN195cO18trsff7hAi3bhm6+VpxjrnSkITsYoNPVnnvDMFJ3PHI15QiwXAi4 px30xhRh/YNzokGJO6Q0mIDQ4lz5SQz0TRdzWThU7KCc4Pw2jhdJsA8qylxw0YQAgW ad3M7zi/83wU5d6V3PPnY2sT9ifgvMUe2iLsGyYXNbFSN/5ayU86Eg18Wy/DZ3/0EX WpEMND3Jt01hOyEblW2diNyYlQspYewuNbj1U/2IJ5xJ8ntkKrSyfA4eBYwmCh2WwH M3nvOequ7HuCg== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define flags to be used as delimeter in migration file stream. Added .save_setup and .save_cleanup functions. Mapped & unmapped migration region from these functions at source during saving or pre-copy phase. Set VFIO device state depending on VM's state. During live migration, VM is running when .save_setup is called, _SAVING | _RUNNING state is set for VFIO device. During save-restore, VM is paused, _SAVING state is set for VFIO de= vice. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/trace-events | 2 ++ 2 files changed, 93 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index f650fe9fc3c8..8e8adaa25779 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -8,12 +8,15 @@ */ =20 #include "qemu/osdep.h" +#include "qemu/main-loop.h" +#include "qemu/cutils.h" #include =20 #include "sysemu/runstate.h" #include "hw/vfio/vfio-common.h" #include "cpu.h" #include "migration/migration.h" +#include "migration/vmstate.h" #include "migration/qemu-file.h" #include "migration/register.h" #include "migration/blocker.h" @@ -25,6 +28,17 @@ #include "trace.h" #include "hw/hw.h" =20 +/* + * Flags used as delimiter: + * 0xffffffff =3D> MSB 32-bit all 1s + * 0xef10 =3D> emulated (virtual) function IO + * 0x0000 =3D> 16-bits reserved for flags + */ +#define VFIO_MIG_FLAG_END_OF_STATE (0xffffffffef100001ULL) +#define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) +#define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) +#define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) + static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int cou= nt, off_t off, bool iswrite) { @@ -166,6 +180,65 @@ static int vfio_migration_set_state(VFIODevice *vbased= ev, uint32_t mask, return 0; } =20 +/* ---------------------------------------------------------------------- = */ + +static int vfio_save_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + int ret; + + trace_vfio_save_setup(vbasedev->name); + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); + + if (migration->region.mmaps) { + qemu_mutex_lock_iothread(); + ret =3D vfio_region_mmap(&migration->region); + qemu_mutex_unlock_iothread(); + if (ret) { + error_report("%s: Failed to mmap VFIO migration region %d: %s", + vbasedev->name, migration->region.nr, + strerror(-ret)); + error_report("%s: Falling back to slow path", vbasedev->name); + } + } + + ret =3D vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("%s: Failed to set state SAVING", vbasedev->name); + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + return 0; +} + +static void vfio_save_cleanup(void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + + if (migration->region.mmaps) { + vfio_region_unmap(&migration->region); + } + trace_vfio_save_cleanup(vbasedev->name); +} + +static SaveVMHandlers savevm_vfio_handlers =3D { + .save_setup =3D vfio_save_setup, + .save_cleanup =3D vfio_save_cleanup, +}; + +/* ---------------------------------------------------------------------- = */ + static void vfio_vmstate_change(void *opaque, int running, RunState state) { VFIODevice *vbasedev =3D opaque; @@ -225,6 +298,8 @@ static int vfio_migration_init(VFIODevice *vbasedev, struct vfio_region_info *info) { int ret =3D -EINVAL; + char id[256] =3D ""; + Object *obj; =20 if (!vbasedev->ops->vfio_get_object) { return ret; @@ -241,6 +316,22 @@ static int vfio_migration_init(VFIODevice *vbasedev, return ret; } =20 + obj =3D vbasedev->ops->vfio_get_object(vbasedev); + + if (obj) { + DeviceState *dev =3D DEVICE(obj); + char *oid =3D vmstate_if_get_id(VMSTATE_IF(dev)); + + if (oid) { + pstrcpy(id, sizeof(id), oid); + pstrcat(id, sizeof(id), "/"); + g_free(oid); + } + } + pstrcat(id, sizeof(id), "vfio"); + + register_savevm_live(id, VMSTATE_INSTANCE_ID_ANY, 1, &savevm_vfio_hand= lers, + vbasedev); vbasedev->vm_state =3D qemu_add_vm_change_state_handler(vfio_vmstate_c= hange, vbasedev); vbasedev->migration_state.notify =3D vfio_migration_state_notifier; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index bcb3fa7314d7..982d8dccb219 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -152,3 +152,5 @@ vfio_migration_probe(const char *name, uint32_t index) = " (%s) Region %d" vfio_migration_set_state(const char *name, uint32_t state) " (%s) state %d" vfio_vmstate_change(const char *name, int running, const char *reason, uin= t32_t dev_state) " (%s) running %d reason %s device state %d" vfio_migration_state_notifier(const char *name, const char *state) " (%s) = state %s" +vfio_save_setup(const char *name) " (%s)" +vfio_save_cleanup(const char *name) " (%s)" --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819788; cv=none; d=zohomail.com; s=zohoarc; b=GwFYn13PvDT765QfYafXjsdR3y4KpjVJ40SZQ9G7MjZWtxC/nz7QGXNlG3Z5PfGmDKP5jGtpKs0JfgC6OZ5ucNCw1Ckb8AZIwqX/44gfKLYbXGeVheK3GTyGjPv2MbL55TTvpFOk5oPU8Xvlf8VexGDa7gXONENoyWAOiYZ2XyY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819788; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pVvP7aCF3tlbLFaFsjhGKbqqTsjvNichpKhJ/Ji6V1M=; b=W6o8/3yB79F4uhVsoIUvRhsGM8ISqzKoFRnYtLaZ2C0J5BEljyLQ3wbFeDTGSj2WdIfNZbOr1ukE60nMQi2TRfFvcX7FqM/igxlXSPiEY5Y7qzx9NJKkoUmmQXK4gTubXvZ0/EaXa1WRG+C+XVprCiizDwVDhn23eWCVh1jLW0w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819788525804.0514106453469; Tue, 22 Sep 2020 17:09:48 -0700 (PDT) Received: from localhost ([::1]:39824 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsLz-0004LJ-7Q for importer@patchew.org; Tue, 22 Sep 2020 20:09:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46994) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFf-0003c0-9l for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:15 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:16943) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFc-0001BN-QB for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:14 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:56:32 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:00 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:57:52 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 08/17] vfio: Add save state functions to SaveVMHandlers Date: Wed, 23 Sep 2020 04:54:10 +0530 Message-ID: <1600817059-26721-9-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600818992; bh=pVvP7aCF3tlbLFaFsjhGKbqqTsjvNichpKhJ/Ji6V1M=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=HrLiX0+H8siMmR4WzpaYQJznVux8LeYN+MCSqBRjQr9GW3S6Sb2Qm8hXjLHvsBRMw kUR/QQET0JwnmziOeGVfu7+b2fBziYO3XXilWMm/LN9dN0JScZx536KqHREd1vUCXM KDjXn+6hdmzNdT0P8k4UgvrYx2nvZ63YamvUSWxGYSjDBg3hkTGu5+R17H7Jjmn1vn YM/w7iQWgb5Mm3XWDRAEzA1+1pqaEECEiN5tYRR+tdVEVSahjgOeKd46tkf/+LQqjp hR654f7rSHdNv2O8uOOZHp+wXueDhy4URI3P8txHaGkbxE8ly+F8NWl5MzwCuSGxSR QNakQ8FnulLYw== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.143; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate24.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:06 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added .save_live_pending, .save_live_iterate and .save_live_complete_precopy functions. These functions handles pre-copy and stop-and-copy phase. In _SAVING|_RUNNING device state or pre-copy phase: - read pending_bytes. If pending_bytes > 0, go through below steps. - read data_offset - indicates kernel driver to write data to staging buffer. - read data_size - amount of data in bytes written by vendor driver in migration region. - read data_size bytes of data from data_offset in the migration region. - Write data packet to file stream as below: {VFIO_MIG_FLAG_DEV_DATA_STATE, data_size, actual data, VFIO_MIG_FLAG_END_OF_STATE } In _SAVING device state or stop-and-copy phase a. read config space of device and save to migration file stream. This doesn't need to be from vendor driver. Any other special config state from driver can be saved as data in following iteration. b. read pending_bytes. If pending_bytes > 0, go through below steps. c. read data_offset - indicates kernel driver to write data to staging buffer. d. read data_size - amount of data in bytes written by vendor driver in migration region. e. read data_size bytes of data from data_offset in the migration region. f. Write data packet as below: {VFIO_MIG_FLAG_DEV_DATA_STATE, data_size, actual data} g. iterate through steps b to f while (pending_bytes > 0) h. Write {VFIO_MIG_FLAG_END_OF_STATE} When data region is mapped, its user's responsibility to read data from data_offset of data_size before moving to next steps. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/migration.c | 273 ++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/trace-events | 6 + include/hw/vfio/vfio-common.h | 1 + 3 files changed, 280 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 8e8adaa25779..4611bb972228 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -180,6 +180,154 @@ static int vfio_migration_set_state(VFIODevice *vbase= dev, uint32_t mask, return 0; } =20 +static void *get_data_section_size(VFIORegion *region, uint64_t data_offse= t, + uint64_t data_size, uint64_t *size) +{ + void *ptr =3D NULL; + uint64_t limit =3D 0; + int i; + + if (!region->mmaps) { + if (size) { + *size =3D data_size; + } + return ptr; + } + + for (i =3D 0; i < region->nr_mmaps; i++) { + VFIOMmap *map =3D region->mmaps + i; + + if ((data_offset >=3D map->offset) && + (data_offset < map->offset + map->size)) { + + /* check if data_offset is within sparse mmap areas */ + ptr =3D map->mmap + data_offset - map->offset; + if (size) { + *size =3D MIN(data_size, map->offset + map->size - data_of= fset); + } + break; + } else if ((data_offset < map->offset) && + (!limit || limit > map->offset)) { + /* + * data_offset is not within sparse mmap areas, find size of + * non-mapped area. Check through all list since region->mmaps= list + * is not sorted. + */ + limit =3D map->offset; + } + } + + if (!ptr && size) { + *size =3D limit ? limit - data_offset : data_size; + } + return ptr; +} + +static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *s= ize) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region; + uint64_t data_offset =3D 0, data_size =3D 0, sz; + int ret; + + ret =3D vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), + region->fd_offset + offsetof(struct vfio_device_migration_= info, + data_offset)); + if (ret < 0) { + return ret; + } + + ret =3D vfio_mig_read(vbasedev, &data_size, sizeof(data_size), + region->fd_offset + offsetof(struct vfio_device_migration_= info, + data_size)); + if (ret < 0) { + return ret; + } + + trace_vfio_save_buffer(vbasedev->name, data_offset, data_size, + migration->pending_bytes); + + qemu_put_be64(f, data_size); + sz =3D data_size; + + while (sz) { + void *buf =3D NULL; + uint64_t sec_size; + bool buf_allocated =3D false; + + buf =3D get_data_section_size(region, data_offset, sz, &sec_size); + + if (!buf) { + buf =3D g_try_malloc(sec_size); + if (!buf) { + error_report("%s: Error allocating buffer ", __func__); + return -ENOMEM; + } + buf_allocated =3D true; + + ret =3D vfio_mig_read(vbasedev, buf, sec_size, + region->fd_offset + data_offset); + if (ret < 0) { + g_free(buf); + return ret; + } + } + + qemu_put_buffer(f, buf, sec_size); + + if (buf_allocated) { + g_free(buf); + } + sz -=3D sec_size; + data_offset +=3D sec_size; + } + + ret =3D qemu_file_get_error(f); + + if (!ret && size) { + *size =3D data_size; + } + + return ret; +} + +static int vfio_update_pending(VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIORegion *region =3D &migration->region; + uint64_t pending_bytes =3D 0; + int ret; + + ret =3D vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), + region->fd_offset + offsetof(struct vfio_device_migration_= info, + pending_bytes)); + if (ret < 0) { + migration->pending_bytes =3D 0; + return ret; + } + + migration->pending_bytes =3D pending_bytes; + trace_vfio_update_pending(vbasedev->name, pending_bytes); + return 0; +} + +static int vfio_save_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_STATE); + + if (vbasedev->ops && vbasedev->ops->vfio_save_config) { + vbasedev->ops->vfio_save_config(vbasedev, f); + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + trace_vfio_save_device_config_state(vbasedev->name); + + return qemu_file_get_error(f); +} + /* ---------------------------------------------------------------------- = */ =20 static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -232,9 +380,134 @@ static void vfio_save_cleanup(void *opaque) trace_vfio_save_cleanup(vbasedev->name); } =20 +static void vfio_save_pending(QEMUFile *f, void *opaque, + uint64_t threshold_size, + uint64_t *res_precopy_only, + uint64_t *res_compatible, + uint64_t *res_postcopy_only) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + int ret; + + ret =3D vfio_update_pending(vbasedev); + if (ret) { + return; + } + + *res_precopy_only +=3D migration->pending_bytes; + + trace_vfio_save_pending(vbasedev->name, *res_precopy_only, + *res_postcopy_only, *res_compatible); +} + +static int vfio_save_iterate(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + uint64_t data_size; + int ret; + + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); + + if (migration->pending_bytes =3D=3D 0) { + ret =3D vfio_update_pending(vbasedev); + if (ret) { + return ret; + } + + if (migration->pending_bytes =3D=3D 0) { + /* indicates data finished, goto complete phase */ + return 1; + } + } + + ret =3D vfio_save_buffer(f, vbasedev, &data_size); + + if (ret) { + error_report("%s: vfio_save_buffer failed %s", vbasedev->name, + strerror(errno)); + return ret; + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + trace_vfio_save_iterate(vbasedev->name, data_size); + + return 0; +} + +static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + uint64_t data_size; + int ret; + + ret =3D vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_RUNNING, + VFIO_DEVICE_STATE_SAVING); + if (ret) { + error_report("%s: Failed to set state STOP and SAVING", + vbasedev->name); + return ret; + } + + ret =3D vfio_save_device_config_state(f, opaque); + if (ret) { + return ret; + } + + ret =3D vfio_update_pending(vbasedev); + if (ret) { + return ret; + } + + while (migration->pending_bytes > 0) { + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); + ret =3D vfio_save_buffer(f, vbasedev, &data_size); + if (ret < 0) { + error_report("%s: Failed to save buffer", vbasedev->name); + return ret; + } + + if (data_size =3D=3D 0) { + break; + } + + ret =3D vfio_update_pending(vbasedev); + if (ret) { + return ret; + } + } + + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + + ret =3D vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_SAVING, = 0); + if (ret) { + error_report("%s: Failed to set state STOPPED", vbasedev->name); + return ret; + } + + trace_vfio_save_complete_precopy(vbasedev->name); + return ret; +} + static SaveVMHandlers savevm_vfio_handlers =3D { .save_setup =3D vfio_save_setup, .save_cleanup =3D vfio_save_cleanup, + .save_live_pending =3D vfio_save_pending, + .save_live_iterate =3D vfio_save_iterate, + .save_live_complete_precopy =3D vfio_save_complete_precopy, }; =20 /* ---------------------------------------------------------------------- = */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 982d8dccb219..118b5547c921 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -154,3 +154,9 @@ vfio_vmstate_change(const char *name, int running, cons= t char *reason, uint32_t vfio_migration_state_notifier(const char *name, const char *state) " (%s) = state %s" vfio_save_setup(const char *name) " (%s)" vfio_save_cleanup(const char *name) " (%s)" +vfio_save_buffer(const char *name, uint64_t data_offset, uint64_t data_siz= e, uint64_t pending) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64" pending 0x%= "PRIx64 +vfio_update_pending(const char *name, uint64_t pending) " (%s) pending 0x%= "PRIx64 +vfio_save_device_config_state(const char *name) " (%s)" +vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, u= int64_t compatible) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" compati= ble 0x%"PRIx64 +vfio_save_iterate(const char *name, int data_size) " (%s) data_size %d" +vfio_save_complete_precopy(const char *name) " (%s)" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 49c7c7a0e29a..471e444a364c 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -60,6 +60,7 @@ typedef struct VFIORegion { =20 typedef struct VFIOMigration { VFIORegion region; + uint64_t pending_bytes; } VFIOMigration; =20 typedef struct VFIOAddressSpace { --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819738; cv=none; d=zohomail.com; s=zohoarc; b=VQrAbzp0VPQa3ZixHP0IehQ/mv30pfRtvCSgCzH4fPg2l/yFuItUcV0rB9ZtUPC/NeSie+XhptwIx312N3GdbQwbbCk+jgZw3nWiO40NUdr42WC9zbIB5EfU/tOppvrRiptSutnrUk11Esp5gQ38Bwivxyo7WcgAo7bWy6n/hBk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819738; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=NW36ok0ydjozvt6K6p61iEu7N8msCJxGXa9UzMYgyyQ=; b=Q1vJPMiz5d/o5QdvULmGl5atfK1Y9+9eME0rPnOaj7k/kDlMHLaPpazDBWRST3AURN6sA9i0BC7/o1AY3j/cZ/Ja5gyRRkis/T8mCg1Nadg/jWEe6tAZXTs4o4IE2m/7zcI9g02YlyCWGUyn3bpGENKBKvJArBmtWGxy8mfA5hQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819738121326.6942087808203; Tue, 22 Sep 2020 17:08:58 -0700 (PDT) Received: from localhost ([::1]:37688 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsLA-0003QE-JU for importer@patchew.org; Tue, 22 Sep 2020 20:08:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47040) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFj-0003nV-TC for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:19 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:16952) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFh-0001CE-QZ for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:19 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:56:41 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:09 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:01 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 09/17] vfio: Add load state functions to SaveVMHandlers Date: Wed, 23 Sep 2020 04:54:11 +0530 Message-ID: <1600817059-26721-10-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819001; bh=NW36ok0ydjozvt6K6p61iEu7N8msCJxGXa9UzMYgyyQ=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=YoVCqt+2HefPOhho/jlGpuBPTuLUzzkA6nosvvzW/oxB7Jj/57XU/xtYHNqhhFHkC LkCg1Hix6wpo1a0pOXPUJw1Mj5A/VAdab9Tm7dj0RKrfBfhIabWYX7HyjiPspNq2RS Wf3ueXcKuBFPTI0f4GMGkKFzLCxvEXjrhGrcdOEB0fH5xHD6piFwKYXuTnM1UDxoAn LFHhl7EV7LFil8NFvYdUOg8TSrsYn/donctx+igF+Um7AkledkmbGu86tQd5+AqFQX yCLG//e29khgpF8qIkYg7t/ysSAhuJVNo9D8R9qcWntPhLfHrW37CZk6vBd4FaiNBB BemH6LDfpkW2g== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.143; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate24.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:06 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Sequence during _RESUMING device state: While data for this device is available, repeat below steps: a. read data_offset from where user application should write data. b. write data of data_size to migration region from data_offset. c. write data_size which indicates vendor driver that data is written in staging buffer. For user, data is opaque. User should write data in the same order as received. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Reviewed-by: Dr. David Alan Gilbert --- hw/vfio/migration.c | 170 +++++++++++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/trace-events | 3 + 2 files changed, 173 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4611bb972228..ffd70282dd0e 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -328,6 +328,33 @@ static int vfio_save_device_config_state(QEMUFile *f, = void *opaque) return qemu_file_get_error(f); } =20 +static int vfio_load_device_config_state(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + uint64_t data; + + if (vbasedev->ops && vbasedev->ops->vfio_load_config) { + int ret; + + ret =3D vbasedev->ops->vfio_load_config(vbasedev, f); + if (ret) { + error_report("%s: Failed to load device config space", + vbasedev->name); + return ret; + } + } + + data =3D qemu_get_be64(f); + if (data !=3D VFIO_MIG_FLAG_END_OF_STATE) { + error_report("%s: Failed loading device config space, " + "end flag incorrect 0x%"PRIx64, vbasedev->name, data); + return -EINVAL; + } + + trace_vfio_load_device_config_state(vbasedev->name); + return qemu_file_get_error(f); +} + /* ---------------------------------------------------------------------- = */ =20 static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -502,12 +529,155 @@ static int vfio_save_complete_precopy(QEMUFile *f, v= oid *opaque) return ret; } =20 +static int vfio_load_setup(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + int ret =3D 0; + + if (migration->region.mmaps) { + ret =3D vfio_region_mmap(&migration->region); + if (ret) { + error_report("%s: Failed to mmap VFIO migration region %d: %s", + vbasedev->name, migration->region.nr, + strerror(-ret)); + error_report("%s: Falling back to slow path", vbasedev->name); + } + } + + ret =3D vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_MASK, + VFIO_DEVICE_STATE_RESUMING); + if (ret) { + error_report("%s: Failed to set state RESUMING", vbasedev->name); + } + return ret; +} + +static int vfio_load_cleanup(void *opaque) +{ + vfio_save_cleanup(opaque); + return 0; +} + +static int vfio_load_state(QEMUFile *f, void *opaque, int version_id) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + int ret =3D 0; + uint64_t data, data_size; + + data =3D qemu_get_be64(f); + while (data !=3D VFIO_MIG_FLAG_END_OF_STATE) { + + trace_vfio_load_state(vbasedev->name, data); + + switch (data) { + case VFIO_MIG_FLAG_DEV_CONFIG_STATE: + { + ret =3D vfio_load_device_config_state(f, opaque); + if (ret) { + return ret; + } + break; + } + case VFIO_MIG_FLAG_DEV_SETUP_STATE: + { + data =3D qemu_get_be64(f); + if (data =3D=3D VFIO_MIG_FLAG_END_OF_STATE) { + return ret; + } else { + error_report("%s: SETUP STATE: EOS not found 0x%"PRIx64, + vbasedev->name, data); + return -EINVAL; + } + break; + } + case VFIO_MIG_FLAG_DEV_DATA_STATE: + { + VFIORegion *region =3D &migration->region; + uint64_t data_offset =3D 0, size; + + data_size =3D size =3D qemu_get_be64(f); + if (data_size =3D=3D 0) { + break; + } + + ret =3D vfio_mig_read(vbasedev, &data_offset, sizeof(data_offs= et), + region->fd_offset + + offsetof(struct vfio_device_migration_info, + data_offset)); + if (ret < 0) { + return ret; + } + + trace_vfio_load_state_device_data(vbasedev->name, data_offset, + data_size); + + while (size) { + void *buf =3D NULL; + uint64_t sec_size; + bool buf_alloc =3D false; + + buf =3D get_data_section_size(region, data_offset, size, + &sec_size); + + if (!buf) { + buf =3D g_try_malloc(sec_size); + if (!buf) { + error_report("%s: Error allocating buffer ", __fun= c__); + return -ENOMEM; + } + buf_alloc =3D true; + } + + qemu_get_buffer(f, buf, sec_size); + + if (buf_alloc) { + ret =3D vfio_mig_write(vbasedev, buf, sec_size, + region->fd_offset + data_offset); + g_free(buf); + + if (ret < 0) { + return ret; + } + } + size -=3D sec_size; + data_offset +=3D sec_size; + } + + ret =3D vfio_mig_write(vbasedev, &data_size, sizeof(data_size), + region->fd_offset + + offsetof(struct vfio_device_migration_info, data_si= ze)); + if (ret < 0) { + return ret; + } + break; + } + + default: + error_report("%s: Unknown tag 0x%"PRIx64, vbasedev->name, data= ); + return -EINVAL; + } + + data =3D qemu_get_be64(f); + ret =3D qemu_file_get_error(f); + if (ret) { + return ret; + } + } + + return ret; +} + static SaveVMHandlers savevm_vfio_handlers =3D { .save_setup =3D vfio_save_setup, .save_cleanup =3D vfio_save_cleanup, .save_live_pending =3D vfio_save_pending, .save_live_iterate =3D vfio_save_iterate, .save_live_complete_precopy =3D vfio_save_complete_precopy, + .load_setup =3D vfio_load_setup, + .load_cleanup =3D vfio_load_cleanup, + .load_state =3D vfio_load_state, }; =20 /* ---------------------------------------------------------------------- = */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 118b5547c921..94ba4696f0c6 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -160,3 +160,6 @@ vfio_save_device_config_state(const char *name) " (%s)" vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, u= int64_t compatible) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" compati= ble 0x%"PRIx64 vfio_save_iterate(const char *name, int data_size) " (%s) data_size %d" vfio_save_complete_precopy(const char *name) " (%s)" +vfio_load_device_config_state(const char *name) " (%s)" +vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 +vfio_load_state_device_data(const char *name, uint64_t data_offset, uint64= _t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819848; cv=none; d=zohomail.com; s=zohoarc; b=iA4ryAwXSuKwVyD7T2BA3sQ+nviZ/+nXrhlwVu8CS6+kK8vvrCfl5U9Pfvlp0NS3lWWfY/47POlaiVtXpbO+RacZpN3EMAUQ8S7AOVi3KkHiXQLOW7SpLFKpHriSHlCoVy+Ef/kHvNRycyWQX2vFxql3eMimdIjur8aCjV4oW+Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819848; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=fEPTqkC5VZ5UkBSS8ITszrI81DuuxIvvZ91XQ2lvszA=; b=dCA4Qrolc+sgK9lWACANuNUadZBxXnV6hkogvqtgg+9JX8mkbIdczYD+8KlJpUfRl0Mfg9m64hUjsnIgsc0M1piDZSWlZ2VuZDJwAJeMOql4K2IOHrFvxjQGNDiLtHyIeSFT8XjCCdsTtSVtSZfU33GWE3/Q9hnX47sHpP+tGW4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819848459882.8047537234781; Tue, 22 Sep 2020 17:10:48 -0700 (PDT) Received: from localhost ([::1]:42022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsMx-0005Nl-69 for importer@patchew.org; Tue, 22 Sep 2020 20:10:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47064) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFu-00044u-7h for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:30 -0400 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:16969) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsFs-0001Cm-Fx for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:29 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:56:51 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:17 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:10 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 10/17] memory: Set DIRTY_MEMORY_MIGRATION when IOMMU is enabled Date: Wed, 23 Sep 2020 04:54:12 +0530 Message-ID: <1600817059-26721-11-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819011; bh=fEPTqkC5VZ5UkBSS8ITszrI81DuuxIvvZ91XQ2lvszA=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=n8ELG+6XbZttSd6+Mkm5N34MG/NkfbSzdH6EeOKofRbS7X3kwzN7EKid6RW0rIPds uaEjLhqsvuV+Lfo09Qew47bYTc27HAuvXUoAdg85eJFrOkYo7pwKth28TarRDqxr3k OawLjxgSuHpRdTc7BuxfyK5v2kMfEgDpXOe3rJAVxo856XwJ5G84ifEwfbovRfD1IY Xv3wx/8wwVGLlv5+xeQHCg5BiWtx6p8lNy2IbFtmrXmPm1JQfnb1wQ2SbqFc4FXMu7 fJ7t6aiAgAePasaW53I2im3dSap4qFRkpjfwFLLR4X667kN+vXAQNCkIP+8YHsDkkV 88NdlNj22Rpeg== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.143; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate24.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:06 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" mr->ram_block is NULL when mr->is_iommu is true, then fr.dirty_log_mask wasn't set correctly due to which memory listener's log_sync doesn't get called. This patch returns log_mask with DIRTY_MEMORY_MIGRATION set when IOMMU is enabled. Signed-off-by: Kirti Wankhede --- softmmu/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/softmmu/memory.c b/softmmu/memory.c index d030eb6f7cea..0c1460394ace 100644 --- a/softmmu/memory.c +++ b/softmmu/memory.c @@ -1777,7 +1777,7 @@ bool memory_region_is_ram_device(MemoryRegion *mr) uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr) { uint8_t mask =3D mr->dirty_log_mask; - if (global_dirty_log && mr->ram_block) { + if (global_dirty_log && (mr->ram_block || memory_region_is_iommu(mr)))= { mask |=3D (1 << DIRTY_MEMORY_MIGRATION); } return mask; --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819673; cv=none; d=zohomail.com; s=zohoarc; b=NkFpPcBTNxvMuuHVuXYgNZ1/mgW3iIFoSWoZYJ1/lucrbVdXbQPqEbzQ/h+mzBNJpzf7TmJQz048DY52aPslpdk4RJEalXzUpL5rvJc54PxsqtABnty3a7y5rE+TYcwFneQx68WtfSXAVucOFJf32AXEw2XgQwouvyNh69vem8A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819673; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=IjQiJvybIhsE1kXhU0QcmULhWUF1j/2sXxzQzst6YSU=; b=SBfGBWjLwXyJFnhN62x7xKwl6b5agkWPx9OT6RRN/NHM0dDgam10MoCl6m8Opshtm3pWAEQi962rv+1zip8WWmJ0Hxs2XLPr4P2ubKtVW82tp+sKF/GPQEkxB+6WMcYDdFFqCxm0IgTK1J5NHcj2f8KNoyaimZ/hLxl9QY2aQBM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 160081967364687.2699057490264; Tue, 22 Sep 2020 17:07:53 -0700 (PDT) Received: from localhost ([::1]:34220 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsK8-0001xm-8f for importer@patchew.org; Tue, 22 Sep 2020 20:07:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47118) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsG9-0004Ms-Lc for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:46 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3843) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsG6-0001DG-SQ for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:45 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:57:54 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:26 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:17 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 11/17] vfio: Get migration capability flags for container Date: Wed, 23 Sep 2020 04:54:13 +0530 Message-ID: <1600817059-26721-12-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819074; bh=IjQiJvybIhsE1kXhU0QcmULhWUF1j/2sXxzQzst6YSU=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=BjKyBclwAZh6ljk854tvmcv7zy4r8MFQux0GrzdIK/1xXtxXh9QlVYfrXftYva6bd DBaZxO+Nnic36wR7KnPJwmpBJoL9AD+HPHpF++MGXoCE7HrVb5d4P5x0rWUUEzbTha RwgLjF8HixpaNFuvN9mLKOHyUUHS7IT70REowAo6obbcq3Jkl1T4wwzjVEglbQ+FeH IKBU7MV3zc6M6v9RpZcjyFpFcr2ANhlSt6h5jHt7eC0951tEChpfPN4AspA+tfbo70 CtMzowfc8wysCpnkS5keJfYsH5YYYtTvCyTLl/Cq212SMqqHHgnmOA/4MEJHTOW1DX R12hd2ArFk5xA== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.64; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate25.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:12 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, Eric Auger , changpeng.liu@intel.com, eskultet@redhat.com, Shameer Kolothum , Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added helper functions to get IOMMU info capability chain. Added function to get migration capability information from that capability chain for IOMMU container. Similar change was proposed earlier: https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg03759.html Disable migration for devices if IOMMU module doesn't support migration capability. Signed-off-by: Kirti Wankhede Cc: Shameer Kolothum Cc: Eric Auger --- hw/vfio/common.c | 90 +++++++++++++++++++++++++++++++++++++++= ---- hw/vfio/migration.c | 7 +++- include/hw/vfio/vfio-common.h | 3 ++ 3 files changed, 91 insertions(+), 9 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index c6e98b8d61be..d4959c036dd1 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1228,6 +1228,75 @@ static int vfio_init_container(VFIOContainer *contai= ner, int group_fd, return 0; } =20 +static int vfio_get_iommu_info(VFIOContainer *container, + struct vfio_iommu_type1_info **info) +{ + + size_t argsz =3D sizeof(struct vfio_iommu_type1_info); + + *info =3D g_new0(struct vfio_iommu_type1_info, 1); +again: + (*info)->argsz =3D argsz; + + if (ioctl(container->fd, VFIO_IOMMU_GET_INFO, *info)) { + g_free(*info); + *info =3D NULL; + return -errno; + } + + if (((*info)->argsz > argsz)) { + argsz =3D (*info)->argsz; + *info =3D g_realloc(*info, argsz); + goto again; + } + + return 0; +} + +static struct vfio_info_cap_header * +vfio_get_iommu_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) +{ + struct vfio_info_cap_header *hdr; + void *ptr =3D info; + + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { + return NULL; + } + + for (hdr =3D ptr + info->cap_offset; hdr !=3D ptr; hdr =3D ptr + hdr->= next) { + if (hdr->id =3D=3D id) { + return hdr; + } + } + + return NULL; +} + +static void vfio_get_iommu_info_migration(VFIOContainer *container, + struct vfio_iommu_type1_info *inf= o) +{ + struct vfio_info_cap_header *hdr; + struct vfio_iommu_type1_info_cap_migration *cap_mig; + + hdr =3D vfio_get_iommu_info_cap(info, VFIO_IOMMU_TYPE1_INFO_CAP_MIGRAT= ION); + if (!hdr) { + return; + } + + cap_mig =3D container_of(hdr, struct vfio_iommu_type1_info_cap_migrati= on, + header); + + /* + * cpu_physical_memory_set_dirty_lebitmap() expects pages in bitmap of + * TARGET_PAGE_SIZE to mark those dirty. + */ + if (cap_mig->pgsize_bitmap & TARGET_PAGE_SIZE) { + container->dirty_pages_supported =3D true; + container->max_dirty_bitmap_size =3D cap_mig->max_dirty_bitmap_siz= e; + container->dirty_pgsizes =3D cap_mig->pgsize_bitmap; + } +} + static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, Error **errp) { @@ -1297,6 +1366,7 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, container->space =3D space; container->fd =3D fd; container->error =3D NULL; + container->dirty_pages_supported =3D false; QLIST_INIT(&container->giommu_list); QLIST_INIT(&container->hostwin_list); =20 @@ -1309,7 +1379,7 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, case VFIO_TYPE1v2_IOMMU: case VFIO_TYPE1_IOMMU: { - struct vfio_iommu_type1_info info; + struct vfio_iommu_type1_info *info; =20 /* * FIXME: This assumes that a Type1 IOMMU can map any 64-bit @@ -1318,15 +1388,19 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as, * existing Type1 IOMMUs generally support any IOVA we're * going to actually try in practice. */ - info.argsz =3D sizeof(info); - ret =3D ioctl(fd, VFIO_IOMMU_GET_INFO, &info); - /* Ignore errors */ - if (ret || !(info.flags & VFIO_IOMMU_INFO_PGSIZES)) { + ret =3D vfio_get_iommu_info(container, &info); + + if (ret || !(info->flags & VFIO_IOMMU_INFO_PGSIZES)) { /* Assume 4k IOVA page size */ - info.iova_pgsizes =3D 4096; + info->iova_pgsizes =3D 4096; } - vfio_host_win_add(container, 0, (hwaddr)-1, info.iova_pgsizes); - container->pgsizes =3D info.iova_pgsizes; + vfio_host_win_add(container, 0, (hwaddr)-1, info->iova_pgsizes); + container->pgsizes =3D info->iova_pgsizes; + + if (!ret) { + vfio_get_iommu_info_migration(container, info); + } + g_free(info); break; } case VFIO_SPAPR_TCE_v2_IOMMU: diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index ffd70282dd0e..4306f6316417 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -786,9 +786,14 @@ static int vfio_migration_init(VFIODevice *vbasedev, =20 int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) { + VFIOContainer *container =3D vbasedev->group->container; struct vfio_region_info *info =3D NULL; Error *local_err =3D NULL; - int ret; + int ret =3D -ENOTSUP; + + if (!container->dirty_pages_supported) { + goto add_blocker; + } =20 ret =3D vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION, VFIO_REGION_SUBTYPE_MIGRATION, &info); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 471e444a364c..0a1651eda2d0 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -79,6 +79,9 @@ typedef struct VFIOContainer { unsigned iommu_type; Error *error; bool initialized; + bool dirty_pages_supported; + uint64_t dirty_pgsizes; + uint64_t max_dirty_bitmap_size; unsigned long pgsizes; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819900; cv=none; d=zohomail.com; s=zohoarc; b=ZmBIHCGHe68kE+wOJpEngVf+rn8r3DZ/0VOU+R/jfy2723fOZR8FEaiLD2JxViZnTXD8Kk5uxu/0LdFmZh9rMhHtx1G7ko792zMCw/8779ffxruEBKEfH7J2ZbJ2HkqBkFRzQnz7lOyMdlfNSyM+4wYqCxE997J8dqLm508fYKc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819900; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=KQNeKmVobDDnc4FBJBVSw+dTSvPqpsdz5wXJvSZjuuI=; b=COWwgfLhHVR6c+CRZYHoJfgcPI9PWzku1weVJ8uD6cXQMIjONgcBSQ//A79YPTock5AZb4VHj4Cty1DxZPduzmArADZPxqMvrOljzOQPXnfUDhXs5E9NhfiALOZRIT1pIlMHIUcRRWFXyqXzVVoSpp66FVI3QwRoTWCTPL2ulxM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819900103938.1608536033394; Tue, 22 Sep 2020 17:11:40 -0700 (PDT) Received: from localhost ([::1]:44098 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsNm-0006N6-Qc for importer@patchew.org; Tue, 22 Sep 2020 20:11:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsG9-0004Ma-7x for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:45 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14503) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsG6-0001DI-UN for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:44 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:58:28 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:34 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:26 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 12/17] vfio: Add function to start and stop dirty pages tracking Date: Wed, 23 Sep 2020 04:54:14 +0530 Message-ID: <1600817059-26721-13-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819108; bh=KQNeKmVobDDnc4FBJBVSw+dTSvPqpsdz5wXJvSZjuuI=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=Sy/N+jWtjl9rltaJEZMETqDEwWF3mFSDoXLWiaEuOpbKFv0+uILakyjfLiKatBU5t eD4uiKJ7mAbPU9iuKgp71SG4ZsJqW2eekcPMkH+dky3XB0sO0UDwEQbUruLfvXGulQ s+0GEOLxDNwgELlQ03qkNuRjhFq2NdCSRroMM3Mli4LEeOlTiS7Ylj2zyfCD6JLIwH 0lsQ4isozQI2Zfuf4cafS+A1atMcDNsDdH4ze3HkABleMqxKchezu244F7OLnuDkvJ k3vJfKj2UnXjY2354Q6ALy19AfLUakRiH2MTZc98WkAGgvb8gdudmdOPRIR0Rgoo/d HOo7Q0XUBz17g== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Call VFIO_IOMMU_DIRTY_PAGES ioctl to start and stop dirty pages tracking for VFIO devices. Signed-off-by: Kirti Wankhede Reviewed-by: Dr. David Alan Gilbert --- hw/vfio/migration.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4306f6316417..822b68b4e015 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -11,6 +11,7 @@ #include "qemu/main-loop.h" #include "qemu/cutils.h" #include +#include =20 #include "sysemu/runstate.h" #include "hw/vfio/vfio-common.h" @@ -355,6 +356,32 @@ static int vfio_load_device_config_state(QEMUFile *f, = void *opaque) return qemu_file_get_error(f); } =20 +static int vfio_set_dirty_page_tracking(VFIODevice *vbasedev, bool start) +{ + int ret; + VFIOContainer *container =3D vbasedev->group->container; + struct vfio_iommu_type1_dirty_bitmap dirty =3D { + .argsz =3D sizeof(dirty), + }; + + if (start) { + if (vbasedev->device_state & VFIO_DEVICE_STATE_SAVING) { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; + } else { + return -EINVAL; + } + } else { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, errno); + } + return ret; +} + /* ---------------------------------------------------------------------- = */ =20 static int vfio_save_setup(QEMUFile *f, void *opaque) @@ -386,6 +413,11 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return ret; } =20 + ret =3D vfio_set_dirty_page_tracking(vbasedev, true); + if (ret) { + return ret; + } + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); =20 ret =3D qemu_file_get_error(f); @@ -401,6 +433,8 @@ static void vfio_save_cleanup(void *opaque) VFIODevice *vbasedev =3D opaque; VFIOMigration *migration =3D vbasedev->migration; =20 + vfio_set_dirty_page_tracking(vbasedev, false); + if (migration->region.mmaps) { vfio_region_unmap(&migration->region); } @@ -734,6 +768,8 @@ static void vfio_migration_state_notifier(Notifier *not= ifier, void *data) if (ret) { error_report("%s: Failed to set state RUNNING", vbasedev->name= ); } + + vfio_set_dirty_page_tracking(vbasedev, false); } } =20 --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819977; cv=none; d=zohomail.com; s=zohoarc; b=cW07HlmtM2+Twp8eER50uChOiJgF8KtSOE6sQAiynhtKAHicIHAvaBVN2we79leqPdZKIpiS0D+x/zL1+9xrN14g1Q/hqt5od5yikJ5iHk427KaFBp/89d+SzyHc6N4SEKueC7XfBReqetnpGNSceAHAaxlha7U4GFYOwl6Lu/4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819977; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=5Ma7HM62w7SSO2Rkrr9jT3RsK/CFCDmVr0rCCyQXllg=; b=aOzxlhQV09hOI5FXAtytXXEp78/UyN/EUGq1JDOEqcj9XonRtNMiAZFvHbK1C6cf7xvDQMtB/iJW2f9CsBozfdUypAwWwExsv0ZFkjhYUS9N0v2IHQm15AY4TdFc9vgbLxS5hEDa0xU9m08oTy4MuAzlQLHg8pwxWLa5EchRUW4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819976994999.9047609238055; Tue, 22 Sep 2020 17:12:56 -0700 (PDT) Received: from localhost ([::1]:46182 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsP0-0007H6-Lr for importer@patchew.org; Tue, 22 Sep 2020 20:12:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGK-0004aT-Lb for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:56 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3864) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGI-0001F1-FE for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:03:56 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:58:05 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:42 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:34 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 13/17] vfio: create mapped iova list when vIOMMU is enabled Date: Wed, 23 Sep 2020 04:54:15 +0530 Message-ID: <1600817059-26721-14-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819085; bh=5Ma7HM62w7SSO2Rkrr9jT3RsK/CFCDmVr0rCCyQXllg=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=T5GPpw+lGnkWDBO1bxe0aaxuWTDhTJZeq+bKgFZnJrDFG1y4hc+6OTRyMbjpr2StE d/cqHddUyAtd584KIS8gcREqFFvAaCeTsIrXqJVb5jTfTHr3m7QD0wmnWcqEE+QIsE KVG5awCAp6ElApmos6ASyYfvNmwssLPW6zPlMKjeX8bw4CPEikiyzpZGAFkkXNOOQb OCE5RwbfsSsu+VWnbLV3xRdN+0OPSWO0lEtPv5ATBcpFMCFzh2PUeepmgxnZE/lD2+ T887AWG1pwRfIsSSwfLF8qITacbkJeB1uhmavTVL/jkabh+msdeeNwvZ+QuKpeIggi 9p/Kh/mH9vKeg== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.64; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate25.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:12 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create mapped iova list when vIOMMU is enabled. For each mapped iova save translated address. Add node to list on MAP and remove node from list on UNMAP. This list is used to track dirty pages during migration. Signed-off-by: Kirti Wankhede --- hw/vfio/common.c | 58 ++++++++++++++++++++++++++++++++++++++-= ---- include/hw/vfio/vfio-common.h | 8 ++++++ 2 files changed, 60 insertions(+), 6 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index d4959c036dd1..dc56cded2d95 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -407,8 +407,8 @@ static bool vfio_listener_skipped_section(MemoryRegionS= ection *section) } =20 /* Called with rcu_read_lock held. */ -static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, void **vaddr, - bool *read_only) +static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, + ram_addr_t *ram_addr, bool *read_only) { MemoryRegion *mr; hwaddr xlat; @@ -439,8 +439,17 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb, void = **vaddr, return false; } =20 - *vaddr =3D memory_region_get_ram_ptr(mr) + xlat; - *read_only =3D !writable || mr->readonly; + if (vaddr) { + *vaddr =3D memory_region_get_ram_ptr(mr) + xlat; + } + + if (ram_addr) { + *ram_addr =3D memory_region_get_ram_addr(mr) + xlat; + } + + if (read_only) { + *read_only =3D !writable || mr->readonly; + } =20 return true; } @@ -450,7 +459,6 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOM= MUTLBEntry *iotlb) VFIOGuestIOMMU *giommu =3D container_of(n, VFIOGuestIOMMU, n); VFIOContainer *container =3D giommu->container; hwaddr iova =3D iotlb->iova + giommu->iommu_offset; - bool read_only; void *vaddr; int ret; =20 @@ -466,7 +474,10 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IO= MMUTLBEntry *iotlb) rcu_read_lock(); =20 if ((iotlb->perm & IOMMU_RW) !=3D IOMMU_NONE) { - if (!vfio_get_vaddr(iotlb, &vaddr, &read_only)) { + ram_addr_t ram_addr; + bool read_only; + + if (!vfio_get_xlat_addr(iotlb, &vaddr, &ram_addr, &read_only)) { goto out; } /* @@ -484,8 +495,28 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IO= MMUTLBEntry *iotlb) "0x%"HWADDR_PRIx", %p) =3D %d (%m)", container, iova, iotlb->addr_mask + 1, vaddr, ret); + } else { + VFIOIovaRange *iova_range; + + iova_range =3D g_malloc0(sizeof(*iova_range)); + iova_range->iova =3D iova; + iova_range->size =3D iotlb->addr_mask + 1; + iova_range->ram_addr =3D ram_addr; + + QLIST_INSERT_HEAD(&giommu->iova_list, iova_range, next); } } else { + VFIOIovaRange *iova_range, *tmp; + + QLIST_FOREACH_SAFE(iova_range, &giommu->iova_list, next, tmp) { + if (iova_range->iova >=3D iova && + iova_range->iova + iova_range->size <=3D iova + + iotlb->addr_mask + = 1) { + QLIST_REMOVE(iova_range, next); + g_free(iova_range); + } + } + ret =3D vfio_dma_unmap(container, iova, iotlb->addr_mask + 1); if (ret) { error_report("vfio_dma_unmap(%p, 0x%"HWADDR_PRIx", " @@ -642,6 +673,7 @@ static void vfio_listener_region_add(MemoryListener *li= stener, g_free(giommu); goto fail; } + QLIST_INIT(&giommu->iova_list); QLIST_INSERT_HEAD(&container->giommu_list, giommu, giommu_next); memory_region_iommu_replay(giommu->iommu, &giommu->n); =20 @@ -740,6 +772,13 @@ static void vfio_listener_region_del(MemoryListener *l= istener, QLIST_FOREACH(giommu, &container->giommu_list, giommu_next) { if (MEMORY_REGION(giommu->iommu) =3D=3D section->mr && giommu->n.start =3D=3D section->offset_within_region) { + VFIOIovaRange *iova_range, *tmp; + + QLIST_FOREACH_SAFE(iova_range, &giommu->iova_list, next, t= mp) { + QLIST_REMOVE(iova_range, next); + g_free(iova_range); + } + memory_region_unregister_iommu_notifier(section->mr, &giommu->n); QLIST_REMOVE(giommu, giommu_next); @@ -1541,6 +1580,13 @@ static void vfio_disconnect_container(VFIOGroup *gro= up) QLIST_REMOVE(container, next); =20 QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, t= mp) { + VFIOIovaRange *iova_range, *itmp; + + QLIST_FOREACH_SAFE(iova_range, &giommu->iova_list, next, itmp)= { + QLIST_REMOVE(iova_range, next); + g_free(iova_range); + } + memory_region_unregister_iommu_notifier( MEMORY_REGION(giommu->iommu), &giommu->n); QLIST_REMOVE(giommu, giommu_next); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 0a1651eda2d0..aa7524fe2cc5 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -89,11 +89,19 @@ typedef struct VFIOContainer { QLIST_ENTRY(VFIOContainer) next; } VFIOContainer; =20 +typedef struct VFIOIovaRange { + hwaddr iova; + size_t size; + ram_addr_t ram_addr; + QLIST_ENTRY(VFIOIovaRange) next; +} VFIOIovaRange; + typedef struct VFIOGuestIOMMU { VFIOContainer *container; IOMMUMemoryRegion *iommu; hwaddr iommu_offset; IOMMUNotifier n; + QLIST_HEAD(, VFIOIovaRange) iova_list; QLIST_ENTRY(VFIOGuestIOMMU) giommu_next; } VFIOGuestIOMMU; =20 --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600820058; cv=none; d=zohomail.com; s=zohoarc; b=kl/H7Ihna6R9wX7e4aMzd5iZUB/HXpC+cdAzqD0ov+Yi1ejvtQPvvX5yTV1RGsbM0CyKCY4foCU8no4Y75E23F0/IEB+pklIoUKN7HSvEJy1C6gg0gIZizqCGUDg35NlqvsltyJV5r97/Ho6D5A+4SnNrXZYA2lJV0SHD5v2G10= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600820058; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=emk1CjW5v4q7knnP0vNjt4v5zUL2vfwWF7VrfDAnSos=; b=OzKXkRMagmLiBA65GD8OhTwURPzWvjPaBm4KUDEqiQAcDXY6wKZRoKRTbdWhUikg2JX8Xz4j8vksOtQzLk2Dxki9vwmaH0yjDEu/wR6Mpk2st0cshJWkyr7afpxeHGY78KXsZyCLK0jAcqaqoTljBxbNmaMx6CdERCgL3cQItWs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 160082005897355.45578513247801; Tue, 22 Sep 2020 17:14:18 -0700 (PDT) Received: from localhost ([::1]:48386 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsQL-0008EE-GR for importer@patchew.org; Tue, 22 Sep 2020 20:14:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47192) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGT-0004wT-U2 for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:11 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3869) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGR-0001FQ-KL for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:05 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:58:14 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:50 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:42 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 14/17] vfio: Add vfio_listener_log_sync to mark dirty pages Date: Wed, 23 Sep 2020 04:54:16 +0530 Message-ID: <1600817059-26721-15-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819094; bh=emk1CjW5v4q7knnP0vNjt4v5zUL2vfwWF7VrfDAnSos=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=hqFxmgDnmvVWQxkRab1x8MLRq6i14pvJUBPp96+NSA/ZjhuMlm48Wdog7sqm1Va/Q TM7W6ZrEcIpwybYMNr8TLbn9j5d+n8iE1Cf1tSEP9JvSa79fRf1uouFK5QAh0IAgFy kup85iQz9OKp/DCwhbbIATE+t8D6ljGEgz0d0l54W7qxONegc1BwS98kgi8e0XIdE6 RRea4MPpAs2bi7pNK0wu90SnLXOwQj5KmDaJDh69+qEUico10QNookKqTGjIAgWshz PC+fh+FCs4z1XCDfuCInxj2ld2UddQaAuUM5Y9UhDnq3kHDCIDhQOI0YF4nvd0SSce 6dsWAB+6iPdTA== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.64; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate25.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:12 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vfio_listener_log_sync gets list of dirty pages from container using VFIO_IOMMU_GET_DIRTY_BITMAP ioctl and mark those pages dirty when all devices are stopped and saving state. Return early for the RAM block section of mapped MMIO region. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- hw/vfio/common.c | 136 +++++++++++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/trace-events | 1 + 2 files changed, 137 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index dc56cded2d95..c36f275951c4 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -29,6 +29,7 @@ #include "hw/vfio/vfio.h" #include "exec/address-spaces.h" #include "exec/memory.h" +#include "exec/ram_addr.h" #include "hw/hw.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" @@ -37,6 +38,7 @@ #include "sysemu/reset.h" #include "trace.h" #include "qapi/error.h" +#include "migration/migration.h" =20 VFIOGroupList vfio_group_list =3D QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -287,6 +289,33 @@ const MemoryRegionOps vfio_region_ops =3D { }; =20 /* + * Device state interfaces + */ + +static bool vfio_devices_all_stopped_and_saving(VFIOContainer *container) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + MigrationState *ms =3D migrate_get_current(); + + if (!migration_is_setup_or_active(ms->state)) { + return false; + } + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if ((vbasedev->device_state & VFIO_DEVICE_STATE_SAVING) && + !(vbasedev->device_state & VFIO_DEVICE_STATE_RUNNING)) { + continue; + } else { + return false; + } + } + } + return true; +} + +/* * DMA - Mapping and unmapping for the "type1" IOMMU interface used on x86 */ static int vfio_dma_unmap(VFIOContainer *container, @@ -851,9 +880,116 @@ static void vfio_listener_region_del(MemoryListener *= listener, } } =20 +static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr) +{ + struct vfio_iommu_type1_dirty_bitmap *dbitmap; + struct vfio_iommu_type1_dirty_bitmap_get *range; + uint64_t pages; + int ret; + + dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); + + dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); + dbitmap->flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; + range =3D (struct vfio_iommu_type1_dirty_bitmap_get *)&dbitmap->data; + range->iova =3D iova; + range->size =3D size; + + /* + * cpu_physical_memory_set_dirty_lebitmap() expects pages in bitmap of + * TARGET_PAGE_SIZE to mark those dirty. Hence set bitmap's pgsize to + * TARGET_PAGE_SIZE. + */ + range->bitmap.pgsize =3D TARGET_PAGE_SIZE; + + pages =3D TARGET_PAGE_ALIGN(range->size) >> TARGET_PAGE_BITS; + range->bitmap.size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / + BITS_PER_BYTE; + range->bitmap.data =3D g_try_malloc0(range->bitmap.size); + if (!range->bitmap.data) { + ret =3D -ENOMEM; + goto err_out; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + if (ret) { + error_report("Failed to get dirty bitmap for iova: 0x%llx " + "size: 0x%llx err: %d", + range->iova, range->size, errno); + goto err_out; + } + + cpu_physical_memory_set_dirty_lebitmap((uint64_t *)range->bitmap.data, + ram_addr, pages); + + trace_vfio_get_dirty_bitmap(container->fd, range->iova, range->size, + range->bitmap.size, ram_addr); +err_out: + g_free(range->bitmap.data); + g_free(dbitmap); + + return ret; +} + +static int vfio_sync_dirty_bitmap(VFIOContainer *container, + MemoryRegionSection *section) +{ + VFIOGuestIOMMU *giommu =3D NULL; + ram_addr_t ram_addr; + uint64_t iova, size; + int ret =3D 0; + + if (memory_region_is_iommu(section->mr)) { + + QLIST_FOREACH(giommu, &container->giommu_list, giommu_next) { + if (MEMORY_REGION(giommu->iommu) =3D=3D section->mr && + giommu->n.start =3D=3D section->offset_within_region) { + VFIOIovaRange *iova_range; + + QLIST_FOREACH(iova_range, &giommu->iova_list, next) { + ret =3D vfio_get_dirty_bitmap(container, iova_range->i= ova, + iova_range->size, iova_range->ram_= addr); + if (ret) { + break; + } + } + break; + } + } + + } else { + iova =3D TARGET_PAGE_ALIGN(section->offset_within_address_space); + size =3D int128_get64(section->size); + + ram_addr =3D memory_region_get_ram_addr(section->mr) + + section->offset_within_region + iova - + TARGET_PAGE_ALIGN(section->offset_within_address_space); + + ret =3D vfio_get_dirty_bitmap(container, iova, size, ram_addr); + } + + return ret; +} + +static void vfio_listerner_log_sync(MemoryListener *listener, + MemoryRegionSection *section) +{ + VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); + + if (vfio_listener_skipped_section(section)) { + return; + } + + if (vfio_devices_all_stopped_and_saving(container)) { + vfio_sync_dirty_bitmap(container, section); + } +} + static const MemoryListener vfio_memory_listener =3D { .region_add =3D vfio_listener_region_add, .region_del =3D vfio_listener_region_del, + .log_sync =3D vfio_listerner_log_sync, }; =20 static void vfio_listener_release(VFIOContainer *container) diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 94ba4696f0c6..6140bb726343 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -163,3 +163,4 @@ vfio_save_complete_precopy(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_offset, uint64= _t data_size) " (%s) Offset 0x%"PRIx64" size 0x%"PRIx64 +vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitma= p_size, uint64_t start) "container fd=3D%d, iova=3D0x%"PRIx64" size=3D 0x%"= PRIx64" bitmap_size=3D0x%"PRIx64" start=3D0x%"PRIx64 --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600820115; cv=none; d=zohomail.com; s=zohoarc; b=WY2kgLkPb5RVpZFlosuXZf2tb5l/4PGXxdcvD/tpDdhrr62Bbpw8ocoZ9lroAlVksWpOe/GQ+pDgfziDlIPOLA3aQIIFvSHaVWpyfbw+KTXMOb7yJUJTTE8CcDzuwkZY9mGQ2d3VOAWyyg0/WRaDz7xbToEsbQGpMUCqgywmrJE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600820115; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=nTr15cRb6GSma9DjbXdPuJ6ztZZMoYN/ekEt+Mzxqj4=; b=fGdjT90gWaQbgoSFjFaiCLbS06r0payFTzk3jidsuyc5R1V9vkwN/AM6iTVK5sTWuuAV0JV4yIKpX7vDObMTdtUuTH5tz05hRWsZnFTWq+qLjUg4oCYMitjBi/q8P8oLWitL9aRlaNJTtPh6AHwzbHdPPP+Qlyh2wrbdSy9uKPs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600820115667320.4725133966716; Tue, 22 Sep 2020 17:15:15 -0700 (PDT) Received: from localhost ([::1]:50480 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsRG-0000fp-CM for importer@patchew.org; Tue, 22 Sep 2020 20:15:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGh-0004zu-7C for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:20 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14576) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGa-0001Ft-FI for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:16 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:58:54 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:58:58 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:50 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 15/17] vfio: Add ioctl to get dirty pages bitmap during dma unmap. Date: Wed, 23 Sep 2020 04:54:17 +0530 Message-ID: <1600817059-26721-16-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819134; bh=nTr15cRb6GSma9DjbXdPuJ6ztZZMoYN/ekEt+Mzxqj4=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=FMCMpsn2umsQXiV8H/8XEWgoXCxIoNYZMR1lx6i/HC+5hdBUT6VFIUlTlqSWARPno InvattLv3FNtR3+xW4ScgwnJ6e7U57qeSs0zQF1tO2HiE2C8ClB59pxpmoW1uOcbuE p0BR5hcpuToMoV7NN269VuJ5lCtReWqSTu+QYwteAHxgbfoiIfas9w/0nNV6FOadm6 jdq6qhI2NZpkL/UQex8QUQrWSaW1wZkyXn2iqtM17vI6OyrvF3W4VXo4+aoRmYb8nn ASIhMn3Ojuo3nVnV29B90Xd4NMtJkqlnojfg/t/+UpYdZQ4azYqWbmuKCBt7x33ljq 0AXSlil+QF3bA== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With vIOMMU, IO virtual address range can get unmapped while in pre-copy phase of migration. In that case, unmap ioctl should return pages pinned in that range and QEMU should find its correcponding guest physical addresses and report those dirty. Suggested-by: Alex Williamson Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- Note: Comments from v25 for this patch are not addressed in this series. https://www.mail-archive.com/qemu-devel@nongnu.org/msg714646.html Need to investigate more on the points raised in previous version. hw/vfio/common.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++= +--- 1 file changed, 86 insertions(+), 4 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index c36f275951c4..7eeaa368187a 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -315,11 +315,88 @@ static bool vfio_devices_all_stopped_and_saving(VFIOC= ontainer *container) return true; } =20 +static bool vfio_devices_all_running_and_saving(VFIOContainer *container) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + MigrationState *ms =3D migrate_get_current(); + + if (!migration_is_setup_or_active(ms->state)) { + return false; + } + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if ((vbasedev->device_state & VFIO_DEVICE_STATE_SAVING) && + (vbasedev->device_state & VFIO_DEVICE_STATE_RUNNING)) { + continue; + } else { + return false; + } + } + } + return true; +} + +static int vfio_dma_unmap_bitmap(VFIOContainer *container, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb) +{ + struct vfio_iommu_type1_dma_unmap *unmap; + struct vfio_bitmap *bitmap; + uint64_t pages =3D TARGET_PAGE_ALIGN(size) >> TARGET_PAGE_BITS; + int ret; + + unmap =3D g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); + + unmap->argsz =3D sizeof(*unmap) + sizeof(*bitmap); + unmap->iova =3D iova; + unmap->size =3D size; + unmap->flags |=3D VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP; + bitmap =3D (struct vfio_bitmap *)&unmap->data; + + /* + * cpu_physical_memory_set_dirty_lebitmap() expects pages in bitmap of + * TARGET_PAGE_SIZE to mark those dirty. Hence set bitmap_pgsize to + * TARGET_PAGE_SIZE. + */ + + bitmap->pgsize =3D TARGET_PAGE_SIZE; + bitmap->size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / + BITS_PER_BYTE; + + if (bitmap->size > container->max_dirty_bitmap_size) { + error_report("UNMAP: Size of bitmap too big 0x%llx", bitmap->size); + ret =3D -E2BIG; + goto unmap_exit; + } + + bitmap->data =3D g_try_malloc0(bitmap->size); + if (!bitmap->data) { + ret =3D -ENOMEM; + goto unmap_exit; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + if (!ret) { + cpu_physical_memory_set_dirty_lebitmap((uint64_t *)bitmap->data, + iotlb->translated_addr, pages); + } else { + error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); + } + + g_free(bitmap->data); +unmap_exit: + g_free(unmap); + return ret; +} + /* * DMA - Mapping and unmapping for the "type1" IOMMU interface used on x86 */ static int vfio_dma_unmap(VFIOContainer *container, - hwaddr iova, ram_addr_t size) + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb) { struct vfio_iommu_type1_dma_unmap unmap =3D { .argsz =3D sizeof(unmap), @@ -328,6 +405,11 @@ static int vfio_dma_unmap(VFIOContainer *container, .size =3D size, }; =20 + if (iotlb && container->dirty_pages_supported && + vfio_devices_all_running_and_saving(container)) { + return vfio_dma_unmap_bitmap(container, iova, size, iotlb); + } + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { /* * The type1 backend has an off-by-one bug in the kernel (71a7d3d7= 8e3c @@ -375,7 +457,7 @@ static int vfio_dma_map(VFIOContainer *container, hwadd= r iova, * the VGA ROM space. */ if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0 || - (errno =3D=3D EBUSY && vfio_dma_unmap(container, iova, size) =3D= =3D 0 && + (errno =3D=3D EBUSY && vfio_dma_unmap(container, iova, size, NULL)= =3D=3D 0 && ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0)) { return 0; } @@ -546,7 +628,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOM= MUTLBEntry *iotlb) } } =20 - ret =3D vfio_dma_unmap(container, iova, iotlb->addr_mask + 1); + ret =3D vfio_dma_unmap(container, iova, iotlb->addr_mask + 1, iotl= b); if (ret) { error_report("vfio_dma_unmap(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx") =3D %d (%m)", @@ -857,7 +939,7 @@ static void vfio_listener_region_del(MemoryListener *li= stener, } =20 if (try_unmap) { - ret =3D vfio_dma_unmap(container, iova, int128_get64(llsize)); + ret =3D vfio_dma_unmap(container, iova, int128_get64(llsize), NULL= ); if (ret) { error_report("vfio_dma_unmap(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx") =3D %d (%m)", --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600820214; cv=none; d=zohomail.com; s=zohoarc; b=Qu5xoPn1pyWlh/xzY4fzflcuqg/dqmT409IQ/SYqedbDlmsCPnS6XmvtvJXJhyAFN7q/ttoilVh1AZmKxNB+0qPTQmb74yiRTjx1A/aekWCQ0Cbbpf3KrlsH3uDM+MVZqR2jwVBPieeEzM0MXDYJR+5zEy205Ft9tLpBAjJ/5eo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600820214; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=FDG3vuJyY8+ryOkMz7MgJ657UGMIbqXwmaZOYNmOzn8=; b=HWbLhdLRCa9MYEbBFy55P+ddAEE8adJyCDP3AcARpJ+PmIrJi54+cYVgkzFQeyCUVhC/KUiVs5VOxLP70SXJS+ZtUgGgcTeF0PubTkRRvfsqDqldqfu8ec+BFO3VDLP5+OD1Xme9GSB6cVrfJzTjKYsXmjNQ09LdY1SYItFIBXY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600820214406221.01291146130507; Tue, 22 Sep 2020 17:16:54 -0700 (PDT) Received: from localhost ([::1]:52622 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsSr-0001eu-3G for importer@patchew.org; Tue, 22 Sep 2020 20:16:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGo-000512-Pg for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:28 -0400 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:14580) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGh-0001GH-Oc for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:26 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:59:02 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:59:06 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:58:58 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 16/17] vfio: Make vfio-pci device migration capable Date: Wed, 23 Sep 2020 04:54:18 +0530 Message-ID: <1600817059-26721-17-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819142; bh=FDG3vuJyY8+ryOkMz7MgJ657UGMIbqXwmaZOYNmOzn8=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=Q4C233PYP3gcpvPNkwnOtPFVNJkYQnZgmewMZlubw8ks7tTvFyUbcKAWusu4QD6r5 OQEWTo2ZDayBm8F6103hDWpMicV7HkidjvQApvo+ncenOgF0GjAE0Ucwy/huw5U8Ww trYkgekjKq4WcFtsV1uziGQcSiOJIBnXhbZiS10T0dw5Pgx9wbBkBWb3rTMVNi8lUd 3tPrlW+VnZT7moJ0pdCcILPERvMWEQ41WAiYdQ2lwJeEr9PjY+VbX/rsY1gaaW3nG4 3f4heRwKxJjPTz8ewidRtxXauo3/8cTVAuja99XK4JyrywsOcrHFjS8qypMer/jE6Q c84Joq9JHPX2w== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.65; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate26.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:16 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the device is not a failover primary device, call vfio_migration_probe() and vfio_migration_finalize() to enable migration support for those devices that support it respectively to tear it down again. Removed vfio_pci_vmstate structure. Removed migration blocker from VFIO PCI device specific structure and use migration blocker from generic structure of VFIO device. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Cornelia Huck --- hw/vfio/pci.c | 28 ++++++++-------------------- hw/vfio/pci.h | 1 - 2 files changed, 8 insertions(+), 21 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 9968cc553391..2418a448ebca 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2872,17 +2872,6 @@ static void vfio_realize(PCIDevice *pdev, Error **er= rp) return; } =20 - if (!pdev->failover_pair_id) { - error_setg(&vdev->migration_blocker, - "VFIO device doesn't support migration"); - ret =3D migrate_add_blocker(vdev->migration_blocker, errp); - if (ret) { - error_free(vdev->migration_blocker); - vdev->migration_blocker =3D NULL; - return; - } - } - vdev->vbasedev.name =3D g_path_get_basename(vdev->vbasedev.sysfsdev); vdev->vbasedev.ops =3D &vfio_pci_ops; vdev->vbasedev.type =3D VFIO_DEVICE_TYPE_PCI; @@ -3152,6 +3141,13 @@ static void vfio_realize(PCIDevice *pdev, Error **er= rp) } } =20 + if (!pdev->failover_pair_id) { + ret =3D vfio_migration_probe(&vdev->vbasedev, errp); + if (ret) { + error_report("%s: Migration disabled", vdev->vbasedev.name); + } + } + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); @@ -3166,11 +3162,6 @@ out_teardown: vfio_bars_exit(vdev); error: error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); - if (vdev->migration_blocker) { - migrate_del_blocker(vdev->migration_blocker); - error_free(vdev->migration_blocker); - vdev->migration_blocker =3D NULL; - } } =20 static void vfio_instance_finalize(Object *obj) @@ -3182,10 +3173,6 @@ static void vfio_instance_finalize(Object *obj) vfio_bars_finalize(vdev); g_free(vdev->emulated_config_bits); g_free(vdev->rom); - if (vdev->migration_blocker) { - migrate_del_blocker(vdev->migration_blocker); - error_free(vdev->migration_blocker); - } /* * XXX Leaking igd_opregion is not an oversight, we can't remove the * fw_cfg entry therefore leaking this allocation seems like the safest @@ -3213,6 +3200,7 @@ static void vfio_exitfn(PCIDevice *pdev) } vfio_teardown_msi(vdev); vfio_bars_exit(vdev); + vfio_migration_finalize(&vdev->vbasedev); } =20 static void vfio_pci_reset(DeviceState *dev) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 9f46af7e153f..0e3782b8e38a 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -173,7 +173,6 @@ struct VFIOPCIDevice { bool no_vfio_ioeventfd; bool enable_ramfb; VFIODisplay *dpy; - Error *migration_blocker; Notifier irqchip_change_notifier; }; =20 --=20 2.7.0 From nobody Sat Apr 27 12:33:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; t=1600819626; cv=none; d=zohomail.com; s=zohoarc; b=BuRxQrgN8PuXA2Mutwf1Co+T0SnQSUV1jJeJSm3WtgdVpV2ucPWDbRRcf9loRxlnuxBb9xF+S4J09YvRqAl0nzniyKpKg0wmdHcsscEJM5kP38V+/fLSssdjapuqOG+h+I4BtmxIEpwzuCs8qVS+ktQ8l5/NmLJmhxMYpfS4B/Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1600819626; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=06Dk6GrPxOyP7ELUmZBjCgQ+MM0vUwj1129i0MMRyuM=; b=S8iLMIi3Qn/co+BHTtSKMKRYQzHjdG3hyOzBUGOqpuE6kXtKyUHmqgG328gTsRVdD8Qp48Et82sgbJuLzwdOjbpUkvfRgRdpFGpyoJx1UCwqZx+kKG6psZZAp4H19hibe17QRqABs8J4+OZFuLXYXqwo5QqTMZko5+J5YMD9+Bg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1600819626920691.65770535614; Tue, 22 Sep 2020 17:07:06 -0700 (PDT) Received: from localhost ([::1]:58610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKsJN-0000KI-JC for importer@patchew.org; Tue, 22 Sep 2020 20:07:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47370) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGu-00058R-34 for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:32 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3896) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKsGr-0001H9-4k for qemu-devel@nongnu.org; Tue, 22 Sep 2020 20:04:31 -0400 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 22 Sep 2020 16:58:37 -0700 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 22 Sep 2020 23:59:14 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 22 Sep 2020 23:59:06 +0000 From: Kirti Wankhede To: , Subject: [PATCH v26 17/17] qapi: Add VFIO devices migration stats in Migration stats Date: Wed, 23 Sep 2020 04:54:19 +0530 Message-ID: <1600817059-26721-18-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> References: <1600817059-26721-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1600819117; bh=06Dk6GrPxOyP7ELUmZBjCgQ+MM0vUwj1129i0MMRyuM=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:X-NVConfidentiality:MIME-Version:Content-Type; b=hwdQWlFucYmUAE0C9bUSDAVn0k8t8txzIAlfE4981QPtL+Hnxi9e0kmAMPJee2MvU VeUxOtLCZIDBBnCW9nUF7ha/2JpBz+s8QYi5Yaa10xeL4zIFYfNs+bBWj3WbInzUMS edFImCRiza4njiSyfh40+akcYSUA4ecXcgBdUZEtjbokK9nkMpqEA3VvtlEjb0g+4f i5yFFDGY5VI32p2NIuEii79K05ZQABZoryPCaYFVkM5shL1Db7vP2K9ThTJwa11F+q YifPngjjJj6sjLzGHxvYeI+wKaN2cXHMR7AufvObK8uhn0RCve528Wi++CyCrqknK8 FYAtANcEioqVg== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.228.121.64; envelope-from=kwankhede@nvidia.com; helo=hqnvemgate25.nvidia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/22 19:57:12 X-ACL-Warn: Detected OS = Windows 7 or 8 [fuzzy] X-Spam_score_int: -85 X-Spam_score: -8.6 X-Spam_bar: -------- X-Spam_report: (-8.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.455, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, aik@ozlabs.ru, Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com, qemu-devel@nongnu.org, peterx@redhat.com, Kirti Wankhede , eauger@redhat.com, yi.l.liu@intel.com, quintela@redhat.com, ziye.yang@intel.com, armbru@redhat.com, mlevitsk@redhat.com, pasic@linux.ibm.com, felipe@nutanix.com, zhi.a.wang@intel.com, kevin.tian@intel.com, yan.y.zhao@intel.com, dgilbert@redhat.com, changpeng.liu@intel.com, eskultet@redhat.com, Ken.Xue@amd.com, jonathan.davies@nutanix.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added amount of bytes transferred to the target VM by all VFIO devices Signed-off-by: Kirti Wankhede Reviewed-by: Dr. David Alan Gilbert --- Note: Comments from v25 for this patch are not addressed yet. https://www.mail-archive.com/qemu-devel@nongnu.org/msg715620.html Alex, need more pointer on documentation part raised Markus Armbruster. hw/vfio/common.c | 20 ++++++++++++++++++++ hw/vfio/migration.c | 10 ++++++++++ include/qemu/vfio-helpers.h | 3 +++ migration/migration.c | 14 ++++++++++++++ monitor/hmp-cmds.c | 6 ++++++ qapi/migration.json | 17 +++++++++++++++++ 6 files changed, 70 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 7eeaa368187a..286cdaac8674 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -39,6 +39,7 @@ #include "trace.h" #include "qapi/error.h" #include "migration/migration.h" +#include "qemu/vfio-helpers.h" =20 VFIOGroupList vfio_group_list =3D QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -292,6 +293,25 @@ const MemoryRegionOps vfio_region_ops =3D { * Device state interfaces */ =20 +bool vfio_mig_active(void) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + + if (QLIST_EMPTY(&vfio_group_list)) { + return false; + } + + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->migration_blocker) { + return false; + } + } + } + return true; +} + static bool vfio_devices_all_stopped_and_saving(VFIOContainer *container) { VFIOGroup *group; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 822b68b4e015..c4226fa8b183 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -28,6 +28,7 @@ #include "pci.h" #include "trace.h" #include "hw/hw.h" +#include "qemu/vfio-helpers.h" =20 /* * Flags used as delimiter: @@ -40,6 +41,8 @@ #define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) #define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) =20 +static int64_t bytes_transferred; + static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int cou= nt, off_t off, bool iswrite) { @@ -289,6 +292,7 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vb= asedev, uint64_t *size) *size =3D data_size; } =20 + bytes_transferred +=3D data_size; return ret; } =20 @@ -770,6 +774,7 @@ static void vfio_migration_state_notifier(Notifier *not= ifier, void *data) } =20 vfio_set_dirty_page_tracking(vbasedev, false); + bytes_transferred =3D 0; } } =20 @@ -820,6 +825,11 @@ static int vfio_migration_init(VFIODevice *vbasedev, =20 /* ---------------------------------------------------------------------- = */ =20 +int64_t vfio_mig_bytes_transferred(void) +{ + return bytes_transferred; +} + int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) { VFIOContainer *container =3D vbasedev->group->container; diff --git a/include/qemu/vfio-helpers.h b/include/qemu/vfio-helpers.h index 1f057c2b9e40..26a7df0767b1 100644 --- a/include/qemu/vfio-helpers.h +++ b/include/qemu/vfio-helpers.h @@ -29,4 +29,7 @@ void qemu_vfio_pci_unmap_bar(QEMUVFIOState *s, int index,= void *bar, int qemu_vfio_pci_init_irq(QEMUVFIOState *s, EventNotifier *e, int irq_type, Error **errp); =20 +bool vfio_mig_active(void); +int64_t vfio_mig_bytes_transferred(void); + #endif diff --git a/migration/migration.c b/migration/migration.c index 58a5452471f9..b204bb1f6cd9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -56,6 +56,7 @@ #include "net/announce.h" #include "qemu/queue.h" #include "multifd.h" +#include "qemu/vfio-helpers.h" =20 #define MAX_THROTTLE (32 << 20) /* Migration transfer speed throttli= ng */ =20 @@ -996,6 +997,17 @@ static void populate_disk_info(MigrationInfo *info) } } =20 +static void populate_vfio_info(MigrationInfo *info) +{ +#ifdef CONFIG_LINUX + if (vfio_mig_active()) { + info->has_vfio =3D true; + info->vfio =3D g_malloc0(sizeof(*info->vfio)); + info->vfio->transferred =3D vfio_mig_bytes_transferred(); + } +#endif +} + static void fill_source_migration_info(MigrationInfo *info) { MigrationState *s =3D migrate_get_current(); @@ -1020,6 +1032,7 @@ static void fill_source_migration_info(MigrationInfo = *info) populate_time_info(info, s); populate_ram_info(info, s); populate_disk_info(info); + populate_vfio_info(info); break; case MIGRATION_STATUS_COLO: info->has_status =3D true; @@ -1028,6 +1041,7 @@ static void fill_source_migration_info(MigrationInfo = *info) case MIGRATION_STATUS_COMPLETED: populate_time_info(info, s); populate_ram_info(info, s); + populate_vfio_info(info); break; case MIGRATION_STATUS_FAILED: info->has_status =3D true; diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c index 7711726fd222..40d60d6a6651 100644 --- a/monitor/hmp-cmds.c +++ b/monitor/hmp-cmds.c @@ -355,6 +355,12 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) } monitor_printf(mon, "]\n"); } + + if (info->has_vfio) { + monitor_printf(mon, "vfio device transferred: %" PRIu64 " kbytes\n= ", + info->vfio->transferred >> 10); + } + qapi_free_MigrationInfo(info); } =20 diff --git a/qapi/migration.json b/qapi/migration.json index 675f70bb6734..3535977123d3 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -147,6 +147,18 @@ 'active', 'postcopy-active', 'postcopy-paused', 'postcopy-recover', 'completed', 'failed', 'colo', 'pre-switchover', 'device', 'wait-unplug' ] } +## +# @VfioStats: +# +# Detailed VFIO devices migration statistics +# +# @transferred: amount of bytes transferred to the target VM by VFIO devic= es +# +# Since: 5.1 +# +## +{ 'struct': 'VfioStats', + 'data': {'transferred': 'int' } } =20 ## # @MigrationInfo: @@ -208,11 +220,16 @@ # # @socket-address: Only used for tcp, to know what the real port is (Since= 4.0) # +# @vfio: @VfioStats containing detailed VFIO devices migration statistics, +# only returned if VFIO device is present, migration is supported b= y all +# VFIO devices and status is 'active' or 'completed' (since 5.1) +# # Since: 0.14.0 ## { 'struct': 'MigrationInfo', 'data': {'*status': 'MigrationStatus', '*ram': 'MigrationStats', '*disk': 'MigrationStats', + '*vfio': 'VfioStats', '*xbzrle-cache': 'XBZRLECacheStats', '*total-time': 'int', '*expected-downtime': 'int', --=20 2.7.0