From nobody Sun May 5 01:04:48 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1562386386; cv=none; d=zoho.com; s=zohoarc; b=iLLOV9tlhuDZViV5iXNwQwQvFyBN9+uFdZiOaXc4oGj2G4+O7RhjxglXfTq3D3Sal8lTfEt6c0AoBeIEod4MI/APCaXyzx7azSRGOiflmUUkBFtO81poz6UW9ANCFt+RJGoztcXzorr9HdPH9Rtt9C6QN05HI21di+3U/VtdliY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1562386386; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=YfW1aH+Wpw10vSj+L0zxMi7E9GI7IXk5GtJx8ieye+k=; b=Sb0wLHzmMRSQiK8OQCgeqIqfOrr3Pz9IamBLNImzVh/SzKR4BBxS06CfBg7XQgBMUZeBt7+ywl46a7sQVemDm9cMGvr0jEbd+mdlOiXATith7oWXmntDI5vkWXOeaPYKJVj/oI14JVqtKjWyomdaJOUuKkTTf8+JCTol/u2tfuE= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1562386386361233.47886405941836; Fri, 5 Jul 2019 21:13:06 -0700 (PDT) Received: from localhost ([::1]:57656 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hjc4K-0008SQ-8x for importer@patchew.org; Sat, 06 Jul 2019 00:13:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53252) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hjc1K-0006ES-CN for qemu-devel@nongnu.org; Sat, 06 Jul 2019 00:09:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hjc1I-0000kf-Ec for qemu-devel@nongnu.org; Sat, 06 Jul 2019 00:09:54 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:42977) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hjc1G-0000ii-Iv for qemu-devel@nongnu.org; Sat, 06 Jul 2019 00:09:51 -0400 Received: by mail-pg1-x541.google.com with SMTP id t132so5061031pgb.9 for ; Fri, 05 Jul 2019 21:09:49 -0700 (PDT) Received: from localhost.localdomain ([2405:204:22f:6255:6f39:949d:5942:eaaf]) by smtp.gmail.com with ESMTPSA id q126sm17330847pfq.123.2019.07.05.21.09.45 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 05 Jul 2019 21:09:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YfW1aH+Wpw10vSj+L0zxMi7E9GI7IXk5GtJx8ieye+k=; b=W0SKeNVqym5z1mgVyAkLzSoTBlePkQ5LGRkGPvvn05zGU8qIX2hTd7bm+ZX6vYQL11 rfvO9Ql1UybmuOs6S8GhjG0u24iP9gRpst7UBwT2UyT35yP+CncSs3EgjM1ULUkCDr7v +0g1aPxBvco3S6K0Aud5TwVGg3QHZf6QAIrkVGDHBVPMLEY2NgtY/RD6W3XZXIiAVSpm a+lV6JiqKuwNHS7HUiuEshThuBu/sb+PB4Bf9cumIM5fcnAep6Yd75Y7JHcIz5qYRhlZ Xg85YYu2pEtBwVCWVp4aynFIRIaAljgC9cdnXXKoejMZWk8x1CzAJ2X3TFH/aUTTIoWu 3RdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YfW1aH+Wpw10vSj+L0zxMi7E9GI7IXk5GtJx8ieye+k=; b=DRDJYrwRsp3czxD3oN5m1tHJhHMknwCdTT9P+qX2DswtUiRShAR5A/wRBdSKr96NrH EQ4W53/Yoj9O05B9P54Xxa4j+WKJhTqhyc4Rw4aUHddy59l8qxd+GJ2iPk7i2mMrsqAB 4xVVscCSYGXbRE/+hhw1KiSxNrrBLhsqourg48RXXBEGKWslqWSyRYFkBsADiuHDTAG2 OXpFx+8jUNVo/qPgoQda6G334A3N4p3QplKBn+xwpzbKXKx0US7fvT7pumE/WgyRu4hA V1FzjG0IjOAN88eT+g6KJldE5Is3XO/CFx7hfyo4Lb82rSQHI8UrFO9wBqvwjqNUBUCj Bpbg== X-Gm-Message-State: APjAAAVfIvI9Zpc3IZzdWqVwMzhRtpQIVuJFa3O/AXhxRHWfV178pSwp xgVSnhlqXRaAPOuVdpFtyQfydPdC X-Google-Smtp-Source: APXvYqzPAq0vMI05Jg7SnrRYwfS1j/RbNAd8wPHQFmTynqTZZO1pCZjhnCxFi/eTKQrpox3XdzHGZA== X-Received: by 2002:a63:c20e:: with SMTP id b14mr8805376pgd.96.1562386188503; Fri, 05 Jul 2019 21:09:48 -0700 (PDT) From: Sukrit Bhatnagar To: qemu-devel@nongnu.org Date: Sat, 6 Jul 2019 09:39:39 +0530 Message-Id: <20190706040940.7884-2-skrtbhtngr@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190706040940.7884-1-skrtbhtngr@gmail.com> References: <20190706040940.7884-1-skrtbhtngr@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 Subject: [Qemu-devel] [RFC v2 1/2] hw/pvrdma: make DSR mapping idempotent in load_dsr() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yuval Shaia Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Map to DSR only when there is no mapping done already i.e., when dev->dsr_info.dsr is NULL. This allows the rest of mappings and ring inits to be done by calling load_dsr() when DSR has already been mapped to, somewhere else. Move free_dsr() out of load_dsr() and call it before the latter as and when needed. This aids the case where load_dsr() is called having DSR mapping already done, but the rest of map and init operations are pending, and prevents an unmap of the DSR. Cc: Marcel Apfelbaum Cc: Yuval Shaia Signed-off-by: Sukrit Bhatnagar --- hw/rdma/vmw/pvrdma_main.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index adcf79cd63..6c90db96f9 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -172,15 +172,15 @@ static int load_dsr(PVRDMADev *dev) DSRInfo *dsr_info; struct pvrdma_device_shared_region *dsr; =20 - free_dsr(dev); - - /* Map to DSR */ - dev->dsr_info.dsr =3D rdma_pci_dma_map(pci_dev, dev->dsr_info.dma, - sizeof(struct pvrdma_device_shared_region)); if (!dev->dsr_info.dsr) { - rdma_error_report("Failed to map to DSR"); - rc =3D -ENOMEM; - goto out; + /* Map to DSR */ + dev->dsr_info.dsr =3D rdma_pci_dma_map(pci_dev, dev->dsr_info.dma, + sizeof(struct pvrdma_device_shared_regio= n)); + if (!dev->dsr_info.dsr) { + rdma_error_report("Failed to map to DSR"); + rc =3D -ENOMEM; + goto out; + } } =20 /* Shortcuts */ @@ -402,6 +402,7 @@ static void pvrdma_regs_write(void *opaque, hwaddr addr= , uint64_t val, case PVRDMA_REG_DSRHIGH: trace_pvrdma_regs_write(addr, val, "DSRHIGH", ""); dev->dsr_info.dma |=3D val << 32; + free_dsr(dev); load_dsr(dev); init_dsr_dev_caps(dev); break; --=20 2.21.0 From nobody Sun May 5 01:04:48 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1562386315; cv=none; d=zoho.com; s=zohoarc; b=JglPFyS5Vyem0GfzKXFySriFgfMDuLDxjrgJwbXRhOEh0+GGaRs0OJ3Biwi+DAmEnVimPuy9zyCJJkxBJ5PEFXKt4yNXAm+LCMZE9YYARcRoAt+rj9yAoZhZ9QT4hy+/lWkK9LY8Vo34KSdt/+jPLyqkdCZArLBRGR3j/jj36mc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1562386315; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=fPlaR/RqYJOcV/wH+fB4Exz5LrXEWktWATpatpBFvpM=; b=YrVhi5aupYGquzdu/ZsfbFkrpE6T6jUr/9yQQ+L166Y1Mm5e5fN7osJk6nGhUczyV+FJDfET1x/1/wqqS35bJhnVKXjWUJsMch3ctqJWpsqJz15xj1MIzdoQSStdzIblcwfI86tl1paNurib7n1n4y22UO6xb5nEJDiAwPAiM7s= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1562386315411754.0307434699399; Fri, 5 Jul 2019 21:11:55 -0700 (PDT) Received: from localhost ([::1]:57644 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hjc3B-0007Bt-Fn for importer@patchew.org; Sat, 06 Jul 2019 00:11:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53263) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hjc1M-0006Ek-7n for qemu-devel@nongnu.org; Sat, 06 Jul 2019 00:09:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hjc1K-0000lr-C4 for qemu-devel@nongnu.org; Sat, 06 Jul 2019 00:09:56 -0400 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:38840) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hjc1J-0000kp-6W for qemu-devel@nongnu.org; Sat, 06 Jul 2019 00:09:54 -0400 Received: by mail-pl1-x641.google.com with SMTP id 9so5450089ple.5 for ; Fri, 05 Jul 2019 21:09:53 -0700 (PDT) Received: from localhost.localdomain ([2405:204:22f:6255:6f39:949d:5942:eaaf]) by smtp.gmail.com with ESMTPSA id q126sm17330847pfq.123.2019.07.05.21.09.49 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 05 Jul 2019 21:09:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fPlaR/RqYJOcV/wH+fB4Exz5LrXEWktWATpatpBFvpM=; b=pLUjFOCahEbZMEf0vwt/aothmGLK+23E2hJVmP17C3SdOTaWvTv/DC3q6SXyLGGpra Ay9sgxz2LlU6PmAxkGe4ImDJL5RKl5ie0P9UKSZbQ9cmjaSdFPvr7DWw/lzZVJOuX5nN TbTYz50+a+zThAfhT9ZtdPzYn7TeCT5of+Ym4ZMlsUY9wwInZ9Kmncvz8LOpSCUnrra0 c2ltJE7+6WUFkZwpbb6gL+eHzyb+asdFSI51a1tI68LyKbUYHOvxIDyDO7khFG6nRSFg D5BJQ9DsgDA70ucGaVi9JUM1upV8IL1s+VIi5FS+z0rncpMXXdu8ll9gejqDEF5bEh1F zEqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fPlaR/RqYJOcV/wH+fB4Exz5LrXEWktWATpatpBFvpM=; b=ByCLKXD/pMC0q9BwNlT/cwFXZHrVwFXTtuQbXcNrebcd5kS0XTXzlSVLbyWnHUJQVL GqLoXWTsL8bTr+C+yW0rv9WNME9T3QQvo8Gh/gqhe0BgBQo3C3stfmXU3q8k+Ck9SsB3 1udfshyNTJOsK+zC8gugykobZzfXNRVmJwjU0QCJu5perLGQy9yPlCTxr5ONF6njtz2/ aHzDlIzlH1stJWKsi9WVsU9lOow/O1wiTsyEcATqwnN09044DZ5aTfiE5t5iIGgqek/4 jYCWblWWMgIx44HmgbY1/xWjfPPREiHTD9S+W+7ZfmSpg+QDXdqqTv4VTjWYroZUG2Fp 5XFw== X-Gm-Message-State: APjAAAW6SD6+f6GmQUNQXU6KF36d65ClaU/qG7r3F1kCfNbjtWiNMKCK ib3S6ypTCuM+Wh74zvHswG/v2s1y X-Google-Smtp-Source: APXvYqxS3JGDrFserN4A8HEly63a7Qgii+XdrUVXoXhdsgGWSjoazqnzu0SCue5o0nu5YAtpveuVyw== X-Received: by 2002:a17:902:e282:: with SMTP id cf2mr9538885plb.301.1562386191851; Fri, 05 Jul 2019 21:09:51 -0700 (PDT) From: Sukrit Bhatnagar To: qemu-devel@nongnu.org Date: Sat, 6 Jul 2019 09:39:40 +0530 Message-Id: <20190706040940.7884-3-skrtbhtngr@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190706040940.7884-1-skrtbhtngr@gmail.com> References: <20190706040940.7884-1-skrtbhtngr@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::641 Subject: [Qemu-devel] [RFC v2 2/2] hw/pvrdma: add live migration support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yuval Shaia Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Use VMStateDescription for migrating device state. Currently, 'vmstate_pvrdma' describes the PCI and MSIX state for pvrdma and 'vmstate_pvrdma_dsr_dma' describes a temporary state containing some values obtained only after mapping to dsr in the source. Since the dsr will not be available on dest until we map to the dma address we had on source, these values cannot be migrated directly. Add PVRDMAMigTmp to store this temporary state which consists of dma addresses and ring page information. The 'parent' member is used to refer to the device state (PVRDMADev) so that parent PCI device object is accessible, which is needed to remap to DSR. pvrdma_dsr_dma_pre_save() saves the dsr state into this temporary representation and pvrdma_dsr_dma_post_load() loads it back. This load function also remaps to the dsr and and calls load_dsr() for further map and ring init operations. Please note that this call to load_dsr() can be removed from the migration flow and included in pvrdma_regs_write() to perform a lazy load. As of now, migration will fail if there in an error in load_dsr(). Also, there might be a considerable amount of pages in the rings, which will have dma map operations when the init functions are called. If this takes noticeable time, it might be better to have lazy load instead. Cc: Marcel Apfelbaum Cc: Yuval Shaia Signed-off-by: Sukrit Bhatnagar --- hw/rdma/vmw/pvrdma_main.c | 87 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 6c90db96f9..4a10bd2fc7 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -28,6 +28,7 @@ #include "sysemu/sysemu.h" #include "monitor/monitor.h" #include "hw/rdma/rdma.h" +#include "migration/register.h" =20 #include "../rdma_rm.h" #include "../rdma_backend.h" @@ -593,6 +594,91 @@ static void pvrdma_shutdown_notifier(Notifier *n, void= *opaque) pvrdma_fini(pci_dev); } =20 +struct PVRDMAMigTmp { + PVRDMADev *parent; + uint64_t dma; + uint64_t cmd_slot_dma; + uint64_t resp_slot_dma; + uint32_t cq_ring_pages_num_pages; + uint64_t cq_ring_pages_pdir_dma; + uint32_t async_ring_pages_num_pages; + uint64_t async_ring_pages_pdir_dma; +}; + +static int pvrdma_dsr_dma_pre_save(void *opaque) +{ + struct PVRDMAMigTmp *tmp =3D opaque; + DSRInfo *dsr_info =3D &tmp->parent->dsr_info; + struct pvrdma_device_shared_region *dsr =3D dsr_info->dsr; + + tmp->dma =3D dsr_info->dma; + tmp->cmd_slot_dma =3D dsr->cmd_slot_dma; + tmp->resp_slot_dma =3D dsr->resp_slot_dma; + tmp->cq_ring_pages_num_pages =3D dsr->cq_ring_pages.num_pages; + tmp->cq_ring_pages_pdir_dma =3D dsr->cq_ring_pages.pdir_dma; + tmp->async_ring_pages_num_pages =3D dsr->async_ring_pages.num_pages; + tmp->async_ring_pages_pdir_dma =3D dsr->async_ring_pages.pdir_dma; + + return 0; +} + +static int pvrdma_dsr_dma_post_load(void *opaque, int version_id) +{ + struct PVRDMAMigTmp *tmp =3D opaque; + PVRDMADev *dev =3D tmp->parent; + PCIDevice *pci_dev =3D PCI_DEVICE(dev); + DSRInfo *dsr_info =3D &dev->dsr_info; + struct pvrdma_device_shared_region *dsr; + + dsr_info->dma =3D tmp->dma; + dsr_info->dsr =3D rdma_pci_dma_map(pci_dev, dsr_info->dma, + sizeof(struct pvrdma_device_shared_region)= ); + if (!dsr_info->dsr) { + rdma_error_report("Failed to map to DSR"); + return -ENOMEM; + } + + dsr =3D dsr_info->dsr; + dsr->cmd_slot_dma =3D tmp->cmd_slot_dma; + dsr->resp_slot_dma =3D tmp->resp_slot_dma; + dsr->cq_ring_pages.num_pages =3D tmp->cq_ring_pages_num_pages; + dsr->cq_ring_pages.pdir_dma =3D tmp->cq_ring_pages_pdir_dma; + dsr->async_ring_pages.num_pages =3D tmp->async_ring_pages_num_pages; + dsr->async_ring_pages.pdir_dma =3D tmp->async_ring_pages_pdir_dma; + + return load_dsr(dev); +} + +static const VMStateDescription vmstate_pvrdma_dsr_dma =3D { + .name =3D "pvrdma-dsr-dma", + .pre_save =3D pvrdma_dsr_dma_pre_save, + .post_load =3D pvrdma_dsr_dma_post_load, + .fields =3D (VMStateField[]) { + VMSTATE_UINT64(dma, struct PVRDMAMigTmp), + VMSTATE_UINT64(cmd_slot_dma, struct PVRDMAMigTmp), + VMSTATE_UINT64(resp_slot_dma, struct PVRDMAMigTmp), + VMSTATE_UINT32(async_ring_pages_num_pages, struct PVRDMAMigTmp= ), + VMSTATE_UINT64(async_ring_pages_pdir_dma, struct PVRDMAMigTmp), + VMSTATE_UINT32(cq_ring_pages_num_pages, struct PVRDMAMigTmp), + VMSTATE_UINT64(cq_ring_pages_pdir_dma, struct PVRDMAMigTmp), + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription vmstate_pvrdma =3D { + .name =3D "pvrdma", + .version_id =3D 1, + .minimum_version_id =3D 1, + .fields =3D (VMStateField[]) { + VMSTATE_PCI_DEVICE(parent_obj, PVRDMADev), + VMSTATE_MSIX(parent_obj, PVRDMADev), + VMSTATE_WITH_TMP(PVRDMADev, + struct PVRDMAMigTmp, + vmstate_pvrdma_dsr_dma), + VMSTATE_END_OF_LIST() + } +}; + static void pvrdma_realize(PCIDevice *pdev, Error **errp) { int rc =3D 0; @@ -688,6 +774,7 @@ static void pvrdma_class_init(ObjectClass *klass, void = *data) =20 dc->desc =3D "RDMA Device"; dc->props =3D pvrdma_dev_properties; + dc->vmsd =3D &vmstate_pvrdma; set_bit(DEVICE_CATEGORY_NETWORK, dc->categories); =20 ir->print_statistics =3D pvrdma_print_statistics; --=20 2.21.0