From nobody Tue Apr 30 05:05:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507068923559999.0835426084119; Tue, 3 Oct 2017 15:15:23 -0700 (PDT) Received: from localhost ([::1]:60478 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzVTD-0000f8-SS for importer@patchew.org; Tue, 03 Oct 2017 18:15:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52619) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzVQg-0007Sc-Ky for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:12:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dzVQd-0002if-Ea for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:12:42 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49372) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dzVQd-0002iG-7y for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:12:39 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4F52781227 for ; Tue, 3 Oct 2017 22:12:38 +0000 (UTC) Received: from gimli.home (ovpn-116-96.phx2.redhat.com [10.3.116.96]) by smtp.corp.redhat.com (Postfix) with ESMTP id 23D635C549; Tue, 3 Oct 2017 22:12:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 4F52781227 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=alex.williamson@redhat.com From: Alex Williamson To: qemu-devel@nongnu.org Date: Tue, 03 Oct 2017 16:12:36 -0600 Message-ID: <20171003221230.17156.56507.stgit@gimli.home> In-Reply-To: <20171003221012.17156.23559.stgit@gimli.home> References: <20171003221012.17156.23559.stgit@gimli.home> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 03 Oct 2017 22:12:38 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 1/3] vfio/pci: Do not unwind on error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 If vfio_add_std_cap() errors then going to out prepends irrelevant errors for capabilities we haven't attempted to add as we unwind our recursive stack. Just return error. Fixes: 7ef165b9a8d9 ("vfio/pci: Pass an error object to vfio_add_capabiliti= es") Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 31e1edf44745..916d365dfab3 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1826,7 +1826,7 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint= 8_t pos, Error **errp) if (next) { ret =3D vfio_add_std_cap(vdev, next, errp); if (ret) { - goto out; + return ret; } } else { /* Begin the rebuild, use QEMU emulated list bits */ @@ -1862,7 +1862,7 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint= 8_t pos, Error **errp) ret =3D pci_add_capability(pdev, cap_id, pos, size, errp); break; } -out: + if (ret < 0) { error_prepend(errp, "failed to add PCI capability 0x%x[0x%x]@0x%x: ", From nobody Tue Apr 30 05:05:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507068863687531.1622181657249; Tue, 3 Oct 2017 15:14:23 -0700 (PDT) Received: from localhost ([::1]:60469 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzVS2-00089E-Kl for importer@patchew.org; Tue, 03 Oct 2017 18:14:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52729) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzVQu-0007Z6-4z for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:12:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dzVQq-0002t2-V6 for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:12:56 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52168) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dzVQq-0002sJ-OF for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:12:52 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E5CAC61467 for ; Tue, 3 Oct 2017 22:12:51 +0000 (UTC) Received: from gimli.home (ovpn-116-96.phx2.redhat.com [10.3.116.96]) by smtp.corp.redhat.com (Postfix) with ESMTP id B95A118C52; Tue, 3 Oct 2017 22:12:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com E5CAC61467 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=alex.williamson@redhat.com From: Alex Williamson To: qemu-devel@nongnu.org Date: Tue, 03 Oct 2017 16:12:49 -0600 Message-ID: <20171003221243.17156.78713.stgit@gimli.home> In-Reply-To: <20171003221012.17156.23559.stgit@gimli.home> References: <20171003221012.17156.23559.stgit@gimli.home> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 03 Oct 2017 22:12:52 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 2/3] vfio/pci: Add virtual capabilities quirk infrastructure X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 If the hypervisor needs to add purely virtual capabilties, give us a hook through quirks to do that. Note that we determine the maximum size for a capability based on the physical device, if we insert a virtual capability, that can change. Therefore if maximum size is smaller after added virt capabilities, use that. Signed-off-by: Alex Williamson --- hw/vfio/pci-quirks.c | 4 ++++ hw/vfio/pci.c | 8 ++++++++ hw/vfio/pci.h | 1 + 3 files changed, 13 insertions(+) diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index 349085ea12bc..40aaae76feb9 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -1850,3 +1850,7 @@ void vfio_setup_resetfn_quirk(VFIOPCIDevice *vdev) break; } } +int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **errp) +{ + return 0; +} diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 916d365dfab3..bfeaaef22d00 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1833,8 +1833,16 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uin= t8_t pos, Error **errp) pdev->config[PCI_CAPABILITY_LIST] =3D 0; vdev->emulated_config_bits[PCI_CAPABILITY_LIST] =3D 0xff; vdev->emulated_config_bits[PCI_STATUS] |=3D PCI_STATUS_CAP_LIST; + + ret =3D vfio_add_virt_caps(vdev, errp); + if (ret) { + return ret; + } } =20 + /* Scale down size, esp in case virt caps were added above */ + size =3D MIN(size, vfio_std_cap_max_size(pdev, pos)); + /* Use emulated next pointer to allow dropping caps */ pci_set_byte(vdev->emulated_config_bits + pos + PCI_CAP_LIST_NEXT, 0xf= f); =20 diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index a8366bb2a74a..958cee058b3b 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -160,6 +160,7 @@ void vfio_bar_quirk_setup(VFIOPCIDevice *vdev, int nr); void vfio_bar_quirk_exit(VFIOPCIDevice *vdev, int nr); void vfio_bar_quirk_finalize(VFIOPCIDevice *vdev, int nr); void vfio_setup_resetfn_quirk(VFIOPCIDevice *vdev); +int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **errp); =20 int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp); =20 From nobody Tue Apr 30 05:05:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507068988940807.0778977453848; Tue, 3 Oct 2017 15:16:28 -0700 (PDT) Received: from localhost ([::1]:60481 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzVUG-0001Gg-SW for importer@patchew.org; Tue, 03 Oct 2017 18:16:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52795) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzVRC-0007mO-6t for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:13:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dzVR8-00036P-VM for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:13:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35382) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dzVR8-00035j-Lw for qemu-devel@nongnu.org; Tue, 03 Oct 2017 18:13:10 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C78F0356D9 for ; Tue, 3 Oct 2017 22:13:09 +0000 (UTC) Received: from gimli.home (ovpn-116-96.phx2.redhat.com [10.3.116.96]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8DE5517DE3; Tue, 3 Oct 2017 22:13:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com C78F0356D9 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=alex.williamson@redhat.com From: Alex Williamson To: qemu-devel@nongnu.org Date: Tue, 03 Oct 2017 16:13:07 -0600 Message-ID: <20171003221257.17156.88241.stgit@gimli.home> In-Reply-To: <20171003221012.17156.23559.stgit@gimli.home> References: <20171003221012.17156.23559.stgit@gimli.home> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Tue, 03 Oct 2017 22:13:09 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 3/3] vfio/pci: Add NVIDIA GPUDirect Cliques support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 NVIDIA has defined a specification for creating GPUDirect "cliques", where devices with the same clique ID support direct peer-to-peer DMA. When running on bare-metal, tools like NVIDIA's p2pBandwidthLatencyTest (part of cuda-samples) determine which GPUs can support peer-to-peer based on chipset and topology. When running in a VM, these tools have no visibility to the physical hardware support or topology. This option allows the user to specify hints via a vendor defined capability. For instance: This enables two cliques. The first is a singleton clique with ID 0, for the first hostdev defined in the XML (note that since cliques define peer-to-peer sets, singleton clique offer no benefit). The subsequent two hostdevs are both added to clique ID 1, indicating peer-to-peer is possible between these devices. QEMU only provides validation that the clique ID is valid and applied to an NVIDIA graphics device, any validation that the resulting cliques are functional and valid is the user's responsibility. The NVIDIA specification allows a 4-bit clique ID, thus valid values are 0-15. Signed-off-by: Alex Williamson --- hw/vfio/pci-quirks.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++= ++++ hw/vfio/pci.c | 5 ++ hw/vfio/pci.h | 3 + 3 files changed, 118 insertions(+) diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index 40aaae76feb9..14291c2a16b2 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -14,6 +14,7 @@ #include "qemu/error-report.h" #include "qemu/range.h" #include "qapi/error.h" +#include "qapi/visitor.h" #include "hw/nvram/fw_cfg.h" #include "pci.h" #include "trace.h" @@ -1850,7 +1851,116 @@ void vfio_setup_resetfn_quirk(VFIOPCIDevice *vdev) break; } } + +/* + * The NVIDIA GPUDirect P2P Vendor capability allows the user to specify + * devices as a member of a clique. Devices within the same clique ID + * are capable of direct P2P. It's the user's responsibility that this + * is correct. The spec says that this may reside at any unused config + * offset, but reserves and recommends hypervisors place this at C8h. + * The spec also states that the hypervisor should place this capability + * at the end of the capability list, thus next is defined as 0h. + * + * +----------------+----------------+----------------+----------------+ + * | sig 7:0 ('P') | vndr len (8h) | next (0h) | cap id (9h) | + * +----------------+----------------+----------------+----------------+ + * | rsvd 15:7(0h),id 6:3,ver 2:0(0h)| sig 23:8 ('P2') | + * +---------------------------------+---------------------------------+ + * + * https://lists.gnu.org/archive/html/qemu-devel/2017-08/pdfUda5iEpgOS.pdf + */ +static void get_nv_gpudirect_clique_id(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + DeviceState *dev =3D DEVICE(obj); + Property *prop =3D opaque; + uint8_t *ptr =3D qdev_get_prop_ptr(dev, prop); + + visit_type_uint8(v, name, ptr, errp); +} + +static void set_nv_gpudirect_clique_id(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + DeviceState *dev =3D DEVICE(obj); + Property *prop =3D opaque; + uint8_t value, *ptr =3D qdev_get_prop_ptr(dev, prop); + Error *local_err =3D NULL; + + if (dev->realized) { + qdev_prop_set_after_realize(dev, name, errp); + return; + } + + visit_type_uint8(v, name, &value, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + + if (value & ~0xF) { + error_setg(errp, "Property %s: valid range 0-15", name); + return; + } + + *ptr =3D value; +} + +const PropertyInfo qdev_prop_nv_gpudirect_clique =3D { + .name =3D "uint4", + .description =3D "NVIDIA GPUDirect Clique ID (0 - 15)", + .get =3D get_nv_gpudirect_clique_id, + .set =3D set_nv_gpudirect_clique_id, +}; + +static int vfio_add_nv_gpudirect_cap(VFIOPCIDevice *vdev, Error **errp) +{ + PCIDevice *pdev =3D &vdev->pdev; + int ret, pos =3D 0xC8; + + if (vdev->nv_gpudirect_clique =3D=3D 0xFF) { + return 0; + } + + if (!vfio_pci_is(vdev, PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID)) { + error_setg(errp, "NVIDIA GPUDirect Clique ID: invalid device vendo= r"); + return -EINVAL; + } + + if (pci_get_byte(pdev->config + PCI_CLASS_DEVICE + 1) !=3D + PCI_BASE_CLASS_DISPLAY) { + error_setg(errp, "NVIDIA GPUDirect Clique ID: unsupported PCI clas= s"); + return -EINVAL; + } + + ret =3D pci_add_capability(pdev, PCI_CAP_ID_VNDR, pos, 8, errp); + if (ret < 0) { + error_prepend(errp, "Failed to add NVIDIA GPUDirect cap: "); + return ret; + } + + memset(vdev->emulated_config_bits + pos, 0xFF, 8); + pos +=3D PCI_CAP_FLAGS; + pci_set_byte(pdev->config + pos++, 8); + pci_set_byte(pdev->config + pos++, 'P'); + pci_set_byte(pdev->config + pos++, '2'); + pci_set_byte(pdev->config + pos++, 'P'); + pci_set_byte(pdev->config + pos++, vdev->nv_gpudirect_clique << 3); + pci_set_byte(pdev->config + pos, 0); + + return 0; +} + int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **errp) { + int ret; + + ret =3D vfio_add_nv_gpudirect_cap(vdev, errp); + if (ret) { + return ret; + } + return 0; } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bfeaaef22d00..9e86db7c3b6d 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2970,6 +2970,8 @@ static void vfio_instance_init(Object *obj) vdev->host.bus =3D ~0U; vdev->host.slot =3D ~0U; vdev->host.function =3D ~0U; + + vdev->nv_gpudirect_clique =3D 0xFF; } =20 static Property vfio_pci_dev_properties[] =3D { @@ -2994,6 +2996,9 @@ static Property vfio_pci_dev_properties[] =3D { DEFINE_PROP_UINT32("x-pci-sub-device-id", VFIOPCIDevice, sub_device_id, PCI_ANY_ID), DEFINE_PROP_UINT32("x-igd-gms", VFIOPCIDevice, igd_gms, 0), + DEFINE_PROP_UNSIGNED_NODEFAULT("x-nv-gpudirect-clique", VFIOPCIDevice, + nv_gpudirect_clique, + qdev_prop_nv_gpudirect_clique, uint8_t), /* * TODO - support passed fds... is this necessary? * DEFINE_PROP_STRING("vfiofd", VFIOPCIDevice, vfiofd_name), diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 958cee058b3b..502a5755b944 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -135,6 +135,7 @@ typedef struct VFIOPCIDevice { int32_t bootindex; uint32_t igd_gms; uint8_t pm_cap; + uint8_t nv_gpudirect_clique; bool pci_aer; bool req_enabled; bool has_flr; @@ -162,6 +163,8 @@ void vfio_bar_quirk_finalize(VFIOPCIDevice *vdev, int n= r); void vfio_setup_resetfn_quirk(VFIOPCIDevice *vdev); int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **errp); =20 +extern const PropertyInfo qdev_prop_nv_gpudirect_clique; + int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp); =20 int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev,