From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542098984216976.3500475674723; Tue, 13 Nov 2018 00:49:44 -0800 (PST) Received: from localhost ([::1]:52736 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUO8-000079-1z for importer@patchew.org; Tue, 13 Nov 2018 03:49:36 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45206) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGx-0002x8-TH for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:13 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU6w-0006VM-2M for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:51 -0500 Received: from [107.173.13.209] (port=60982 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU6u-00068t-4I; Tue, 13 Nov 2018 03:31:48 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id EB9C8AE80494; Tue, 13 Nov 2018 03:31:09 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:30:58 +1100 Message-Id: <20181113083104.2692-2-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 1/7] vfio/spapr: Fix indirect levels calculation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The current code assumes that we can address more bits on a PCI bus for DMA than we really can. Limit to the known tested maximum of 55 bits and assume 64K IOMMU pages. Signed-off-by: Alexey Kardashevskiy --- hw/vfio/spapr.c | 3 ++- hw/vfio/trace-events | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/hw/vfio/spapr.c b/hw/vfio/spapr.c index becf71a..f5fdc53 100644 --- a/hw/vfio/spapr.c +++ b/hw/vfio/spapr.c @@ -183,7 +183,7 @@ int vfio_spapr_create_window(VFIOContainer *container, entries =3D create.window_size >> create.page_shift; pages =3D MAX((entries * sizeof(uint64_t)) / getpagesize(), 1); pages =3D MAX(pow2ceil(pages), 1); /* Round up */ - create.levels =3D ctz64(pages) / 6 + 1; + create.levels =3D MAX(1, (55 - create.page_shift) / 16); =20 ret =3D ioctl(container->fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create); if (ret) { @@ -200,6 +200,7 @@ int vfio_spapr_create_window(VFIOContainer *container, return -EINVAL; } trace_vfio_spapr_create_window(create.page_shift, + create.levels, create.window_size, create.start_addr); *pgsize =3D pagesize; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index a85e866..db730f3 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -128,6 +128,6 @@ vfio_prereg_listener_region_add_skip(uint64_t start, ui= nt64_t end) "0x%"PRIx64" vfio_prereg_listener_region_del_skip(uint64_t start, uint64_t end) "0x%"PR= Ix64" - 0x%"PRIx64 vfio_prereg_register(uint64_t va, uint64_t size, int ret) "va=3D0x%"PRIx64= " size=3D0x%"PRIx64" ret=3D%d" vfio_prereg_unregister(uint64_t va, uint64_t size, int ret) "va=3D0x%"PRIx= 64" size=3D0x%"PRIx64" ret=3D%d" -vfio_spapr_create_window(int ps, uint64_t ws, uint64_t off) "pageshift=3D0= x%x winsize=3D0x%"PRIx64" offset=3D0x%"PRIx64 +vfio_spapr_create_window(int ps, unsigned int levels, uint64_t ws, uint64_= t off) "pageshift=3D0x%x levels=3D%u winsize=3D0x%"PRIx64" offset=3D0x%"PRI= x64 vfio_spapr_remove_window(uint64_t off) "offset=3D0x%"PRIx64 vfio_spapr_group_attach(int groupfd, int tablefd) "Attached groupfd %d to = liobn fd %d" --=20 2.17.1 From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542099101128145.18933723182943; Tue, 13 Nov 2018 00:51:41 -0800 (PST) Received: from localhost ([::1]:52750 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUQ1-0001Tb-SB for importer@patchew.org; Tue, 13 Nov 2018 03:51:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGy-0002rc-5f for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU6P-00058x-8c for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:20 -0500 Received: from [107.173.13.209] (port=60845 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU6O-00056c-PU; Tue, 13 Nov 2018 03:31:17 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id B09B0AE80496; Tue, 13 Nov 2018 03:31:12 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:30:59 +1100 Message-Id: <20181113083104.2692-3-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 2/7] linux-header: Update for new capabilities X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This pulls new NVIDIA V100 and NPU VFIO region types. The kernel changes are not in upstream yet so this does not have a tag. Signed-off-by: Alexey Kardashevskiy --- include/standard-headers/drm/drm_fourcc.h | 48 ++++++++++++ include/standard-headers/linux/ethtool.h | 15 +--- .../linux/input-event-codes.h | 18 +++++ include/standard-headers/linux/pci_regs.h | 1 + linux-headers/asm-arm/unistd-common.h | 1 + linux-headers/asm-arm64/unistd.h | 1 + linux-headers/asm-generic/unistd.h | 2 + linux-headers/linux/kvm.h | 10 +++ linux-headers/linux/vfio.h | 76 +++++++++++++++++++ 9 files changed, 160 insertions(+), 12 deletions(-) diff --git a/include/standard-headers/drm/drm_fourcc.h b/include/standard-h= eaders/drm/drm_fourcc.h index b53f8d7..bd0248a 100644 --- a/include/standard-headers/drm/drm_fourcc.h +++ b/include/standard-headers/drm/drm_fourcc.h @@ -29,11 +29,50 @@ extern "C" { #endif =20 +/** + * DOC: overview + * + * In the DRM subsystem, framebuffer pixel formats are described using the + * fourcc codes defined in `include/uapi/drm/drm_fourcc.h`. In addition to= the + * fourcc code, a Format Modifier may optionally be provided, in order to + * further describe the buffer's format - for example tiling or compressio= n. + * + * Format Modifiers + * ---------------- + * + * Format modifiers are used in conjunction with a fourcc code, forming a + * unique fourcc:modifier pair. This format:modifier pair must fully defin= e the + * format and data layout of the buffer, and should be the only way to des= cribe + * that particular buffer. + * + * Having multiple fourcc:modifier pairs which describe the same layout sh= ould + * be avoided, as such aliases run the risk of different drivers exposing + * different names for the same data format, forcing userspace to understa= nd + * that they are aliases. + * + * Format modifiers may change any property of the buffer, including the n= umber + * of planes and/or the required allocation size. Format modifiers are + * vendor-namespaced, and as such the relationship between a fourcc code a= nd a + * modifier is specific to the modifer being used. For example, some modif= iers + * may preserve meaning - such as number of planes - from the fourcc code, + * whereas others may not. + * + * Vendors should document their modifier usage in as much detail as + * possible, to ensure maximum compatibility across devices, drivers and + * applications. + * + * The authoritative list of format modifier codes is found in + * `include/uapi/drm/drm_fourcc.h` + */ + #define fourcc_code(a, b, c, d) ((uint32_t)(a) | ((uint32_t)(b) << 8) | \ ((uint32_t)(c) << 16) | ((uint32_t)(d) << 24)) =20 #define DRM_FORMAT_BIG_ENDIAN (1<<31) /* format is big endian instead of l= ittle endian */ =20 +/* Reserve 0 for the invalid format specifier */ +#define DRM_FORMAT_INVALID 0 + /* color index */ #define DRM_FORMAT_C8 fourcc_code('C', '8', ' ', ' ') /* [7:0] C */ =20 @@ -298,6 +337,15 @@ extern "C" { */ #define DRM_FORMAT_MOD_SAMSUNG_64_32_TILE fourcc_mod_code(SAMSUNG, 1) =20 +/* + * Tiled, 16 (pixels) x 16 (lines) - sized macroblocks + * + * This is a simple tiled layout using tiles of 16x16 pixels in a row-major + * layout. For YCbCr formats Cb/Cr components are taken in such a way that + * they correspond to their 16x16 luma block. + */ +#define DRM_FORMAT_MOD_SAMSUNG_16_16_TILE fourcc_mod_code(SAMSUNG, 2) + /* * Qualcomm Compressed Format * diff --git a/include/standard-headers/linux/ethtool.h b/include/standard-he= aders/linux/ethtool.h index 57ffcb5..6ec5aeb 100644 --- a/include/standard-headers/linux/ethtool.h +++ b/include/standard-headers/linux/ethtool.h @@ -91,10 +91,6 @@ * %ETHTOOL_GSET to get the current values before making specific * changes and then applying them with %ETHTOOL_SSET. * - * Drivers that implement set_settings() should validate all fields - * other than @cmd that are not described as read-only or deprecated, - * and must ignore all fields described as read-only. - * * Deprecated fields should be ignored by both users and drivers. */ struct ethtool_cmd { @@ -1800,14 +1796,9 @@ enum ethtool_reset_flags { * rejected. * * Deprecated %ethtool_cmd fields transceiver, maxtxpkt and maxrxpkt - * are not available in %ethtool_link_settings. Until all drivers are - * converted to ignore them or to the new %ethtool_link_settings API, - * for both queries and changes, users should always try - * %ETHTOOL_GLINKSETTINGS first, and if it fails with -ENOTSUPP stick - * only to %ETHTOOL_GSET and %ETHTOOL_SSET consistently. If it - * succeeds, then users should stick to %ETHTOOL_GLINKSETTINGS and - * %ETHTOOL_SLINKSETTINGS (which would support drivers implementing - * either %ethtool_cmd or %ethtool_link_settings). + * are not available in %ethtool_link_settings. These fields will be + * always set to zero in %ETHTOOL_GSET reply and %ETHTOOL_SSET will + * fail if any of them is set to non-zero value. * * Users should assume that all fields not marked read-only are * writable and subject to validation by the driver. They should use diff --git a/include/standard-headers/linux/input-event-codes.h b/include/s= tandard-headers/linux/input-event-codes.h index 9e6a8ba..792b8df 100644 --- a/include/standard-headers/linux/input-event-codes.h +++ b/include/standard-headers/linux/input-event-codes.h @@ -708,6 +708,15 @@ #define REL_DIAL 0x07 #define REL_WHEEL 0x08 #define REL_MISC 0x09 +/* + * 0x0a is reserved and should not be used in input drivers. + * It was used by HID as REL_MISC+1 and userspace needs to detect if + * the next REL_* event is correct or is just REL_MISC + n. + * We define here REL_RESERVED so userspace can rely on it and detect + * the situation described above. + */ +#define REL_RESERVED 0x0a +#define REL_WHEEL_HI_RES 0x0b #define REL_MAX 0x0f #define REL_CNT (REL_MAX+1) =20 @@ -744,6 +753,15 @@ =20 #define ABS_MISC 0x28 =20 +/* + * 0x2e is reserved and should not be used in input drivers. + * It was used by HID as ABS_MISC+6 and userspace needs to detect if + * the next ABS_* event is correct or is just ABS_MISC + n. + * We define here ABS_RESERVED so userspace can rely on it and detect + * the situation described above. + */ +#define ABS_RESERVED 0x2e + #define ABS_MT_SLOT 0x2f /* MT slot being modified */ #define ABS_MT_TOUCH_MAJOR 0x30 /* Major axis of touching ellipse */ #define ABS_MT_TOUCH_MINOR 0x31 /* Minor axis (omit if circular) */ diff --git a/include/standard-headers/linux/pci_regs.h b/include/standard-h= eaders/linux/pci_regs.h index ee556cc..e1e9888 100644 --- a/include/standard-headers/linux/pci_regs.h +++ b/include/standard-headers/linux/pci_regs.h @@ -52,6 +52,7 @@ #define PCI_COMMAND_INTX_DISABLE 0x400 /* INTx Emulation Disable */ =20 #define PCI_STATUS 0x06 /* 16 bits */ +#define PCI_STATUS_IMM_READY 0x01 /* Immediate Readiness */ #define PCI_STATUS_INTERRUPT 0x08 /* Interrupt status */ #define PCI_STATUS_CAP_LIST 0x10 /* Support Capability List */ #define PCI_STATUS_66MHZ 0x20 /* Support 66 MHz PCI 2.1 bus */ diff --git a/linux-headers/asm-arm/unistd-common.h b/linux-headers/asm-arm/= unistd-common.h index 60c2d93..8c84bcf 100644 --- a/linux-headers/asm-arm/unistd-common.h +++ b/linux-headers/asm-arm/unistd-common.h @@ -355,5 +355,6 @@ #define __NR_pkey_free (__NR_SYSCALL_BASE + 396) #define __NR_statx (__NR_SYSCALL_BASE + 397) #define __NR_rseq (__NR_SYSCALL_BASE + 398) +#define __NR_io_pgetevents (__NR_SYSCALL_BASE + 399) =20 #endif /* _ASM_ARM_UNISTD_COMMON_H */ diff --git a/linux-headers/asm-arm64/unistd.h b/linux-headers/asm-arm64/uni= std.h index 5072cbd..dae1584 100644 --- a/linux-headers/asm-arm64/unistd.h +++ b/linux-headers/asm-arm64/unistd.h @@ -16,5 +16,6 @@ */ =20 #define __ARCH_WANT_RENAMEAT +#define __ARCH_WANT_NEW_STAT =20 #include diff --git a/linux-headers/asm-generic/unistd.h b/linux-headers/asm-generic= /unistd.h index df4bedb..538546e 100644 --- a/linux-headers/asm-generic/unistd.h +++ b/linux-headers/asm-generic/unistd.h @@ -242,10 +242,12 @@ __SYSCALL(__NR_tee, sys_tee) /* fs/stat.c */ #define __NR_readlinkat 78 __SYSCALL(__NR_readlinkat, sys_readlinkat) +#if defined(__ARCH_WANT_NEW_STAT) || defined(__ARCH_WANT_STAT64) #define __NR3264_fstatat 79 __SC_3264(__NR3264_fstatat, sys_fstatat64, sys_newfstatat) #define __NR3264_fstat 80 __SC_3264(__NR3264_fstat, sys_fstat64, sys_newfstat) +#endif =20 /* fs/sync.c */ #define __NR_sync 81 diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index f11a7eb..95481af 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -757,6 +757,15 @@ struct kvm_ppc_resize_hpt { =20 #define KVM_S390_SIE_PAGE_OFFSET 1 =20 +/* + * On arm64, machine type can be used to request the physical + * address size for the VM. Bits[7-0] are reserved for the guest + * PA size shift (i.e, log2(PA_Size)). For backward compatibility, + * value 0 implies the default IPA size, 40bits. + */ +#define KVM_VM_TYPE_ARM_IPA_SIZE_MASK 0xffULL +#define KVM_VM_TYPE_ARM_IPA_SIZE(x) \ + ((x) & KVM_VM_TYPE_ARM_IPA_SIZE_MASK) /* * ioctls for /dev/kvm fds: */ @@ -965,6 +974,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_COALESCED_PIO 162 #define KVM_CAP_HYPERV_ENLIGHTENED_VMCS 163 #define KVM_CAP_EXCEPTION_PAYLOAD 164 +#define KVM_CAP_ARM_VM_IPA_SIZE 165 =20 #ifdef KVM_CAP_IRQ_ROUTING =20 diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index ceb6453..fea9390 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -303,6 +303,70 @@ struct vfio_region_info_cap_type { #define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2) #define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) =20 +#define VFIO_REGION_TYPE_GFX (1) +#define VFIO_REGION_SUBTYPE_GFX_EDID (1) + +/** + * struct vfio_region_gfx_edid - EDID region layout. + * + * Set display link state and EDID blob. + * + * The EDID blob has monitor information such as brand, name, serial + * number, physical size, supported video modes and more. + * + * This special region allows userspace (typically qemu) set a virtual + * EDID for the virtual monitor, which allows a flexible display + * configuration. + * + * For the edid blob spec look here: + * https://en.wikipedia.org/wiki/Extended_Display_Identification_Data + * + * On linux systems you can find the EDID blob in sysfs: + * /sys/class/drm/${card}/${connector}/edid + * + * You can use the edid-decode ulility (comes with xorg-x11-utils) to + * decode the EDID blob. + * + * @edid_offset: location of the edid blob, relative to the + * start of the region (readonly). + * @edid_max_size: max size of the edid blob (readonly). + * @edid_size: actual edid size (read/write). + * @link_state: display link state (read/write). + * VFIO_DEVICE_GFX_LINK_STATE_UP: Monitor is turned on. + * VFIO_DEVICE_GFX_LINK_STATE_DOWN: Monitor is turned off. + * @max_xres: max display width (0 =3D=3D no limitation, readonly). + * @max_yres: max display height (0 =3D=3D no limitation, readonly). + * + * EDID update protocol: + * (1) set link-state to down. + * (2) update edid blob and size. + * (3) set link-state to up. + */ +struct vfio_region_gfx_edid { + __u32 edid_offset; + __u32 edid_max_size; + __u32 edid_size; + __u32 max_xres; + __u32 max_yres; + __u32 link_state; +#define VFIO_DEVICE_GFX_LINK_STATE_UP 1 +#define VFIO_DEVICE_GFX_LINK_STATE_DOWN 2 +}; + +/* 10de vendor sub-type + * + * NVIDIA GPU NVlink2 RAM is coherent RAM mapped onto the host address spa= ce. + */ +#define VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM (1) + +/* + * 1014 vendor sub-type + * + * IBM NPU NVlink2 ATSD (Address Translation Shootdown) register of NPU + * to do TLB invalidation on a GPU. + */ +#define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1) + /* * The MSIX mappable capability informs that MSIX data of a BAR can be mma= pped * which allows direct access to non-MSIX registers which happened to be w= ithin @@ -313,6 +377,18 @@ struct vfio_region_info_cap_type { */ #define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 =20 +/* + * Capability with compressed real address (aka SSA - small system address) + * where GPU RAM is mapped on a system bus. Used by a GPU for DMA routing. + */ +#define VFIO_REGION_INFO_CAP_NPU2 4 + +struct vfio_region_info_cap_npu2 { + struct vfio_info_cap_header header; + __u64 tgt; + /* size is defined in VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM */ +}; + /** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) --=20 2.17.1 From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542098640797293.55883683116326; Tue, 13 Nov 2018 00:44:00 -0800 (PST) Received: from localhost ([::1]:52704 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUIh-0004G2-4M for importer@patchew.org; Tue, 13 Nov 2018 03:43:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45024) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGw-0002pd-2D for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU6w-0006X7-Kc for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:51 -0500 Received: from [107.173.13.209] (port=32768 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU6w-0006Rt-Bc; Tue, 13 Nov 2018 03:31:50 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 9EF2FAE80498; Tue, 13 Nov 2018 03:31:15 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:31:00 +1100 Message-Id: <20181113083104.2692-4-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 3/7] pci: Move NVIDIA vendor id to the rest of ids X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" sPAPR code will use it too so move it from VFIO to the common code. Signed-off-by: Alexey Kardashevskiy Reviewed-by: Alistair Francis Reviewed-by: David Gibson --- include/hw/pci/pci_ids.h | 2 ++ hw/vfio/pci-quirks.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h index 63acc72..3ed7d10 100644 --- a/include/hw/pci/pci_ids.h +++ b/include/hw/pci/pci_ids.h @@ -271,4 +271,6 @@ =20 #define PCI_VENDOR_ID_SYNOPSYS 0x16C3 =20 +#define PCI_VENDOR_ID_NVIDIA 0x10de + #endif diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index eae31c7..40a1200 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -526,8 +526,6 @@ static void vfio_probe_ati_bar2_quirk(VFIOPCIDevice *vd= ev, int nr) * note it for future reference. */ =20 -#define PCI_VENDOR_ID_NVIDIA 0x10de - /* * Nvidia has several different methods to get to config space, the * nouveu project has several of these documented here: --=20 2.17.1 From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542098855003462.81988654982183; Tue, 13 Nov 2018 00:47:35 -0800 (PST) Received: from localhost ([::1]:52724 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUM9-000705-Og for importer@patchew.org; Tue, 13 Nov 2018 03:47:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45651) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGt-00036a-BE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU6y-0006b5-2L for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:53 -0500 Received: from [107.173.13.209] (port=32778 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU6x-0006Yt-JY; Tue, 13 Nov 2018 03:31:51 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 63B5AAE80548; Tue, 13 Nov 2018 03:31:18 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:31:01 +1100 Message-Id: <20181113083104.2692-5-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 4/7] vfio/nvidia-v100: Disable VBIOS update X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The NVIDIA V100 GPUs often come in several instances on the same system board where they are connected directly via out of band fabric called "NVLink". In order to make GPUs talk to each other, NVLink has to be enabled on both GPUs and this is guaranteed by the firmware by providing special MMIO registers to disable NVLink till GPU is reset. This blocks GPU VBIOS update to add an extra level of assurance that the firmware does not get reflashed with a malicious firmware which does not implement NVLink disabling mechanism. Signed-off-by: Alexey Kardashevskiy --- NVIDIA firmwares come signed and GPUs do not accept unsigned images anyway so this is probably overkill, or not? Also, there is no available documentation on the magic value of 0x22408; however it does help as the nvflash upgrade tool stops working with this applied. --- hw/vfio/pci.h | 1 + include/hw/pci/pci_ids.h | 1 + hw/vfio/pci-quirks.c | 26 ++++++++++++++++++++++++++ hw/vfio/pci.c | 2 ++ hw/vfio/trace-events | 1 + 5 files changed, 31 insertions(+) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index b1ae4c0..f4c5fb6 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -163,6 +163,7 @@ typedef struct VFIOPCIDevice { bool no_kvm_msi; bool no_kvm_msix; bool no_geforce_quirks; + bool no_nvidia_v100_quirks; bool no_kvm_ioeventfd; bool no_vfio_ioeventfd; bool enable_ramfb; diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h index 3ed7d10..2140dad 100644 --- a/include/hw/pci/pci_ids.h +++ b/include/hw/pci/pci_ids.h @@ -272,5 +272,6 @@ #define PCI_VENDOR_ID_SYNOPSYS 0x16C3 =20 #define PCI_VENDOR_ID_NVIDIA 0x10de +#define PCI_VENDOR_ID_NVIDIA_V100_SXM2 0x1db1 =20 #endif diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index 40a1200..2796837 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -996,6 +996,31 @@ static void vfio_probe_nvidia_bar0_quirk(VFIOPCIDevice= *vdev, int nr) trace_vfio_quirk_nvidia_bar0_probe(vdev->vbasedev.name); } =20 +static void vfio_probe_nvidia_v100_bar0_quirk(VFIOPCIDevice *vdev, int nr) +{ + VFIOQuirk *quirk; + + if (vdev->no_nvidia_v100_quirks || + !vfio_pci_is(vdev, PCI_VENDOR_ID_NVIDIA, + PCI_VENDOR_ID_NVIDIA_V100_SXM2) || + nr !=3D 0) { + return; + } + + quirk =3D vfio_quirk_alloc(1); + + memory_region_init_io(quirk->mem, OBJECT(vdev), + NULL, quirk, + "vfio-nvidia-v100_bar0-block-quirk", + 4); + memory_region_add_subregion_overlap(vdev->bars[nr].region.mem, + 0x22408, quirk->mem, 1); + + QLIST_INSERT_HEAD(&vdev->bars[nr].quirks, quirk, next); + + trace_vfio_quirk_nvidia_v100_bar0_probe(vdev->vbasedev.name); +} + /* * TODO - Some Nvidia devices provide config access to their companion HDA * device and even to their parent bridge via these config space mirrors. @@ -1853,6 +1878,7 @@ void vfio_bar_quirk_setup(VFIOPCIDevice *vdev, int nr) vfio_probe_ati_bar2_quirk(vdev, nr); vfio_probe_nvidia_bar5_quirk(vdev, nr); vfio_probe_nvidia_bar0_quirk(vdev, nr); + vfio_probe_nvidia_v100_bar0_quirk(vdev, nr); vfio_probe_rtl8168_bar2_quirk(vdev, nr); vfio_probe_igd_bar4_quirk(vdev, nr); } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5c7bd96..7848b28 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3203,6 +3203,8 @@ static Property vfio_pci_dev_properties[] =3D { DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false), DEFINE_PROP_BOOL("x-no-geforce-quirks", VFIOPCIDevice, no_geforce_quirks, false), + DEFINE_PROP_BOOL("x-no-nvidia-v100-quirks", VFIOPCIDevice, + no_nvidia_v100_quirks, false), DEFINE_PROP_BOOL("x-no-kvm-ioeventfd", VFIOPCIDevice, no_kvm_ioeventfd, false), DEFINE_PROP_BOOL("x-no-vfio-ioeventfd", VFIOPCIDevice, no_vfio_ioevent= fd, diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index db730f3..adfa75e 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -68,6 +68,7 @@ vfio_quirk_nvidia_bar5_state(const char *name, const char= *state) "%s %s" vfio_quirk_nvidia_bar5_probe(const char *name) "%s" vfio_quirk_nvidia_bar0_msi_ack(const char *name) "%s" vfio_quirk_nvidia_bar0_probe(const char *name) "%s" +vfio_quirk_nvidia_v100_bar0_probe(const char *name) "%s" vfio_quirk_rtl8168_fake_latch(const char *name, uint64_t val) "%s 0x%"PRIx= 64 vfio_quirk_rtl8168_msix_write(const char *name, uint16_t offset, uint64_t = val) "%s MSI-X table write[0x%x]: 0x%"PRIx64 vfio_quirk_rtl8168_msix_read(const char *name, uint16_t offset, uint64_t v= al) "%s MSI-X table read[0x%x]: 0x%"PRIx64 --=20 2.17.1 From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542098644758782.7995121987441; Tue, 13 Nov 2018 00:44:04 -0800 (PST) Received: from localhost ([::1]:52703 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUIb-0004DW-U4 for importer@patchew.org; Tue, 13 Nov 2018 03:43:53 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45106) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGr-0002s3-Q9 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU70-0006iR-IJ for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:55 -0500 Received: from [107.173.13.209] (port=32783 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU70-0006go-8V; Tue, 13 Nov 2018 03:31:54 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 531A3AE805A6; Tue, 13 Nov 2018 03:31:21 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:31:02 +1100 Message-Id: <20181113083104.2692-6-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 5/7] spapr-iommu: Always advertise the maximum possible DMA window size X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When deciding about the huge DMA window, the typical Linux pseries guest uses the maximum allowed RAM size as the upper limit. We did the same on QEMU side to match that logic. Now we are going to support GPU RAM pass through which is not available at the guest boot time as it requires the guest driver interaction. As the result, the guest requests a smaller window than it should. Therefore the guest needs to be patched to understand this new memory and so does QEMU. Instead of reimplementing here whatever solution we will choose for the guest, this advertises the biggest possible window size limited by 32 bit (as defined by LoPAPR). This seems to be safe as: 1. The guest visible emulated table is allocated in KVM (actual pages are allocated in page fault handler) and QEMU (actual pages are allocated when changed); 2. The hardware table (and corresponding userspace addresses cache) supports sparse allocation and also checks for locked_vm limit so it is unable to cause the host any damage. Signed-off-by: Alexey Kardashevskiy --- hw/ppc/spapr_rtas_ddw.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-) diff --git a/hw/ppc/spapr_rtas_ddw.c b/hw/ppc/spapr_rtas_ddw.c index 329feb1..df60351 100644 --- a/hw/ppc/spapr_rtas_ddw.c +++ b/hw/ppc/spapr_rtas_ddw.c @@ -96,9 +96,8 @@ static void rtas_ibm_query_pe_dma_window(PowerPCCPU *cpu, uint32_t nret, target_ulong rets) { sPAPRPHBState *sphb; - uint64_t buid, max_window_size; + uint64_t buid; uint32_t avail, addr, pgmask =3D 0; - MachineState *machine =3D MACHINE(spapr); =20 if ((nargs !=3D 3) || (nret !=3D 5)) { goto param_error_exit; @@ -114,27 +113,15 @@ static void rtas_ibm_query_pe_dma_window(PowerPCCPU *= cpu, /* Translate page mask to LoPAPR format */ pgmask =3D spapr_page_mask_to_query_mask(sphb->page_size_mask); =20 - /* - * This is "Largest contiguous block of TCEs allocated specifically - * for (that is, are reserved for) this PE". - * Return the maximum number as maximum supported RAM size was in 4K p= ages. - */ - if (machine->ram_size =3D=3D machine->maxram_size) { - max_window_size =3D machine->ram_size; - } else { - max_window_size =3D machine->device_memory->base + - memory_region_size(&machine->device_memory->mr); - } - avail =3D SPAPR_PCI_DMA_MAX_WINDOWS - spapr_phb_get_active_win_num(sph= b); =20 rtas_st(rets, 0, RTAS_OUT_SUCCESS); rtas_st(rets, 1, avail); - rtas_st(rets, 2, max_window_size >> SPAPR_TCE_PAGE_SHIFT); + rtas_st(rets, 2, 0xFFFFFFFF); /* Largest contiguous block of TCEs */ rtas_st(rets, 3, pgmask); rtas_st(rets, 4, 0); /* DMA migration mask, not supported */ =20 - trace_spapr_iommu_ddw_query(buid, addr, avail, max_window_size, pgmask= ); + trace_spapr_iommu_ddw_query(buid, addr, avail, 0xFFFFFFFF, pgmask); return; =20 param_error_exit: --=20 2.17.1 From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542098785182673.7326009397852; Tue, 13 Nov 2018 00:46:25 -0800 (PST) Received: from localhost ([::1]:52720 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUL1-00067h-UK for importer@patchew.org; Tue, 13 Nov 2018 03:46:23 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45311) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGy-000332-0m for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:13 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU6a-0005gQ-3X for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:36 -0500 Received: from [107.173.13.209] (port=60915 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU6Z-0005c1-Gt; Tue, 13 Nov 2018 03:31:27 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 185DDAE807DB; Tue, 13 Nov 2018 03:31:23 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:31:03 +1100 Message-Id: <20181113083104.2692-7-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 6/7] vfio: Make vfio_get_region_info_cap public X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This makes vfio_get_region_info_cap() to be used in quirks. Signed-off-by: Alexey Kardashevskiy --- include/hw/vfio/vfio-common.h | 2 ++ hw/vfio/common.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1b434d0..abc82f9 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -189,6 +189,8 @@ int vfio_get_region_info(VFIODevice *vbasedev, int inde= x, int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, uint32_t subtype, struct vfio_region_info **i= nfo); bool vfio_has_region_cap(VFIODevice *vbasedev, int region, uint16_t cap_ty= pe); +struct vfio_info_cap_header * +vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id); #endif extern const MemoryListener vfio_prereg_listener; =20 diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 7c185e5a..f58b487 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -711,7 +711,7 @@ static void vfio_listener_release(VFIOContainer *contai= ner) } } =20 -static struct vfio_info_cap_header * +struct vfio_info_cap_header * vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id) { struct vfio_info_cap_header *hdr; --=20 2.17.1 From nobody Mon May 6 09:13:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542098937963936.4876186576114; Tue, 13 Nov 2018 00:48:57 -0800 (PST) Received: from localhost ([::1]:52731 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUNT-00084V-Vz for importer@patchew.org; Tue, 13 Nov 2018 03:48:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45341) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMUGx-00033B-VW for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:42:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMU6s-0006H3-51 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 03:31:49 -0500 Received: from [107.173.13.209] (port=60915 helo=ozlabs.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMU6r-0005c1-1o; Tue, 13 Nov 2018 03:31:46 -0500 Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id D371AAE807E5; Tue, 13 Nov 2018 03:31:26 -0500 (EST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 13 Nov 2018 19:31:04 +1100 Message-Id: <20181113083104.2692-8-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113083104.2692-1-aik@ozlabs.ru> References: <20181113083104.2692-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [PATCH qemu RFC 7/7] spapr: Add NVLink2 pass through support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jose Ricardo Ziviani , Alexey Kardashevskiy , Alistair Popple , Alex Williamson , Sam Bobroff , Piotr Jaroszynski , qemu-ppc@nongnu.org, =?UTF-8?q?Leonardo=20Augusto=20Guimar=C3=A3es=20Garcia?= , Oliver O'Halloran , Reza Arbab , David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The NVIDIA V100 GPU comes with some on-board RAM which is mapped into the host memory space and accessible as normal RAM via NVLink bus. The VFIO-PCI driver implements special regions for such GPU and emulated NVLink bridge (below referred as NPU). The POWER9 CPU also provides address translation services which includes TLB invalidation register exposes via the NVLink bridge; the feature is called "ATSD". This adds a quirk to VFIO to map the memory and create an MR; the new MR is stored in a GPU as a QOM link. The sPAPR PCI uses this to get the MR and map it to the system address space. Another quirk does the same for ATSD. This adds 3 additional steps to the FDT builder in spapr-pci: 1. Search for specific GPUs and NPUs, collects findings in sPAPRPHBState; 2. Adds several properties in the DT: "ibm,npu", "ibm,gpu", "memory-block", and some other. These are required by the guest platform and GPU driver; this also adds a new made-up compatible type for a PHB to signal a modified guest that this particular PHB needs the default DMA window removed as these GPUs have limited DMA mask size (way lower than usual 59); 3. Adds new memory blocks with one addition - they have "linux,memory-usable" property configured in the way which prevents the guest from onlining it automatically as it needs to be deferred till the guest GPU driver trains NVLink. A couple of notes: - this changes the FDT rendeder as doing 1-2-3 from sPAPRPHBClass::realize impossible - devices are not yet attached; - this does not add VFIO quirk MRs to the system address space as the address is selected in sPAPRPHBState, similar to MMIO. This puts new memory nodes in a separate NUMA node to replicate the host system setup as close as possible (the GPU driver relies on this too). This adds fake NPU nodes to make the guest platform code work, specifically "ibm,npu-link-index". Signed-off-by: Alexey Kardashevskiy --- hw/vfio/pci.h | 2 + include/hw/pci-host/spapr.h | 28 ++++ include/hw/ppc/spapr.h | 3 +- hw/ppc/spapr.c | 14 +- hw/ppc/spapr_pci.c | 256 +++++++++++++++++++++++++++++++++++- hw/vfio/pci-quirks.c | 93 +++++++++++++ hw/vfio/pci.c | 14 ++ hw/vfio/trace-events | 3 + 8 files changed, 408 insertions(+), 5 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index f4c5fb6..b8954cc 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -195,6 +195,8 @@ int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp= ); int vfio_pci_igd_opregion_init(VFIOPCIDevice *vdev, struct vfio_region_info *info, Error **errp); +int vfio_pci_nvlink2_ram_init(VFIOPCIDevice *vdev, Error **errp); +int vfio_pci_npu2_atsd_init(VFIOPCIDevice *vdev, Error **errp); =20 void vfio_display_reset(VFIOPCIDevice *vdev); int vfio_display_probe(VFIOPCIDevice *vdev, Error **errp); diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h index 7c66c38..1f8ebf3 100644 --- a/include/hw/pci-host/spapr.h +++ b/include/hw/pci-host/spapr.h @@ -87,6 +87,24 @@ struct sPAPRPHBState { uint32_t mig_liobn; hwaddr mig_mem_win_addr, mig_mem_win_size; hwaddr mig_io_win_addr, mig_io_win_size; + hwaddr nv2_gpa_win_addr; + hwaddr nv2_atsd_win_addr; + + struct spapr_phb_pci_nvgpu_config { + uint64_t nv2_ram; + uint64_t nv2_atsd; + int num; + struct { + int links; + uint64_t tgt; + uint64_t gpa; + PCIDevice *gpdev; + uint64_t atsd[3]; + PCIDevice *npdev[3]; + } gpus[6]; + uint64_t atsd[64]; /* Big Endian (BE), ready for the DT */ + int atsd_num; + } nvgpus; }; =20 #define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL @@ -104,6 +122,16 @@ struct sPAPRPHBState { =20 #define SPAPR_PCI_MSI_WINDOW 0x40000000000ULL =20 +#define PHANDLE_PCIDEV(phb, pdev) (0x12000000 | \ + (((phb)->index) << 16) | ((pdev)->dev= fn)) +#define PHANDLE_GPURAM(phb, n) (0x110000FF | ((n) << 8) | \ + (((phb)->index) << 16)) +#define GPURAM_ASSOCIATIVITY(phb, n) (255 - ((phb)->index * 3 + (n))) +#define SPAPR_PCI_NV2RAM64_WIN_BASE 0x10000000000ULL /* 1 TiB */ +#define SPAPR_PCI_NV2RAM64_WIN_SIZE 0x02000000000ULL +#define PHANDLE_NVLINK(phb, gn, nn) (0x00130000 | (((phb)->index) << 8) |= \ + ((gn) << 4) | (nn)) + static inline qemu_irq spapr_phb_lsi_qirq(struct sPAPRPHBState *phb, int p= in) { sPAPRMachineState *spapr =3D SPAPR_MACHINE(qdev_get_machine()); diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index f5dcaf4..0ceca47 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -108,7 +108,8 @@ struct sPAPRMachineClass { void (*phb_placement)(sPAPRMachineState *spapr, uint32_t index, uint64_t *buid, hwaddr *pio,=20 hwaddr *mmio32, hwaddr *mmio64, - unsigned n_dma, uint32_t *liobns, Error **errp); + unsigned n_dma, uint32_t *liobns, hwaddr *nv2gpa, + hwaddr *nv2atsd, Error **errp); sPAPRResizeHPT resize_hpt_default; sPAPRCapabilities default_caps; sPAPRIrq *irq; diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 38a8218..760b0b5 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -3723,7 +3723,8 @@ static const CPUArchIdList *spapr_possible_cpu_arch_i= ds(MachineState *machine) static void spapr_phb_placement(sPAPRMachineState *spapr, uint32_t index, uint64_t *buid, hwaddr *pio, hwaddr *mmio32, hwaddr *mmio64, - unsigned n_dma, uint32_t *liobns, Error **= errp) + unsigned n_dma, uint32_t *liobns, + hwaddr *nv2gpa, hwaddr *nv2atsd, Error **e= rrp) { /* * New-style PHB window placement. @@ -3770,6 +3771,11 @@ static void spapr_phb_placement(sPAPRMachineState *s= papr, uint32_t index, *pio =3D SPAPR_PCI_BASE + index * SPAPR_PCI_IO_WIN_SIZE; *mmio32 =3D SPAPR_PCI_BASE + (index + 1) * SPAPR_PCI_MEM32_WIN_SIZE; *mmio64 =3D SPAPR_PCI_BASE + (index + 1) * SPAPR_PCI_MEM64_WIN_SIZE; + + *nv2gpa =3D SPAPR_PCI_NV2RAM64_WIN_BASE + + (index + 1) * SPAPR_PCI_NV2RAM64_WIN_SIZE; + + *nv2atsd =3D SPAPR_PCI_BASE + (index + 8192) * 0x10000; } =20 static ICSState *spapr_ics_get(XICSFabric *dev, int irq) @@ -4182,7 +4188,8 @@ DEFINE_SPAPR_MACHINE(2_8, "2.8", false); static void phb_placement_2_7(sPAPRMachineState *spapr, uint32_t index, uint64_t *buid, hwaddr *pio, hwaddr *mmio32, hwaddr *mmio64, - unsigned n_dma, uint32_t *liobns, Error **er= rp) + unsigned n_dma, uint32_t *liobns, + hwaddr *nv2_gpa, hwaddr *nv2atsd, Error **er= rp) { /* Legacy PHB placement for pseries-2.7 and earlier machine types */ const uint64_t base_buid =3D 0x800000020000000ULL; @@ -4226,6 +4233,9 @@ static void phb_placement_2_7(sPAPRMachineState *spap= r, uint32_t index, * fallback behaviour of automatically splitting a large "32-bit" * window into contiguous 32-bit and 64-bit windows */ + + *nv2_gpa =3D 0; + *nv2atsd =3D 0; } =20 static void spapr_machine_2_7_instance_options(MachineState *machine) diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index 58afa46..417ea1d 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -1249,6 +1249,7 @@ static uint32_t spapr_phb_get_pci_drc_index(sPAPRPHBS= tate *phb, static void spapr_populate_pci_child_dt(PCIDevice *dev, void *fdt, int off= set, sPAPRPHBState *sphb) { + int i, j; ResourceProps rp; bool is_bridge =3D false; int pci_status; @@ -1349,6 +1350,56 @@ static void spapr_populate_pci_child_dt(PCIDevice *d= ev, void *fdt, int offset, if (sphb->pcie_ecs && pci_is_express(dev)) { _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type", 0x= 1)); } + + for (i =3D 0; i < sphb->nvgpus.num; ++i) { + PCIDevice *gpdev =3D sphb->nvgpus.gpus[i].gpdev; + + if (dev =3D=3D gpdev) { + uint32_t npus[sphb->nvgpus.gpus[i].links]; + + for (j =3D 0; j < sphb->nvgpus.gpus[i].links; ++j) { + PCIDevice *npdev =3D sphb->nvgpus.gpus[i].npdev[j]; + + npus[j] =3D cpu_to_be32(PHANDLE_PCIDEV(sphb, npdev)); + } + _FDT(fdt_setprop(fdt, offset, "ibm,npu", npus, + j * sizeof(npus[0]))); + _FDT((fdt_setprop_cell(fdt, offset, "phandle", + PHANDLE_PCIDEV(sphb, dev)))); + } else { + for (j =3D 0; j < sphb->nvgpus.gpus[i].links; ++j) { + if (dev !=3D sphb->nvgpus.gpus[i].npdev[j]) { + continue; + } + + _FDT((fdt_setprop_cell(fdt, offset, "phandle", + PHANDLE_PCIDEV(sphb, dev)))); + + _FDT(fdt_setprop_cell(fdt, offset, "ibm,gpu", + PHANDLE_PCIDEV(sphb, gpdev))); + + _FDT((fdt_setprop_cell(fdt, offset, "ibm,nvlink", + PHANDLE_NVLINK(sphb, i, j)))); + + /* + * If we ever want to emulate GPU RAM at the same location= as on + * the host - here is the encoding GPA->TGT: + * + * gta =3D ((sphb->nv2_gpa >> 42) & 0x1) << 42; + * gta |=3D ((sphb->nv2_gpa >> 45) & 0x3) << 43; + * gta |=3D ((sphb->nv2_gpa >> 49) & 0x3) << 45; + * gta |=3D sphb->nv2_gpa & ((1UL << 43) - 1); + */ + _FDT(fdt_setprop_cell(fdt, offset, "memory-region", + PHANDLE_GPURAM(sphb, i))); + _FDT(fdt_setprop_u64(fdt, offset, "ibm,device-tgt-addr", + sphb->nvgpus.gpus[i].tgt)); + /* _FDT(fdt_setprop_cell(fdt, offset, "ibm,nvlink", 0x164)= ); */ + /* Unknown magic value of 9 */ + _FDT(fdt_setprop_cell(fdt, offset, "ibm,nvlink-speed", 9)); + } + } + } } =20 /* create OF node for pci device and required OF DT properties */ @@ -1582,7 +1633,9 @@ static void spapr_phb_realize(DeviceState *dev, Error= **errp) smc->phb_placement(spapr, sphb->index, &sphb->buid, &sphb->io_win_addr, &sphb->mem_win_addr, &sphb->mem64_win_addr, - windows_supported, sphb->dma_liobn, &local_err); + windows_supported, sphb->dma_liobn, + &sphb->nv2_gpa_win_addr, + &sphb->nv2_atsd_win_addr, &local_err); if (local_err) { error_propagate(errp, local_err); return; @@ -1829,6 +1882,8 @@ static Property spapr_phb_properties[] =3D { pre_2_8_migration, false), DEFINE_PROP_BOOL("pcie-extended-configuration-space", sPAPRPHBState, pcie_ecs, true), + DEFINE_PROP_UINT64("gpa", sPAPRPHBState, nv2_gpa_win_addr, 0), + DEFINE_PROP_UINT64("atsd", sPAPRPHBState, nv2_atsd_win_addr, 0), DEFINE_PROP_END_OF_LIST(), }; =20 @@ -2068,6 +2123,73 @@ static void spapr_phb_pci_enumerate(sPAPRPHBState *p= hb) =20 } =20 +static void spapr_phb_pci_find_nvgpu(PCIBus *bus, PCIDevice *pdev, void *o= paque) +{ + struct spapr_phb_pci_nvgpu_config *nvgpus =3D opaque; + PCIBus *sec_bus; + Object *mr_gpu, *mr_npu; + uint64_t tgt =3D 0, gpa, atsd; + int i; + + mr_gpu =3D object_property_get_link(OBJECT(pdev), "nvlink2-mr[0]", NUL= L); + mr_npu =3D object_property_get_link(OBJECT(pdev), "nvlink2-atsd-mr[0]"= , NULL); + if (mr_gpu) { + tgt =3D object_property_get_uint(mr_gpu, "tgt", NULL); + gpa =3D nvgpus->nv2_ram; + nvgpus->nv2_ram +=3D memory_region_size(MEMORY_REGION(mr_gpu)); + } else if (mr_npu) { + tgt =3D object_property_get_uint(mr_npu, "tgt", NULL); + atsd =3D nvgpus->nv2_atsd; + nvgpus->atsd[nvgpus->atsd_num] =3D cpu_to_be64(atsd); + ++nvgpus->atsd_num; + nvgpus->nv2_atsd +=3D memory_region_size(MEMORY_REGION(mr_npu)); + } + + if (tgt) { + for (i =3D 0; i < nvgpus->num; ++i) { + if (nvgpus->gpus[i].tgt =3D=3D tgt) { + break; + } + } + + if (i =3D=3D nvgpus->num) { + if (nvgpus->num =3D=3D ARRAY_SIZE(nvgpus->gpus)) { + return; + } + ++nvgpus->num; + } + + nvgpus->gpus[i].tgt =3D tgt; + if (mr_gpu) { + g_assert(!nvgpus->gpus[i].gpdev); + nvgpus->gpus[i].gpdev =3D pdev; + nvgpus->gpus[i].gpa =3D gpa; + } else { + int j =3D nvgpus->gpus[i].links; + + ++nvgpus->gpus[i].links; + + g_assert(mr_npu); + g_assert(!nvgpus->gpus[i].npdev[j]); + nvgpus->gpus[i].npdev[j] =3D pdev; + nvgpus->gpus[i].atsd[j] =3D atsd; + } + } + + if ((pci_default_read_config(pdev, PCI_HEADER_TYPE, 1) !=3D + PCI_HEADER_TYPE_BRIDGE)) { + return; + } + + sec_bus =3D pci_bridge_get_sec_bus(PCI_BRIDGE(pdev)); + if (!sec_bus) { + return; + } + + pci_for_each_device(sec_bus, pci_bus_num(sec_bus), + spapr_phb_pci_find_nvgpu, opaque); +} + int spapr_populate_pci_dt(sPAPRPHBState *phb, uint32_t xics_phandle, void = *fdt, uint32_t nr_msis) { @@ -2127,7 +2249,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, uint32_= t xics_phandle, void *fdt, =20 /* Write PHB properties */ _FDT(fdt_setprop_string(fdt, bus_off, "device_type", "pci")); - _FDT(fdt_setprop_string(fdt, bus_off, "compatible", "IBM,Logical_PHB")= ); + _FDT(fdt_setprop_cell(fdt, bus_off, "#address-cells", 0x3)); _FDT(fdt_setprop_cell(fdt, bus_off, "#size-cells", 0x2)); _FDT(fdt_setprop_cell(fdt, bus_off, "#interrupt-cells", 0x1)); @@ -2186,6 +2308,45 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, uint32= _t xics_phandle, void *fdt, spapr_phb_pci_enumerate(phb); _FDT(fdt_setprop_cell(fdt, bus_off, "qemu,phb-enumerated", 0x1)); =20 + for (i =3D 0; i < phb->nvgpus.num; ++i) { + PCIDevice *gpdev =3D phb->nvgpus.gpus[i].gpdev; + Object *nvlink2_mrobj =3D object_property_get_link(OBJECT(gpdev), + "nvlink2-mr[0]", = NULL); + MemoryRegion *mr =3D MEMORY_REGION(nvlink2_mrobj); + + memory_region_del_subregion(get_system_memory(), mr); + + for (j =3D 0; j < phb->nvgpus.gpus[i].links; ++j) { + PCIDevice *npdev =3D phb->nvgpus.gpus[i].npdev[j]; + Object *nvlink2_mrobj; + nvlink2_mrobj =3D object_property_get_link(OBJECT(npdev), + "nvlink2-atsd-mr[0]", + NULL); + if (nvlink2_mrobj) { + MemoryRegion *mr =3D MEMORY_REGION(nvlink2_mrobj); + memory_region_del_subregion(get_system_memory(), mr); + } + } + } + + memset(&phb->nvgpus, 0, sizeof(phb->nvgpus)); + phb->nvgpus.nv2_ram =3D phb->nv2_gpa_win_addr; + phb->nvgpus.nv2_atsd =3D phb->nv2_atsd_win_addr; + pci_for_each_device(bus, pci_bus_num(bus), + spapr_phb_pci_find_nvgpu, &phb->nvgpus); + if (phb->nvgpus.num) { + const char compat_npu[] =3D "IBM,Logical_PHB\x00IBM,npu-vphb"; + + /* 1 GPU and at least one NVLink2 */ + _FDT(fdt_setprop(fdt, bus_off, "compatible", compat_npu, + sizeof(compat_npu))); + _FDT((fdt_setprop(fdt, bus_off, "ibm,mmio-atsd", phb->nvgpus.atsd, + phb->nvgpus.atsd_num * + sizeof(phb->nvgpus.atsd[0])))); + } else { + _FDT(fdt_setprop_string(fdt, bus_off, "compatible", "IBM,Logical_P= HB")); + } + /* Populate tree nodes with PCI devices attached */ s_fdt.fdt =3D fdt; s_fdt.node_off =3D bus_off; @@ -2200,6 +2361,97 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, uint32= _t xics_phandle, void *fdt, return ret; } =20 + /* NVLink: Add memory nodes; map GPU RAM and ATSD */ + for (i =3D 0; i < phb->nvgpus.num; ++i) { + PCIDevice *gpdev =3D phb->nvgpus.gpus[i].gpdev; + Object *nvlink2_mrobj =3D object_property_get_link(OBJECT(gpdev), + "nvlink2-mr[0]", = NULL); + char *mem_name; + int off; + /* For some reason NVLink2 wants a separate NUMA node for its RAM = */ + uint32_t at =3D cpu_to_be32(GPURAM_ASSOCIATIVITY(phb, i)); + uint32_t associativity[] =3D { cpu_to_be32(0x4), at, at, at, at }; + uint64_t nv2_size =3D object_property_get_uint(nvlink2_mrobj, + "size", NULL); + uint64_t mem_reg_property[2] =3D { + cpu_to_be64(phb->nvgpus.gpus[i].gpa), cpu_to_be64(nv2_size) }; + + mem_name =3D g_strdup_printf("memory@" TARGET_FMT_lx, + phb->nvgpus.gpus[i].gpa); + off =3D fdt_add_subnode(fdt, 0, mem_name); + _FDT(off); + _FDT((fdt_setprop_string(fdt, off, "device_type", "memory"))); + _FDT((fdt_setprop(fdt, off, "reg", mem_reg_property, + sizeof(mem_reg_property)))); + _FDT((fdt_setprop(fdt, off, "ibm,associativity", associativity, + sizeof(associativity)))); + + _FDT((fdt_setprop_string(fdt, off, "compatible", + "ibm,coherent-device-memory"))); + mem_reg_property[1] =3D 0; + _FDT((fdt_setprop(fdt, off, "linux,usable-memory", mem_reg_propert= y, + sizeof(mem_reg_property)))); + /*_FDT((fdt_setprop_cell(fdt, off, "ibm,chip-id", phb->index))); */ + _FDT((fdt_setprop_cell(fdt, off, "phandle", PHANDLE_GPURAM(phb, i)= ))); + + g_free(mem_name); + } + + /* NVLink: Add fake NPU Links for NPU bridge's "ibm,nvlink" property */ + if (phb->nvgpus.num) { + char *npuname =3D g_strdup_printf("npuphb%d", phb->index); + int npuoff =3D fdt_add_subnode(fdt, 0, npuname); + int linkidx =3D 0; + + _FDT(npuoff); + _FDT(fdt_setprop_cell(fdt, npuoff, "#address-cells", 1)); + _FDT(fdt_setprop_cell(fdt, npuoff, "#size-cells", 0)); + _FDT((fdt_setprop_string(fdt, npuoff, "compatible", "ibm,power9-np= u"))); + g_free(npuname); + + for (i =3D 0; i < phb->nvgpus.num; ++i) { + for (j =3D 0; j < phb->nvgpus.gpus[i].links; ++j) { + char *linkname =3D g_strdup_printf("link@%d", linkidx); + int off =3D fdt_add_subnode(fdt, npuoff, linkname); + + _FDT(off); + _FDT((fdt_setprop_cell(fdt, off, "reg", linkidx))); + _FDT((fdt_setprop_string(fdt, off, "compatible", + "ibm,npu-link"))); + _FDT((fdt_setprop_cell(fdt, off, "phandle", + PHANDLE_NVLINK(phb, i, j)))); + _FDT((fdt_setprop_cell(fdt, off, "ibm,npu-link-index", + linkidx))); + g_free(linkname); + ++linkidx; + } + } + } + + for (i =3D 0; i < phb->nvgpus.num; ++i) { + PCIDevice *gpdev =3D phb->nvgpus.gpus[i].gpdev; + Object *nvlink2_mrobj =3D object_property_get_link(OBJECT(gpdev), + "nvlink2-mr[0]", = NULL); + MemoryRegion *mr =3D MEMORY_REGION(nvlink2_mrobj); + + memory_region_add_subregion(get_system_memory(), + phb->nvgpus.gpus[i].gpa, mr); + + for (j =3D 0; j < phb->nvgpus.gpus[i].links; ++j) { + PCIDevice *npdev =3D phb->nvgpus.gpus[i].npdev[j]; + Object *nvlink2_mrobj; + nvlink2_mrobj =3D object_property_get_link(OBJECT(npdev), + "nvlink2-atsd-mr[0]", + NULL); + if (nvlink2_mrobj) { + MemoryRegion *mr =3D MEMORY_REGION(nvlink2_mrobj); + memory_region_add_subregion(get_system_memory(), + phb->nvgpus.gpus[i].atsd[j], + mr); + } + } + } + return 0; } =20 diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index 2796837..e655dbc 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -2206,3 +2206,96 @@ int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **= errp) =20 return 0; } + +static void vfio_pci_npu2_atsd_get_tgt(Object *obj, Visitor *v, + const char *name, + void *opaque, Error **errp) +{ + uint64_t tgt =3D (uint64_t) opaque; + visit_type_uint64(v, name, &tgt, errp); +} + +int vfio_pci_nvlink2_ram_init(VFIOPCIDevice *vdev, Error **errp) +{ + int ret; + void *p; + struct vfio_region_info *nv2region =3D NULL; + struct vfio_info_cap_header *hdr; + struct vfio_region_info_cap_npu2 *cap; + MemoryRegion *nv2mr =3D g_malloc0(sizeof(*nv2mr)); + + ret =3D vfio_get_dev_region_info(&vdev->vbasedev, + VFIO_REGION_TYPE_PCI_VENDOR_TYPE | + PCI_VENDOR_ID_NVIDIA, + VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM, + &nv2region); + if (ret) { + return ret; + } + + p =3D mmap(NULL, nv2region->size, PROT_READ | PROT_WRITE | PROT_EXEC, + MAP_SHARED, vdev->vbasedev.fd, nv2region->offset); + + if (!p) { + return -errno; + } + + memory_region_init_ram_ptr(nv2mr, OBJECT(vdev), "nvlink2-mr", + nv2region->size, p); + + hdr =3D vfio_get_region_info_cap(nv2region, VFIO_REGION_INFO_CAP_NPU2); + cap =3D (struct vfio_region_info_cap_npu2 *) hdr; + + object_property_add(OBJECT(nv2mr), "tgt", "uint64", + vfio_pci_npu2_atsd_get_tgt, NULL, NULL, + (void *) cap->tgt, NULL); + trace_vfio_pci_nvidia_gpu_ram_setup_quirk(vdev->vbasedev.name, cap->tg= t, + nv2region->size); + g_free(nv2region); + + return 0; +} + +int vfio_pci_npu2_atsd_init(VFIOPCIDevice *vdev, Error **errp) +{ + int ret; + void *p; + struct vfio_region_info *atsd_region =3D NULL; + struct vfio_info_cap_header *hdr; + struct vfio_region_info_cap_npu2 *cap; + MemoryRegion *atsd_mr =3D g_malloc0(sizeof(*atsd_mr)); + + ret =3D vfio_get_dev_region_info(&vdev->vbasedev, + VFIO_REGION_TYPE_PCI_VENDOR_TYPE | + PCI_VENDOR_ID_IBM, + VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD, + &atsd_region); + if (ret) { + return ret; + } + + p =3D mmap(NULL, atsd_region->size, PROT_READ | PROT_WRITE | PROT_EXEC, + MAP_SHARED, vdev->vbasedev.fd, atsd_region->offset); + + if (!p) { + return -errno; + } + + memory_region_init_ram_device_ptr(atsd_mr, OBJECT(vdev), + "nvlink2-atsd-mr", + atsd_region->size, + p); + + hdr =3D vfio_get_region_info_cap(atsd_region, VFIO_REGION_INFO_CAP_NPU= 2); + cap =3D (struct vfio_region_info_cap_npu2 *) hdr; + + object_property_add(OBJECT(atsd_mr), "tgt", "uint64", + vfio_pci_npu2_atsd_get_tgt, NULL, NULL, + (void *) cap->tgt, NULL); + + trace_vfio_pci_npu2_setup_quirk(vdev->vbasedev.name, cap->tgt, + atsd_region->size); + g_free(atsd_region); + + return 0; +} diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7848b28..d7de202 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3074,6 +3074,20 @@ static void vfio_realize(PCIDevice *pdev, Error **er= rp) goto out_teardown; } =20 + if (vdev->vendor_id =3D=3D PCI_VENDOR_ID_NVIDIA && vdev->device_id =3D= =3D 0x1db1) { + ret =3D vfio_pci_nvlink2_ram_init(vdev, errp); + if (ret) { + error_report("Failed to setup GPU RAM"); + } + } + + if (vdev->vendor_id =3D=3D PCI_VENDOR_ID_IBM && vdev->device_id =3D=3D= 0x04ea) { + ret =3D vfio_pci_npu2_atsd_init(vdev, errp); + if (ret) { + error_report("Failed to setup ATSD"); + } + } + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index adfa75e..7595009 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -88,6 +88,9 @@ vfio_pci_igd_opregion_enabled(const char *name) "%s" vfio_pci_igd_host_bridge_enabled(const char *name) "%s" vfio_pci_igd_lpc_bridge_enabled(const char *name) "%s" =20 +vfio_pci_nvidia_gpu_ram_setup_quirk(const char *name, uint64_t tgt, uint64= _t size) "%s tgt=3D0x%"PRIx64" size=3D0x%"PRIx64 +vfio_pci_npu2_setup_quirk(const char *name, uint64_t tgt, uint64_t size) "= %s tgt=3D0x%"PRIx64" size=3D0x%"PRIx64 + # hw/vfio/common.c vfio_region_write(const char *name, int index, uint64_t addr, uint64_t dat= a, unsigned size) " (%s:region%d+0x%"PRIx64", 0x%"PRIx64 ", %d)" vfio_region_read(char *name, int index, uint64_t addr, unsigned size, uint= 64_t data) " (%s:region%d+0x%"PRIx64", %d) =3D 0x%"PRIx64 --=20 2.17.1