From nobody Sun May 19 13:07:53 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linux.ibm.com ARC-Seal: i=1; a=rsa-sha256; t=1588962165; cv=none; d=zohomail.com; s=zohoarc; b=a4yaQKjIIju7TGIXqvoXYwNIqHX85pM30uW9bCTucFua+/gcO43B+P3xJQVM/Rm99XBd8yrLTJDT0sTzI8v/F67CPZ0Kjw5+XZUUiczkmqxdndbgFZXBJ254icShNDNFgCqjWiBENkw5/L/lLKQ6Nh9jgfQd2PTXf87Ss6WONOI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588962165; h=Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Sender:Subject:To; bh=lguEb2CMEoPzPrUlfTDItvkJZUL/bzXLOrlnl/LvqqQ=; b=PZGb/NsFieDRe0S8eecE0QWC6nfyXCtPVbDeT/5gMflaQgdYfFwKOCiKgCZ3K+tN1J9Uxx4OZ7LjUgMkAWHZQdkOPBD4bIJYly6zGkxJ31F1q2j01nXaZq20yEXUJZ6sFLwKnZyclTF2viX2BZEibHHRrfJ7tUpXlFnazw74LP0= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588962165126513.2747444867888; Fri, 8 May 2020 11:22:45 -0700 (PDT) Received: from localhost ([::1]:37842 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jX7dz-0004Mg-Gg for importer@patchew.org; Fri, 08 May 2020 14:22:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54330) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jX7Hi-0004io-Jf; Fri, 08 May 2020 13:59:42 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:43626) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jX7Hh-0002k1-61; Fri, 08 May 2020 13:59:42 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 048HXm7c056383; Fri, 8 May 2020 13:59:32 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 30vtt449a5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 08 May 2020 13:59:32 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 048HtLYx023932; Fri, 8 May 2020 17:59:31 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma03dal.us.ibm.com with ESMTP id 30s0g88vgg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 08 May 2020 17:59:31 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 048HxUOb55050590 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 8 May 2020 17:59:30 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AC370AC05B; Fri, 8 May 2020 17:59:30 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 31B3FAC059; Fri, 8 May 2020 17:59:30 +0000 (GMT) Received: from arbab-laptop.localdomain (unknown [9.65.228.143]) by b01ledav006.gho.pok.ibm.com (Postfix) with SMTP; Fri, 8 May 2020 17:59:30 +0000 (GMT) Received: by arbab-laptop.localdomain (Postfix, from userid 152845) id C0D854606EE; Fri, 8 May 2020 12:59:27 -0500 (CDT) From: Reza Arbab To: David Gibson , qemu-ppc@nongnu.org, qemu-devel@nongnu.org Subject: [PATCH] spapr: Add a new level of NUMA for GPUs Date: Fri, 8 May 2020 12:59:27 -0500 Message-Id: <20200508175927.21791-1-arbab@linux.ibm.com> X-Mailer: git-send-email 2.18.1 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-08_16:2020-05-08, 2020-05-08 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 adultscore=0 spamscore=0 clxscore=1011 priorityscore=1501 suspectscore=0 bulkscore=0 lowpriorityscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005080147 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=148.163.156.1; envelope-from=arbab@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/08 13:59:37 X-ACL-Warn: Detected OS = Linux 3.x [generic] X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, KHOP_DYNAMIC=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Mailman-Approved-At: Fri, 08 May 2020 14:22:03 -0400 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kardashevskiy , Daniel Henrique Barboza , Leonardo Augusto Guimaraes Garcia Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" NUMA nodes corresponding to GPU memory currently have the same affinity/distance as normal memory nodes. Add a third NUMA associativity reference point enabling us to give GPU nodes more distance. Before, `numactl -H` output in a guest with 4 GPUs (nodes 2-5): node distances: node 0 1 2 3 4 5 0: 10 40 40 40 40 40 1: 40 10 40 40 40 40 2: 40 40 10 40 40 40 3: 40 40 40 10 40 40 4: 40 40 40 40 10 40 5: 40 40 40 40 40 10 After: node distances: node 0 1 2 3 4 5 0: 10 40 80 80 80 80 1: 40 10 80 80 80 80 2: 80 80 10 80 80 80 3: 80 80 80 10 80 80 4: 80 80 80 80 10 80 5: 80 80 80 80 80 10 These are the same distances as on the host, mirroring the change made to host firmware in skiboot commit f845a648b8cb ("numa/associativity: Add a new level of NUMA for GPU's"). Signed-off-by: Reza Arbab --- hw/ppc/spapr.c | 6 +++++- hw/ppc/spapr_pci_nvlink2.c | 8 +++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index c18eab0a2305..53567f98f0c6 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -892,7 +892,11 @@ static void spapr_dt_rtas(SpaprMachineState *spapr, vo= id *fdt) int rtas; GString *hypertas =3D g_string_sized_new(256); GString *qemu_hypertas =3D g_string_sized_new(256); - uint32_t refpoints[] =3D { cpu_to_be32(0x4), cpu_to_be32(0x4) }; + uint32_t refpoints[] =3D { + cpu_to_be32(0x4), + cpu_to_be32(0x4), + cpu_to_be32(0x2), + }; uint64_t max_device_addr =3D MACHINE(spapr)->device_memory->base + memory_region_size(&MACHINE(spapr)->device_memory->mr); uint32_t lrdr_capacity[] =3D { diff --git a/hw/ppc/spapr_pci_nvlink2.c b/hw/ppc/spapr_pci_nvlink2.c index 8332d5694e46..f2cb26019e88 100644 --- a/hw/ppc/spapr_pci_nvlink2.c +++ b/hw/ppc/spapr_pci_nvlink2.c @@ -37,8 +37,6 @@ #define PHANDLE_NVLINK(phb, gn, nn) (0x00130000 | (((phb)->index) << 8) |= \ ((gn) << 4) | (nn)) =20 -#define SPAPR_GPU_NUMA_ID (cpu_to_be32(1)) - typedef struct SpaprPhbPciNvGpuSlot { uint64_t tgt; uint64_t gpa; @@ -361,9 +359,9 @@ void spapr_phb_nvgpu_ram_populate_dt(SpaprPhbState *sph= b, void *fdt) "nvlink2-mr[0]", NULL); uint32_t associativity[] =3D { cpu_to_be32(0x4), - SPAPR_GPU_NUMA_ID, - SPAPR_GPU_NUMA_ID, - SPAPR_GPU_NUMA_ID, + cpu_to_be32(nvslot->numa_id), + cpu_to_be32(nvslot->numa_id), + cpu_to_be32(nvslot->numa_id), cpu_to_be32(nvslot->numa_id) }; uint64_t size =3D object_property_get_uint(nv_mrobj, "size", NULL); --=20 2.18.1