From nobody Tue Feb 10 00:59:46 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1599192337; cv=none; d=zohomail.com; s=zohoarc; b=CLt3JKXWLEnaUlmhkxppIpjzYVJN9gH4pcvKTyJ0Pz9Qh57ZPQaRKmnndsAhDSGB7J/CP2HbOPNvBkJxUYcjW6cgro2iSCKjUZXPDiSJxahN1pq0BidsQucx/93hJH9pN4FOPpNchBO3pFoJokV5bb1LrRp96YRaNoFXkLH5Dwc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1599192337; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=tI3kWYpFycLBTcBLiTUv6HtR3lJFpkl0kib+ZVlpt7c=; b=O/7KDXjSBustUCtc+bUT33K7z2VinyIKFCqILoBcY8kohhE3DIHEP6hg16fKX9GrSON6tjoAsm/9URh4aVimdJHcg8rXFMtND8ZD2ouEHevVal4FYH0USZh/xQrKG2WaK6iIdrDnAtk68VGg1WiWPuUspFOJFYoTGDHg7jVcVnI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1599192337579710.5263487318485; Thu, 3 Sep 2020 21:05:37 -0700 (PDT) Received: from localhost ([::1]:46446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kE2ym-0007x4-M1 for importer@patchew.org; Fri, 04 Sep 2020 00:05:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55024) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kE2i3-0003hp-SK; Thu, 03 Sep 2020 23:48:19 -0400 Received: from ozlabs.org ([2401:3900:2:1::2]:42665) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kE2i0-0004z6-5C; Thu, 03 Sep 2020 23:48:19 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 4BjNst6dRlz9sWH; Fri, 4 Sep 2020 13:47:30 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1599191250; bh=EMhX4EaQtYbgZPvdJDLhYZH3ThfzDd5O94XDFPoOgFM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bPtFLfWo/excwiLHRoEOV6wvwMAcCYYxOhQHm6GZJql9AoT/x3xbVBUR7VCcfQERI PfNdCZDLhfPg34rr0R9MRBsAy7gqkmXszvk+9TWe2psb9tHunX7SW1gH+x/PHeh1ba LTfDYwhPxWndnLfqBnpP4Bu882UVMWbUyBBG+vfc= From: David Gibson To: peter.maydell@linaro.org Subject: [PULL 27/30] spapr: introduce SpaprMachineState::numa_assoc_array Date: Fri, 4 Sep 2020 13:47:16 +1000 Message-Id: <20200904034719.673626-28-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200904034719.673626-1-david@gibson.dropbear.id.au> References: <20200904034719.673626-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2401:3900:2:1::2; envelope-from=dgibson@ozlabs.org; helo=ozlabs.org X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: danielhb413@gmail.com, qemu-devel@nongnu.org, groug@kaod.org, qemu-ppc@nongnu.org, bauerman@linux.ibm.com, David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Daniel Henrique Barboza The next step to centralize all NUMA/associativity handling in the spapr machine is to create a 'one stop place' for all things ibm,associativity. This patch introduces numa_assoc_array, a 2 dimensional array that will store all ibm,associativity arrays of all NUMA nodes. This array is initialized in a new spapr_numa_associativity_init() function, called in spapr_machine_init(). It is being initialized with the same values used in other ibm,associativity properties around spapr files (i.e. all zeros, last value is node_id). The idea is to remove all hardcoded definitions and FDT writes of ibm,associativity arrays, doing instead a call to the new helper spapr_numa_write_associativity_dt() helper, that will be able to write the DT with the correct values. We'll start small, handling the trivial cases first. The remaining instances of ibm,associativity will be handled next. Signed-off-by: Daniel Henrique Barboza Message-Id: <20200903220639.563090-2-danielhb413@gmail.com> Signed-off-by: David Gibson --- hw/ppc/spapr.c | 23 ++++++++++------------- hw/ppc/spapr_numa.c | 30 ++++++++++++++++++++++++++++++ hw/ppc/spapr_nvdimm.c | 19 +++++++------------ hw/ppc/spapr_pci.c | 9 ++------- include/hw/ppc/spapr.h | 12 ++++++++++++ include/hw/ppc/spapr_numa.h | 11 +++++++++++ include/hw/ppc/spapr_nvdimm.h | 2 +- 7 files changed, 73 insertions(+), 33 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index a45912acac..1ad6f59863 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -314,14 +314,9 @@ static void add_str(GString *s, const gchar *s1) g_string_append_len(s, s1, strlen(s1) + 1); } =20 -static int spapr_dt_memory_node(void *fdt, int nodeid, hwaddr start, - hwaddr size) +static int spapr_dt_memory_node(SpaprMachineState *spapr, void *fdt, int n= odeid, + hwaddr start, hwaddr size) { - uint32_t associativity[] =3D { - cpu_to_be32(0x4), /* length */ - cpu_to_be32(0x0), cpu_to_be32(0x0), - cpu_to_be32(0x0), cpu_to_be32(nodeid) - }; char mem_name[32]; uint64_t mem_reg_property[2]; int off; @@ -335,8 +330,7 @@ static int spapr_dt_memory_node(void *fdt, int nodeid, = hwaddr start, _FDT((fdt_setprop_string(fdt, off, "device_type", "memory"))); _FDT((fdt_setprop(fdt, off, "reg", mem_reg_property, sizeof(mem_reg_property)))); - _FDT((fdt_setprop(fdt, off, "ibm,associativity", associativity, - sizeof(associativity)))); + spapr_numa_write_associativity_dt(spapr, fdt, off, nodeid); return off; } =20 @@ -649,7 +643,7 @@ static int spapr_dt_memory(SpaprMachineState *spapr, vo= id *fdt) if (!mem_start) { /* spapr_machine_init() checks for rma_size <=3D node0_size * already */ - spapr_dt_memory_node(fdt, i, 0, spapr->rma_size); + spapr_dt_memory_node(spapr, fdt, i, 0, spapr->rma_size); mem_start +=3D spapr->rma_size; node_size -=3D spapr->rma_size; } @@ -661,7 +655,7 @@ static int spapr_dt_memory(SpaprMachineState *spapr, vo= id *fdt) sizetmp =3D 1ULL << ctzl(mem_start); } =20 - spapr_dt_memory_node(fdt, i, mem_start, sizetmp); + spapr_dt_memory_node(spapr, fdt, i, mem_start, sizetmp); node_size -=3D sizetmp; mem_start +=3D sizetmp; } @@ -1275,7 +1269,7 @@ void *spapr_build_fdt(SpaprMachineState *spapr, bool = reset, size_t space) =20 /* NVDIMM devices */ if (mc->nvdimm_supported) { - spapr_dt_persistent_memory(fdt); + spapr_dt_persistent_memory(spapr, fdt); } =20 return fdt; @@ -2810,6 +2804,9 @@ static void spapr_machine_init(MachineState *machine) */ spapr->gpu_numa_id =3D MAX(1, machine->numa_state->num_nodes); =20 + /* Init numa_assoc_array */ + spapr_numa_associativity_init(spapr, machine); + if ((!kvm_enabled() || kvmppc_has_cap_mmu_radix()) && ppc_type_check_compat(machine->cpu_type, CPU_POWERPC_LOGICAL_3_00,= 0, spapr->max_compat_pvr)) { @@ -3394,7 +3391,7 @@ int spapr_lmb_dt_populate(SpaprDrc *drc, SpaprMachine= State *spapr, addr =3D spapr_drc_index(drc) * SPAPR_MEMORY_BLOCK_SIZE; node =3D object_property_get_uint(OBJECT(drc->dev), PC_DIMM_NODE_PROP, &error_abort); - *fdt_start_offset =3D spapr_dt_memory_node(fdt, node, addr, + *fdt_start_offset =3D spapr_dt_memory_node(spapr, fdt, node, addr, SPAPR_MEMORY_BLOCK_SIZE); return 0; } diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index cdf3288cbd..f6b6fe648f 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -15,6 +15,36 @@ #include "hw/ppc/spapr_numa.h" #include "hw/ppc/fdt.h" =20 + +void spapr_numa_associativity_init(SpaprMachineState *spapr, + MachineState *machine) +{ + int nb_numa_nodes =3D machine->numa_state->num_nodes; + int i; + + /* + * For all associativity arrays: first position is the size, + * position MAX_DISTANCE_REF_POINTS is always the numa_id, + * represented by the index 'i'. + * + * This will break on sparse NUMA setups, when/if QEMU starts + * to support it, because there will be no more guarantee that + * 'i' will be a valid node_id set by the user. + */ + for (i =3D 0; i < nb_numa_nodes; i++) { + spapr->numa_assoc_array[i][0] =3D cpu_to_be32(MAX_DISTANCE_REF_POI= NTS); + spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] =3D cpu_to_be3= 2(i); + } +} + +void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, + int offset, int nodeid) +{ + _FDT((fdt_setprop(fdt, offset, "ibm,associativity", + spapr->numa_assoc_array[nodeid], + sizeof(spapr->numa_assoc_array[nodeid])))); +} + /* * Helper that writes ibm,associativity-reference-points and * max-associativity-domains in the RTAS pointed by @rtas diff --git a/hw/ppc/spapr_nvdimm.c b/hw/ppc/spapr_nvdimm.c index 5188e2f503..63872054f3 100644 --- a/hw/ppc/spapr_nvdimm.c +++ b/hw/ppc/spapr_nvdimm.c @@ -31,6 +31,7 @@ #include "hw/ppc/fdt.h" #include "qemu/range.h" #include "sysemu/sysemu.h" +#include "hw/ppc/spapr_numa.h" =20 void spapr_nvdimm_validate(HotplugHandler *hotplug_dev, NVDIMMDevice *nvdi= mm, uint64_t size, Error **errp) @@ -117,8 +118,8 @@ void spapr_create_nvdimm_dr_connectors(SpaprMachineStat= e *spapr) } =20 =20 -static int spapr_dt_nvdimm(void *fdt, int parent_offset, - NVDIMMDevice *nvdimm) +static int spapr_dt_nvdimm(SpaprMachineState *spapr, void *fdt, + int parent_offset, NVDIMMDevice *nvdimm) { int child_offset; char *buf; @@ -128,11 +129,6 @@ static int spapr_dt_nvdimm(void *fdt, int parent_offse= t, &error_abort); uint64_t slot =3D object_property_get_uint(OBJECT(nvdimm), PC_DIMM_SLO= T_PROP, &error_abort); - uint32_t associativity[] =3D { - cpu_to_be32(0x4), /* length */ - cpu_to_be32(0x0), cpu_to_be32(0x0), - cpu_to_be32(0x0), cpu_to_be32(node) - }; uint64_t lsize =3D nvdimm->label_size; uint64_t size =3D object_property_get_int(OBJECT(nvdimm), PC_DIMM_SIZE= _PROP, NULL); @@ -152,8 +148,7 @@ static int spapr_dt_nvdimm(void *fdt, int parent_offset, _FDT((fdt_setprop_string(fdt, child_offset, "compatible", "ibm,pmemory= "))); _FDT((fdt_setprop_string(fdt, child_offset, "device_type", "ibm,pmemor= y"))); =20 - _FDT((fdt_setprop(fdt, child_offset, "ibm,associativity", associativit= y, - sizeof(associativity)))); + spapr_numa_write_associativity_dt(spapr, fdt, child_offset, node); =20 buf =3D qemu_uuid_unparse_strdup(&nvdimm->uuid); _FDT((fdt_setprop_string(fdt, child_offset, "ibm,unit-guid", buf))); @@ -179,12 +174,12 @@ int spapr_pmem_dt_populate(SpaprDrc *drc, SpaprMachin= eState *spapr, { NVDIMMDevice *nvdimm =3D NVDIMM(drc->dev); =20 - *fdt_start_offset =3D spapr_dt_nvdimm(fdt, 0, nvdimm); + *fdt_start_offset =3D spapr_dt_nvdimm(spapr, fdt, 0, nvdimm); =20 return 0; } =20 -void spapr_dt_persistent_memory(void *fdt) +void spapr_dt_persistent_memory(SpaprMachineState *spapr, void *fdt) { int offset =3D fdt_subnode_offset(fdt, 0, "persistent-memory"); GSList *iter, *nvdimms =3D nvdimm_get_device_list(); @@ -202,7 +197,7 @@ void spapr_dt_persistent_memory(void *fdt) for (iter =3D nvdimms; iter; iter =3D iter->next) { NVDIMMDevice *nvdimm =3D iter->data; =20 - spapr_dt_nvdimm(fdt, offset, nvdimm); + spapr_dt_nvdimm(spapr, fdt, offset, nvdimm); } g_slist_free(nvdimms); =20 diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index 0a418f1e67..4d97ff6c70 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -52,6 +52,7 @@ #include "sysemu/kvm.h" #include "sysemu/hostmem.h" #include "sysemu/numa.h" +#include "hw/ppc/spapr_numa.h" =20 /* Copied from the kernel arch/powerpc/platforms/pseries/msi.c */ #define RTAS_QUERY_FN 0 @@ -2321,11 +2322,6 @@ int spapr_dt_phb(SpaprMachineState *spapr, SpaprPhbS= tate *phb, cpu_to_be32(1), cpu_to_be32(RTAS_IBM_RESET_PE_DMA_WINDOW) }; - uint32_t associativity[] =3D {cpu_to_be32(0x4), - cpu_to_be32(0x0), - cpu_to_be32(0x0), - cpu_to_be32(0x0), - cpu_to_be32(phb->numa_node)}; SpaprTceTable *tcet; SpaprDrc *drc; Error *err =3D NULL; @@ -2358,8 +2354,7 @@ int spapr_dt_phb(SpaprMachineState *spapr, SpaprPhbSt= ate *phb, =20 /* Advertise NUMA via ibm,associativity */ if (phb->numa_node !=3D -1) { - _FDT(fdt_setprop(fdt, bus_off, "ibm,associativity", associativity, - sizeof(associativity))); + spapr_numa_write_associativity_dt(spapr, fdt, bus_off, phb->numa_n= ode); } =20 /* Build the interrupt-map, this must matches what is done diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index a1e230ad39..9a63380801 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -105,6 +105,16 @@ typedef enum { =20 #define FDT_MAX_SIZE 0x100000 =20 +/* + * NUMA related macros. MAX_DISTANCE_REF_POINTS was taken + * from Taken from Linux kernel arch/powerpc/mm/numa.h. + * + * NUMA_ASSOC_SIZE is the base array size of an ibm,associativity + * array for any non-CPU resource. + */ +#define MAX_DISTANCE_REF_POINTS 4 +#define NUMA_ASSOC_SIZE (MAX_DISTANCE_REF_POINTS + 1) + typedef struct SpaprCapabilities SpaprCapabilities; struct SpaprCapabilities { uint8_t caps[SPAPR_CAP_NUM]; @@ -231,6 +241,8 @@ struct SpaprMachineState { unsigned gpu_numa_id; SpaprTpmProxy *tpm_proxy; =20 + uint32_t numa_assoc_array[MAX_NODES][NUMA_ASSOC_SIZE]; + Error *fwnmi_migration_blocker; }; =20 diff --git a/include/hw/ppc/spapr_numa.h b/include/hw/ppc/spapr_numa.h index 7a370a8768..a2a4df55f7 100644 --- a/include/hw/ppc/spapr_numa.h +++ b/include/hw/ppc/spapr_numa.h @@ -13,8 +13,19 @@ #ifndef HW_SPAPR_NUMA_H #define HW_SPAPR_NUMA_H =20 +#include "hw/boards.h" #include "hw/ppc/spapr.h" =20 +/* + * Having both SpaprMachineState and MachineState as arguments + * feels odd, but it will spare a MACHINE() call inside the + * function. spapr_machine_init() is the only caller for it, and + * it has both pointers resolved already. + */ +void spapr_numa_associativity_init(SpaprMachineState *spapr, + MachineState *machine); void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rta= s); +void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, + int offset, int nodeid); =20 #endif /* HW_SPAPR_NUMA_H */ diff --git a/include/hw/ppc/spapr_nvdimm.h b/include/hw/ppc/spapr_nvdimm.h index 10a6d9dbbc..3eb344e8e9 100644 --- a/include/hw/ppc/spapr_nvdimm.h +++ b/include/hw/ppc/spapr_nvdimm.h @@ -27,7 +27,7 @@ QEMU_BUILD_BUG_ON(SPAPR_MINIMUM_SCM_BLOCK_SIZE % SPAPR_ME= MORY_BLOCK_SIZE); =20 int spapr_pmem_dt_populate(SpaprDrc *drc, SpaprMachineState *spapr, void *fdt, int *fdt_start_offset, Error **errp); -void spapr_dt_persistent_memory(void *fdt); +void spapr_dt_persistent_memory(SpaprMachineState *spapr, void *fdt); void spapr_nvdimm_validate(HotplugHandler *hotplug_dev, NVDIMMDevice *nvdi= mm, uint64_t size, Error **errp); void spapr_add_nvdimm(DeviceState *dev, uint64_t slot, Error **errp); --=20 2.26.2