From nobody Wed May 8 01:48:36 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1630974448841188.2211233126087; Mon, 6 Sep 2021 17:27:28 -0700 (PDT) Received: from localhost ([::1]:37204 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNOxT-0006cZ-Le for importer@patchew.org; Mon, 06 Sep 2021 20:27:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59038) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNOvn-0003rq-14; Mon, 06 Sep 2021 20:25:43 -0400 Received: from mail-qv1-xf34.google.com ([2607:f8b0:4864:20::f34]:33724) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNOvl-0005h7-9F; Mon, 06 Sep 2021 20:25:42 -0400 Received: by mail-qv1-xf34.google.com with SMTP id a5so4864381qvq.0; Mon, 06 Sep 2021 17:25:40 -0700 (PDT) Received: from rekt.COMFAST ([179.247.136.122]) by smtp.gmail.com with ESMTPSA id 207sm7892056qkh.45.2021.09.06.17.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Sep 2021 17:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ewk+Cj943TQXjiQMHnCOouOqj0zsxj96uycEsJ7iacA=; b=hGxtgFR4QIt4a/c7no6t3I16GC6p4+o6thEPdvdVTmvuNdHoKTmyQK6vbS8YmC+V3M eq46biZVPmUwqh64PwI39dGYfDUNonUx/5t7zVaW4H+3YwnXEZUKt15ytTxlTJbTp9mN OtUr+oep73lWPSX0HnJ3fmPS3NkIUfPRf8Md5dV5T1Y0gan3U6YueTFydOop8oh0smHz WOHIdUH6Gx0sUbLatTvFQTDulgOe3pn/NY9996yDscVGTlGF/vM0w7w6IDUGMyIJr7bF ZrsSXZjEFPL/76m2Dk8/6oV1pZWeHsDbwrQQemHdMurhMIYTO6sTTUkpUYEHW6ojw7jX /nTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ewk+Cj943TQXjiQMHnCOouOqj0zsxj96uycEsJ7iacA=; b=IouIgTGPgXQwBlGvnkanWbGGKpAmtmsWHyDzPcitm5aVOtxBiBDOrEzYAEp7cyueWH 4sbRY+djaIcDRaYFBBulbB2Xh58z75rm1F564iB/5g3ofr0f2DMjJQ4RK2AdBIEL1xhb lay1pTPNPQvQMUWjDbgMNfcyQ4obwh8xK90R7AWMsP2LFySZQkzEMwsc+/T4x8FMbacb dTsRzAJ66/qZhvS1cMUEDjAZcjh2+n+z8zoZM+ZSZ5LlFn4nEZoYRCD5HSiEAhSftx+h riyNvRcVL7GsjRoozYSnEXfLfhV31eWZAvQPBNjq2ubfKvJnRs6v5w3Hb9ORZH2DftqU dU7w== X-Gm-Message-State: AOAM530D/Qq7enQ2J/DBmiJZLlCRa9BMtPmPG1iT6GbE4Zf5z3N3niin Dwd8reNctO1aJbfT765YR5prSJezaGc= X-Google-Smtp-Source: ABdhPJwsXN3m36/ldZc2Cmjcj/F89RFm5Z3/EHjoY9OBVAM5WQEpR50IIDtlOymmQn0wWKUEdpd0dQ== X-Received: by 2002:a05:6214:1142:: with SMTP id b2mr14792764qvt.0.1630974339955; Mon, 06 Sep 2021 17:25:39 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v5 1/4] spapr: move NUMA associativity init to machine reset Date: Mon, 6 Sep 2021 21:25:24 -0300 Message-Id: <20210907002527.412013-2-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907002527.412013-1-danielhb413@gmail.com> References: <20210907002527.412013-1-danielhb413@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::f34; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf34.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1630974449481100005 Content-Type: text/plain; charset="utf-8" At this moment we only support one form of NUMA affinity, FORM1. This allows us to init the internal structures during machine_init(), and given that NUMA distances won't change during the guest lifetime we don't need to bother with that again. We're about to introduce FORM2, a new NUMA affinity mode for pSeries guests. This means that we'll only be certain about the affinity mode being used after client architecture support. This also means that the guest can switch affinity modes in machine reset. Let's prepare the ground for the FORM2 support by moving the NUMA internal data init from machine_init() to machine_reset(). Change the name to spapr_numa_associativity_reset() to make it clearer that this is a function that can be called multiple times during the guest lifecycle. We're also simplifying its current API since this method will be called during CAS time (do_client_architecture_support()) later on and there's no MachineState pointer already solved there. Signed-off-by: Daniel Henrique Barboza Reviewed-by: Greg Kurz --- hw/ppc/spapr.c | 6 +++--- hw/ppc/spapr_numa.c | 4 ++-- include/hw/ppc/spapr_numa.h | 9 +-------- 3 files changed, 6 insertions(+), 13 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index d39fd4e644..8e1ff6cd10 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -1621,6 +1621,9 @@ static void spapr_machine_reset(MachineState *machine) */ spapr_irq_reset(spapr, &error_fatal); =20 + /* Reset numa_assoc_array */ + spapr_numa_associativity_reset(spapr); + /* * There is no CAS under qtest. Simulate one to please the code that * depends on spapr->ov5_cas. This is especially needed to test device @@ -2808,9 +2811,6 @@ static void spapr_machine_init(MachineState *machine) =20 spapr->gpu_numa_id =3D spapr_numa_initial_nvgpu_numa_id(machine); =20 - /* Init numa_assoc_array */ - spapr_numa_associativity_init(spapr, machine); - if ((!kvm_enabled() || kvmppc_has_cap_mmu_radix()) && ppc_type_check_compat(machine->cpu_type, CPU_POWERPC_LOGICAL_3_00,= 0, spapr->max_compat_pvr)) { diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 779f18b994..9ee4b479fe 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -155,10 +155,10 @@ static void spapr_numa_define_associativity_domains(S= paprMachineState *spapr) =20 } =20 -void spapr_numa_associativity_init(SpaprMachineState *spapr, - MachineState *machine) +void spapr_numa_associativity_reset(SpaprMachineState *spapr) { SpaprMachineClass *smc =3D SPAPR_MACHINE_GET_CLASS(spapr); + MachineState *machine =3D MACHINE(spapr); int nb_numa_nodes =3D machine->numa_state->num_nodes; int i, j, max_nodes_with_gpus; bool using_legacy_numa =3D spapr_machine_using_legacy_numa(spapr); diff --git a/include/hw/ppc/spapr_numa.h b/include/hw/ppc/spapr_numa.h index 6f9f02d3de..0e457bba57 100644 --- a/include/hw/ppc/spapr_numa.h +++ b/include/hw/ppc/spapr_numa.h @@ -16,14 +16,7 @@ #include "hw/boards.h" #include "hw/ppc/spapr.h" =20 -/* - * Having both SpaprMachineState and MachineState as arguments - * feels odd, but it will spare a MACHINE() call inside the - * function. spapr_machine_init() is the only caller for it, and - * it has both pointers resolved already. - */ -void spapr_numa_associativity_init(SpaprMachineState *spapr, - MachineState *machine); +void spapr_numa_associativity_reset(SpaprMachineState *spapr); void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rta= s); void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, int offset, int nodeid); --=20 2.31.1 From nobody Wed May 8 01:48:36 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1630974543585650.0228086555508; Mon, 6 Sep 2021 17:29:03 -0700 (PDT) Received: from localhost ([::1]:42176 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNOz0-0001Yq-L9 for importer@patchew.org; Mon, 06 Sep 2021 20:29:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59050) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNOvp-0003wC-SK; Mon, 06 Sep 2021 20:25:45 -0400 Received: from mail-qv1-xf2d.google.com ([2607:f8b0:4864:20::f2d]:34424) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNOvn-0005iA-6e; Mon, 06 Sep 2021 20:25:45 -0400 Received: by mail-qv1-xf2d.google.com with SMTP id e18so4837618qvo.1; Mon, 06 Sep 2021 17:25:42 -0700 (PDT) Received: from rekt.COMFAST ([179.247.136.122]) by smtp.gmail.com with ESMTPSA id 207sm7892056qkh.45.2021.09.06.17.25.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Sep 2021 17:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=78x2NwMDsIaWD5wj157LidavEe4AZL2cMvVX/tKpj/k=; b=pZWODoMSPeMH/p7UIQb7jD0z5uvExyT+LcKRqZL8MzH4lLCjos48rWA1+/pX4oo1bj 4yiTJn/Zeu7HHI8qUuDeBG2Fo/+gT3b8Evoxh2uHZDEKwprkcrY4Z6lAlb2z9DEVUIdP BFnrpNgnSv4N5eAn/43mmK66jqdy74yRlCiL9Vi1NHdYCYKamaaB0+ZWul9Su/0y1Go3 JpR8Wuv0bo/dLH/dh3hNE0lS7NvQDQZzuW9Nu+ZbyEBuwB1Kzdn+oHR21Ypt6IYcw8TN UiBRI4rqxX74BpwKACtgA5iGIjVM+0DAoTDXmqy72JlD8+foArKZfrfIQ/5TVTBu8lNH /9Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=78x2NwMDsIaWD5wj157LidavEe4AZL2cMvVX/tKpj/k=; b=tchw7V/hmYcNA7UOPSR4sL2rCVSV/8WbS+0hsBAneAYE7agPJGtg2erviqmHwTfLpO 8NVJ/HcrnJP8JxuDwhPtIZ0n6k/whZafFKbKKJvaSaUMWMYOLy2P+wXfQ0OS+XAO9MX/ oNMwOHLQbk4NerEinabkZnH2j4MhNTbOm8ZxMhP9tosYhyhrgBWx/8djv9y+BlDvzIZT iHDDnbrRu4gM0ZjwbdRFaZq2UMAlufHsx94ma5jJNZx2/DJa9xByTWtckzgLQsG3wdme /qQzTPwiMBdPW/Ijetr2zoknl1wnsiTjQxuk91FFKNG2MZKpOGNd+KW88C7SWklOSTVy jkvw== X-Gm-Message-State: AOAM5339ysVRfqrOiw372ICG+P0dnG/+7wgcE3Ph7Fj/DGcVOqhJCXGZ 2ffHz+B4Sk9UQce2pDXA+ssH1lzw8sM= X-Google-Smtp-Source: ABdhPJx9CjRrno/9Zanpd8gEM+ORDmFN0qS2UwWf8KdmMGxMNeu1Ufipcv+K/vJBbMbeObEoZFvYRA== X-Received: by 2002:a05:6214:3008:: with SMTP id ke8mr14709663qvb.17.1630974341880; Mon, 06 Sep 2021 17:25:41 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v5 2/4] spapr_numa.c: split FORM1 code into helpers Date: Mon, 6 Sep 2021 21:25:25 -0300 Message-Id: <20210907002527.412013-3-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907002527.412013-1-danielhb413@gmail.com> References: <20210907002527.412013-1-danielhb413@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::f2d; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf2d.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1630974545404100001 Content-Type: text/plain; charset="utf-8" The upcoming FORM2 NUMA affinity will support asymmetric NUMA topologies and doesn't need be concerned with all the legacy support for older pseries FORM1 guests. We're also not going to calculate associativity domains based on numa distance (via spapr_numa_define_associativity_domains) since the distances will be written directly into new DT properties. Let's split FORM1 code into its own functions to allow for easier insertion of FORM2 logic later on. Reviewed-by: Greg Kurz Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_numa.c | 61 +++++++++++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 9ee4b479fe..84636cb86a 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -155,6 +155,32 @@ static void spapr_numa_define_associativity_domains(Sp= aprMachineState *spapr) =20 } =20 +/* + * Set NUMA machine state data based on FORM1 affinity semantics. + */ +static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, + MachineState *machine) +{ + bool using_legacy_numa =3D spapr_machine_using_legacy_numa(spapr); + + /* + * Legacy NUMA guests (pseries-5.1 and older, or guests with only + * 1 NUMA node) will not benefit from anything we're going to do + * after this point. + */ + if (using_legacy_numa) { + return; + } + + if (!spapr_numa_is_symmetrical(machine)) { + error_report("Asymmetrical NUMA topologies aren't supported " + "in the pSeries machine"); + exit(EXIT_FAILURE); + } + + spapr_numa_define_associativity_domains(spapr); +} + void spapr_numa_associativity_reset(SpaprMachineState *spapr) { SpaprMachineClass *smc =3D SPAPR_MACHINE_GET_CLASS(spapr); @@ -210,22 +236,7 @@ void spapr_numa_associativity_reset(SpaprMachineState = *spapr) spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] =3D cpu_to_be3= 2(i); } =20 - /* - * Legacy NUMA guests (pseries-5.1 and older, or guests with only - * 1 NUMA node) will not benefit from anything we're going to do - * after this point. - */ - if (using_legacy_numa) { - return; - } - - if (!spapr_numa_is_symmetrical(machine)) { - error_report("Asymmetrical NUMA topologies aren't supported " - "in the pSeries machine"); - exit(EXIT_FAILURE); - } - - spapr_numa_define_associativity_domains(spapr); + spapr_numa_FORM1_affinity_init(spapr, machine); } =20 void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, @@ -302,12 +313,8 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineS= tate *spapr, void *fdt, return ret; } =20 -/* - * Helper that writes ibm,associativity-reference-points and - * max-associativity-domains in the RTAS pointed by @rtas - * in the DT @fdt. - */ -void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rta= s) +static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, + void *fdt, int rtas) { MachineState *ms =3D MACHINE(spapr); SpaprMachineClass *smc =3D SPAPR_MACHINE_GET_CLASS(spapr); @@ -365,6 +372,16 @@ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr= , void *fdt, int rtas) maxdomains, sizeof(maxdomains))); } =20 +/* + * Helper that writes ibm,associativity-reference-points and + * max-associativity-domains in the RTAS pointed by @rtas + * in the DT @fdt. + */ +void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rta= s) +{ + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); +} + static target_ulong h_home_node_associativity(PowerPCCPU *cpu, SpaprMachineState *spapr, target_ulong opcode, --=20 2.31.1 From nobody Wed May 8 01:48:36 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1630974656129281.19950964014106; Mon, 6 Sep 2021 17:30:56 -0700 (PDT) Received: from localhost ([::1]:44502 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNP0p-0003FE-57 for importer@patchew.org; Mon, 06 Sep 2021 20:30:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59086) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNOw5-0004eo-4p; Mon, 06 Sep 2021 20:26:01 -0400 Received: from mail-qv1-xf2c.google.com ([2607:f8b0:4864:20::f2c]:40500) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNOvr-0005k3-Lj; Mon, 06 Sep 2021 20:26:00 -0400 Received: by mail-qv1-xf2c.google.com with SMTP id 93so4822885qva.7; Mon, 06 Sep 2021 17:25:45 -0700 (PDT) Received: from rekt.COMFAST ([179.247.136.122]) by smtp.gmail.com with ESMTPSA id 207sm7892056qkh.45.2021.09.06.17.25.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Sep 2021 17:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tW0QOQSSP0XPRH7DVGBI3ZVFDrH57j+4eTdaY13/ywY=; b=i8/h24/TROmwjpA5HWHMug+J8REhWhzFDp2tcuGwj8fccyAp1SWvzVDGUnenwsStpQ o5DwxTcnSzrEdHYk6RrrBRf5s/j3NMzSy/4RdDV3yEd/g2tzUWUKSnyKHvBpPzZuB3C/ ZdTsS8W91IC3PyhlnTYChY6HzkO5E+vfvh+1a6ukMFGmZZbbHcp2xmTYxtJ41ay5eu2e v0KDhnsTyP4UdUWENsqmk5Y1/Oy7loyRgRkDPsxRDcFSLKyjTA8I3swmsH1psljUP/6x wBVKqqPyINxbLxn+Z95AXOVpxuTBnOdhEzXz4ZU86Hi/7/bgk00cLJBi2L3BCtLBMBtU 4mvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tW0QOQSSP0XPRH7DVGBI3ZVFDrH57j+4eTdaY13/ywY=; b=RVSmV5GPJCvils0IpW0lrEfEeP78jdxSShG3kW6+VyLXovbMBwqY+3iNrGSNrVAyHD sBKLyarpA6C8GTQwGk6U/87NCO/sC/GgeyMApowhphx4DkZYjcPa3eo4sVjbUKcObfkR wUx/0+Hpxhst4oct53F+I8sxw1EFt0eIS9IV3GwR3k7R8CmUWu9ILwXpZ9yH4V5ChOLl fMdmQgLpUCELw8/TbRkYjePVS9ZnVrtn6qUS4v1CGRmANpiv1rLJOG+sNaK/OGaaRIvW JdVZjqpKaSPPF6dIsj7NkWCz+aTZ1UWzXpZv9iB9rS2Eip3BhzMMW7NTJeLL1zlbdGCE pxPQ== X-Gm-Message-State: AOAM532RY7ljVzIkE9Rw2N+Fo6lPN0ScJUho3iYKcVcybgz2vkaXlbQA 2Kgjj1kqusEYLBAtcEt/DhCxGbBXcj0= X-Google-Smtp-Source: ABdhPJxNRFOcZaOXuw2RwsvyAMHETuLb333jpO8k8fWXf9yl+3phcLLGx0cPPpDFvk0BJJspVNqTGw== X-Received: by 2002:a0c:8d0f:: with SMTP id r15mr14593347qvb.1.1630974343927; Mon, 06 Sep 2021 17:25:43 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v5 3/4] spapr_numa.c: base FORM2 NUMA affinity support Date: Mon, 6 Sep 2021 21:25:26 -0300 Message-Id: <20210907002527.412013-4-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907002527.412013-1-danielhb413@gmail.com> References: <20210907002527.412013-1-danielhb413@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::f2c; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf2c.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1630974657232100001 Content-Type: text/plain; charset="utf-8" The main feature of FORM2 affinity support is the separation of NUMA distances from ibm,associativity information. This allows for a more flexible and straightforward NUMA distance assignment without relying on complex associations between several levels of NUMA via ibm,associativity matches. Another feature is its extensibility. This base support contains the facilities for NUMA distance assignment, but in the future more facilities will be added for latency, performance, bandwidth and so on. This patch implements the base FORM2 affinity support as follows: - the use of FORM2 associativity is indicated by using bit 2 of byte 5 of ibm,architecture-vec-5. A FORM2 aware guest can choose to use FORM1 or FORM2 affinity. Setting both forms will default to FORM2. We're not advertising FORM2 for pseries-6.1 and older machine versions to prevent guest visible changes in those; - call spapr_numa_associativity_reset() in do_client_architecture_support() if FORM2 is chosen. This will avoid re-initializing FORM1 artifacts that were already initialized in spapr_machine_reset(); - ibm,associativity-reference-points has a new semantic. Instead of being used to calculate distances via NUMA levels, it's now used to indicate the primary domain index in the ibm,associativity domain of each resource. In our case it's set to {0x4}, matching the position where we already place logical_domain_id; - two new RTAS DT artifacts are introduced: ibm,numa-lookup-index-table and ibm,numa-distance-table. The index table is used to list all the NUMA logical domains of the platform, in ascending order, and allows for spartial NUMA configurations (although QEMU ATM doesn't support that). ibm,numa-distance-table is an array that contains all the distances from the first NUMA node to all other nodes, then the second NUMA node distances to all other nodes and so on; - spapr_post_load changes: since we're adding a new NUMA affinity that isn't compatible with the existing one, migration must be handled accordingly because we can't be certain of whether the guest went through CAS in the source. The solution chosen is to initiate the NUMA associativity data in spapr_post_load() unconditionally. The worst case would be to write the DT twice if the guest is in pre-CAS stage. Otherwise, we're making sure that a FORM1 guest will have the spapr->numa_assoc_array initialized with the proper information based on user distance, something that we're not doing with FORM2. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 24 +++++++ hw/ppc/spapr_hcall.c | 10 +++ hw/ppc/spapr_numa.c | 135 +++++++++++++++++++++++++++++++++++- include/hw/ppc/spapr.h | 1 + include/hw/ppc/spapr_ovec.h | 1 + 5 files changed, 170 insertions(+), 1 deletion(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 8e1ff6cd10..8d98e3b08a 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -1789,6 +1789,22 @@ static int spapr_post_load(void *opaque, int version= _id) return err; } =20 + /* + * NUMA data init is made in CAS time. There is no reliable + * way of telling whether the guest already went through CAS + * in the source due to how spapr_ov5_cas_needed works: a + * FORM1 guest can be migrated with ov5_cas empty regardless + * of going through CAS first. + * + * One solution is to always call numa_associativity_reset. The + * downside is that a guest migrated before CAS will run + * numa_associativity_reset again when going through it, but + * at least we're making sure spapr->numa_assoc_array will be + * initialized and hotplug operations won't fail in both before + * and after CAS migration cases. + */ + spapr_numa_associativity_reset(spapr); + return err; } =20 @@ -2755,6 +2771,11 @@ static void spapr_machine_init(MachineState *machine) =20 spapr_ovec_set(spapr->ov5, OV5_FORM1_AFFINITY); =20 + /* Do not advertise FORM2 support for pseries-6.1 and older */ + if (!smc->pre_6_2_numa_affinity) { + spapr_ovec_set(spapr->ov5, OV5_FORM2_AFFINITY); + } + /* advertise support for dedicated HP event source to guests */ if (spapr->use_hotplug_event_source) { spapr_ovec_set(spapr->ov5, OV5_HP_EVT); @@ -4700,8 +4721,11 @@ DEFINE_SPAPR_MACHINE(6_2, "6.2", true); */ static void spapr_machine_6_1_class_options(MachineClass *mc) { + SpaprMachineClass *smc =3D SPAPR_MACHINE_CLASS(mc); + spapr_machine_6_2_class_options(mc); compat_props_add(mc->compat_props, hw_compat_6_1, hw_compat_6_1_len); + smc->pre_6_2_numa_affinity =3D true; } =20 DEFINE_SPAPR_MACHINE(6_1, "6.1", false); diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index 0e9a5b2e40..7efbe93f4b 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -11,6 +11,7 @@ #include "helper_regs.h" #include "hw/ppc/spapr.h" #include "hw/ppc/spapr_cpu_core.h" +#include "hw/ppc/spapr_numa.h" #include "mmu-hash64.h" #include "cpu-models.h" #include "trace.h" @@ -1197,6 +1198,15 @@ target_ulong do_client_architecture_support(PowerPCC= PU *cpu, spapr->cas_pre_isa3_guest =3D !spapr_ovec_test(ov1_guest, OV1_PPC_3_00= ); spapr_ovec_cleanup(ov1_guest); =20 + /* + * If the guest chooses FORM2 we need to reset the associativity + * information - it is being defaulted to FORM1 during + * spapr_machine_reset(). + */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + spapr_numa_associativity_reset(spapr); + } + /* * Ensure the guest asks for an interrupt mode we support; * otherwise terminate the boot. diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 84636cb86a..ca276e16cb 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -202,6 +202,24 @@ void spapr_numa_associativity_reset(SpaprMachineState = *spapr) spapr->numa_assoc_array[i][0] =3D cpu_to_be32(MAX_DISTANCE_REF_POI= NTS); spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] =3D cpu_to_be3= 2(i); =20 + /* + * For FORM2 affinity, zero all the non-used associativity + * domains (all domains but the last). This step is necessary + * because FORM2 will always be populated on top of FORM1 + * (via spapr_machine_reset()), which populates them. + * + * The kernel doesn't mind these non-used domains but zeroing + * them out makes it clearer to userspace that they are not + * being used. + */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + for (j =3D 1; j < MAX_DISTANCE_REF_POINTS; j++) { + spapr->numa_assoc_array[i][j] =3D 0; + } + + continue; + } + /* * Fill all associativity domains of non-zero NUMA nodes with * node_id. This is required because the default value (0) is @@ -236,7 +254,16 @@ void spapr_numa_associativity_reset(SpaprMachineState = *spapr) spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] =3D cpu_to_be3= 2(i); } =20 - spapr_numa_FORM1_affinity_init(spapr, machine); + /* + * We test for !FORM2 instead of testing for FORM1 because, + * as per spapr_ov5_cas_needed, setting FORM1 is not enough + * to get ov5_cas migrated, but setting FORM2 is. Since we're + * dealing with either FORM1 or FORM2, test for the option + * that is guaranteed to be set after a migration. + */ + if (!spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + spapr_numa_FORM1_affinity_init(spapr, machine); + } } =20 void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, @@ -313,6 +340,107 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachine= State *spapr, void *fdt, return ret; } =20 +static void spapr_numa_FORM2_write_rtas_tables(SpaprMachineState *spapr, + void *fdt, int rtas) +{ + MachineState *ms =3D MACHINE(spapr); + NodeInfo *numa_info =3D ms->numa_state->nodes; + int nb_numa_nodes =3D ms->numa_state->num_nodes; + int distance_table_entries =3D nb_numa_nodes * nb_numa_nodes; + g_autofree uint32_t *lookup_index_table =3D NULL; + g_autofree uint32_t *distance_table =3D NULL; + int src, dst, i, distance_table_size; + uint8_t *node_distances; + + /* + * ibm,numa-lookup-index-table: array with length and a + * list of NUMA ids present in the guest. + */ + lookup_index_table =3D g_new0(uint32_t, nb_numa_nodes + 1); + lookup_index_table[0] =3D cpu_to_be32(nb_numa_nodes); + + for (i =3D 0; i < nb_numa_nodes; i++) { + lookup_index_table[i + 1] =3D cpu_to_be32(i); + } + + _FDT(fdt_setprop(fdt, rtas, "ibm,numa-lookup-index-table", + lookup_index_table, + (nb_numa_nodes + 1) * sizeof(uint32_t))); + + /* + * ibm,numa-distance-table: contains all node distances. First + * element is the size of the table as uint32, followed up + * by all the uint8 distances from the first NUMA node, then all + * distances from the second NUMA node and so on. + * + * ibm,numa-lookup-index-table is used by guest to navigate this + * array because NUMA ids can be sparse (node 0 is the first, + * node 8 is the second ...). + */ + distance_table =3D g_new0(uint32_t, distance_table_entries + 1); + distance_table[0] =3D cpu_to_be32(distance_table_entries); + + node_distances =3D (uint8_t *)&distance_table[1]; + i =3D 0; + + for (src =3D 0; src < nb_numa_nodes; src++) { + for (dst =3D 0; dst < nb_numa_nodes; dst++) { + node_distances[i++] =3D numa_info[src].distance[dst]; + } + } + + distance_table_size =3D distance_table_entries * sizeof(uint8_t) + + sizeof(uint32_t); + _FDT(fdt_setprop(fdt, rtas, "ibm,numa-distance-table", + distance_table, distance_table_size)); +} + +/* + * This helper could be compressed in a single function with + * FORM1 logic since we're setting the same DT values, with the + * difference being a call to spapr_numa_FORM2_write_rtas_tables() + * in the end. The separation was made to avoid clogging FORM1 code + * which already has to deal with compat modes from previous + * QEMU machine types. + */ +static void spapr_numa_FORM2_write_rtas_dt(SpaprMachineState *spapr, + void *fdt, int rtas) +{ + MachineState *ms =3D MACHINE(spapr); + uint32_t number_nvgpus_nodes =3D spapr->gpu_numa_id - + spapr_numa_initial_nvgpu_numa_id(ms); + + /* + * In FORM2, ibm,associativity-reference-points will point to + * the element in the ibm,associativity array that contains the + * primary domain index. This value (in our case, the numa-id) is + * then used as an index to retrieve all other attributes of the + * node (distance, bandwidth, latency) via ibm,numa-lookup-index-table + * and other ibm,numa-*-table properties. + */ + uint32_t refpoints[] =3D { + cpu_to_be32(0x4), + }; + + uint32_t nr_refpoints =3D ARRAY_SIZE(refpoints); + uint32_t maxdomain =3D ms->numa_state->num_nodes + number_nvgpus_nodes; + uint32_t maxdomains[] =3D { + cpu_to_be32(4), + cpu_to_be32(maxdomain), + cpu_to_be32(maxdomain), + cpu_to_be32(maxdomain), + cpu_to_be32(maxdomain) + }; + + _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", + refpoints, nr_refpoints * sizeof(refpoints[0]))); + + _FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", + maxdomains, sizeof(maxdomains))); + + spapr_numa_FORM2_write_rtas_tables(spapr, fdt, rtas); +} + static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) { @@ -379,6 +507,11 @@ static void spapr_numa_FORM1_write_rtas_dt(SpaprMachin= eState *spapr, */ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rta= s) { + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + spapr_numa_FORM2_write_rtas_dt(spapr, fdt, rtas); + return; + } + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); } =20 diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 637652ad16..21b1cf9ebf 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -145,6 +145,7 @@ struct SpaprMachineClass { hwaddr rma_limit; /* clamp the RMA to this size */ bool pre_5_1_assoc_refpoints; bool pre_5_2_numa_associativity; + bool pre_6_2_numa_affinity; =20 bool (*phb_placement)(SpaprMachineState *spapr, uint32_t index, uint64_t *buid, hwaddr *pio, diff --git a/include/hw/ppc/spapr_ovec.h b/include/hw/ppc/spapr_ovec.h index 48b716a060..c3e8b98e7e 100644 --- a/include/hw/ppc/spapr_ovec.h +++ b/include/hw/ppc/spapr_ovec.h @@ -49,6 +49,7 @@ typedef struct SpaprOptionVector SpaprOptionVector; /* option vector 5 */ #define OV5_DRCONF_MEMORY OV_BIT(2, 2) #define OV5_FORM1_AFFINITY OV_BIT(5, 0) +#define OV5_FORM2_AFFINITY OV_BIT(5, 2) #define OV5_HP_EVT OV_BIT(6, 5) #define OV5_HPT_RESIZE OV_BIT(6, 7) #define OV5_DRMEM_V2 OV_BIT(22, 0) --=20 2.31.1 From nobody Wed May 8 01:48:36 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1630974445087258.69937856020863; Mon, 6 Sep 2021 17:27:25 -0700 (PDT) Received: from localhost ([::1]:36934 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNOxP-0006SH-Ux for importer@patchew.org; Mon, 06 Sep 2021 20:27:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59068) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNOvs-00044c-Tk; Mon, 06 Sep 2021 20:25:48 -0400 Received: from mail-qk1-x72b.google.com ([2607:f8b0:4864:20::72b]:34807) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNOvr-0005km-3u; Mon, 06 Sep 2021 20:25:48 -0400 Received: by mail-qk1-x72b.google.com with SMTP id a66so8444243qkc.1; Mon, 06 Sep 2021 17:25:46 -0700 (PDT) Received: from rekt.COMFAST ([179.247.136.122]) by smtp.gmail.com with ESMTPSA id 207sm7892056qkh.45.2021.09.06.17.25.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Sep 2021 17:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=e0OAJGK7QQ67l/s45XfqlUOoltbSPMh/cftld7lIQUY=; b=XL7F/brNfhGHKSiMqb3FqtJ6w6/tBGPZtz4ev06XurnVhmqYfsRK+ziZoIQYHpAhjE C8M+9n7Mm1R5KDKuBQz96YXIXZuoQeFBwH4I1Fd4nFyC38Q8lHzb1QucCip2D+3kCkzh fCsFUpxJCCEV0Z5CM6X73hIfZhDJro27rPtfk9HmflhJXCqh8tGpZ/QFVt68WsfS0JiI dDfZJmG5NVL4czHY4ZnCxKwqiCHdaNRlk12lXZSzjCJGgEotiSIiHeEPh/eeHzc5kl3y VdpgdjSoL6/1i7PH3V1KWXc4w0CCl+yIOyu4IHjnLKzM0EqnW4o1EUXav5hwN5UIL112 Gxbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e0OAJGK7QQ67l/s45XfqlUOoltbSPMh/cftld7lIQUY=; b=FD7iBFO9rUS1Nelarul/uYLzsbLpTgZ9aCIpM9ei9mI7Yl1s08qpCgG3wvkMRuLMt3 3/bomZrBh+Z/5ot4BRJDakLTJ3+Sl653i2EFnh5h52Fh8v02v4OF01bC1wt8UcZkz3Ux MoRoJzFG40ReWOPF5Ig5NzFxqlz1ifrr4NrxNYNLDiMgoKYhcF8WLRpHen7JXQevHT1U eDAMeb9gtdkpMrdZ6lXsGpZg7jF44kGm9PbqFbcNqRs6BDiA2dCxRCSYW9iU1L8zP3dM 00Uh/1OWp7X+2shfQ90sUeIHJt8YfHs/5i4C2fAcKt0meT8eaXVZfH4jDsticI+2E7ob srOQ== X-Gm-Message-State: AOAM5309YANfHJfS906WQLMBBZJ74gA/MdnUUoUpc8xV3vY/BN/lk19y XyzpN72ip/jKaxIkZVmihlRiTruwuTE= X-Google-Smtp-Source: ABdhPJwp8v8gvnyy57uIf9GK9eX3wdbvWVn1qKMsUxKUVKjsiZHnWl8yiv9HACfamkN5XRA5Js3xCQ== X-Received: by 2002:a05:620a:4541:: with SMTP id u1mr13569614qkp.106.1630974345630; Mon, 06 Sep 2021 17:25:45 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v5 4/4] spapr: move FORM1 verifications to do_client_architecture_support() Date: Mon, 6 Sep 2021 21:25:27 -0300 Message-Id: <20210907002527.412013-5-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907002527.412013-1-danielhb413@gmail.com> References: <20210907002527.412013-1-danielhb413@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::72b; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x72b.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1630974447408100001 Content-Type: text/plain; charset="utf-8" FORM2 NUMA affinity is prepared to deal with empty (memory/cpu less) NUMA nodes. This is used by the DAX KMEM driver to locate a PAPR SCM device that has a different latency than the original NUMA node from the regular memory. FORM2 is also enable to deal with asymmetric NUMA distances gracefully, something that our FORM1 implementation doesn't do. Move these FORM1 verifications to a new function and wait until after CAS, when we're sure that we're sticking with FORM1, to enforce them. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 33 ------------------------- hw/ppc/spapr_hcall.c | 6 +++++ hw/ppc/spapr_numa.c | 49 ++++++++++++++++++++++++++++++++----- include/hw/ppc/spapr_numa.h | 1 + 4 files changed, 50 insertions(+), 39 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 8d98e3b08a..c974c07fb8 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -2797,39 +2797,6 @@ static void spapr_machine_init(MachineState *machine) /* init CPUs */ spapr_init_cpus(spapr); =20 - /* - * check we don't have a memory-less/cpu-less NUMA node - * Firmware relies on the existing memory/cpu topology to provide the - * NUMA topology to the kernel. - * And the linux kernel needs to know the NUMA topology at start - * to be able to hotplug CPUs later. - */ - if (machine->numa_state->num_nodes) { - for (i =3D 0; i < machine->numa_state->num_nodes; ++i) { - /* check for memory-less node */ - if (machine->numa_state->nodes[i].node_mem =3D=3D 0) { - CPUState *cs; - int found =3D 0; - /* check for cpu-less node */ - CPU_FOREACH(cs) { - PowerPCCPU *cpu =3D POWERPC_CPU(cs); - if (cpu->node_id =3D=3D i) { - found =3D 1; - break; - } - } - /* memory-less and cpu-less node */ - if (!found) { - error_report( - "Memory-less/cpu-less nodes are not supported (node= %d)", - i); - exit(1); - } - } - } - - } - spapr->gpu_numa_id =3D spapr_numa_initial_nvgpu_numa_id(machine); =20 if ((!kvm_enabled() || kvmppc_has_cap_mmu_radix()) && diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index 7efbe93f4b..27ee713600 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -1202,9 +1202,15 @@ target_ulong do_client_architecture_support(PowerPCC= PU *cpu, * If the guest chooses FORM2 we need to reset the associativity * information - it is being defaulted to FORM1 during * spapr_machine_reset(). + * + * If we're sure that we'll be using FORM1, verify now if we have + * a configuration or condition that is not available for FORM1 + * (namely asymmetric NUMA topologies and empty NUMA nodes). */ if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { spapr_numa_associativity_reset(spapr); + } else { + spapr_numa_check_FORM1_constraints(MACHINE(spapr)); } =20 /* diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index ca276e16cb..0c57d03184 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -155,6 +155,49 @@ static void spapr_numa_define_associativity_domains(Sp= aprMachineState *spapr) =20 } =20 +void spapr_numa_check_FORM1_constraints(MachineState *machine) +{ + int i; + + if (!spapr_numa_is_symmetrical(machine)) { + error_report("Asymmetrical NUMA topologies aren't supported " + "in the pSeries machine"); + exit(EXIT_FAILURE); + } + + /* + * check we don't have a memory-less/cpu-less NUMA node + * Firmware relies on the existing memory/cpu topology to provide the + * NUMA topology to the kernel. + * And the linux kernel needs to know the NUMA topology at start + * to be able to hotplug CPUs later. + */ + if (machine->numa_state->num_nodes) { + for (i =3D 0; i < machine->numa_state->num_nodes; ++i) { + /* check for memory-less node */ + if (machine->numa_state->nodes[i].node_mem =3D=3D 0) { + CPUState *cs; + int found =3D 0; + /* check for cpu-less node */ + CPU_FOREACH(cs) { + PowerPCCPU *cpu =3D POWERPC_CPU(cs); + if (cpu->node_id =3D=3D i) { + found =3D 1; + break; + } + } + /* memory-less and cpu-less node */ + if (!found) { + error_report( + "Memory-less/cpu-less nodes are not supported (node= %d)", + i); + exit(EXIT_FAILURE); + } + } + } + } +} + /* * Set NUMA machine state data based on FORM1 affinity semantics. */ @@ -172,12 +215,6 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachin= eState *spapr, return; } =20 - if (!spapr_numa_is_symmetrical(machine)) { - error_report("Asymmetrical NUMA topologies aren't supported " - "in the pSeries machine"); - exit(EXIT_FAILURE); - } - spapr_numa_define_associativity_domains(spapr); } =20 diff --git a/include/hw/ppc/spapr_numa.h b/include/hw/ppc/spapr_numa.h index 0e457bba57..b5a19cb3f1 100644 --- a/include/hw/ppc/spapr_numa.h +++ b/include/hw/ppc/spapr_numa.h @@ -25,5 +25,6 @@ int spapr_numa_fixup_cpu_dt(SpaprMachineState *spapr, voi= d *fdt, int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *f= dt, int offset); unsigned int spapr_numa_initial_nvgpu_numa_id(MachineState *machine); +void spapr_numa_check_FORM1_constraints(MachineState *machine); =20 #endif /* HW_SPAPR_NUMA_H */ --=20 2.31.1