From nobody Sat May 11 04:51:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1597417515; cv=none; d=zohomail.com; s=zohoarc; b=H255uFcksDjZLqA0b1UqKwm7U9Y5ctSEe+JWL9wp4Gk+yGS+puFbZZ6FK4fnyBUM2hxTzE9Y7Ft/7kBUl1vT+p9u3pT1GuAru/gnlKfvIHLA+2wghcCZHs97FQe9WBq2GQiypFdc1RDdlfxd7IJ3Itp8b1639Msr5gZDlvXoSNI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1597417515; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=WH6R+4rSY4u3LmeeTD82r1Wn7ei6algO6PAfKhNyHjQ=; b=mbnTA0a9h9bbjN/ztIeUL1YpLJRAd//Qi0i6K255rTdMYlvgFiRfY7xdz6JpTBA6dXtvRtpt01R3RcmhX6/pUpwftjFDlVIg5Vu59Z27OSwyQlc0wHbM1tklAjwxBDciSm8FxSPxgbXMEk5Cy7u0CePIoPahJ/K5NedGorOCwjU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1597417515377247.04752233314218; Fri, 14 Aug 2020 08:05:15 -0700 (PDT) Received: from localhost ([::1]:38792 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k6bGb-00026L-2u for importer@patchew.org; Fri, 14 Aug 2020 11:05:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45974) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k6bFg-00015I-Hp; Fri, 14 Aug 2020 11:04:16 -0400 Received: from smtpout1.mo804.mail-out.ovh.net ([79.137.123.220]:59065) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k6bFe-0004ic-Fh; Fri, 14 Aug 2020 11:04:16 -0400 Received: from mxplan5.mail.ovh.net (unknown [10.108.20.149]) by mo804.mail-out.ovh.net (Postfix) with ESMTPS id 294A1572BC00; Fri, 14 Aug 2020 17:04:01 +0200 (CEST) Received: from kaod.org (37.59.142.97) by DAG4EX1.mxp5.local (172.16.2.31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Fri, 14 Aug 2020 17:04:00 +0200 Authentication-Results: garm.ovh; auth=pass (GARM-97G00115a63b35-ec6f-4d0c-9884-f6fc30d70b31, 373B2072066AFC5654FE77EA66C144DA3E99FD02) smtp.auth=clg@kaod.org From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: David Gibson Subject: [PATCH] spapr/xive: Allocate IPIs from the vCPU contexts Date: Fri, 14 Aug 2020 17:03:58 +0200 Message-ID: <20200814150358.1682513-1-clg@kaod.org> X-Mailer: git-send-email 2.25.4 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [37.59.142.97] X-ClientProxiedBy: DAG8EX2.mxp5.local (172.16.2.72) To DAG4EX1.mxp5.local (172.16.2.31) X-Ovh-Tracer-GUID: bb78967b-5f56-4fd0-87eb-b0d83842c9a8 X-Ovh-Tracer-Id: 12096950076797389606 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: -100 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeduiedrleejgdejkecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfqggfjpdevjffgvefmvefgnecuuegrihhlohhuthemucehtddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvffufffkofggtgfgihesthekredtredtjeenucfhrhhomhepveorughrihgtucfnvgcuifhorghtvghruceotghlgheskhgrohgurdhorhhgqeenucggtffrrghtthgvrhhnpeefvdeutddvieekkeeuhfekudejjefggffghfetgfelgfevveefgefhvdegtdelveenucfkpheptddrtddrtddrtddpfeejrdehledrudegvddrleejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmohguvgepshhmthhpqdhouhhtpdhhvghlohepmhigphhlrghnhedrmhgrihhlrdhovhhhrdhnvghtpdhinhgvtheptddrtddrtddrtddpmhgrihhlfhhrohhmpegtlhhgsehkrghougdrohhrghdprhgtphhtthhopegtlhhgsehkrghougdrohhrgh Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=79.137.123.220; envelope-from=clg@kaod.org; helo=smtpout1.mo804.mail-out.ovh.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/14 11:04:02 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , qemu-ppc@nongnu.org, Greg Kurz , Gustavo Romero Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" When QEMU switches to the XIVE interrupt mode, it performs a kvmppc_xive_source_reset() which creates all the guest interrupts at the level of the KVM device. These interrupts are backed by real HW interrupts from the IPI interrupt pool of the XIVE controller. Currently, this is done from the QEMU main thread, which results in allocating all interrupts from the chip on which QEMU is running. IPIs are not distributed across the system and the load is not well balanced across the interrupt controllers. Change the vCPU IPI allocation to run from the vCPU context in order to allocate the associated XIVE IPI interrupt on the chip on which the vCPU is running. This gives a chance to a better distribution of the IPIs when the guest has a lot of vCPUs. When the vCPUs are pinned, it makes the IPI local to the chip of the vCPU which reduces rerouting between interrupt controllers and gives better performance. This is only possible for running vCPUs. The IPIs of hot plugable vCPUs will still be allocated in the context of the QEMU main thread. Device interrupts are treated the same. To improve placement, we would need some information on the chip owning the virtual source or HW source in case of passthrough. This requires changes in PAPR. Signed-off-by: C=C3=A9dric Le Goater --- hw/intc/spapr_xive_kvm.c | 50 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/hw/intc/spapr_xive_kvm.c b/hw/intc/spapr_xive_kvm.c index c6958f2da218..553fd7fd8f56 100644 --- a/hw/intc/spapr_xive_kvm.c +++ b/hw/intc/spapr_xive_kvm.c @@ -223,6 +223,47 @@ void kvmppc_xive_sync_source(SpaprXive *xive, uint32_t= lisn, Error **errp) NULL, true, errp); } =20 +/* + * Allocate the IPIs from the vCPU context. This will allocate the + * XIVE IPI interrupt on the chip on which the vCPU is running. This + * gives a better distribution of IPIs when the guest has a lot of + * vCPUs. When the vCPU are pinned, the IPIs are local which reduces + * rerouting between interrupt controllers and gives better + * performance. + */ +typedef struct { + SpaprXive *xive; + int ipi; + Error *err; + int rc; +} XiveInitIPI; + +static void kvmppc_xive_reset_ipi_on_cpu(CPUState *cs, run_on_cpu_data arg) +{ + XiveInitIPI *s =3D arg.host_ptr; + uint64_t state =3D 0; + + s->rc =3D kvm_device_access(s->xive->fd, KVM_DEV_XIVE_GRP_SOURCE, s->i= pi, + &state, true, &s->err); +} + +static int kvmppc_xive_reset_ipi(SpaprXive *xive, int ipi, Error **errp) +{ + PowerPCCPU *cpu =3D spapr_find_cpu(ipi); + XiveInitIPI s =3D { + .xive =3D xive, + .ipi =3D ipi, + .err =3D NULL, + .rc =3D 0, + }; + + run_on_cpu(CPU(cpu), kvmppc_xive_reset_ipi_on_cpu, RUN_ON_CPU_HOST_PTR= (&s)); + if (s.err) { + error_propagate(errp, s.err); + } + return s.rc; +} + /* * At reset, the interrupt sources are simply created and MASKED. We * only need to inform the KVM XIVE device about their type: LSI or @@ -230,11 +271,20 @@ void kvmppc_xive_sync_source(SpaprXive *xive, uint32_= t lisn, Error **errp) */ int kvmppc_xive_source_reset_one(XiveSource *xsrc, int srcno, Error **errp) { + MachineState *machine =3D MACHINE(qdev_get_machine()); SpaprXive *xive =3D SPAPR_XIVE(xsrc->xive); uint64_t state =3D 0; =20 assert(xive->fd !=3D -1); =20 + /* + * IPIs are special. Allocate the IPIs from the vCPU context for + * those running. Hotplugged CPUs will the QEMU context. + */ + if (srcno < machine->smp.cpus) { + return kvmppc_xive_reset_ipi(xive, srcno, errp); + } + if (xive_source_irq_is_lsi(xsrc, srcno)) { state |=3D KVM_XIVE_LEVEL_SENSITIVE; if (xsrc->status[srcno] & XIVE_STATUS_ASSERTED) { --=20 2.25.4