From nobody Fri Mar 29 07:18:04 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1565679680; cv=none; d=zoho.com; s=zohoarc; b=ap3Kbj9DX+rUnHTnPBA3O1p93kbtn9t0GPJos9ZaxISrJ4wt5cNwgC7rLGHzgLXHzHS1Gu1g9kDOokDOSFcEyKS+rZsxilJ+xZIRDO/y43scGY6LxGxbsCPY+VjbEcqNJvTPkIPfIf/oekVuO/gnDAijgHAvC9gIV455muo6gcM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565679680; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=YqzRe9dFtdV4i0L49oAqqkfSweI1gbRjPkzhbU7TmVY=; b=HhiEO6lGHgYjL1ooOPzu5lHqqNd7acJU283LrsGRmQLYtv5MT4ptKEIOiBT48UNjpX9bXeUxgTgpnn3Ird0+IMiQkXHT+urk1yRS3E1kzts2OGdynGnXLVEmifFD+Ig4MYuLrLqzK9qiXnCju8N8ZOBY3c+Dh0CU5t391pYJlpI= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1565679680955297.8778164118829; Tue, 13 Aug 2019 00:01:20 -0700 (PDT) Received: from localhost ([::1]:49804 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxQo4-0006tb-2I for importer@patchew.org; Tue, 13 Aug 2019 03:01:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57996) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxQmI-0004z1-QY for qemu-devel@nongnu.org; Tue, 13 Aug 2019 02:59:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hxQmH-0003M7-Cx for qemu-devel@nongnu.org; Tue, 13 Aug 2019 02:59:30 -0400 Received: from ozlabs.org ([2401:3900:2:1::2]:60877) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hxQmG-0003JG-KT; Tue, 13 Aug 2019 02:59:29 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 4673VM6VF1z9sNk; Tue, 13 Aug 2019 16:59:23 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1565679563; bh=9y0WaK5rfiuDHqIDJtUaKVJD71EgDPNCe5GIyMLdzw4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YI2O5IDDIXiVlXrwPPwCqPPMZMRHOHUjaCzswojpcIPDJQb9AExkaNkv/0NG6ZZgg ahpsH+w8uPqSXbrAB7i65RD/e2r3bEpG5eEeiPfCCUY/9aAWpAJ/O0Ds2cGEnZe4fT 080WLAzu5vPND77fPLzBhOiJiAzX8wxxCNt7EibI= From: David Gibson To: peter.maydell@linaro.org Date: Tue, 13 Aug 2019 16:59:19 +1000 Message-Id: <20190813065920.23203-2-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190813065920.23203-1-david@gibson.dropbear.id.au> References: <20190813065920.23203-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 Subject: [Qemu-devel] [PULL 1/2] spapr: Reset CAS & IRQ subsystem after devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: clg@kaod.org, David Gibson , qemu-ppc@nongnu.org, qemu-devel@nongnu.org, groug@kaod.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This fixes a nasty regression in qemu-4.1 for the 'pseries' machine, caused by the new "dual" interrupt controller model. Specifically, qemu can crash when used with KVM if a 'system_reset' is requested while there's active I/O in the guest. The problem is that in spapr_machine_reset() we: 1. Reset the CAS vector state spapr_ovec_cleanup(spapr->ov5_cas); 2. Reset all devices qemu_devices_reset() 3. Reset the irq subsystem spapr_irq_reset(); However (1) implicitly changes the interrupt delivery mode, because whether we're using XICS or XIVE depends on the CAS state. We don't properly initialize the new irq mode until (3) though - in particular setting up the KVM devices. During (2), we can temporarily drop the BQL allowing some irqs to be delivered which will go to an irq system that's not properly set up. Specifically, if the previous guest was in (KVM) XIVE mode, the CAS reset will put us back in XICS mode. kvm_kernel_irqchip() still returns true, because XIVE was using KVM, however XICs doesn't have its KVM components intialized and kernel_xics_fd =3D=3D -1. When the irq is delivered it goes via ics_kvm_set_irq() which assert()s that kernel_xics_fd !=3D -1. This change addresses the problem by delaying the CAS reset until after the devices reset. The device reset should quiesce all the devices so we won't get irqs delivered while we mess around with the IRQ. The CAS reset and irq re-initialize should also now be under the same BQL critical section so nothing else should be able to interrupt it either. We also move the spapr_irq_msi_reset() used in one of the legacy irq modes, since it logically makes sense at the same point as the spapr_irq_reset() (it's essentially an equivalent operation for older machine types). Since we don't need to switch between different interrupt controllers for those old machine types it shouldn't actually be broken in those cases though. Cc: C=C3=A9dric Le Goater Fixes: b2e22477 "spapr: add a 'reset' method to the sPAPR IRQ backend" Fixes: 13db0cd9 "spapr: introduce a new sPAPR IRQ backend supporting XIVE and XICS" Signed-off-by: David Gibson --- hw/ppc/spapr.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 821f0d4a49..12ed4b065c 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -1726,6 +1726,18 @@ static void spapr_machine_reset(MachineState *machin= e) spapr_setup_hpt_and_vrma(spapr); } =20 + /* + * NVLink2-connected GPU RAM needs to be placed on a separate NUMA nod= e. + * We assign a new numa ID per GPU in spapr_pci_collect_nvgpu() which = is + * called from vPHB reset handler so we initialize the counter here. + * If no NUMA is configured from the QEMU side, we start from 1 as GPU= RAM + * must be equally distant from any other node. + * The final value of spapr->gpu_numa_id is going to be written to + * max-associativity-domains in spapr_build_fdt(). + */ + spapr->gpu_numa_id =3D MAX(1, nb_numa_nodes); + qemu_devices_reset(); + /* * If this reset wasn't generated by CAS, we should reset our * negotiated options and start from scratch @@ -1741,18 +1753,6 @@ static void spapr_machine_reset(MachineState *machin= e) spapr_irq_msi_reset(spapr); } =20 - /* - * NVLink2-connected GPU RAM needs to be placed on a separate NUMA nod= e. - * We assign a new numa ID per GPU in spapr_pci_collect_nvgpu() which = is - * called from vPHB reset handler so we initialize the counter here. - * If no NUMA is configured from the QEMU side, we start from 1 as GPU= RAM - * must be equally distant from any other node. - * The final value of spapr->gpu_numa_id is going to be written to - * max-associativity-domains in spapr_build_fdt(). - */ - spapr->gpu_numa_id =3D MAX(1, nb_numa_nodes); - qemu_devices_reset(); - /* * This is fixing some of the default configuration of the XIVE * devices. To be called after the reset of the machine devices. --=20 2.21.0 From nobody Fri Mar 29 07:18:04 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1565679671; cv=none; d=zoho.com; s=zohoarc; b=EEdGwO4TOcnrM5KguWYms5wojIXJozfsY90qqx5SiYbaneSB7YMaN8n2AZLFzJYf9SPpq9KwoI+BI8IUl+BKS4ml78IkipHLiMZcLc40pbEl1CMTPpZLGW8cTMRIag+A7x0tx1igMyXW2Y5IcF3EaevdNd5Qq89g6e2hnpm8bfY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565679671; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=HsCUE+/RnlC/Qcq9iHauHUfl2roWxw2PE26rPGty+U0=; b=T1amcLy/h3nKVk70ICB6fGWvQOYNTrm80G9Dcq6zkf1vhx5IU1V5rRWBTDJFtUtVJnQsvuytqhwYwnsiGcc4X0Aptj9lZpiS0QR2a79fPomFrdK+rC4yWV522eqgP6uIa5Fucb0M+Pc8OrJX4liu02ta9ZCO6fO6iQ34JDl+LE0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1565679670680543.3164402010126; Tue, 13 Aug 2019 00:01:10 -0700 (PDT) Received: from localhost ([::1]:49802 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxQnr-0006UZ-OJ for importer@patchew.org; Tue, 13 Aug 2019 03:01:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57977) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hxQmH-0004yv-LS for qemu-devel@nongnu.org; Tue, 13 Aug 2019 02:59:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hxQmG-0003Lk-Gf for qemu-devel@nongnu.org; Tue, 13 Aug 2019 02:59:29 -0400 Received: from ozlabs.org ([203.11.71.1]:41121) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hxQmF-0003JA-79; Tue, 13 Aug 2019 02:59:28 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 4673VN02RGz9sNy; Tue, 13 Aug 2019 16:59:23 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1565679564; bh=emrIgzu+XTunHT9T/aAO9YQzehEDIDPRz+sDuCSkRv0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gBo+s5apgl0rcWHCsC+SZWw6CnnolcfEcWweMB1VrIvP+KGXU4iGx+cd7lnMopWZu FcWxCvbJTB/coFyx2pZMHGyW3F3offAYHAUge6G2ONqa8Tl6vej82HL65UZLtpEfQa iD9M7W8Sru3z/RyJme0rRrTHa4+WURGAc3DNXiTU= From: David Gibson To: peter.maydell@linaro.org Date: Tue, 13 Aug 2019 16:59:20 +1000 Message-Id: <20190813065920.23203-3-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190813065920.23203-1-david@gibson.dropbear.id.au> References: <20190813065920.23203-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 203.11.71.1 Subject: [Qemu-devel] [PULL 2/2] spapr/xive: Fix migration of hot-plugged CPUs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: clg@kaod.org, David Gibson , qemu-ppc@nongnu.org, qemu-devel@nongnu.org, groug@kaod.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: C=C3=A9dric Le Goater The migration sequence of a guest using the XIVE exploitation mode relies on the fact that the states of all devices are restored before the machine is. This is not true for hot-plug devices such as CPUs which state come after the machine. This breaks migration because the thread interrupt context registers are not correctly set. Fix migration of hotplugged CPUs by restoring their context in the 'post_load' handler of the XiveTCTX model. Fixes: 277dd3d7712a ("spapr/xive: add migration support for KVM") Signed-off-by: C=C3=A9dric Le Goater Message-Id: <20190813064853.29310-1-clg@kaod.org> Signed-off-by: David Gibson --- hw/intc/spapr_xive_kvm.c | 19 +++++++++++++++++-- hw/intc/xive.c | 21 ++++++++++++++++++++- include/hw/ppc/xive.h | 1 + 3 files changed, 38 insertions(+), 3 deletions(-) diff --git a/hw/intc/spapr_xive_kvm.c b/hw/intc/spapr_xive_kvm.c index 3bf8e7a20e..8898615c69 100644 --- a/hw/intc/spapr_xive_kvm.c +++ b/hw/intc/spapr_xive_kvm.c @@ -72,11 +72,17 @@ static void kvm_cpu_disable_all(void) * XIVE Thread Interrupt Management context (KVM) */ =20 -static void kvmppc_xive_cpu_set_state(XiveTCTX *tctx, Error **errp) +void kvmppc_xive_cpu_set_state(XiveTCTX *tctx, Error **errp) { + SpaprXive *xive =3D SPAPR_MACHINE(qdev_get_machine())->xive; uint64_t state[2]; int ret; =20 + /* The KVM XIVE device is not in use yet */ + if (xive->fd =3D=3D -1) { + return; + } + /* word0 and word1 of the OS ring. */ state[0] =3D *((uint64_t *) &tctx->regs[TM_QW1_OS]); =20 @@ -655,7 +661,16 @@ int kvmppc_xive_post_load(SpaprXive *xive, int version= _id) } } =20 - /* Restore the thread interrupt contexts */ + /* + * Restore the thread interrupt contexts of initial CPUs. + * + * The context of hotplugged CPUs is restored later, by the + * 'post_load' handler of the XiveTCTX model because they are not + * available at the time the SpaprXive 'post_load' method is + * called. We can not restore the context of all CPUs in the + * 'post_load' handler of XiveTCTX because the machine is not + * necessarily connected to the KVM device at that time. + */ CPU_FOREACH(cs) { PowerPCCPU *cpu =3D POWERPC_CPU(cs); =20 diff --git a/hw/intc/xive.c b/hw/intc/xive.c index cf77bdb7d3..da148e9f6f 100644 --- a/hw/intc/xive.c +++ b/hw/intc/xive.c @@ -615,12 +615,31 @@ static int vmstate_xive_tctx_pre_save(void *opaque) return 0; } =20 +static int vmstate_xive_tctx_post_load(void *opaque, int version_id) +{ + Error *local_err =3D NULL; + + if (kvm_irqchip_in_kernel()) { + /* + * Required for hotplugged CPU, for which the state comes + * after all states of the machine. + */ + kvmppc_xive_cpu_set_state(XIVE_TCTX(opaque), &local_err); + if (local_err) { + error_report_err(local_err); + return -1; + } + } + + return 0; +} + static const VMStateDescription vmstate_xive_tctx =3D { .name =3D TYPE_XIVE_TCTX, .version_id =3D 1, .minimum_version_id =3D 1, .pre_save =3D vmstate_xive_tctx_pre_save, - .post_load =3D NULL, /* handled by the sPAPRxive model */ + .post_load =3D vmstate_xive_tctx_post_load, .fields =3D (VMStateField[]) { VMSTATE_BUFFER(regs, XiveTCTX), VMSTATE_END_OF_LIST() diff --git a/include/hw/ppc/xive.h b/include/hw/ppc/xive.h index 55c53c7417..736335174a 100644 --- a/include/hw/ppc/xive.h +++ b/include/hw/ppc/xive.h @@ -438,5 +438,6 @@ void kvmppc_xive_source_set_irq(void *opaque, int srcno= , int val); void kvmppc_xive_cpu_connect(XiveTCTX *tctx, Error **errp); void kvmppc_xive_cpu_synchronize_state(XiveTCTX *tctx, Error **errp); void kvmppc_xive_cpu_get_state(XiveTCTX *tctx, Error **errp); +void kvmppc_xive_cpu_set_state(XiveTCTX *tctx, Error **errp); =20 #endif /* PPC_XIVE_H */ --=20 2.21.0