From nobody Thu May 16 08:36:39 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1497023678232257.39452759690016; Fri, 9 Jun 2017 08:54:38 -0700 (PDT) Received: from localhost ([::1]:55456 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMFA-0004Aq-Ts for importer@patchew.org; Fri, 09 Jun 2017 11:54:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44888) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMDg-00037u-A2 for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dJMDe-00050u-JJ for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46728) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dJMDa-0004vv-Lx; Fri, 09 Jun 2017 11:52:58 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6C5AD20B1C; Fri, 9 Jun 2017 15:52:57 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-222.ams2.redhat.com [10.36.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id A6032955A2; Fri, 9 Jun 2017 15:52:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 6C5AD20B1C Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=eric.auger@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 6C5AD20B1C From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Fri, 9 Jun 2017 17:52:30 +0200 Message-Id: <1497023553-18411-2-git-send-email-eric.auger@redhat.com> In-Reply-To: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> References: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 09 Jun 2017 15:52:57 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 1/4] kvm-all: Pass an error object to kvm_device_access X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wei@redhat.com, drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, zhaoshenglong@huawei.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" In some circumstances, we don't want to abort if the kvm_device_access fails. This will be the case during ITS migration, in case the ITS table save/restore fails because the guest did not program the vITS correctly. So let's pass an error object to the function and return the ioctl value. New callers will be able to make a decision upon this returned value. Existing callers pass &error_abort which will cause the function to abort on failure. Signed-off-by: Eric Auger Reviewed-by: Juan Quintela --- hw/intc/arm_gic_kvm.c | 9 +++++---- hw/intc/arm_gicv3_its_kvm.c | 2 +- hw/intc/arm_gicv3_kvm.c | 14 +++++++------- include/sysemu/kvm.h | 11 +++++++---- kvm-all.c | 13 +++++++------ 5 files changed, 27 insertions(+), 22 deletions(-) diff --git a/hw/intc/arm_gic_kvm.c b/hw/intc/arm_gic_kvm.c index af5cd36..ae095d0 100644 --- a/hw/intc/arm_gic_kvm.c +++ b/hw/intc/arm_gic_kvm.c @@ -100,14 +100,14 @@ static void kvm_gicd_access(GICState *s, int offset, = int cpu, uint32_t *val, bool write) { kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_DIST_REGS, - KVM_VGIC_ATTR(offset, cpu), val, write); + KVM_VGIC_ATTR(offset, cpu), val, write, &error_abort= ); } =20 static void kvm_gicc_access(GICState *s, int offset, int cpu, uint32_t *val, bool write) { kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_REGS, - KVM_VGIC_ATTR(offset, cpu), val, write); + KVM_VGIC_ATTR(offset, cpu), val, write, &error_abort= ); } =20 #define for_each_irq_reg(_ctr, _max_irq, _field_width) \ @@ -538,13 +538,14 @@ static void kvm_arm_gic_realize(DeviceState *dev, Err= or **errp) if (kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS,= 0)) { uint32_t numirqs =3D s->num_irq; kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, 0, - &numirqs, true); + &numirqs, true, &error_abort); } /* Tell the kernel to complete VGIC initialization now */ if (kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, KVM_DEV_ARM_VGIC_CTRL_INIT)) { kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, - KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true); + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true, + &error_abort); } } else if (ret !=3D -ENODEV && ret !=3D -ENOTSUP) { error_setg_errno(errp, -ret, "error creating in-kernel VGIC"); diff --git a/hw/intc/arm_gicv3_its_kvm.c b/hw/intc/arm_gicv3_its_kvm.c index a0441d6..340c2b0 100644 --- a/hw/intc/arm_gicv3_its_kvm.c +++ b/hw/intc/arm_gicv3_its_kvm.c @@ -78,7 +78,7 @@ static void kvm_arm_its_realize(DeviceState *dev, Error *= *errp) =20 /* explicit init of the ITS */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, - KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true); + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true, &error_abort= ); =20 /* register the base address */ kvm_arm_register_device(&s->iomem_its_cntrl, -1, KVM_DEV_ARM_VGIC_GRP_= ADDR, diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 4ee2baa..b70ee27 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -93,7 +93,7 @@ static inline void kvm_gicd_access(GICv3State *s, int off= set, { kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_DIST_REGS, KVM_VGIC_ATTR(offset, 0), - val, write); + val, write, &error_abort); } =20 static inline void kvm_gicr_access(GICv3State *s, int offset, int cpu, @@ -101,7 +101,7 @@ static inline void kvm_gicr_access(GICv3State *s, int o= ffset, int cpu, { kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_REDIST_REGS, KVM_VGIC_ATTR(offset, s->cpu[cpu].gicr_typer), - val, write); + val, write, &error_abort); } =20 static inline void kvm_gicc_access(GICv3State *s, uint64_t reg, int cpu, @@ -109,7 +109,7 @@ static inline void kvm_gicc_access(GICv3State *s, uint6= 4_t reg, int cpu, { kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, KVM_VGIC_ATTR(reg, s->cpu[cpu].gicr_typer), - val, write); + val, write, &error_abort); } =20 static inline void kvm_gic_line_level_access(GICv3State *s, int irq, int c= pu, @@ -119,7 +119,7 @@ static inline void kvm_gic_line_level_access(GICv3State= *s, int irq, int cpu, KVM_VGIC_ATTR(irq, s->cpu[cpu].gicr_typer) | (VGIC_LEVEL_INFO_LINE_LEVEL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT), - val, write); + val, write, &error_abort); } =20 /* Loop through each distributor IRQ related register; since bits @@ -630,7 +630,7 @@ static void arm_gicv3_icc_reset(CPUARMState *env, const= ARMCPRegInfo *ri) /* Initialize to actual HW supported configuration */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, KVM_VGIC_ATTR(ICC_CTLR_EL1, cpu->mp_affinity), - &c->icc_ctlr_el1[GICV3_NS], false); + &c->icc_ctlr_el1[GICV3_NS], false, &error_abort); =20 c->icc_ctlr_el1[GICV3_S] =3D c->icc_ctlr_el1[GICV3_NS]; } @@ -717,11 +717,11 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, E= rror **errp) } =20 kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_NR_IRQS, - 0, &s->num_irq, true); + 0, &s->num_irq, true, &error_abort); =20 /* Tell the kernel to complete VGIC initialization now */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, - KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true); + KVM_DEV_ARM_VGIC_CTRL_INIT, NULL, true, &error_abort= ); =20 kvm_arm_register_device(&s->iomem_dist, -1, KVM_DEV_ARM_VGIC_GRP_ADDR, KVM_VGIC_V3_ADDR_TYPE_DIST, s->dev_fd); diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index a45c145..1e91613 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -294,12 +294,15 @@ int kvm_device_check_attr(int fd, uint32_t group, uin= t64_t attr); * @attr: the attribute of that group to set or get * @val: pointer to a storage area for the value * @write: true for set and false for get operation + * @errp: error object handle * - * This function is not allowed to fail. Use kvm_device_check_attr() - * in order to check for the availability of optional attributes. + * Returns: 0 on success + * < 0 on error + * Use kvm_device_check_attr() in order to check for the availability + * of optional attributes. */ -void kvm_device_access(int fd, int group, uint64_t attr, - void *val, bool write); +int kvm_device_access(int fd, int group, uint64_t attr, + void *val, bool write, Error **errp); =20 /** * kvm_create_device - create a KVM device for the device control API diff --git a/kvm-all.c b/kvm-all.c index 494b925..2f9d605 100644 --- a/kvm-all.c +++ b/kvm-all.c @@ -23,6 +23,7 @@ #include "qemu/option.h" #include "qemu/config-file.h" #include "qemu/error-report.h" +#include "qapi/error.h" #include "hw/hw.h" #include "hw/pci/msi.h" #include "hw/pci/msix.h" @@ -2214,8 +2215,8 @@ int kvm_device_check_attr(int dev_fd, uint32_t group,= uint64_t attr) return kvm_device_ioctl(dev_fd, KVM_HAS_DEVICE_ATTR, &attribute) ? 0 := 1; } =20 -void kvm_device_access(int fd, int group, uint64_t attr, - void *val, bool write) +int kvm_device_access(int fd, int group, uint64_t attr, + void *val, bool write, Error **errp) { struct kvm_device_attr kvmattr; int err; @@ -2229,11 +2230,11 @@ void kvm_device_access(int fd, int group, uint64_t = attr, write ? KVM_SET_DEVICE_ATTR : KVM_GET_DEVICE_AT= TR, &kvmattr); if (err < 0) { - error_report("KVM_%s_DEVICE_ATTR failed: %s", - write ? "SET" : "GET", strerror(-err)); - error_printf("Group %d attr 0x%016" PRIx64 "\n", group, attr); - abort(); + error_setg_errno(errp, -err, + "KVM_%s_DEVICE_ATTR failed: Group %d attr 0x%016"= PRIx64, + write ? "SET" : "GET", group, attr); } + return err; } =20 /* Return 1 on success, 0 on failure */ --=20 2.5.5 From nobody Thu May 16 08:36:39 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1497023784737681.6365126719709; Fri, 9 Jun 2017 08:56:24 -0700 (PDT) Received: from localhost ([::1]:55467 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMGt-0005PT-7r for importer@patchew.org; Fri, 09 Jun 2017 11:56:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45002) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMDu-0003Jk-Is for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dJMDt-0005FR-7Y for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53294) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dJMDn-00059F-Da; Fri, 09 Jun 2017 11:53:11 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4553C334587; Fri, 9 Jun 2017 15:53:10 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-222.ams2.redhat.com [10.36.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id BEB50955A2; Fri, 9 Jun 2017 15:52:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 4553C334587 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=eric.auger@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 4553C334587 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Fri, 9 Jun 2017 17:52:31 +0200 Message-Id: <1497023553-18411-3-git-send-email-eric.auger@redhat.com> In-Reply-To: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> References: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Fri, 09 Jun 2017 15:53:10 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 2/4] hw/intc/arm_gicv3_its: Implement state save/restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wei@redhat.com, drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, zhaoshenglong@huawei.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We need to handle both registers and ITS tables. While register handling is standard, ITS table handling is more challenging since the kernel API is devised so that the tables are flushed into guest RAM and not in vmstate buffers. Flushing the ITS tables on device pre_save() is too late since the guest RAM is already saved at this point. Table flushing needs to happen when we are sure the vcpus are stopped and before the last dirty page saving. The right point is RUN_STATE_FINISH_MIGRATE but sometimes the VM gets stopped before migration launch so let's simply flush the tables each time the VM gets stopped. For regular ITS registers we just can use vmstate pre_save() and post_load() callbacks. Signed-off-by: Eric Auger --- v5 -> v6: - use &s->ctlr directly when writing GITS_CTLR - don't abort on -EFAULT - s->iidr reset in gicv3_its_common_reset --- hw/intc/arm_gicv3_its_common.c | 10 ++++ hw/intc/arm_gicv3_its_kvm.c | 105 +++++++++++++++++++++++++++++= ++++ include/hw/intc/arm_gicv3_its_common.h | 8 +++ 3 files changed, 123 insertions(+) diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c index 9d67c5c..696c11c 100644 --- a/hw/intc/arm_gicv3_its_common.c +++ b/hw/intc/arm_gicv3_its_common.c @@ -49,6 +49,15 @@ static const VMStateDescription vmstate_its =3D { .pre_save =3D gicv3_its_pre_save, .post_load =3D gicv3_its_post_load, .unmigratable =3D true, + .fields =3D (VMStateField[]) { + VMSTATE_UINT32(ctlr, GICv3ITSState), + VMSTATE_UINT32(iidr, GICv3ITSState), + VMSTATE_UINT64(cbaser, GICv3ITSState), + VMSTATE_UINT64(cwriter, GICv3ITSState), + VMSTATE_UINT64(creadr, GICv3ITSState), + VMSTATE_UINT64_ARRAY(baser, GICv3ITSState, 8), + VMSTATE_END_OF_LIST() + }, }; =20 static MemTxResult gicv3_its_trans_read(void *opaque, hwaddr offset, @@ -118,6 +127,7 @@ static void gicv3_its_common_reset(DeviceState *dev) s->cbaser =3D 0; s->cwriter =3D 0; s->creadr =3D 0; + s->iidr =3D 0; memset(&s->baser, 0, sizeof(s->baser)); =20 gicv3_its_post_load(s, 0); diff --git a/hw/intc/arm_gicv3_its_kvm.c b/hw/intc/arm_gicv3_its_kvm.c index 340c2b0..4cd8f5f 100644 --- a/hw/intc/arm_gicv3_its_kvm.c +++ b/hw/intc/arm_gicv3_its_kvm.c @@ -53,6 +53,33 @@ static int kvm_its_send_msi(GICv3ITSState *s, uint32_t v= alue, uint16_t devid) return kvm_vm_ioctl(kvm_state, KVM_SIGNAL_MSI, &msi); } =20 +/** + * vm_change_state_handler - VM change state callback aiming at flushing + * ITS tables into guest RAM + * + * The tables get flushed to guest RAM whenever the VM gets stopped. + */ +static void vm_change_state_handler(void *opaque, int running, + RunState state) +{ + GICv3ITSState *s =3D (GICv3ITSState *)opaque; + Error *err =3D NULL; + int ret; + + if (running) { + return; + } + + ret =3D kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_ITS_SAVE_TABLES, NULL, true, &err); + if (err) { + error_report_err(err); + } + if (ret < 0 && ret !=3D -EFAULT) { + abort(); + } +} + static void kvm_arm_its_realize(DeviceState *dev, Error **errp) { GICv3ITSState *s =3D ARM_GICV3_ITS_COMMON(dev); @@ -89,6 +116,8 @@ static void kvm_arm_its_realize(DeviceState *dev, Error = **errp) kvm_msi_use_devid =3D true; kvm_gsi_direct_mapping =3D false; kvm_msi_via_irqfd_allowed =3D kvm_irqfds_enabled(); + + qemu_add_vm_change_state_handler(vm_change_state_handler, s); } =20 static void kvm_arm_its_init(Object *obj) @@ -102,6 +131,80 @@ static void kvm_arm_its_init(Object *obj) &error_abort); } =20 +/** + * kvm_arm_its_pre_save - handles the saving of ITS registers. + * ITS tables are flushed into guest RAM separately and earlier, + * through the VM change state handler, since at the moment pre_save() + * is called, the guest RAM has already been saved. + */ +static void kvm_arm_its_pre_save(GICv3ITSState *s) +{ + int i; + + for (i =3D 0; i < 8; i++) { + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_BASER + i * 8, &s->baser[i], false, + &error_abort); + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR, &s->ctlr, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CBASER, &s->cbaser, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CREADR, &s->creadr, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CWRITER, &s->cwriter, false, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_IIDR, &s->iidr, false, &error_abort); +} + +/** + * kvm_arm_its_post_load - Restore both the ITS registers and tables + */ +static void kvm_arm_its_post_load(GICv3ITSState *s) +{ + int i; + + if (!s->iidr) { + return; + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_IIDR, &s->iidr, true, &error_abort); + + /* + * must be written before GITS_CREADR since GITS_CBASER write + * access resets GITS_CREADR. + */ + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CBASER, &s->cbaser, true, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CREADR, &s->creadr, true, &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CWRITER, &s->cwriter, true, &error_abort); + + + for (i =3D 0; i < 8; i++) { + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_BASER + i * 8, &s->baser[i], true, + &error_abort); + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_ITS_RESTORE_TABLES, NULL, true, + &error_abort); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR, &s->ctlr, true, &error_abort); +} + static void kvm_arm_its_class_init(ObjectClass *klass, void *data) { DeviceClass *dc =3D DEVICE_CLASS(klass); @@ -109,6 +212,8 @@ static void kvm_arm_its_class_init(ObjectClass *klass, = void *data) =20 dc->realize =3D kvm_arm_its_realize; icc->send_msi =3D kvm_its_send_msi; + icc->pre_save =3D kvm_arm_its_pre_save; + icc->post_load =3D kvm_arm_its_post_load; } =20 static const TypeInfo kvm_arm_its_info =3D { diff --git a/include/hw/intc/arm_gicv3_its_common.h b/include/hw/intc/arm_g= icv3_its_common.h index 1ba1894..fd1fe64 100644 --- a/include/hw/intc/arm_gicv3_its_common.h +++ b/include/hw/intc/arm_gicv3_its_common.h @@ -28,6 +28,13 @@ #define ITS_TRANS_SIZE 0x10000 #define ITS_SIZE (ITS_CONTROL_SIZE + ITS_TRANS_SIZE) =20 +#define GITS_CTLR 0x0 +#define GITS_IIDR 0x4 +#define GITS_CBASER 0x80 +#define GITS_CWRITER 0x88 +#define GITS_CREADR 0x90 +#define GITS_BASER 0x100 + struct GICv3ITSState { SysBusDevice parent_obj; =20 @@ -43,6 +50,7 @@ struct GICv3ITSState { =20 /* Registers */ uint32_t ctlr; + uint32_t iidr; uint64_t cbaser; uint64_t cwriter; uint64_t creadr; --=20 2.5.5 From nobody Thu May 16 08:36:39 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1497023785445709.4532728233; Fri, 9 Jun 2017 08:56:25 -0700 (PDT) Received: from localhost ([::1]:55468 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMGt-0005Q2-VZ for importer@patchew.org; Fri, 09 Jun 2017 11:56:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45003) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMDu-0003Jl-Iy for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dJMDt-0005GJ-Mx for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41946) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dJMDr-0005Cn-FI; Fri, 09 Jun 2017 11:53:15 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 64791C04B92D; Fri, 9 Jun 2017 15:53:14 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-222.ams2.redhat.com [10.36.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9832F955AC; Fri, 9 Jun 2017 15:53:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 64791C04B92D Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=eric.auger@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 64791C04B92D From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Fri, 9 Jun 2017 17:52:32 +0200 Message-Id: <1497023553-18411-4-git-send-email-eric.auger@redhat.com> In-Reply-To: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> References: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Fri, 09 Jun 2017 15:53:14 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 3/4] hw/intc/arm_gicv3_kvm: Implement pending table save X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wei@redhat.com, drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, zhaoshenglong@huawei.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds the flush of the LPI pending bits into the redistributor pending tables. This happens on VM stop. There is no explicit restore as the tables are implicitly sync'ed on ITS table restore and on LPI enable at redistributor level. Signed-off-by: Eric Auger --- v5 -> v6: - do not abort on -EFAULT --- hw/intc/arm_gicv3_kvm.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index b70ee27..6051c77 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -25,6 +25,7 @@ #include "hw/sysbus.h" #include "qemu/error-report.h" #include "sysemu/kvm.h" +#include "sysemu/sysemu.h" #include "kvm_arm.h" #include "gicv3_internal.h" #include "vgic_common.h" @@ -680,6 +681,35 @@ static const ARMCPRegInfo gicv3_cpuif_reginfo[] =3D { REGINFO_SENTINEL }; =20 +/** + * vm_change_state_handler - VM change state callback aiming at flushing + * RDIST pending tables into guest RAM + * + * The tables get flushed to guest RAM whenever the VM gets stopped. + */ +static void vm_change_state_handler(void *opaque, int running, + RunState state) +{ + GICv3State *s =3D (GICv3State *)opaque; + Error *err =3D NULL; + int ret; + + if (running) { + return; + } + + ret =3D kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES, + NULL, true, &err); + if (err) { + error_report_err(err); + } + if (ret < 0 && ret !=3D -EFAULT) { + abort(); + } +} + + static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) { GICv3State *s =3D KVM_ARM_GICV3(dev); @@ -751,6 +781,10 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Er= ror **errp) return; } } + if (kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CTRL, + KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES)) { + qemu_add_vm_change_state_handler(vm_change_state_handler, s); + } } =20 static void kvm_arm_gicv3_class_init(ObjectClass *klass, void *data) --=20 2.5.5 From nobody Thu May 16 08:36:39 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1497023929164511.04773366722134; Fri, 9 Jun 2017 08:58:49 -0700 (PDT) Received: from localhost ([::1]:55473 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJMJD-00075L-Jf for importer@patchew.org; Fri, 09 Jun 2017 11:58:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45087) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dJME3-0003Qz-FB for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dJME2-0005Px-Eg for qemu-devel@nongnu.org; Fri, 09 Jun 2017 11:53:27 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60322) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dJMDy-0005KB-76; Fri, 09 Jun 2017 11:53:22 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 11DF5C0587DB; Fri, 9 Jun 2017 15:53:21 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-222.ams2.redhat.com [10.36.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id B5D29955A2; Fri, 9 Jun 2017 15:53:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 11DF5C0587DB Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=eric.auger@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 11DF5C0587DB From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Fri, 9 Jun 2017 17:52:33 +0200 Message-Id: <1497023553-18411-5-git-send-email-eric.auger@redhat.com> In-Reply-To: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> References: <1497023553-18411-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 09 Jun 2017 15:53:21 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 4/4] hw/intc/arm_gicv3_its: Allow save/restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wei@redhat.com, drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, zhaoshenglong@huawei.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We change the restoration priority of both the GICv3 and ITS. The GICv3 must be restored before the ITS and the ITS needs to be restored before PCIe devices since it translates their MSI transactions. Signed-off-by: Eric Auger Reviewed-by: Juan Quintela --- v5 -> v6: - fix the migration_blocker message (s/vGICv3/vITS) --- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_its_common.c | 2 +- hw/intc/arm_gicv3_its_kvm.c | 24 ++++++++++++------------ include/migration/vmstate.h | 2 ++ 4 files changed, 16 insertions(+), 13 deletions(-) diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index c6493d6..4228b7c 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -145,6 +145,7 @@ static const VMStateDescription vmstate_gicv3 =3D { .minimum_version_id =3D 1, .pre_save =3D gicv3_pre_save, .post_load =3D gicv3_post_load, + .priority =3D MIG_PRI_GICV3, .fields =3D (VMStateField[]) { VMSTATE_UINT32(gicd_ctlr, GICv3State), VMSTATE_UINT32_ARRAY(gicd_statusr, GICv3State, 2), diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c index 696c11c..68b20fc 100644 --- a/hw/intc/arm_gicv3_its_common.c +++ b/hw/intc/arm_gicv3_its_common.c @@ -48,7 +48,7 @@ static const VMStateDescription vmstate_its =3D { .name =3D "arm_gicv3_its", .pre_save =3D gicv3_its_pre_save, .post_load =3D gicv3_its_post_load, - .unmigratable =3D true, + .priority =3D MIG_PRI_GICV3_ITS, .fields =3D (VMStateField[]) { VMSTATE_UINT32(ctlr, GICv3ITSState), VMSTATE_UINT32(iidr, GICv3ITSState), diff --git a/hw/intc/arm_gicv3_its_kvm.c b/hw/intc/arm_gicv3_its_kvm.c index 4cd8f5f..1f8991b 100644 --- a/hw/intc/arm_gicv3_its_kvm.c +++ b/hw/intc/arm_gicv3_its_kvm.c @@ -85,18 +85,6 @@ static void kvm_arm_its_realize(DeviceState *dev, Error = **errp) GICv3ITSState *s =3D ARM_GICV3_ITS_COMMON(dev); Error *local_err =3D NULL; =20 - /* - * Block migration of a KVM GICv3 ITS device: the API for saving and - * restoring the state in the kernel is not yet available - */ - error_setg(&s->migration_blocker, "vITS migration is not implemented"); - migrate_add_blocker(s->migration_blocker, &local_err); - if (local_err) { - error_propagate(errp, local_err); - error_free(s->migration_blocker); - return; - } - s->dev_fd =3D kvm_create_device(kvm_state, KVM_DEV_TYPE_ARM_VGIC_ITS, = false); if (s->dev_fd < 0) { error_setg_errno(errp, -s->dev_fd, "error creating in-kernel ITS"); @@ -113,6 +101,18 @@ static void kvm_arm_its_realize(DeviceState *dev, Erro= r **errp) =20 gicv3_its_init_mmio(s, NULL); =20 + if (!kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR)) { + error_setg(&s->migration_blocker, "This operating system kernel " + "does not support vITS migration"); + migrate_add_blocker(s->migration_blocker, &local_err); + if (local_err) { + error_propagate(errp, local_err); + error_free(s->migration_blocker); + return; + } + } + kvm_msi_use_devid =3D true; kvm_gsi_direct_mapping =3D false; kvm_msi_via_irqfd_allowed =3D kvm_irqfds_enabled(); diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h index 8a3e9e6..2f4c4a9 100644 --- a/include/migration/vmstate.h +++ b/include/migration/vmstate.h @@ -187,6 +187,8 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT =3D 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ + MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX, } MigrationPriority; =20 --=20 2.5.5