From nobody Sun May 5 13:08:44 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1488800982192567.3921247560154; Mon, 6 Mar 2017 03:49:42 -0800 (PST) Received: from localhost ([::1]:43095 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ckr92-0007z4-VX for importer@patchew.org; Mon, 06 Mar 2017 06:49:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59481) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ckr7s-0007iK-UC for qemu-devel@nongnu.org; Mon, 06 Mar 2017 06:48:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ckr7r-0006FC-Ju for qemu-devel@nongnu.org; Mon, 06 Mar 2017 06:48:28 -0500 Received: from mx1.redhat.com ([209.132.183.28]:55412) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ckr7o-0006A0-Fd; Mon, 06 Mar 2017 06:48:24 -0500 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7F6F3F3D4F; Mon, 6 Mar 2017 11:48:24 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-100.ams2.redhat.com [10.36.116.100]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v26BmGgB027200; Mon, 6 Mar 2017 06:48:20 -0500 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Mon, 6 Mar 2017 12:48:12 +0100 Message-Id: <1488800894-22421-2-git-send-email-eric.auger@redhat.com> In-Reply-To: <1488800894-22421-1-git-send-email-eric.auger@redhat.com> References: <1488800894-22421-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Mon, 06 Mar 2017 11:48:24 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 1/3] linux-headers: Update for vITS save/restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is a linux header update against 4.11-rc1 plus the non upstreamed ITS migration series. https://github.com/eauger/linux/tree/v4.11-rc1-its-mig-v3 It aims at enhancing the KVM user API with vITS save/restore capability. This consists in two new groups for the ARM_VGIC_ITS KVM device, named: KVM_DEV_ARM_VGIC_GRP_ITS_REGS, KVM_DEV_ARM_VGIC_GRP_ITS_TABLES. Signed-off-by: Eric Auger --- include/standard-headers/linux/virtio_pci.h | 2 +- linux-headers/asm-arm/kvm.h | 2 ++ linux-headers/asm-arm64/kvm.h | 2 ++ linux-headers/linux/userfaultfd.h | 16 +++++++++++----- 4 files changed, 16 insertions(+), 6 deletions(-) diff --git a/include/standard-headers/linux/virtio_pci.h b/include/standard= -headers/linux/virtio_pci.h index 9262acd..eba3ba4 100644 --- a/include/standard-headers/linux/virtio_pci.h +++ b/include/standard-headers/linux/virtio_pci.h @@ -79,7 +79,7 @@ * configuration space */ #define VIRTIO_PCI_CONFIG_OFF(msix_enabled) ((msix_enabled) ? 24 : 20) /* Deprecated: please use VIRTIO_PCI_CONFIG_OFF instead */ -#define VIRTIO_PCI_CONFIG(dev) VIRTIO_PCI_CONFIG_OFF((dev)->msix_enabled) +#define VIRTIO_PCI_CONFIG(dev) VIRTIO_PCI_CONFIG_OFF((dev)->pci_dev->msix_= enabled) =20 /* Virtio ABI version, this must match exactly */ #define VIRTIO_PCI_ABI_VERSION 0 diff --git a/linux-headers/asm-arm/kvm.h b/linux-headers/asm-arm/kvm.h index 1101d55..c8b9a68 100644 --- a/linux-headers/asm-arm/kvm.h +++ b/linux-headers/asm-arm/kvm.h @@ -192,6 +192,8 @@ struct kvm_arch_memory_slot { #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5 #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 +#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_ITS_TABLES 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h index 651ec30..a8d7d67 100644 --- a/linux-headers/asm-arm64/kvm.h +++ b/linux-headers/asm-arm64/kvm.h @@ -212,6 +212,8 @@ struct kvm_arch_memory_slot { #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5 #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 +#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_ITS_TABLES 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/linux-headers/linux/userfaultfd.h b/linux-headers/linux/userfa= ultfd.h index 2ed5dc3..9d8f313 100644 --- a/linux-headers/linux/userfaultfd.h +++ b/linux-headers/linux/userfaultfd.h @@ -18,9 +18,11 @@ * means the userland is reading). */ #define UFFD_API ((__u64)0xAA) -#define UFFD_API_FEATURES (UFFD_FEATURE_EVENT_FORK | \ +#define UFFD_API_FEATURES (UFFD_FEATURE_EVENT_EXIT | \ + UFFD_FEATURE_EVENT_FORK | \ UFFD_FEATURE_EVENT_REMAP | \ - UFFD_FEATURE_EVENT_MADVDONTNEED | \ + UFFD_FEATURE_EVENT_REMOVE | \ + UFFD_FEATURE_EVENT_UNMAP | \ UFFD_FEATURE_MISSING_HUGETLBFS | \ UFFD_FEATURE_MISSING_SHMEM) #define UFFD_API_IOCTLS \ @@ -92,7 +94,7 @@ struct uffd_msg { struct { __u64 start; __u64 end; - } madv_dn; + } remove; =20 struct { /* unused reserved fields */ @@ -109,7 +111,9 @@ struct uffd_msg { #define UFFD_EVENT_PAGEFAULT 0x12 #define UFFD_EVENT_FORK 0x13 #define UFFD_EVENT_REMAP 0x14 -#define UFFD_EVENT_MADVDONTNEED 0x15 +#define UFFD_EVENT_REMOVE 0x15 +#define UFFD_EVENT_UNMAP 0x16 +#define UFFD_EVENT_EXIT 0x17 =20 /* flags for UFFD_EVENT_PAGEFAULT */ #define UFFD_PAGEFAULT_FLAG_WRITE (1<<0) /* If this was a write fault */ @@ -155,9 +159,11 @@ struct uffdio_api { #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) #define UFFD_FEATURE_EVENT_REMAP (1<<2) -#define UFFD_FEATURE_EVENT_MADVDONTNEED (1<<3) +#define UFFD_FEATURE_EVENT_REMOVE (1<<3) #define UFFD_FEATURE_MISSING_HUGETLBFS (1<<4) #define UFFD_FEATURE_MISSING_SHMEM (1<<5) +#define UFFD_FEATURE_EVENT_UNMAP (1<<6) +#define UFFD_FEATURE_EVENT_EXIT (1<<7) __u64 features; =20 __u64 ioctls; --=20 2.5.5 From nobody Sun May 5 13:08:44 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1488801070469511.41715466835626; Mon, 6 Mar 2017 03:51:10 -0800 (PST) Received: from localhost ([::1]:43108 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ckrAS-0000mg-4z for importer@patchew.org; Mon, 06 Mar 2017 06:51:08 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59513) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ckr7v-0007k8-8X for qemu-devel@nongnu.org; Mon, 06 Mar 2017 06:48:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ckr7u-0006Hu-22 for qemu-devel@nongnu.org; Mon, 06 Mar 2017 06:48:31 -0500 Received: from mx1.redhat.com ([209.132.183.28]:60838) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ckr7q-0006Cv-Rz; Mon, 06 Mar 2017 06:48:27 -0500 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D5E313B721; Mon, 6 Mar 2017 11:48:26 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-100.ams2.redhat.com [10.36.116.100]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v26BmGgC027200; Mon, 6 Mar 2017 06:48:23 -0500 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Mon, 6 Mar 2017 12:48:13 +0100 Message-Id: <1488800894-22421-3-git-send-email-eric.auger@redhat.com> In-Reply-To: <1488800894-22421-1-git-send-email-eric.auger@redhat.com> References: <1488800894-22421-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 06 Mar 2017 11:48:26 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 2/3] hw/intc/arm_gicv3_its: Implement state save/restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We need to handle both registers and ITS tables. While register handling is standard, ITS table handling is more challenging since the kernel API is devised so that the tables are flushed into guest RAM and not in vmstate buffers. Flushing the ITS tables on device pre_save() is too late since the guest RAM had already been saved at this point. Table flushing needs to happen when we are sure the vcpus are stopped and before the last dirty page saving. The right point is RUN_STATE_FINISH_MIGRATE but sometimes the VM gets stopped before migration launch so let's simply flush the tables each time the VM gets stopped. For regular ITS registers we just can use vmstate pre_save and post_load callbacks. Signed-off-by: Eric Auger --- --- hw/intc/arm_gicv3_its_common.c | 8 ++++ hw/intc/arm_gicv3_its_kvm.c | 86 ++++++++++++++++++++++++++++++= ++++ include/hw/intc/arm_gicv3_its_common.h | 6 +++ 3 files changed, 100 insertions(+) diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c index 9d67c5c..75b9f04 100644 --- a/hw/intc/arm_gicv3_its_common.c +++ b/hw/intc/arm_gicv3_its_common.c @@ -49,6 +49,14 @@ static const VMStateDescription vmstate_its =3D { .pre_save =3D gicv3_its_pre_save, .post_load =3D gicv3_its_post_load, .unmigratable =3D true, + .fields =3D (VMStateField[]) { + VMSTATE_UINT32(ctlr, GICv3ITSState), + VMSTATE_UINT64(cbaser, GICv3ITSState), + VMSTATE_UINT64(cwriter, GICv3ITSState), + VMSTATE_UINT64(creadr, GICv3ITSState), + VMSTATE_UINT64_ARRAY(baser, GICv3ITSState, 8), + VMSTATE_END_OF_LIST() + }, }; =20 static MemTxResult gicv3_its_trans_read(void *opaque, hwaddr offset, diff --git a/hw/intc/arm_gicv3_its_kvm.c b/hw/intc/arm_gicv3_its_kvm.c index bd4f3aa..45e57d6 100644 --- a/hw/intc/arm_gicv3_its_kvm.c +++ b/hw/intc/arm_gicv3_its_kvm.c @@ -53,6 +53,24 @@ static int kvm_its_send_msi(GICv3ITSState *s, uint32_t v= alue, uint16_t devid) return kvm_vm_ioctl(kvm_state, KVM_SIGNAL_MSI, &msi); } =20 +/** + * vm_change_state_handler - VM change state callback aiming at flushing + * ITS tables into guest RAM + * + * The tables get flushed to guest RAM whenever the VM gets stopped. + */ +static void vm_change_state_handler(void *opaque, int running, + RunState state) +{ + GICv3ITSState *s =3D (GICv3ITSState *)opaque; + + if (running) { + return; + } + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_TABLES, + 0, NULL, false); +} + static void kvm_arm_its_realize(DeviceState *dev, Error **errp) { GICv3ITSState *s =3D ARM_GICV3_ITS_COMMON(dev); @@ -89,6 +107,8 @@ static void kvm_arm_its_realize(DeviceState *dev, Error = **errp) kvm_msi_use_devid =3D true; kvm_gsi_direct_mapping =3D false; kvm_msi_via_irqfd_allowed =3D kvm_irqfds_enabled(); + + qemu_add_vm_change_state_handler(vm_change_state_handler, s); } =20 static void kvm_arm_its_init(Object *obj) @@ -102,6 +122,70 @@ static void kvm_arm_its_init(Object *obj) &error_abort); } =20 +/** + * kvm_arm_its_get - handles the saving of ITS registers. + * ITS tables, being flushed into guest RAM needs to be saved before + * the pre_save() callback, hence the migration state change notifiers + */ +static void kvm_arm_its_get(GICv3ITSState *s) +{ + uint64_t reg; + int i; + + for (i =3D 0; i < 8; i++) { + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_BASER + i * 8, &s->baser[i], false); + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR, ®, false); + s->ctlr =3D extract64(reg, 0, 32); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CBASER, &s->cbaser, false); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CREADR, &s->creadr, false); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CWRITER, &s->cwriter, false); +} + +/** + * kvm_arm_its_put - Restore both the ITS registers and guest RAM tables + * ITS tables, being flushed into guest RAM needs to be saved before + * the pre_save() callback. The restoration order matters since there + * are dependencies between register settings, as specified by the + * architecture specification + */ +static void kvm_arm_its_put(GICv3ITSState *s) +{ + uint64_t reg; + int i; + + /* must be written before GITS_CREADR since it resets this latter*/ + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CBASER, &s->cbaser, true); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CREADR, &s->creadr, true); + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CWRITER, &s->cwriter, true); + + for (i =3D 0; i < 8; i++) { + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_BASER + i * 8, &s->baser[i], true); + } + + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_TABLES, + 0, NULL, true); + + reg =3D s->ctlr; + kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR, ®, true); +} + static void kvm_arm_its_class_init(ObjectClass *klass, void *data) { DeviceClass *dc =3D DEVICE_CLASS(klass); @@ -109,6 +193,8 @@ static void kvm_arm_its_class_init(ObjectClass *klass, = void *data) =20 dc->realize =3D kvm_arm_its_realize; icc->send_msi =3D kvm_its_send_msi; + icc->pre_save =3D kvm_arm_its_get; + icc->post_load =3D kvm_arm_its_put; } =20 static const TypeInfo kvm_arm_its_info =3D { diff --git a/include/hw/intc/arm_gicv3_its_common.h b/include/hw/intc/arm_g= icv3_its_common.h index 1ba1894..ed5d6df 100644 --- a/include/hw/intc/arm_gicv3_its_common.h +++ b/include/hw/intc/arm_gicv3_its_common.h @@ -28,6 +28,12 @@ #define ITS_TRANS_SIZE 0x10000 #define ITS_SIZE (ITS_CONTROL_SIZE + ITS_TRANS_SIZE) =20 +#define GITS_CTLR 0x0 +#define GITS_CBASER 0x80 +#define GITS_CWRITER 0x88 +#define GITS_CREADR 0x90 +#define GITS_BASER 0x100 + struct GICv3ITSState { SysBusDevice parent_obj; =20 --=20 2.5.5 From nobody Sun May 5 13:08:44 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1488801081959615.2509975952125; Mon, 6 Mar 2017 03:51:21 -0800 (PST) Received: from localhost ([::1]:43109 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ckrAe-0000wF-PZ for importer@patchew.org; Mon, 06 Mar 2017 06:51:20 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59552) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ckr82-0007qY-JK for qemu-devel@nongnu.org; Mon, 06 Mar 2017 06:48:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ckr81-0006Oi-J4 for qemu-devel@nongnu.org; Mon, 06 Mar 2017 06:48:38 -0500 Received: from mx1.redhat.com ([209.132.183.28]:38648) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ckr7v-0006JI-47; Mon, 06 Mar 2017 06:48:31 -0500 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3936163E2C; Mon, 6 Mar 2017 11:48:31 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-116-100.ams2.redhat.com [10.36.116.100]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v26BmGgD027200; Mon, 6 Mar 2017 06:48:26 -0500 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, peter.maydell@linaro.org, qemu-arm@nongnu.org, qemu-devel@nongnu.org Date: Mon, 6 Mar 2017 12:48:14 +0100 Message-Id: <1488800894-22421-4-git-send-email-eric.auger@redhat.com> In-Reply-To: <1488800894-22421-1-git-send-email-eric.auger@redhat.com> References: <1488800894-22421-1-git-send-email-eric.auger@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Mon, 06 Mar 2017 11:48:31 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 3/3] hw/intc/arm_gicv3_its: Allow save/restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: drjones@redhat.com, vijay.kilari@gmail.com, quintela@redhat.com, dgilbert@redhat.com, peterx@redhat.com, Vijaya.Kumar@cavium.com, christoffer.dall@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We change the restoration priority of both the GICv3 and ITS. The GICv3 must be restored before the ITS and the ITS needs to be restored before PCIe devices since it translates their MSI transactions. We typically observe the virtio-pci-net device sending MSI transactions very early (even before the first vcpu run) which looks weird. It appears that not servicing those transactions cause the virtio-pci-net to stall. Signed-off-by: Eric Auger --- v1 -> v2: - handle case where migrate_add_blocker fails - add comments along with ITS and GICv3 migration priorities --- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_its_common.c | 3 ++- hw/intc/arm_gicv3_its_kvm.c | 23 +++++++++++------------ include/migration/vmstate.h | 2 ++ 4 files changed, 16 insertions(+), 13 deletions(-) diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index c6493d6..4228b7c 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -145,6 +145,7 @@ static const VMStateDescription vmstate_gicv3 =3D { .minimum_version_id =3D 1, .pre_save =3D gicv3_pre_save, .post_load =3D gicv3_post_load, + .priority =3D MIG_PRI_GICV3, .fields =3D (VMStateField[]) { VMSTATE_UINT32(gicd_ctlr, GICv3State), VMSTATE_UINT32_ARRAY(gicd_statusr, GICv3State, 2), diff --git a/hw/intc/arm_gicv3_its_common.c b/hw/intc/arm_gicv3_its_common.c index 75b9f04..854709f 100644 --- a/hw/intc/arm_gicv3_its_common.c +++ b/hw/intc/arm_gicv3_its_common.c @@ -48,7 +48,8 @@ static const VMStateDescription vmstate_its =3D { .name =3D "arm_gicv3_its", .pre_save =3D gicv3_its_pre_save, .post_load =3D gicv3_its_post_load, - .unmigratable =3D true, + .unmigratable =3D false, + .priority =3D MIG_PRI_GICV3_ITS, .fields =3D (VMStateField[]) { VMSTATE_UINT32(ctlr, GICv3ITSState), VMSTATE_UINT64(cbaser, GICv3ITSState), diff --git a/hw/intc/arm_gicv3_its_kvm.c b/hw/intc/arm_gicv3_its_kvm.c index 45e57d6..3bd2873 100644 --- a/hw/intc/arm_gicv3_its_kvm.c +++ b/hw/intc/arm_gicv3_its_kvm.c @@ -76,18 +76,6 @@ static void kvm_arm_its_realize(DeviceState *dev, Error = **errp) GICv3ITSState *s =3D ARM_GICV3_ITS_COMMON(dev); Error *local_err =3D NULL; =20 - /* - * Block migration of a KVM GICv3 ITS device: the API for saving and - * restoring the state in the kernel is not yet available - */ - error_setg(&s->migration_blocker, "vITS migration is not implemented"); - migrate_add_blocker(s->migration_blocker, &local_err); - if (local_err) { - error_propagate(errp, local_err); - error_free(s->migration_blocker); - return; - } - s->dev_fd =3D kvm_create_device(kvm_state, KVM_DEV_TYPE_ARM_VGIC_ITS, = false); if (s->dev_fd < 0) { error_setg_errno(errp, -s->dev_fd, "error creating in-kernel ITS"); @@ -104,6 +92,17 @@ static void kvm_arm_its_realize(DeviceState *dev, Error= **errp) =20 gicv3_its_init_mmio(s, NULL); =20 + if (!kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS, + GITS_CTLR)) { + error_setg(&s->migration_blocker, "vITS migration is not implement= ed"); + migrate_add_blocker(s->migration_blocker, &local_err); + if (local_err) { + error_propagate(errp, local_err); + error_free(s->migration_blocker); + return; + } + } + kvm_msi_use_devid =3D true; kvm_gsi_direct_mapping =3D false; kvm_msi_via_irqfd_allowed =3D kvm_irqfds_enabled(); diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h index f2dbf84..8dab9c7 100644 --- a/include/migration/vmstate.h +++ b/include/migration/vmstate.h @@ -198,6 +198,8 @@ enum VMStateFlags { typedef enum { MIG_PRI_DEFAULT =3D 0, MIG_PRI_IOMMU, /* Must happen before PCI devices */ + MIG_PRI_GICV3_ITS, /* Must happen before PCI devices */ + MIG_PRI_GICV3, /* Must happen before the ITS */ MIG_PRI_MAX, } MigrationPriority; =20 --=20 2.5.5