From nobody Sat May 4 00:38:03 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1565085007; cv=none; d=zoho.com; s=zohoarc; b=Co3+ZqHOAKRZKiFV2bg+wrn2CTZ4CCPmqGpRo1t67mCP5nBDKg/AheFF/LNYx2Cq5EB0s3jNt7I2aT1BjM1dZfYLYxZaJsVNWV9iQ4n0srgbvRhaZk2GYHw41TUZOYzm67hibbUXXvCGLmPcRG307iYvrdmG1jRT3HiMjYI5Mrs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565085007; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=wrlrtsqLD9/rr6h8ApKz8JJGoqapxL3euFoFmN4eK5o=; b=c47B94kfaXVmS3p61IcQglJ+Jen40JhA/v98lGwaBvD3egp4ETs8ju74pelddz6ET8jOfnvuxRnUjfzj6enzzsbM/O3Eu8FQOp8LgvL16qdHZrRYfn3rS+YBRezsdTMN1HzTjaEW7ildlZZxN4eJ7sLdsL6pe+ilWGgi0r82w28= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1565085007052279.6479833674201; Tue, 6 Aug 2019 02:50:07 -0700 (PDT) Received: from localhost ([::1]:59946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huw6Y-0005hQ-50 for importer@patchew.org; Tue, 06 Aug 2019 05:50:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36859) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huw5G-0003LV-HP for qemu-devel@nongnu.org; Tue, 06 Aug 2019 05:48:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1huw5F-0002f2-57 for qemu-devel@nongnu.org; Tue, 06 Aug 2019 05:48:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35672) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1huw5E-0002eo-Ti; Tue, 06 Aug 2019 05:48:45 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3ED297FDEC; Tue, 6 Aug 2019 09:48:44 +0000 (UTC) Received: from dell-r430-03.lab.eng.brq.redhat.com (dell-r430-03.lab.eng.brq.redhat.com [10.37.153.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id C5B595D9C5; Tue, 6 Aug 2019 09:48:42 +0000 (UTC) From: Igor Mammedov To: qemu-devel@nongnu.org Date: Tue, 6 Aug 2019 05:48:33 -0400 Message-Id: <20190806094834.7691-2-imammedo@redhat.com> In-Reply-To: <20190806094834.7691-1-imammedo@redhat.com> References: <20190806094834.7691-1-imammedo@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 06 Aug 2019 09:48:44 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH for-4.2 v4 1/2] kvm: s390: split too big memory section on several memslots X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: thuth@redhat.com, david@redhat.com, cohuck@redhat.com, borntraeger@de.ibm.com, qemu-s390x@nongnu.org, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Max memslot size supported by kvm on s390 is 8Tb, move logic of splitting RAM in chunks upto 8T to KVM code. This way it will hide KVM specific restrictions in KVM code and won't affect baord level design decisions. Which would allow us to avoid misusing memory_region_allocate_system_memory() API and eventually use a single hostmem backend for guest RAM. Signed-off-by: Igor Mammedov --- v4: * fix compilation issue (Christian Borntraeger ) * advance HVA along with GPA in kvm_set_phys_mem() (Christian Borntraeger ) patch prepares only KVM side for switching to single RAM memory region another patch will take care of dropping manual RAM partitioning in s390 code. --- include/sysemu/kvm_int.h | 1 + accel/kvm/kvm-all.c | 80 +++++++++++++++++++++++--------------- hw/s390x/s390-virtio-ccw.c | 9 ----- target/s390x/kvm.c | 12 ++++++ 4 files changed, 62 insertions(+), 40 deletions(-) diff --git a/include/sysemu/kvm_int.h b/include/sysemu/kvm_int.h index 31df465fdc..7f7520bce2 100644 --- a/include/sysemu/kvm_int.h +++ b/include/sysemu/kvm_int.h @@ -41,4 +41,5 @@ typedef struct KVMMemoryListener { void kvm_memory_listener_register(KVMState *s, KVMMemoryListener *kml, AddressSpace *as, int as_id); =20 +void kvm_set_max_memslot_size(hwaddr max_slot_size); #endif diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index f450f25295..d87f855ea4 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -138,6 +138,7 @@ bool kvm_direct_msi_allowed; bool kvm_ioeventfd_any_length_allowed; bool kvm_msi_use_devid; static bool kvm_immediate_exit; +static hwaddr kvm_max_slot_size =3D ~0; =20 static const KVMCapabilityInfo kvm_required_capabilites[] =3D { KVM_CAP_INFO(USER_MEMORY), @@ -951,6 +952,14 @@ kvm_check_extension_list(KVMState *s, const KVMCapabil= ityInfo *list) return NULL; } =20 +void kvm_set_max_memslot_size(hwaddr max_slot_size) +{ + g_assert( + ROUND_UP(max_slot_size, qemu_real_host_page_size) =3D=3D max_slot_= size + ); + kvm_max_slot_size =3D max_slot_size; +} + static void kvm_set_phys_mem(KVMMemoryListener *kml, MemoryRegionSection *section, bool add) { @@ -958,7 +967,7 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml, int err; MemoryRegion *mr =3D section->mr; bool writeable =3D !mr->readonly && !mr->rom_device; - hwaddr start_addr, size; + hwaddr start_addr, size, slot_size; void *ram; =20 if (!memory_region_is_ram(mr)) { @@ -983,41 +992,50 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml, kvm_slots_lock(kml); =20 if (!add) { - mem =3D kvm_lookup_matching_slot(kml, start_addr, size); - if (!mem) { - goto out; - } - if (mem->flags & KVM_MEM_LOG_DIRTY_PAGES) { - kvm_physical_sync_dirty_bitmap(kml, section); - } + do { + slot_size =3D kvm_max_slot_size < size ? kvm_max_slot_size : s= ize; + mem =3D kvm_lookup_matching_slot(kml, start_addr, slot_size); + if (!mem) { + goto out; + } + if (mem->flags & KVM_MEM_LOG_DIRTY_PAGES) { + kvm_physical_sync_dirty_bitmap(kml, section); + } =20 - /* unregister the slot */ - g_free(mem->dirty_bmap); - mem->dirty_bmap =3D NULL; - mem->memory_size =3D 0; - mem->flags =3D 0; - err =3D kvm_set_user_memory_region(kml, mem, false); - if (err) { - fprintf(stderr, "%s: error unregistering slot: %s\n", - __func__, strerror(-err)); - abort(); - } + /* unregister the slot */ + g_free(mem->dirty_bmap); + mem->dirty_bmap =3D NULL; + mem->memory_size =3D 0; + mem->flags =3D 0; + err =3D kvm_set_user_memory_region(kml, mem, false); + if (err) { + fprintf(stderr, "%s: error unregistering slot: %s\n", + __func__, strerror(-err)); + abort(); + } + start_addr +=3D slot_size; + } while ((size -=3D slot_size)); goto out; } =20 /* register the new slot */ - mem =3D kvm_alloc_slot(kml); - mem->memory_size =3D size; - mem->start_addr =3D start_addr; - mem->ram =3D ram; - mem->flags =3D kvm_mem_flags(mr); - - err =3D kvm_set_user_memory_region(kml, mem, true); - if (err) { - fprintf(stderr, "%s: error registering slot: %s\n", __func__, - strerror(-err)); - abort(); - } + do { + slot_size =3D kvm_max_slot_size < size ? kvm_max_slot_size : size; + mem =3D kvm_alloc_slot(kml); + mem->memory_size =3D slot_size; + mem->start_addr =3D start_addr; + mem->ram =3D ram; + mem->flags =3D kvm_mem_flags(mr); + + err =3D kvm_set_user_memory_region(kml, mem, true); + if (err) { + fprintf(stderr, "%s: error registering slot: %s\n", __func__, + strerror(-err)); + abort(); + } + start_addr +=3D slot_size; + ram +=3D slot_size; + } while ((size -=3D slot_size)); =20 out: kvm_slots_unlock(kml); diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 5b6a9a4e55..0c03ffb7c7 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -151,15 +151,6 @@ static void virtio_ccw_register_hcalls(void) virtio_ccw_hcall_early_printk); } =20 -/* - * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages - * as the dirty bitmap must be managed by bitops that take an int as - * position indicator. If we have a guest beyond that we will split off - * new subregions. The split must happen on a segment boundary (1MB). - */ -#define KVM_MEM_MAX_NR_PAGES ((1ULL << 31) - 1) -#define SEG_MSK (~0xfffffULL) -#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SE= G_MSK) static void s390_memory_init(ram_addr_t mem_size) { MemoryRegion *sysmem =3D get_system_memory(); diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c index 6e814c230b..6b1428a760 100644 --- a/target/s390x/kvm.c +++ b/target/s390x/kvm.c @@ -28,6 +28,7 @@ #include "cpu.h" #include "internal.h" #include "kvm_s390x.h" +#include "sysemu/kvm_int.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "qemu/timer.h" @@ -122,6 +123,16 @@ #define VCPU_IRQ_BUF_SIZE(max_cpus) (sizeof(struct kvm_s390_irq) * \ (max_cpus + NR_LOCAL_IRQS)) =20 +/* + * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages + * as the dirty bitmap must be managed by bitops that take an int as + * position indicator. If we have a guest beyond that we will split off + * new subregions. The split must happen on a segment boundary (1MB). + */ +#define KVM_MEM_MAX_NR_PAGES ((1ULL << 31) - 1) +#define SEG_MSK (~0xfffffULL) +#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SE= G_MSK) + static CPUWatchpoint hw_watchpoint; /* * We don't use a list because this structure is also used to transmit the @@ -347,6 +358,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) */ /* kvm_vm_enable_cap(s, KVM_CAP_S390_AIS, 0); */ =20 + kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES); return 0; } =20 --=20 2.18.1 From nobody Sat May 4 00:38:03 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1565084974; cv=none; d=zoho.com; s=zohoarc; b=cYwvtsI2F0RsahttKkB3oLwo7X+rfz5I0Xf+E/2eOXs2cCNzTiqbj6yu8Ec8YmEjj5jHR6vo8PZQcfD8iJvompNofJv2nTUNZH3BCcAFnz/vRGVncEtiacezb7toGMqw4bb8WdmKiVU4M8Kok+5iLlt85TqaIqJZyjLOwfVWBLU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565084974; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=U3DXhIpalKwD2XPQjOWWWZo+ec+MpqnECkTFBeCDyVo=; b=PgaC+3QqQ8NKHI+KblgbdiZMCu9KixmSZGW03rqQljCR2omcdI+ysH7oxbuiSckGQpai9sdS/xdcnScQ4OeLkdagCK7xd2H1z/hjQU8fbFZQ2FyahY+PWVFXD5OvGITvJpnL63PT9nF8Mbr17KWQ6qZ8M22mZacOnulO3M2LI+s= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1565084974957650.6708371429611; Tue, 6 Aug 2019 02:49:34 -0700 (PDT) Received: from localhost ([::1]:59932 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huw62-0004TG-2T for importer@patchew.org; Tue, 06 Aug 2019 05:49:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36880) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huw5H-0003Ma-TW for qemu-devel@nongnu.org; Tue, 06 Aug 2019 05:48:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1huw5G-0002fa-Qh for qemu-devel@nongnu.org; Tue, 06 Aug 2019 05:48:47 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51042) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1huw5G-0002fK-LQ; Tue, 06 Aug 2019 05:48:46 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F181B30FB8E9; Tue, 6 Aug 2019 09:48:45 +0000 (UTC) Received: from dell-r430-03.lab.eng.brq.redhat.com (dell-r430-03.lab.eng.brq.redhat.com [10.37.153.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 83DF85D9C5; Tue, 6 Aug 2019 09:48:44 +0000 (UTC) From: Igor Mammedov To: qemu-devel@nongnu.org Date: Tue, 6 Aug 2019 05:48:34 -0400 Message-Id: <20190806094834.7691-3-imammedo@redhat.com> In-Reply-To: <20190806094834.7691-1-imammedo@redhat.com> References: <20190806094834.7691-1-imammedo@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Tue, 06 Aug 2019 09:48:46 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH for-4.2 v4 2/2] s390: do not call memory_region_allocate_system_memory() multiple times X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: thuth@redhat.com, david@redhat.com, cohuck@redhat.com, borntraeger@de.ibm.com, qemu-s390x@nongnu.org, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" s390 was trying to solve limited KVM memslot size issue by abusing memory_region_allocate_system_memory(), which breaks API contract where the function might be called only once. Beside an invalid use of API, the approach also introduced migration issue, since RAM chunks for each KVM_SLOT_MAX_BYTES are transferred in migration stream as separate RAMBlocks. After discussion [1], it was agreed to break migration from older QEMU for guest with RAM >8Tb (as it was relatively new (since 2.12) and considered to be not actually used downstream). Migration should keep working for guests with less than 8TB and for more than 8TB with QEMU 4.2 and newer binary. In case user tries to migrate more than 8TB guest, between incompatible QEMU versions, migration should fail gracefully due to non-exiting RAMBlock ID or RAMBlock size mismatch. Taking in account above and that now KVM code is able to split too big MemorySection into several memslots, stop abusing memory_region_allocate_system_memory() and use only one memory region for RAM. 1) [PATCH RFC v2 4/4] s390: do not call memory_region_allocate_system_memo= ry() multiple times Signed-off-by: Igor Mammedov Reviewed-by: David Hildenbrand --- v3: - drop migration compat code PS: I don't have access to a suitable system to test it. --- hw/s390x/s390-virtio-ccw.c | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 0c03ffb7c7..e30058df38 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -154,27 +154,12 @@ static void virtio_ccw_register_hcalls(void) static void s390_memory_init(ram_addr_t mem_size) { MemoryRegion *sysmem =3D get_system_memory(); - ram_addr_t chunk, offset =3D 0; - unsigned int number =3D 0; + MemoryRegion *ram =3D g_new(MemoryRegion, 1); Error *local_err =3D NULL; - gchar *name; =20 /* allocate RAM for core */ - name =3D g_strdup_printf("s390.ram"); - while (mem_size) { - MemoryRegion *ram =3D g_new(MemoryRegion, 1); - uint64_t size =3D mem_size; - - /* KVM does not allow memslots >=3D 8 TB */ - chunk =3D MIN(size, KVM_SLOT_MAX_BYTES); - memory_region_allocate_system_memory(ram, NULL, name, chunk); - memory_region_add_subregion(sysmem, offset, ram); - mem_size -=3D chunk; - offset +=3D chunk; - g_free(name); - name =3D g_strdup_printf("s390.ram.%u", ++number); - } - g_free(name); + memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size); + memory_region_add_subregion(sysmem, 0, ram); =20 /* * Configure the maximum page size. As no memory devices were created --=20 2.18.1