From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682599; cv=none; d=zohomail.com; s=zohoarc; b=H0jA3QyyZxfz75s1AxhsfLRM/HpbB+vwSBFWTOoUOFAW8CeX4BaCDLG8O7qWyz5KQoaQFWCcKJN+nzwmzfsJDrTnbs4cbSAmoxrcnetzI9PiYmXjLfAEoIRD6eCtlsYvIWejtkMeBILP8VihaJBighHJStEne8oLxQJ1V+5zDHg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682599; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=a7SKC2TVqKQzjslWVnkLc/vK9me7ZpW6UmUARx4xlI4=; b=Y3JeGX/ofWnqfiuM8lj6lQMinEjQnRnFI+kHRJn/Mhanhq1VDjeDwtJWt8ZVBSM5gi7j+nlVXcWTqjm3ZD9aPcWpEqhWag6E++Y+dcJG4urDbDf4rAENXhhJIQT+S8tpRPBEcSX90cReZXFTXbZn+HhS5a/XeBW85XfCQNtuziw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1675682599611301.3004682296156; Mon, 6 Feb 2023 03:23:19 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzZf-0004EM-8h; Mon, 06 Feb 2023 06:22:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzZZ-0004DA-E6 for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:22:12 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzZX-0003Mh-5y for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:22:09 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-517-qAks29C5OHq9GLqAIKiCbg-1; Mon, 06 Feb 2023 06:21:00 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B64AC800050; Mon, 6 Feb 2023 11:20:59 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id C54AC492B21; Mon, 6 Feb 2023 11:20:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a7SKC2TVqKQzjslWVnkLc/vK9me7ZpW6UmUARx4xlI4=; b=M+fITCXuShcVrPrkxjPsBZ6CDbRDfyslBIIsxVpZuKAj98uMdsHifozHRWsSwGdmAS0xpl 8gJa7jCCv6lbbOq7hzcdBZXDRgrIqM//fPXuqdocSb5Uh/rwPmhMH3BSifVXfhfcbpd0WG knXkCpM1ZM9xfXqHYPgklgWOhYFXe6o= X-MC-Unique: qAks29C5OHq9GLqAIKiCbg-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 1/8] linux-headers: Update for dirty ring Date: Mon, 6 Feb 2023 19:20:03 +0800 Message-Id: <20230206112010.99871-2-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682600896100007 Content-Type: text/plain; charset="utf-8" Signed-off-by: Gavin Shan --- linux-headers/asm-arm64/kvm.h | 1 + linux-headers/linux/kvm.h | 2 ++ 2 files changed, 3 insertions(+) diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h index 4bf2d7246e..a7cfefb3a8 100644 --- a/linux-headers/asm-arm64/kvm.h +++ b/linux-headers/asm-arm64/kvm.h @@ -43,6 +43,7 @@ #define __KVM_HAVE_VCPU_EVENTS =20 #define KVM_COALESCED_MMIO_PAGE_OFFSET 1 +#define KVM_DIRTY_LOG_PAGE_OFFSET 64 =20 #define KVM_REG_SIZE(id) \ (1U << (((id) & KVM_REG_SIZE_MASK) >> KVM_REG_SIZE_SHIFT)) diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index ebdafa576d..5b4e0e4e68 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -1175,6 +1175,8 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_VM_DISABLE_NX_HUGE_PAGES 220 #define KVM_CAP_S390_ZPCI_OP 221 #define KVM_CAP_S390_CPU_TOPOLOGY 222 +#define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223 +#define KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP 225 =20 #ifdef KVM_CAP_IRQ_ROUTING =20 --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682556; cv=none; d=zohomail.com; s=zohoarc; b=NbYgftfvDCFzOlVVFagxXRwz98/qRZT7cnZlSAhKZ8+G6PBG7zcKK95avr3NpwiYBZTq8iRcDa3mj0UJ9W+C0udG1rEnrUxUQ3gWu2yrm9wt3fBhKzkl8ObF7XdNFQs8oU0i5A7rg/nWuPt1+8FGJfjS4DnTe9OcgWjEBNivm4Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682556; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=/Q1skwqJamEhJJMNEc5BkiWW2OHQ6d6mDlGtktQruOA=; b=TxlHNaA4Vm0nzZXK/p4MXZRhIXXsDyTDw98XAqHfBw6Iwrw9iqhbiB/tSyymTltEx6OGl8L1OrX1dW+7Kmb/0oUGEfnzFWTJQCTEmssZZdtk8IckSGHDW75eOiykpPpV7GjIElLxtC/bv+aTYAq7CIVSNfizzESfbpH9MGywxEs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1675682556649419.95591123070915; Mon, 6 Feb 2023 03:22:36 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzZp-0004I4-WD; Mon, 06 Feb 2023 06:22:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzZi-0004FT-NB for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:22:20 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzZg-0003Nz-0e for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:22:17 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-160-DutwdtJMP5qQUYAfgfdfIw-1; Mon, 06 Feb 2023 06:21:05 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 28DD5811E9C; Mon, 6 Feb 2023 11:21:05 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5609C492B21; Mon, 6 Feb 2023 11:21:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Q1skwqJamEhJJMNEc5BkiWW2OHQ6d6mDlGtktQruOA=; b=ZYLLrMGxiuKQnsQTlivGsYl/zV7aR++pKJWDNUILmWH+/Qqa/PQOeFhIAxyFi2Yq4FICTw Go4PTb/OzhS00xYkq7qNEOqVKYhU29J/Rx9r5BKRzZ1TDAbp+uz2GNlm9fJCMK+0ZVL9jq DsAxcUJnvSQY2C7fResX/E6vgMZkpNI= X-MC-Unique: DutwdtJMP5qQUYAfgfdfIw-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 2/8] memory: Add last stage indicator to global dirty log synchronization Date: Mon, 6 Feb 2023 19:20:04 +0800 Message-Id: <20230206112010.99871-3-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682558554100003 Content-Type: text/plain; charset="utf-8" The global dirty log synchronization is used when KVM and dirty ring are enabled. There is a particularity for ARM64 where the backup bitmap is used to track dirty pages in non-running-vcpu situations. It means the dirty ring works with the combination of ring buffer and backup bitmap. The dirty bits in the backup bitmap needs to collected in the last stage of live migration. In order to identify the last stage of live migration and pass it down, an extra parameter is added to the relevant functions and callback. This last stage information isn't used yet. No functional change intended. Signed-off-by: Gavin Shan --- accel/kvm/kvm-all.c | 2 +- include/exec/memory.h | 5 +++-- migration/dirtyrate.c | 4 ++-- migration/ram.c | 6 +++--- softmmu/memory.c | 10 +++++----- 5 files changed, 14 insertions(+), 13 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 9b26582655..01a6a026af 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -1554,7 +1554,7 @@ static void kvm_log_sync(MemoryListener *listener, kvm_slots_unlock(); } =20 -static void kvm_log_sync_global(MemoryListener *l) +static void kvm_log_sync_global(MemoryListener *l, bool last_stage) { KVMMemoryListener *kml =3D container_of(l, KVMMemoryListener, listener= ); KVMState *s =3D kvm_state; diff --git a/include/exec/memory.h b/include/exec/memory.h index 2e602a2fad..75b2fd9f48 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -929,8 +929,9 @@ struct MemoryListener { * its @log_sync must be NULL. Vice versa. * * @listener: The #MemoryListener. + * @last_stage: The last stage to synchronize the log during migration */ - void (*log_sync_global)(MemoryListener *listener); + void (*log_sync_global)(MemoryListener *listener, bool last_stage); =20 /** * @log_clear: @@ -2408,7 +2409,7 @@ MemoryRegionSection memory_region_find(MemoryRegion *= mr, * * Synchronizes the dirty page log for all address spaces. */ -void memory_global_dirty_log_sync(void); +void memory_global_dirty_log_sync(bool last_stage); =20 /** * memory_global_dirty_log_sync: synchronize the dirty log for all memory diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index 4bfb97fc68..aecc8142e4 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -101,7 +101,7 @@ void global_dirty_log_change(unsigned int flag, bool st= art) static void global_dirty_log_sync(unsigned int flag, bool one_shot) { qemu_mutex_lock_iothread(); - memory_global_dirty_log_sync(); + memory_global_dirty_log_sync(false); if (one_shot) { memory_global_dirty_log_stop(flag); } @@ -553,7 +553,7 @@ static void calculate_dirtyrate_dirty_bitmap(struct Dir= tyRateConfig config) * skip it unconditionally and start dirty tracking * from 2'round of log sync */ - memory_global_dirty_log_sync(); + memory_global_dirty_log_sync(false); =20 /* * reset page protect manually and unconditionally. diff --git a/migration/ram.c b/migration/ram.c index 334309f1c6..d1b9b270ec 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1188,7 +1188,7 @@ static void migration_bitmap_sync(RAMState *rs) } =20 trace_migration_bitmap_sync_start(); - memory_global_dirty_log_sync(); + memory_global_dirty_log_sync(false); =20 qemu_mutex_lock(&rs->bitmap_mutex); WITH_RCU_READ_LOCK_GUARD() { @@ -3817,7 +3817,7 @@ void colo_incoming_start_dirty_log(void) qemu_mutex_lock_iothread(); qemu_mutex_lock_ramlist(); =20 - memory_global_dirty_log_sync(); + memory_global_dirty_log_sync(false); WITH_RCU_READ_LOCK_GUARD() { RAMBLOCK_FOREACH_NOT_IGNORED(block) { ramblock_sync_dirty_bitmap(ram_state, block); @@ -4114,7 +4114,7 @@ void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 - memory_global_dirty_log_sync(); + memory_global_dirty_log_sync(false); WITH_RCU_READ_LOCK_GUARD() { RAMBLOCK_FOREACH_NOT_IGNORED(block) { ramblock_sync_dirty_bitmap(ram_state, block); diff --git a/softmmu/memory.c b/softmmu/memory.c index 9d64efca26..1cc36ef028 100644 --- a/softmmu/memory.c +++ b/softmmu/memory.c @@ -2224,7 +2224,7 @@ void memory_region_set_dirty(MemoryRegion *mr, hwaddr= addr, * If memory region `mr' is NULL, do global sync. Otherwise, sync * dirty bitmap for the specified memory region. */ -static void memory_region_sync_dirty_bitmap(MemoryRegion *mr) +static void memory_region_sync_dirty_bitmap(MemoryRegion *mr, bool last_st= age) { MemoryListener *listener; AddressSpace *as; @@ -2254,7 +2254,7 @@ static void memory_region_sync_dirty_bitmap(MemoryReg= ion *mr) * is to do a global sync, because we are not capable to * sync in a finer granularity. */ - listener->log_sync_global(listener); + listener->log_sync_global(listener, last_stage); trace_memory_region_sync_dirty(mr ? mr->name : "(all)", listen= er->name, 1); } } @@ -2318,7 +2318,7 @@ DirtyBitmapSnapshot *memory_region_snapshot_and_clear= _dirty(MemoryRegion *mr, { DirtyBitmapSnapshot *snapshot; assert(mr->ram_block); - memory_region_sync_dirty_bitmap(mr); + memory_region_sync_dirty_bitmap(mr, false); snapshot =3D cpu_physical_memory_snapshot_and_clear_dirty(mr, addr, si= ze, client); memory_global_after_dirty_log_sync(); return snapshot; @@ -2844,9 +2844,9 @@ bool memory_region_present(MemoryRegion *container, h= waddr addr) return mr && mr !=3D container; } =20 -void memory_global_dirty_log_sync(void) +void memory_global_dirty_log_sync(bool last_stage) { - memory_region_sync_dirty_bitmap(NULL); + memory_region_sync_dirty_bitmap(NULL, last_stage); } =20 void memory_global_after_dirty_log_sync(void) --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682622; cv=none; d=zohomail.com; s=zohoarc; b=PqAyR6e2XDfPnatoWUs9Vjv/LEUo1UIIiguRnCa3YSK/N9hQ+S3zmV8hD54wEfkUfMh0Pc8PGa82aIE1RxRKVzLLlfZcIRRj5PDXD6j0CXnWtMrZfm1Bhpqaa8JWj/ViT88PTY/WoyEE+24DlfJz1qwxzdkwtQ8CgqZ2Aoo/QcI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682622; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=qtsL8ERd4ZdT7V/YvGYXethl/MqAVLTyFEEwTNfoqy4=; b=iKfmudFpNVgH7ElqFvkJ8bCihdxts1taCyKSQP74xtXwOh4Wcvpu192XJ6ptvr36Fdw4N8gtNvjsoJcE6gtF3tYrH+Zjrbi4pJdDz7YpkfhvdyBi7RLorBHRXDl8SvP96Ce7Ln5W3ur9ZQ8+UZcxP79IAHVeAEzBNuR2mURLYDo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1675682622788535.1359358477372; Mon, 6 Feb 2023 03:23:42 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzZr-0004IZ-CX; Mon, 06 Feb 2023 06:22:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzZm-0004Fn-Bb for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:22:25 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzZj-0003Ob-0B for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:22:21 -0500 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-616-Bx8nWHSlOYuyHWpfgOyq6g-1; Mon, 06 Feb 2023 06:21:10 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6E51F1C06EC2; Mon, 6 Feb 2023 11:21:10 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id BAD8A492B21; Mon, 6 Feb 2023 11:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qtsL8ERd4ZdT7V/YvGYXethl/MqAVLTyFEEwTNfoqy4=; b=MmaIOC3AXcZ51mHErx1eYNcvrO2RNC9mgwYNpJ+K7GzT3AMotcCIyIV32USKmsqOwavWlI hUkkD8kHapXQTfw+vhWtGRYbRyf5y6683W4uK7Yk8nTZSRf5Y7viaTx8BND0QXM+kIsIqK Yd552Jn6JhR4Z8FKrOxrunn+NsHnAw0= X-MC-Unique: Bx8nWHSlOYuyHWpfgOyq6g-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 3/8] migration: Add last stage indicator to global dirty log synchronization Date: Mon, 6 Feb 2023 19:20:05 +0800 Message-Id: <20230206112010.99871-4-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682623293100001 Content-Type: text/plain; charset="utf-8" For the pre-copy live migration scenario, the last stage indicator is needed for KVM backend to collect the dirty pages from the backup bitmap when dirty ring is used. The indicator isn't used so far. No functional change intended. Signed-off-by: Gavin Shan --- migration/ram.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index d1b9b270ec..6c9642edee 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1176,7 +1176,7 @@ static void migration_trigger_throttle(RAMState *rs) } } =20 -static void migration_bitmap_sync(RAMState *rs) +static void migration_bitmap_sync(RAMState *rs, bool last_stage) { RAMBlock *block; int64_t end_time; @@ -1188,7 +1188,7 @@ static void migration_bitmap_sync(RAMState *rs) } =20 trace_migration_bitmap_sync_start(); - memory_global_dirty_log_sync(false); + memory_global_dirty_log_sync(last_stage); =20 qemu_mutex_lock(&rs->bitmap_mutex); WITH_RCU_READ_LOCK_GUARD() { @@ -1222,7 +1222,7 @@ static void migration_bitmap_sync(RAMState *rs) } } =20 -static void migration_bitmap_sync_precopy(RAMState *rs) +static void migration_bitmap_sync_precopy(RAMState *rs, bool last_stage) { Error *local_err =3D NULL; =20 @@ -1235,7 +1235,7 @@ static void migration_bitmap_sync_precopy(RAMState *r= s) local_err =3D NULL; } =20 - migration_bitmap_sync(rs); + migration_bitmap_sync(rs, last_stage); =20 if (precopy_notify(PRECOPY_NOTIFY_AFTER_BITMAP_SYNC, &local_err)) { error_report_err(local_err); @@ -2844,7 +2844,7 @@ void ram_postcopy_send_discard_bitmap(MigrationState = *ms) RCU_READ_LOCK_GUARD(); =20 /* This should be our last sync, the src is now paused */ - migration_bitmap_sync(rs); + migration_bitmap_sync(rs, false); =20 /* Easiest way to make sure we don't resume in the middle of a host-pa= ge */ rs->pss[RAM_CHANNEL_PRECOPY].last_sent_block =3D NULL; @@ -3034,7 +3034,7 @@ static void ram_init_bitmaps(RAMState *rs) /* We don't use dirty log with background snapshots */ if (!migrate_background_snapshot()) { memory_global_dirty_log_start(GLOBAL_DIRTY_MIGRATION); - migration_bitmap_sync_precopy(rs); + migration_bitmap_sync_precopy(rs, false); } } qemu_mutex_unlock_ramlist(); @@ -3349,7 +3349,7 @@ static int ram_save_complete(QEMUFile *f, void *opaqu= e) =20 WITH_RCU_READ_LOCK_GUARD() { if (!migration_in_postcopy()) { - migration_bitmap_sync_precopy(rs); + migration_bitmap_sync_precopy(rs, true); } =20 ram_control_before_iterate(f, RAM_CONTROL_FINISH); @@ -3407,7 +3407,7 @@ static void ram_save_pending(QEMUFile *f, void *opaqu= e, uint64_t max_size, remaining_size < max_size) { qemu_mutex_lock_iothread(); WITH_RCU_READ_LOCK_GUARD() { - migration_bitmap_sync_precopy(rs); + migration_bitmap_sync_precopy(rs, false); } qemu_mutex_unlock_iothread(); remaining_size =3D rs->migration_dirty_pages * TARGET_PAGE_SIZE; --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682599; cv=none; d=zohomail.com; s=zohoarc; b=DNZ8heYI9zGesOWv9gOd5oBRyQ8Ym/Ndt7DvP1jqU8AxBgxDatZG+LhHJnFr8H8zdkoDrEbgGDLkkdItSjxCi9lywbSna1OTWD4uzzmt8m+YtnxOVeeIXNjZPDwmTjkdhmKi233HNkesrIJfCBMDwqU8KkRkGL7SAR9lCYEfrWo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682599; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=+pBZkvwO/MSacYruR8vBHavjUuE24O9czHcjOtXLm1A=; b=EJXbm0yMzn3q18dfa3ksauoEKwau2MIgmuEKuOXBG8yktqSeMRsb+lMZeXFkx50jlge+s+w8j4KRd9sHkjaK0kG0ulxE6uz2bgEu2o1+DEFAmw9fklKLHvE43BbTJNHopoPCObmEjqVaZwAoREJ44AhaFR19yf369x4vTxGr1vE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1675682599426288.04754354883744; Mon, 6 Feb 2023 03:23:19 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzaS-00053N-SR; Mon, 06 Feb 2023 06:23:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaR-0004zl-0v for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:03 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaP-0003XM-71 for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:02 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-90-l4xcpkjLNuORLG3vQhMfmw-1; Mon, 06 Feb 2023 06:21:22 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D65A41875041; Mon, 6 Feb 2023 11:21:21 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id F04E2492B21; Mon, 6 Feb 2023 11:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+pBZkvwO/MSacYruR8vBHavjUuE24O9czHcjOtXLm1A=; b=Q1Czvk/7vagchrxbSnhgWCIoyK5Suq8AkFWKnnIr4XaIDmahcGaEbm0Ni6jtoR7Nx2dfQU eSOutn+TkFwZfSeQ82sEVrinjJHyKzVulayolZNFhKR09B0HcEZpDMRBeG3gnGOkfy0SJx xkFp/XeJe6Qd+pckEr3cKVe768YOc5w= X-MC-Unique: l4xcpkjLNuORLG3vQhMfmw-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 4/8] kvm: Introduce secondary dirty bitmap Date: Mon, 6 Feb 2023 19:20:06 +0800 Message-Id: <20230206112010.99871-5-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682600922100009 Content-Type: text/plain; charset="utf-8" When dirty ring is enabled on ARM64, the backup bitmap may be used to track the dirty pages in no-running-vcpu situations. The original bitmap is the primary one, used for the dirty ring buffer. We need the secondary bitmap to collect the backup bitmap for ARM64. No functional change intended. Signed-off-by: Gavin Shan --- accel/kvm/kvm-all.c | 50 ++++++++++++++++++++++++++++++---------- include/sysemu/kvm_int.h | 1 + 2 files changed, 39 insertions(+), 12 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 01a6a026af..1a93985574 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -553,13 +553,29 @@ static void kvm_log_stop(MemoryListener *listener, } } =20 +static unsigned long *kvm_slot_dirty_bitmap(KVMSlot *slot, bool primary) +{ + if (primary) { + return slot->dirty_bmap; + } + + return slot->dirty_bmap + + slot->dirty_bmap_size / sizeof(slot->dirty_bmap[0]); +} + /* get kvm's dirty pages bitmap and update qemu's */ -static void kvm_slot_sync_dirty_pages(KVMSlot *slot) +static void kvm_slot_sync_dirty_pages(KVMSlot *slot, bool primary) { + KVMState *s =3D kvm_state; + unsigned long *bmap =3D kvm_slot_dirty_bitmap(slot, primary); ram_addr_t start =3D slot->ram_start_offset; ram_addr_t pages =3D slot->memory_size / qemu_real_host_page_size(); =20 - cpu_physical_memory_set_dirty_lebitmap(slot->dirty_bmap, start, pages); + if (!s->kvm_dirty_ring_with_bitmap && !primary) { + return; + } + + cpu_physical_memory_set_dirty_lebitmap(bmap, start, pages); } =20 static void kvm_slot_reset_dirty_pages(KVMSlot *slot) @@ -572,6 +588,9 @@ static void kvm_slot_reset_dirty_pages(KVMSlot *slot) /* Allocate the dirty bitmap for a slot */ static void kvm_slot_init_dirty_bitmap(KVMSlot *mem) { + KVMState *s =3D kvm_state; + hwaddr bitmap_size, alloc_size; + if (!(mem->flags & KVM_MEM_LOG_DIRTY_PAGES) || mem->dirty_bmap) { return; } @@ -593,9 +612,11 @@ static void kvm_slot_init_dirty_bitmap(KVMSlot *mem) * And mem->memory_size is aligned to it (otherwise this mem can't * be registered to KVM). */ - hwaddr bitmap_size =3D ALIGN(mem->memory_size / qemu_real_host_page_si= ze(), - /*HOST_LONG_BITS*/ 64) / 8; - mem->dirty_bmap =3D g_malloc0(bitmap_size); + bitmap_size =3D ALIGN(mem->memory_size / qemu_real_host_page_size(), + /*HOST_LONG_BITS*/ 64) / 8; + alloc_size =3D s->kvm_dirty_ring_with_bitmap ? 2 * bitmap_size : bitma= p_size; + + mem->dirty_bmap =3D g_malloc0(alloc_size); mem->dirty_bmap_size =3D bitmap_size; } =20 @@ -603,12 +624,16 @@ static void kvm_slot_init_dirty_bitmap(KVMSlot *mem) * Sync dirty bitmap from kernel to KVMSlot.dirty_bmap, return true if * succeeded, false otherwise */ -static bool kvm_slot_get_dirty_log(KVMState *s, KVMSlot *slot) +static bool kvm_slot_get_dirty_log(KVMState *s, KVMSlot *slot, bool primar= y) { struct kvm_dirty_log d =3D {}; int ret; =20 - d.dirty_bitmap =3D slot->dirty_bmap; + if (!s->kvm_dirty_ring_with_bitmap && !primary) { + return false; + } + + d.dirty_bitmap =3D kvm_slot_dirty_bitmap(slot, primary); d.slot =3D slot->slot | (slot->as_id << 16); ret =3D kvm_vm_ioctl(s, KVM_GET_DIRTY_LOG, &d); =20 @@ -839,8 +864,8 @@ static void kvm_physical_sync_dirty_bitmap(KVMMemoryLis= tener *kml, /* We don't have a slot if we want to trap every access. */ return; } - if (kvm_slot_get_dirty_log(s, mem)) { - kvm_slot_sync_dirty_pages(mem); + if (kvm_slot_get_dirty_log(s, mem, true)) { + kvm_slot_sync_dirty_pages(mem, true); } start_addr +=3D slot_size; size -=3D slot_size; @@ -1353,9 +1378,9 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml, if (kvm_state->kvm_dirty_ring_size) { kvm_dirty_ring_reap_locked(kvm_state, NULL); } else { - kvm_slot_get_dirty_log(kvm_state, mem); + kvm_slot_get_dirty_log(kvm_state, mem, true); } - kvm_slot_sync_dirty_pages(mem); + kvm_slot_sync_dirty_pages(mem, true); } =20 /* unregister the slot */ @@ -1572,7 +1597,7 @@ static void kvm_log_sync_global(MemoryListener *l, bo= ol last_stage) for (i =3D 0; i < s->nr_slots; i++) { mem =3D &kml->slots[i]; if (mem->memory_size && mem->flags & KVM_MEM_LOG_DIRTY_PAGES) { - kvm_slot_sync_dirty_pages(mem); + kvm_slot_sync_dirty_pages(mem, true); /* * This is not needed by KVM_GET_DIRTY_LOG because the * ioctl will unconditionally overwrite the whole region. @@ -3701,6 +3726,7 @@ static void kvm_accel_instance_init(Object *obj) s->kernel_irqchip_split =3D ON_OFF_AUTO_AUTO; /* KVM dirty ring is by default off */ s->kvm_dirty_ring_size =3D 0; + s->kvm_dirty_ring_with_bitmap =3D false; s->notify_vmexit =3D NOTIFY_VMEXIT_OPTION_RUN; s->notify_window =3D 0; } diff --git a/include/sysemu/kvm_int.h b/include/sysemu/kvm_int.h index 60b520a13e..fdd5b1bde0 100644 --- a/include/sysemu/kvm_int.h +++ b/include/sysemu/kvm_int.h @@ -115,6 +115,7 @@ struct KVMState } *as; uint64_t kvm_dirty_ring_bytes; /* Size of the per-vcpu dirty ring */ uint32_t kvm_dirty_ring_size; /* Number of dirty GFNs per ring */ + bool kvm_dirty_ring_with_bitmap; struct KVMDirtyRingReaper reaper; NotifyVmexitOption notify_vmexit; uint32_t notify_window; --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682634; cv=none; d=zohomail.com; s=zohoarc; b=hO0PvkoftDWqRGpn2aRq70mFUiyDbAz2OfKDV5kn79XFvaSKvoWZOxnIHvTo2EA3ki8XFBscXoF9b9mUZXRpUB60rd/cYG6bBZmnGNp55Eub1MreqI1Hw9FQ6TBSViNpCn2G0mvRfEIL/dfHi5EjXM3L/ZgnxWrb8ah4SOuKYh4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682634; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=AV1dHV/DKMAbDep2jVZbQdk/CCup6inKTxE45seX6JQ=; b=gr07dE8pjnrfhsuPGgpR0BussNPrsg0aqhkUFhYlXe79xNw+OAgjrweTkxMVmK5l0xlm3qYLkKAqhGpl7Kc9ewkfzdO7CbktyXBiqG9SrTSBjU6734g10lu5tZ+cXzbgF+eET3NlSY4cCALIp1ZnaRAmpiwcqYXXT6vRQhVr80I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 167568263421419.70314147867714; Mon, 6 Feb 2023 03:23:54 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzaW-0005K8-Lx; Mon, 06 Feb 2023 06:23:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaU-0005DA-Ae for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:06 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaP-0003Xr-M8 for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:06 -0500 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-363-M3c4unt1NFyNVhKtRVjbGw-1; Mon, 06 Feb 2023 06:21:27 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5EC891C05151; Mon, 6 Feb 2023 11:21:27 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F46B492B21; Mon, 6 Feb 2023 11:21:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682581; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AV1dHV/DKMAbDep2jVZbQdk/CCup6inKTxE45seX6JQ=; b=ZBGclPXnqZPZewW0+0PvCEgIuwKLeFsbcLynMlIzRDwV/5GSaY6yprhLc3+QEiSZ3xMul7 TZwSJKcE4R9BIPwnvNSlKa1RLHZPIwxgn/2ApCH4n1lY1mp6SshCB09lKm/whZsiGYBT6b /1uO1SaCTrXREKxkrkkfgL4+uj59mCY= X-MC-Unique: M3c4unt1NFyNVhKtRVjbGw-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 5/8] kvm: Synchronize secondary bitmap in last stage Date: Mon, 6 Feb 2023 19:20:07 +0800 Message-Id: <20230206112010.99871-6-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682635354100003 Content-Type: text/plain; charset="utf-8" In the last stage of live migration or memory slot removal, the backup bitmap needs to be synchronized. Signed-off-by: Gavin Shan --- accel/kvm/kvm-all.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 1a93985574..9ec117c441 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -1377,10 +1377,12 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml, */ if (kvm_state->kvm_dirty_ring_size) { kvm_dirty_ring_reap_locked(kvm_state, NULL); + kvm_slot_get_dirty_log(kvm_state, mem, false); } else { kvm_slot_get_dirty_log(kvm_state, mem, true); } kvm_slot_sync_dirty_pages(mem, true); + kvm_slot_sync_dirty_pages(mem, false); } =20 /* unregister the slot */ @@ -1604,6 +1606,11 @@ static void kvm_log_sync_global(MemoryListener *l, b= ool last_stage) * However kvm dirty ring has no such side effect. */ kvm_slot_reset_dirty_pages(mem); + + if (s->kvm_dirty_ring_with_bitmap && last_stage && + kvm_slot_get_dirty_log(s, mem, false)) { + kvm_slot_sync_dirty_pages(mem, false); + } } } kvm_slots_unlock(); --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682657; cv=none; d=zohomail.com; s=zohoarc; b=blQM+4x+LNyzc7fbCzjphhHmopFXdFuWq2XNepyWqt00DFq+NVwReVzW6mhlJdRh/p7aMiT7JFUJcRBC40Kzqcsx8NWNo4NzKG5ARENGPDF0a1ZMaR0MOx5U39MGxS+1GxoRAvhT9ooaJRG4J7HKLadbLVZY/UInkR0hQAKaBVU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682657; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=S7zj9j3WG5yOw3N8S6DF7Pu5F0DlcQE5xV5NIX1uE+g=; b=gBb9VYhlX2ptNrSCzzMym6KjNjaFmLILL1xlpXkAZmljNicUgJeDeaEGqi5abfPghMeVkhwdpSV0eBOfQHAxuVktonsJ3ypUXDBs5YLCkZi//q4nxT5IWEZ6Yyo9Hth7JUpNfdLylcz+2xhxTKOHChzb3wlVDMHfr1y844zSCMA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 167568265798124.61267244300973; Mon, 6 Feb 2023 03:24:17 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzaV-0005FS-2Y; Mon, 06 Feb 2023 06:23:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaS-00052v-Gt for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:04 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaP-0003XT-DG for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:04 -0500 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-388-csO-Z1grO3ugrvZ2gXhW5A-1; Mon, 06 Feb 2023 06:21:36 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE19B10395C6; Mon, 6 Feb 2023 11:21:32 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id E54B2492B21; Mon, 6 Feb 2023 11:21:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S7zj9j3WG5yOw3N8S6DF7Pu5F0DlcQE5xV5NIX1uE+g=; b=flgD1Jj3tWO/cluuxxFRhKlF1JCOPNVJUsfOGvu0F0Fu/bcp2xwbwWnwxxUgYZeaPSRG6/ G+sHld/aCOAgaAWhzKpa1zfjAJODIY5Cf4nOtvhz4nqB2cr2mAb7rl3lPXms7bWReTNQQ1 5ES2AhprNVv9caSXJyFwu/SEWYcgpMQ= X-MC-Unique: csO-Z1grO3ugrvZ2gXhW5A-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 6/8] kvm: Add helper kvm_dirty_ring_init() Date: Mon, 6 Feb 2023 19:20:08 +0800 Message-Id: <20230206112010.99871-7-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682659507100003 Content-Type: text/plain; charset="utf-8" Due to multiple capabilities associated with the dirty ring for different architectures: KVM_CAP_DIRTY_{LOG_RING, LOG_RING_ACQ_REL} for x86 and arm64 separately. There will be more to be done in order to support the dirty ring for arm64. Lets add helper kvm_dirty_ring_init() to enable the dirty ring. With this, the code looks a bit clean. No functional change intended. Signed-off-by: Gavin Shan --- accel/kvm/kvm-all.c | 73 ++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 27 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 9ec117c441..399ef0f7e6 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -1476,6 +1476,49 @@ static int kvm_dirty_ring_reaper_init(KVMState *s) return 0; } =20 +static int kvm_dirty_ring_init(KVMState *s) +{ + uint64_t ring_bytes; + int ret; + + /* + * Read the max supported pages. Fall back to dirty logging mode + * if the dirty ring isn't supported. + */ + ret =3D kvm_vm_check_extension(s, KVM_CAP_DIRTY_LOG_RING); + if (ret <=3D 0) { + warn_report("KVM dirty ring not available, using bitmap method"); + s->kvm_dirty_ring_size =3D 0; + return 0; + } + + ring_bytes =3D s->kvm_dirty_ring_size * sizeof(struct kvm_dirty_gfn); + if (ring_bytes > ret) { + error_report("KVM dirty ring size %" PRIu32 " too big " + "(maximum is %ld). Please use a smaller value.", + s->kvm_dirty_ring_size, + (long)ret / sizeof(struct kvm_dirty_gfn)); + ret =3D -EINVAL; + goto out; + } + + ret =3D kvm_vm_enable_cap(s, KVM_CAP_DIRTY_LOG_RING, 0, ring_bytes); + if (ret) { + error_report("Enabling of KVM dirty ring failed: %s. " + "Suggested minimum value is 1024.", strerror(-ret)); + ret =3D -EIO; + } + +out: + if (ret) { + s->kvm_dirty_ring_size =3D 0; + } else { + s->kvm_dirty_ring_bytes =3D ring_bytes; + } + + return ret; +} + static void kvm_region_add(MemoryListener *listener, MemoryRegionSection *section) { @@ -2545,33 +2588,9 @@ static int kvm_init(MachineState *ms) * dirty logging mode */ if (s->kvm_dirty_ring_size > 0) { - uint64_t ring_bytes; - - ring_bytes =3D s->kvm_dirty_ring_size * sizeof(struct kvm_dirty_gf= n); - - /* Read the max supported pages */ - ret =3D kvm_vm_check_extension(s, KVM_CAP_DIRTY_LOG_RING); - if (ret > 0) { - if (ring_bytes > ret) { - error_report("KVM dirty ring size %" PRIu32 " too big " - "(maximum is %ld). Please use a smaller valu= e.", - s->kvm_dirty_ring_size, - (long)ret / sizeof(struct kvm_dirty_gfn)); - ret =3D -EINVAL; - goto err; - } - - ret =3D kvm_vm_enable_cap(s, KVM_CAP_DIRTY_LOG_RING, 0, ring_b= ytes); - if (ret) { - error_report("Enabling of KVM dirty ring failed: %s. " - "Suggested minimum value is 1024.", strerror(= -ret)); - goto err; - } - - s->kvm_dirty_ring_bytes =3D ring_bytes; - } else { - warn_report("KVM dirty ring not available, using bitmap metho= d"); - s->kvm_dirty_ring_size =3D 0; + ret =3D kvm_dirty_ring_init(s); + if (ret < 0) { + goto err; } } =20 --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682647; cv=none; d=zohomail.com; s=zohoarc; b=lUUNK8+DM0RDLYdnKvVwgcvc7jvX3Y4ciELLe5Dhr70Yw0S+pK8SsmO4BySY+9oJ1KBBjpMn0He5NO5m2nXK7hJ4RLGr/Bv3kPlUdB/1TrMTy/9+jvgYPiph9VGGK8Pt9PjQ0TS5qYSKHt0NAjELHbsZmQobZHW/1sQgTNmIXgI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682647; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=URENfRLE4MQd654yowZvE5OyBk8WhhSip8pYfRKwdxs=; b=hpZuKKvs3t/ZGJy239uErQpZzmqeOsQ+6jipVSFmQ6jyA99QNDlryj6s0LCdTi+CxoRaIRriCoZ+ZAL/UMoCrDQRPPRHhLHX6q2xSJvZXltViZm0FVfx9b29ihbVbM8uTw0vaH7xLmaLkNUy5aPlUL+UVayiQit4xst69qJ9nXo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 167568264707816.43999530464737; Mon, 6 Feb 2023 03:24:07 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzaV-0005H6-HE; Mon, 06 Feb 2023 06:23:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaT-0005Au-NH for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:05 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaP-0003Xe-M1 for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:05 -0500 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-281-6PszlWFZOBWBDUH4zbgUTQ-1; Mon, 06 Feb 2023 06:21:38 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3B55C3C0E45A; Mon, 6 Feb 2023 11:21:38 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4BE9E492B21; Mon, 6 Feb 2023 11:21:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=URENfRLE4MQd654yowZvE5OyBk8WhhSip8pYfRKwdxs=; b=RrC5AABQ74Dl4WKJt+X+kZXgi4rnYNyFnDob2TfAjSf+XHN2ci0OCa+F6xK2ge/+bHHRCv B5l3hVDCUkdny1YDHE1rrDTcfYEcQzkl7vQNHGPELiCQI0Wjll+Oe1ZxvVHNqtSuR3vGla STqZjDdnL33YRWFd+/zJyt6/48koBVE= X-MC-Unique: 6PszlWFZOBWBDUH4zbgUTQ-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 7/8] hw/arm/virt: Enable backup bitmap for dirty ring Date: Mon, 6 Feb 2023 19:20:09 +0800 Message-Id: <20230206112010.99871-8-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682647458100001 Content-Type: text/plain; charset="utf-8" When KVM device "kvm-arm-gicv3" or "arm-its-kvm" is used, we have to enable the backup bitmap for the dirty ring. Otherwise, the migration will fail because those two devices are using the backup bitmap to track dirty guest memory, corresponding to various hardware tables. Signed-off-by: Gavin Shan Reviewed-by: Juan Quintela --- hw/arm/virt.c | 26 ++++++++++++++++++++++++++ target/arm/kvm64.c | 25 +++++++++++++++++++++++++ target/arm/kvm_arm.h | 12 ++++++++++++ 3 files changed, 63 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index ba47728288..b91b5972bc 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2025,6 +2025,8 @@ static void machvirt_init(MachineState *machine) VirtMachineClass *vmc =3D VIRT_MACHINE_GET_CLASS(machine); MachineClass *mc =3D MACHINE_GET_CLASS(machine); const CPUArchIdList *possible_cpus; + const char *gictype =3D NULL; + const char *itsclass =3D NULL; MemoryRegion *sysmem =3D get_system_memory(); MemoryRegion *secure_sysmem =3D NULL; MemoryRegion *tag_sysmem =3D NULL; @@ -2072,6 +2074,30 @@ static void machvirt_init(MachineState *machine) */ finalize_gic_version(vms); =20 + /* + * When "kvm-arm-gicv3" or "arm-its-kvm" is used, the backup dirty + * bitmap has to be enabled for KVM dirty ring, before any memory + * slot is added. Otherwise, the migration will fail with the dirty + * ring. + */ + if (kvm_enabled()) { + if (vms->gic_version !=3D VIRT_GIC_VERSION_2) { + gictype =3D gicv3_class_name(); + } + + if (vms->gic_version !=3D VIRT_GIC_VERSION_2 && vms->its) { + itsclass =3D its_class_name(); + } + + if (((gictype && !strcmp(gictype, "kvm-arm-gicv3")) || + (itsclass && !strcmp(itsclass, "arm-its-kvm"))) && + !kvm_arm_enable_dirty_ring_with_bitmap()) { + error_report("Failed to enable the backup bitmap for " + "KVM dirty ring"); + exit(1); + } + } + if (vms->secure) { /* * The Secure view of the world is the same as the NonSecure, diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c index 1197253d12..91960e1cd3 100644 --- a/target/arm/kvm64.c +++ b/target/arm/kvm64.c @@ -764,6 +764,31 @@ bool kvm_arm_steal_time_supported(void) return kvm_check_extension(kvm_state, KVM_CAP_STEAL_TIME); } =20 +bool kvm_arm_enable_dirty_ring_with_bitmap(void) +{ + int ret; + + /* No need to enable the backup bitmap if dirty ring isn't enabled */ + if (!kvm_dirty_ring_enabled()) { + return true; + } + + ret =3D kvm_check_extension(kvm_state, KVM_CAP_DIRTY_LOG_RING_WITH_BIT= MAP); + if (!ret) { + return false; + } + + ret =3D kvm_vm_enable_cap(kvm_state, KVM_CAP_DIRTY_LOG_RING_WITH_BITMA= P, + 0, 1); + if (ret) { + return false; + } + + kvm_state->kvm_dirty_ring_with_bitmap =3D true; + + return true; +} + QEMU_BUILD_BUG_ON(KVM_ARM64_SVE_VQ_MIN !=3D 1); =20 uint32_t kvm_arm_sve_get_vls(CPUState *cs) diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index 99017b635c..402281e61a 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -282,6 +282,13 @@ void kvm_arm_steal_time_finalize(ARMCPU *cpu, Error **= errp); */ bool kvm_arm_steal_time_supported(void); =20 +/** + * kvm_arm_enable_dirty_ring_with_bitmap: + * Returns: true if KVM dirty ring's backup bitmap is enabled + * and false otherwise. + */ +bool kvm_arm_enable_dirty_ring_with_bitmap(void); + /** * kvm_arm_aarch32_supported: * @@ -395,6 +402,11 @@ static inline bool kvm_arm_steal_time_supported(void) return false; } =20 +static inline bool kvm_arm_enable_dirty_ring_with_bitmap(void) +{ + return false; +} + /* * These functions should never actually be called without KVM support. */ --=20 2.23.0 From nobody Fri May 17 03:13:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1675682598; cv=none; d=zohomail.com; s=zohoarc; b=CQLvMHR25gLOvcl35UihYWwvlbzjcIrwLNdLasEFqvMx+TNWj1i0ZGzzVwKbQj5T5Ue3rwnamwmEavMS18JxgUF8gA5gsOr8Qm3vPFsFohy1U/ccks11G4Vg9jmmMHiS69I9HVRzCKiKpcz4fuwZ/xwnFTWmZAQy0TqFYyXjEKk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675682598; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ezT0/aoBW/YY0l1s0UAK7lpF6HyMwc+sYc1e4IW+uoI=; b=YJ7HFsp1Piqc5TcqbLegOrvlARSQyN6LVm3X39ZKOgQluEtjkpIGEkDGgm3T7FXpa9UYPf5EWnF/DBdKCBCglHgcMvKZazht8T2HMQjUog4A2x7yfxSuPN9SC0NQx0HpsTDyFsnE/6n4Jz1JiZxJETPWpxtc1zZa3AwRdENK4+s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 167568259885226.04964571764583; Mon, 6 Feb 2023 03:23:18 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pOzaU-0005Cr-CP; Mon, 06 Feb 2023 06:23:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaS-00052j-Ev for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:04 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pOzaP-0003Xa-E4 for qemu-devel@nongnu.org; Mon, 06 Feb 2023 06:23:04 -0500 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-597-I_iCKIeoN4eCqfiLZIGa2g-1; Mon, 06 Feb 2023 06:21:45 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8B951C08964; Mon, 6 Feb 2023 11:21:44 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-63.bne.redhat.com [10.64.54.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id C0EA2492B23; Mon, 6 Feb 2023 11:21:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675682580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ezT0/aoBW/YY0l1s0UAK7lpF6HyMwc+sYc1e4IW+uoI=; b=INXrfSNpMwxYSkSvuoU7HGck0BAEY3X9m43YkAMMxb4auZkY1tPKT3bjT3csjRy0ydUAEa vYaPV37Ucq9GNcfpLCFpyDTnTQeUthN5il7zxgQO0U0fdI47NnDi/Z57RyCKY2dXmvR12I 2icI15DFJbJRl8ct+qB+/z8oU1toO2U= X-MC-Unique: I_iCKIeoN4eCqfiLZIGa2g-1 From: Gavin Shan To: qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, pbonzini@redhat.com, peter.maydell@linaro.org, peterx@redhat.com, david@redhat.com, philmd@linaro.org, mst@redhat.com, cohuck@redhat.com, quintela@redhat.com, dgilbert@redhat.com, maz@kernel.org, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: [PATCH RFCv1 8/8] kvm: Enable dirty ring for arm64 Date: Mon, 6 Feb 2023 19:20:10 +0800 Message-Id: <20230206112010.99871-9-gshan@redhat.com> In-Reply-To: <20230206112010.99871-1-gshan@redhat.com> References: <20230206112010.99871-1-gshan@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1675682600900100008 Content-Type: text/plain; charset="utf-8" arm64 has different capability from x86 to enable the dirty ring, which is KVM_CAP_DIRTY_LOG_RING_ACQ_REL. To enable it in kvm_dirty_ring_init() when KVM_CAP_DIRTY_LOG_RING isn't supported. Signed-off-by: Gavin Shan Reviewed-by: Juan Quintela --- accel/kvm/kvm-all.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 399ef0f7e6..8ab31865eb 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -1479,13 +1479,19 @@ static int kvm_dirty_ring_reaper_init(KVMState *s) static int kvm_dirty_ring_init(KVMState *s) { uint64_t ring_bytes; + unsigned int capability =3D KVM_CAP_DIRTY_LOG_RING; int ret; =20 /* * Read the max supported pages. Fall back to dirty logging mode * if the dirty ring isn't supported. */ - ret =3D kvm_vm_check_extension(s, KVM_CAP_DIRTY_LOG_RING); + ret =3D kvm_vm_check_extension(s, capability); + if (ret <=3D 0) { + capability =3D KVM_CAP_DIRTY_LOG_RING_ACQ_REL; + ret =3D kvm_vm_check_extension(s, capability); + } + if (ret <=3D 0) { warn_report("KVM dirty ring not available, using bitmap method"); s->kvm_dirty_ring_size =3D 0; @@ -1502,7 +1508,7 @@ static int kvm_dirty_ring_init(KVMState *s) goto out; } =20 - ret =3D kvm_vm_enable_cap(s, KVM_CAP_DIRTY_LOG_RING, 0, ring_bytes); + ret =3D kvm_vm_enable_cap(s, capability, 0, ring_bytes); if (ret) { error_report("Enabling of KVM dirty ring failed: %s. " "Suggested minimum value is 1024.", strerror(-ret)); --=20 2.23.0