From nobody Mon Feb 9 09:28:27 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1584095613; cv=none; d=zohomail.com; s=zohoarc; b=Q44JILydExFcrrX2SWKjVo0B4BtE5zvGYKPNRvZyeL3JoVrcUSpcITzZ+k7pIKqW3puV+IOSg34Ed+WqBr98Euvhk9wdCCvUJhSiXB5ZAdNE0UHbJGj3SHJMuIgHR2M+MP9/4KgytJQ51UH/z1+tYlBENXEndOI7wOqKoxkIe8w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584095613; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=fciT879zSkhcAGmUXRf98FMw7LxC/2yx1e2Pv/uU6aM=; b=MPiP1EBR82Lf3GsUMXNQUbNN+K9qv7SvlTqRMDF7+7cr2/QcVt+eDFEq+5YXYUBJTWCORb7S9qrZg+9SWg/3Ns4xDwPz625/ZziSV8pJJERa/lqvqvZQCFortdUppFOdnG6TJU2ODuJD9ZmJXt5M9SgLYTphIfx6Jw8k6NuVQGE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584095613855416.55209439273517; Fri, 13 Mar 2020 03:33:33 -0700 (PDT) Received: from localhost ([::1]:56686 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jChdE-00032D-S1 for importer@patchew.org; Fri, 13 Mar 2020 06:33:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39014) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jChae-0008MD-R2 for qemu-devel@nongnu.org; Fri, 13 Mar 2020 06:30:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jChad-0005wq-1f for qemu-devel@nongnu.org; Fri, 13 Mar 2020 06:30:52 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:22709 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jChac-0005ud-TP for qemu-devel@nongnu.org; Fri, 13 Mar 2020 06:30:50 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-58-gFIuVPbOMPacTUoVdhessQ-1; Fri, 13 Mar 2020 06:30:48 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7BEB4107ACCA; Fri, 13 Mar 2020 10:30:47 +0000 (UTC) Received: from dgilbert-t580.localhost (ovpn-117-92.ams2.redhat.com [10.36.117.92]) by smtp.corp.redhat.com (Postfix) with ESMTP id 71F418FBE1; Fri, 13 Mar 2020 10:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584095450; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fciT879zSkhcAGmUXRf98FMw7LxC/2yx1e2Pv/uU6aM=; b=CIzCsZyeWgDEiGJj8Ga1L4zfjWLg71uRI+WAkFd6rhlV6VEPrdCZA2OC2JIh/RQMZvSwdO jdiQRIcJV4vpGlCCGDGIbsPrVJT6+fI1tXFt4FL/yztDhuJg5dTWsR3gjKDfWY+ulcmAkP WSE+CPa6rdjbmkuysLS/yEODM35zjj8= X-MC-Unique: gFIuVPbOMPacTUoVdhessQ-1 From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, zhang.zhanghailiang@huawei.com, zhukeqian1@huawei.com, quintela@redhat.com Subject: [PULL 3/5] COLO: Optimize memory back-up process Date: Fri, 13 Mar 2020 10:30:37 +0000 Message-Id: <20200313103039.150133-4-dgilbert@redhat.com> In-Reply-To: <20200313103039.150133-1-dgilbert@redhat.com> References: <20200313103039.150133-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" From: zhanghailiang This patch will reduce the downtime of VM for the initial process, Previously, we copied all these memory in preparing stage of COLO while we need to stop VM, which is a time-consuming process. Here we optimize it by a trick, back-up every page while in migration process while COLO is enabled, though it affects the speed of the migration, but it obviously reduce the downtime of back-up all SVM'S memory in COLO preparing stage. Signed-off-by: zhanghailiang Message-Id: <20200224065414.36524-5-zhang.zhanghailiang@huawei.com> Signed-off-by: Dr. David Alan Gilbert minor typo fixes --- migration/colo.c | 3 +++ migration/ram.c | 66 +++++++++++++++++++++++++++++++++++------------- migration/ram.h | 1 + 3 files changed, 53 insertions(+), 17 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 93c5a452fb..44942c4e23 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -26,6 +26,7 @@ #include "qemu/main-loop.h" #include "qemu/rcu.h" #include "migration/failover.h" +#include "migration/ram.h" #ifdef CONFIG_REPLICATION #include "replication.h" #endif @@ -845,6 +846,8 @@ void *colo_process_incoming_thread(void *opaque) */ qemu_file_set_blocking(mis->from_src_file, true); =20 + colo_incoming_start_dirty_log(); + bioc =3D qio_channel_buffer_new(COLO_BUFFER_BASE_SIZE); fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); diff --git a/migration/ram.c b/migration/ram.c index 02cfd76d19..31f4a9dc09 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2994,7 +2994,6 @@ int colo_init_ram_cache(void) } return -errno; } - memcpy(block->colo_cache, block->host, block->used_length); } } =20 @@ -3008,19 +3007,36 @@ int colo_init_ram_cache(void) =20 RAMBLOCK_FOREACH_NOT_IGNORED(block) { unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; - block->bmap =3D bitmap_new(pages); - bitmap_set(block->bmap, 0, pages); } } - ram_state =3D g_new0(RAMState, 1); - ram_state->migration_dirty_pages =3D 0; - qemu_mutex_init(&ram_state->bitmap_mutex); - memory_global_dirty_log_start(); =20 + ram_state_init(&ram_state); return 0; } =20 +/* TODO: duplicated with ram_init_bitmaps */ +void colo_incoming_start_dirty_log(void) +{ + RAMBlock *block =3D NULL; + /* For memory_global_dirty_log_start below. */ + qemu_mutex_lock_iothread(); + qemu_mutex_lock_ramlist(); + + memory_global_dirty_log_sync(); + WITH_RCU_READ_LOCK_GUARD() { + RAMBLOCK_FOREACH_NOT_IGNORED(block) { + ramblock_sync_dirty_bitmap(ram_state, block); + /* Discard this dirty bitmap record */ + bitmap_zero(block->bmap, block->max_length >> TARGET_PAGE_BITS= ); + } + memory_global_dirty_log_start(); + } + ram_state->migration_dirty_pages =3D 0; + qemu_mutex_unlock_ramlist(); + qemu_mutex_unlock_iothread(); +} + /* It is need to hold the global lock to call this helper */ void colo_release_ram_cache(void) { @@ -3040,9 +3056,7 @@ void colo_release_ram_cache(void) } } } - qemu_mutex_destroy(&ram_state->bitmap_mutex); - g_free(ram_state); - ram_state =3D NULL; + ram_state_cleanup(&ram_state); } =20 /** @@ -3356,7 +3370,7 @@ static int ram_load_precopy(QEMUFile *f) =20 while (!ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host =3D NULL; + void *host =3D NULL, *host_bak =3D NULL; uint8_t ch; =20 /* @@ -3387,20 +3401,35 @@ static int ram_load_precopy(QEMUFile *f) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 + host =3D host_from_ram_block_offset(block, addr); /* - * After going into COLO, we should load the Page into colo_ca= che. + * After going into COLO stage, we should not load the page + * into SVM's memory directly, we put them into colo_cache fir= stly. + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Previously, we copied all these memory in preparing stage o= f COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects = the + * speed of the migration, but it obviously reduce the downtim= e of + * back-up all SVM'S memory in COLO preparing stage. */ - if (migration_incoming_in_colo_state()) { - host =3D colo_cache_from_block_offset(block, addr); - } else { - host =3D host_from_ram_block_offset(block, addr); + if (migration_incoming_colo_enabled()) { + if (migration_incoming_in_colo_state()) { + /* In COLO stage, put all pages into cache temporarily= */ + host =3D colo_cache_from_block_offset(block, addr); + } else { + /* + * In migration stage but before COLO stage, + * Put all pages into both cache and SVM's memory. + */ + host_bak =3D colo_cache_from_block_offset(block, addr); + } } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - if (!migration_incoming_in_colo_state()) { ramblock_recv_bitmap_set(block, host); } @@ -3514,6 +3543,9 @@ static int ram_load_precopy(QEMUFile *f) if (!ret) { ret =3D qemu_file_get_error(f); } + if (!ret && host_bak) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } =20 ret |=3D wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index a553d40751..5ceaff7cb4 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -66,5 +66,6 @@ int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *= rb); /* ram cache */ int colo_init_ram_cache(void); void colo_release_ram_cache(void); +void colo_incoming_start_dirty_log(void); =20 #endif --=20 2.24.1