From nobody Wed Oct 29 22:59:57 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1525866419834960.8670448467002; Wed, 9 May 2018 04:46:59 -0700 (PDT) Received: from localhost ([::1]:55885 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fGNYg-0002BM-SQ for importer@patchew.org; Wed, 09 May 2018 07:46:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58764) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fGNDQ-0004ut-R6 for qemu-devel@nongnu.org; Wed, 09 May 2018 07:25:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fGNDP-0001g3-Jj for qemu-devel@nongnu.org; Wed, 09 May 2018 07:25:00 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:53776 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fGNDP-0001f5-D7 for qemu-devel@nongnu.org; Wed, 09 May 2018 07:24:59 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 07A9D8182D24 for ; Wed, 9 May 2018 11:24:59 +0000 (UTC) Received: from secure.mitica (ovpn-117-237.ams2.redhat.com [10.36.117.237]) by smtp.corp.redhat.com (Postfix) with ESMTP id 00E08215CDA7; Wed, 9 May 2018 11:24:57 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 9 May 2018 13:23:54 +0200 Message-Id: <20180509112406.6183-30-quintela@redhat.com> In-Reply-To: <20180509112406.6183-1-quintela@redhat.com> References: <20180509112406.6183-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 09 May 2018 11:24:59 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 09 May 2018 11:24:59 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PULL 29/41] migration: synchronize dirty bitmap for resume X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Peter Xu This patch implements the first part of core RAM resume logic for postcopy. ram_resume_prepare() is provided for the work. When the migration is interrupted by network failure, the dirty bitmap on the source side will be meaningless, because even the dirty bit is cleared, it is still possible that the sent page was lost along the way to destination. Here instead of continue the migration with the old dirty bitmap on source, we ask the destination side to send back its received bitmap, then invert it to be our initial dirty bitmap. The source side send thread will issue the MIG_CMD_RECV_BITMAP requests, once per ramblock, to ask for the received bitmap. On destination side, MIG_RP_MSG_RECV_BITMAP will be issued, along with the requested bitmap. Data will be received on the return-path thread of source, and the main migration thread will be notified when all the ramblock bitmaps are synchronized. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Message-Id: <20180502104740.12123-17-peterx@redhat.com> Signed-off-by: Juan Quintela --- migration/migration.c | 2 ++ migration/migration.h | 1 + migration/ram.c | 47 ++++++++++++++++++++++++++++++++++++++++++ migration/trace-events | 4 ++++ 4 files changed, 54 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index b0217c4823..19ef8b05b1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2967,6 +2967,7 @@ static void migration_instance_finalize(Object *obj) qemu_sem_destroy(&ms->pause_sem); qemu_sem_destroy(&ms->postcopy_pause_sem); qemu_sem_destroy(&ms->postcopy_pause_rp_sem); + qemu_sem_destroy(&ms->rp_state.rp_sem); error_free(ms->error); } =20 @@ -2999,6 +3000,7 @@ static void migration_instance_init(Object *obj) =20 qemu_sem_init(&ms->postcopy_pause_sem, 0); qemu_sem_init(&ms->postcopy_pause_rp_sem, 0); + qemu_sem_init(&ms->rp_state.rp_sem, 0); } =20 /* diff --git a/migration/migration.h b/migration/migration.h index 556964d9d9..b4438ccb65 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -135,6 +135,7 @@ struct MigrationState QEMUFile *from_dst_file; QemuThread rp_thread; bool error; + QemuSemaphore rp_sem; } rp_state; =20 double mbps; diff --git a/migration/ram.c b/migration/ram.c index 5542843adc..b16eabcfb9 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -54,6 +54,7 @@ #include "migration/block.h" #include "sysemu/sysemu.h" #include "qemu/uuid.h" +#include "savevm.h" =20 /***********************************************************/ /* ram save/restore */ @@ -3364,6 +3365,38 @@ static bool ram_has_postcopy(void *opaque) return migrate_postcopy_ram(); } =20 +/* Sync all the dirty bitmap with destination VM. */ +static int ram_dirty_bitmap_sync_all(MigrationState *s, RAMState *rs) +{ + RAMBlock *block; + QEMUFile *file =3D s->to_dst_file; + int ramblock_count =3D 0; + + trace_ram_dirty_bitmap_sync_start(); + + RAMBLOCK_FOREACH(block) { + qemu_savevm_send_recv_bitmap(file, block->idstr); + trace_ram_dirty_bitmap_request(block->idstr); + ramblock_count++; + } + + trace_ram_dirty_bitmap_sync_wait(); + + /* Wait until all the ramblocks' dirty bitmap synced */ + while (ramblock_count--) { + qemu_sem_wait(&s->rp_state.rp_sem); + } + + trace_ram_dirty_bitmap_sync_complete(); + + return 0; +} + +static void ram_dirty_bitmap_reload_notify(MigrationState *s) +{ + qemu_sem_post(&s->rp_state.rp_sem); +} + /* * Read the received bitmap, revert it as the initial dirty bitmap. * This is only used when the postcopy migration is paused but wants @@ -3438,12 +3471,25 @@ int ram_dirty_bitmap_reload(MigrationState *s, RAMB= lock *block) =20 trace_ram_dirty_bitmap_reload_complete(block->idstr); =20 + /* + * We succeeded to sync bitmap for current ramblock. If this is + * the last one to sync, we need to notify the main send thread. + */ + ram_dirty_bitmap_reload_notify(s); + ret =3D 0; out: free(le_bitmap); return ret; } =20 +static int ram_resume_prepare(MigrationState *s, void *opaque) +{ + RAMState *rs =3D *(RAMState **)opaque; + + return ram_dirty_bitmap_sync_all(s, rs); +} + static SaveVMHandlers savevm_ram_handlers =3D { .save_setup =3D ram_save_setup, .save_live_iterate =3D ram_save_iterate, @@ -3455,6 +3501,7 @@ static SaveVMHandlers savevm_ram_handlers =3D { .save_cleanup =3D ram_save_cleanup, .load_setup =3D ram_load_setup, .load_cleanup =3D ram_load_cleanup, + .resume_prepare =3D ram_resume_prepare, }; =20 void ram_mig_init(void) diff --git a/migration/trace-events b/migration/trace-events index be36fbccfe..53243e17ec 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -82,8 +82,12 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PR= Ix64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset= : 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: 0x%zx len: 0x%zx" +ram_dirty_bitmap_request(char *str) "%s" ram_dirty_bitmap_reload_begin(char *str) "%s" ram_dirty_bitmap_reload_complete(char *str) "%s" +ram_dirty_bitmap_sync_start(void) "" +ram_dirty_bitmap_sync_wait(void) "" +ram_dirty_bitmap_sync_complete(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.17.0