From nobody Tue Feb 10 12:57:50 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1501230492445731.61413252474; Fri, 28 Jul 2017 01:28:12 -0700 (PDT) Received: from localhost ([::1]:46756 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1db0d0-000612-Vf for importer@patchew.org; Fri, 28 Jul 2017 04:28:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47380) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1db0Jf-0001Rp-R5 for qemu-devel@nongnu.org; Fri, 28 Jul 2017 04:08:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1db0Je-0001bu-DX for qemu-devel@nongnu.org; Fri, 28 Jul 2017 04:08:11 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42252) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1db0Je-0001bF-4n for qemu-devel@nongnu.org; Fri, 28 Jul 2017 04:08:10 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 07650C024B1A; Fri, 28 Jul 2017 08:08:09 +0000 (UTC) Received: from pxdev.xzpeter.org.com (dhcp-15-224.nay.redhat.com [10.66.15.224]) by smtp.corp.redhat.com (Postfix) with ESMTP id D0CE8600C2; Fri, 28 Jul 2017 08:08:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 07650C024B1A Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=peterx@redhat.com From: Peter Xu To: qemu-devel@nongnu.org Date: Fri, 28 Jul 2017 16:06:31 +0800 Message-Id: <1501229198-30588-23-git-send-email-peterx@redhat.com> In-Reply-To: <1501229198-30588-1-git-send-email-peterx@redhat.com> References: <1501229198-30588-1-git-send-email-peterx@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Fri, 28 Jul 2017 08:08:09 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC 22/29] migration: new message MIG_RP_MSG_RECV_BITMAP X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Laurent Vivier , Andrea Arcangeli , Juan Quintela , Alexey Perevalov , peterx@redhat.com, "Dr . David Alan Gilbert" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introducing new return path message MIG_RP_MSG_RECV_BITMAP to send received bitmap of ramblock back to source. This is the reply message of MIG_CMD_RECV_BITMAP, it contains not only the header (including the ramblock name), and it was appended with the whole ramblock received bitmap on the destination side. When the source receives such a reply message (MIG_RP_MSG_RECV_BITMAP), it parses it, convert it to the dirty bitmap by reverting the bits. Signed-off-by: Peter Xu --- migration/migration.c | 62 ++++++++++++++++++++++++++++++++++++++++++ migration/migration.h | 2 ++ migration/ram.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++= ++++ migration/ram.h | 2 ++ migration/savevm.c | 2 +- migration/trace-events | 2 ++ 6 files changed, 143 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index e498fa4..c2b85ac 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -92,6 +92,7 @@ enum mig_rp_message_type { =20 MIG_RP_MSG_REQ_PAGES_ID, /* data (start: be64, len: be32, id: string) = */ MIG_RP_MSG_REQ_PAGES, /* data (start: be64, len: be32) */ + MIG_RP_MSG_RECV_BITMAP, /* send recved_bitmap back to source */ =20 MIG_RP_MSG_MAX }; @@ -450,6 +451,39 @@ void migrate_send_rp_pong(MigrationIncomingState *mis, migrate_send_rp_message(mis, MIG_RP_MSG_PONG, sizeof(buf), &buf); } =20 +void migrate_send_rp_recv_bitmap(MigrationIncomingState *mis, + char *block_name) +{ + char buf[512]; + int len; + int64_t res; + + /* + * First, we send the header part. It contains only the len of + * idstr, and the idstr itself. + */ + len =3D strlen(block_name); + buf[0] =3D len; + memcpy(buf + 1, block_name, len); + + migrate_send_rp_message(mis, MIG_RP_MSG_RECV_BITMAP, len + 1, buf); + + /* + * Next, we dump the received bitmap to the stream. + * + * TODO: currently we are safe since we are the only one that is + * using the to_src_file handle (fault thread is still paused), + * and it's ok even not taking the mutex. However the best way is + * to take the lock before sending the message header, and release + * the lock after sending the bitmap. + */ + qemu_mutex_lock(&mis->rp_mutex); + res =3D ramblock_recv_bitmap_send(mis->to_src_file, block_name); + qemu_mutex_unlock(&mis->rp_mutex); + + trace_migrate_send_rp_recv_bitmap(block_name, res); +} + MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp) { MigrationCapabilityStatusList *head =3D NULL; @@ -1560,6 +1594,7 @@ static struct rp_cmd_args { [MIG_RP_MSG_PONG] =3D { .len =3D 4, .name =3D "PONG" }, [MIG_RP_MSG_REQ_PAGES] =3D { .len =3D 12, .name =3D "REQ_PAGES" }, [MIG_RP_MSG_REQ_PAGES_ID] =3D { .len =3D -1, .name =3D "REQ_PAGES_ID= " }, + [MIG_RP_MSG_RECV_BITMAP] =3D { .len =3D -1, .name =3D "RECV_BITMAP"= }, [MIG_RP_MSG_MAX] =3D { .len =3D -1, .name =3D "MAX" }, }; =20 @@ -1604,6 +1639,19 @@ static bool postcopy_pause_return_path_thread(Migrat= ionState *s) return true; } =20 +static int migrate_handle_rp_recv_bitmap(MigrationState *s, char *block_na= me) +{ + RAMBlock *block =3D qemu_ram_block_by_name(block_name); + + if (!block) { + error_report("%s: invalid block name '%s'", __func__, block_name); + return -EINVAL; + } + + /* Fetch the received bitmap and refresh the dirty bitmap */ + return ram_dirty_bitmap_reload(s, block); +} + /* * Handles messages sent on the return path towards the source VM * @@ -1709,6 +1757,20 @@ retry: migrate_handle_rp_req_pages(ms, (char *)&buf[13], start, len); break; =20 + case MIG_RP_MSG_RECV_BITMAP: + if (header_len < 1) { + error_report("%s: missing block name", __func__); + mark_source_rp_bad(ms); + goto out; + } + /* Format: len (1B) + idstr (<255B). This ends the idstr. */ + buf[buf[0] + 1] =3D '\0'; + if (migrate_handle_rp_recv_bitmap(ms, (char *)(buf + 1))) { + mark_source_rp_bad(ms); + goto out; + } + break; + default: break; } diff --git a/migration/migration.h b/migration/migration.h index 574fedd..4d38308 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -204,5 +204,7 @@ void migrate_send_rp_pong(MigrationIncomingState *mis, uint32_t value); int migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbn= ame, ram_addr_t start, size_t len); +void migrate_send_rp_recv_bitmap(MigrationIncomingState *mis, + char *block_name); =20 #endif diff --git a/migration/ram.c b/migration/ram.c index 7f4cb0f..d543483 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -182,6 +182,32 @@ void ramblock_recv_bitmap_clear(void *host_addr, RAMBl= ock *rb) } =20 /* + * Format: bitmap_size (8 bytes) + whole_bitmap (N bytes). + * + * Returns >0 if success with sent bytes, or <0 if error. + */ +int64_t ramblock_recv_bitmap_send(QEMUFile *file, char *block_name) +{ + RAMBlock *block =3D qemu_ram_block_by_name(block_name); + uint64_t size; + + /* We should have made sure that the block exists */ + assert(block); + + /* Size of the bitmap, in bytes */ + size =3D (block->max_length >> TARGET_PAGE_BITS) / 8; + qemu_put_be64(file, size); + qemu_put_buffer(file, (const uint8_t *)block->receivedmap, size); + qemu_fflush(file); + + if (qemu_file_get_error(file)) { + return qemu_file_get_error(file); + } + + return sizeof(size) + size; +} + +/* * An outstanding page request, on the source, having been received * and queued */ @@ -2705,6 +2731,54 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) return ret; } =20 +/* + * Read the received bitmap, revert it as the initial dirty bitmap. + * This is only used when the postcopy migration is paused but wants + * to resume from a middle point. + */ +int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *block) +{ + QEMUFile *file =3D s->rp_state.from_dst_file; + uint64_t local_size =3D (block->max_length >> TARGET_PAGE_BITS) / 8; + uint64_t size; + + if (s->state !=3D MIGRATION_STATUS_POSTCOPY_RECOVER) { + error_report("%s: incorrect state %s", __func__, + MigrationStatus_lookup[s->state]); + return -EINVAL; + } + + size =3D qemu_get_be64(file); + + /* The size of the bitmap should match with our ramblock */ + if (size !=3D local_size) { + error_report("%s: ramblock '%s' bitmap size mismatch " + "(0x%lx !=3D 0x%lx)", __func__, block->idstr, + size, local_size); + return -EINVAL; + } + + /* + * We are still during migration (though paused). The dirty bitmap + * won't change. We can directly modify it. + */ + size =3D qemu_get_buffer(file, (uint8_t *)block->bmap, local_size); + + if (qemu_file_get_error(file)) { + return qemu_file_get_error(file); + } + + /* + * What we received is "received bitmap". Revert it as the initial + * dirty bitmap for this ramblock. + */ + bitmap_invert(block->bmap, block->max_length >> TARGET_PAGE_BITS); + + trace_ram_dirty_bitmap_reload(block->idstr); + + return 0; +} + static SaveVMHandlers savevm_ram_handlers =3D { .save_setup =3D ram_save_setup, .save_live_iterate =3D ram_save_iterate, diff --git a/migration/ram.h b/migration/ram.h index 84e8623..86eb973 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -58,5 +58,7 @@ void ramblock_recv_bitmap_set(void *host_addr, RAMBlock *= rb); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t len); void ramblock_recv_bitmap_clear(void *host_addr, RAMBlock *rb); +int64_t ramblock_recv_bitmap_send(QEMUFile *file, char *block_name); +int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); =20 #endif diff --git a/migration/savevm.c b/migration/savevm.c index 0ab13c0..def9213 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1755,7 +1755,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomin= gState *mis, return -EINVAL; } =20 - /* TODO: send the bitmap back to source */ + migrate_send_rp_recv_bitmap(mis, block_name); =20 return 0; } diff --git a/migration/trace-events b/migration/trace-events index ca7b43f..ed69551 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -77,6 +77,7 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRI= x64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset= : %" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: %zx len: %zx" +ram_dirty_bitmap_reload(char *str) "%s" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" @@ -88,6 +89,7 @@ migrate_fd_cancel(void) "" migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) = "in %s at %zx len %zx" migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpo= st) "pending size %" PRIu64 " max %" PRIu64 " (post=3D%" PRIu64 " nonpost= =3D%" PRIu64 ")" migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d" +migrate_send_rp_recv_bitmap(char *name, int64_t size) "block '%s' size 0x%= "PRIi64 migration_completion_file_err(void) "" migration_completion_postcopy_end(void) "" migration_completion_postcopy_end_after_complete(void) "" --=20 2.7.4