From nobody Fri May 3 17:25:35 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=ucloud.cn ARC-Seal: i=1; a=rsa-sha256; t=1593769183; cv=none; d=zohomail.com; s=zohoarc; b=glnKRvbtMDc1Gzzq7PJqqR+D1Fv5TDIO8MLjRJYr9+Fi0fV+oerb/iDa3KIfNL5rtcA6CcmpVP3lC2uOCwUWmr2fODdDEpU16RlVr6Fu2amUMBPNjASdKNqtaD7nJWdiaOqOslhtSTB7OPVxIjlMJlrkTVh97zxMe+iQ0HPr4kI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1593769183; h=Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:Sender:Subject:To; bh=8d2v9Tt7m6lbIQ77KwIvxwyG3dP/6uYSYZsbdSR9/po=; b=h4ehQT9D/JJUMkSxoYbSWgYAb8DcewPEvQ62ygXN/YI/lFM53iMHnBZjEQ4p+d14gU2/+s6fz71+au6v8dC13D+cc/fmC42GSNDJb28N2W0P5prCobkhjAXtFceneGAOlrt2c+wJFFLEVhS+OSaoEQWLV0+5xm5GO8MqBM6gqHQ= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1593769182848940.9935322957834; Fri, 3 Jul 2020 02:39:42 -0700 (PDT) Received: from localhost ([::1]:39742 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jrIAX-0004vt-Hr for importer@patchew.org; Fri, 03 Jul 2020 05:39:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47298) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jrI9t-0004Hb-HO for qemu-devel@nongnu.org; Fri, 03 Jul 2020 05:39:01 -0400 Received: from m9785.mail.qiye.163.com ([220.181.97.85]:61376) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jrI9n-0005cs-GU for qemu-devel@nongnu.org; Fri, 03 Jul 2020 05:39:00 -0400 Received: from localhost.localdomain (unknown [106.75.220.3]) by m9785.mail.qiye.163.com (Hmail) with ESMTPA id DE2BE5C161B for ; Fri, 3 Jul 2020 17:29:00 +0800 (CST) From: Bingsong Si To: qemu-devel@nongnu.org Subject: [PATCH] ram: add support for dirty page tracking Date: Fri, 3 Jul 2020 17:28:53 +0800 Message-Id: <20200703092853.1448582-1-owen.si@ucloud.cn> X-Mailer: git-send-email 2.18.4 X-HM-Spam-Status: e1kfGhgUHx5ZQUtXWQgYFAkeWUFZSFVITUhCQkJDT0hOQkxPSllXWShZQU lCN1dZLVlBSVdZDwkaFQgSH1lBWR0iNQs4HDlIDQMfGFANTh4BEhUzOhxWVlVMSUsoSVlXWQkOFx 4IWUFZNTQpNjo3JCkuNz5ZV1kWGg8SFR0UWUFZNDBZBg++ X-HM-Sender-Digest: e1kMHhlZQR0aFwgeV1kSHx4VD1lBWUc6Pgw6NBw6Lj8aGBRDShMWCk5C Qw0wChdVSlVKTkJITE1DTk9KSEpLVTMWGhIXVRQMHhVVCBI7DhgXFA4fVRgVRVlXWRILWUFZSktN VUxOVUlJS1VIWVdZCAFZQUJISUo3Bg++ X-HM-Tid: 0a731401fb092087kuqyde2be5c161b Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=220.181.97.85; envelope-from=owen.si@ucloud.cn; helo=m9785.mail.qiye.163.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/03 05:29:02 X-ACL-Warn: Detected OS = Linux 3.1-3.10 [fuzzy] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" In production, the VM with insentive memory activity maybe failed to migrat= e, because of the change of memory in the VM greater than the throughtput of t= he network interface, and we want to identify it before migration. 1. dirty tracking start: virsh qemu-monitor-command --hmp dirty_track 2. wait some time, stop dirty tracking: virsh qemu-monitor-command --hmp dirty_track_stop Dirty rate: 607 pages/s Signed-off-by: Bingsong Si --- hmp-commands.hx | 26 ++++++++++++++ include/monitor/hmp.h | 2 ++ migration/migration.c | 5 +++ migration/ram.c | 65 +++++++++++++++++++++++++++++++++ migration/ram.h | 5 +++ migration/savevm.c | 83 +++++++++++++++++++++++++++++++++++++++++++ migration/savevm.h | 2 ++ 7 files changed, 188 insertions(+) diff --git a/hmp-commands.hx b/hmp-commands.hx index 60f395c276..05a688286b 100644 --- a/hmp-commands.hx +++ b/hmp-commands.hx @@ -1819,6 +1819,32 @@ SRST Set QOM property *property* of object at location *path* to value *value* ERST =20 + { + .name =3D "dirty_track", + .args_type =3D "", + .params =3D "", + .help =3D "track dirty pages rate", + .cmd =3D hmp_dirty_track, + }, + +SRST +``dirty_track`` + Track dirty pages rate. +ERST + + { + .name =3D "dirty_track_stop", + .args_type =3D "", + .params =3D "", + .help =3D "stop current dirty pages track", + .cmd =3D hmp_dirty_track_stop, + }, + +SRST +``dirty_track_stop`` + Stop current dirty pages track. +ERST + { .name =3D "info", .args_type =3D "item:s?", diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h index c986cfd28b..c139fe8758 100644 --- a/include/monitor/hmp.h +++ b/include/monitor/hmp.h @@ -130,5 +130,7 @@ void hmp_hotpluggable_cpus(Monitor *mon, const QDict *q= dict); void hmp_info_vm_generation_id(Monitor *mon, const QDict *qdict); void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict); void hmp_info_sev(Monitor *mon, const QDict *qdict); +void hmp_dirty_track(Monitor *mon, const QDict *qdict); +void hmp_dirty_track_stop(Monitor *mon, const QDict *qdict); =20 #endif diff --git a/migration/migration.c b/migration/migration.c index 481a590f72..5550afafe6 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1964,6 +1964,11 @@ static bool migrate_prepare(MigrationState *s, bool = blk, bool blk_inc, { Error *local_err =3D NULL; =20 + if (dirty_track_is_running()) { + error_setg(errp, "There is a dirty tracking process in progress"); + return false; + } + if (resume) { if (s->state !=3D MIGRATION_STATUS_POSTCOPY_PAUSED) { error_setg(errp, "Cannot resume if there is no " diff --git a/migration/ram.c b/migration/ram.c index 069b6e30bc..03a5e44617 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3780,6 +3780,71 @@ static int ram_resume_prepare(MigrationState *s, voi= d *opaque) return 0; } =20 +void dirty_track_init(void) +{ + RAMBlock *block; + + if (ram_bytes_total()) { + RAMBLOCK_FOREACH_NOT_IGNORED(block) { + unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; + + block->bmap =3D bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state =3D g_new0(RAMState, 1); + ram_state->migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); +} + +uint64_t dirty_track_dirty_pages(void) +{ + return ram_state->migration_dirty_pages; +} + +void dirty_track_sync(void) +{ + RAMBlock *block =3D NULL; + unsigned long offset =3D 0; + + memory_global_dirty_log_sync(); + rcu_read_lock(); + RAMBLOCK_FOREACH_NOT_IGNORED(block) { + ramblock_sync_dirty_bitmap(ram_state, block); + } + rcu_read_unlock(); + + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + test_and_clear_bit(offset, block->bmap); + } + } + + rcu_read_unlock(); +} + +void dirty_track_cleanup(void) +{ + RAMBlock *block; + + memory_global_dirty_log_stop(); + RAMBLOCK_FOREACH_NOT_IGNORED(block) { + g_free(block->bmap); + block->bmap =3D NULL; + } + + g_free(ram_state); + ram_state =3D NULL; +} + static SaveVMHandlers savevm_ram_handlers =3D { .save_setup =3D ram_save_setup, .save_live_iterate =3D ram_save_iterate, diff --git a/migration/ram.h b/migration/ram.h index 2eeaacfa13..104c48285c 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -69,4 +69,9 @@ void colo_flush_ram_cache(void); void colo_release_ram_cache(void); void colo_incoming_start_dirty_log(void); =20 +void dirty_track_init(void); +uint64_t dirty_track_dirty_pages(void); +void dirty_track_sync(void); +void dirty_track_cleanup(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index b979ea6e7f..0be70e6528 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -63,6 +63,8 @@ #include "migration/colo.h" #include "qemu/bitmap.h" #include "net/announce.h" +#include "monitor/monitor.h" +#include "monitor/hmp.h" =20 const unsigned int postcopy_ram_discard_version =3D 0; =20 @@ -171,6 +173,15 @@ static QEMUFile *qemu_fopen_bdrv(BlockDriverState *bs,= int is_writable) return qemu_fopen_ops(bs, &bdrv_read_ops); } =20 +static struct DirtyTrackState { + QemuThread thread; + int dirty_pages_rate; + bool quit; +} current_dirty_track_state =3D { + { .thread =3D 0 }, + .dirty_pages_rate =3D 0, + .quit =3D false, +}; =20 /* QEMUFile timer support. * Not in qemu-file.c to not add qemu-timer.c as dependency to qemu-file.c @@ -2747,6 +2758,78 @@ int save_snapshot(const char *name, Error **errp) return ret; } =20 +static void *dirty_track_thread(void *opaque) +{ + int64_t initial_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + struct DirtyTrackState *s =3D opaque; + int64_t current_time; + uint64_t time_spent; + + for (;;) { + dirty_track_sync(); + if (s->quit) { + current_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + time_spent =3D current_time - initial_time; + + if (time_spent) { + s->dirty_pages_rate =3D dirty_track_dirty_pages() * 1000 / + time_spent; + } + break; + } + usleep(1000 * 100); + } + return NULL; +} + +bool dirty_track_is_running(void) +{ + return !!current_dirty_track_state.thread.thread; +} + +void hmp_dirty_track(Monitor *mon, const QDict *qdict) +{ + MigrationState *s =3D migrate_get_current(); + + if (migration_is_running(s->state)) { + error_report(QERR_MIGRATION_ACTIVE); + return; + } + + if (runstate_check(RUN_STATE_INMIGRATE)) { + error_report("Guest is waiting for an incoming migration"); + return; + } + + if (dirty_track_is_running()) { + error_report("There is a dirty tracking process in progress"); + return; + } + + dirty_track_init(); + qemu_thread_create(¤t_dirty_track_state.thread, "dirty tracking", + dirty_track_thread, ¤t_dirty_track_state, + QEMU_THREAD_JOINABLE); +} + +void hmp_dirty_track_stop(Monitor *mon, const QDict *qdict) +{ + if (current_dirty_track_state.thread.thread =3D=3D 0) { + error_report("There is no dirty tracking process in progress"); + return; + } + + current_dirty_track_state.quit =3D true; + qemu_thread_join(¤t_dirty_track_state.thread); + monitor_printf(mon, "Dirty rate: %d pages/s\n", + current_dirty_track_state.dirty_pages_rate); + + dirty_track_cleanup(); + current_dirty_track_state.thread.thread =3D 0; + current_dirty_track_state.dirty_pages_rate =3D 0; + current_dirty_track_state.quit =3D false; +} + void qmp_xen_save_devices_state(const char *filename, bool has_live, bool = live, Error **errp) { diff --git a/migration/savevm.h b/migration/savevm.h index ba64a7e271..216b9b7396 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -65,4 +65,6 @@ void qemu_loadvm_state_cleanup(void); int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); int qemu_load_device_state(QEMUFile *f); =20 +bool dirty_track_is_running(void); + #endif --=20 2.18.4