From nobody Tue Feb 10 11:14:53 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1623936150554479.9421287873845; Thu, 17 Jun 2021 06:22:30 -0700 (PDT) Received: from localhost ([::1]:32874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltryX-0004Ry-DF for importer@patchew.org; Thu, 17 Jun 2021 09:22:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45132) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltroa-00083p-Cm for qemu-devel@nongnu.org; Thu, 17 Jun 2021 09:12:12 -0400 Received: from prt-mail.chinatelecom.cn ([42.123.76.220]:42643 helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltroW-0001iL-Bm for qemu-devel@nongnu.org; Thu, 17 Jun 2021 09:12:12 -0400 Received: from clientip-202.80.192.39?logid-7be0abd77ce94142b7bfe1792c57913a (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 1E311280029; Thu, 17 Jun 2021 21:12:07 +0800 (CST) Received: from ([172.18.0.218]) by app0025 with ESMTP id 31985a2c383249099411102687ed3b3c for qemu-devel@nongnu.org; Thu Jun 17 21:12:07 2021 HMM_SOURCE_IP: 172.18.0.218:48906.1413600847 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 31985a2c383249099411102687ed3b3c X-filter-score: filter<0> X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 172.18.0.218 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel@nongnu.org Subject: [PATCH v6 7/7] migration/dirtyrate: implement dirty-ring dirtyrate calculation Date: Thu, 17 Jun 2021 21:15:44 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.220; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Juan Quintela , Hyman , "Dr. David Alan Gilbert" , Peter Xu , Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" From: Hyman Huang(=E9=BB=84=E5=8B=87) use dirty ring feature to implement dirtyrate calculation. introduce mode option in qmp calc_dirty_rate to specify what method should be used when calculating dirtyrate, either page-sampling or dirty-ring should be passed. introduce "dirty_ring:-r" option in hmp calc_dirty_rate to indicate dirty ring method should be used for calculation. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- hmp-commands.hx | 7 +- migration/dirtyrate.c | 192 +++++++++++++++++++++++++++++++++++++++++++++= ---- migration/trace-events | 2 + qapi/migration.json | 16 ++++- 4 files changed, 200 insertions(+), 17 deletions(-) diff --git a/hmp-commands.hx b/hmp-commands.hx index 8e45bce..f7fc9d7 100644 --- a/hmp-commands.hx +++ b/hmp-commands.hx @@ -1738,8 +1738,9 @@ ERST =20 { .name =3D "calc_dirty_rate", - .args_type =3D "second:l,sample_pages_per_GB:l?", - .params =3D "second [sample_pages_per_GB]", - .help =3D "start a round of guest dirty rate measurement", + .args_type =3D "dirty_ring:-r,second:l,sample_pages_per_GB:l?", + .params =3D "[-r] second [sample_pages_per_GB]", + .help =3D "start a round of guest dirty rate measurement (us= ing -d to" + "\n\t\t\t specify dirty ring as the method of calcul= ation)", .cmd =3D hmp_calc_dirty_rate, }, diff --git a/migration/dirtyrate.c b/migration/dirtyrate.c index 8a9dcf7..a130b5d 100644 --- a/migration/dirtyrate.c +++ b/migration/dirtyrate.c @@ -16,6 +16,7 @@ #include "cpu.h" #include "exec/ramblock.h" #include "qemu/rcu_queue.h" +#include "qemu/main-loop.h" #include "qapi/qapi-commands-migration.h" #include "ram.h" #include "trace.h" @@ -23,6 +24,14 @@ #include "monitor/hmp.h" #include "monitor/monitor.h" #include "qapi/qmp/qdict.h" +#include "sysemu/kvm.h" +#include "sysemu/runstate.h" +#include "exec/memory.h" + +typedef struct DirtyPageRecord { + uint64_t start_pages; + uint64_t end_pages; +} DirtyPageRecord; =20 static int CalculatingState =3D DIRTY_RATE_STATUS_UNSTARTED; static struct DirtyRateStat DirtyStat; @@ -71,19 +80,35 @@ static int dirtyrate_set_state(int *state, int old_stat= e, int new_state) =20 static struct DirtyRateInfo *query_dirty_rate_info(void) { + int i; int64_t dirty_rate =3D DirtyStat.dirty_rate; struct DirtyRateInfo *info =3D g_malloc0(sizeof(DirtyRateInfo)); - - if (qatomic_read(&CalculatingState) =3D=3D DIRTY_RATE_STATUS_MEASURED)= { - info->has_dirty_rate =3D true; - info->dirty_rate =3D dirty_rate; - } + DirtyRateVcpuList *head =3D NULL, **tail =3D &head; =20 info->status =3D CalculatingState; info->start_time =3D DirtyStat.start_time; info->calc_time =3D DirtyStat.calc_time; info->sample_pages =3D DirtyStat.sample_pages; + info->mode =3D dirtyrate_mode; +=20 + if (qatomic_read(&CalculatingState) =3D=3D DIRTY_RATE_STATUS_MEASURED)= { + info->has_dirty_rate =3D true; + info->dirty_rate =3D dirty_rate; =20 + if (dirtyrate_mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) { + /* set sample_pages with 0 to indicate page sampling isn't ena= bled */ + info->sample_pages =3D 0; + info->has_vcpu_dirty_rate =3D true; + for (i =3D 0; i < DirtyStat.dirty_ring.nvcpu; i++) { + DirtyRateVcpu *rate =3D g_malloc0(sizeof(DirtyRateVcpu)); + rate->id =3D DirtyStat.dirty_ring.rates[i].id; + rate->dirty_rate =3D DirtyStat.dirty_ring.rates[i].dirty_r= ate; + QAPI_LIST_APPEND(tail, rate); + } + info->vcpu_dirty_rate =3D head; + } + } + =20 trace_query_dirty_rate_info(DirtyRateStatus_str(CalculatingState)); =20 return info; @@ -114,7 +139,11 @@ static void init_dirtyrate_stat(int64_t start_time, =20 static void cleanup_dirtyrate_stat(struct DirtyRateConfig config) { - /* TODO */ + /* last calc-dirty-rate qmp use dirty ring mode */ + if (dirtyrate_mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) { + free(DirtyStat.dirty_ring.rates); + DirtyStat.dirty_ring.rates =3D NULL; + } } =20 static void update_dirtyrate_stat(struct RamblockDirtyInfo *info) @@ -351,7 +380,96 @@ static bool compare_page_hash_info(struct RamblockDirt= yInfo *info, return true; } =20 -static void calculate_dirtyrate(struct DirtyRateConfig config) +static inline void record_dirtypages(DirtyPageRecord *dirty_pages, + CPUState *cpu, bool start) +{ + if (start) { + dirty_pages[cpu->cpu_index].start_pages =3D cpu->dirty_pages; + } else { + dirty_pages[cpu->cpu_index].end_pages =3D cpu->dirty_pages; + } +} + +static void dirtyrate_global_dirty_log_start(void) +{ + qemu_mutex_lock_iothread(); + memory_global_dirty_log_start(GLOBAL_DIRTY_DIRTY_RATE); + qemu_mutex_unlock_iothread(); +} + +static void dirtyrate_global_dirty_log_stop(void) +{ + qemu_mutex_lock_iothread(); + memory_global_dirty_log_stop(GLOBAL_DIRTY_DIRTY_RATE); + qemu_mutex_unlock_iothread(); +} + +static int64_t do_calculate_dirtyrate_vcpu(DirtyPageRecord dirty_pages) +{ + uint64_t memory_size_MB; + int64_t time_s; + uint64_t increased_dirty_pages =3D + dirty_pages.end_pages - dirty_pages.start_pages; + + memory_size_MB =3D (increased_dirty_pages * TARGET_PAGE_SIZE) >> 20; + time_s =3D DirtyStat.calc_time; + + return memory_size_MB / time_s; +} + +static void calculate_dirtyrate_dirty_ring(struct DirtyRateConfig config) +{ + CPUState *cpu; + int64_t msec =3D 0; + int64_t start_time; + uint64_t dirtyrate =3D 0; + uint64_t dirtyrate_sum =3D 0; + DirtyPageRecord *dirty_pages; + int nvcpu =3D 0; + int i =3D 0; + + CPU_FOREACH(cpu) { + nvcpu++; + } + + dirty_pages =3D malloc(sizeof(*dirty_pages) * nvcpu); + + DirtyStat.dirty_ring.nvcpu =3D nvcpu; + DirtyStat.dirty_ring.rates =3D malloc(sizeof(DirtyRateVcpu) * nvcpu); + + dirtyrate_global_dirty_log_start(); + + CPU_FOREACH(cpu) { + record_dirtypages(dirty_pages, cpu, true); + } + + start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); + DirtyStat.start_time =3D start_time / 1000; + + msec =3D config.sample_period_seconds * 1000; + msec =3D set_sample_page_period(msec, start_time); + DirtyStat.calc_time =3D msec / 1000; + + CPU_FOREACH(cpu) { + record_dirtypages(dirty_pages, cpu, false); + } + + dirtyrate_global_dirty_log_stop(); + + for (i =3D 0; i < DirtyStat.dirty_ring.nvcpu; i++) { + dirtyrate =3D do_calculate_dirtyrate_vcpu(dirty_pages[i]); + trace_dirtyrate_do_calculate_vcpu(i, dirtyrate); + + DirtyStat.dirty_ring.rates[i].id =3D i; + DirtyStat.dirty_ring.rates[i].dirty_rate =3D dirtyrate; + dirtyrate_sum +=3D dirtyrate; + } + + DirtyStat.dirty_rate =3D dirtyrate_sum; + free(dirty_pages); +} + +static void calculate_dirtyrate_sample_vm(struct DirtyRateConfig config) { struct RamblockDirtyInfo *block_dinfo =3D NULL; int block_count =3D 0; @@ -382,6 +500,17 @@ out: free_ramblock_dirty_info(block_dinfo, block_count); } =20 +static void calculate_dirtyrate(struct DirtyRateConfig config) +{ + if (config.mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) { + calculate_dirtyrate_dirty_ring(config); + } else { + calculate_dirtyrate_sample_vm(config); + } + + trace_dirtyrate_calculate(DirtyStat.dirty_rate); +} + void *get_dirtyrate_thread(void *arg) { struct DirtyRateConfig config =3D *(struct DirtyRateConfig *)arg; @@ -407,8 +536,12 @@ void *get_dirtyrate_thread(void *arg) return NULL; } =20 -void qmp_calc_dirty_rate(int64_t calc_time, bool has_sample_pages, - int64_t sample_pages, Error **errp) +void qmp_calc_dirty_rate(int64_t calc_time, + bool has_sample_pages, + int64_t sample_pages, + bool has_mode, + DirtyRateMeasureMode mode, + Error **errp) { static struct DirtyRateConfig config; QemuThread thread; @@ -430,6 +563,15 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_s= ample_pages, return; } =20 + if (!has_mode) { + mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + } + + if (has_sample_pages && mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING= ) { + error_setg(errp, "either sample-pages or dirty-ring can be specifi= ed."); + return; + } + if (has_sample_pages) { if (!is_sample_pages_valid(sample_pages)) { error_setg(errp, "sample-pages is out of range[%d, %d].", @@ -442,6 +584,16 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_s= ample_pages, } =20 /* + * dirty ring mode only works when kvm dirty ring is enabled. + */ + if ((mode =3D=3D DIRTY_RATE_MEASURE_MODE_DIRTY_RING) && + !kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty ring is disabled, use sample-pages method " + "or remeasure later."); + return; + } + + /* * Init calculation state as unstarted. */ ret =3D dirtyrate_set_state(&CalculatingState, CalculatingState, @@ -453,7 +605,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sa= mple_pages, =20 config.sample_period_seconds =3D calc_time; config.sample_pages_per_gigabytes =3D sample_pages; - config.mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + config.mode =3D mode; =20 cleanup_dirtyrate_stat(config); =20 @@ -461,7 +613,7 @@ void qmp_calc_dirty_rate(int64_t calc_time, bool has_sa= mple_pages, * update dirty rate mode so that we can figure out what mode has * been used in last calculation **/ - dirtyrate_mode =3D DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING; + dirtyrate_mode =3D mode; =20 start_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME) / 1000; init_dirtyrate_stat(start_time, config); @@ -487,12 +639,23 @@ void hmp_info_dirty_rate(Monitor *mon, const QDict *q= dict) info->sample_pages); monitor_printf(mon, "Period: %"PRIi64" (sec)\n", info->calc_time); + monitor_printf(mon, "Mode: %s\n", + DirtyRateMeasureMode_str(info->mode)); monitor_printf(mon, "Dirty rate: "); if (info->has_dirty_rate) { monitor_printf(mon, "%"PRIi64" (MB/s)\n", info->dirty_rate); + if (info->has_vcpu_dirty_rate) { + DirtyRateVcpuList *rate, *head =3D info->vcpu_dirty_rate; + for (rate =3D head; rate !=3D NULL; rate =3D rate->next) { + monitor_printf(mon, "vcpu[%"PRIi64"], Dirty rate: %"PRIi64= "\n", + rate->value->id, rate->value->dirty_rate); + } + } } else { monitor_printf(mon, "(not ready)\n"); } + + qapi_free_DirtyRateVcpuList(info->vcpu_dirty_rate);=20 g_free(info); } =20 @@ -501,6 +664,10 @@ void hmp_calc_dirty_rate(Monitor *mon, const QDict *qd= ict) int64_t sec =3D qdict_get_try_int(qdict, "second", 0); int64_t sample_pages =3D qdict_get_try_int(qdict, "sample_pages_per_GB= ", -1); bool has_sample_pages =3D (sample_pages !=3D -1); + bool dirty_ring =3D qdict_get_try_bool(qdict, "dirty_ring", false); + DirtyRateMeasureMode mode =3D + (dirty_ring ? DIRTY_RATE_MEASURE_MODE_DIRTY_RING : + DIRTY_RATE_MEASURE_MODE_PAGE_SAMPLING); Error *err =3D NULL; =20 if (!sec) { @@ -508,7 +675,8 @@ void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdi= ct) return; } =20 - qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, &err); + qmp_calc_dirty_rate(sec, has_sample_pages, sample_pages, true, + mode, &err); if (err) { hmp_handle_error(mon, err); return; diff --git a/migration/trace-events b/migration/trace-events index 860c4f4..3186929 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -330,6 +330,8 @@ get_ramblock_vfn_hash(const char *idstr, uint64_t vfn, = uint32_t crc) "ramblock n calc_page_dirty_rate(const char *idstr, uint32_t new_crc, uint32_t old_crc= ) "ramblock name: %s, new crc: %" PRIu32 ", old crc: %" PRIu32 skip_sample_ramblock(const char *idstr, uint64_t ramblock_size) "ramblock = name: %s, ramblock size: %" PRIu64 find_page_matched(const char *idstr) "ramblock %s addr or size changed" +dirtyrate_calculate(int64_t dirtyrate) "dirty rate: %" PRIi64 " MB/s" +dirtyrate_do_calculate_vcpu(int idx, uint64_t rate) "vcpu[%d]: %"PRIu64 " = MB/s" =20 # block.c migration_block_init_shared(const char *blk_device_name) "Start migration = for %s with shared base image" diff --git a/qapi/migration.json b/qapi/migration.json index 7395305..e3d21a8 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1773,6 +1773,12 @@ # @sample-pages: page count per GB for sample dirty pages # the default value is 512 (since 6.1) # +# @mode: mode containing method of calculate dirtyrate includes +# 'page-sampling' and 'dirty-ring' (Since 6.1) +# +# @vcpu-dirty-rate: dirtyrate for each vcpu if dirty-ring +# mode specified (Since 6.1) +# # Since: 5.2 # ## @@ -1781,7 +1787,9 @@ 'status': 'DirtyRateStatus', 'start-time': 'int64', 'calc-time': 'int64', - 'sample-pages': 'uint64'} } + 'sample-pages': 'uint64', + 'mode': 'DirtyRateMeasureMode', + '*vcpu-dirty-rate': [ 'DirtyRateVcpu' ] } } =20 ## # @calc-dirty-rate: @@ -1793,6 +1801,9 @@ # @sample-pages: page count per GB for sample dirty pages # the default value is 512 (since 6.1) # +# @mode: mechanism of calculating dirtyrate includes +# 'page-sampling' and 'dirty-ring' (Since 6.1) +# # Since: 5.2 # # Example: @@ -1801,7 +1812,8 @@ # ## { 'command': 'calc-dirty-rate', 'data': {'calc-time': 'int64', - '*sample-pages': 'int'} } + '*sample-pages': 'int', + '*mode': 'DirtyRateMeasureMode'} } =20 ## # @query-dirty-rate: --=20 1.8.3.1