From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564383067255.8847163144486; Thu, 16 Feb 2023 08:19:43 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgyf-00080F-Hj; Thu, 16 Feb 2023 11:19:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgyd-0007xY-21 for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:19 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgya-000526-RQ for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:18 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 1C63A2800E2; Fri, 17 Feb 2023 00:19:02 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id 606cdf52dece466c84aacc876dd3eb98 for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:09 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 606cdf52dece466c84aacc876dd3eb98 X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 01/10] dirtylimit: Fix overflow when computing MB Date: Fri, 17 Feb 2023 00:18:30 +0800 Message-Id: <255bcba59296e91e6be756c1e6e540afeabd94f3.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564385901100001 From: Hyman Huang(=E9=BB=84=E5=8B=87) Coverity points out a overflow problem when computing MB, dirty_ring_size and TARGET_PAGE_SIZE are both 32 bits, multiplication will be done as a 32-bit operation, which could overflow. Simplify the formula. Meanwhile, fix spelling mistake of variable name. Reported-by: Peter Maydell Signed-off-by: Peter Maydell Signed-off-by: Richard Henderson Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Reviewed-by: Peter Xu --- softmmu/dirtylimit.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index c56f0f58c8..065ed18afc 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -235,14 +235,14 @@ static inline int64_t dirtylimit_dirty_ring_full_time= (uint64_t dirtyrate) { static uint64_t max_dirtyrate; uint32_t dirty_ring_size =3D kvm_dirty_ring_size(); - uint64_t dirty_ring_size_meory_MB =3D - dirty_ring_size * TARGET_PAGE_SIZE >> 20; + uint32_t dirty_ring_size_memory_MB =3D + dirty_ring_size >> (20 - TARGET_PAGE_BITS); =20 if (max_dirtyrate < dirtyrate) { max_dirtyrate =3D dirtyrate; } =20 - return dirty_ring_size_meory_MB * 1000000 / max_dirtyrate; + return dirty_ring_size_memory_MB * 1000000ULL / max_dirtyrate; } =20 static inline bool dirtylimit_done(uint64_t quota, --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564387370675.0268240065692; Thu, 16 Feb 2023 08:19:47 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgyk-00086s-3g; Thu, 16 Feb 2023 11:19:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgyd-0007yA-J9 for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:19 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgyb-000532-Oo for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:19 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 6E2F22800E1; Fri, 17 Feb 2023 00:19:09 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id d38a131cea4c4d9391f8998fd4db1557 for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:14 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: d38a131cea4c4d9391f8998fd4db1557 X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 02/10] softmmu/dirtylimit: Add parameter check for hmp "set_vcpu_dirty_limit" Date: Fri, 17 Feb 2023 00:18:31 +0800 Message-Id: <1ee7b30a16ebc76bf5503a7c5a8919f3d813edbd.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564388738100001 From: Hyman Huang(=E9=BB=84=E5=8B=87) dirty_rate paraemter of hmp command "set_vcpu_dirty_limit" is invalid if less than 0, so add parameter check for it. Note that this patch also delete the unsolicited help message and clean up the code. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Signed-off-by: Markus Armbruster Reviewed-by: Peter Xu --- softmmu/dirtylimit.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 065ed18afc..dcab9bf2b1 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -514,14 +514,15 @@ void hmp_set_vcpu_dirty_limit(Monitor *mon, const QDi= ct *qdict) int64_t cpu_index =3D qdict_get_try_int(qdict, "cpu_index", -1); Error *err =3D NULL; =20 - qmp_set_vcpu_dirty_limit(!!(cpu_index !=3D -1), cpu_index, dirty_rate,= &err); - if (err) { - hmp_handle_error(mon, err); - return; + if (dirty_rate < 0) { + error_setg(&err, "invalid dirty page limit %ld", dirty_rate); + goto out; } =20 - monitor_printf(mon, "[Please use 'info vcpu_dirty_limit' to query " - "dirty limit for virtual CPU]\n"); + qmp_set_vcpu_dirty_limit(!!(cpu_index !=3D -1), cpu_index, dirty_rate,= &err); + +out: + hmp_handle_error(mon, err); } =20 static struct DirtyLimitInfo *dirtylimit_query_vcpu(int cpu_index) --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564446888355.1940513797683; Thu, 16 Feb 2023 08:20:46 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgyw-0008Bk-GT; Thu, 16 Feb 2023 11:19:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgyh-000866-LU for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:25 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgyf-00054c-EY for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:23 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 861F52800E3; Fri, 17 Feb 2023 00:19:14 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id f08dd78b756545d9a1458872790a429d for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:18 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: huangy81@chinatelecom.cn X-Transaction-ID: f08dd78b756545d9a1458872790a429d X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson Subject: [PATCH v4 03/10] kvm: dirty-ring: Fix race with vcpu creation Date: Fri, 17 Feb 2023 00:18:32 +0800 Message-Id: <1d14deb6684bcb7de1c9633c5bd21113988cc698.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564447073100001 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Peter Xu It's possible that we want to reap a dirty ring on a vcpu that is during creation, because the vcpu is put onto list (CPU_FOREACH visible) before initialization of the structures. In this case: qemu_init_vcpu x86_cpu_realizefn cpu_exec_realizefn cpu_list_add <---- can be probed by CPU_FOREACH qemu_init_vcpu cpus_accel->create_vcpu_thread(cpu); kvm_init_vcpu map kvm_dirty_gfns <--- kvm_dirty_gfns valid Don't try to reap dirty ring on vcpus during creation or it'll crash. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=3D2124756 Reported-by: Xiaohui Li Signed-off-by: Peter Xu --- accel/kvm/kvm-all.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 9b26582655..47483cdfa0 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -685,6 +685,15 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, C= PUState *cpu) uint32_t ring_size =3D s->kvm_dirty_ring_size; uint32_t count =3D 0, fetch =3D cpu->kvm_fetch_index; =20 + /* + * It's possible that we race with vcpu creation code where the vcpu is + * put onto the vcpus list but not yet initialized the dirty ring + * structures. If so, skip it. + */ + if (!cpu->created) { + return 0; + } + assert(dirty_gfns && ring_size); trace_kvm_dirty_ring_reap_vcpu(cpu->cpu_index); =20 --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564411786771.3580960952468; Thu, 16 Feb 2023 08:20:11 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz8-0000l9-PN; Thu, 16 Feb 2023 11:19:50 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgz6-0000GT-8R for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:48 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgyk-000569-SM for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:46 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id F1DB62800E4; Fri, 17 Feb 2023 00:19:18 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id c249d8d1bceb4b5ebed569a933f4addc for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:22 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: c249d8d1bceb4b5ebed569a933f4addc X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 04/10] qapi/migration: Introduce x-vcpu-dirty-limit-period parameter Date: Fri, 17 Feb 2023 00:18:33 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564412910100001 From: Hyman Huang(=E9=BB=84=E5=8B=87) Introduce "x-vcpu-dirty-limit-period" migration experimental parameter, which is in the range of 1 to 1000ms and used to make dirtyrate calculation period configurable. Currently with the "x-vcpu-dirty-limit-period" varies, the total time of live migration changes, test results show the optimal value of "x-vcpu-dirty-limit-period" ranges from 500ms to 1000 ms. "x-vcpu-dirty-limit-period" should be made stable once it proves best value can not be determined with developer's experiments. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Signed-off-by: Markus Armbruster --- migration/migration-hmp-cmds.c | 8 ++++++++ migration/migration.c | 27 +++++++++++++++++++++++++++ qapi/migration.json | 33 ++++++++++++++++++++++++++------- 3 files changed, 61 insertions(+), 7 deletions(-) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index ef25bc8929..3bc751bec9 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -345,6 +345,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const Q= Dict *qdict) } } } + + monitor_printf(mon, "%s: %" PRIu64 " ms\n", + MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERI= OD), + params->x_vcpu_dirty_limit_period); } =20 qapi_free_MigrationParameters(params); @@ -601,6 +605,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDi= ct *qdict) error_setg(&err, "The block-bitmap-mapping parameter can only be s= et " "through QMP"); break; + case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD: + p->has_x_vcpu_dirty_limit_period =3D true; + visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err); + break; default: assert(0); } diff --git a/migration/migration.c b/migration/migration.c index 90fca70cb7..6162f048ae 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -119,6 +119,8 @@ #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5 #define DEFAULT_MIGRATE_ANNOUNCE_STEP 100 =20 +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* microsecond= */ + static NotifierList migration_state_notifiers =3D NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); =20 @@ -1016,6 +1018,9 @@ MigrationParameters *qmp_query_migrate_parameters(Err= or **errp) s->parameters.block_bitmap_mapping); } =20 + params->has_x_vcpu_dirty_limit_period =3D true; + params->x_vcpu_dirty_limit_period =3D s->parameters.x_vcpu_dirty_limit= _period; + return params; } =20 @@ -1660,6 +1665,15 @@ static bool migrate_params_check(MigrationParameters= *params, Error **errp) } #endif =20 + if (params->has_x_vcpu_dirty_limit_period && + (params->x_vcpu_dirty_limit_period < 1 || + params->x_vcpu_dirty_limit_period > 1000)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "x-vcpu-dirty-limit-period", + "a value between 1 and 1000"); + return false; + } + return true; } =20 @@ -1759,6 +1773,10 @@ static void migrate_params_test_apply(MigrateSetPara= meters *params, dest->has_block_bitmap_mapping =3D true; dest->block_bitmap_mapping =3D params->block_bitmap_mapping; } + + if (params->has_x_vcpu_dirty_limit_period) { + dest->x_vcpu_dirty_limit_period =3D params->x_vcpu_dirty_limit_per= iod; + } } =20 static void migrate_params_apply(MigrateSetParameters *params, Error **err= p) @@ -1881,6 +1899,11 @@ static void migrate_params_apply(MigrateSetParameter= s *params, Error **errp) QAPI_CLONE(BitmapMigrationNodeAliasList, params->block_bitmap_mapping); } + + if (params->has_x_vcpu_dirty_limit_period) { + s->parameters.x_vcpu_dirty_limit_period =3D + params->x_vcpu_dirty_limit_period; + } } =20 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) @@ -4513,6 +4536,9 @@ static Property migration_properties[] =3D { DEFINE_PROP_STRING("tls-creds", MigrationState, parameters.tls_creds), DEFINE_PROP_STRING("tls-hostname", MigrationState, parameters.tls_host= name), DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz), + DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState, + parameters.x_vcpu_dirty_limit_period, + DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD), =20 /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -4602,6 +4628,7 @@ static void migration_instance_init(Object *obj) params->has_announce_max =3D true; params->has_announce_rounds =3D true; params->has_announce_step =3D true; + params->has_x_vcpu_dirty_limit_period =3D true; =20 qemu_sem_init(&ms->postcopy_pause_sem, 0); qemu_sem_init(&ms->postcopy_pause_rp_sem, 0); diff --git a/qapi/migration.json b/qapi/migration.json index c84fa10e86..f43e4061b4 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -775,9 +775,13 @@ # names are mapped to themselves. Nodes are mapped= to their # block device name if there is one, and to their n= ode name # otherwise. (Since 5.2) +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty lim= it during +# live migration. Should be in the range 1 to = 1000ms, +# defaults to 1000ms. (Since 8.0) # # Features: -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -795,8 +799,9 @@ 'multifd-channels', 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', - 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'multifd-zlib-level', 'multifd-zstd-level', + 'block-bitmap-mapping', + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable']= } ] } =20 ## # @MigrateSetParameters: @@ -941,8 +946,13 @@ # block device name if there is one, and to their n= ode name # otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty lim= it during +# live migration. Should be in the range 1 to = 1000ms, +# defaults to 1000ms. (Since 8.0) +# # Features: -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -976,7 +986,9 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } }= } =20 ## # @migrate-set-parameters: @@ -1141,8 +1153,13 @@ # block device name if there is one, and to their n= ode name # otherwise. (Since 5.2) # +# @x-vcpu-dirty-limit-period: Periodic time (in milliseconds) of dirty lim= it during +# live migration. Should be in the range 1 to = 1000ms, +# defaults to 1000ms. (Since 8.0) +# # Features: -# @unstable: Member @x-checkpoint-delay is experimental. +# @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period +# are experimental. # # Since: 2.4 ## @@ -1174,7 +1191,9 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*x-vcpu-dirty-limit-period': { 'type': 'uint64', + 'features': [ 'unstable' ] } }= } =20 ## # @query-migrate-parameters: --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564431958799.6590877202672; Thu, 16 Feb 2023 08:20:31 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgzB-0001Fj-8E; Thu, 16 Feb 2023 11:19:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgzA-00013u-6P for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:52 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz6-00056e-KJ for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:51 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 6D26C2800E0; Fri, 17 Feb 2023 00:19:23 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id e29e24eb84094dd8bbb670df53ad86fb for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:27 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: e29e24eb84094dd8bbb670df53ad86fb X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 05/10] qapi/migration: Introduce vcpu-dirty-limit parameters Date: Fri, 17 Feb 2023 00:18:34 +0800 Message-Id: <901dc3fb01cb4bee7209e22de0fc5ca4386d313d.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564433071100001 From: Hyman Huang(=E9=BB=84=E5=8B=87) Introduce "vcpu-dirty-limit" migration parameter used to limit dirty page rate during live migration. "vcpu-dirty-limit" and "x-vcpu-dirty-limit-period" are two dirty-limit-related migration parameters, which can be set before and during live migration by qmp migrate-set-parameters. This two parameters are used to help implement the dirty page rate limit algo of migration. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Acked-by: Peter Xu --- migration/migration-hmp-cmds.c | 8 ++++++++ migration/migration.c | 23 +++++++++++++++++++++++ qapi/migration.json | 18 +++++++++++++++--- 3 files changed, 46 insertions(+), 3 deletions(-) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 3bc751bec9..a61ec80d9d 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -349,6 +349,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const Q= Dict *qdict) monitor_printf(mon, "%s: %" PRIu64 " ms\n", MigrationParameter_str(MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERI= OD), params->x_vcpu_dirty_limit_period); + + monitor_printf(mon, "%s: %" PRIu64 " MB/s\n", + MigrationParameter_str(MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT), + params->vcpu_dirty_limit); } =20 qapi_free_MigrationParameters(params); @@ -609,6 +613,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDi= ct *qdict) p->has_x_vcpu_dirty_limit_period =3D true; visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err); break; + case MIGRATION_PARAMETER_VCPU_DIRTY_LIMIT: + p->has_vcpu_dirty_limit =3D true; + visit_type_size(v, param, &p->vcpu_dirty_limit, &err); + break; default: assert(0); } diff --git a/migration/migration.c b/migration/migration.c index 6162f048ae..e479c86575 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -120,6 +120,7 @@ #define DEFAULT_MIGRATE_ANNOUNCE_STEP 100 =20 #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* microsecond= */ +#define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ =20 static NotifierList migration_state_notifiers =3D NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); @@ -1021,6 +1022,9 @@ MigrationParameters *qmp_query_migrate_parameters(Err= or **errp) params->has_x_vcpu_dirty_limit_period =3D true; params->x_vcpu_dirty_limit_period =3D s->parameters.x_vcpu_dirty_limit= _period; =20 + params->has_vcpu_dirty_limit =3D true; + params->vcpu_dirty_limit =3D s->parameters.vcpu_dirty_limit; + return params; } =20 @@ -1674,6 +1678,14 @@ static bool migrate_params_check(MigrationParameters= *params, Error **errp) return false; } =20 + if (params->has_vcpu_dirty_limit && + (params->vcpu_dirty_limit < 1)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "vcpu_dirty_limit", + "is invalid, it must greater then 1 MB/s"); + return false; + } + return true; } =20 @@ -1777,6 +1789,10 @@ static void migrate_params_test_apply(MigrateSetPara= meters *params, if (params->has_x_vcpu_dirty_limit_period) { dest->x_vcpu_dirty_limit_period =3D params->x_vcpu_dirty_limit_per= iod; } + + if (params->has_vcpu_dirty_limit) { + dest->vcpu_dirty_limit =3D params->vcpu_dirty_limit; + } } =20 static void migrate_params_apply(MigrateSetParameters *params, Error **err= p) @@ -1904,6 +1920,9 @@ static void migrate_params_apply(MigrateSetParameters= *params, Error **errp) s->parameters.x_vcpu_dirty_limit_period =3D params->x_vcpu_dirty_limit_period; } + if (params->has_vcpu_dirty_limit) { + s->parameters.vcpu_dirty_limit =3D params->vcpu_dirty_limit; + } } =20 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) @@ -4539,6 +4558,9 @@ static Property migration_properties[] =3D { DEFINE_PROP_UINT64("x-vcpu-dirty-limit-period", MigrationState, parameters.x_vcpu_dirty_limit_period, DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD), + DEFINE_PROP_UINT64("vcpu-dirty-limit", MigrationState, + parameters.vcpu_dirty_limit, + DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT), =20 /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -4629,6 +4651,7 @@ static void migration_instance_init(Object *obj) params->has_announce_rounds =3D true; params->has_announce_step =3D true; params->has_x_vcpu_dirty_limit_period =3D true; + params->has_vcpu_dirty_limit =3D true; =20 qemu_sem_init(&ms->postcopy_pause_sem, 0); qemu_sem_init(&ms->postcopy_pause_rp_sem, 0); diff --git a/qapi/migration.json b/qapi/migration.json index f43e4061b4..d33cc2d582 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -779,6 +779,9 @@ # live migration. Should be in the range 1 to = 1000ms, # defaults to 1000ms. (Since 8.0) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.0) +# # Features: # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period # are experimental. @@ -801,7 +804,8 @@ 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', 'block-bitmap-mapping', - { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable']= } ] } + { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable']= }, + 'vcpu-dirty-limit'] } =20 ## # @MigrateSetParameters: @@ -950,6 +954,9 @@ # live migration. Should be in the range 1 to = 1000ms, # defaults to 1000ms. (Since 8.0) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.0) +# # Features: # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period # are experimental. @@ -988,7 +995,8 @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } }= } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } =20 ## # @migrate-set-parameters: @@ -1157,6 +1165,9 @@ # live migration. Should be in the range 1 to = 1000ms, # defaults to 1000ms. (Since 8.0) # +# @vcpu-dirty-limit: Dirtyrate limit (MB/s) during live migration. +# Defaults to 1. (Since 8.0) +# # Features: # @unstable: Members @x-checkpoint-delay and @x-vcpu-dirty-limit-period # are experimental. @@ -1193,7 +1204,8 @@ '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', - 'features': [ 'unstable' ] } }= } + 'features': [ 'unstable' ] }, + '*vcpu-dirty-limit': 'uint64'} } =20 ## # @query-migrate-parameters: --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564465421521.8752276787225; Thu, 16 Feb 2023 08:21:05 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgzA-00014k-Bf; Thu, 16 Feb 2023 11:19:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgz8-0000hk-DH for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:50 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz5-00056r-VJ for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:50 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id E536D2800E5; Fri, 17 Feb 2023 00:19:27 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id 4ba4661da9464483a51dfe42991691a4 for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:31 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 4ba4661da9464483a51dfe42991691a4 X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 06/10] migration: Introduce dirty-limit capability Date: Fri, 17 Feb 2023 00:18:35 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564467191100009 From: Hyman Huang(=E9=BB=84=E5=8B=87) Introduce migration dirty-limit capability, which can be turned on before live migration and limit dirty page rate durty live migration. Introduce migrate_dirty_limit function to help check if dirty-limit capability enabled during live migration. Meanwhile, refactor vcpu_dirty_rate_stat_collect so that period can be configured instead of hardcoded. dirty-limit capability is kind of like auto-converge but using dirty limit instead of traditional cpu-throttle to throttle guest down. To enable this feature, turn on the dirty-limit capability before live migration using migrate-set-capabilities, and set the parameters "x-vcpu-dirty-limit-period", "vcpu-dirty-limit" suitably to speed up convergence. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Acked-by: Peter Xu --- migration/migration.c | 25 +++++++++++++++++++++++++ migration/migration.h | 1 + qapi/migration.json | 4 +++- softmmu/dirtylimit.c | 11 ++++++++++- 4 files changed, 39 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index e479c86575..f890e5966a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -64,6 +64,7 @@ #include "yank_functions.h" #include "sysemu/qtest.h" #include "ui/qemu-spice.h" +#include "sysemu/kvm.h" =20 #define MAX_THROTTLE (128 << 20) /* Migration transfer speed throttl= ing */ =20 @@ -1444,6 +1445,20 @@ static bool migrate_caps_check(bool *cap_list, } } =20 + if (cap_list[MIGRATION_CAPABILITY_DIRTY_LIMIT]) { + if (cap_list[MIGRATION_CAPABILITY_AUTO_CONVERGE]) { + error_setg(errp, "dirty-limit conflicts with auto-converge" + " either of then available currently"); + return false; + } + + if (!kvm_enabled() || !kvm_dirty_ring_enabled()) { + error_setg(errp, "dirty-limit requires KVM with accelerator" + " property 'dirty-ring-size' set"); + return false; + } + } + return true; } =20 @@ -2635,6 +2650,15 @@ bool migrate_auto_converge(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE]; } =20 +bool migrate_dirty_limit(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_LIMIT]; +} + bool migrate_zero_blocks(void) { MigrationState *s; @@ -4583,6 +4607,7 @@ static Property migration_properties[] =3D { DEFINE_PROP_MIG_CAP("x-zero-copy-send", MIGRATION_CAPABILITY_ZERO_COPY_SEND), #endif + DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT), =20 DEFINE_PROP_END_OF_LIST(), }; diff --git a/migration/migration.h b/migration/migration.h index 2da2f8a164..cd2e9bfeea 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -418,6 +418,7 @@ bool migrate_ignore_shared(void); bool migrate_validate_uuid(void); =20 bool migrate_auto_converge(void); +bool migrate_dirty_limit(void); bool migrate_use_multifd(void); bool migrate_pause_before_switchover(void); int migrate_multifd_channels(void); diff --git a/qapi/migration.json b/qapi/migration.json index d33cc2d582..b7a92be055 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -477,6 +477,8 @@ # will be handled faster. This is a performance featur= e and # should not affect the correctness of postcopy migrati= on. # (since 7.1) +# @dirty-limit: Use dirty-limit to throttle down guest if enabled. +# (since 8.0) # # Features: # @unstable: Members @x-colo and @x-ignore-shared are experimental. @@ -492,7 +494,7 @@ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, 'validate-uuid', 'background-snapshot', - 'zero-copy-send', 'postcopy-preempt'] } + 'zero-copy-send', 'postcopy-preempt', 'dirty-limit'] } =20 ## # @MigrationCapabilityStatus: diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index dcab9bf2b1..52d1b2c6fa 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -22,6 +22,8 @@ #include "exec/memory.h" #include "hw/boards.h" #include "sysemu/kvm.h" +#include "migration/misc.h" +#include "migration/migration.h" #include "trace.h" =20 /* @@ -74,11 +76,18 @@ static bool dirtylimit_quit; =20 static void vcpu_dirty_rate_stat_collect(void) { + MigrationState *s =3D migrate_get_current(); VcpuStat stat; int i =3D 0; + int64_t period =3D DIRTYLIMIT_CALC_TIME_MS; + + if (migrate_dirty_limit() && + migration_is_active(s)) { + period =3D s->parameters.x_vcpu_dirty_limit_period; + } =20 /* calculate vcpu dirtyrate */ - vcpu_calculate_dirtyrate(DIRTYLIMIT_CALC_TIME_MS, + vcpu_calculate_dirtyrate(period, &stat, GLOBAL_DIRTY_LIMIT, false); --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564401347902.8714036084093; Thu, 16 Feb 2023 08:20:01 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgzC-0001UG-Fb; Thu, 16 Feb 2023 11:19:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgz9-0000yE-Oj for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:51 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz6-00057L-Kr for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:51 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 6D4812800E6; Fri, 17 Feb 2023 00:19:32 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id 925b61c1a00b48d7a910b9772666548d for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:36 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 925b61c1a00b48d7a910b9772666548d X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 07/10] migration: Refactor auto-converge capability logic Date: Fri, 17 Feb 2023 00:18:36 +0800 Message-Id: <7973ba24842fe86264bc674f6c36c0b9e7540d88.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564402868100005 From: Hyman Huang(=E9=BB=84=E5=8B=87) Check if block migration is running before throttling guest down in auto-converge way. Note that this modification is kind of like code clean, because block migration does not depend on auto-converge capability, so the order of checks can be adjusted. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Acked-by: Peter Xu --- migration/ram.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 521912385d..3e5dff4068 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1200,7 +1200,11 @@ static void migration_trigger_throttle(RAMState *rs) /* During block migration the auto-converge logic incorrectly detects * that ram migration makes no progress. Avoid this by disabling the * throttling logic during the bulk phase of block migration. */ - if (migrate_auto_converge() && !blk_mig_bulk_active()) { + if (blk_mig_bulk_active()) { + return; + } + + if (migrate_auto_converge()) { /* The following detection logic can be refined later. For now: Check to see if the ratio between dirtied bytes and the approx. amount of bytes that just got transferred since the last time --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564460926971.2175563386377; Thu, 16 Feb 2023 08:21:00 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgzD-0001gW-Qi; Thu, 16 Feb 2023 11:19:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgzB-0001Hy-B1 for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:53 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz5-00057T-Uo for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:53 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 9F1492800E9; Fri, 17 Feb 2023 00:19:36 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id 7f2bdb0d682a4716855b3fe0cd6ad4d2 for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:41 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 7f2bdb0d682a4716855b3fe0cd6ad4d2 X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 08/10] migration: Implement dirty-limit convergence algo Date: Fri, 17 Feb 2023 00:18:37 +0800 Message-Id: <9166468b7365583486b48314254ab11a5e687780.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564463179100001 From: Hyman Huang(=E9=BB=84=E5=8B=87) Implement dirty-limit convergence algo for live migration, which is kind of like auto-converge algo but using dirty-limit instead of cpu throttle to make migration convergent. Enable dirty page limit if dirty_rate_high_cnt greater than 2 when dirty-limit capability enabled, Disable dirty-limit if migration be cancled. Note that "set_vcpu_dirty_limit", "cancel_vcpu_dirty_limit" commands are not allowed during dirty-limit live migration. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Signed-off-by: Markus Armbruster --- migration/migration.c | 3 ++ migration/ram.c | 63 ++++++++++++++++++++++++++++++++---------- migration/trace-events | 1 + softmmu/dirtylimit.c | 22 +++++++++++++++ 4 files changed, 74 insertions(+), 15 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index f890e5966a..7ccbc07257 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -256,6 +256,9 @@ void migration_cancel(const Error *error) if (error) { migrate_set_error(current_migration, error); } + if (migrate_dirty_limit()) { + qmp_cancel_vcpu_dirty_limit(false, -1, NULL); + } migrate_fd_cancel(current_migration); } =20 diff --git a/migration/ram.c b/migration/ram.c index 3e5dff4068..24d26b5135 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -45,6 +45,7 @@ #include "qapi/error.h" #include "qapi/qapi-types-migration.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qapi-commands-migration.h" #include "qapi/qmp/qerror.h" #include "trace.h" #include "exec/ram_addr.h" @@ -57,6 +58,8 @@ #include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" +#include "sysemu/dirtylimit.h" +#include "sysemu/kvm.h" =20 #include "hw/boards.h" /* for machine_dump_guest_core() */ =20 @@ -1188,6 +1191,30 @@ static void migration_update_rates(RAMState *rs, int= 64_t end_time) } } =20 +/* + * Enable dirty-limit to throttle down the guest + */ +static void migration_dirty_limit_guest(void) +{ + static int64_t quota_dirtyrate; + MigrationState *s =3D migrate_get_current(); + + /* + * If dirty limit already enabled and migration parameter + * vcpu-dirty-limit untouched. + */ + if (dirtylimit_in_service() && + quota_dirtyrate =3D=3D s->parameters.vcpu_dirty_limit) { + return; + } + + quota_dirtyrate =3D s->parameters.vcpu_dirty_limit; + + /* Set or update quota dirty limit */ + qmp_set_vcpu_dirty_limit(false, -1, quota_dirtyrate, NULL); + trace_migration_dirty_limit_guest(quota_dirtyrate); +} + static void migration_trigger_throttle(RAMState *rs) { MigrationState *s =3D migrate_get_current(); @@ -1197,26 +1224,32 @@ static void migration_trigger_throttle(RAMState *rs) uint64_t bytes_dirty_period =3D rs->num_dirty_pages_period * TARGET_PA= GE_SIZE; uint64_t bytes_dirty_threshold =3D bytes_xfer_period * threshold / 100; =20 - /* During block migration the auto-converge logic incorrectly detects - * that ram migration makes no progress. Avoid this by disabling the - * throttling logic during the bulk phase of block migration. */ - if (blk_mig_bulk_active()) { - return; - } + /* + * The following detection logic can be refined later. For now: + * Check to see if the ratio between dirtied bytes and the approx. + * amount of bytes that just got transferred since the last time + * we were in this routine reaches the threshold. If that happens + * twice, start or increase throttling. + */ =20 - if (migrate_auto_converge()) { - /* The following detection logic can be refined later. For now: - Check to see if the ratio between dirtied bytes and the approx. - amount of bytes that just got transferred since the last time - we were in this routine reaches the threshold. If that happens - twice, start or increase throttling. */ + if ((bytes_dirty_period > bytes_dirty_threshold) && + (++rs->dirty_rate_high_cnt >=3D 2)) { + rs->dirty_rate_high_cnt =3D 0; + /* + * During block migration the auto-converge logic incorrectly dete= cts + * that ram migration makes no progress. Avoid this by disabling t= he + * throttling logic during the bulk phase of block migration + */ + if (blk_mig_bulk_active()) { + return; + } =20 - if ((bytes_dirty_period > bytes_dirty_threshold) && - (++rs->dirty_rate_high_cnt >=3D 2)) { + if (migrate_auto_converge()) { trace_migration_throttle(); - rs->dirty_rate_high_cnt =3D 0; mig_throttle_guest_down(bytes_dirty_period, bytes_dirty_threshold); + } else if (migrate_dirty_limit()) { + migration_dirty_limit_guest(); } } } diff --git a/migration/trace-events b/migration/trace-events index 67b65a70ff..a689807a49 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -91,6 +91,7 @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, uns= igned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx" migration_throttle(void) "" +migration_dirty_limit_guest(int64_t dirtyrate) "guest dirty page rate limi= t %" PRIi64 " MB/s" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: sta= rt: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%= s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" ram_load_postcopy_loop(int channel, uint64_t addr, int flags) "chan=3D%d a= ddr=3D0x%" PRIx64 " flags=3D0x%x" diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index 52d1b2c6fa..ae77ebe5c5 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -438,6 +438,8 @@ void qmp_cancel_vcpu_dirty_limit(bool has_cpu_index, int64_t cpu_index, Error **errp) { + MigrationState *ms =3D migrate_get_current(); + if (!kvm_enabled() || !kvm_dirty_ring_enabled()) { return; } @@ -451,6 +453,15 @@ void qmp_cancel_vcpu_dirty_limit(bool has_cpu_index, return; } =20 + if (migration_is_running(ms->state) && + (!qemu_thread_is_self(&ms->thread)) && + migrate_dirty_limit() && + dirtylimit_in_service()) { + error_setg(errp, "can't cancel dirty page limit while" + " migration is running"); + return; + } + dirtylimit_state_lock(); =20 if (has_cpu_index) { @@ -486,6 +497,8 @@ void qmp_set_vcpu_dirty_limit(bool has_cpu_index, uint64_t dirty_rate, Error **errp) { + MigrationState *ms =3D migrate_get_current(); + if (!kvm_enabled() || !kvm_dirty_ring_enabled()) { error_setg(errp, "dirty page limit feature requires KVM with" " accelerator property 'dirty-ring-size' set'"); @@ -502,6 +515,15 @@ void qmp_set_vcpu_dirty_limit(bool has_cpu_index, return; } =20 + if (migration_is_running(ms->state) && + (!qemu_thread_is_self(&ms->thread)) && + migrate_dirty_limit() && + dirtylimit_in_service()) { + error_setg(errp, "can't cancel dirty page limit while" + " migration is running"); + return; + } + dirtylimit_state_lock(); =20 if (!dirtylimit_in_service()) { --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564441955288.5182624777734; Thu, 16 Feb 2023 08:20:41 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgzG-0001vH-6d; Thu, 16 Feb 2023 11:19:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgzE-0001kx-7e for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:56 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz6-00057o-Ks for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:55 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 1F51D2800E2; Fri, 17 Feb 2023 00:19:42 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id 39ca26f00da34716880fb8ea48d74c5c for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:46 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 39ca26f00da34716880fb8ea48d74c5c X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 09/10] migration: Extend query-migrate to provide dirty page limit info Date: Fri, 17 Feb 2023 00:18:38 +0800 Message-Id: <92b74c98dbb6df18cda73149088083d276a660a3.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564443077100001 From: Hyman Huang(=E9=BB=84=E5=8B=87) Extend query-migrate to provide throttle time and estimated ring full time with dirty-limit capability enabled, through which we can observe if dirty limit take effect during live migration. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) Signed-off-by: Markus Armbruster --- include/sysemu/dirtylimit.h | 2 ++ migration/migration-hmp-cmds.c | 10 +++++++++ migration/migration.c | 10 +++++++++ qapi/migration.json | 15 ++++++++++++- softmmu/dirtylimit.c | 39 ++++++++++++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 1 deletion(-) diff --git a/include/sysemu/dirtylimit.h b/include/sysemu/dirtylimit.h index 8d2c1f3a6b..410a2bc0b6 100644 --- a/include/sysemu/dirtylimit.h +++ b/include/sysemu/dirtylimit.h @@ -34,4 +34,6 @@ void dirtylimit_set_vcpu(int cpu_index, void dirtylimit_set_all(uint64_t quota, bool enable); void dirtylimit_vcpu_execute(CPUState *cpu); +int64_t dirtylimit_throttle_time_per_round(void); +int64_t dirtylimit_ring_full_time(void); #endif diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index a61ec80d9d..1f090faec5 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -171,6 +171,16 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->cpu_throttle_percentage); } =20 + if (info->has_dirty_limit_throttle_time_per_round) { + monitor_printf(mon, "dirty-limit throttle time: %" PRIi64 " us\n", + info->dirty_limit_throttle_time_per_round); + } + + if (info->has_dirty_limit_ring_full_time) { + monitor_printf(mon, "dirty-limit ring full time: %" PRIi64 " us\n", + info->dirty_limit_ring_full_time); + } + if (info->has_postcopy_blocktime) { monitor_printf(mon, "postcopy blocktime: %u\n", info->postcopy_blocktime); diff --git a/migration/migration.c b/migration/migration.c index 7ccbc07257..1f1e1f2268 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -65,6 +65,7 @@ #include "sysemu/qtest.h" #include "ui/qemu-spice.h" #include "sysemu/kvm.h" +#include "sysemu/dirtylimit.h" =20 #define MAX_THROTTLE (128 << 20) /* Migration transfer speed throttl= ing */ =20 @@ -1203,6 +1204,15 @@ static void populate_ram_info(MigrationInfo *info, M= igrationState *s) info->ram->remaining =3D ram_bytes_remaining(); info->ram->dirty_pages_rate =3D ram_counters.dirty_pages_rate; } + + if (migrate_dirty_limit() && dirtylimit_in_service()) { + info->has_dirty_limit_throttle_time_per_round =3D true; + info->dirty_limit_throttle_time_per_round =3D + dirtylimit_throttle_time_per_round(); + + info->has_dirty_limit_ring_full_time =3D true; + info->dirty_limit_ring_full_time =3D dirtylimit_us_ring_full(); + } } =20 static void populate_disk_info(MigrationInfo *info) diff --git a/qapi/migration.json b/qapi/migration.json index b7a92be055..f511771101 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -242,6 +242,17 @@ # Present and non-empty when migration is blocked. # (since 6.0) # +# @dirty-limit-throttle-time-per-round: Maximum throttle time (in microsec= onds) of virtual +# CPUs each dirty ring full round, w= hich shows how +# MigrationCapability dirty-limit af= fects the guest +# during live migration. (since 8.0) +# +# @dirty-limit-ring-full-time: Estimated average dirty ring full time (in = microseconds) +# each dirty ring full round, note that the v= alue equals +# dirty ring memory size divided by average d= irty page rate +# of virtual CPU, which can be used to observ= e the average +# memory load of virtual CPU indirectly. (sin= ce 8.0) +# # Since: 0.14 ## { 'struct': 'MigrationInfo', @@ -259,7 +270,9 @@ '*postcopy-blocktime' : 'uint32', '*postcopy-vcpu-blocktime': ['uint32'], '*compression': 'CompressionStats', - '*socket-address': ['SocketAddress'] } } + '*socket-address': ['SocketAddress'], + '*dirty-limit-throttle-time-per-round': 'int64', + '*dirty-limit-ring-full-time': 'int64'} } =20 ## # @query-migrate: diff --git a/softmmu/dirtylimit.c b/softmmu/dirtylimit.c index ae77ebe5c5..3c07844a11 100644 --- a/softmmu/dirtylimit.c +++ b/softmmu/dirtylimit.c @@ -568,6 +568,45 @@ static struct DirtyLimitInfo *dirtylimit_query_vcpu(in= t cpu_index) return info; } =20 +/* Return the max throttle time of each virtual CPU */ +int64_t dirtylimit_throttle_time_per_round(void) +{ + CPUState *cpu; + int64_t max =3D 0; + + CPU_FOREACH(cpu) { + if (cpu->throttle_us_per_full > max) { + max =3D cpu->throttle_us_per_full; + } + } + + return max; +} + +/* + * Estimate average dirty ring full time of each virtaul CPU. + * Return -1 if guest doesn't dirty memory. + */ +int64_t dirtylimit_us_ring_full(void) +{ + CPUState *cpu; + uint64_t curr_rate =3D 0; + int nvcpus =3D 0; + + CPU_FOREACH(cpu) { + if (cpu->running) { + nvcpus++; + curr_rate +=3D vcpu_dirty_rate_get(cpu->cpu_index); + } + } + + if (!curr_rate || !nvcpus) { + return -1; + } + + return dirtylimit_dirty_ring_full_time(curr_rate / nvcpus); +} + static struct DirtyLimitInfoList *dirtylimit_query_all(void) { int i, index; --=20 2.17.1 From nobody Sat May 18 21:00:25 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1676564445863184.69032042758067; Thu, 16 Feb 2023 08:20:45 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgzF-0001om-1P; Thu, 16 Feb 2023 11:19:57 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pSgzC-0001U4-C4 for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:54 -0500 Received: from prt-mail.chinatelecom.cn ([42.123.76.223] helo=chinatelecom.cn) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pSgz9-00058h-UJ for qemu-devel@nongnu.org; Thu, 16 Feb 2023 11:19:54 -0500 Received: from clientip-118.116.19.27 (unknown [172.18.0.188]) by chinatelecom.cn (HERMES) with SMTP id 8C2602800E4; Fri, 17 Feb 2023 00:19:46 +0800 (CST) Received: from ([118.116.19.27]) by app0023 with ESMTP id 564a714ee3cd4f278a55ec88c00a63f2 for qemu-devel@nongnu.org; Fri, 17 Feb 2023 00:19:50 CST HMM_SOURCE_IP: 172.18.0.188:50698.1319324123 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP X-189-SAVE-TO-SEND: +huangy81@chinatelecom.cn X-Transaction-ID: 564a714ee3cd4f278a55ec88c00a63f2 X-Real-From: huangy81@chinatelecom.cn X-Receive-IP: 118.116.19.27 X-MEDUSA-Status: 0 From: huangy81@chinatelecom.cn To: qemu-devel Cc: Markus Armbruster , Peter Xu , "Dr. David Alan Gilbert" , Juan Quintela , Thomas Huth , Paolo Bonzini , Eric Blake , Peter Maydell , Richard Henderson , =?UTF-8?q?Hyman=20Huang=28=E9=BB=84=E5=8B=87=29?= Subject: [PATCH v4 10/10] tests: Add migration dirty-limit capability test Date: Fri, 17 Feb 2023 00:18:39 +0800 Message-Id: <7b156d0bb1cc3841066034d312ca379ef4c98ab5.1676563222.git.huangy81@chinatelecom.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=42.123.76.223; envelope-from=huangy81@chinatelecom.cn; helo=chinatelecom.cn X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1676564447119100002 From: Hyman Huang(=E9=BB=84=E5=8B=87) Add migration dirty-limit capability test if kernel support dirty ring. Migration dirty-limit capability introduce dirty limit capability, two parameters: x-vcpu-dirty-limit-period and vcpu-dirty-limit are introduced to implement the live migration with dirty limit. The test case does the following things: 1. start src, dst vm and enable dirty-limit capability 2. start migrate and set cancel it to check if dirty limit stop working. 3. restart dst vm 4. start migrate and enable dirty-limit capability 5. check if migration satisfy the convergence condition during pre-switchover phase. Signed-off-by: Hyman Huang(=E9=BB=84=E5=8B=87) --- tests/qtest/migration-test.c | 157 +++++++++++++++++++++++++++++++++++ 1 file changed, 157 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 109bc8e7b1..6aad86e572 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2434,6 +2434,161 @@ static void test_vcpu_dirty_limit(void) dirtylimit_stop_vm(vm); } =20 +static void migrate_dirty_limit_wait_showup(QTestState *from, + const int64_t period, + const int64_t value) +{ + /* Enable dirty limit capability */ + migrate_set_capability(from, "dirty-limit", true); + + /* Set dirty limit parameters */ + migrate_set_parameter_int(from, "x-vcpu-dirty-limit-period", period); + migrate_set_parameter_int(from, "vcpu-dirty-limit", value); + + /* Make sure migrate can't converge */ + migrate_ensure_non_converge(from); + + /* To check limit rate after precopy */ + migrate_set_capability(from, "pause-before-switchover", true); + + /* Wait for the serial output from the source */ + wait_for_serial("src_serial"); +} + +/* + * This test does: + * source target + * migrate_incoming + * migrate + * migrate_cancel + * restart target + * migrate + * + * And see that if dirty limit works correctly + */ +static void test_migrate_dirty_limit(void) +{ + g_autofree char *uri =3D g_strdup_printf("unix:%s/migsocket", tmpfs); + QTestState *from, *to; + int64_t remaining, throttle_us_per_full; + /* + * We want the test to be stable and as fast as possible. + * E.g., with 1Gb/s bandwith migration may pass without dirty limit, + * so we need to decrease a bandwidth. + */ + const int64_t dirtylimit_period =3D 1000, dirtylimit_value =3D 50; + const int64_t max_bandwidth =3D 400000000; /* ~400Mb/s */ + const int64_t downtime_limit =3D 250; /* 250ms */ + /* + * We migrate through unix-socket (> 500Mb/s). + * Thus, expected migration speed ~=3D bandwidth limit (< 500Mb/s). + * So, we can predict expected_threshold + */ + const int64_t expected_threshold =3D max_bandwidth * downtime_limit / = 1000; + int max_try_count =3D 10; + MigrateCommon args =3D { + .start =3D { + .hide_stderr =3D true, + .use_dirty_ring =3D true, + }, + .listen_uri =3D uri, + .connect_uri =3D uri, + }; + + /* Start src, dst vm */ + if (test_migrate_start(&from, &to, args.listen_uri, &args.start)) { + return; + } + + /* Prepare for dirty limit migration and wait src vm show up */ + migrate_dirty_limit_wait_showup(from, dirtylimit_period, dirtylimit_va= lue); + + /* Start migrate */ + migrate_qmp(from, uri, "{}"); + + /* Wait for dirty limit throttle begin */ + throttle_us_per_full =3D 0; + while (throttle_us_per_full =3D=3D 0) { + throttle_us_per_full =3D + read_migrate_property_int(from, + "dirty-limit-throttle-time-per-round"); + usleep(100); + g_assert_false(got_stop); + } + + /* Now cancel migrate and wait for dirty limit throttle switch off */ + migrate_cancel(from); + wait_for_migration_status(from, "cancelled", NULL); + + /* Check if dirty limit throttle switched off, set timeout 1ms */ + do { + throttle_us_per_full =3D + read_migrate_property_int(from, + "dirty-limit-throttle-time-per-round"); + usleep(100); + g_assert_false(got_stop); + } while (throttle_us_per_full !=3D 0 && --max_try_count); + + /* Assert dirty limit is not in service */ + g_assert_cmpint(throttle_us_per_full, =3D=3D, 0); + + args =3D (MigrateCommon) { + .start =3D { + .only_target =3D true, + .use_dirty_ring =3D true, + }, + .listen_uri =3D uri, + .connect_uri =3D uri, + }; + + /* Restart dst vm, src vm already show up so we needn't wait anymore */ + if (test_migrate_start(&from, &to, args.listen_uri, &args.start)) { + return; + } + + /* Start migrate */ + migrate_qmp(from, uri, "{}"); + + /* Wait for dirty limit throttle begin */ + throttle_us_per_full =3D 0; + while (throttle_us_per_full =3D=3D 0) { + throttle_us_per_full =3D + read_migrate_property_int(from, + "dirty-limit-throttle-time-per-round"); + usleep(100); + g_assert_false(got_stop); + } + + /* + * The dirty limit rate should equals the return value of + * query-vcpu-dirty-limit if dirty limit cap set + */ + g_assert_cmpint(dirtylimit_value, =3D=3D, get_limit_rate(from)); + + /* Now, we have tested if dirty limit works, let it converge */ + migrate_set_parameter_int(from, "downtime-limit", downtime_limit); + migrate_set_parameter_int(from, "max-bandwidth", max_bandwidth); + + /* + * Wait for pre-switchover status to check if migration + * satisfy the convergence condition + */ + wait_for_migration_status(from, "pre-switchover", NULL); + + remaining =3D read_ram_property_int(from, "remaining"); + g_assert_cmpint(remaining, <, + (expected_threshold + expected_threshold / 100)); + + migrate_continue(from, "pre-switchover"); + + qtest_qmp_eventwait(to, "RESUME"); + + wait_for_serial("dest_serial"); + wait_for_migration_complete(from); + + test_migrate_end(from, to, true); +} + static bool kvm_dirty_ring_supported(void) { #if defined(__linux__) && defined(HOST_X86_64) @@ -2604,6 +2759,8 @@ int main(int argc, char **argv) test_precopy_unix_dirty_ring); qtest_add_func("/migration/vcpu_dirty_limit", test_vcpu_dirty_limit); + qtest_add_func("/migration/dirty_limit", + test_migrate_dirty_limit); } =20 ret =3D g_test_run(); --=20 2.17.1