From nobody Mon May 13 20:25:06 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1688647483; cv=none; d=zohomail.com; s=zohoarc; b=U557PvTnXiZVq9tcaSh5//MqzjyG75U7mM5FFjr0rX+JKboxUa6IZ6DlWk5esBLx+3lqoQu9yhh2qfVssltlthfXUBJd+2L9ec2h/2AFnBLqESVXbzek2k4l8bAi8dJttO1VT8hB1ZT34wqhrwkykK0907TOW2SiDaEPvhUQhVs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1688647483; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=AM4AZOXzO48xBAvOX8su4Bp5y3blPa41EcclzlrL8B8=; b=OkRb+rjydrUYlCyKehYCrtCcYZjDf0e3E1c6a048kqJYCXN944mft1bSpyhOU6AGSXlb+zvqJkiIye8f4f3pjgXXy+DPACx63irhnyXmmOZeyBLzxDnd/oTwRYlhaRCd8j/vyKEX9V0hYmaK2c2T94UyInkDfpxJgRYXPa2K55o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 168864748373796.99007035638078; Thu, 6 Jul 2023 05:44:43 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHOKo-00032g-Kw; Thu, 06 Jul 2023 08:43:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHOKm-000325-V4 for qemu-devel@nongnu.org; Thu, 06 Jul 2023 08:43:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHOKk-0004Tp-Pd for qemu-devel@nongnu.org; Thu, 06 Jul 2023 08:43:44 -0400 Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-58-nZb96MwINj2DUKZhk3_-Jw-1; Thu, 06 Jul 2023 08:43:36 -0400 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6364867fa8aso1480526d6.1 for ; Thu, 06 Jul 2023 05:43:36 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id d13-20020a0ce44d000000b006301d3cab9csm832816qvm.27.2023.07.06.05.43.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 05:43:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688647422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AM4AZOXzO48xBAvOX8su4Bp5y3blPa41EcclzlrL8B8=; b=A20bjuuOJlq1QC357daKj6B3ogg0tj8fZdcg1dq7uKCLvCj3CdvS+qSM0JN8+3R2KO8n1L +liR0ktUS2wpKQNRzYSjMlItoY3JUL1xG51GjMP98HN160BmS7fL/+Dmh5DkiLTtDA9Wk2 P30JLxk4jdvhXMef7jSD3j4gkbaanOE= X-MC-Unique: nZb96MwINj2DUKZhk3_-Jw-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688647415; x=1691239415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AM4AZOXzO48xBAvOX8su4Bp5y3blPa41EcclzlrL8B8=; b=NIbDt4/8sWJtKqCUEAvrTslNXZOyNzicxWuQBK1V57/53aKabmi7RWhv1FnjRG6HbN hnTO1ODPd0b12QidoPuyoTFmIoNESEZQu55HwiEWGr3qCwhmkZap9y3b52LpLcawSEQR G2nRomRZRvTkY0fyS1FmhAen+A6NquOW7c/1ZBvhXsQGhxZuPXFyyuvZ/sWuUyXu2ycG C76T7wACn9kuWWZOdrLrHa46QquT+mVy9Lm56iHu+zmDu8GDak5ftsvUXaUDWtEbS3Rz n1hK5XGpiOomcjycf9DCndRruztRYZSflDSp16H3Lb2c/sHqwzhgqhaoAJFpkIMWNfgh UkpA== X-Gm-Message-State: ABy/qLZOBBx4Wo2SCwBqfLz0BsEdhtIJ9DkGthVz2UUQ3i1RiGtpA9S0 hL7maYfWe7eY8tGcGWIVJRrO3F66EOjSwaWSS/jmuD1z74z3QxqnRhgGOET5/LrP5oYEs4sd3pT tTdbhHK/C+mKpSTaFTbtMYYWxHqSIKNW1VYBet2SX+R21qZ43c5SDKemSqlKGAuTFKgy8K5m7 X-Received: by 2002:a05:6214:5298:b0:635:ec47:bfa4 with SMTP id kj24-20020a056214529800b00635ec47bfa4mr1663904qvb.4.1688647415337; Thu, 06 Jul 2023 05:43:35 -0700 (PDT) X-Google-Smtp-Source: APBJJlHO8n7BcT11rZw8SD6QdRrCABjpywWmGT67cKqcqV+5Yuwe1+0frgkjXSbc/NsKFZItYs/xSQ== X-Received: by 2002:a05:6214:5298:b0:635:ec47:bfa4 with SMTP id kj24-20020a056214529800b00635ec47bfa4mr1663871qvb.4.1688647414802; Thu, 06 Jul 2023 05:43:34 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Juan Quintela , Paolo Bonzini , peterx@redhat.com, Avihai Horon , Leonardo Bras Soares Passos , Laurent Vivier , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eric Blake , Markus Armbruster , Thomas Huth Subject: [PATCH v5 1/2] migration: switchover-hold parameter Date: Thu, 6 Jul 2023 08:43:30 -0400 Message-ID: <20230706124331.377939-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706124331.377939-1-peterx@redhat.com> References: <20230706124331.377939-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1688647485354100001 Content-Type: text/plain; charset="utf-8" Add a new migration parameter switchover-hold which can block src qemu migration from switching over to dest from running. One can set this flag to true so src qemu will keep iterating the VM data, not switching over to dest even if it can. It means now live migration works somehow like COLO; we keep syncing data from src to dst without stopping. When the user is ready for the switchover, one can set the parameter from true->false. That'll contain a implicit kick to migration thread to be alive and re-evaluate the switchover decision. This can be used in two cases so far in my mind: (1) One can use this parameter to start pre-heating migration (but not really migrating, so a migrate-cancel will cancel the preheat). When the user wants to really migrate, just clear the flag. It'll in most cases migrate immediately because most pages are already synced. (2) Can also be used as a clean way to do qtest, in many of the precopy tests we have requirement to run after 1 iteration without completing the precopy migration. Before that we have either set bandwidth to ridiculous low value, or tricks on detecting guest memory change over some adhoc guest memory position. Now we can simply set this flag then we know precopy won't complete and will just keep going. Here we leveraged a sem to make sure migration thread won't busy spin on a physical cpu, meanwhile provide a timedwait() of 10ms so it can still try its best to sync with dest QEMU from time to time. Note that the sem is prone to outdated counts but it's benign, please refer to the comment above the semaphore definition for more information. Signed-off-by: Peter Xu --- qapi/migration.json | 25 ++++++++++-- migration/migration.h | 17 +++++++++ migration/migration-hmp-cmds.c | 7 ++++ migration/migration.c | 69 ++++++++++++++++++++++++++++++++-- migration/options.c | 17 +++++++++ 5 files changed, 128 insertions(+), 7 deletions(-) diff --git a/qapi/migration.json b/qapi/migration.json index 47dfef0278..c050081555 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -789,6 +789,15 @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @switchover-hold: Whether we should hold-off precopy switchover from +# src to dest QEMU, even if we can finish migration in the +# downtime specified. By default off, so precopy migration will +# complete as soon as possible. One can set it to explicitly keep +# iterating during precopy migration until set the flag to false +# again to kick off the final switchover. Note, this does not +# affect postcopy switchover, because the user can control that +# using "migrate-start-postcopy" command explicitly. (Since 8.1) +# # Features: # # @unstable: Member @x-checkpoint-delay is experimental. @@ -810,7 +819,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level' ,'multifd-zstd-level', - 'block-bitmap-mapping' ] } + 'block-bitmap-mapping', 'switchover-hold' ] } =20 ## # @MigrateSetParameters: @@ -945,6 +954,10 @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @switchover-hold: Whether we should hold-off precopy switchover from +# src to dest QEMU. For more details, please refer to +# MigrationParameter entry of the same field. (Since 8.1) +# # Features: # # @unstable: Member @x-checkpoint-delay is experimental. @@ -982,7 +995,8 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*switchover-hold': 'bool' } } =20 ## # @migrate-set-parameters: @@ -1137,6 +1151,10 @@ # Nodes are mapped to their block device name if there is one, and # to their node name otherwise. (Since 5.2) # +# @switchover-hold: Whether we should hold-off precopy switchover from +# src to dest QEMU. For more details, please refer to +# MigrationParameter entry of the same field. (Since 8.1) +# # Features: # # @unstable: Member @x-checkpoint-delay is experimental. @@ -1171,7 +1189,8 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', - '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } + '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], + '*switchover-hold': 'bool' } } =20 ## # @query-migrate-parameters: diff --git a/migration/migration.h b/migration/migration.h index a80b22b703..6b31a4b371 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -453,6 +453,23 @@ struct MigrationState { * switchover has been received. */ bool switchover_acked; + + /* + * Only migration thread will wait on it when switchover_hold=3D=3Dtru= e. + * + * Only qmp set param will kick it when switching switchover_hold from + * true->false. + * + * NOTE: outdated sem count here is benign. E.g., when this is posted, + * the 1st migration got cancelled, then start the 2nd migration, or + * when someone sets the flag from true->false->true->false.. because + * any outdated sem count will only let the migration thread to run one + * more loop (timedwait() will eat the outdated count) when reaching + * the completion phase, then in the next loop it'll sleep again. The + * important thing here OTOH is when the migration thread is sleeping + * we can always kick it out of the sleep, which we will always do. + */ + QemuSemaphore switchover_hold_sem; }; =20 void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 9885d7c9f7..c3b3860f00 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -338,6 +338,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QD= ict *qdict) monitor_printf(mon, "%s: '%s'\n", MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ), params->tls_authz); + monitor_printf(mon, "%s: %s\n", + MigrationParameter_str(MIGRATION_PARAMETER_SWITCHOVER_HOLD), + params->switchover_hold ? "on" : "off"); =20 if (params->has_block_bitmap_mapping) { const BitmapMigrationNodeAliasList *bmnal; @@ -616,6 +619,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDi= ct *qdict) p->has_announce_step =3D true; visit_type_size(v, param, &p->announce_step, &err); break; + case MIGRATION_PARAMETER_SWITCHOVER_HOLD: + p->has_switchover_hold =3D true; + visit_type_bool(v, param, &p->switchover_hold, &err); + break; case MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING: error_setg(&err, "The block-bitmap-mapping parameter can only be s= et " "through QMP"); diff --git a/migration/migration.c b/migration/migration.c index 096e8191d1..d75c2bd63c 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2721,6 +2721,67 @@ static bool migration_can_switchover(MigrationState = *s) return s->switchover_acked; } =20 +static bool +migration_should_complete(MigrationState *s, uint64_t pending_size) +{ + /* Need an explicit ACK from dst? */ + if (!migration_can_switchover(s)) { + return false; + } + + /* We still have large pending data to send? */ + if (pending_size && (pending_size >=3D s->threshold_size)) { + return false; + } + + /* The user doesn't want us to switchover yet for precopy */ + if (!migration_in_postcopy() && s->parameters.switchover_hold) { + /* + * Note: when reaching here it probably means we've migrated almost + * everything and ready to switchover. If user asked not to switch + * wait for a short period and respond to kicks immediately. + * + * If we wait too long, there can be a lot of dirty data generated, + * while we could have done something to sync data between src/dst. + * + * If we wait too short, migration thread can eat most/all cpu + * resource looping over switchover_hold. + * + * Make it 10ms which seems to be a good intermediate value. + */ + qemu_sem_timedwait(&s->switchover_hold_sem, 10); + + /* + * Return false here always even if user changed it, because we'd + * like to re-evaluate everything (e.g. pending_size). + */ + return false; + } + + return true; +} + +static bool +migration_should_start_postcopy(MigrationState *s, uint64_t must_precopy) +{ + /* If we're already in postcopy phase, don't bother */ + if (migration_in_postcopy()) { + return false; + } + + /* Need an explicit ACK from dst? */ + if (!migration_can_switchover(s)) { + return false; + } + + /* We still have lots of thing that must be migrated in precopy */ + if (must_precopy > s->threshold_size) { + return false; + } + + return qatomic_read(&s->start_postcopy); +} + /* Migration thread iteration status */ typedef enum { MIG_ITERATE_RESUME, /* Resume current iteration */ @@ -2736,7 +2797,6 @@ static MigIterateState migration_iteration_run(Migrat= ionState *s) { uint64_t must_precopy, can_postcopy; bool in_postcopy =3D s->state =3D=3D MIGRATION_STATUS_POSTCOPY_ACTIVE; - bool can_switchover =3D migration_can_switchover(s); =20 qemu_savevm_state_pending_estimate(&must_precopy, &can_postcopy); uint64_t pending_size =3D must_precopy + can_postcopy; @@ -2749,15 +2809,14 @@ static MigIterateState migration_iteration_run(Migr= ationState *s) trace_migrate_pending_exact(pending_size, must_precopy, can_postco= py); } =20 - if ((!pending_size || pending_size < s->threshold_size) && can_switcho= ver) { + if (migration_should_complete(s, pending_size)) { trace_migration_thread_low_pending(pending_size); migration_completion(s); return MIG_ITERATE_BREAK; } =20 /* Still a significant amount to transfer */ - if (!in_postcopy && must_precopy <=3D s->threshold_size && can_switcho= ver && - qatomic_read(&s->start_postcopy)) { + if (migration_should_start_postcopy(s, must_precopy)) { if (postcopy_start(s)) { error_report("%s: postcopy failed to start", __func__); } @@ -3314,6 +3373,7 @@ static void migration_instance_finalize(Object *obj) qemu_sem_destroy(&ms->rp_state.rp_sem); qemu_sem_destroy(&ms->rp_state.rp_pong_acks); qemu_sem_destroy(&ms->postcopy_qemufile_src_sem); + qemu_sem_destroy(&ms->switchover_hold_sem); error_free(ms->error); } =20 @@ -3336,6 +3396,7 @@ static void migration_instance_init(Object *obj) qemu_sem_init(&ms->rate_limit_sem, 0); qemu_sem_init(&ms->wait_unplug_sem, 0); qemu_sem_init(&ms->postcopy_qemufile_src_sem, 0); + qemu_sem_init(&ms->switchover_hold_sem, 0); qemu_mutex_init(&ms->qemu_file_lock); } =20 diff --git a/migration/options.c b/migration/options.c index 5a9505adf7..aac658fb2d 100644 --- a/migration/options.c +++ b/migration/options.c @@ -163,6 +163,8 @@ Property migration_properties[] =3D { DEFINE_PROP_STRING("tls-creds", MigrationState, parameters.tls_creds), DEFINE_PROP_STRING("tls-hostname", MigrationState, parameters.tls_host= name), DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz), + DEFINE_PROP_BOOL("switchover-hold", MigrationState, + parameters.switchover_hold, false), =20 /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -900,6 +902,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error= **errp) params->announce_rounds =3D s->parameters.announce_rounds; params->has_announce_step =3D true; params->announce_step =3D s->parameters.announce_step; + params->has_switchover_hold =3D true; + params->switchover_hold =3D s->parameters.switchover_hold; =20 if (s->parameters.has_block_bitmap_mapping) { params->has_block_bitmap_mapping =3D true; @@ -940,6 +944,7 @@ void migrate_params_init(MigrationParameters *params) params->has_announce_max =3D true; params->has_announce_rounds =3D true; params->has_announce_step =3D true; + params->has_switchover_hold =3D true; } =20 /* @@ -1194,6 +1199,9 @@ static void migrate_params_test_apply(MigrateSetParam= eters *params, if (params->has_announce_step) { dest->announce_step =3D params->announce_step; } + if (params->has_switchover_hold) { + dest->switchover_hold =3D params->switchover_hold; + } =20 if (params->has_block_bitmap_mapping) { dest->has_block_bitmap_mapping =3D true; @@ -1307,6 +1315,15 @@ static void migrate_params_apply(MigrateSetParameter= s *params, Error **errp) if (params->has_announce_step) { s->parameters.announce_step =3D params->announce_step; } + if (params->has_switchover_hold) { + bool old =3D s->parameters.switchover_hold; + bool new =3D params->switchover_hold; + + s->parameters.switchover_hold =3D params->switchover_hold; + if (old && !new) { + qemu_sem_post(&s->switchover_hold_sem); + } + } =20 if (params->has_block_bitmap_mapping) { qapi_free_BitmapMigrationNodeAliasList( --=20 2.41.0 From nobody Mon May 13 20:25:06 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1688647464; cv=none; d=zohomail.com; s=zohoarc; b=PCxHAU0c5tIthSEX2Oqka8S59tNaTZ5ZjkTG3qdgtVto/rcFzzMLZmTT8ojGFgJyGUOvwxwS6LCEgsY3AD9091483Y5TNYcJhWBC4geoFFfFXDRnozxOqzCD8MlNGqIsMgpkS9fwfiBDng7vW88hbeo/RglT17BwNWYfugwBd3c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1688647464; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=RiGJvmAjJgDhh+5Hyb+zEN0NsPKDhwAESazt+bfQzMs=; b=jcjcrijSExMfj44tcoIzQS+V5etP8jfjPzH2GEl0vD2NugFIPTNTC/wbDbAR2cT/BOa3bWDZOl4W++6IOIy8O7APAfwhuVGFUYHx3L6VvEqlf8OR4eU4HJM0MXEl4gUmW5xHHQcIwqV015UiHOecauYqHEOungy8Cl7kzWsDBF0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 168864746416690.45392596226725; Thu, 6 Jul 2023 05:44:24 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qHOKj-00031A-VV; Thu, 06 Jul 2023 08:43:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHOKi-00030l-M3 for qemu-devel@nongnu.org; Thu, 06 Jul 2023 08:43:40 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qHOKg-0004T7-Rd for qemu-devel@nongnu.org; Thu, 06 Jul 2023 08:43:40 -0400 Received: from mail-ua1-f72.google.com (mail-ua1-f72.google.com [209.85.222.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-290-eq1I6gH5NMClJ-JO0C7dAw-1; Thu, 06 Jul 2023 08:43:36 -0400 Received: by mail-ua1-f72.google.com with SMTP id a1e0cc1a2514c-78f229d2217so21858241.0 for ; Thu, 06 Jul 2023 05:43:36 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id d13-20020a0ce44d000000b006301d3cab9csm832816qvm.27.2023.07.06.05.43.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 05:43:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688647417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RiGJvmAjJgDhh+5Hyb+zEN0NsPKDhwAESazt+bfQzMs=; b=Ep28P1HRE68L/b9/pfP8gc/j2S7fqF2WcOouQTqgiMDIaCw+oCukTqCMwK5cgKx9b4MZQg VszGvBl7mmXbYd68yrIrw14+aI4YbuOVqHLtz2DnHJh6aA9kExkjfQkvuZJM9PQOvdFRqc vpj2jss43t7/qgqtZ0cUQK3E2hvx2dM= X-MC-Unique: eq1I6gH5NMClJ-JO0C7dAw-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688647416; x=1691239416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RiGJvmAjJgDhh+5Hyb+zEN0NsPKDhwAESazt+bfQzMs=; b=JgNoDgS1ESditItcabxQP/jwgstF69zTEaxAoljS48c8dPATiLb9IWaY72pmh1bUc8 BkoVkc/NBXgBTNDRHtRT2rz5PgsxJI7ikq9pReDFXs6UPYfFPjHW4HQou3cZK0cIfSQb ZhIXOUPMQ098VXgJZ8Kd0rr0NuhWDMlV7YwN176hC/sBhs/nbetzuq0F185L/aTKBQr0 Fy+QgZxEUqu3N3mhV5EXVoZ/oVsskIcdSoAvdUBiYmjEfWlR0o2gTJsNS5myISx35VSs Nt9kG9Ao02A+vaxtVl5DXORF3/rDT7Zmt+Aj7nkdTQJKdDCBJQGyzfC0yvPG0lf1fq4j qeCA== X-Gm-Message-State: ABy/qLYFT1x06L9qKQAHT3hwpzOru0mFObVQv5ckLAhgwYOaU7hJ5OUG CZwsy3WzgOOFW5wbvV4szjFBlNV/VQfbxDwaEJM/fpYctcUXNOHlk9mrX/for3oLYFT8/wuA1+r cYWSj54gyp+sO71ehhtyJPi7V/fz4BgsuQ/zVi8RQcjedJqp43/WjcyFOYWUkn5orPB5o85e3 X-Received: by 2002:a05:6122:98c:b0:471:6e5c:dddd with SMTP id g12-20020a056122098c00b004716e5cddddmr943067vkd.0.1688647416246; Thu, 06 Jul 2023 05:43:36 -0700 (PDT) X-Google-Smtp-Source: APBJJlErclCTxpx246s4PPD78RMjctbejivSctqFmzThdLf35YyylyViefr8slTvdAx6GxS18PNxdQ== X-Received: by 2002:a05:6122:98c:b0:471:6e5c:dddd with SMTP id g12-20020a056122098c00b004716e5cddddmr943044vkd.0.1688647415813; Thu, 06 Jul 2023 05:43:35 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Juan Quintela , Paolo Bonzini , peterx@redhat.com, Avihai Horon , Leonardo Bras Soares Passos , Laurent Vivier , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eric Blake , Markus Armbruster , Thomas Huth Subject: [PATCH v5 2/2] qtest/migration: Use switchover-hold to speedup Date: Thu, 6 Jul 2023 08:43:31 -0400 Message-ID: <20230706124331.377939-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706124331.377939-1-peterx@redhat.com> References: <20230706124331.377939-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1688647466262100005 Content-Type: text/plain; charset="utf-8" This solution is heavily based on Daniel's original approach here, but hopefully a cleaner way to impl: https://lore.kernel.org/r/20230601161347.1803440-11-berrange@redhat.com The difference is we use the switchover-hold flag rather than tuning bw+downtime to guide test convergence, comparing to use the magic offset. This can achieve similar goal of previous patch "tests/qtest: massively speed up migration-test" but without magic offset to write or monitoring. With this flag, we can safely always run migration tests with full speed (bw=3D0). However for postcopy tests, when with above bw=3D0, it's easy to happen that right after switching to postcopy there're merely no page left, so the postcopy paths are not well tested. To remedy that, don't wait for a full iteration but switch to postcopy in the 1st iteration, adding a precopy bw limit (200MB/s for now, running 100ms) so it should guarantee enough pages left for postcopy. One pity is that normally postcopy switchover happens after 1-2 rounds of precopy, so qtest doesn't follow that anymore (while it was trying to). However it also means previous postcopy tests never tested the case where there're holes in pages (pages that never got migrated during precopy), now we cover that too which is actually also a valid scenario for postcopy. The initial solution can reduce migration-test time from 8min to 1min40s, this patch can further reduce it from 1m40s to <1m per my local test. While at it, add migrate_set_bandwidth_[pre|post]copy() and use them. Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 39 +++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index b9cc194100..d5584d07a9 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -433,16 +433,23 @@ static void migrate_set_parameter_bool(QTestState *wh= o, const char *parameter, =20 static void migrate_ensure_non_converge(QTestState *who) { - /* Can't converge with 1ms downtime + 3 mbs bandwidth limit */ - migrate_set_parameter_int(who, "max-bandwidth", 3 * 1000 * 1000); - migrate_set_parameter_int(who, "downtime-limit", 1); + /* Hold off switchover for precopy only */ + migrate_set_parameter_bool(who, "switchover-hold", true); } =20 static void migrate_ensure_converge(QTestState *who) { - /* Should converge with 30s downtime + 1 gbs bandwidth limit */ - migrate_set_parameter_int(who, "max-bandwidth", 1 * 1000 * 1000 * 1000= ); - migrate_set_parameter_int(who, "downtime-limit", 30 * 1000); + migrate_set_parameter_bool(who, "switchover-hold", false); +} + +static void migrate_set_bandwidth_precopy(QTestState *who, int bw) +{ + migrate_set_parameter_int(who, "max-bandwidth", bw); +} + +static void migrate_set_bandwidth_postcopy(QTestState *who, int bw) +{ + migrate_set_parameter_int(who, "max-postcopy-bandwidth", bw); } =20 static void migrate_pause(QTestState *who) @@ -736,6 +743,14 @@ static int test_migrate_start(QTestState **from, QTest= State **to, unlink(shmem_path); } =20 + /* + * By default, use full speed for precopy in qtests as that will reduce + * the time of testing. The default bandwidth (128MB/s) may be too slow + * in this case. Specific test can overwrite this value after the + * function returns but before starting migration. + */ + migrate_set_bandwidth_precopy(*from, 0); + return 0; } =20 @@ -1168,9 +1183,13 @@ static int migrate_postcopy_prepare(QTestState **fro= m_ptr, /* Wait for the first serial output from the source */ wait_for_serial("src_serial"); =20 + /* + * Limit precopy to 200MB/s for 0.1 sec, so we guarantee to leave + * enough pages (total-20MB) for remote page fault processes later. + */ + migrate_set_bandwidth_precopy(from, 200 * 1024 * 1024); migrate_qmp(from, uri, "{}"); - - wait_for_migration_pass(from); + usleep(100000); =20 *from_ptr =3D from; *to_ptr =3D to; @@ -1270,7 +1289,7 @@ static void test_postcopy_recovery_common(MigrateComm= on *args) } =20 /* Turn postcopy speed down, 4K/s is slow enough on any machines */ - migrate_set_parameter_int(from, "max-postcopy-bandwidth", 4096); + migrate_set_bandwidth_postcopy(from, 4096); =20 /* Now we start the postcopy */ migrate_postcopy_start(from, to); @@ -1314,7 +1333,7 @@ static void test_postcopy_recovery_common(MigrateComm= on *args) migrate_qmp(from, uri, "{'resume': true}"); =20 /* Restore the postcopy bandwidth to unlimited */ - migrate_set_parameter_int(from, "max-postcopy-bandwidth", 0); + migrate_set_bandwidth_postcopy(from, 0); =20 migrate_postcopy_complete(from, to, args); } --=20 2.41.0