From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749359924598.4912428647899; Wed, 11 Oct 2017 12:15:59 -0700 (PDT) Received: from localhost ([::1]:42322 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MTu-0006QH-Ap for importer@patchew.org; Wed, 11 Oct 2017 15:15:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55986) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MRy-0005Fq-GB for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MRt-0004ox-IK for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:50 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36798) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MRt-0004oX-Bq for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:45 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 677485F7A6; Wed, 11 Oct 2017 19:13:44 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id DBCBE6C407; Wed, 11 Oct 2017 19:13:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 677485F7A6 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:11 +0100 Message-Id: <20171011191317.24157-2-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 11 Oct 2017 19:13:44 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/7] migration: Add 'pause-before-device' capability X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" When 'pause-before-device' is enabled, the outgoing migration will pause after completing any outstanding IOs but before serialising the device state. At this point the management layer gets the chance to clean up any device jobs or other device users before the migration completes. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 9 +++++++++ migration/migration.h | 1 + qapi/migration.json | 5 ++++- 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index 98429dc843..df8f8c9f14 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1521,6 +1521,15 @@ bool migrate_use_multifd(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_X_MULTIFD]; } =20 +bool migrate_pause_before_device(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_PAUSE_BEFORE_DEVIC= E]; +} + int migrate_multifd_channels(void) { MigrationState *s; diff --git a/migration/migration.h b/migration/migration.h index b83cceadc4..37feea5453 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -177,6 +177,7 @@ bool migrate_zero_blocks(void); =20 bool migrate_auto_converge(void); bool migrate_use_multifd(void); +bool migrate_pause_before_device(void); int migrate_multifd_channels(void); int migrate_multifd_page_count(void); =20 diff --git a/qapi/migration.json b/qapi/migration.json index f8b365e3f5..420b56f194 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -341,6 +341,9 @@ # @return-path: If enabled, migration will use the return path even # for precopy. (since 2.10) # +# @pause-before-device: Pause outgoing migration before serialising device +# state but after finalising any IO. (since 2.11) +# # @x-multifd: Use more than one fd for migration (since 2.11) # # Since: 1.2 @@ -348,7 +351,7 @@ { 'enum': 'MigrationCapability', 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram', - 'block', 'return-path', 'x-multifd' ] } + 'block', 'return-path', 'pause-before-device', 'x-multifd' ] } =20 ## # @MigrationCapabilityStatus: --=20 2.13.6 From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749365005491.26392437670336; Wed, 11 Oct 2017 12:16:05 -0700 (PDT) Received: from localhost ([::1]:42323 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MTw-0006QK-Cm for importer@patchew.org; Wed, 11 Oct 2017 15:15:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55997) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MRz-0005GC-3x for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MRy-0004ql-6n for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:51 -0400 Received: from mx1.redhat.com ([209.132.183.28]:56044) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MRy-0004qP-0C for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:50 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1D27B3E2A6; Wed, 11 Oct 2017 19:13:49 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id AFA1117C47; Wed, 11 Oct 2017 19:13:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 1D27B3E2A6 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:12 +0100 Message-Id: <20171011191317.24157-3-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 11 Oct 2017 19:13:49 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/7] migration: Add 'pause-before-device' and 'device' statuses X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add two statuses for use when the 'pause-before-device' capability is enabled. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 6 ++++++ qapi/migration.json | 8 +++++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index df8f8c9f14..e1a87c3d23 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -526,6 +526,8 @@ static bool migration_is_setup_or_active(int state) case MIGRATION_STATUS_ACTIVE: case MIGRATION_STATUS_POSTCOPY_ACTIVE: case MIGRATION_STATUS_SETUP: + case MIGRATION_STATUS_PAUSE_BEFORE_DEVICE: + case MIGRATION_STATUS_DEVICE: return true; =20 default: @@ -600,6 +602,8 @@ MigrationInfo *qmp_query_migrate(Error **errp) case MIGRATION_STATUS_ACTIVE: case MIGRATION_STATUS_CANCELLING: case MIGRATION_STATUS_POSTCOPY_ACTIVE: + case MIGRATION_STATUS_PAUSE_BEFORE_DEVICE: + case MIGRATION_STATUS_DEVICE: /* TODO add some postcopy stats */ info->has_status =3D true; info->has_total_time =3D true; @@ -1183,6 +1187,8 @@ bool migration_is_idle(void) case MIGRATION_STATUS_ACTIVE: case MIGRATION_STATUS_POSTCOPY_ACTIVE: case MIGRATION_STATUS_COLO: + case MIGRATION_STATUS_PAUSE_BEFORE_DEVICE: + case MIGRATION_STATUS_DEVICE: return false; case MIGRATION_STATUS__MAX: g_assert_not_reached(); diff --git a/qapi/migration.json b/qapi/migration.json index 420b56f194..00f2f6cc09 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -96,12 +96,18 @@ # @colo: VM is in the process of fault tolerance, VM can not get into this # state unless colo capability is enabled for migration. (since 2.8) # +# @pause-before-device: Paused before device serialisation. (since 2.11) +# +# @device: During device serialisation when pause-before-device is enabled +# (since 2.11) +# # Since: 2.3 # ## { 'enum': 'MigrationStatus', 'data': [ 'none', 'setup', 'cancelling', 'cancelled', - 'active', 'postcopy-active', 'completed', 'failed', 'colo' ] } + 'active', 'postcopy-active', 'completed', 'failed', 'colo', + 'pause-before-device', 'device' ] } =20 ## # @MigrationInfo: --=20 2.13.6 From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749488088980.679028267061; Wed, 11 Oct 2017 12:18:08 -0700 (PDT) Received: from localhost ([::1]:42328 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MW6-0008L5-AX for importer@patchew.org; Wed, 11 Oct 2017 15:18:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56018) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MS1-0005Hu-Mu for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MS0-0004s2-J9 for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41460) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MS0-0004rg-9n for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:52 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6739A3A261; Wed, 11 Oct 2017 19:13:51 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 65E731898B; Wed, 11 Oct 2017 19:13:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 6739A3A261 Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:13 +0100 Message-Id: <20171011191317.24157-4-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Wed, 11 Oct 2017 19:13:51 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 3/7] migration: Wait for semaphore before completing migration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Wait for a semaphore before completing the migration, if the previously added capability was enabled. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ migration/migration.h | 3 +++ 2 files changed, 50 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index e1a87c3d23..b411a7bb63 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1967,6 +1967,46 @@ fail: } =20 /** + * migration_maybe_pause: Pause if required to by migrate_pause_before_dev= ice + * called with the iothread locked + * Returns: 0 on success + */ +static int migration_maybe_pause(MigrationState *s, int *current_active_st= ate) +{ + int ret; + if (!migrate_pause_before_device()) { + return 0; + } + ret =3D bdrv_inactivate_all(); + if (ret) { + error_report("%s: bdrv_inactivate_all() failed (%d)", + __func__, ret); + return ret; + } + + s->block_inactive =3D true; + + /* Since leaving this state is not atomic with posting the semaphore + * it's possible that someone could have issued multiple migrate_conti= nue + * and the semaphore is incorrectly positive at this point; + * the docs say it's undefined to reinit a semaphore that's already + * init'd, so use timedwait to eat up any existing posts. + */ + while (qemu_sem_timedwait(&s->pause_sem, 1) =3D=3D 0); + + qemu_mutex_unlock_iothread(); + migrate_set_state(&s->state, *current_active_state, + MIGRATION_STATUS_PAUSE_BEFORE_DEVICE); + qemu_sem_wait(&s->pause_sem); + migrate_set_state(&s->state, MIGRATION_STATUS_PAUSE_BEFORE_DEVICE, + MIGRATION_STATUS_DEVICE); + *current_active_state =3D MIGRATION_STATUS_DEVICE; + qemu_mutex_lock_iothread(); + + return s->state =3D=3D MIGRATION_STATUS_DEVICE ? 0 : -EINVAL; +} + +/** * migration_completion: Used by migration_thread when there's not much le= ft. * The caller 'breaks' the loop when this returns. * @@ -1992,6 +2032,11 @@ static void migration_completion(MigrationState *s, = int current_active_state, bool inactivate =3D !migrate_colo_enabled(); ret =3D vm_stop_force_state(RUN_STATE_FINISH_MIGRATE); if (ret >=3D 0) { + ret =3D migration_maybe_pause(s, ¤t_active_state); + /* If this worked it will already have inactivated */ + inactivate &=3D !migrate_pause_before_device(); + } + if (ret >=3D 0) { qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX); ret =3D qemu_savevm_state_complete_precopy(s->to_dst_file,= false, inactivate); @@ -2372,6 +2417,7 @@ static void migration_instance_finalize(Object *obj) =20 g_free(params->tls_hostname); g_free(params->tls_creds); + qemu_sem_destroy(&ms->pause_sem); } =20 static void migration_instance_init(Object *obj) @@ -2382,6 +2428,7 @@ static void migration_instance_init(Object *obj) ms->state =3D MIGRATION_STATUS_NONE; ms->xbzrle_cache_size =3D DEFAULT_MIGRATE_CACHE_SIZE; ms->mbps =3D -1; + qemu_sem_init(&ms->pause_sem, 0); =20 params->tls_hostname =3D g_strdup(""); params->tls_creds =3D g_strdup(""); diff --git a/migration/migration.h b/migration/migration.h index 37feea5453..447e8b3f79 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -121,6 +121,9 @@ struct MigrationState /* Flag set once the migration thread called bdrv_inactivate_all */ bool block_inactive; =20 + /* Migration is paused due to pause-before-device */ + QemuSemaphore pause_sem; + /* The semaphore is used to notify COLO thread that failover is finish= ed */ QemuSemaphore colo_exit_sem; =20 --=20 2.13.6 From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749597279630.7293706502098; Wed, 11 Oct 2017 12:19:57 -0700 (PDT) Received: from localhost ([::1]:42338 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MXq-000126-Ii for importer@patchew.org; Wed, 11 Oct 2017 15:19:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56029) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MS3-0005Je-Iw for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MS2-0004tQ-QO for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:47824) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MS2-0004sx-KK for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:13:54 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B4FD280E7A; Wed, 11 Oct 2017 19:13:53 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B24001898B; Wed, 11 Oct 2017 19:13:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com B4FD280E7A Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:14 +0100 Message-Id: <20171011191317.24157-5-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Wed, 11 Oct 2017 19:13:53 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 4/7] migration: migrate-continue X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" A new qmp command allows the caller to continue from a given paused state. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 11 +++++++++++ qapi/migration.json | 17 +++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index b411a7bb63..be03e8ff0b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1368,6 +1368,17 @@ void qmp_migrate_cancel(Error **errp) migrate_fd_cancel(migrate_get_current()); } =20 +void qmp_migrate_continue(MigrationStatus state, Error **errp) +{ + MigrationState *s =3D migrate_get_current(); + if (s->state !=3D state) { + error_setg(errp, "Migration not in expected state: %s", + MigrationStatus_str(s->state)); + return; + } + qemu_sem_post(&s->pause_sem); +} + void qmp_migrate_set_cache_size(int64_t value, Error **errp) { MigrationState *s =3D migrate_get_current(); diff --git a/qapi/migration.json b/qapi/migration.json index 00f2f6cc09..a95eb19528 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -877,6 +877,23 @@ { 'command': 'migrate_cancel' } =20 ## +# @migrate-continue: +# +# Continue migration when it's in a paused state. +# +# @state: The state the migration is currently expected to be in +# +# Returns: nothing on success +# Since: 2.11 +# Example: +# +# -> { "execute": "migrate-continue" , "arguments": +# { "state": "pause-before-device" } } +# <- { "return": {} } +## +{ 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} } + +## # @migrate_set_downtime: # # Set maximum tolerated downtime for migration. --=20 2.13.6 From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749383546201.82196532849946; Wed, 11 Oct 2017 12:16:23 -0700 (PDT) Received: from localhost ([::1]:42324 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MUL-0006kh-Oj for importer@patchew.org; Wed, 11 Oct 2017 15:16:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56092) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MSI-0005Vd-Oi for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MSD-0004yP-Rq for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:10 -0400 Received: from mx1.redhat.com ([209.132.183.28]:58746) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MSD-0004xz-LV for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:05 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BDE9E7D0FF; Wed, 11 Oct 2017 19:14:04 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 099FC1898B; Wed, 11 Oct 2017 19:13:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com BDE9E7D0FF Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:15 +0100 Message-Id: <20171011191317.24157-6-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 11 Oct 2017 19:14:04 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 5/7] migrate: HMP migate_continue X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" HMP equivalent to the just added migrate-continue Unpause a migrate paused at a given state. Signed-off-by: Dr. David Alan Gilbert --- hmp-commands.hx | 12 ++++++++++++ hmp.c | 13 +++++++++++++ hmp.h | 1 + 3 files changed, 26 insertions(+) diff --git a/hmp-commands.hx b/hmp-commands.hx index 1941e19932..4afd57cf5f 100644 --- a/hmp-commands.hx +++ b/hmp-commands.hx @@ -959,7 +959,19 @@ STEXI @item migrate_cancel @findex migrate_cancel Cancel the current VM migration. +ETEXI =20 + { + .name =3D "migrate_continue", + .args_type =3D "state:s", + .params =3D "state", + .help =3D "Continue migration from the given paused state", + .cmd =3D hmp_migrate_continue, + }, +STEXI +@item migrate_continue @var{state} +@findex migrate_continue +Continue migration from the paused state @var{state} ETEXI =20 { diff --git a/hmp.c b/hmp.c index 739d330f4e..5fd22a6ea8 100644 --- a/hmp.c +++ b/hmp.c @@ -1490,6 +1490,19 @@ void hmp_migrate_cancel(Monitor *mon, const QDict *q= dict) qmp_migrate_cancel(NULL); } =20 +void hmp_migrate_continue(Monitor *mon, const QDict *qdict) +{ + Error *err =3D NULL; + const char *state =3D qdict_get_str(qdict, "state"); + int val =3D qapi_enum_parse(&MigrationStatus_lookup, state, -1, &err); + + if (val >=3D 0) { + qmp_migrate_continue(val, &err); + } + + hmp_handle_error(mon, &err); +} + void hmp_migrate_incoming(Monitor *mon, const QDict *qdict) { Error *err =3D NULL; diff --git a/hmp.h b/hmp.h index 3605003e4c..a6f56b1f29 100644 --- a/hmp.h +++ b/hmp.h @@ -68,6 +68,7 @@ void hmp_savevm(Monitor *mon, const QDict *qdict); void hmp_delvm(Monitor *mon, const QDict *qdict); void hmp_info_snapshots(Monitor *mon, const QDict *qdict); void hmp_migrate_cancel(Monitor *mon, const QDict *qdict); +void hmp_migrate_continue(Monitor *mon, const QDict *qdict); void hmp_migrate_incoming(Monitor *mon, const QDict *qdict); void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict); void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict); --=20 2.13.6 From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749472184189.4946809932111; Wed, 11 Oct 2017 12:17:52 -0700 (PDT) Received: from localhost ([::1]:42327 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MVp-0008Bj-F0 for importer@patchew.org; Wed, 11 Oct 2017 15:17:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56160) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MST-0005br-3B for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MSN-000542-W0 for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:21 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51072) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MSN-00053d-Pn for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:15 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E38C21293; Wed, 11 Oct 2017 19:14:14 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 105336C407; Wed, 11 Oct 2017 19:14:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com E38C21293 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:16 +0100 Message-Id: <20171011191317.24157-7-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 11 Oct 2017 19:14:15 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 6/7] migration: allow cancel to unpause X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" If a migration_cancel is issued during the new paused state, kick the pause_sem to get to unpause so it can cancel. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index be03e8ff0b..fa42918270 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1108,6 +1108,10 @@ static void migrate_fd_cancel(MigrationState *s) if (!migration_is_setup_or_active(old_state)) { break; } + /* If the migration is paused, kick it out of the pause */ + if (old_state =3D=3D MIGRATION_STATUS_PAUSE_BEFORE_DEVICE) { + qemu_sem_post(&s->pause_sem); + } migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLIN= G); } while (s->state !=3D MIGRATION_STATUS_CANCELLING); =20 --=20 2.13.6 From nobody Thu Mar 28 18:26:00 2024 Delivered-To: importer@patchew.org Received-SPF: temperror (zoho.com: Error in retrieving data from DNS) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1507749692617906.14870465916; Wed, 11 Oct 2017 12:21:32 -0700 (PDT) Received: from localhost ([::1]:42349 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MZ6-0001qy-74 for importer@patchew.org; Wed, 11 Oct 2017 15:21:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56170) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e2MSU-0005cj-3t for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e2MST-00056c-2b for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:22 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60394) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1e2MSS-00056E-QB for qemu-devel@nongnu.org; Wed, 11 Oct 2017 15:14:21 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id EDED37E427; Wed, 11 Oct 2017 19:14:19 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-166.ams2.redhat.com [10.36.117.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 38DAF6C407; Wed, 11 Oct 2017 19:14:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com EDED37E427 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, kwolf@redhat.com, jdenemar@redhat.com, wangjie88@huawei.com, quintela@redhat.com, peterx@redhat.com, mreitz@redhat.com Date: Wed, 11 Oct 2017 20:13:17 +0100 Message-Id: <20171011191317.24157-8-dgilbert@redhat.com> In-Reply-To: <20171011191317.24157-1-dgilbert@redhat.com> References: <20171011191317.24157-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 11 Oct 2017 19:14:20 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 7/7] migration: pause-before-device for postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fuweiwei2@huawei.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_6 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add pause-before-device support for postcopy. After starting postcopy it will transition active->pause_before_device->postcopy_active Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index fa42918270..bdb6a30995 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -104,6 +104,9 @@ enum mig_rp_message_type { static MigrationState *current_migration; =20 static bool migration_object_check(MigrationState *ms, Error **errp); +static int migration_maybe_pause(MigrationState *s, + int *current_active_state, + int new_state); =20 void migration_object_init(void) { @@ -1829,8 +1832,11 @@ static int postcopy_start(MigrationState *ms, bool *= old_vm_running) QEMUFile *fb; int64_t time_at_stop =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); bool restart_block =3D false; - migrate_set_state(&ms->state, MIGRATION_STATUS_ACTIVE, - MIGRATION_STATUS_POSTCOPY_ACTIVE); + int cur_state =3D MIGRATION_STATUS_ACTIVE; + if (!migrate_pause_before_device()) { + migrate_set_state(&ms->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_POSTCOPY_ACTIVE); + } =20 trace_postcopy_start(); qemu_mutex_lock_iothread(); @@ -1850,6 +1856,12 @@ static int postcopy_start(MigrationState *ms, bool *= old_vm_running) } restart_block =3D true; =20 + ret =3D migration_maybe_pause(ms, &cur_state, + MIGRATION_STATUS_POSTCOPY_ACTIVE); + if (ret < 0) { + goto fail; + } + /* * Cause any non-postcopiable, but iterative devices to * send out their final data. @@ -1986,7 +1998,9 @@ fail: * called with the iothread locked * Returns: 0 on success */ -static int migration_maybe_pause(MigrationState *s, int *current_active_st= ate) +static int migration_maybe_pause(MigrationState *s, + int *current_active_state, + int new_state) { int ret; if (!migrate_pause_before_device()) { @@ -2014,11 +2028,11 @@ static int migration_maybe_pause(MigrationState *s,= int *current_active_state) MIGRATION_STATUS_PAUSE_BEFORE_DEVICE); qemu_sem_wait(&s->pause_sem); migrate_set_state(&s->state, MIGRATION_STATUS_PAUSE_BEFORE_DEVICE, - MIGRATION_STATUS_DEVICE); - *current_active_state =3D MIGRATION_STATUS_DEVICE; + new_state); + *current_active_state =3D new_state; qemu_mutex_lock_iothread(); =20 - return s->state =3D=3D MIGRATION_STATUS_DEVICE ? 0 : -EINVAL; + return s->state =3D=3D new_state ? 0 : -EINVAL; } =20 /** @@ -2047,7 +2061,8 @@ static void migration_completion(MigrationState *s, i= nt current_active_state, bool inactivate =3D !migrate_colo_enabled(); ret =3D vm_stop_force_state(RUN_STATE_FINISH_MIGRATE); if (ret >=3D 0) { - ret =3D migration_maybe_pause(s, ¤t_active_state); + ret =3D migration_maybe_pause(s, ¤t_active_state, + MIGRATION_STATUS_DEVICE); /* If this worked it will already have inactivated */ inactivate &=3D !migrate_pause_before_device(); } --=20 2.13.6