From nobody Sat Feb 7 23:43:22 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1549469359600694.0550020872302; Wed, 6 Feb 2019 08:09:19 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AC5AD13924; Wed, 6 Feb 2019 16:09:16 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5437764438; Wed, 6 Feb 2019 16:09:16 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id F3DE33F602; Wed, 6 Feb 2019 16:09:14 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x16G9DQA020424 for ; Wed, 6 Feb 2019 11:09:13 -0500 Received: by smtp.corp.redhat.com (Postfix) id 9CF8384E4; Wed, 6 Feb 2019 16:09:13 +0000 (UTC) Received: from virval.usersys.redhat.com (unknown [10.43.2.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6DB804142 for ; Wed, 6 Feb 2019 16:09:13 +0000 (UTC) Received: by virval.usersys.redhat.com (Postfix, from userid 500) id 203631002A8; Wed, 6 Feb 2019 17:09:12 +0100 (CET) From: Jiri Denemark To: libvir-list@redhat.com Date: Wed, 6 Feb 2019 17:08:59 +0100 Message-Id: <3559c92d647dbfd27afc67d704a22127bf6cf89e.1549469222.git.jdenemar@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 1/3] Public API for parallel migration X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Wed, 06 Feb 2019 16:09:18 +0000 (UTC) Content-Type: text/plain; charset="utf-8" This patch adds a new VIR_MIGRATE_PARALLEL flag for migration APIs which will ask the hypervisor to use multiple parallel connections for migrating a domain. The number of parallel connections can be set using VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS typed parameter. Signed-off-by: Jiri Denemark --- include/libvirt/libvirt-domain.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-dom= ain.h index d99f9b690b..b79a817ce5 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -831,6 +831,12 @@ typedef enum { */ VIR_MIGRATE_TLS =3D (1 << 16), =20 + /* Send memory pages to the destination host through several network + * connections. See VIR_MIGRATE_PARAM_PARALLEL_* parameters for + * configuring the parallel migration. + */ + VIR_MIGRATE_PARALLEL =3D (1 << 17), + } virDomainMigrateFlags; =20 =20 @@ -1025,6 +1031,14 @@ typedef enum { */ # define VIR_MIGRATE_PARAM_AUTO_CONVERGE_INCREMENT "auto_converge.increme= nt" =20 +/** + * VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS: + * + * virDomainMigrate* params field: number of connections used during paral= lel + * migration. As VIR_TYPED_PARAM_INT. + */ +# define VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS "parallel.connections" + /* Domain migration. */ virDomainPtr virDomainMigrate (virDomainPtr domain, virConnectPtr dconn, unsigned long flags, const char *dname, --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Sat Feb 7 23:43:22 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1549469373539664.6785853447091; Wed, 6 Feb 2019 08:09:33 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 924B4C070142; Wed, 6 Feb 2019 16:09:29 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DCCEA7C811; Wed, 6 Feb 2019 16:09:26 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 523E33F605; Wed, 6 Feb 2019 16:09:26 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x16G9DcZ020433 for ; Wed, 6 Feb 2019 11:09:13 -0500 Received: by smtp.corp.redhat.com (Postfix) id C599264438; Wed, 6 Feb 2019 16:09:13 +0000 (UTC) Received: from virval.usersys.redhat.com (unknown [10.43.2.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 70FB4649BF for ; Wed, 6 Feb 2019 16:09:13 +0000 (UTC) Received: by virval.usersys.redhat.com (Postfix, from userid 500) id 260CA1011C8; Wed, 6 Feb 2019 17:09:12 +0100 (CET) From: Jiri Denemark To: libvir-list@redhat.com Date: Wed, 6 Feb 2019 17:09:00 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 2/3] qemu: Add support for parallel migration X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Wed, 06 Feb 2019 16:09:31 +0000 (UTC) Content-Type: text/plain; charset="utf-8" The VIR_MIGRATE_PARALLEL flag is implemented using QEMU's multifd migration capability and the corresponding multifd-channels migration parameter. Signed-off-by: Jiri Denemark --- Notes: QEMU still uses the x- prefix for multifd capability and multifd-channe= ls, but Juan already sent a series of patches to make multifd migration ful= ly supported by dropping the experimental prefix. src/qemu/qemu_migration.c | 9 ++++++--- src/qemu/qemu_migration.h | 5 ++++- src/qemu/qemu_migration_params.c | 18 ++++++++++++++++++ src/qemu/qemu_migration_params.h | 2 ++ 4 files changed, 30 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 1433b2c2f3..d66313d35e 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3823,10 +3823,13 @@ qemuMigrationSrcPerformNative(virQEMUDriverPtr driv= er, } } =20 - if (STRNEQ(uribits->scheme, "rdma")) - spec.destType =3D MIGRATION_DEST_CONNECT_HOST; - else + /* RDMA and multi-fd migration requires QEMU to connect to the destina= tion + * itself. + */ + if (STREQ(uribits->scheme, "rdma") || (flags & VIR_MIGRATE_PARALLEL)) spec.destType =3D MIGRATION_DEST_HOST; + else + spec.destType =3D MIGRATION_DEST_CONNECT_HOST; spec.dest.host.protocol =3D uribits->scheme; spec.dest.host.name =3D uribits->server; spec.dest.host.port =3D uribits->port; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index ca73d3e467..fea5000a4d 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -57,7 +57,9 @@ VIR_MIGRATE_AUTO_CONVERGE | \ VIR_MIGRATE_RDMA_PIN_ALL | \ VIR_MIGRATE_POSTCOPY | \ - VIR_MIGRATE_TLS) + VIR_MIGRATE_TLS | \ + VIR_MIGRATE_PARALLEL | \ + 0) =20 /* All supported migration parameters and their types. */ # define QEMU_MIGRATION_PARAMETERS \ @@ -80,6 +82,7 @@ VIR_MIGRATE_PARAM_AUTO_CONVERGE_INITIAL, VIR_TYPED_PARAM_INT, \ VIR_MIGRATE_PARAM_AUTO_CONVERGE_INCREMENT, VIR_TYPED_PARAM_INT, \ VIR_MIGRATE_PARAM_BANDWIDTH_POSTCOPY, VIR_TYPED_PARAM_ULLONG, \ + VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS, VIR_TYPED_PARAM_INT, \ NULL =20 =20 diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_par= ams.c index 67070b9d08..7908aa95cd 100644 --- a/src/qemu/qemu_migration_params.c +++ b/src/qemu/qemu_migration_params.c @@ -86,6 +86,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability, QEMU_MIGRATION_CAP= _LAST, "compress", "pause-before-switchover", "late-block-activate", + "multifd", ); =20 =20 @@ -103,6 +104,7 @@ VIR_ENUM_IMPL(qemuMigrationParam, QEMU_MIGRATION_PARAM_= LAST, "block-incremental", "xbzrle-cache-size", "max-postcopy-bandwidth", + "multifd-channels", ); =20 typedef struct _qemuMigrationParamsAlwaysOnItem qemuMigrationParamsAlwaysO= nItem; @@ -151,6 +153,10 @@ static const qemuMigrationParamsFlagMapItem qemuMigrat= ionParamsFlagMap[] =3D { {VIR_MIGRATE_POSTCOPY, QEMU_MIGRATION_CAP_POSTCOPY, QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, + + {VIR_MIGRATE_PARALLEL, + QEMU_MIGRATION_CAP_MULTIFD, + QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, }; =20 /* Translation from VIR_MIGRATE_PARAM_* typed parameters to @@ -184,6 +190,10 @@ static const qemuMigrationParamsTPMapItem qemuMigratio= nParamsTPMap[] =3D { .unit =3D 1024 * 1024, /* MB/s */ .param =3D QEMU_MIGRATION_PARAM_MAX_POSTCOPY_BANDWIDTH, .party =3D QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, + + {.typedParam =3D VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS, + .param =3D QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS, + .party =3D QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, }; =20 static const qemuMigrationParamType qemuMigrationParamTypes[] =3D { @@ -199,6 +209,7 @@ static const qemuMigrationParamType qemuMigrationParamT= ypes[] =3D { [QEMU_MIGRATION_PARAM_BLOCK_INCREMENTAL] =3D QEMU_MIGRATION_PARAM_TYPE= _BOOL, [QEMU_MIGRATION_PARAM_XBZRLE_CACHE_SIZE] =3D QEMU_MIGRATION_PARAM_TYPE= _ULL, [QEMU_MIGRATION_PARAM_MAX_POSTCOPY_BANDWIDTH] =3D QEMU_MIGRATION_PARAM= _TYPE_ULL, + [QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS] =3D QEMU_MIGRATION_PARAM_TYPE_= INT, }; verify(ARRAY_CARDINALITY(qemuMigrationParamTypes) =3D=3D QEMU_MIGRATION_PA= RAM_LAST); =20 @@ -535,6 +546,13 @@ qemuMigrationParamsFromFlags(virTypedParameterPtr para= ms, goto error; } =20 + if (migParams->params[QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS].set && + !(flags & VIR_MIGRATE_PARALLEL)) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Turn parallel migration on to tune it")); + goto error; + } + if (qemuMigrationParamsSetCompression(params, nparams, flags, migParam= s) < 0) goto error; =20 diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_par= ams.h index 2460684a00..959fdd5854 100644 --- a/src/qemu/qemu_migration_params.h +++ b/src/qemu/qemu_migration_params.h @@ -38,6 +38,7 @@ typedef enum { QEMU_MIGRATION_CAP_COMPRESS, QEMU_MIGRATION_CAP_PAUSE_BEFORE_SWITCHOVER, QEMU_MIGRATION_CAP_LATE_BLOCK_ACTIVATE, + QEMU_MIGRATION_CAP_MULTIFD, =20 QEMU_MIGRATION_CAP_LAST } qemuMigrationCapability; @@ -56,6 +57,7 @@ typedef enum { QEMU_MIGRATION_PARAM_BLOCK_INCREMENTAL, QEMU_MIGRATION_PARAM_XBZRLE_CACHE_SIZE, QEMU_MIGRATION_PARAM_MAX_POSTCOPY_BANDWIDTH, + QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS, =20 QEMU_MIGRATION_PARAM_LAST } qemuMigrationParam; --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Sat Feb 7 23:43:22 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1549469380116743.6440063343473; Wed, 6 Feb 2019 08:09:40 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A019B3697F; Wed, 6 Feb 2019 16:09:35 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5A8A0649D8; Wed, 6 Feb 2019 16:09:32 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id F08F73F605; Wed, 6 Feb 2019 16:09:31 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x16G9FuB020461 for ; Wed, 6 Feb 2019 11:09:15 -0500 Received: by smtp.corp.redhat.com (Postfix) id C735B1001F54; Wed, 6 Feb 2019 16:09:15 +0000 (UTC) Received: from virval.usersys.redhat.com (unknown [10.43.2.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 98A14100194A for ; Wed, 6 Feb 2019 16:09:13 +0000 (UTC) Received: by virval.usersys.redhat.com (Postfix, from userid 500) id 2B3FE1011C9; Wed, 6 Feb 2019 17:09:12 +0100 (CET) From: Jiri Denemark To: libvir-list@redhat.com Date: Wed, 6 Feb 2019 17:09:01 +0100 Message-Id: <0346bdb869ae7b2d3492231c605e712ea7b85206.1549469222.git.jdenemar@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 3/3] virsh: Add options for parallel migration X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 06 Feb 2019 16:09:38 +0000 (UTC) Content-Type: text/plain; charset="utf-8" Signed-off-by: Jiri Denemark --- tools/virsh-domain.c | 19 +++++++++++++++++++ tools/virsh.pod | 7 +++++++ 2 files changed, 26 insertions(+) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index db0d5d4dcc..94cade5757 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -10561,6 +10561,14 @@ static const vshCmdOptDef opts_migrate[] =3D { .type =3D VSH_OT_INT, .help =3D N_("post-copy migration bandwidth limit in MiB/s") }, + {.name =3D "parallel", + .type =3D VSH_OT_BOOL, + .help =3D N_("enable parallel migration") + }, + {.name =3D "parallel-connections", + .type =3D VSH_OT_INT, + .help =3D N_("number of connections for parallel migration") + }, {.name =3D NULL} }; =20 @@ -10766,6 +10774,14 @@ doMigrate(void *opaque) goto save_error; } =20 + if (vshCommandOptInt(ctl, cmd, "parallel-connections", &intOpt) < 0) + goto out; + if (intOpt && + virTypedParamsAddInt(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS, + intOpt) < 0) + goto save_error; + if (vshCommandOptBool(cmd, "live")) flags |=3D VIR_MIGRATE_LIVE; if (vshCommandOptBool(cmd, "p2p")) @@ -10814,6 +10830,9 @@ doMigrate(void *opaque) if (vshCommandOptBool(cmd, "tls")) flags |=3D VIR_MIGRATE_TLS; =20 + if (vshCommandOptBool(cmd, "parallel")) + flags |=3D VIR_MIGRATE_PARALLEL; + if (flags & VIR_MIGRATE_PEER2PEER || vshCommandOptBool(cmd, "direct"))= { if (virDomainMigrateToURI3(dom, desturi, params, nparams, flags) = =3D=3D 0) ret =3D '0'; diff --git a/tools/virsh.pod b/tools/virsh.pod index 67edb57b14..7604c7536e 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -1904,6 +1904,7 @@ I I [I] [I]= [I] [I] [I<--auto-converge>] [I] [I] [I<--persistent-xml> B] [I<--tls>] [I<--postcopy-bandwidth> B] +[I<--parallel> [I B]] =20 Migrate domain to another host. Add I<--live> for live migration; <--p2p> for peer-2-peer migration; I<--direct> for direct migration; or I<--tunnel= led> @@ -1997,6 +1998,12 @@ Providing I<--tls> causes the migration to use the h= ost configured TLS setup the migration of the domain. Usage requires proper TLS setup for both sour= ce and target. =20 +I<--parallel> option will cause migration data to be sent over multiple +parallel connections. The number of such connections can be set using +I. Parallel connections may help with saturating the +network link between the source and the target and thus speeding up the +migration. + Running migration can be canceled by interrupting virsh (usually using C) or by B command sent from another virsh instance. =20 --=20 2.20.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list