From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409150385729.67057400632; Mon, 13 Mar 2017 05:45:50 -0700 (PDT) Received: from localhost ([::1]:51884 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPM8-0004Gl-55 for importer@patchew.org; Mon, 13 Mar 2017 08:45:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56366) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPL7-0004F3-Hb for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPL5-0005mk-VR for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:41 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33464) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPL5-0005mL-Mg for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:39 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CF3C861B86 for ; Mon, 13 Mar 2017 12:44:39 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPr012445; Mon, 13 Mar 2017 08:44:38 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:19 +0100 Message-Id: <20170313124434.1043-2-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 13 Mar 2017 12:44:39 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 01/16] qio: create new qio_channel_write_all X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The function waits until it is able to write the full iov. Signed-off-by: Juan Quintela --- include/io/channel.h | 23 +++++++++++++++++++++++ io/channel.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/qemu-file-channel.c | 29 +---------------------------- 3 files changed, 63 insertions(+), 28 deletions(-) diff --git a/include/io/channel.h b/include/io/channel.h index 5d48906..f786c4f 100644 --- a/include/io/channel.h +++ b/include/io/channel.h @@ -269,6 +269,29 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc, Error **errp); =20 /** + * qio_channel_writev_all: + * @ioc: the channel object + * @iov: the array of memory regions to write data from + * @niov: the length of the @iov array + * @errp: pointer to a NULL-initialized error object + * + * Write data to the IO channel, reading it from the + * memory regions referenced by @iov. Each element + * in the @iov will be fully sent, before the next + * one is used. The @niov parameter specifies the + * total number of elements in @iov. + * + * It is required for all @iov data to be fully + * sent. + * + * Returns: the number of bytes sent, or -1 on error, + */ +ssize_t qio_channel_writev_all(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + Error **erp); + +/** * qio_channel_readv: * @ioc: the channel object * @iov: the array of memory regions to read data into diff --git a/io/channel.c b/io/channel.c index cdf7454..c5a1bd5 100644 --- a/io/channel.c +++ b/io/channel.c @@ -22,6 +22,7 @@ #include "io/channel.h" #include "qapi/error.h" #include "qemu/main-loop.h" +#include "qemu/iov.h" =20 bool qio_channel_has_feature(QIOChannel *ioc, QIOChannelFeature feature) @@ -85,6 +86,44 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc, } =20 =20 + +ssize_t qio_channel_writev_all(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + Error **errp) +{ + ssize_t done =3D 0; + struct iovec *local_iov =3D g_new(struct iovec, niov); + struct iovec *local_iov_head =3D local_iov; + unsigned int nlocal_iov =3D niov; + + nlocal_iov =3D iov_copy(local_iov, nlocal_iov, + iov, niov, + 0, iov_size(iov, niov)); + + while (nlocal_iov > 0) { + ssize_t len; + len =3D qio_channel_writev(ioc, local_iov, nlocal_iov, errp); + if (len =3D=3D QIO_CHANNEL_ERR_BLOCK) { + qio_channel_wait(ioc, G_IO_OUT); + continue; + } + if (len < 0) { + error_setg_errno(errp, EIO, + "Channel was not able to write full iov"); + done =3D -1; + goto cleanup; + } + + iov_discard_front(&local_iov, &nlocal_iov, len); + done +=3D len; + } + + cleanup: + g_free(local_iov_head); + return done; +} + ssize_t qio_channel_readv(QIOChannel *ioc, const struct iovec *iov, size_t niov, diff --git a/migration/qemu-file-channel.c b/migration/qemu-file-channel.c index 45c13f1..dd95e06 100644 --- a/migration/qemu-file-channel.c +++ b/migration/qemu-file-channel.c @@ -34,35 +34,8 @@ static ssize_t channel_writev_buffer(void *opaque, int64_t pos) { QIOChannel *ioc =3D QIO_CHANNEL(opaque); - ssize_t done =3D 0; - struct iovec *local_iov =3D g_new(struct iovec, iovcnt); - struct iovec *local_iov_head =3D local_iov; - unsigned int nlocal_iov =3D iovcnt; =20 - nlocal_iov =3D iov_copy(local_iov, nlocal_iov, - iov, iovcnt, - 0, iov_size(iov, iovcnt)); - - while (nlocal_iov > 0) { - ssize_t len; - len =3D qio_channel_writev(ioc, local_iov, nlocal_iov, NULL); - if (len =3D=3D QIO_CHANNEL_ERR_BLOCK) { - qio_channel_wait(ioc, G_IO_OUT); - continue; - } - if (len < 0) { - /* XXX handle Error objects */ - done =3D -EIO; - goto cleanup; - } - - iov_discard_front(&local_iov, &nlocal_iov, len); - done +=3D len; - } - - cleanup: - g_free(local_iov_head); - return done; + return qio_channel_writev_all(ioc, iov, iovcnt, NULL); } =20 =20 --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409299637178.20737930906068; Mon, 13 Mar 2017 05:48:19 -0700 (PDT) Received: from localhost ([::1]:51898 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPOY-0006KO-7Z for importer@patchew.org; Mon, 13 Mar 2017 08:48:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56387) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPL8-0004FO-IW for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPL7-0005oS-HP for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:42 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38096) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPL7-0005ne-94 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:41 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 63F396AAC7 for ; Mon, 13 Mar 2017 12:44:41 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPs012445; Mon, 13 Mar 2017 08:44:40 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:20 +0100 Message-Id: <20170313124434.1043-3-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 13 Mar 2017 12:44:41 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 02/16] qio: create new qio_channel_read_all X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It is the symmetric function from qio_channel_write_all Signed-off-by: Juan Quintela --- include/io/channel.h | 23 +++++++++++++++++++++++ io/channel.c | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/include/io/channel.h b/include/io/channel.h index f786c4f..2f55831 100644 --- a/include/io/channel.h +++ b/include/io/channel.h @@ -269,6 +269,29 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc, Error **errp); =20 /** + * qio_channel_readv_all: + * @ioc: the channel object + * @iov: the array of memory regions to read data into + * @niov: the length of the @iov array + * @errp: pointer to a NULL-initialized error object + * + * Read data from the IO channel, storing it in the + * memory regions referenced by @iov. Each element + * in the @iov will be fully populated with data + * before the next one is used. The @niov parameter + * specifies the total number of elements in @iov. + * + * Returns: the number of bytes read, or -1 on error, + * or QIO_CHANNEL_ERR_BLOCK if no data is available + * and the channel is non-blocking + */ +ssize_t qio_channel_readv_all(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + Error **errp); + + +/** * qio_channel_writev_all: * @ioc: the channel object * @iov: the array of memory regions to write data from diff --git a/io/channel.c b/io/channel.c index c5a1bd5..82203ef 100644 --- a/io/channel.c +++ b/io/channel.c @@ -87,6 +87,43 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc, =20 =20 =20 +ssize_t qio_channel_readv_all(QIOChannel *ioc, + const struct iovec *iov, + size_t niov, + Error **errp) +{ + ssize_t done =3D 0; + struct iovec *local_iov =3D g_new(struct iovec, niov); + struct iovec *local_iov_head =3D local_iov; + unsigned int nlocal_iov =3D niov; + + nlocal_iov =3D iov_copy(local_iov, nlocal_iov, + iov, niov, + 0, iov_size(iov, niov)); + + while (nlocal_iov > 0) { + ssize_t len; + len =3D qio_channel_readv(ioc, local_iov, nlocal_iov, errp); + if (len =3D=3D QIO_CHANNEL_ERR_BLOCK) { + qio_channel_wait(ioc, G_IO_OUT); + continue; + } + if (len < 0) { + error_setg_errno(errp, EIO, + "Channel was not able to read full iov"); + done =3D -1; + goto cleanup; + } + + iov_discard_front(&local_iov, &nlocal_iov, len); + done +=3D len; + } + + cleanup: + g_free(local_iov_head); + return done; +} + ssize_t qio_channel_writev_all(QIOChannel *ioc, const struct iovec *iov, size_t niov, --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409146249656.6583614826369; Mon, 13 Mar 2017 05:45:46 -0700 (PDT) Received: from localhost ([::1]:51885 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPM6-0004H7-Oe for importer@patchew.org; Mon, 13 Mar 2017 08:45:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56404) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPL9-0004Fw-RC for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPL8-0005pc-Vn for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:43 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33470) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPL8-0005ow-Py for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:42 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E962D61BA8 for ; Mon, 13 Mar 2017 12:44:42 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPt012445; Mon, 13 Mar 2017 08:44:41 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:21 +0100 Message-Id: <20170313124434.1043-4-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 13 Mar 2017 12:44:42 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 03/16] migration: Test for disabled features on reception X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Right now, if we receive a compressed page while this features are disabled, Bad Things (TM) can happen. Just add a test for them. Signed-off-by: Juan Quintela -- I had XBZRLE here also, but it don't need extra resources on destination, only on source. Additionally libvirt don't enable it on destination, so don't put it here. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 719425b..65419c1 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2514,7 +2514,7 @@ static int ram_load_postcopy(QEMUFile *f) =20 static int ram_load(QEMUFile *f, void *opaque, int version_id) { - int flags =3D 0, ret =3D 0; + int flags =3D 0, ret =3D 0, invalid_flags; static uint64_t seq_iter; int len =3D 0; /* @@ -2531,6 +2531,11 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) ret =3D -EINVAL; } =20 + invalid_flags =3D 0; + + if (!migrate_use_compression()) { + invalid_flags |=3D RAM_SAVE_FLAG_COMPRESS_PAGE; + } /* This RCU critical section can be very long running. * When RCU reclaims in the code start to become numerous, * it will be necessary to reduce the granularity of this @@ -2551,6 +2556,15 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) flags =3D addr & ~TARGET_PAGE_MASK; addr &=3D TARGET_PAGE_MASK; =20 + if (flags & invalid_flags) { + if (flags & invalid_flags & RAM_SAVE_FLAG_COMPRESS_PAGE) { + error_report("Received an unexpected compressed page"); + } + + ret =3D -EINVAL; + break; + } + if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409444020237.8086330616054; Mon, 13 Mar 2017 05:50:44 -0700 (PDT) Received: from localhost ([::1]:51908 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPQs-0008Id-QF for importer@patchew.org; Mon, 13 Mar 2017 08:50:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56419) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLB-0004Gs-7k for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLA-0005r6-J7 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:45 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43086) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLA-0005qP-DX for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:44 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 82469C0092A2 for ; Mon, 13 Mar 2017 12:44:44 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPu012445; Mon, 13 Mar 2017 08:44:43 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:22 +0100 Message-Id: <20170313124434.1043-5-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 13 Mar 2017 12:44:44 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 04/16] migration: Don't create decompression threads if not enabled X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Juan Quintela -- I removed the [HACK] part because previous patch just check that compression pages are not received. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 65419c1..aa51dbd 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2306,6 +2306,9 @@ void migrate_decompress_threads_create(void) { int i, thread_count; =20 + if (!migrate_use_compression()) { + return; + } thread_count =3D migrate_decompress_threads(); decompress_threads =3D g_new0(QemuThread, thread_count); decomp_param =3D g_new0(DecompressParam, thread_count); @@ -2327,6 +2330,9 @@ void migrate_decompress_threads_join(void) { int i, thread_count; =20 + if (!migrate_use_compression()) { + return; + } thread_count =3D migrate_decompress_threads(); for (i =3D 0; i < thread_count; i++) { qemu_mutex_lock(&decomp_param[i].mutex); --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409295082138.77021973172282; Mon, 13 Mar 2017 05:48:15 -0700 (PDT) Received: from localhost ([::1]:51897 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPOV-0006Hj-Kq for importer@patchew.org; Mon, 13 Mar 2017 08:48:11 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56431) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLC-0004IH-RP for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLC-0005ry-2u for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59082) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLB-0005rW-TR for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:46 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0A2A23A76A8 for ; Mon, 13 Mar 2017 12:44:46 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPv012445; Mon, 13 Mar 2017 08:44:44 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:23 +0100 Message-Id: <20170313124434.1043-6-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 13 Mar 2017 12:44:46 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 05/16] migration: Add multifd capability X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- include/migration/migration.h | 1 + migration/migration.c | 9 +++++++++ qapi-schema.json | 6 ++++-- 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/include/migration/migration.h b/include/migration/migration.h index 5720c88..d48934b 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -323,6 +323,7 @@ bool migrate_postcopy_ram(void); bool migrate_zero_blocks(void); =20 bool migrate_auto_converge(void); +bool migrate_use_multifd(void); =20 int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_buf, int slen, uint8_t *dst, int dlen); diff --git a/migration/migration.c b/migration/migration.c index 3dab684..0e5b0bf 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1409,6 +1409,15 @@ bool migrate_use_events(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS]; } =20 +bool migrate_use_multifd(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_X_MULTIFD]; +} + int migrate_use_xbzrle(void) { MigrationState *s; diff --git a/qapi-schema.json b/qapi-schema.json index 32b4a4b..d21934b 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -868,12 +868,14 @@ # @release-ram: if enabled, qemu will free the migrated ram pages on the s= ource # during postcopy-ram migration. (since 2.9) # +# @x-multifd: Use more than one fd for migration (since 2.9) +# # Since: 1.2 ## { 'enum': 'MigrationCapability', 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', - 'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram'] } - + 'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram', + 'x-multifd'] } ## # @MigrationCapabilityStatus: # --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409315657144.66828282845904; Mon, 13 Mar 2017 05:48:35 -0700 (PDT) Received: from localhost ([::1]:51901 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPOo-0006bO-DL for importer@patchew.org; Mon, 13 Mar 2017 08:48:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56459) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLG-0004L7-1Q for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLD-0005sp-Nu for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:50 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46170) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLD-0005sI-F4 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:47 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 969CE7712 for ; Mon, 13 Mar 2017 12:44:47 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPw012445; Mon, 13 Mar 2017 08:44:46 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:24 +0100 Message-Id: <20170313124434.1043-7-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Mon, 13 Mar 2017 12:44:47 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 06/16] migration: Create x-multifd-threads parameter X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Indicates the number of threads that we would create. By default we create 2 threads. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert -- Catch inconsistent defaults. Thanks Eric Signed-off-by: Juan Quintela --- hmp.c | 8 ++++++++ include/migration/migration.h | 2 ++ migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 13 +++++++++++-- 4 files changed, 44 insertions(+), 2 deletions(-) diff --git a/hmp.c b/hmp.c index 261843f..7d6db63 100644 --- a/hmp.c +++ b/hmp.c @@ -323,6 +323,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QD= ict *qdict) monitor_printf(mon, " %s: %" PRId64, MigrationParameter_lookup[MIGRATION_PARAMETER_X_CHECKPOINT_DEL= AY], params->x_checkpoint_delay); + monitor_printf(mon, " %s: %" PRId64, + MigrationParameter_lookup[MIGRATION_PARAMETER_X_MULTIFD_THREAD= S], + params->x_multifd_threads); monitor_printf(mon, "\n"); } =20 @@ -1400,6 +1403,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const Q= Dict *qdict) p.has_x_checkpoint_delay =3D true; use_int_value =3D true; break; + case MIGRATION_PARAMETER_X_MULTIFD_THREADS: + p.has_x_multifd_threads =3D true; + use_int_value =3D true; + break; } =20 if (use_int_value) { @@ -1417,6 +1424,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QD= ict *qdict) p.cpu_throttle_increment =3D valueint; p.downtime_limit =3D valueint; p.x_checkpoint_delay =3D valueint; + p.x_multifd_threads =3D valueint; } =20 qmp_migrate_set_parameters(&p, &err); diff --git a/include/migration/migration.h b/include/migration/migration.h index d48934b..2950270 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -265,6 +265,8 @@ bool migration_in_postcopy(MigrationState *); bool migration_in_postcopy_after_devices(MigrationState *); MigrationState *migrate_get_current(void); =20 +int migrate_multifd_threads(void); + void migrate_compress_threads_create(void); void migrate_compress_threads_join(void); void migrate_decompress_threads_create(void); diff --git a/migration/migration.c b/migration/migration.c index 0e5b0bf..cc9440b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -71,6 +71,7 @@ * Note: Please change this default value to 10000 when we support hybrid = mode. */ #define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_MULTIFD_THREADS 2 =20 static NotifierList migration_state_notifiers =3D NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); @@ -105,6 +106,7 @@ MigrationState *migrate_get_current(void) .max_bandwidth =3D MAX_THROTTLE, .downtime_limit =3D DEFAULT_MIGRATE_SET_DOWNTIME, .x_checkpoint_delay =3D DEFAULT_MIGRATE_X_CHECKPOINT_DELAY, + .x_multifd_threads =3D DEFAULT_MIGRATE_MULTIFD_THREADS, }, }; =20 @@ -596,6 +598,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error= **errp) params->downtime_limit =3D s->parameters.downtime_limit; params->has_x_checkpoint_delay =3D true; params->x_checkpoint_delay =3D s->parameters.x_checkpoint_delay; + params->has_x_multifd_threads =3D true; + params->x_multifd_threads =3D s->parameters.x_multifd_threads; =20 return params; } @@ -860,6 +864,13 @@ void qmp_migrate_set_parameters(MigrationParameters *p= arams, Error **errp) "x_checkpoint_delay", "is invalid, it should be positive"); } + if (params->has_x_multifd_threads && + (params->x_multifd_threads < 1 || params->x_multifd_threads > 255)= ) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "multifd_threads", + "is invalid, it should be in the range of 1 to 255"); + return; + } =20 if (params->has_compress_level) { s->parameters.compress_level =3D params->compress_level; @@ -901,6 +912,9 @@ void qmp_migrate_set_parameters(MigrationParameters *pa= rams, Error **errp) colo_checkpoint_notify(s); } } + if (params->has_x_multifd_threads) { + s->parameters.x_multifd_threads =3D params->x_multifd_threads; + } } =20 =20 @@ -1418,6 +1432,15 @@ bool migrate_use_multifd(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_X_MULTIFD]; } =20 +int migrate_multifd_threads(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->parameters.x_multifd_threads; +} + int migrate_use_xbzrle(void) { MigrationState *s; diff --git a/qapi-schema.json b/qapi-schema.json index d21934b..b7cb26d 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -985,13 +985,17 @@ # @x-checkpoint-delay: The delay time (in ms) between two COLO checkpoints= in # periodic mode. (Since 2.8) # +# @x-multifd-threads: Number of threads used to migrate data in parallel +# The default value is 2 (since 2.9) +# # Since: 2.4 ## { 'enum': 'MigrationParameter', 'data': ['compress-level', 'compress-threads', 'decompress-threads', 'cpu-throttle-initial', 'cpu-throttle-increment', 'tls-creds', 'tls-hostname', 'max-bandwidth', - 'downtime-limit', 'x-checkpoint-delay' ] } + 'downtime-limit', 'x-checkpoint-delay', + 'x-multifd-threads'] } =20 ## # @migrate-set-parameters: @@ -1054,6 +1058,10 @@ # # @x-checkpoint-delay: the delay time between two COLO checkpoints. (Since= 2.8) # +# +# @x-multifd-threads: Number of threads used to migrate data in parallel +# The default value is 2 (since 2.9) +# # Since: 2.4 ## { 'struct': 'MigrationParameters', @@ -1066,7 +1074,8 @@ '*tls-hostname': 'str', '*max-bandwidth': 'int', '*downtime-limit': 'int', - '*x-checkpoint-delay': 'int'} } + '*x-checkpoint-delay': 'int', + '*x-multifd-threads': 'int'} } =20 ## # @query-migrate-parameters: --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409443063186.47858920768113; Mon, 13 Mar 2017 05:50:43 -0700 (PDT) Received: from localhost ([::1]:51906 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPQs-0008Gw-Kn for importer@patchew.org; Mon, 13 Mar 2017 08:50:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56461) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLG-0004LR-B0 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLF-0005tb-AF for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:50 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43154) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLF-0005tB-1m for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:49 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 20CEDC05AA40 for ; Mon, 13 Mar 2017 12:44:49 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaPx012445; Mon, 13 Mar 2017 08:44:47 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:25 +0100 Message-Id: <20170313124434.1043-8-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 13 Mar 2017 12:44:49 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 07/16] migration: Create x-multifd-group parameter X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Indicates how many pages we are going to send in each bach to a multifd thread. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert -- Be consistent with defaults and documentation Signed-off-by: Juan Quintela --- hmp.c | 8 ++++++++ include/migration/migration.h | 1 + migration/migration.c | 23 +++++++++++++++++++++++ qapi-schema.json | 11 +++++++++-- 4 files changed, 41 insertions(+), 2 deletions(-) diff --git a/hmp.c b/hmp.c index 7d6db63..ab02773 100644 --- a/hmp.c +++ b/hmp.c @@ -326,6 +326,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QD= ict *qdict) monitor_printf(mon, " %s: %" PRId64, MigrationParameter_lookup[MIGRATION_PARAMETER_X_MULTIFD_THREAD= S], params->x_multifd_threads); + monitor_printf(mon, " %s: %" PRId64, + MigrationParameter_lookup[MIGRATION_PARAMETER_X_MULTIFD_GROUP], + params->x_multifd_group); monitor_printf(mon, "\n"); } =20 @@ -1407,6 +1410,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const Q= Dict *qdict) p.has_x_multifd_threads =3D true; use_int_value =3D true; break; + case MIGRATION_PARAMETER_X_MULTIFD_GROUP: + p.has_x_multifd_group =3D true; + use_int_value =3D true; + break; } =20 if (use_int_value) { @@ -1425,6 +1432,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QD= ict *qdict) p.downtime_limit =3D valueint; p.x_checkpoint_delay =3D valueint; p.x_multifd_threads =3D valueint; + p.x_multifd_group =3D valueint; } =20 qmp_migrate_set_parameters(&p, &err); diff --git a/include/migration/migration.h b/include/migration/migration.h index 2950270..bacde15 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -266,6 +266,7 @@ bool migration_in_postcopy_after_devices(MigrationState= *); MigrationState *migrate_get_current(void); =20 int migrate_multifd_threads(void); +int migrate_multifd_group(void); =20 void migrate_compress_threads_create(void); void migrate_compress_threads_join(void); diff --git a/migration/migration.c b/migration/migration.c index cc9440b..4cc45a4 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -72,6 +72,7 @@ */ #define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 #define DEFAULT_MIGRATE_MULTIFD_THREADS 2 +#define DEFAULT_MIGRATE_MULTIFD_GROUP 16 =20 static NotifierList migration_state_notifiers =3D NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); @@ -107,6 +108,7 @@ MigrationState *migrate_get_current(void) .downtime_limit =3D DEFAULT_MIGRATE_SET_DOWNTIME, .x_checkpoint_delay =3D DEFAULT_MIGRATE_X_CHECKPOINT_DELAY, .x_multifd_threads =3D DEFAULT_MIGRATE_MULTIFD_THREADS, + .x_multifd_group =3D DEFAULT_MIGRATE_MULTIFD_GROUP, }, }; =20 @@ -600,6 +602,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error= **errp) params->x_checkpoint_delay =3D s->parameters.x_checkpoint_delay; params->has_x_multifd_threads =3D true; params->x_multifd_threads =3D s->parameters.x_multifd_threads; + params->has_x_multifd_group =3D true; + params->x_multifd_group =3D s->parameters.x_multifd_group; =20 return params; } @@ -871,6 +875,13 @@ void qmp_migrate_set_parameters(MigrationParameters *p= arams, Error **errp) "is invalid, it should be in the range of 1 to 255"); return; } + if (params->has_x_multifd_group && + (params->x_multifd_group < 1 || params->x_multifd_group > 1000= 0)) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, + "multifd_group", + "is invalid, it should be in the range of 1 to 10000"); + return; + } =20 if (params->has_compress_level) { s->parameters.compress_level =3D params->compress_level; @@ -915,6 +926,9 @@ void qmp_migrate_set_parameters(MigrationParameters *pa= rams, Error **errp) if (params->has_x_multifd_threads) { s->parameters.x_multifd_threads =3D params->x_multifd_threads; } + if (params->has_x_multifd_group) { + s->parameters.x_multifd_group =3D params->x_multifd_group; + } } =20 =20 @@ -1441,6 +1455,15 @@ int migrate_multifd_threads(void) return s->parameters.x_multifd_threads; } =20 +int migrate_multifd_group(void) +{ + MigrationState *s; + + s =3D migrate_get_current(); + + return s->parameters.x_multifd_group; +} + int migrate_use_xbzrle(void) { MigrationState *s; diff --git a/qapi-schema.json b/qapi-schema.json index b7cb26d..33a6267 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -988,6 +988,9 @@ # @x-multifd-threads: Number of threads used to migrate data in parallel # The default value is 2 (since 2.9) # +# @x-multifd-group: Number of pages sent together to a thread +# The default value is 16 (since 2.9) +# # Since: 2.4 ## { 'enum': 'MigrationParameter', @@ -995,7 +998,7 @@ 'cpu-throttle-initial', 'cpu-throttle-increment', 'tls-creds', 'tls-hostname', 'max-bandwidth', 'downtime-limit', 'x-checkpoint-delay', - 'x-multifd-threads'] } + 'x-multifd-threads', 'x-multifd-group'] } =20 ## # @migrate-set-parameters: @@ -1062,6 +1065,9 @@ # @x-multifd-threads: Number of threads used to migrate data in parallel # The default value is 2 (since 2.9) # +# @x-multifd-group: Number of pages sent together in a bunch +# The default value is 16 (since 2.9) +# # Since: 2.4 ## { 'struct': 'MigrationParameters', @@ -1075,7 +1081,8 @@ '*max-bandwidth': 'int', '*downtime-limit': 'int', '*x-checkpoint-delay': 'int', - '*x-multifd-threads': 'int'} } + '*x-multifd-threads': 'int', + '*x-multifd-group': 'int'} } =20 ## # @query-migrate-parameters: --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409162455766.0945020547097; Mon, 13 Mar 2017 05:46:02 -0700 (PDT) Received: from localhost ([::1]:51886 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPMM-0004Ps-34 for importer@patchew.org; Mon, 13 Mar 2017 08:45:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56477) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLH-0004Mv-WE for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLG-0005uU-Ov for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33564) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLG-0005u6-Go for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:50 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A4CAF61BB9 for ; Mon, 13 Mar 2017 12:44:50 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ0012445; Mon, 13 Mar 2017 08:44:49 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:26 +0100 Message-Id: <20170313124434.1043-9-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 13 Mar 2017 12:44:50 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 08/16] migration: Create multifd migration threads X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Creation of the threads, nothing inside yet. Signed-off-by: Juan Quintela -- Use pointers instead of long array names Move to use semaphores instead of conditions as paolo suggestion Put all the state inside one struct. Use a counter for the number of threads created. Needed during cancellatio= n. Add error return to thread creation Add id field Signed-off-by: Juan Quintela --- include/migration/migration.h | 4 + migration/migration.c | 15 ++++ migration/ram.c | 188 ++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 207 insertions(+) diff --git a/include/migration/migration.h b/include/migration/migration.h index bacde15..e8b9fcb 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -267,6 +267,10 @@ MigrationState *migrate_get_current(void); =20 int migrate_multifd_threads(void); int migrate_multifd_group(void); +int migrate_multifd_send_threads_create(void); +void migrate_multifd_send_threads_join(void); +int migrate_multifd_recv_threads_create(void); +void migrate_multifd_recv_threads_join(void); =20 void migrate_compress_threads_create(void); void migrate_compress_threads_join(void); diff --git a/migration/migration.c b/migration/migration.c index 4cc45a4..5bbd688 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -348,6 +348,7 @@ static void process_incoming_migration_bh(void *opaque) MIGRATION_STATUS_FAILED); error_report_err(local_err); migrate_decompress_threads_join(); + migrate_multifd_recv_threads_join(); exit(EXIT_FAILURE); } =20 @@ -372,6 +373,7 @@ static void process_incoming_migration_bh(void *opaque) runstate_set(global_state_get_runstate()); } migrate_decompress_threads_join(); + migrate_multifd_recv_threads_join(); /* * This must happen after any state changes since as soon as an extern= al * observer sees this event they might start to prod at the VM assuming @@ -438,6 +440,7 @@ static void process_incoming_migration_co(void *opaque) MIGRATION_STATUS_FAILED); error_report("load of migration failed: %s", strerror(-ret)); migrate_decompress_threads_join(); + migrate_multifd_recv_threads_join(); exit(EXIT_FAILURE); } =20 @@ -450,6 +453,11 @@ void migration_fd_process_incoming(QEMUFile *f) Coroutine *co =3D qemu_coroutine_create(process_incoming_migration_co,= f); =20 migrate_decompress_threads_create(); + if (migrate_multifd_recv_threads_create() !=3D 0) { + /* We haven't been able to create multifd threads + nothing better to do */ + exit(EXIT_FAILURE); + } qemu_file_set_blocking(f, false); qemu_coroutine_enter(co); } @@ -983,6 +991,7 @@ static void migrate_fd_cleanup(void *opaque) qemu_mutex_lock_iothread(); =20 migrate_compress_threads_join(); + migrate_multifd_send_threads_join(); qemu_fclose(s->to_dst_file); s->to_dst_file =3D NULL; } @@ -2144,6 +2153,12 @@ void migrate_fd_connect(MigrationState *s) } =20 migrate_compress_threads_create(); + if (migrate_multifd_send_threads_create() !=3D 0) { + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, + MIGRATION_STATUS_FAILED); + migrate_fd_cleanup(s); + return; + } qemu_thread_create(&s->thread, "live_migration", migration_thread, s, QEMU_THREAD_JOINABLE); s->migration_thread_running =3D true; diff --git a/migration/ram.c b/migration/ram.c index aa51dbd..ee32fa8 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -382,6 +382,194 @@ void migrate_compress_threads_create(void) } } =20 +/* Multiple fd's */ + +struct MultiFDSendParams { + int id; + QemuThread thread; + QemuSemaphore sem; + QemuMutex mutex; + bool quit; +}; +typedef struct MultiFDSendParams MultiFDSendParams; + +struct { + MultiFDSendParams *params; + /* number o6 created threads */ + int count; +} *multifd_send_state; + +static void terminate_multifd_send_threads(void) +{ + int i; + + for (i =3D 0; i < multifd_send_state->count; i++) { + MultiFDSendParams *p =3D &multifd_send_state->params[i]; + + qemu_mutex_lock(&p->mutex); + p->quit =3D true; + qemu_sem_post(&p->sem); + qemu_mutex_unlock(&p->mutex); + } +} + +void migrate_multifd_send_threads_join(void) +{ + int i; + + if (!migrate_use_multifd()) { + return; + } + terminate_multifd_send_threads(); + for (i =3D 0; i < multifd_send_state->count; i++) { + MultiFDSendParams *p =3D &multifd_send_state->params[i]; + + qemu_thread_join(&p->thread); + qemu_mutex_destroy(&p->mutex); + qemu_sem_destroy(&p->sem); + } + g_free(multifd_send_state->params); + multifd_send_state->params =3D NULL; + g_free(multifd_send_state); + multifd_send_state =3D NULL; +} + +static void *multifd_send_thread(void *opaque) +{ + MultiFDSendParams *p =3D opaque; + + while (true) { + qemu_mutex_lock(&p->mutex); + if (p->quit) { + qemu_mutex_unlock(&p->mutex); + break; + } + qemu_mutex_unlock(&p->mutex); + qemu_sem_wait(&p->sem); + } + + return NULL; +} + +int migrate_multifd_send_threads_create(void) +{ + int i, thread_count; + + if (!migrate_use_multifd()) { + return 0; + } + thread_count =3D migrate_multifd_threads(); + multifd_send_state =3D g_malloc0(sizeof(*multifd_send_state)); + multifd_send_state->params =3D g_new0(MultiFDSendParams, thread_count); + multifd_send_state->count =3D 0; + for (i =3D 0; i < thread_count; i++) { + char thread_name[15]; + MultiFDSendParams *p =3D &multifd_send_state->params[i]; + + qemu_mutex_init(&p->mutex); + qemu_sem_init(&p->sem, 0); + p->quit =3D false; + p->id =3D i; + snprintf(thread_name, 15, "multifd_send_%d", i); + qemu_thread_create(&p->thread, thread_name, multifd_send_thread, p, + QEMU_THREAD_JOINABLE); + multifd_send_state->count++; + } + return 0; +} + +struct MultiFDRecvParams { + int id; + QemuThread thread; + QemuSemaphore sem; + QemuMutex mutex; + bool quit; +}; +typedef struct MultiFDRecvParams MultiFDRecvParams; + +struct { + MultiFDRecvParams *params; + /* number o6 created threads */ + int count; +} *multifd_recv_state; + +static void terminate_multifd_recv_threads(void) +{ + int i; + + for (i =3D 0; i < multifd_recv_state->count; i++) { + MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; + + qemu_mutex_lock(&p->mutex); + p->quit =3D true; + qemu_sem_post(&p->sem); + qemu_mutex_unlock(&p->mutex); + } +} + +void migrate_multifd_recv_threads_join(void) +{ + int i; + + if (!migrate_use_multifd()) { + return; + } + terminate_multifd_recv_threads(); + for (i =3D 0; i < multifd_recv_state->count; i++) { + MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; + + qemu_thread_join(&p->thread); + qemu_mutex_destroy(&p->mutex); + qemu_sem_destroy(&p->sem); + } + g_free(multifd_recv_state->params); + multifd_recv_state->params =3D NULL; + g_free(multifd_recv_state); + multifd_recv_state =3D NULL; +} + +static void *multifd_recv_thread(void *opaque) +{ + MultiFDRecvParams *p =3D opaque; + + while (true) { + qemu_mutex_lock(&p->mutex); + if (p->quit) { + qemu_mutex_unlock(&p->mutex); + break; + } + qemu_mutex_unlock(&p->mutex); + qemu_sem_wait(&p->sem); + } + + return NULL; +} + +int migrate_multifd_recv_threads_create(void) +{ + int i, thread_count; + + if (!migrate_use_multifd()) { + return 0; + } + thread_count =3D migrate_multifd_threads(); + multifd_recv_state =3D g_malloc0(sizeof(*multifd_recv_state)); + multifd_recv_state->params =3D g_new0(MultiFDRecvParams, thread_count); + multifd_recv_state->count =3D 0; + for (i =3D 0; i < thread_count; i++) { + MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; + + qemu_mutex_init(&p->mutex); + qemu_sem_init(&p->sem, 0); + p->quit =3D false; + p->id =3D i; + qemu_thread_create(&p->thread, "multifd_recv", multifd_recv_thread= , p, + QEMU_THREAD_JOINABLE); + multifd_recv_state->count++; + } + return 0; +} + /** * save_page_header: Write page header to wire * --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409458344515.6691060873436; Mon, 13 Mar 2017 05:50:58 -0700 (PDT) Received: from localhost ([::1]:51909 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPR8-0008WA-0E for importer@patchew.org; Mon, 13 Mar 2017 08:50:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56489) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLJ-0004OR-Hm for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLI-0005vm-AG for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:32860) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLI-0005v7-1f for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:52 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 363B683F46 for ; Mon, 13 Mar 2017 12:44:52 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ1012445; Mon, 13 Mar 2017 08:44:50 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:27 +0100 Message-Id: <20170313124434.1043-10-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Mon, 13 Mar 2017 12:44:52 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 09/16] migration: Start of multiple fd work X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We create new channels for each new thread created. We only send through them a character to be sure that we are creating the channels in the right order. Signed-off-by: Juan Quintela -- Split SocketArgs into incoming and outgoing args Signed-off-by: Juan Quintela --- include/migration/migration.h | 7 +++++ migration/ram.c | 35 ++++++++++++++++++++++ migration/socket.c | 67 +++++++++++++++++++++++++++++++++++++++= ++-- 3 files changed, 106 insertions(+), 3 deletions(-) diff --git a/include/migration/migration.h b/include/migration/migration.h index e8b9fcb..cbb049d 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -23,6 +23,7 @@ #include "exec/cpu-common.h" #include "qemu/coroutine_int.h" #include "qom/object.h" +#include "io/channel.h" =20 #define QEMU_VM_FILE_MAGIC 0x5145564d #define QEMU_VM_FILE_VERSION_COMPAT 0x00000002 @@ -235,6 +236,12 @@ void tcp_start_incoming_migration(const char *host_por= t, Error **errp); =20 void tcp_start_outgoing_migration(MigrationState *s, const char *host_port= , Error **errp); =20 +QIOChannel *socket_recv_channel_create(void); +int socket_recv_channel_destroy(QIOChannel *recv); +int socket_recv_channel_close_listening(void); +QIOChannel *socket_send_channel_create(void); +int socket_send_channel_destroy(QIOChannel *send); + void unix_start_incoming_migration(const char *path, Error **errp); =20 void unix_start_outgoing_migration(MigrationState *s, const char *path, Er= ror **errp); diff --git a/migration/ram.c b/migration/ram.c index ee32fa8..7833e6f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -387,7 +387,9 @@ void migrate_compress_threads_create(void) struct MultiFDSendParams { int id; QemuThread thread; + QIOChannel *c; QemuSemaphore sem; + QemuSemaphore init; QemuMutex mutex; bool quit; }; @@ -427,6 +429,8 @@ void migrate_multifd_send_threads_join(void) qemu_thread_join(&p->thread); qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->init); + socket_send_channel_destroy(p->c); } g_free(multifd_send_state->params); multifd_send_state->params =3D NULL; @@ -438,6 +442,11 @@ static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p =3D opaque; =20 + char start =3D 's'; + + qio_channel_write(p->c, &start, 1, &error_abort); + qemu_sem_post(&p->init); + while (true) { qemu_mutex_lock(&p->mutex); if (p->quit) { @@ -468,12 +477,20 @@ int migrate_multifd_send_threads_create(void) =20 qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); + qemu_sem_init(&p->init, 0); p->quit =3D false; p->id =3D i; + p->c =3D socket_send_channel_create(); + if (!p->c) { + error_report("Error creating a send channel"); + migrate_multifd_send_threads_join(); + return -1; + } snprintf(thread_name, 15, "multifd_send_%d", i); qemu_thread_create(&p->thread, thread_name, multifd_send_thread, p, QEMU_THREAD_JOINABLE); multifd_send_state->count++; + qemu_sem_wait(&p->init); } return 0; } @@ -481,6 +498,8 @@ int migrate_multifd_send_threads_create(void) struct MultiFDRecvParams { int id; QemuThread thread; + QIOChannel *c; + QemuSemaphore init; QemuSemaphore sem; QemuMutex mutex; bool quit; @@ -521,6 +540,8 @@ void migrate_multifd_recv_threads_join(void) qemu_thread_join(&p->thread); qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->init); + socket_send_channel_destroy(multifd_recv_state->params[i].c); } g_free(multifd_recv_state->params); multifd_recv_state->params =3D NULL; @@ -531,6 +552,10 @@ void migrate_multifd_recv_threads_join(void) static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p =3D opaque; + char start; + + qio_channel_read(p->c, &start, 1, &error_abort); + qemu_sem_post(&p->init); =20 while (true) { qemu_mutex_lock(&p->mutex); @@ -561,12 +586,22 @@ int migrate_multifd_recv_threads_create(void) =20 qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); + qemu_sem_init(&p->init, 0); p->quit =3D false; p->id =3D i; + p->c =3D socket_recv_channel_create(); + + if (!p->c) { + error_report("Error creating a recv channel"); + migrate_multifd_recv_threads_join(); + return -1; + } qemu_thread_create(&p->thread, "multifd_recv", multifd_recv_thread= , p, QEMU_THREAD_JOINABLE); multifd_recv_state->count++; + qemu_sem_wait(&p->init); } + socket_recv_channel_close_listening(); return 0; } =20 diff --git a/migration/socket.c b/migration/socket.c index 13966f1..58a16b5 100644 --- a/migration/socket.c +++ b/migration/socket.c @@ -24,6 +24,65 @@ #include "io/channel-socket.h" #include "trace.h" =20 +struct SocketIncomingArgs { + QIOChannelSocket *ioc; +} incoming_args; + +QIOChannel *socket_recv_channel_create(void) +{ + QIOChannelSocket *sioc; + Error *err =3D NULL; + + sioc =3D qio_channel_socket_accept(QIO_CHANNEL_SOCKET(incoming_args.io= c), + &err); + if (!sioc) { + error_report("could not accept migration connection (%s)", + error_get_pretty(err)); + return NULL; + } + return QIO_CHANNEL(sioc); +} + +int socket_recv_channel_destroy(QIOChannel *recv) +{ + /* Remove channel */ + object_unref(OBJECT(send)); + return 0; +} + +/* we have created all the recv channels, we can close the main one */ +int socket_recv_channel_close_listening(void) +{ + /* Close listening socket as its no longer needed */ + qio_channel_close(QIO_CHANNEL(incoming_args.ioc), NULL); + return 0; +} + +struct SocketOutgoingArgs { + SocketAddress *saddr; + Error **errp; +} outgoing_args; + +QIOChannel *socket_send_channel_create(void) +{ + QIOChannelSocket *sioc =3D qio_channel_socket_new(); + + qio_channel_socket_connect_sync(sioc, outgoing_args.saddr, + outgoing_args.errp); + qio_channel_set_delay(QIO_CHANNEL(sioc), false); + return QIO_CHANNEL(sioc); +} + +int socket_send_channel_destroy(QIOChannel *send) +{ + /* Remove channel */ + object_unref(OBJECT(send)); + if (outgoing_args.saddr) { + qapi_free_SocketAddress(outgoing_args.saddr); + outgoing_args.saddr =3D NULL; + } + return 0; +} =20 static SocketAddress *tcp_build_address(const char *host_port, Error **err= p) { @@ -97,6 +156,10 @@ static void socket_start_outgoing_migration(MigrationSt= ate *s, struct SocketConnectData *data =3D g_new0(struct SocketConnectData, 1); =20 data->s =3D s; + + outgoing_args.saddr =3D saddr; + outgoing_args.errp =3D errp; + if (saddr->type =3D=3D SOCKET_ADDRESS_KIND_INET) { data->hostname =3D g_strdup(saddr->u.inet.data->host); } @@ -107,7 +170,6 @@ static void socket_start_outgoing_migration(MigrationSt= ate *s, socket_outgoing_migration, data, socket_connect_data_free); - qapi_free_SocketAddress(saddr); } =20 void tcp_start_outgoing_migration(MigrationState *s, @@ -154,8 +216,6 @@ static gboolean socket_accept_incoming_migration(QIOCha= nnel *ioc, object_unref(OBJECT(sioc)); =20 out: - /* Close listening socket as its no longer needed */ - qio_channel_close(ioc, NULL); return FALSE; /* unregister */ } =20 @@ -164,6 +224,7 @@ static void socket_start_incoming_migration(SocketAddre= ss *saddr, Error **errp) { QIOChannelSocket *listen_ioc =3D qio_channel_socket_new(); + incoming_args.ioc =3D listen_ioc; =20 qio_channel_set_name(QIO_CHANNEL(listen_ioc), "migration-socket-listener"); --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409574878822.3348551990124; Mon, 13 Mar 2017 05:52:54 -0700 (PDT) Received: from localhost ([::1]:51922 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPT3-0001yB-Jc for importer@patchew.org; Mon, 13 Mar 2017 08:52:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56501) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLL-0004Q4-9a for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLJ-0005wU-S1 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:39344) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLJ-0005w7-JU for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:53 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B8D5E3B731 for ; Mon, 13 Mar 2017 12:44:53 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ2012445; Mon, 13 Mar 2017 08:44:52 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:28 +0100 Message-Id: <20170313124434.1043-11-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 13 Mar 2017 12:44:53 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 10/16] migration: Create ram_multifd_page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The function still don't use multifd, but we have simplified ram_save_page, xbzrle and RDMA stuff is gone. We have added a new counter and a new flag for this type of pages. Signed-off-by: Juan Quintela --- hmp.c | 2 + include/migration/migration.h | 1 + migration/migration.c | 1 + migration/ram.c | 98 +++++++++++++++++++++++++++++++++++++++= +++- qapi-schema.json | 4 +- 5 files changed, 103 insertions(+), 3 deletions(-) diff --git a/hmp.c b/hmp.c index ab02773..1238d0f 100644 --- a/hmp.c +++ b/hmp.c @@ -223,6 +223,8 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) monitor_printf(mon, "postcopy request count: %" PRIu64 "\n", info->ram->postcopy_requests); } + monitor_printf(mon, "multifd: %" PRIu64 " pages\n", + info->ram->multifd); } =20 if (info->has_disk) { diff --git a/include/migration/migration.h b/include/migration/migration.h index cbb049d..bd152c5 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -301,6 +301,7 @@ uint64_t xbzrle_mig_pages_transferred(void); uint64_t xbzrle_mig_pages_overflow(void); uint64_t xbzrle_mig_pages_cache_miss(void); double xbzrle_mig_cache_miss_rate(void); +uint64_t multifd_mig_pages_transferred(void); =20 void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); void ram_debug_dump_bitmap(unsigned long *todump, bool expected); diff --git a/migration/migration.c b/migration/migration.c index 5bbd688..a32e4ad 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -661,6 +661,7 @@ static void populate_ram_info(MigrationInfo *info, Migr= ationState *s) info->ram->mbps =3D s->mbps; info->ram->dirty_sync_count =3D s->dirty_sync_count; info->ram->postcopy_requests =3D s->postcopy_requests; + info->ram->multifd =3D multifd_mig_pages_transferred(); =20 if (s->state !=3D MIGRATION_STATUS_COMPLETED) { info->ram->remaining =3D ram_bytes_remaining(); diff --git a/migration/ram.c b/migration/ram.c index 7833e6f..ccd7fe9 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -61,6 +61,7 @@ static uint64_t bitmap_sync_count; #define RAM_SAVE_FLAG_XBZRLE 0x40 /* 0x80 is reserved in migration.h start with 0x100 next */ #define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100 +#define RAM_SAVE_FLAG_MULTIFD_PAGE 0x200 =20 static uint8_t *ZERO_TARGET_PAGE; =20 @@ -141,6 +142,7 @@ typedef struct AccountingInfo { uint64_t dup_pages; uint64_t skipped_pages; uint64_t norm_pages; + uint64_t multifd_pages; uint64_t iterations; uint64_t xbzrle_bytes; uint64_t xbzrle_pages; @@ -211,6 +213,11 @@ uint64_t xbzrle_mig_pages_overflow(void) return acct_info.xbzrle_overflows; } =20 +uint64_t multifd_mig_pages_transferred(void) +{ + return acct_info.multifd_pages; +} + /* This is the last block that we have visited serching for dirty pages */ static RAMBlock *last_seen_block; @@ -385,13 +392,18 @@ void migrate_compress_threads_create(void) /* Multiple fd's */ =20 struct MultiFDSendParams { + /* not changed */ int id; QemuThread thread; QIOChannel *c; QemuSemaphore sem; QemuSemaphore init; QemuMutex mutex; + /* protected by param mutex */ bool quit; + uint8_t *address; + /* protected by multifd mutex */ + bool done; }; typedef struct MultiFDSendParams MultiFDSendParams; =20 @@ -399,6 +411,8 @@ struct { MultiFDSendParams *params; /* number o6 created threads */ int count; + QemuMutex mutex; + QemuSemaphore sem; } *multifd_send_state; =20 static void terminate_multifd_send_threads(void) @@ -441,11 +455,11 @@ void migrate_multifd_send_threads_join(void) static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p =3D opaque; - char start =3D 's'; =20 qio_channel_write(p->c, &start, 1, &error_abort); qemu_sem_post(&p->init); + qemu_sem_post(&multifd_send_state->sem); =20 while (true) { qemu_mutex_lock(&p->mutex); @@ -453,6 +467,15 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_unlock(&p->mutex); break; } + if (p->address) { + p->address =3D 0; + qemu_mutex_unlock(&p->mutex); + qemu_mutex_lock(&multifd_send_state->mutex); + p->done =3D true; + qemu_mutex_unlock(&multifd_send_state->mutex); + qemu_sem_post(&multifd_send_state->sem); + continue; + } qemu_mutex_unlock(&p->mutex); qemu_sem_wait(&p->sem); } @@ -471,6 +494,8 @@ int migrate_multifd_send_threads_create(void) multifd_send_state =3D g_malloc0(sizeof(*multifd_send_state)); multifd_send_state->params =3D g_new0(MultiFDSendParams, thread_count); multifd_send_state->count =3D 0; + qemu_mutex_init(&multifd_send_state->mutex); + qemu_sem_init(&multifd_send_state->sem, 0); for (i =3D 0; i < thread_count; i++) { char thread_name[15]; MultiFDSendParams *p =3D &multifd_send_state->params[i]; @@ -480,6 +505,8 @@ int migrate_multifd_send_threads_create(void) qemu_sem_init(&p->init, 0); p->quit =3D false; p->id =3D i; + p->done =3D true; + p->address =3D 0; p->c =3D socket_send_channel_create(); if (!p->c) { error_report("Error creating a send channel"); @@ -495,6 +522,30 @@ int migrate_multifd_send_threads_create(void) return 0; } =20 +static int multifd_send_page(uint8_t *address) +{ + int i; + MultiFDSendParams *p =3D NULL; /* make happy gcc */ + + qemu_sem_wait(&multifd_send_state->sem); + qemu_mutex_lock(&multifd_send_state->mutex); + for (i =3D 0; i < multifd_send_state->count; i++) { + p =3D &multifd_send_state->params[i]; + + if (p->done) { + p->done =3D false; + break; + } + } + qemu_mutex_unlock(&multifd_send_state->mutex); + qemu_mutex_lock(&p->mutex); + p->address =3D address; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); + + return 0; +} + struct MultiFDRecvParams { int id; QemuThread thread; @@ -1050,6 +1101,34 @@ static int ram_save_page(MigrationState *ms, QEMUFil= e *f, PageSearchStatus *pss, return pages; } =20 +static int ram_multifd_page(QEMUFile *f, PageSearchStatus *pss, + bool last_stage, uint64_t *bytes_transferred) +{ + int pages; + uint8_t *p; + RAMBlock *block =3D pss->block; + ram_addr_t offset =3D pss->offset; + + p =3D block->host + offset; + + if (block =3D=3D last_sent_block) { + offset |=3D RAM_SAVE_FLAG_CONTINUE; + } + pages =3D save_zero_page(f, block, offset, p, bytes_transferred); + if (pages =3D=3D -1) { + *bytes_transferred +=3D + save_page_header(f, block, offset | RAM_SAVE_FLAG_MULTIFD_PAGE= ); + qemu_put_buffer(f, p, TARGET_PAGE_SIZE); + multifd_send_page(p); + *bytes_transferred +=3D TARGET_PAGE_SIZE; + pages =3D 1; + acct_info.norm_pages++; + acct_info.multifd_pages++; + } + + return pages; +} + static int do_compress_ram_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) { @@ -1495,6 +1574,8 @@ static int ram_save_target_page(MigrationState *ms, Q= EMUFile *f, res =3D ram_save_compressed_page(ms, f, pss, last_stage, bytes_transferred); + } else if (migrate_use_multifd()) { + res =3D ram_multifd_page(f, pss, last_stage, bytes_transferred= ); } else { res =3D ram_save_page(ms, f, pss, last_stage, bytes_transferred); @@ -2765,6 +2846,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) if (!migrate_use_compression()) { invalid_flags |=3D RAM_SAVE_FLAG_COMPRESS_PAGE; } + + if (!migrate_use_multifd()) { + invalid_flags |=3D RAM_SAVE_FLAG_MULTIFD_PAGE; + } /* This RCU critical section can be very long running. * When RCU reclaims in the code start to become numerous, * it will be necessary to reduce the granularity of this @@ -2789,13 +2874,17 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) if (flags & invalid_flags & RAM_SAVE_FLAG_COMPRESS_PAGE) { error_report("Received an unexpected compressed page"); } + if (flags & invalid_flags & RAM_SAVE_FLAG_MULTIFD_PAGE) { + error_report("Received an unexpected multifd page"); + } =20 ret =3D -EINVAL; break; } =20 if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE | - RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { + RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE | + RAM_SAVE_FLAG_MULTIFD_PAGE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 host =3D host_from_ram_block_offset(block, addr); @@ -2882,6 +2971,11 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) break; } break; + + case RAM_SAVE_FLAG_MULTIFD_PAGE: + qemu_get_buffer(f, host, TARGET_PAGE_SIZE); + break; + case RAM_SAVE_FLAG_EOS: /* normal exit */ break; diff --git a/qapi-schema.json b/qapi-schema.json index 33a6267..0286b75 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -574,6 +574,7 @@ # # @postcopy-requests: The number of page requests received from the destin= ation # (since 2.7) +# @multifd: number of pages sent with multifd (since 2.9) # # Since: 0.14.0 ## @@ -582,7 +583,8 @@ 'duplicate': 'int', 'skipped': 'int', 'normal': 'int', 'normal-bytes': 'int', 'dirty-pages-rate' : 'int', 'mbps' : 'number', 'dirty-sync-count' : 'int', - 'postcopy-requests' : 'int' } } + 'postcopy-requests' : 'int', + 'multifd' : 'int'} } =20 ## # @XBZRLECacheStats: --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409358259592.7769215100649; Mon, 13 Mar 2017 05:49:18 -0700 (PDT) Received: from localhost ([::1]:51902 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPPW-0007By-RD for importer@patchew.org; Mon, 13 Mar 2017 08:49:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56511) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLM-0004R0-CG for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLL-0005x8-Ds for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:56 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44168) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLL-0005wf-6W for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:55 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 49B9D8123F for ; Mon, 13 Mar 2017 12:44:55 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ3012445; Mon, 13 Mar 2017 08:44:53 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:29 +0100 Message-Id: <20170313124434.1043-12-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Mon, 13 Mar 2017 12:44:55 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 11/16] migration: Really use multiple pages at a time X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We now send several pages at a time each time that we wakeup a thread. Signed-off-by: Juan Quintela -- Use iovec's insead of creating the equivalent. Signed-off-by: Juan Quintela --- migration/ram.c | 46 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index ccd7fe9..4914240 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -391,6 +391,13 @@ void migrate_compress_threads_create(void) =20 /* Multiple fd's */ =20 + +typedef struct { + int num; + int size; + struct iovec *iov; +} multifd_pages_t; + struct MultiFDSendParams { /* not changed */ int id; @@ -401,7 +408,7 @@ struct MultiFDSendParams { QemuMutex mutex; /* protected by param mutex */ bool quit; - uint8_t *address; + multifd_pages_t pages; /* protected by multifd mutex */ bool done; }; @@ -467,8 +474,8 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_unlock(&p->mutex); break; } - if (p->address) { - p->address =3D 0; + if (p->pages.num) { + p->pages.num =3D 0; qemu_mutex_unlock(&p->mutex); qemu_mutex_lock(&multifd_send_state->mutex); p->done =3D true; @@ -483,6 +490,13 @@ static void *multifd_send_thread(void *opaque) return NULL; } =20 +static void multifd_init_group(multifd_pages_t *pages) +{ + pages->num =3D 0; + pages->size =3D migrate_multifd_group(); + pages->iov =3D g_malloc0(pages->size * sizeof(struct iovec)); +} + int migrate_multifd_send_threads_create(void) { int i, thread_count; @@ -506,7 +520,7 @@ int migrate_multifd_send_threads_create(void) p->quit =3D false; p->id =3D i; p->done =3D true; - p->address =3D 0; + multifd_init_group(&p->pages); p->c =3D socket_send_channel_create(); if (!p->c) { error_report("Error creating a send channel"); @@ -524,8 +538,23 @@ int migrate_multifd_send_threads_create(void) =20 static int multifd_send_page(uint8_t *address) { - int i; + int i, j; MultiFDSendParams *p =3D NULL; /* make happy gcc */ + static multifd_pages_t pages; + static bool once; + + if (!once) { + multifd_init_group(&pages); + once =3D true; + } + + pages.iov[pages.num].iov_base =3D address; + pages.iov[pages.num].iov_len =3D TARGET_PAGE_SIZE; + pages.num++; + + if (pages.num < (pages.size - 1)) { + return UINT16_MAX; + } =20 qemu_sem_wait(&multifd_send_state->sem); qemu_mutex_lock(&multifd_send_state->mutex); @@ -539,7 +568,12 @@ static int multifd_send_page(uint8_t *address) } qemu_mutex_unlock(&multifd_send_state->mutex); qemu_mutex_lock(&p->mutex); - p->address =3D address; + p->pages.num =3D pages.num; + for (j =3D 0; j < pages.size; j++) { + p->pages.iov[j].iov_base =3D pages.iov[j].iov_base; + p->pages.iov[j].iov_len =3D pages.iov[j].iov_len; + } + pages.num =3D 0; qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem); =20 --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409572995805.2213711053308; Mon, 13 Mar 2017 05:52:52 -0700 (PDT) Received: from localhost ([::1]:51920 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPT0-0001t5-Pj for importer@patchew.org; Mon, 13 Mar 2017 08:52:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56540) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLP-0004Ts-L7 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLM-0005xy-TU for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:59 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46238) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLM-0005xZ-NT for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:56 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D49C04DD44 for ; Mon, 13 Mar 2017 12:44:56 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ4012445; Mon, 13 Mar 2017 08:44:55 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:30 +0100 Message-Id: <20170313124434.1043-13-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Mon, 13 Mar 2017 12:44:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 12/16] migration: Send the fd number which we are going to use for this page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We are still sending the page through the main channel, that would change later in the series Signed-off-by: Juan Quintela --- migration/ram.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 4914240..6f5ca50 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -577,7 +577,7 @@ static int multifd_send_page(uint8_t *address) qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->sem); =20 - return 0; + return i; } =20 struct MultiFDRecvParams { @@ -1139,6 +1139,7 @@ static int ram_multifd_page(QEMUFile *f, PageSearchSt= atus *pss, bool last_stage, uint64_t *bytes_transferred) { int pages; + uint16_t fd_num; uint8_t *p; RAMBlock *block =3D pss->block; ram_addr_t offset =3D pss->offset; @@ -1152,8 +1153,10 @@ static int ram_multifd_page(QEMUFile *f, PageSearchS= tatus *pss, if (pages =3D=3D -1) { *bytes_transferred +=3D save_page_header(f, block, offset | RAM_SAVE_FLAG_MULTIFD_PAGE= ); + fd_num =3D multifd_send_page(p); + qemu_put_be16(f, fd_num); + *bytes_transferred +=3D 2; /* size of fd_num */ qemu_put_buffer(f, p, TARGET_PAGE_SIZE); - multifd_send_page(p); *bytes_transferred +=3D TARGET_PAGE_SIZE; pages =3D 1; acct_info.norm_pages++; @@ -2898,6 +2901,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; void *host =3D NULL; + uint16_t fd_num; uint8_t ch; =20 addr =3D qemu_get_be64(f); @@ -3007,6 +3011,11 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) break; =20 case RAM_SAVE_FLAG_MULTIFD_PAGE: + fd_num =3D qemu_get_be16(f); + if (fd_num !=3D 0) { + /* this is yet an unused variable, changed later */ + fd_num =3D fd_num; + } qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break; =20 --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 148940970507030.15207659261182; Mon, 13 Mar 2017 05:55:05 -0700 (PDT) Received: from localhost ([::1]:51956 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPV7-0003rz-KT for importer@patchew.org; Mon, 13 Mar 2017 08:55:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56541) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLP-0004Tt-LZ for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLO-0005ym-Ip for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:59 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33634) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLO-0005yN-AB for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:44:58 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6DA4761BB9 for ; Mon, 13 Mar 2017 12:44:58 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ5012445; Mon, 13 Mar 2017 08:44:57 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:31 +0100 Message-Id: <20170313124434.1043-14-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 13 Mar 2017 12:44:58 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 13/16] migration: Create thread infrastructure for multifd recv side X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We make the locking and the transfer of information specific, even if we are still receiving things through the main thread. Signed-off-by: Juan Quintela --- migration/ram.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++---= ---- 1 file changed, 60 insertions(+), 8 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 6f5ca50..3b1a2dc 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -44,6 +44,7 @@ #include "exec/ram_addr.h" #include "qemu/rcu_queue.h" #include "migration/colo.h" +#include "qemu/iov.h" =20 static int dirty_rate_high_cnt; =20 @@ -536,7 +537,7 @@ int migrate_multifd_send_threads_create(void) return 0; } =20 -static int multifd_send_page(uint8_t *address) +static uint16_t multifd_send_page(uint8_t *address, bool last_page) { int i, j; MultiFDSendParams *p =3D NULL; /* make happy gcc */ @@ -552,8 +553,10 @@ static int multifd_send_page(uint8_t *address) pages.iov[pages.num].iov_len =3D TARGET_PAGE_SIZE; pages.num++; =20 - if (pages.num < (pages.size - 1)) { - return UINT16_MAX; + if (!last_page) { + if (pages.num < (pages.size - 1)) { + return UINT16_MAX; + } } =20 qemu_sem_wait(&multifd_send_state->sem); @@ -581,13 +584,18 @@ static int multifd_send_page(uint8_t *address) } =20 struct MultiFDRecvParams { + /* not changed */ int id; QemuThread thread; QIOChannel *c; QemuSemaphore init; + QemuSemaphore ready; QemuSemaphore sem; QemuMutex mutex; + /* proteced by param mutex */ bool quit; + multifd_pages_t pages; + bool done; }; typedef struct MultiFDRecvParams MultiFDRecvParams; =20 @@ -641,6 +649,7 @@ static void *multifd_recv_thread(void *opaque) =20 qio_channel_read(p->c, &start, 1, &error_abort); qemu_sem_post(&p->init); + qemu_sem_post(&p->ready); =20 while (true) { qemu_mutex_lock(&p->mutex); @@ -648,6 +657,13 @@ static void *multifd_recv_thread(void *opaque) qemu_mutex_unlock(&p->mutex); break; } + if (p->pages.num) { + p->pages.num =3D 0; + p->done =3D true; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->ready); + continue; + } qemu_mutex_unlock(&p->mutex); qemu_sem_wait(&p->sem); } @@ -672,8 +688,11 @@ int migrate_multifd_recv_threads_create(void) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->init, 0); + qemu_sem_init(&p->ready, 0); p->quit =3D false; p->id =3D i; + p->done =3D false; + multifd_init_group(&p->pages); p->c =3D socket_recv_channel_create(); =20 if (!p->c) { @@ -690,6 +709,42 @@ int migrate_multifd_recv_threads_create(void) return 0; } =20 +static void multifd_recv_page(uint8_t *address, uint16_t fd_num) +{ + int thread_count; + MultiFDRecvParams *p; + static multifd_pages_t pages; + static bool once; + + if (!once) { + multifd_init_group(&pages); + once =3D true; + } + + pages.iov[pages.num].iov_base =3D address; + pages.iov[pages.num].iov_len =3D TARGET_PAGE_SIZE; + pages.num++; + + if (fd_num =3D=3D UINT16_MAX) { + return; + } + + thread_count =3D migrate_multifd_threads(); + assert(fd_num < thread_count); + p =3D &multifd_recv_state->params[fd_num]; + + qemu_sem_wait(&p->ready); + + qemu_mutex_lock(&p->mutex); + p->done =3D false; + iov_copy(p->pages.iov, pages.num, pages.iov, pages.num, 0, + iov_size(pages.iov, pages.num)); + p->pages.num =3D pages.num; + pages.num =3D 0; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); +} + /** * save_page_header: Write page header to wire * @@ -1153,7 +1208,7 @@ static int ram_multifd_page(QEMUFile *f, PageSearchSt= atus *pss, if (pages =3D=3D -1) { *bytes_transferred +=3D save_page_header(f, block, offset | RAM_SAVE_FLAG_MULTIFD_PAGE= ); - fd_num =3D multifd_send_page(p); + fd_num =3D multifd_send_page(p, migration_dirty_pages =3D=3D 1); qemu_put_be16(f, fd_num); *bytes_transferred +=3D 2; /* size of fd_num */ qemu_put_buffer(f, p, TARGET_PAGE_SIZE); @@ -3012,10 +3067,7 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) =20 case RAM_SAVE_FLAG_MULTIFD_PAGE: fd_num =3D qemu_get_be16(f); - if (fd_num !=3D 0) { - /* this is yet an unused variable, changed later */ - fd_num =3D fd_num; - } + multifd_recv_page(host, fd_num); qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break; =20 --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409705019508.97010787173303; Mon, 13 Mar 2017 05:55:05 -0700 (PDT) Received: from localhost ([::1]:51955 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPV5-0003oa-Pu for importer@patchew.org; Mon, 13 Mar 2017 08:54:59 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56572) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLR-0004Vg-6t for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLQ-0005zb-5d for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:01 -0400 Received: from mx1.redhat.com ([209.132.183.28]:39398) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLP-0005z5-TU for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:00 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 075443B716 for ; Mon, 13 Mar 2017 12:45:00 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ6012445; Mon, 13 Mar 2017 08:44:58 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:32 +0100 Message-Id: <20170313124434.1043-15-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 13 Mar 2017 12:45:00 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 14/16] migration: Test new fd infrastructure X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We just send the address through the alternate channels and test that it is ok. Signed-off-by: Juan Quintela --- migration/ram.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 3b1a2dc..32cc678 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -476,8 +476,26 @@ static void *multifd_send_thread(void *opaque) break; } if (p->pages.num) { + int i; + int num; + + num =3D p->pages.num; p->pages.num =3D 0; qemu_mutex_unlock(&p->mutex); + + for (i =3D 0; i < num; i++) { + if (qio_channel_write(p->c, + (const char *)&p->pages.iov[i].iov_b= ase, + sizeof(uint8_t *), &error_abort) + !=3D sizeof(uint8_t *)) { + MigrationState *s =3D migrate_get_current(); + + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + terminate_multifd_send_threads(); + return NULL; + } + } qemu_mutex_lock(&multifd_send_state->mutex); p->done =3D true; qemu_mutex_unlock(&multifd_send_state->mutex); @@ -645,6 +663,7 @@ void migrate_multifd_recv_threads_join(void) static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p =3D opaque; + uint8_t *recv_address; char start; =20 qio_channel_read(p->c, &start, 1, &error_abort); @@ -658,7 +677,38 @@ static void *multifd_recv_thread(void *opaque) break; } if (p->pages.num) { + int i; + int num; + + num =3D p->pages.num; p->pages.num =3D 0; + + for (i =3D 0; i < num; i++) { + if (qio_channel_read(p->c, + (char *)&recv_address, + sizeof(uint8_t *), &error_abort) + !=3D sizeof(uint8_t *)) { + MigrationState *s =3D migrate_get_current(); + + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + terminate_multifd_recv_threads(); + return NULL; + } + if (recv_address !=3D p->pages.iov[i].iov_base) { + MigrationState *s =3D migrate_get_current(); + + printf("We received %p what we were expecting %p (%d)\= n", + recv_address, + p->pages.iov[i].iov_base, i); + + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + terminate_multifd_recv_threads(); + return NULL; + } + } + p->done =3D true; qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->ready); --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409891522904.7797280128474; Mon, 13 Mar 2017 05:58:11 -0700 (PDT) Received: from localhost ([::1]:51992 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPY5-0006JU-7I for importer@patchew.org; Mon, 13 Mar 2017 08:58:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56639) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLX-0004eF-MK for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLR-00060N-Lh for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59266) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLR-0005zy-DD for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:01 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8649C1556C for ; Mon, 13 Mar 2017 12:45:01 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ7012445; Mon, 13 Mar 2017 08:45:00 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:33 +0100 Message-Id: <20170313124434.1043-16-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 13 Mar 2017 12:45:01 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 15/16] migration: Transfer pages over new channels X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We switch for sending the page number to send real pages. Signed-off-by: Juan Quintela -- Remove the HACK bit, now we have the function that calculates the size of a page exported. Signed-off-by: Juan Quintela --- migration/migration.c | 14 ++++++++---- migration/ram.c | 59 +++++++++++++++++------------------------------= ---- 2 files changed, 29 insertions(+), 44 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index a32e4ad..18d6955 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1967,7 +1967,8 @@ static void *migration_thread(void *opaque) /* Used by the bandwidth calcs, updated later */ int64_t initial_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); int64_t setup_start =3D qemu_clock_get_ms(QEMU_CLOCK_HOST); - int64_t initial_bytes =3D 0; + int64_t qemu_file_bytes =3D 0; + int64_t multifd_pages =3D 0; int64_t max_size =3D 0; int64_t start_time =3D initial_time; int64_t end_time; @@ -2051,9 +2052,13 @@ static void *migration_thread(void *opaque) } current_time =3D qemu_clock_get_ms(QEMU_CLOCK_REALTIME); if (current_time >=3D initial_time + BUFFER_DELAY) { - uint64_t transferred_bytes =3D qemu_ftell(s->to_dst_file) - - initial_bytes; uint64_t time_spent =3D current_time - initial_time; + uint64_t qemu_file_bytes_now =3D qemu_ftell(s->to_dst_file); + uint64_t multifd_pages_now =3D multifd_mig_pages_transferred(); + uint64_t transferred_bytes =3D + (qemu_file_bytes_now - qemu_file_bytes) + + (multifd_pages_now - multifd_pages) * + (1ul << qemu_target_page_bits()); double bandwidth =3D (double)transferred_bytes / time_spent; max_size =3D bandwidth * s->parameters.downtime_limit; =20 @@ -2070,7 +2075,8 @@ static void *migration_thread(void *opaque) =20 qemu_file_reset_rate_limit(s->to_dst_file); initial_time =3D current_time; - initial_bytes =3D qemu_ftell(s->to_dst_file); + qemu_file_bytes =3D qemu_file_bytes_now; + multifd_pages =3D multifd_pages_now; } if (qemu_file_rate_limit(s->to_dst_file)) { /* usleep expects microseconds */ diff --git a/migration/ram.c b/migration/ram.c index 32cc678..e213a49 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -476,25 +476,21 @@ static void *multifd_send_thread(void *opaque) break; } if (p->pages.num) { - int i; int num; =20 num =3D p->pages.num; p->pages.num =3D 0; qemu_mutex_unlock(&p->mutex); =20 - for (i =3D 0; i < num; i++) { - if (qio_channel_write(p->c, - (const char *)&p->pages.iov[i].iov_b= ase, - sizeof(uint8_t *), &error_abort) - !=3D sizeof(uint8_t *)) { - MigrationState *s =3D migrate_get_current(); + if (qio_channel_writev_all(p->c, p->pages.iov, + num, &error_abort) + !=3D num * TARGET_PAGE_SIZE) { + MigrationState *s =3D migrate_get_current(); =20 - migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, - MIGRATION_STATUS_FAILED); - terminate_multifd_send_threads(); - return NULL; - } + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + terminate_multifd_send_threads(); + return NULL; } qemu_mutex_lock(&multifd_send_state->mutex); p->done =3D true; @@ -663,7 +659,6 @@ void migrate_multifd_recv_threads_join(void) static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p =3D opaque; - uint8_t *recv_address; char start; =20 qio_channel_read(p->c, &start, 1, &error_abort); @@ -677,38 +672,21 @@ static void *multifd_recv_thread(void *opaque) break; } if (p->pages.num) { - int i; int num; =20 num =3D p->pages.num; p->pages.num =3D 0; =20 - for (i =3D 0; i < num; i++) { - if (qio_channel_read(p->c, - (char *)&recv_address, - sizeof(uint8_t *), &error_abort) - !=3D sizeof(uint8_t *)) { - MigrationState *s =3D migrate_get_current(); + if (qio_channel_readv_all(p->c, p->pages.iov, + num, &error_abort) + !=3D num * TARGET_PAGE_SIZE) { + MigrationState *s =3D migrate_get_current(); =20 - migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, - MIGRATION_STATUS_FAILED); - terminate_multifd_recv_threads(); - return NULL; - } - if (recv_address !=3D p->pages.iov[i].iov_base) { - MigrationState *s =3D migrate_get_current(); - - printf("We received %p what we were expecting %p (%d)\= n", - recv_address, - p->pages.iov[i].iov_base, i); - - migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, - MIGRATION_STATUS_FAILED); - terminate_multifd_recv_threads(); - return NULL; - } + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + terminate_multifd_recv_threads(); + return NULL; } - p->done =3D true; qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->ready); @@ -1260,8 +1238,10 @@ static int ram_multifd_page(QEMUFile *f, PageSearchS= tatus *pss, save_page_header(f, block, offset | RAM_SAVE_FLAG_MULTIFD_PAGE= ); fd_num =3D multifd_send_page(p, migration_dirty_pages =3D=3D 1); qemu_put_be16(f, fd_num); + if (fd_num !=3D UINT16_MAX) { + qemu_fflush(f); + } *bytes_transferred +=3D 2; /* size of fd_num */ - qemu_put_buffer(f, p, TARGET_PAGE_SIZE); *bytes_transferred +=3D TARGET_PAGE_SIZE; pages =3D 1; acct_info.norm_pages++; @@ -3118,7 +3098,6 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) case RAM_SAVE_FLAG_MULTIFD_PAGE: fd_num =3D qemu_get_be16(f); multifd_recv_page(host, fd_num); - qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break; =20 case RAM_SAVE_FLAG_EOS: --=20 2.9.3 From nobody Thu Nov 6 14:25:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1489409800077367.9679456123208; Mon, 13 Mar 2017 05:56:40 -0700 (PDT) Received: from localhost ([::1]:51971 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPWc-0005Eu-Tk for importer@patchew.org; Mon, 13 Mar 2017 08:56:34 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56608) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cnPLU-0004ZR-Ac for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cnPLT-00060k-8g for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38268) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cnPLT-00060Y-07 for qemu-devel@nongnu.org; Mon, 13 Mar 2017 08:45:03 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1DA566AAC7 for ; Mon, 13 Mar 2017 12:45:03 +0000 (UTC) Received: from secure.mitica (ovpn-117-36.ams2.redhat.com [10.36.117.36]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2DCiaQ8012445; Mon, 13 Mar 2017 08:45:01 -0400 From: Juan Quintela To: qemu-devel@nongnu.org Date: Mon, 13 Mar 2017 13:44:34 +0100 Message-Id: <20170313124434.1043-17-quintela@redhat.com> In-Reply-To: <20170313124434.1043-1-quintela@redhat.com> References: <20170313124434.1043-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 13 Mar 2017 12:45:03 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 16/16] migration: Flush receive queue X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amit.shah@redhat.com, dgilbert@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Each time that we sync the bitmap, it is a possiblity that we receive a page that is being processed by a different thread. We fix this problem just making sure that we wait for all receiving threads to finish its work before we procedeed with the next stage. We are low on page flags, so we use a combination that is not valid to emit that message: MULTIFD_PAGE and COMPRESSED. I tried to make a migration command for it, but it don't work because we sync the bitmap sometimes when we have already sent the beggining of the section, so I just added a new page flag. Signed-off-by: Juan Quintela --- include/migration/migration.h | 1 + migration/ram.c | 57 +++++++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 58 insertions(+) diff --git a/include/migration/migration.h b/include/migration/migration.h index bd152c5..86023ff 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -278,6 +278,7 @@ int migrate_multifd_send_threads_create(void); void migrate_multifd_send_threads_join(void); int migrate_multifd_recv_threads_create(void); void migrate_multifd_recv_threads_join(void); +void qemu_savevm_send_multifd_flush(QEMUFile *f); =20 void migrate_compress_threads_create(void); void migrate_compress_threads_join(void); diff --git a/migration/ram.c b/migration/ram.c index e213a49..a93c4ae 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -64,6 +64,13 @@ static uint64_t bitmap_sync_count; #define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100 #define RAM_SAVE_FLAG_MULTIFD_PAGE 0x200 =20 +/* We are getting low on pages flags, so we start using combinations + When we need to flush a page, we sent it as + RAM_SAVE_FLAG_MULTIFD_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE + We don't allow that combination +*/ + + static uint8_t *ZERO_TARGET_PAGE; =20 static inline bool is_zero_range(uint8_t *p, uint64_t size) @@ -392,6 +399,9 @@ void migrate_compress_threads_create(void) =20 /* Multiple fd's */ =20 +/* Indicates if we have synced the bitmap and we need to assure that + target has processeed all previous pages */ +bool multifd_needs_flush; =20 typedef struct { int num; @@ -605,9 +615,11 @@ struct MultiFDRecvParams { QemuSemaphore init; QemuSemaphore ready; QemuSemaphore sem; + QemuCond cond_sync; QemuMutex mutex; /* proteced by param mutex */ bool quit; + bool sync; multifd_pages_t pages; bool done; }; @@ -648,6 +660,7 @@ void migrate_multifd_recv_threads_join(void) qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); qemu_sem_destroy(&p->init); + qemu_cond_destroy(&p->cond_sync); socket_send_channel_destroy(multifd_recv_state->params[i].c); } g_free(multifd_recv_state->params); @@ -688,6 +701,10 @@ static void *multifd_recv_thread(void *opaque) return NULL; } p->done =3D true; + if (p->sync) { + qemu_cond_signal(&p->cond_sync); + p->sync =3D false; + } qemu_mutex_unlock(&p->mutex); qemu_sem_post(&p->ready); continue; @@ -717,9 +734,11 @@ int migrate_multifd_recv_threads_create(void) qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->init, 0); qemu_sem_init(&p->ready, 0); + qemu_cond_init(&p->cond_sync); p->quit =3D false; p->id =3D i; p->done =3D false; + p->sync =3D false; multifd_init_group(&p->pages); p->c =3D socket_recv_channel_create(); =20 @@ -773,6 +792,27 @@ static void multifd_recv_page(uint8_t *address, uint16= _t fd_num) qemu_sem_post(&p->sem); } =20 +static int multifd_flush(void) +{ + int i, thread_count; + + if (!migrate_use_multifd()) { + return 0; + } + thread_count =3D migrate_multifd_threads(); + for (i =3D 0; i < thread_count; i++) { + MultiFDRecvParams *p =3D &multifd_recv_state->params[i]; + + qemu_mutex_lock(&p->mutex); + while (!p->done) { + p->sync =3D true; + qemu_cond_wait(&p->cond_sync, &p->mutex); + } + qemu_mutex_unlock(&p->mutex); + } + return 0; +} + /** * save_page_header: Write page header to wire * @@ -789,6 +829,12 @@ static size_t save_page_header(QEMUFile *f, RAMBlock *= block, ram_addr_t offset) { size_t size, len; =20 + if (multifd_needs_flush && + (offset & RAM_SAVE_FLAG_MULTIFD_PAGE)) { + offset |=3D RAM_SAVE_FLAG_COMPRESS; + multifd_needs_flush =3D false; + } + qemu_put_be64(f, offset); size =3D 8; =20 @@ -2526,6 +2572,9 @@ static int ram_save_complete(QEMUFile *f, void *opaqu= e) =20 if (!migration_in_postcopy(migrate_get_current())) { migration_bitmap_sync(); + if (migrate_use_multifd()) { + multifd_needs_flush =3D true; + } } =20 ram_control_before_iterate(f, RAM_CONTROL_FINISH); @@ -2567,6 +2616,9 @@ static void ram_save_pending(QEMUFile *f, void *opaqu= e, uint64_t max_size, qemu_mutex_lock_iothread(); rcu_read_lock(); migration_bitmap_sync(); + if (migrate_use_multifd()) { + multifd_needs_flush =3D true; + } rcu_read_unlock(); qemu_mutex_unlock_iothread(); remaining_size =3D ram_save_remaining() * TARGET_PAGE_SIZE; @@ -3005,6 +3057,11 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) break; } =20 + if ((flags & (RAM_SAVE_FLAG_MULTIFD_PAGE | RAM_SAVE_FLAG_COMPRESS)) + =3D=3D (RAM_SAVE_FLAG_MULTIFD_PAGE | RAM_SAVE_FLAG_COMPR= ESS)) { + multifd_flush(); + flags =3D flags & ~RAM_SAVE_FLAG_COMPRESS; + } if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE | RAM_SAVE_FLAG_MULTIFD_PAGE)) { --=20 2.9.3