From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850246636563.7167379541489; Sat, 22 Apr 2017 01:37:26 -0700 (PDT) Received: from localhost ([::1]:34734 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qXl-0001G0-7D for importer@patchew.org; Sat, 22 Apr 2017 04:37:25 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59875) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNZ-0001qj-TD for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNW-0000jo-PZ for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:53 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3402 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNV-0000jG-T2 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:50 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71123; Sat, 22 Apr 2017 16:26:27 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:16 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:41 +0800 Message-ID: <1492849558-17540-2-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090201.58FB13B4.0008, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 339b3fca4e9c846839fa64341d32c787 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 01/18] net/colo: Add notifier/callback related helpers for filter X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We will use this notifier to help COLO to notify filter object to do something, like do checkpoint, or process failover event. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Signed-off-by: Li Zhijian --- net/colo.c | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++ net/colo.h | 19 +++++++++++ 2 files changed, 124 insertions(+) diff --git a/net/colo.c b/net/colo.c index 8cc166b..8aef670 100644 --- a/net/colo.c +++ b/net/colo.c @@ -15,6 +15,7 @@ #include "qemu/osdep.h" #include "trace.h" #include "net/colo.h" +#include "qapi/error.h" =20 uint32_t connection_key_hash(const void *opaque) { @@ -209,3 +210,107 @@ Connection *connection_get(GHashTable *connection_tra= ck_table, =20 return conn; } + +static gboolean +filter_notify_prepare(GSource *source, gint *timeout) +{ + *timeout =3D -1; + + return FALSE; +} + +static gboolean +filter_notify_check(GSource *source) +{ + FilterNotifier *notify =3D (FilterNotifier *)source; + + return notify->pfd.revents & (G_IO_IN | G_IO_HUP | G_IO_ERR); +} + +static gboolean +filter_notify_dispatch(GSource *source, + GSourceFunc callback, + gpointer user_data) +{ + FilterNotifier *notify =3D (FilterNotifier *)source; + int revents; + uint64_t value; + int ret; + + revents =3D notify->pfd.revents & notify->pfd.events; + if (revents & (G_IO_IN | G_IO_HUP | G_IO_ERR)) { + ret =3D filter_notifier_get(notify, &value); + if (notify->cb && !ret) { + notify->cb(notify, value); + } + } + return TRUE; +} + +static void +filter_notify_finalize(GSource *source) +{ + FilterNotifier *notify =3D (FilterNotifier *)source; + + event_notifier_cleanup(¬ify->event); +} + +static GSourceFuncs notifier_source_funcs =3D { + filter_notify_prepare, + filter_notify_check, + filter_notify_dispatch, + filter_notify_finalize, +}; + +FilterNotifier *filter_notifier_new(FilterNotifierCallback *cb, + void *opaque, Error **errp) +{ + FilterNotifier *notify; + int ret; + + notify =3D (FilterNotifier *)g_source_new(¬ifier_source_funcs, + sizeof(FilterNotifier)); + ret =3D event_notifier_init(¬ify->event, false); + if (ret < 0) { + error_setg_errno(errp, -ret, "Failed to initialize event notifier"= ); + goto fail; + } + notify->pfd.fd =3D event_notifier_get_fd(¬ify->event); + notify->pfd.events =3D G_IO_IN | G_IO_HUP | G_IO_ERR; + notify->cb =3D cb; + notify->opaque =3D opaque; + g_source_add_poll(¬ify->source, ¬ify->pfd); + + return notify; + +fail: + g_source_destroy(¬ify->source); + return NULL; +} + +int filter_notifier_set(FilterNotifier *notify, uint64_t value) +{ + ssize_t ret; + + do { + ret =3D write(notify->event.wfd, &value, sizeof(value)); + } while (ret < 0 && errno =3D=3D EINTR); + + /* EAGAIN is fine, a read must be pending. */ + if (ret < 0 && errno !=3D EAGAIN) { + return -errno; + } + return 0; +} + +int filter_notifier_get(FilterNotifier *notify, uint64_t *value) +{ + ssize_t len; + + /* Drain the notify pipe. For eventfd, only 8 bytes will be read. */ + do { + len =3D read(notify->event.rfd, value, sizeof(*value)); + } while ((len =3D=3D -1 && errno =3D=3D EINTR)); + + return len !=3D sizeof(*value) ? -1 : 0; +} diff --git a/net/colo.h b/net/colo.h index cd9027f..b586db3 100644 --- a/net/colo.h +++ b/net/colo.h @@ -19,6 +19,7 @@ #include "qemu/jhash.h" #include "qemu/timer.h" #include "slirp/tcp.h" +#include "qemu/event_notifier.h" =20 #define HASHTABLE_MAX_SIZE 16384 =20 @@ -89,4 +90,22 @@ void connection_hashtable_reset(GHashTable *connection_t= rack_table); Packet *packet_new(const void *data, int size); void packet_destroy(void *opaque, void *user_data); =20 +typedef void FilterNotifierCallback(void *opaque, int value); +typedef struct FilterNotifier { + GSource source; + EventNotifier event; + GPollFD pfd; + FilterNotifierCallback *cb; + void *opaque; +} FilterNotifier; + +FilterNotifier *filter_notifier_new(FilterNotifierCallback *cb, + void *opaque, Error **errp); +int filter_notifier_set(FilterNotifier *notify, uint64_t value); +int filter_notifier_get(FilterNotifier *notify, uint64_t *value); + +enum { + COLO_CHECKPOINT =3D 2, + COLO_FAILOVER, +}; #endif /* QEMU_COLO_PROXY_H */ --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850131754162.00036742801728; Sat, 22 Apr 2017 01:35:31 -0700 (PDT) Received: from localhost ([::1]:34727 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qVt-0008Jq-MK for importer@patchew.org; Sat, 22 Apr 2017 04:35:29 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59874) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNZ-0001qi-T1 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNY-0000k0-EU for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:53 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3403 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNX-0000jT-I5 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:52 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71115; Sat, 22 Apr 2017 16:26:26 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:18 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:42 +0800 Message-ID: <1492849558-17540-3-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090206.58FB13B4.0024, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 17aec0c49a4bec067be8602fdd9f8988 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 02/18] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo-compare.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++= ++++ net/colo-compare.h | 6 +++++ 2 files changed, 84 insertions(+) create mode 100644 net/colo-compare.h diff --git a/net/colo-compare.c b/net/colo-compare.c index 97bf0e5..3adccfb 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,17 +29,24 @@ #include "qemu/sockets.h" #include "qapi-visit.h" #include "net/colo.h" +#include "net/colo-compare.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) =20 +static QTAILQ_HEAD(, CompareState) net_compares =3D + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 =20 +static QemuMutex event_mtx =3D { .lock =3D PTHREAD_MUTEX_INITIALIZER }; +static QemuCond event_complete_cond =3D { .cond =3D PTHREAD_COND_INITIALIZ= ER }; +static int event_unhandled_count; /* + CompareState ++ | | @@ -87,6 +94,10 @@ typedef struct CompareState { =20 GMainContext *worker_context; GMainLoop *compare_loop; + /* Used for COLO to notify compare to do something */ + FilterNotifier *notifier; + + QTAILQ_ENTRY(CompareState) next; } CompareState; =20 typedef struct CompareClass { @@ -417,6 +428,11 @@ static void colo_compare_connection(void *opaque, void= *user_data) while (!g_queue_is_empty(&conn->primary_list) && !g_queue_is_empty(&conn->secondary_list)) { pkt =3D g_queue_pop_tail(&conn->primary_list); + if (!pkt) { + error_report("colo-compare pop pkt failed"); + return; + } + switch (conn->ip_proto) { case IPPROTO_TCP: result =3D g_queue_find_custom(&conn->secondary_list, @@ -538,6 +554,53 @@ static gboolean check_old_packet_regular(void *opaque) return TRUE; } =20 +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + int ret; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + ret =3D filter_notifier_set(s->notifier, event); + if (ret < 0) { + error_setg_errno(errp, -ret, "Failed to write value to eventfd= "); + goto fail; + } + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + +fail: + qemu_mutex_unlock(&event_mtx); +} + +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque, int event) +{ + FilterNotifier *notify =3D opaque; + CompareState *s =3D notify->opaque; + + switch (event) { + case COLO_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void *colo_compare_thread(void *opaque) { CompareState *s =3D opaque; @@ -558,10 +621,15 @@ static void *colo_compare_thread(void *opaque) (GSourceFunc)check_old_packet_regular, s, NULL); g_source_attach(timeout_source, s->worker_context); =20 + s->notifier =3D filter_notifier_new(colo_compare_handle_event, s, NULL= ); + g_source_attach(&s->notifier->source, s->worker_context); + qemu_sem_post(&s->thread_ready); =20 g_main_loop_run(s->compare_loop); =20 + g_source_destroy(&s->notifier->source); + g_source_unref(&s->notifier->source); g_source_destroy(timeout_source); g_source_unref(timeout_source); =20 @@ -706,6 +774,8 @@ static void colo_compare_complete(UserCreatable *uc, Er= ror **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize); =20 + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); =20 s->connection_track_table =3D g_hash_table_new_full(connection_key_has= h, @@ -765,6 +835,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s =3D COLO_COMPARE(obj); + CompareState *tmp =3D NULL; =20 qemu_chr_fe_set_handlers(&s->chr_pri_in, NULL, NULL, NULL, NULL, s->worker_context, true); @@ -777,6 +848,13 @@ static void colo_compare_finalize(Object *obj) } qemu_thread_join(&s->thread); =20 + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); =20 diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..c9c62f5 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,6 @@ +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850689411121.10961427520635; Sat, 22 Apr 2017 01:44:49 -0700 (PDT) Received: from localhost ([::1]:34765 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qeu-0008TS-3d for importer@patchew.org; Sat, 22 Apr 2017 04:44:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60095) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qOH-0002TH-LY for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qOE-00013h-Hp for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:37 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3406 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qOD-00012z-VS for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:34 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71121; Sat, 22 Apr 2017 16:26:27 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:19 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:43 +0800 Message-ID: <1492849558-17540-4-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090205.58FB13B4.0003, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: a1a154054fcb0bf78a7cdd9340f03e13 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 03/18] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Cc: Jason Wang Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 ++++++++++++++++++++++++++++---- net/colo-compare.h | 2 ++ 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 3adccfb..bb234dd 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -30,6 +30,7 @@ #include "qapi-visit.h" #include "net/colo.h" #include "net/colo-compare.h" +#include "migration/migration.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -38,6 +39,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =3D QTAILQ_HEAD_INITIALIZER(net_compares); =20 +static NotifierList colo_compare_notifiers =3D + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -384,6 +388,22 @@ static int colo_old_packet_check_one(Packet *pkt, int6= 4_t *check_time) } } =20 +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static void colo_old_packet_check_one_conn(void *opaque, void *user_data) { @@ -397,7 +417,7 @@ static void colo_old_packet_check_one_conn(void *opaque, =20 if (result) { /* do checkpoint will flush old packet */ - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); } } =20 @@ -415,7 +435,10 @@ static void colo_old_packet_check(void *opaque) =20 /* * Called from the compare thread on the primary - * for compare connection + * for compare connection. + * TODO: Reconstruct this function, we should hold the max handled sequence + * of the connect, Don't trigger a checkpoint request if we only get packe= ts + * from one side (primary or secondary). */ static void colo_compare_connection(void *opaque, void *user_data) { @@ -464,11 +487,12 @@ static void colo_compare_connection(void *opaque, voi= d *user_data) /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_tail(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index c9c62f5..a0b573e 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -2,5 +2,7 @@ #define QEMU_COLO_COMPARE_H =20 void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); =20 #endif /* QEMU_COLO_COMPARE_H */ --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849721584970.5875455494031; Sat, 22 Apr 2017 01:28:41 -0700 (PDT) Received: from localhost ([::1]:34694 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qPI-0002xQ-4J for importer@patchew.org; Sat, 22 Apr 2017 04:28:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59786) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jK-37 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNO-0000hc-UK for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3399 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNO-0000gp-2J for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:42 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71118; Sat, 22 Apr 2017 16:26:26 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:19 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:44 +0800 Message-ID: <1492849558-17540-5-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090201.58FB13B4.001D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 1ca117d73de3f963d57429163b9e5f79 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 04/18] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Cc: Jason Wang Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 2 +- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index c19eb3f..a3344ce 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -21,8 +21,11 @@ #include "migration/failover.h" #include "replication.h" #include "qmp-commands.h" +#include "net/colo-compare.h" +#include "net/colo.h" =20 static bool vmstate_loading; +static Notifier packets_compare_notifier; =20 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) =20 @@ -332,6 +335,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + colo_notify_compares_event(NULL, COLO_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ s->params.blk =3D 0; s->params.shared =3D 0; @@ -390,6 +398,11 @@ out: return ret; } =20 +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -406,6 +419,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 + packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -451,11 +467,21 @@ out: qemu_fclose(fb); } =20 - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -548,6 +574,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -567,6 +598,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -620,6 +656,8 @@ void *colo_process_incoming_thread(void *opaque) } =20 vmstate_loading =3D false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); =20 if (failover_get_state() =3D=3D FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 353f272..2ade2aa 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -70,7 +70,7 @@ /* The delay time (in ms) between two COLO checkpoints * Note: Please change this default value to 10000 when we support hybrid = mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) =20 static NotifierList migration_state_notifiers =3D NOTIFIER_LIST_INITIALIZER(migration_state_notifiers); --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850684573832.1044203789253; Sat, 22 Apr 2017 01:44:44 -0700 (PDT) Received: from localhost ([::1]:34764 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qep-0008P3-2n for importer@patchew.org; Sat, 22 Apr 2017 04:44:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59947) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNq-000248-PL for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNp-0000vP-MT for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:10 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3405 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNo-0000uu-Qq for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:09 -0400 Received: from 172.30.72.54 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.54]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71133; Sat, 22 Apr 2017 16:26:32 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:20 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:45 +0800 Message-ID: <1492849558-17540-6-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090201.58FB13BC.0045, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 2e373c194345f165b783610d08a7f786 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 05/18] COLO: Handle shutdown command for VM in COLO state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If VM is in COLO FT state, we need to do some extra works before starting normal shutdown process. Secondary VM will ignore the shutdown command if users issue it directly to Secondary VM. COLO will capture shutdown command and after shutdown request from user. Cc: Paolo Bonzini Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- include/migration/colo.h | 1 + include/sysemu/sysemu.h | 3 +++ migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++= +- qapi-schema.json | 4 +++- vl.c | 19 ++++++++++++++++--- 5 files changed, 68 insertions(+), 5 deletions(-) diff --git a/include/migration/colo.h b/include/migration/colo.h index 2bbff9e..aadd040 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -37,4 +37,5 @@ COLOMode get_colo_mode(void); void colo_do_failover(MigrationState *s); =20 void colo_checkpoint_notify(void *opaque); +bool colo_handle_shutdown(void); #endif diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h index 16175f7..8054f53 100644 --- a/include/sysemu/sysemu.h +++ b/include/sysemu/sysemu.h @@ -49,6 +49,8 @@ typedef enum WakeupReason { QEMU_WAKEUP_REASON_OTHER, } WakeupReason; =20 +extern int colo_shutdown_requested; + void qemu_system_reset_request(void); void qemu_system_suspend_request(void); void qemu_register_suspend_notifier(Notifier *notifier); @@ -56,6 +58,7 @@ void qemu_system_wakeup_request(WakeupReason reason); void qemu_system_wakeup_enable(WakeupReason reason, bool enabled); void qemu_register_wakeup_notifier(Notifier *notifier); void qemu_system_shutdown_request(void); +void qemu_system_shutdown_request_core(void); void qemu_system_powerdown_request(void); void qemu_register_powerdown_notifier(Notifier *notifier); void qemu_system_debug_request(void); diff --git a/migration/colo.c b/migration/colo.c index a3344ce..c4fc865 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -384,6 +384,21 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + if (colo_shutdown_requested) { + colo_send_message(s->to_dst_file, COLO_MESSAGE_GUEST_SHUTDOWN, + &local_err); + if (local_err) { + error_free(local_err); + /* Go on the shutdown process and throw the error message */ + error_report("Failed to send shutdown message to SVM"); + } + qemu_fflush(s->to_dst_file); + colo_shutdown_requested =3D 0; + qemu_system_shutdown_request_core(); + /* Fix me: Just let the colo thread exit ? */ + qemu_thread_exit(0); + } + ret =3D 0; =20 qemu_mutex_lock_iothread(); @@ -449,7 +464,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 - qemu_sem_wait(&s->colo_checkpoint_sem); + if (!colo_shutdown_requested) { + qemu_sem_wait(&s->colo_checkpoint_sem); + } =20 ret =3D colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { @@ -534,6 +551,16 @@ static void colo_wait_handle_message(QEMUFile *f, int = *checkpoint_request, case COLO_MESSAGE_CHECKPOINT_REQUEST: *checkpoint_request =3D 1; break; + case COLO_MESSAGE_GUEST_SHUTDOWN: + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + qemu_system_shutdown_request_core(); + qemu_mutex_unlock_iothread(); + /* + * The main thread will be exit and terminate the whole + * process, do need some cleanup ? + */ + qemu_thread_exit(0); default: *checkpoint_request =3D 0; error_setg(errp, "Got unknown COLO message: %d", msg); @@ -696,3 +723,20 @@ out: =20 return NULL; } + +bool colo_handle_shutdown(void) +{ + /* + * If VM is in COLO-FT mode, we need do some significant work before + * respond to the shutdown request. Besides, Secondary VM will ignore + * the shutdown request from users. + */ + if (migration_incoming_in_colo_state()) { + return true; + } + if (migration_in_colo_state()) { + colo_shutdown_requested =3D 1; + return true; + } + return false; +} diff --git a/qapi-schema.json b/qapi-schema.json index 01b087f..4b3e1b7 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -1187,12 +1187,14 @@ # # @vmstate-loaded: VM's state has been loaded by SVM. # +# @guest-shutdown: shutdown requested from PVM to SVM. (Since 2.9) +# # Since: 2.8 ## { 'enum': 'COLOMessage', 'data': [ 'checkpoint-ready', 'checkpoint-request', 'checkpoint-reply', 'vmstate-send', 'vmstate-size', 'vmstate-received', - 'vmstate-loaded' ] } + 'vmstate-loaded', 'guest-shutdown' ] } =20 ## # @COLOMode: diff --git a/vl.c b/vl.c index 0b4ed52..72638c9 100644 --- a/vl.c +++ b/vl.c @@ -1611,6 +1611,8 @@ static NotifierList wakeup_notifiers =3D NOTIFIER_LIST_INITIALIZER(wakeup_notifiers); static uint32_t wakeup_reason_mask =3D ~(1 << QEMU_WAKEUP_REASON_NONE); =20 +int colo_shutdown_requested; + int qemu_shutdown_requested_get(void) { return shutdown_requested; @@ -1737,7 +1739,10 @@ void qemu_system_guest_panicked(GuestPanicInformatio= n *info) void qemu_system_reset_request(void) { if (no_reboot) { - shutdown_requested =3D 1; + qemu_system_shutdown_request(); + if (!shutdown_requested) {/* colo handle it ? */ + return; + } } else { reset_requested =3D 1; } @@ -1810,14 +1815,22 @@ void qemu_system_killed(int signal, pid_t pid) qemu_notify_event(); } =20 -void qemu_system_shutdown_request(void) +void qemu_system_shutdown_request_core(void) { - trace_qemu_system_shutdown_request(); replay_shutdown_request(); shutdown_requested =3D 1; qemu_notify_event(); } =20 +void qemu_system_shutdown_request(void) +{ + trace_qemu_system_shutdown_request(); + if (colo_handle_shutdown()) { + return; + } + qemu_system_shutdown_request_core(); +} + static void qemu_system_powerdown(void) { qapi_event_send_powerdown(&error_abort); --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849994182917.7065828676707; Sat, 22 Apr 2017 01:33:14 -0700 (PDT) Received: from localhost ([::1]:34714 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qTg-0006UF-NN for importer@patchew.org; Sat, 22 Apr 2017 04:33:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59798) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jN-Jj for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNQ-0000hv-0X for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3400 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNP-0000gr-4n for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:43 -0400 Received: from 172.30.72.54 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.54]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71131; Sat, 22 Apr 2017 16:26:32 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:21 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:46 +0800 Message-ID: <1492849558-17540-7-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090201.58FB13B8.0033, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 70a33e34cc7dbc6ff92406d65e97d8f7 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 06/18] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Xie Changlong , Max Reitz , Stefan Hajnoczi Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Cc: Stefan Hajnoczi Cc: Kevin Wolf Cc: Max Reitz Cc: Xie Changlong --- migration/colo.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++= +++ migration/migration.c | 16 ++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c4fc865..9949293 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -23,6 +23,9 @@ #include "qmp-commands.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "qapi-event.h" +#include "block/block.h" +#include "replication.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -57,6 +60,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; =20 /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -74,6 +78,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); =20 + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); /* recover runstate to normal migration finish state */ @@ -111,6 +120,7 @@ static void primary_vm_do_failover(void) { MigrationState *s =3D migrate_get_current(); int old_state; + Error *local_err =3D NULL; =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -134,6 +144,13 @@ static void primary_vm_do_failover(void) FailoverStatus_lookup[old_state]); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err =3D NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -345,6 +362,15 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, s->params.shared =3D 0; qemu_savevm_state_header(fb); qemu_savevm_state_begin(fb, &s->params); + + /* We call this API although this may do nothing on primary side. */ + qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + qemu_mutex_unlock_iothread(); + if (local_err) { + goto out; + } + qemu_mutex_lock_iothread(); qemu_savevm_state_complete_precopy(fb, false); qemu_mutex_unlock_iothread(); @@ -451,6 +477,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -554,6 +586,7 @@ static void colo_wait_handle_message(QEMUFile *f, int *= checkpoint_request, case COLO_MESSAGE_GUEST_SHUTDOWN: qemu_mutex_lock_iothread(); vm_stop_force_state(RUN_STATE_COLO); + replication_stop_all(false, NULL); qemu_system_shutdown_request_core(); qemu_mutex_unlock_iothread(); /* @@ -602,6 +635,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -682,6 +720,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 2ade2aa..755ea54 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -394,6 +394,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis =3D migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err =3D NULL; =20 mis->from_src_file =3D f; mis->largest_page_size =3D qemu_ram_pagesize_largest(); @@ -425,6 +426,21 @@ static void process_incoming_migration_co(void *opaque) =20 /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + migrate_decompress_threads_join(); + exit(EXIT_FAILURE); + } + /* If we get an error here, just exit qemu. */ + blk_resume_after_migration(&local_err); + if (local_err) { + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850392378289.99058161230823; Sat, 22 Apr 2017 01:39:52 -0700 (PDT) Received: from localhost ([::1]:34741 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qa6-0003UZ-QR for importer@patchew.org; Sat, 22 Apr 2017 04:39:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59924) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNm-00021N-Cp for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNj-0000tR-87 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:06 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3404 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNi-0000sJ-Bw for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:03 -0400 Received: from 172.30.72.54 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.54]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71130; Sat, 22 Apr 2017 16:26:31 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:22 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:47 +0800 Message-ID: <1492849558-17540-8-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090206.58FB13B8.001A, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 7bd5165cece5ae24b6279018157c7aae X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 07/18] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, "Dr . David Alan Gilbert" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary= side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram ca= che always the same as PVM's memory at every checkpoint, then we flush this cac= hed ram to SVM after we receive all PVM's state. Cc: Dr. David Alan Gilbert Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian --- v2: - Move colo_init_ram_cache() and colo_release_ram_cache() out of incoming thread since both of them need the global lock, if we keep colo_release_ram_cache() in incoming thread, there are potential dead-lock. - Remove bool ram_cache_enable flag, use migration_incoming_in_state() inst= ead. - Remove the Reviewd-by tag because of the above changes. --- include/exec/ram_addr.h | 1 + include/migration/migration.h | 4 +++ migration/migration.c | 6 ++++ migration/ram.c | 71 +++++++++++++++++++++++++++++++++++++++= +++- 4 files changed, 81 insertions(+), 1 deletion(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index c9ddcd0..0b3d77c 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/include/migration/migration.h b/include/migration/migration.h index ba1a16c..ba765eb 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -360,4 +360,8 @@ uint64_t ram_pagesize_summary(void); PostcopyState postcopy_state_get(void); /* Set the state and return the old state */ PostcopyState postcopy_state_set(PostcopyState new_state); + +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); #endif diff --git a/migration/migration.c b/migration/migration.c index 755ea54..7419404 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -441,6 +441,10 @@ static void process_incoming_migration_co(void *opaque) error_report_err(local_err); exit(EXIT_FAILURE); } + if (colo_init_ram_cache() < 0) { + error_report("Init ram cache failed"); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); @@ -449,6 +453,8 @@ static void process_incoming_migration_co(void *opaque) =20 /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } =20 if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index f48664e..05d1b06 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2265,6 +2265,20 @@ static inline void *host_from_ram_block_offset(RAMBl= ock *block, return block->host + offset; } =20 +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2605,7 +2619,12 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + /* After going into COLO, we should load the Page into colo_ca= che */ + if (migration_incoming_in_colo_state()) { + host =3D colo_cache_from_block_offset(block, addr); + } else { + host =3D host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; @@ -2712,6 +2731,56 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) return ret; } =20 +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =3D qemu_anon_ram_alloc(block->used_length, NULL= ); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %= s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + memcpy(block->colo_cache, block->host, block->used_length); + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + rcu_read_unlock(); +} + static SaveVMHandlers savevm_ram_handlers =3D { .save_live_setup =3D ram_save_setup, .save_live_iterate =3D ram_save_iterate, --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849722362719.4423800207059; Sat, 22 Apr 2017 01:28:42 -0700 (PDT) Received: from localhost ([::1]:34696 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qPJ-0002yG-0J for importer@patchew.org; Sat, 22 Apr 2017 04:28:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59779) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jG-1c for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNO-0000hS-Hb for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3398 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNN-0000g5-UV for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:42 -0400 Received: from 172.30.72.54 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.54]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71129; Sat, 22 Apr 2017 16:26:31 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:23 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:48 +0800 Message-ID: <1492849558-17540-9-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090201.58FB13B7.0064, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: cb9ae118bf01d1331167615eb36b019f X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 08/18] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Cc: Juan Quintela Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 05d1b06..0653a24 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2268,6 +2268,9 @@ static inline void *host_from_ram_block_offset(RAMBlo= ck *block, static inline void *colo_cache_from_block_offset(RAMBlock *block, ram_addr_t offset) { + unsigned long *bitmap; + long k; + if (!offset_in_ramblock(block, offset)) { return NULL; } @@ -2276,6 +2279,17 @@ static inline void *colo_cache_from_block_offset(RAM= Block *block, __func__, block->idstr); return NULL; } + + k =3D (memory_region_get_ram_addr(block->mr) + offset) >> TARGET_PAGE_= BITS; + bitmap =3D atomic_rcu_read(&ram_state.ram_bitmap)->bmap; + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(k, bitmap)) { + ram_state.migration_dirty_pages++; + } return block->colo_cache + offset; } =20 @@ -2752,6 +2766,15 @@ int colo_init_ram_cache(void) memcpy(block->colo_cache, block->host, block->used_length); } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap to= gether + * with to decide which page in cache should be flushed into SVM's RAM.= Here + * we use the same name 'ram_bitmap' as for migration. + */ + ram_state.ram_bitmap =3D g_new0(RAMBitmap, 1); + ram_state.ram_bitmap->bmap =3D bitmap_new(last_ram_page()); + ram_state.migration_dirty_pages =3D 0; + return 0; =20 out_locked: @@ -2770,6 +2793,12 @@ out_locked: void colo_release_ram_cache(void) { RAMBlock *block; + RAMBitmap *bitmap =3D ram_state.ram_bitmap; + + atomic_rcu_set(&ram_state.ram_bitmap, NULL); + if (bitmap) { + call_rcu(bitmap, migration_bitmap_free, rcu); + } =20 rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849722308500.7508419697074; Sat, 22 Apr 2017 01:28:42 -0700 (PDT) Received: from localhost ([::1]:34693 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qPG-0002wY-78 for importer@patchew.org; Sat, 22 Apr 2017 04:28:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59777) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jE-0g for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNN-0000gx-KS for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:3397 helo=dggrg03-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNM-0000g4-OT for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:41 -0400 Received: from 172.30.72.54 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.54]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMH71128; Sat, 22 Apr 2017 16:26:31 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:24 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:49 +0800 Message-ID: <1492849558-17540-10-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090201.58FB13B7.0048, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: e19224d6e1f03c86629b2269880f9fa2 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.189 Subject: [Qemu-devel] [PATCH v2 09/18] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkp= oint time. So, the content of SVM's RAM cache will always be same with PVM's mem= ory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- include/migration/migration.h | 1 + migration/ram.c | 40 ++++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 3 files changed, 43 insertions(+) diff --git a/include/migration/migration.h b/include/migration/migration.h index ba765eb..2aa7654 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -364,4 +364,5 @@ PostcopyState postcopy_state_set(PostcopyState new_stat= e); /* ram cache */ int colo_init_ram_cache(void); void colo_release_ram_cache(void); +void colo_flush_ram_cache(void); #endif diff --git a/migration/ram.c b/migration/ram.c index 0653a24..df10d4b 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2602,6 +2602,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) bool postcopy_running =3D postcopy_state_get() >=3D POSTCOPY_INCOMING_= LISTENING; /* ADVISE is earlier, it shows the source has the postcopy capability = on */ bool postcopy_advised =3D postcopy_state_get() >=3D POSTCOPY_INCOMING_= ADVISE; + bool need_flush =3D false; =20 seq_iter++; =20 @@ -2636,6 +2637,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) /* After going into COLO, we should load the Page into colo_ca= che */ if (migration_incoming_in_colo_state()) { host =3D colo_cache_from_block_offset(block, addr); + need_flush =3D true; } else { host =3D host_from_ram_block_offset(block, addr); } @@ -2742,6 +2744,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && ram_cache_enable && need_flush) { + colo_flush_ram_cache(); + } return ret; } =20 @@ -2810,6 +2816,40 @@ void colo_release_ram_cache(void) rcu_read_unlock(); } =20 +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +void colo_flush_ram_cache(void) +{ + RAMBlock *block =3D NULL; + void *dst_host; + void *src_host; + unsigned long offset =3D 0; + + trace_colo_flush_ram_cache_begin(ram_state.migration_dirty_pages); + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(&ram_state, block, offset); + migration_bitmap_clear_dirty(&ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + dst_host =3D block->host + (offset << TARGET_PAGE_BITS); + src_host =3D block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state.migration_dirty_pages =3D=3D 0); +} + static SaveVMHandlers savevm_ram_handlers =3D { .save_live_setup =3D ram_save_setup, .save_live_iterate =3D ram_save_iterate, diff --git a/migration/trace-events b/migration/trace-events index b8f01a2..93f4337 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -72,6 +72,8 @@ ram_discard_range(const char *rbname, uint64_t start, siz= e_t len) "%s: start: %" ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: %zx len: %zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 149285053758558.68699159625146; Sat, 22 Apr 2017 01:42:17 -0700 (PDT) Received: from localhost ([::1]:34755 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qcS-00067V-5j for importer@patchew.org; Sat, 22 Apr 2017 04:42:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59935) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNp-00022P-1F for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNl-0000uj-Pj for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:09 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3469 helo=dggrg02-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNl-0000tM-5l for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:05 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg02-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMF70965; Sat, 22 Apr 2017 16:26:37 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:26 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:50 +0800 Message-ID: <1492849558-17540-11-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.58FB13BD.006D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: c8df25b891ad8e3040e5360223fc5780 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.188 Subject: [Qemu-devel] [PATCH v2 10/18] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Markus Armbruster , Michael Roth Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x_colo_lost_heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Cc: Markus Armbruster Cc: Michael Roth Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Eric Blake --- migration/colo.c | 19 +++++++++++++++++++ qapi-schema.json | 14 ++++++++++++++ qapi/event.json | 21 +++++++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 9949293..e62da93 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -516,6 +516,18 @@ out: qemu_fclose(fb); } =20 + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -757,6 +769,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } =20 if (fb) { qemu_fclose(fb); diff --git a/qapi-schema.json b/qapi-schema.json index 4b3e1b7..460ca53 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -1233,6 +1233,20 @@ 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } =20 ## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.10 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'request', 'error' ] } + +## # @x-colo-lost-heartbeat: # # Tell qemu that heartbeat is lost, request it to do takeover procedures. diff --git a/qapi/event.json b/qapi/event.json index e80f3f4..924bc6f 100644 --- a/qapi/event.json +++ b/qapi/event.json @@ -441,6 +441,27 @@ 'data': { 'pass': 'int' } } =20 ## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.10 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "reques= t" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## # @ACPI_DEVICE_OST: # # Emitted when guest executes ACPI _OST method. --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849857014559.098736703645; Sat, 22 Apr 2017 01:30:57 -0700 (PDT) Received: from localhost ([::1]:34706 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qRT-0004ut-KE for importer@patchew.org; Sat, 22 Apr 2017 04:30:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59778) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jF-1R for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNN-0000h6-Rw for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3468 helo=dggrg02-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNN-0000gc-95 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:41 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg02-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMF70966; Sat, 22 Apr 2017 16:26:37 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:27 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:51 +0800 Message-ID: <1492849558-17540-12-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.58FB13BD.0075, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 135fb2a0d253225435c516e685275abb X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.188 Subject: [Qemu-devel] [PATCH v2 11/18] savevm: split save/find loadvm_handlers entry into two helper functions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" COLO's checkpoint process is based on migration process, everytime we do checkpoint we will repeat the process of savevm and loadvm. So we will call qemu_loadvm_section_start_full() repeatedly, It will add all migration sections information into loadvm_handlers list everytime, which will lead to memory leak. To fix it, we split the process of saving and finding section entry into two helper functions, we will check if section info was exist in loadvm_handlers list before save it. This modifications have no side effect for normal migration. Cc: Juan Quintela Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/savevm.c | 55 +++++++++++++++++++++++++++++++++++++++-----------= ---- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/migration/savevm.c b/migration/savevm.c index 03ae1bd..f87cd8d 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1836,6 +1836,37 @@ void loadvm_free_handlers(MigrationIncomingState *mi= s) } } =20 +static LoadStateEntry *loadvm_add_section_entry(MigrationIncomingState *mi= s, + SaveStateEntry *se, + uint32_t section_id, + uint32_t version_id) +{ + LoadStateEntry *le; + + /* Add entry */ + le =3D g_malloc0(sizeof(*le)); + + le->se =3D se; + le->section_id =3D section_id; + le->version_id =3D version_id; + QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry); + return le; +} + +static LoadStateEntry *loadvm_find_section_entry(MigrationIncomingState *m= is, + uint32_t section_id) +{ + LoadStateEntry *le; + + QLIST_FOREACH(le, &mis->loadvm_handlers, entry) { + if (le->section_id =3D=3D section_id) { + break; + } + } + + return le; +} + static int qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis) { @@ -1878,15 +1909,12 @@ qemu_loadvm_section_start_full(QEMUFile *f, Migrati= onIncomingState *mis) return -EINVAL; } =20 - /* Add entry */ - le =3D g_malloc0(sizeof(*le)); - - le->se =3D se; - le->section_id =3D section_id; - le->version_id =3D version_id; - QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry); - - ret =3D vmstate_load(f, le->se, le->version_id); + /* Check if we have saved this section info before, if not, save it */ + le =3D loadvm_find_section_entry(mis, section_id); + if (!le) { + le =3D loadvm_add_section_entry(mis, se, section_id, version_id); + } + ret =3D vmstate_load(f, se, version_id); if (ret < 0) { error_report("error while loading state for instance 0x%x of" " device '%s'", instance_id, idstr); @@ -1909,12 +1937,9 @@ qemu_loadvm_section_part_end(QEMUFile *f, MigrationI= ncomingState *mis) section_id =3D qemu_get_be32(f); =20 trace_qemu_loadvm_state_section_partend(section_id); - QLIST_FOREACH(le, &mis->loadvm_handlers, entry) { - if (le->section_id =3D=3D section_id) { - break; - } - } - if (le =3D=3D NULL) { + + le =3D loadvm_find_section_entry(mis, section_id); + if (!le) { error_report("Unknown savevm section %d", section_id); return -EINVAL; } --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849860358376.91154619152815; Sat, 22 Apr 2017 01:31:00 -0700 (PDT) Received: from localhost ([::1]:34708 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qRW-0004x2-Ua for importer@patchew.org; Sat, 22 Apr 2017 04:30:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59784) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jJ-2u for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNO-0000hG-3i for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3467 helo=dggrg02-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNN-0000gb-94 for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:42 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg02-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMF70967; Sat, 22 Apr 2017 16:26:37 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:28 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:52 +0800 Message-ID: <1492849558-17540-13-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.58FB13BD.00AB, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 35e8ff4d74793d5fefbb2fe3886852c1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.188 Subject: [Qemu-devel] [PATCH v2 12/18] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_loadvm_state_begin(), qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- v2: - Use the wrapper qemu_savevm_state_header() to simplify the codes of qemu_save_device_state() (Dave's suggestion) --- include/sysemu/sysemu.h | 6 ++++++ migration/savevm.c | 54 ++++++++++++++++++++++++++++++++++++++++++---= ---- 2 files changed, 53 insertions(+), 7 deletions(-) diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h index 8054f53..0255c4e 100644 --- a/include/sysemu/sysemu.h +++ b/include/sysemu/sysemu.h @@ -132,7 +132,13 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f= , const char *name, uint64_t *start_list, uint64_t *length_list); =20 +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); + int qemu_loadvm_state(QEMUFile *f); +int qemu_loadvm_state_begin(QEMUFile *f); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); =20 extern int autostart; =20 diff --git a/migration/savevm.c b/migration/savevm.c index f87cd8d..8c2ce0b 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -54,6 +54,7 @@ #include "qemu/cutils.h" #include "io/channel-buffer.h" #include "io/channel-file.h" +#include "migration/colo.h" =20 #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -1285,13 +1286,20 @@ done: return ret; } =20 -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true); + qemu_put_byte(f, QEMU_VM_EOF); +} =20 - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; =20 + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1342,8 +1350,6 @@ enum LoadVMExitCodes { LOADVM_QUIT =3D 1, }; =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis= ); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -1957,7 +1963,7 @@ qemu_loadvm_section_part_end(QEMUFile *f, MigrationIn= comingState *mis) return 0; } =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret =3D 0; @@ -2095,6 +2101,40 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } =20 +int qemu_loadvm_state_begin(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; + int ret; + + if (qemu_savevm_state_blocked(&local_err)) { + error_report_err(local_err); + return -EINVAL; + } + /* Load QEMU_VM_SECTION_START section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to loadvm begin work: %d", ret); + } + return ret; +} + +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_vmstate(Monitor *mon, const char *name) { BlockDriverState *bs, *bs1; --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849999509463.4880125085666; Sat, 22 Apr 2017 01:33:19 -0700 (PDT) Received: from localhost ([::1]:34715 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qTm-0006YT-4g for importer@patchew.org; Sat, 22 Apr 2017 04:33:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59782) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jH-2Q for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNO-0000hM-4z for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3465 helo=dggrg02-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNN-0000gW-AK for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:42 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg02-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMF70963; Sat, 22 Apr 2017 16:26:36 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:29 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:53 +0800 Message-ID: <1492849558-17540-14-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.58FB13BC.009B, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 45a6ff49a7115787ee826e58fa577da8 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.188 Subject: [Qemu-devel] [PATCH v2 13/18] COLO: Separate the process of saving/loading ram and device state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We separate the process of saving/loading ram and device state when do checkpoint. We add new helpers for save/load ram/device. With this change, we can directly transfer RAM from primary side to secondary side without using channel-buffer as assistant, which also reduce the size of extra memo= ry was used during checkpoint. Besides, we move the colo_flush_ram_cache to the proper position after the above change. Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 49 +++++++++++++++++++++++++++++++++++++++---------- migration/ram.c | 5 ----- migration/savevm.c | 4 ++++ 3 files changed, 43 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index e62da93..8e27a4c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -357,11 +357,20 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, goto out; } =20 + colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); + if (local_err) { + goto out; + } + /* Disable block migration */ s->params.blk =3D 0; s->params.shared =3D 0; - qemu_savevm_state_header(fb); - qemu_savevm_state_begin(fb, &s->params); + qemu_savevm_state_begin(s->to_dst_file, &s->params); + ret =3D qemu_file_get_error(s->to_dst_file); + if (ret < 0) { + error_report("Save VM state begin error"); + goto out; + } =20 /* We call this API although this may do nothing on primary side. */ qemu_mutex_lock_iothread(); @@ -372,15 +381,21 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, } =20 qemu_mutex_lock_iothread(); - qemu_savevm_state_complete_precopy(fb, false); + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret =3D qemu_save_device_state(fb); qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); - - colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); - if (local_err) { + if (ret < 0) { + error_report("Save device state error"); goto out; } + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -621,6 +636,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err =3D NULL; + int ret; =20 qemu_sem_init(&mis->colo_incoming_sem, 0); =20 @@ -693,6 +709,17 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + ret =3D qemu_loadvm_state_begin(mis->from_src_file); + if (ret < 0) { + error_report("Load vm state begin error, ret=3D%d", ret); + goto out; + } + ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value =3D colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -726,8 +753,10 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(VMRESET_SILENT); vmstate_loading =3D true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + colo_flush_ram_cache(); + ret =3D qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/ram.c b/migration/ram.c index df10d4b..f171a82 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2602,7 +2602,6 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) bool postcopy_running =3D postcopy_state_get() >=3D POSTCOPY_INCOMING_= LISTENING; /* ADVISE is earlier, it shows the source has the postcopy capability = on */ bool postcopy_advised =3D postcopy_state_get() >=3D POSTCOPY_INCOMING_= ADVISE; - bool need_flush =3D false; =20 seq_iter++; =20 @@ -2637,7 +2636,6 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) /* After going into COLO, we should load the Page into colo_ca= che */ if (migration_incoming_in_colo_state()) { host =3D colo_cache_from_block_offset(block, addr); - need_flush =3D true; } else { host =3D host_from_ram_block_offset(block, addr); } @@ -2745,9 +2743,6 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); =20 - if (!ret && ram_cache_enable && need_flush) { - colo_flush_ram_cache(); - } return ret; } =20 diff --git a/migration/savevm.c b/migration/savevm.c index 8c2ce0b..60d346c 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1003,6 +1003,10 @@ void qemu_savevm_state_begin(QEMUFile *f, break; } } + if (migration_in_colo_state()) { + qemu_put_byte(f, QEMU_VM_EOF); + qemu_fflush(f); + } } =20 /* --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492849860387981.6471836572388; Sat, 22 Apr 2017 01:31:00 -0700 (PDT) Received: from localhost ([::1]:34707 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qRW-0004wq-Rt for importer@patchew.org; Sat, 22 Apr 2017 04:30:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59783) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNR-0001jI-2V for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNN-0000h8-SJ for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:45 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:3466 helo=dggrg02-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNN-0000ga-8n for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:41 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg02-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id AMF70964; Sat, 22 Apr 2017 16:26:36 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:30 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:54 +0800 Message-ID: <1492849558-17540-15-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.58FB13BC.00A6, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: d09de5acbdae645441f84fc37b96ad7f X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.188 Subject: [Qemu-devel] [PATCH v2 14/18] COLO: Split qemu_savevm_state_begin out of checkpoint process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It is unnecessary to call qemu_savevm_state_begin() in every checkpoint pro= cess. It mainly sets up devices and does the first device state pass. These data = will not change during the later checkpoint process. So, we split it out of colo_do_checkpoint_transaction(), in this way, we can reduce these data transferring in the subsequent checkpoint. Cc: Juan Quintela Sgned-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 51 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 8e27a4c..66bb5b2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -362,16 +362,6 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 - /* Disable block migration */ - s->params.blk =3D 0; - s->params.shared =3D 0; - qemu_savevm_state_begin(s->to_dst_file, &s->params); - ret =3D qemu_file_get_error(s->to_dst_file); - if (ret < 0) { - error_report("Save VM state begin error"); - goto out; - } - /* We call this API although this may do nothing on primary side. */ qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); @@ -459,6 +449,21 @@ static void colo_compare_notify_checkpoint(Notifier *n= otifier, void *data) colo_checkpoint_notify(data); } =20 +static int colo_prepare_before_save(MigrationState *s) +{ + int ret; + + /* Disable block migration */ + s->params.blk =3D 0; + s->params.shared =3D 0; + qemu_savevm_state_begin(s->to_dst_file, &s->params); + ret =3D qemu_file_get_error(s->to_dst_file); + if (ret < 0) { + error_report("Save VM state begin error"); + } + return ret; +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -478,6 +483,11 @@ static void colo_process_checkpoint(MigrationState *s) packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; colo_compare_register_notifier(&packets_compare_notifier); =20 + ret =3D colo_prepare_before_save(s); + if (ret < 0) { + goto out; + } + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -628,6 +638,17 @@ static void colo_wait_handle_message(QEMUFile *f, int = *checkpoint_request, } } =20 +static int colo_prepare_before_load(QEMUFile *f) +{ + int ret; + + ret =3D qemu_loadvm_state_begin(f); + if (ret < 0) { + error_report("Load VM state begin error, ret =3D %d", ret); + } + return ret; +} + void *colo_process_incoming_thread(void *opaque) { MigrationIncomingState *mis =3D opaque; @@ -662,6 +683,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + ret =3D colo_prepare_before_load(mis->from_src_file); + if (ret < 0) { + goto out; + } + qemu_mutex_lock_iothread(); replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); if (local_err) { @@ -709,11 +735,6 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 - ret =3D qemu_loadvm_state_begin(mis->from_src_file); - if (ret < 0) { - error_report("Load vm state begin error, ret=3D%d", ret); - goto out; - } ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); if (ret < 0) { error_report("Load VM's live state (ram) error"); --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850127641202.1192218281784; Sat, 22 Apr 2017 01:35:27 -0700 (PDT) Received: from localhost ([::1]:34724 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qVn-00088y-M0 for importer@patchew.org; Sat, 22 Apr 2017 04:35:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59848) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNW-0001n5-Kq for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNT-0000jA-IU for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:50 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:3506 helo=dggrg01-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNS-0000hz-UA for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:47 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ANE69258; Sat, 22 Apr 2017 16:26:42 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:31 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:55 +0800 Message-ID: <1492849558-17540-16-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.58FB13C2.00BD, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 4e8300bc417163a375ef9eda6190ac60 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.187 Subject: [Qemu-devel] [PATCH v2 15/18] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , xiecl.fnst@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com, lizhijian@cn.fujitsu.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Cc: Juan Quintela Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- v2: - stop dirty log after exit from COLO state. (Dave) --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index f171a82..7bf3515 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2775,6 +2775,7 @@ int colo_init_ram_cache(void) ram_state.ram_bitmap =3D g_new0(RAMBitmap, 1); ram_state.ram_bitmap->bmap =3D bitmap_new(last_ram_page()); ram_state.migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); =20 return 0; =20 @@ -2798,6 +2799,7 @@ void colo_release_ram_cache(void) =20 atomic_rcu_set(&ram_state.ram_bitmap, NULL); if (bitmap) { + memory_global_dirty_log_stop(); call_rcu(bitmap, migration_bitmap_free, rcu); } =20 @@ -2822,6 +2824,16 @@ void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state.bitmap_mutex); + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + migration_bitmap_sync_range(&ram_state, block, block->offset, + block->used_length); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state.bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state.migration_dirty_pages); rcu_read_lock(); block =3D QLIST_FIRST_RCU(&ram_list.blocks); --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850399072315.38556952687475; Sat, 22 Apr 2017 01:39:59 -0700 (PDT) Received: from localhost ([::1]:34743 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qaD-0003ac-Mb for importer@patchew.org; Sat, 22 Apr 2017 04:39:57 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59897) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNb-0001t2-RH for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNa-0000m4-Sx for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:55 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:3507 helo=dggrg01-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNa-0000jw-9A for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:54 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ANE69254; Sat, 22 Apr 2017 16:26:41 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:32 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:56 +0800 Message-ID: <1492849558-17540-17-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.58FB13C2.0004, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 5db5cb2de09617644d32837130d70b5b X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.187 Subject: [Qemu-devel] [PATCH v2 16/18] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Cc: Jason Wang Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 16 ++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 0c4a2ea..df4510d 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -37,6 +37,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, =20 typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); =20 +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **e= rrp); + typedef struct NetFilterClass { ObjectClass parent_class; =20 @@ -44,6 +46,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -76,4 +79,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sende= r, int iovcnt, void *opaque); =20 +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 1dfd2ca..993b35e 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,7 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" =20 static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +246,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } =20 +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_CHECKPOINT: + break; + case COLO_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc =3D NETFILTER_CLASS(oc); =20 ucc->complete =3D netfilter_complete; + nfc->handle_event =3D dummy_handle_event; } =20 static const TypeInfo netfilter_info =3D { diff --git a/net/net.c b/net/net.c index 0ac3b9e..1373f63 100644 --- a/net/net.c +++ b/net/net.c @@ -1373,6 +1373,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdi= ct) } } =20 +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc =3D NULL; + Error *local_err =3D NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer =3D nc->peer; + type =3D nc->info->type; + if (!peer || type !=3D NET_CLIENT_DRIVER_NIC) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =3D NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 14928502404911016.3329773368243; Sat, 22 Apr 2017 01:37:20 -0700 (PDT) Received: from localhost ([::1]:34732 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qXe-00015d-7z for importer@patchew.org; Sat, 22 Apr 2017 04:37:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59909) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qNf-0001wR-3K for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNc-0000o9-Em for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:59 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:3508 helo=dggrg01-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNb-0000lS-Qr for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:26:56 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ANE69260; Sat, 22 Apr 2017 16:26:42 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:33 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:57 +0800 Message-ID: <1492849558-17540-18-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.58FB13C3.0064, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 03e97cbcae354ed26977b1fd566931dc X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.187 Subject: [Qemu-devel] [PATCH v2 17/18] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Cc: Jason Wang Signed-off-by: zhanghailiang --- net/filter-rewriter.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index c9a6d43..0a90b11 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -22,6 +22,7 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" =20 #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -270,6 +271,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterStat= e *nf, return 0; } =20 +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_d= ata) +{ + Connection *conn =3D (Connection *)value; + + conn->offset =3D 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn =3D (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs =3D FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s =3D FILTER_COLO_REWRITER(nf); @@ -299,6 +337,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, v= oid *data) nfc->setup =3D colo_rewriter_setup; nfc->cleanup =3D colo_rewriter_cleanup; nfc->receive_iov =3D colo_rewriter_receive_iov; + nfc->handle_event =3D colo_rewriter_handle_event; } =20 static const TypeInfo colo_rewriter_info =3D { --=20 1.8.3.1 From nobody Fri May 3 21:36:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1492850544622538.3343298060454; Sat, 22 Apr 2017 01:42:24 -0700 (PDT) Received: from localhost ([::1]:34756 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qcY-0006Dm-VN for importer@patchew.org; Sat, 22 Apr 2017 04:42:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59985) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1qO1-0002Bh-Ei for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1qNt-0000vu-UT for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:17 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:3509 helo=dggrg01-dlp.huawei.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1qNt-0000va-JG for qemu-devel@nongnu.org; Sat, 22 Apr 2017 04:27:13 -0400 Received: from 172.30.72.57 (EHLO DGGEML403-HUB.china.huawei.com) ([172.30.72.57]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ANE69255; Sat, 22 Apr 2017 16:26:42 +0800 (CST) Received: from localhost (10.177.24.212) by DGGEML403-HUB.china.huawei.com (10.3.17.33) with Microsoft SMTP Server id 14.3.301.0; Sat, 22 Apr 2017 16:26:34 +0800 From: zhanghailiang To: , Date: Sat, 22 Apr 2017 16:25:58 +0800 Message-ID: <1492849558-17540-19-git-send-email-zhang.zhanghailiang@huawei.com> X-Mailer: git-send-email 2.7.2.windows.1 In-Reply-To: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> References: <1492849558-17540-1-git-send-email-zhang.zhanghailiang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.24.212] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.58FB13C2.0095, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 04a99677af24dc7b8c4b891133e0db41 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] [fuzzy] X-Received-From: 45.249.212.187 Subject: [Qemu-devel] [PATCH v2 18/18] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lizhijian@cn.fujitsu.com, xiecl.fnst@cn.fujitsu.com, zhanghailiang , zhangchen.fnst@cn.fujitsu.com, quintela@redhat.com, Jason Wang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Notify all net filters about the checkpoint and failover event. Cc: Jason Wang Signed-off-by: zhanghailiang --- migration/colo.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 66bb5b2..62f58c6 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -26,6 +26,7 @@ #include "qapi-event.h" #include "block/block.h" #include "replication.h" +#include "net/filter.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -82,6 +83,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -794,6 +800,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); --=20 1.8.3.1