From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15207546328292.0509898359798626; Sat, 10 Mar 2018 23:50:32 -0800 (PST) Received: from localhost ([::1]:53360 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvkV-0001DD-N8 for importer@patchew.org; Sun, 11 Mar 2018 03:50:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59105) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvfm-00061c-ST for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvfl-0000XA-An for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:38 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:43911) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvfl-0000Vp-36 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:37 -0400 Received: by mail-pf0-x243.google.com with SMTP id j2so3036300pff.10 for ; Sat, 10 Mar 2018 23:45:36 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=frxRX3jL4VVjHYV1oYmTUZX4+eV8sO0DhXbz1gEnl6M=; b=eXaJVWwMXW3AQ4xz0QPoZh0BZSCizLYKLmmMvS1Nt2A6oAYp4dO4WRS6gkH9eXhYAE Ln+/TDRhNwiAy4fuOS7nSVcAD//ikftuYjH4nbIdQ//hU7sDYBP698YWhjFbSW+rhqXo I7hvc6eTZMHWVpcAsOmt7+mZmLnF8ncBFcy+6nZOX/8aihEZhB5j1/Cl+CLebsA6PrYB EfMQJq4oW5YIne/d899ZHDq6+dRz2lEQU6/NfrMvzALy432+2tXfmSlBnmuVaWN7sADL DX+DBiIjvTm6kI11mBdON1iMy2N0ne6zfsDDor7aQwCY5MKLwFbWdrMxHlPz4jiYkmBb Ejdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=frxRX3jL4VVjHYV1oYmTUZX4+eV8sO0DhXbz1gEnl6M=; b=EEh0In61Y9kbOKrMfV4dOOsYYFJ/dMfWokzFOCTKyzMoXOMakIuI5tmuspAnGyohcN QVdRjSapCkCy/N4j5NSsWeUbBxKxUrMZ9wiA8EP+RkPBJNb1E/+Wx1A7rJniHT/oa0yf oM2qOJjpeFuiBe/9myYGogBm2avDrD53Q+GlJu4EKtj/XX85LDIkxXI6QWh8IN0NgrNc +k3VsSLJYCF311xmVuL0ALs3C7W2EbWZvHKCwFa8kONY99NuuLMWt4dibtR+aCh+TZTB QiiT+2UQrwvAqQuTHAZB67RwTNMv/4rQbgVfTOERwlBnnaAJWrfy+gkXfFX3MipEH4Bl rbjQ== X-Gm-Message-State: AElRT7EwiChG4p7/sG+ntpaviccv6X5UW1sUhNmy3unvVs0Ub3mdIwyL DKndd9+JBnN8AD8hG3neS71Ezo5q X-Google-Smtp-Source: AG47ELs2k4y5VB0JT+BojK2NtyNAvMTTPgroIUyxGmtSa7tVH4TH27EPb+iMWuu4pygHrwf+CWlRsQ== X-Received: by 10.99.117.76 with SMTP id f12mr3527508pgn.410.1520754335714; Sat, 10 Mar 2018 23:45:35 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:44:58 +0800 Message-Id: <1520754314-5969-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V6 01/17] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 +++++++++++++++++++++++++++++++++++++++++++++--= ---- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index da6c36d..cd11851 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" =20 #define HASHTABLE_MAX_SIZE 16384 =20 @@ -86,6 +87,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=3D1,ack=3D1' */ } Connection; =20 uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 62dad2d..0909a9a 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } =20 /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -99,15 +99,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=3D1' packets from client side, it acks 'fin=3D1, ac= k=3D1' + * packet from server side. From this point, we can ensure that th= ere + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this ra= re + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_ta= ble. + * + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from client side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D htonl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } =20 return 0; } =20 /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -139,8 +168,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=3D1' packets from server side, it acks 'fin=3D1, ac= k=3D1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=3D1,ack=3D1'= packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from server side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D ntohl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } - return 0; } =20 @@ -190,7 +245,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, =20 if (sender =3D=3D nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +258,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754635984778.9683764863133; Sat, 10 Mar 2018 23:50:35 -0800 (PST) Received: from localhost ([::1]:53361 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvkZ-0001G1-5S for importer@patchew.org; Sun, 11 Mar 2018 03:50:35 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59122) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvfq-00064K-EG for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvfp-0000Z2-A0 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:42 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:38484) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvfp-0000Ym-1H for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:41 -0400 Received: by mail-pf0-x243.google.com with SMTP id d26so3041790pfn.5 for ; Sat, 10 Mar 2018 23:45:40 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jZZL+cB7ywLrmktyU3wExkoQrmvin6i8y6BlcpE3nRg=; b=qJXsXVR9N2V5t5Gei78E0n626LOZXwhKVAVZUPt+HLa/+6FIJJkNdOLqJIWRmEp3RG x7+BZey9zqQQXxpQgu5xpVdO2CSpl2yZiY4X/jBg9CdUpxhen+6a5wsP5z3Jffc1eVN8 DSLSDufrI1xjc1BzMwXz3ZXOqe1NQdKZsokytgxULFDx9vbRQCVH12ook20uN0cxmOwq iJGQ2pXXaWr3Kwr5imHm19sy/tTEwD6nazMZkRzYm4AQiSSVUw28uKGrWothvEbmwuCg kxVVDuRqyUigoTbJDEZ/AcjSsRD5GxkRuYR5Pr+wF69lrazj1pF920ra91lMJJwsPm5H xRHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jZZL+cB7ywLrmktyU3wExkoQrmvin6i8y6BlcpE3nRg=; b=uH0qX2JVTQZ6Ir0mV0DcV92lz4n1mjxhv3q7A6L142JRYNRBws2WfQmIYR6ZF4aIL1 KRJVRVQwB0O2K+C8/3AcqOE51QW6XyGWTm6vQcNkQjZjrI6r+FZHKjYylcaKB7A1dVIf LLthmeQq4JZSGtbYzVsUzq25L0y+wBakzEyU6sDNYBRDCH/hHAHlCr18PC6nHqKl9Dl5 yrtvbchSeVI5wC9HfxJv06e9yNO4TVjH5m2VcIXNZQcoTeUuaLK0xFDQlWa+sRhpylzI BHYzyLIcTba5dNPXQDSZ07HYCy1j28phlvIQQwIXOc9UII3SZChWpCNj3ZDsIxqtJOzZ G0pg== X-Gm-Message-State: AElRT7GSb87VZqT/Bjsb03FnHplIJdBJMcDKG3Wt+XhQ9k3EAts2si6O Y7mGKKx0uSqaiQPS5unv+W+Yyy6p X-Google-Smtp-Source: AG47ELsXCheXLOZlwpHo9imudcmyrd/qnUrWXEgQg/SawT1q6Bx2KkA6cq8NikA49RgVskWVXDyjPA== X-Received: by 10.99.5.137 with SMTP id 131mr3473472pgf.60.1520754339722; Sat, 10 Mar 2018 23:45:39 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:44:59 +0800 Message-Id: <1520754314-5969-4-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V6 02/17] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++++++= ++++ net/colo-compare.h | 22 ++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad..fefb2fc 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" =20 +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); =20 void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 23b2d2c..7ff3ae8 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) =20 +static QTAILQ_HEAD(, CompareState) net_compares =3D + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 =20 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; =20 typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } =20 +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event =3D event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx =3D iothread_get_aio_context(s->iothread); @@ -756,6 +789,28 @@ static void colo_compare_timer_del(CompareState *s) } } =20 +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s =3D opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +824,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); =20 colo_compare_timer_init(s); + s->event_bh =3D qemu_bh_new(colo_compare_handle_event, s); } =20 static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +982,13 @@ static void colo_compare_complete(UserCreatable *uc, E= rror **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); =20 + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); =20 + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table =3D g_hash_table_new_full(connection_key_has= h, connection_key_equal, g_free, @@ -990,6 +1051,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s =3D COLO_COMPARE(obj); + CompareState *tmp =3D NULL; =20 qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1059,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); =20 @@ -1009,6 +1081,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754483097558.4242565972627; Sat, 10 Mar 2018 23:48:03 -0800 (PST) Received: from localhost ([::1]:53346 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvi6-0007Nb-A3 for importer@patchew.org; Sun, 11 Mar 2018 03:48:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59133) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvfu-00067Y-3F for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvft-0000ad-4x for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:46 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:44713) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvfs-0000aJ-V0 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:45 -0400 Received: by mail-pg0-x241.google.com with SMTP id l4so5242101pgp.11 for ; Sat, 10 Mar 2018 23:45:44 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RyVyyiWz6BVrwmCmJeVQGLsDL7nGb6Qbw2cd4DzIUuw=; b=aQWPfoGJ7EkyP/d4BkM59rmFZcRO3gbfkjFmtlMk2yGQC+yu5wuSWxPD5Nbbb8B6wq MN3/BUI5+HIOky5cNI+Mn8XuSjgysSBVkT0rRfnjC02m/Dxb9YVuxKamUq7ca2mHEvSI LiexGYRidw+EyHTXS9rrx3kvna9i9K3vaXX/KlQnJggwHa4zMQSmNm82JSDcQQb5n7yS tsmURBu4Wb7yLSspjPKyYsYfzjurPcBakfmgIiWVh+h6Nn/zfnN6a54+JwUaq2IDy4BA IKg0g8uyV4KKUkGsRTqzroAm9vNWkMM7KAeWjPqe9nXrJaFB74cfX6wrJ/5GZ1Cg1l3A 6SNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RyVyyiWz6BVrwmCmJeVQGLsDL7nGb6Qbw2cd4DzIUuw=; b=mEF+RaqfK3BH15BZPbOO66mP2SkvdxKiGj8KNOJv5CbtSs0nKN3PPvQzcqxitRvs45 6UlmGiCVmRVIWfY8NpHG4MbViuHYNkTPJmmHGA4MGBgEpkDHsZuK3fSqqkcHfs7TwwuP kN6EUrydO3oGy2SEkb7oXqB1SHeNYpQahEbHv77Sm1KK89lar/D8eZ3CcTPDrYSLMKFY b7FNsu59T6zsZ3kLhRA4IhHhzTOqlW6hIjFrFAaaaV2ZzrvG2fJu45VJMWybh6YDogjX cvmKh0RYB+WLKg2BsY78LiBwKOhU6NRT8x+uVRmfGC6kcaiSYE9+eO4gHhAaYbjkcreK eqVw== X-Gm-Message-State: AElRT7HI9IqpTcE6Lgr32iB/VmwOms3bC9HI9u3rvGvqKlgCxYjUuQHg /WKX0ADsthnuLyksZqZET8ec/wFh X-Google-Smtp-Source: AG47ELssxqk2Hy3Bxv2oBqffs28amAbC/8shyew1v68AAsRwhzHa/SFU241fpcdFT8R3uNpX+MJarw== X-Received: by 10.99.124.1 with SMTP id x1mr1479288pgc.318.1520754343669; Sat, 10 Mar 2018 23:45:43 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:00 +0800 Message-Id: <1520754314-5969-5-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH V6 03/17] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 7ff3ae8..05061cd 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =3D QTAILQ_HEAD_INITIALIZER(net_compares); =20 +static NotifierList colo_compare_notifiers =3D + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -561,8 +565,24 @@ static int colo_old_packet_check_one(Packet *pkt, int6= 4_t *check_time) } } =20 +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result =3D NULL; int64_t check_time =3D REGULAR_PACKET_CHECK_MS; @@ -573,10 +593,7 @@ static int colo_old_packet_check_one_conn(Connection *= conn, =20 if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } =20 @@ -620,11 +637,12 @@ static void colo_compare_packet(CompareState *s, Conn= ection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76..22ddd51 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H =20 void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); =20 #endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754860788775.7087844181955; Sat, 10 Mar 2018 23:54:20 -0800 (PST) Received: from localhost ([::1]:53383 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvoC-0004bg-0w for importer@patchew.org; Sun, 11 Mar 2018 03:54:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59144) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvg0-0006DH-9i for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvfx-0000cL-6L for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:52 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:35245) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvfw-0000bw-Uh for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:49 -0400 Received: by mail-pg0-x244.google.com with SMTP id l131so5250602pga.2 for ; Sat, 10 Mar 2018 23:45:48 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nr70ZvGv7z5M6QMDWePDS+ONDj2QkdxJHgd5N8gmR3M=; b=NC02ykvfQxBAtLMqX/KIyhObHkFLiVAwQcrW/C/zAiJF3Tc0WoEJOhCj8vAALcG1Y8 KCLBK8kf+Y12FfVW4EJQx3s9D0RRDy9oPKAO54LNJcS7oBTqlRuDa20bYtylSRqiGmxA GR+SCdtgsHvZQOcYtTHoJ+1irsCfzkzTqslKwYjSB8NUp4PqmupPLrV8VIrY5UucaZHe yQj4E+I5lF5J9uXW8AU5osHSzMJ811odRcX0aaUj2N22c1Zy/KoMaUXckkmo1iNQZrho 1iWlBVjVoZmRa0orq6KAPj+YVp9PUDuarohFM5ldpLs1kPMbdq6euFnJnc3yINLtgSuu MO+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nr70ZvGv7z5M6QMDWePDS+ONDj2QkdxJHgd5N8gmR3M=; b=EBvrqsqGqqa3gTSoyjLH2ZZptK1PWVYCHj0yrOdRBMPwXUBcNrNWl5Jh9qHFH94o3n /n/J16SQtdMgkwEbltuoM8EkKxBPatE4HX35HisTkzE0IVez+GvIund2CIBTZ+zqMTko WLVZFzpd2FGU/kzvYTuY7VJqb24DurzYxggS1molNrOUNSEdXk1ze2Z4uuXKpRVgbxZl reocHtkIpUwfUooG/FrbNg2LLg0h3HQ0WvA5M4V4dtX3fgnQwXdpDS3rEF0R6OgSGXDm 0oyNxqF94fFfSQNXoJ4QDRyeEnUCE9752DvZfoVYz0aZf3cQijKMopCe8nHGDzPvFqcw Sx1g== X-Gm-Message-State: AElRT7Fc8Ml4EOCZyY2OtOqbfv+AIN7yQYi2uTI8ecHtNQtYGGvyWqqA 2tZ5cnXV48rw+svQXxRg4eGHD5IB X-Google-Smtp-Source: AG47ELthybE/lMWp33tesi/UionxrF7efoVETpRE42zkaAi/Q3wYe+06HrYAB9LK62t5kpYe2n2ptA== X-Received: by 10.98.218.7 with SMTP id c7mr4085558pfh.162.1520754347617; Sat, 10 Mar 2018 23:45:47 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:01 +0800 Message-Id: <1520754314-5969-6-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V6 04/17] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067..081df18 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" =20 static bool vmstate_loading; +static Notifier packets_compare_notifier; =20 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) =20 @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } =20 +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 + packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } =20 - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } =20 vmstate_loading =3D false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); =20 if (failover_get_state() =3D=3D FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index e345d0c..7f72f78 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -76,9 +76,9 @@ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) =20 /* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid = mode. + * Note: Please change this default value to 20000 when we support hybrid = mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 =20 --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754814555264.3455555948742; Sat, 10 Mar 2018 23:53:34 -0800 (PST) Received: from localhost ([::1]:53380 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvnE-0003eT-QH for importer@patchew.org; Sun, 11 Mar 2018 03:53:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59155) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvg2-0006E9-JN for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvg1-0000eP-HG for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:54 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:42143) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvg1-0000dy-9A for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:53 -0400 Received: by mail-pf0-x243.google.com with SMTP id a16so3036705pfn.9 for ; Sat, 10 Mar 2018 23:45:53 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dV39DM1htJcTRZ9VSV+34MRY12c//1ZBNeILrPdCi1s=; b=rRc3WQz/Tdq/aPyg6dprNCtqgzWsEQYeqn/EcSC4KrlHiwUXgFazU71qQGI+vugFje UcmjO9tkT1B4+Lsgf9p+hF9OmQCiXH4LFcsnTYDnLCMOqEh6zrDFPokRtiia2evCuKyH DBPA92UBFxCnWU9QLaahSZdcVX5wNRf4llMeWux7BcLellJtLW1Ff4Emu6pW6sBZxsYC 50l/nI169Jfn3HYRV0A1oyl+h9qt1gxnotihmfJQx3Bv31/5Coo0tcaHRW6b6g5ye1cb YJ6GMHKwN9Odw/23O3zKl36vLlk+1GmYMvah4qLsaXr6QGNpVUgPoKZqDi6WSDcVh9C0 q4nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dV39DM1htJcTRZ9VSV+34MRY12c//1ZBNeILrPdCi1s=; b=XjJe1SXpgUET4EtmsCQ4BQfDUdJGpd2A279k/9Oi5sCJEI05rhQChgC9ZmKM/UapJJ xG1ggergU5DoG+BJRznDkKuqyHHhckcqYJbxO2myCE4kPwmhmWjz/Xw0o8UWFUltFex1 NBMKs4WCxjbinfbFxdnmBbDKNTNi349mjTRG3JJfYZen31vs5UmMu8W2TgjELbWhEHvE uAiGi/lEahawLWsaslFa84INAWqc5ZJaRhQ1+knD/DllzlSxJR8iahGw+baufpWR+98J mvZ6KnwQ6br34zNzWjYCxA2nNvUBjERvqc5Hk2JAMGfV4ZKeu/WS8Xe6r3ZSGBEfGdHF 2gPw== X-Gm-Message-State: AElRT7HpO1/xOsFrUrazvtrTJIUKgbuqs/dGTBFkHp5hyZ8UMyIDyF1w rfsa6QIz2dHI4eErjMiWXNjTEk3Y X-Google-Smtp-Source: AG47ELubP35XD8ssnHKbx4AE0NzjtzE/Geem2T6LCh3EnRVGCPSTPidscEcIjawXESHRhyyGnywGvQ== X-Received: by 10.101.97.207 with SMTP id j15mr3380549pgv.266.1520754352020; Sat, 10 Mar 2018 23:45:52 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:02 +0800 Message-Id: <1520754314-5969-7-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V6 05/17] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df18..e06640c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; =20 /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); =20 + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s =3D migrate_get_current(); int old_state; + Error *local_err =3D NULL; =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err =3D NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); =20 @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 7f72f78..128daaa 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -346,6 +346,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis =3D migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err =3D NULL; =20 assert(mis->from_src_file); mis->largest_page_size =3D qemu_ram_pagesize_largest(); @@ -377,6 +378,14 @@ static void process_incoming_migration_co(void *opaque) =20 /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754495499165.54322327108264; Sat, 10 Mar 2018 23:48:15 -0800 (PST) Received: from localhost ([::1]:53347 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euviI-0007Zq-MB for importer@patchew.org; Sun, 11 Mar 2018 03:48:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59173) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvg9-0006I2-82 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvg6-0000gV-3M for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:01 -0400 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:35964) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvg5-0000g4-RA for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:45:58 -0400 Received: by mail-pg0-x241.google.com with SMTP id i14so5252527pgv.3 for ; Sat, 10 Mar 2018 23:45:57 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SPBx8J3iYP9KDnmMYNDMC2qTaHhQpjQcoJU9L4s1kOY=; b=TQ/QDh+R9XrZAJUW+xQQWmdaIuKntRMDPd22IqTT3jsHxqFqPHUIJJ7ZvHrdRcS2Wa hRLZbG9d+D4KcVyYE9/ksCaB/AWi/K5psH1EjfOScuMztFDtOjGlZNn6CJO6YH7VHfcv UfCIrgxzFvyd3QKL/WW8fxfmIe0QaaRWWqY2eM8oyCtuhtIVpZjtMvY5hx+Xs5vsLRLA uKLdYa53PydAd6V75wLA9U/WPWFlgESRtuXMTDDcebosDUPcWbpVFVlZHSLdc3klGbPi uubXzx5CeAgJQmsKKWiFkaIH4/hlvP+HQfO/faXfATbE4AZnF2w4nwxMriR6s444ewx1 ilcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SPBx8J3iYP9KDnmMYNDMC2qTaHhQpjQcoJU9L4s1kOY=; b=Uoi6K1Up7FJevMRVNhApN6P8loFyO7inKTsVR6oq02u9ZQgwWBqIqnFH1zXkQKAtDQ U6UnBaW3A7ziVZG4K5Is/bS5/piq5IZaJ/mCYOZa3SRR9GYujKexvDIZE5OKcrebGGIn 49d4OEnFPI/frXPLfwQruo+ZDvZGgkWsTIFmm+5S9+YN/ZVJby7qsYFVpuoGH5MImLit TUsIbQxqNAJANKAAnuCICNsolrA09KfCJ8zvFb+dgaaGVmVqox5Yg1Kf8jpd0BCpapYV FYx5I2akkXSxXfGmo2F/gwGrG9PhQCu7LTFcORqMpRviNWUM7nluKouJkZH7lerAd+lG HUjA== X-Gm-Message-State: AElRT7EbgqYv42a2aA4HGTIWx7QbCTILmEBl869cK37XSa04XU+QeZjb A8SdO5308I/DDqKS7vFJWUii2Z4Z X-Google-Smtp-Source: AG47ELvF+Og1P+GR8KKTtIJ6rg5e6NWOOkhGy5Soc5kXV8jRAxWvgT43BtEHo3mukMRUcw22X5Svgw== X-Received: by 10.99.119.133 with SMTP id s127mr3195787pgc.441.1520754356509; Sat, 10 Mar 2018 23:45:56 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:03 +0800 Message-Id: <1520754314-5969-8-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH V6 06/17] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 --------------------------------------------= ---- migration/colo.c | 13 ++++++++- migration/migration.c | 23 ++++++++++++++- migration/savevm.c | 19 ++++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 59 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fc..99ce17a 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); =20 /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); =20 diff --git a/migration/Makefile.objs b/migration/Makefile.objs index 99e0380..3099eec 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y +=3D migration.o socket.o fd.o exec.o common-obj-y +=3D tls.o channel.o savevm.o -common-obj-y +=3D colo-comm.o colo.o colo-failover.o +common-obj-y +=3D colo.o colo-failover.o common-obj-y +=3D vmstate.o vmstate-types.o page_cache.o common-obj-y +=3D qemu-file.o global_state.o common-obj-y +=3D qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4d..0000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s =3D opaque; - - s->colo_requested =3D migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state =3D { - .name =3D "COLOState", - .version_id =3D 1, - .minimum_version_id =3D 1, - .pre_save =3D colo_info_pre_save, - .needed =3D colo_info_need, - .fields =3D (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested =3D false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c..c083d36 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } =20 +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); =20 return NULL; } diff --git a/migration/migration.c b/migration/migration.c index 128daaa..47328df 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -268,6 +268,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *= mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } =20 +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled =3D false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled =3D true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -377,7 +393,7 @@ static void process_incoming_migration_co(void *opaque) } =20 /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2365,6 +2381,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } =20 + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); =20 s->setup_time =3D qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index 358c5b5..10a6676 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -54,6 +54,7 @@ #include "qemu/cutils.h" #include "io/channel-buffer.h" #include "io/channel-file.h" +#include "migration/colo.h" =20 #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -80,6 +81,9 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream= */ + + MIG_CMD_ENABLE_COLO, /* Enable COLO */ + MIG_CMD_MAX }; =20 @@ -835,6 +839,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } =20 +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1769,6 +1779,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncom= ingState *mis) return ret; } =20 +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1842,6 +1858,9 @@ static int loadvm_process_command(QEMUFile *f) =20 case MIG_CMD_POSTCOPY_RAM_DISCARD: return loadvm_postcopy_ram_handle_discard(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } =20 return 0; diff --git a/migration/savevm.h b/migration/savevm.h index 295c4a1..041d23c 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -51,6 +51,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, c= onst char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 93961de..b27a3a0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -34,6 +34,7 @@ savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" +savevm_send_colo_enable(void) "" savevm_state_setup(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" diff --git a/vl.c b/vl.c index dae986b..e67750d 100644 --- a/vl.c +++ b/vl.c @@ -4471,8 +4471,6 @@ int main(int argc, char **argv, char **envp) #endif } =20 - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754652805415.53455738366176; Sat, 10 Mar 2018 23:50:52 -0800 (PST) Received: from localhost ([::1]:53367 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvkp-0001ai-UY for importer@patchew.org; Sun, 11 Mar 2018 03:50:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59185) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgB-0006K5-Ky for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgA-0000iV-4d for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:03 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:45829) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvg9-0000hz-RY for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:02 -0400 Received: by mail-pf0-x244.google.com with SMTP id h19so3034226pfd.12 for ; Sat, 10 Mar 2018 23:46:01 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.45.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:45:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FwbfEomgswg5pfZiHLpZIZOP0KsxS2HQUOa6bKdBK5Y=; b=grHhz5z1G5Yn1NPg7iQORGoSiG3rhLz3owQ0t554MHXIVeqXDV4t45smXvUPfp5+HY cuT6wiQgL0M/cOh1uYNL95CRbDM19hcY9/g62ba/eYskuL3nRto5nfQT5hACWd4KLy20 /Te1VZ3xvrs3Tayi+gH1kuaHJQBV91/e0ux4alXoxmmLgW3utiE5E1oRGTP9Fs4YEjEv 4zvQuRhyNUUvlm8oB84eMkZAlxeysFzWhKcthYPIG+pU2DE8f1Wdi3nhdfB45+XQg/rG ORMnnATtKhNF2rsHADx8UL4EVfZHXRuaXbKe+4jRbyuYw4vs/HtgVLIHGrvW8G7o7HOg ucnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FwbfEomgswg5pfZiHLpZIZOP0KsxS2HQUOa6bKdBK5Y=; b=hS0yu/r7Q2VkWOxUgfRgbFgn50AFTgAH6SYyPOvegdvtPf/VSs5AMJ2qobxOGyGffH HvGfEdaH5PIQwuaS9OZdT6BcKgFXJKjcjKtGG+lbaRoXO4X5WX/uTft8J50pgzQnAzvA EU60GadFEQksy36Tf84OBgdPmh7KOmUsLtUJo6lc0kP8xlh0L/8343fXj2tMbitwTAb4 R7vHilbZ3QP+b2G8FvxzLZyqlDEZglkeQI3DVTCxfRKRzlzYDe1f/YSNsVCLxDtfVVQ3 QUumRcXeCtVvWx5kNXNRdD1eSYsKPkBhlUU7QWHFO1BIO0KyzIo1WX5erKkSeUbCKrdD EfAw== X-Gm-Message-State: AElRT7Ewi4xa1LpIhBUf6O8At2shIvvWdPEViHXLJe0mEsYaI6FYGgzO fMUcJWq/zaht1iw//LX+TmFO9+tI X-Google-Smtp-Source: AG47ELuPw78Jasu73cDdczCiQCNeWyLaL3Ym3XDqdXCtgymoQ2BKp2QFYwozkvYLoDZILBb1EaSq5A== X-Received: by 10.101.67.73 with SMTP id k9mr3412504pgq.244.1520754360470; Sat, 10 Mar 2018 23:46:00 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:04 +0800 Message-Id: <1520754314-5969-9-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH V6 07/17] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary= side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram ca= che always the same as PVM's memory at every checkpoint, then we flush this cac= hed ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 99 +++++++++++++++++++++++++++++++++++++++++++++= ++-- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 104 insertions(+), 4 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf2446a..51ec153 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 47328df..4d5e188 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -410,6 +410,8 @@ static void process_incoming_migration_co(void *opaque) =20 /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } =20 if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 3b6c077..86f5c3c 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2470,6 +2470,20 @@ static inline void *host_from_ram_block_offset(RAMBl= ock *block, return block->host + offset; } =20 +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2623,6 +2637,57 @@ static void decompress_data_with_multi_threads(QEMUF= ile *f, qemu_mutex_unlock(&decomp_done_lock); } =20 +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =3D qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %= s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -2636,6 +2701,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); compress_threads_load_setup(); ramblock_recv_map_init(); + return 0; } =20 @@ -2649,6 +2715,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap =3D NULL; } + return 0; } =20 @@ -2862,7 +2929,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) =20 while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host =3D NULL; + void *host =3D NULL, *host_bak =3D NULL; uint8_t ch; =20 addr =3D qemu_get_be64(f); @@ -2882,13 +2949,36 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_ca= che + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Privously, we copied all these memory in preparing stage of= COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects = the + * speed of the migration, but it obviously reduce the downtim= e of + * back-up all SVM'S memory in COLO preparing stage. + */ + if (migration_incoming_in_colo_state()) { + host =3D colo_cache_from_block_offset(block, addr); + /* After goes into COLO state, don't backup it any more */ + if (!migration_incoming_in_colo_state()) { + host_bak =3D host; + } + } + if (!migration_incoming_in_colo_state()) { + host =3D host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } =20 @@ -2983,6 +3073,9 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) if (!ret) { ret =3D qemu_file_get_error(f); } + if (!ret && host_bak && host) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } =20 wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index 53f0021..3a8e5a6 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -63,4 +63,8 @@ int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_ad= dr); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t = nr); =20 +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index 10a6676..0d1f593 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1782,7 +1782,7 @@ static int loadvm_handle_cmd_packaged(MigrationIncomi= ngState *mis) static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } =20 /* --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754669469668.19381484907; Sat, 10 Mar 2018 23:51:09 -0800 (PST) Received: from localhost ([::1]:53369 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvl6-0001mT-Mw for importer@patchew.org; Sun, 11 Mar 2018 03:51:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59218) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgF-0006Ni-7G for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgE-0000kz-7G for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:07 -0400 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:35966) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgE-0000kX-0V for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:06 -0400 Received: by mail-pg0-x243.google.com with SMTP id i14so5252588pgv.3 for ; Sat, 10 Mar 2018 23:46:05 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lxN6SpcWZNUdS22P6NJoYX6YVb5xrWvofqgkGV4t3u0=; b=UbiQFy3Ff/Krtl6gZ0v4XvF5kay/HAbBF0EnRfoZy5vAok5aX1y8yppSmMMnCwgVja Ozw0ieBqQZi9UBEfbOaJ8khbRxVGUN5JBYVGkgTnF4zVnskoyf+W/ykhtiYgOcy47ouu jG4IVHdONVoMrUdtSkzHqqtjW27nePuMUdqf/bkyLbnt8lfqXh5OdvgTzy5aMmyt9VKy MGPHi7ip9FJZ6nQjR1GmCM+oQonZmxhAiJAHY+1NZtfrGNzVrpghW/rN9CVtausRH4zz gXgr9zfoVmaGYjR8XBY/Qbl0tU2lBCycd1YsHznqcO7rtdUIlLEwO5fRawyDFqvlhGFv L+tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lxN6SpcWZNUdS22P6NJoYX6YVb5xrWvofqgkGV4t3u0=; b=eN6xG6514IhIpLJHzNBVQbTgW7kN65vTrLQ53iowDD6kTvSQpcnp8tHYBarpGnJemC RaIyK1iUnlB65S+eXXfyFNONBohmWe6alsr6aGHSvAFaZ93ilMfi4SKBD2fCy0i6cdeN X901DGoHqDVor8wueM5LHtUDjuC4VLUDt/kqatiTkvrOMOjNhg2ooZkeA+Gjgy9Mwxwh gVWchkOMXW4r6EkbcNqr0gSVJVkMrZKpqgRgf+8inNckGF0wPZd5q1ifuX0u76SkLZrH abLSY4FWGsJesEFN55BhByt+JLzxyB6YPfxXJY0a4SyzokvfV0YS8zDesCjKWjEUYeiP 3YAg== X-Gm-Message-State: AElRT7FeJBVI4l8VCqih+fICenqSHtiOlMaLZARe5hy6BwsQEfSEpRZi p/Wp45iOfvOCFKu95jThBW0IqVnz X-Google-Smtp-Source: AG47ELtA03rfALCGlcZwzx/UcoO3mCU8E+cRUYiJxw0mKXc9KCDFxVIehaHdQHbXs0CLpwoRS7EYAg== X-Received: by 10.98.72.10 with SMTP id v10mr4106327pfa.148.1520754364695; Sat, 10 Mar 2018 23:46:04 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:05 +0800 Message-Id: <1520754314-5969-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V6 08/17] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 86f5c3c..4534c7a 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2481,6 +2481,15 @@ static inline void *colo_cache_from_block_offset(RAM= Block *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } =20 @@ -2659,6 +2668,24 @@ int colo_init_ram_cache(void) } } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap to= gether + * with to decide which page in cache should be flushed into SVM's RAM.= Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; + + block->bmap =3D bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state =3D g_new0(RAMState, 1); + ram_state->migration_dirty_pages =3D 0; + return 0; =20 out_locked: @@ -2678,6 +2705,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap =3D NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2686,6 +2717,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state =3D NULL; } =20 /** --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754819612945.1339589677316; Sat, 10 Mar 2018 23:53:39 -0800 (PST) Received: from localhost ([::1]:53381 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvnW-0003wB-Mq for importer@patchew.org; Sun, 11 Mar 2018 03:53:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59247) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgJ-0006PC-Ha for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgI-0000nz-D8 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:11 -0400 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:33496) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgI-0000nb-6t for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:10 -0400 Received: by mail-pg0-x242.google.com with SMTP id g12so5255441pgs.0 for ; Sat, 10 Mar 2018 23:46:10 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XqbOQdGJaaK869FsHU8cyOLMJv+HPHjhy3tE5GWcJ+0=; b=hk9kWxeHputI+fdRYWl1ZBZnRgzbgkIgY0hblHc9dqL5aYnpkW/DtTueTMJQWtXQEO JgErySQVTVpbEO5mt1S1O9x2hc2rGOEEyGDchaYBSVci2OPLW5X3uDcucSwPJAnllwRU xLYZkcJuiMfVihT36fyX9DNXaT6D0CE5fgG0aMDR5fI62VOyk3pTT94iahuB2hvC9zSw uNRkbKNd9+BI92pAFkie7GRAEf/ufYaUtKCPBDL39Ey91k1CczUxWGBBYu8IRn7zMWVa 3DZNNyJmRE0l0epXaWXtsmYFo9Fj7hLMPsBlTUM6zpiQi1FkBW8a4Yy/hzoAuUa3jQ/N sOuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XqbOQdGJaaK869FsHU8cyOLMJv+HPHjhy3tE5GWcJ+0=; b=gLm8GnxNDAugwzxvvP+Te52qYduJKiLazs9iu8GLYgQPRW+73iKhZhkB9/vBvezDRx j2In57wty1biKUH+Oed6vcGcv7qHVWP3q5PFRXKYb7XhHeRjbbcZOYjCEd2OhS0qxend NNCs7SKl+AaKbATUQeQVrb+GPjH6loZvx5aZLDWD+a1afjPLhD5zTbQZpLDx9Yt5/NpB +PR8w4ODFpQ0gOtjDL7O2EUJ+ISsXXvAgbc1NdtBB+eDQnw8qqAX9tgK6IaRXbqgh+ZB V9ZkGE76HmaVIUn2/AZDPQwH3wEuvNdwvTqGWzsWrvroywMebP5SdTOHctgfEicKZVsY L0Lw== X-Gm-Message-State: AElRT7GAQRiNSuhRziB5lhehz1KSOG6t/hIYYUR8n+NsWYid4RVve4ME PNrTFqIN3FJ9SCipVinvJCbCPS0V X-Google-Smtp-Source: AG47ELsmQB7ohrK+yUxY4kQ9pPUtt9va2iAmSYLLaolk2X5TpRBX1mQVGFBF83FnOLQJHNSs/vbUkg== X-Received: by 10.98.152.205 with SMTP id d74mr4124281pfk.115.1520754368862; Sat, 10 Mar 2018 23:46:08 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:06 +0800 Message-Id: <1520754314-5969-11-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V6 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkp= oint time. So, the content of SVM's RAM cache will always be same with PVM's mem= ory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 4534c7a..ff7265e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2927,6 +2927,40 @@ static bool postcopy_is_running(void) return ps >=3D POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_E= ND; } =20 +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block =3D NULL; + void *dst_host; + void *src_host; + unsigned long offset =3D 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + dst_host =3D block->host + (offset << TARGET_PAGE_BITS); + src_host =3D block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages =3D=3D 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags =3D 0, ret =3D 0, invalid_flags =3D 0; @@ -2939,6 +2973,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) bool postcopy_running =3D postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability = on */ bool postcopy_advised =3D postcopy_is_advised(); + bool need_flush =3D false; =20 seq_iter++; =20 @@ -3114,6 +3149,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } =20 diff --git a/migration/trace-events b/migration/trace-events index b27a3a0..a3a124b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRI= x64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset= : 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152075483102930.021630605428527; Sat, 10 Mar 2018 23:53:51 -0800 (PST) Received: from localhost ([::1]:53382 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvni-00047B-56 for importer@patchew.org; Sun, 11 Mar 2018 03:53:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59266) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgN-0006T4-HW for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgM-0000qV-JY for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:15 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:41355) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgM-0000q1-D2 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:14 -0400 Received: by mail-pf0-x241.google.com with SMTP id f80so3040525pfa.8 for ; Sat, 10 Mar 2018 23:46:14 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GD6x7tdnuKsONz6vJ1sF07NbBOLox/mgTLZpcyl+d1w=; b=CJnOHXk3ttK72RpIEK3drqddQVpgrW+25eEgjDOLf6wXeLAkeq3Q9FjQ1sPLfC3jr6 aB0Tni+y+eXHBGtN5PHAJ/cNErpN1pEMwHoqU2ff/Y4AvlQgo/8+ek24VrVpKjh8HVXC B1s1QLvYt0OGOMVVufE0wIJEQmaq88QNlLfFi80Mdw+XogIDZ7JL5QO4HRlx/zX4gp7X 85bmif/N2l48ackvt1I1oVGZrztWWl3niTiF6k9nDc28A/+RRqd1VC2rUGWsjgEHPm+A uV0r4L5YaSWKkaMMW4zho8+1aJ/E+AiSbtJTYODe1015ByUoiS/sgDRHy63IIXPnRVMp Q+6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GD6x7tdnuKsONz6vJ1sF07NbBOLox/mgTLZpcyl+d1w=; b=HNVX8QShO9RWYQxC5FHttP9pusnzYJCvUhE3DCsX3PM3eks/fnHdVmY8pvUMc7Sg5s QQGfZKYSur06JvQBuRnlB5o0VzuHriNVWu/xeQZsHHgHwnQ76ostTuNZYsKXG1YisJE4 eyfjz3zVZD+sWP54ACzRRXlSzu3VMOYxZk05RFgrxznCa8c+HWatt6K2XbsautOyTygC DZenUIsFNBnTF+OFgDgyCkGKma+4ycmFABmNE7CBvi4iRH9ecgle54w3x4jzc5APnwLf gltGAPvcfKGeR+dTHokms8o1TjxcZWbmmTM5Uh6kgua8+iAyPfgVbcxjMZ8pZfLDQcQu Bj1g== X-Gm-Message-State: AElRT7F5SiTgLKxrd3v93h9guje5HmI4ebX/0GCV/bxhSs/kHfd7MJSb +tyUDvxY+P+muWj/y/TGvoD8LIVy X-Google-Smtp-Source: AG47ELuaTYRlf5aM8MUpyiWH70jfNOSvdDIhZEZfqZbvomGf9dtEXWFHeX92bzCmk00bsQOPcoTdbA== X-Received: by 10.98.70.89 with SMTP id t86mr4101978pfa.215.1520754373058; Sat, 10 Mar 2018 23:46:13 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:07 +0800 Message-Id: <1520754314-5969-12-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V6 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c | 20 ++++++++++++++++++++ qapi/migration.json | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d36..8ca6381 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,18 @@ out: qemu_fclose(fb); } =20 + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -744,6 +757,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } =20 if (fb) { qemu_fclose(fb); diff --git a/qapi/migration.json b/qapi/migration.json index 7f465a1..6c6c50e 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -858,6 +858,41 @@ 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } =20 ## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.12 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "reques= t" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.12 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'request', 'error' ] } + +## # @x-colo-lost-heartbeat: # # Tell qemu that heartbeat is lost, request it to do takeover procedures. --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754948988901.3267448216353; Sat, 10 Mar 2018 23:55:48 -0800 (PST) Received: from localhost ([::1]:53395 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvpc-0005uB-8R for importer@patchew.org; Sun, 11 Mar 2018 03:55:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59309) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgT-0006YW-TU for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgQ-0000tF-Ra for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:21 -0400 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:37267) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgQ-0000so-LT for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:18 -0400 Received: by mail-pf0-x242.google.com with SMTP id h11so3039964pfn.4 for ; Sat, 10 Mar 2018 23:46:18 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=acZC30ja156O5Gnyj/WgqIdYeX5z3oaQRSH1m++FKfA=; b=FV630v78OPlCWZc2C8XxtKW6I0gmujscwftiU3X8F3WFJCf8OWQ0u3jXYIOvjRJma7 VWp0MHU7Ek/8O/a8Yd7pYX9JQyslA29xVP65LY4z+THa/rwuJ8UVda471hXkX3wwHPFY cadMqMj1JaR3Yf03rDWqzoxTXIhHIECG8YYVfQxsxRQMfumi5bom+qoNslnp0lIB/5cy QyH8jM9w1BmGi3EYjvCAEg7MkZRUIT89kXDVsOJtemRWLLnZnA7ynETc6DVUqJJfgW58 Tw0lj5oCeIdMv34odSQnAqrgew0RqDZiPf60TRNOsvk3bZl7UrZ04HBpYzaJzt+R4WaD PqeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=acZC30ja156O5Gnyj/WgqIdYeX5z3oaQRSH1m++FKfA=; b=BJ8zh5gl8zFkFGHzpMPperkndCC8hsZCg7TSMUl75gKjPoJ8IAsDawM7seglEvmRyQ gbQF1DlnLZL2jZjLtzj2AeK8fpf7BHo47NnKsMn1Q+b0cN6+ECumVfsuTJBNYxRAsVoT cJRs7eGsgfcF2AKm8uG3k9wUPksEZ1KzaQ2z+8IEwNIGOLXJIRt5OLRUe72JKtaflgov kI4JT/0YSwBpi8fBW6zei0+0TV7H+RfUpOnykV4rhqr7NfDT/21ms9EwTWU7K+1s7LFK D71cyvy2gj719va1vJy6x9QkLIoT7GFIke/mP0c2keDwHbH6qdwGx+jR71LQMk9+8yKm antg== X-Gm-Message-State: AElRT7EGlt8j00oycBFQ2Nemz7dw0mK+fwoqHpqWZRA4mUjqdmudkl0H K2Cbuz8L8bCv0ncVaGncdvpH1+gK X-Google-Smtp-Source: AG47ELvOIsZVeXENnpsN/cC1w8s3SMhIcBFEBdUDtwnnmVKQ3S3kz+UwIeSDOL8ASIM6qIF1Ey34yg== X-Received: by 10.98.65.198 with SMTP id g67mr4113548pfd.127.1520754377343; Sat, 10 Mar 2018 23:46:17 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:08 +0800 Message-Id: <1520754314-5969-13-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V6 11/17] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8ca6381..127db3c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -237,6 +237,37 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } =20 +COLOStatus *qmp_query_colo_status(Error **errp) +{ + MigrationState *m; + MigrationIncomingState *mis; + COLOStatus *s =3D g_new0(COLOStatus, 1); + + if (get_colo_mode() =3D=3D COLO_MODE_PRIMARY) { + m =3D migrate_get_current(); + + if (m->state =3D=3D MIGRATION_STATUS_COLO) { + s->colo_running =3D true; + } else { + s->colo_running =3D false; + } + s->mode =3D COLO_MODE_PRIMARY; + } else { + mis =3D migration_incoming_get_current(); + + if (mis->state =3D=3D MIGRATION_STATUS_COLO) { + s->colo_running =3D true; + } else { + s->colo_running =3D false; + } + s->mode =3D COLO_MODE_SECONDARY; + } + + s->reason =3D failover_get_state(); + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index 6c6c50e..8ccdd21 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1201,3 +1201,36 @@ # Since: 2.9 ## { 'command': 'xen-colo-do-checkpoint' } + +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @colo-running: if true means COLO running well, otherwise COLO have done. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.12 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'colo-running': 'bool', 'reason': 'COLOExi= tReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "colo-running": true, "reason": "req= uest" } } +# +# Since: 2.12 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520755051142682.5043156526979; Sat, 10 Mar 2018 23:57:31 -0800 (PST) Received: from localhost ([::1]:53402 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvrG-0007Hu-BX for importer@patchew.org; Sun, 11 Mar 2018 03:57:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59325) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgW-0006ay-FS for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgV-0000vP-5y for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:24 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:37327) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgU-0000uy-TE for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:23 -0400 Received: by mail-pl0-x244.google.com with SMTP id w12-v6so7606761plp.4 for ; Sat, 10 Mar 2018 23:46:22 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3ulfaou2re/01f94Yo2+jBdgKNZg8UcfXzE4VLleK8s=; b=FPwrXeM5Q7TdqjsZUH7rslfHC+alj13ZaP6uzI7VvHXN96zOKnYEK9RxbmpgFhhy11 v9ZSQb1ROn5D1OfpK7EPdWJa2gJr7gh4P0KhKQ7Ig13xuRByLcfFXFQi7YkeuC7n3/RE LC9QwKJIH5+8hfFEy8D5rmhpdryVDssHoij4wQ2+gUKRjoS53lK3ZPbhoKdWp52K4Prq 3ebQk02H47r33VFTWCKXeQdeB4+d/CdlfS5JVZi8ZyLz9brfn8uZW16ersdG6Rs1pC3g 0UBaE1Juw8dTf40xuiUFy4esHCVvuK+Pmbc/0DXxYxWDVvTl1pibjnEt/y5Yc+zoTRqN TPWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3ulfaou2re/01f94Yo2+jBdgKNZg8UcfXzE4VLleK8s=; b=sGwBWOIGhqccBxNvq3V+xgC2uogJi31wPsmOZ8/Ss7Gh2rJGuzJhUrGzsg4DsvOx8s X7d0IgJSjoPf9WSteu1O1R9QdI0mOwBWjQX2HBLEA2iTaa/ANkmBB53FItvyE7FADlI8 boXPYdtst10gAV5kA/xt1D/Z1Iy3gayVLqDKRnTbjYeWCzrJJIbARkGGyo3BvamjWnHC Rj4xC7OJBtmco2UnOhwyeBaHIHKeWumcgzL2MvrYyd3M3XoZYZnTLNChybqh6J2uvtSe b3KcNz9Eo4SS47c/rw1F1UZkVBwa9+GstFYjVrhfz+Of3rTlRw0R/LWJ15WAIZuRLe4O qBLg== X-Gm-Message-State: AElRT7FoQBKv3hUnLHTfUExETSNpY+3OKoWu61EqrmB18eMAqYnv6PTA wtOcz8TfQsmmGraZDfz6xHF5pEMW X-Google-Smtp-Source: AG47ELu9pq3rT+ZIwEiMAsrFFutDu1i6hHx6Cyy2U9HlmUIDy/x7SmoZDTZxTC96eu06i3p9BZ3huA== X-Received: by 2002:a17:902:8487:: with SMTP id c7-v6mr4349853plo.143.1520754381525; Sat, 10 Mar 2018 23:46:21 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:09 +0800 Message-Id: <1520754314-5969-14-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V6 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 36 ++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 127db3c..f514492 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "sysemu/cpus.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -411,24 +412,31 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, =20 /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); =20 colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret =3D qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -640,6 +648,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err =3D NULL; + int ret; =20 qemu_sem_init(&mis->colo_incoming_sem, 0); =20 @@ -712,6 +721,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value =3D colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -745,8 +764,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading =3D true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret =3D qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 0d1f593..e347f73 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1322,13 +1322,20 @@ done: return ret; } =20 -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} =20 - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; =20 + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1384,8 +1391,6 @@ enum LoadVMExitCodes { LOADVM_QUIT =3D 1, }; =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis= ); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2050,7 +2055,7 @@ void qemu_loadvm_state_cleanup(void) } } =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret =3D 0; @@ -2204,6 +2209,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } =20 +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 041d23c..8d463fd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,8 +52,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, = const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); =20 #endif --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520754968777653.2607074118615; Sat, 10 Mar 2018 23:56:08 -0800 (PST) Received: from localhost ([::1]:53397 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvpw-0006GD-2Z for importer@patchew.org; Sun, 11 Mar 2018 03:56:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59361) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgc-0006eR-BZ for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgZ-0000xW-8p for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:30 -0400 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:33497) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgZ-0000xD-31 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:27 -0400 Received: by mail-pg0-x242.google.com with SMTP id g12so5255580pgs.0 for ; Sat, 10 Mar 2018 23:46:26 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1Ng5StD/qge1fH0589n/nBkXQnchc0g+28X6DXcdkB4=; b=DUKp+QVgv1s4DQN4bexaJvM7MK8EwycHY9p5dvZ5cHJEst6FMhvofpYRT4uC+kEbW1 GXPX/d/20Sb5woIi785z34SKEAMuN++bWljdfWTGyzWIpt7t+FEzGfNZwUwH4AvAgJ1P VTX2gLAZndaE/Qc1yzNHsFloEImMREkzt7ugoCNgtKgYTIQrfOv5WL1R5QzAz8t3zBG6 LadUHO2A9fgGzVVCdMKYC8wPnwpBdRUBbuK6fvhufeemSWrWHdKZSPl74rvYS3KGqu14 AwbrQsDm4288Rrvgqw+Fu3KyyroudRR9Ijg6oZZETkKzTQoW4NwsgRlrM+ojurLQgbTq /pBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1Ng5StD/qge1fH0589n/nBkXQnchc0g+28X6DXcdkB4=; b=ql9HGrrTt0ZTDYodglNUDvyJ9avkTPlDtjzeBHKOMfUDVI4PgYKqTc4J5RScWrA/Jr o9I5c4OjyAIh914ynejuOhhaKyt0Gu4nLOoH0vFZFb+mfmP8OHXt+byd1n7ZCF8z3WsO +rHrT6lgeAk5Ku5NjaaKCfvKA7mIULpp50nsrVHBa8f60MoK1lsUpzznwM/ZWrGCZYe2 j+5JPWhp05XVyz3aGNlwnMEfCXchXM+6Kgacdj64CWZpW9nVL/O3gISSpOX9GziQquGP sBDUby6E/N/BnTsUP32HkPgsuqhPQ/97FxUJ8FGutUamhFUOzE5jO5jghq9SXlx1TIFB dHaA== X-Gm-Message-State: AElRT7G8QA78IUQLxD7KZAsmkhtn/zyyCZWvMbWgR53YIgHlvZvBV4Zu rIejNMAhxjOAeCofEcfQNuSTovKm X-Google-Smtp-Source: AG47ELtX01wzN6h6oGaAye4mModZj6ZTZrSwLwFOP8q+cozfXOVbTl3cCYd+QB4Ci8pstJEJs0uu6w== X-Received: by 10.99.127.65 with SMTP id p1mr3460143pgn.141.1520754385870; Sat, 10 Mar 2018 23:46:25 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:10 +0800 Message-Id: <1520754314-5969-15-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V6 13/17] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index ff7265e..003617e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2685,6 +2685,7 @@ int colo_init_ram_cache(void) } ram_state =3D g_new0(RAMState, 1); ram_state->migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); =20 return 0; =20 @@ -2705,10 +2706,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap =3D NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2938,6 +2941,15 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_lengt= h); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state->bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block =3D QLIST_FIRST_RCU(&ram_list.blocks); --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520755146690768.082965572424; Sat, 10 Mar 2018 23:59:06 -0800 (PST) Received: from localhost ([::1]:53413 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvse-00005O-0R for importer@patchew.org; Sun, 11 Mar 2018 03:58:56 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59375) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvge-0006eY-1H for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgd-0000zu-1d for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:32 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:41507) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgc-0000zU-RU for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:30 -0400 Received: by mail-pg0-x244.google.com with SMTP id w17so1178332pgq.8 for ; Sat, 10 Mar 2018 23:46:30 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xGJfOtyitB72l6WyXk30MaHec1wKPM9nbCkyT8g5f/A=; b=nXjK6FH9wZ+8rh19dfxk0YUj2HOxCHKZX+KjtYj2n66/HVUUGE1DseSCDXJm3tzSGn V20GcG2dtrbx/+8zLb7Wy+3R+jQYKhBdmGoeaDVP7eTMhm4V3A/45Wd83ZScxyp2ZHeI k0Gkhfv3sV36ZxBXC04zuPybPfVqakwJYe/0aLZV6F4eupOSXSEm9Ctru7G0lMQ/8omj t+o+K53ZVZuuJ7lPX8kNkDTG9GplaiNThohM6tMgFRliODihKWCYeNRY6F0EbMCndrti CnEiZIK0LsUODMmh1NZC5yEQz7Ea+IumwF9K5AglJtiBTCxN+kXIR02X7Q8tbyCS2Vnh 4CrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xGJfOtyitB72l6WyXk30MaHec1wKPM9nbCkyT8g5f/A=; b=SfWVCGo9AIZ7JrgIqylfzINd+ji9EIW7mVOki5SLhtVqbAAIcyrvDXqRUJPS3DYT+e htGQDwiGnA4WbNvFZqJsTRFIyNtrDoZDdIt/w00kEpfRAdwHahNzFoVbtEZDFTfH75sa xoomxj7cSRuCn9c23g6cnTGGLRawqvvLNC3TwciQmWLSIS6X4viu9pQlAq0CBgb7Lr7j /53SMgoyPSBqvqKf1/dU80qBXGs3CVZlnqrdjmbKp0TPMylKGRWm8VSk6uFrZhuBnxsQ yMuXW3Ff+r4CZPoVEMblZ/YLG8EBV6WtEaxz0VUMlNXq6xGadUI4ScN8qPjb0ODQw5FP puhw== X-Gm-Message-State: AElRT7GBn0NNnwM1jjZphKXhMBpakbGUb0/AEcN+bCZdrz+CR+3DC1s0 Lql1fS6APoNoS3OcaRnC52e/Uzob X-Google-Smtp-Source: AG47ELtGarmukX2qjjzj/IfjsPyx69AZM9K8p7bCJxKI5ToZLm7UoClDSSMSosHnQPP+1dmwRWaX8w== X-Received: by 10.101.93.82 with SMTP id e18mr3357882pgt.371.1520754389642; Sat, 10 Mar 2018 23:46:29 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:11 +0800 Message-Id: <1520754314-5969-16-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V6 14/17] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6..49da666 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, =20 typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); =20 +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **e= rrp); + typedef struct NetFilterClass { ObjectClass parent_class; =20 @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sende= r, int iovcnt, void *opaque); =20 +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d..0f17eba 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" =20 static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } =20 +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc =3D NETFILTER_CLASS(oc); =20 ucc->complete =3D netfilter_complete; + nfc->handle_event =3D dummy_handle_event; } =20 static const TypeInfo netfilter_info =3D { diff --git a/net/net.c b/net/net.c index 5222e45..5bdbd6a 100644 --- a/net/net.c +++ b/net/net.c @@ -1336,6 +1336,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdi= ct) } } =20 +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc =3D NULL; + Error *local_err =3D NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer =3D nc->peer; + type =3D nc->info->type; + if (!peer || type !=3D NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =3D NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152075507090078.56384931321134; Sat, 10 Mar 2018 23:57:50 -0800 (PST) Received: from localhost ([::1]:53403 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvrZ-0007VB-SM for importer@patchew.org; Sun, 11 Mar 2018 03:57:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59404) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgl-0006l9-3x for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgi-00013D-0q for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:39 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:35567) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgh-00012C-Qd for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:35 -0400 Received: by mail-pl0-x241.google.com with SMTP id w22-v6so7601574pll.2 for ; Sat, 10 Mar 2018 23:46:35 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=krjBqgTGepsjEspcz8jZq5t+zPYFaaJLMjzJbngN1Vw=; b=ohZISyGmd37q+ACVKHMe6+0Lk4caqU3hf4z7ddKzIgfmNqEa/ub12tR4S0tQWeM1ld ++e/M4vffcRq2TMA5anqvDMlEAyKHl0aIILeAyXWJycg1/CdMupQ2yGbULb3/5InDErU UC0XyTXSx6YcqLi95ZXWZ9bRUdd/Yl5zKXjSbWy5RxJW7He/QA6aA5LtP40oXHcWuMqs ksSijr2CWijpUDhbHBf9WUzsv73jHyg/Rqst7jxQtDSgK295JtyLyOtfj3le7lq8Aj9W Y1bnD9dCv564EKqfUOuVaIOaq3A11kSlVKdb8dvNn2EipHAXcKD9YVVLJJwP94al/4fT IOow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=krjBqgTGepsjEspcz8jZq5t+zPYFaaJLMjzJbngN1Vw=; b=eAcAEH6giF+RYp2k4gyKTPB/XmZZ0nD4+jOkv0o5FohySc97ckAUg7Rrg9SAEU81uO vS0sdCI+TI9ztfdnQRBR9VTLxs7qum79Bav1mg7SY5gSRsmR9FsZ3Sa9tHrDR1f2FCJi /kPq754zZFqP2GUnZxumrLnj9i7DHn3leLBiNqz0UaFUUVdcj+e2EIvZgM4mwMoQqRrq /W2cgxNkv66hq/gGKcEUvByVJackWKfmHmt6v0NQmHCA4FFvxgJ+yJc6mm/uxRFYOR3Z lGPL4MLrgcAKLeqzancT3to6G3qO4TszIdngHL2T6lcaDmm80GUT6U2gIKxqmcRpb/5N o7yQ== X-Gm-Message-State: AElRT7E0b+ftNRhPhDwOYzqCpzj+ty3gogvX7a1s1OnY8uHT2Q01+e1T HcLG39VhuYxla2hBMVdh1bM7kZnD X-Google-Smtp-Source: AG47ELv51iD+8NRqRff+v/LxzPxpbIQc5nSiX7WLzo7VhtwiNQ6SW6DQupOsRsqGGTTtVB87B5Ez3w== X-Received: by 2002:a17:902:c5:: with SMTP id a63-v6mr4252068pla.391.1520754394505; Sat, 10 Mar 2018 23:46:34 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:12 +0800 Message-Id: <1520754314-5969-17-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V6 15/17] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index f514492..f3d588e 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "sysemu/cpus.h" +#include "net/filter.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -81,6 +82,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -783,6 +789,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 0909a9a..f3c306c 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,6 +20,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" =20 #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -277,6 +279,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterStat= e *nf, return 0; } =20 +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_d= ata) +{ + Connection *conn =3D (Connection *)value; + + conn->offset =3D 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn =3D (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs =3D FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s =3D FILTER_COLO_REWRITER(nf); @@ -332,6 +371,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, v= oid *data) nfc->setup =3D colo_rewriter_setup; nfc->cleanup =3D colo_rewriter_cleanup; nfc->receive_iov =3D colo_rewriter_receive_iov; + nfc->handle_event =3D colo_rewriter_handle_event; } =20 static const TypeInfo colo_rewriter_info =3D { --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520755217206110.77425458025505; Sun, 11 Mar 2018 00:00:17 -0800 (PST) Received: from localhost ([::1]:53418 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvtw-0000sy-H1 for importer@patchew.org; Sun, 11 Mar 2018 04:00:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59416) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgn-0006nC-Be for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgm-00018h-Ji for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:41 -0400 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:40874) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgm-00018M-Db for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:40 -0400 Received: by mail-pl0-x243.google.com with SMTP id x3-v6so1087412plo.7 for ; Sat, 10 Mar 2018 23:46:40 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0tTQpIVyCNzXoNXHf3yDuCTrarUzSq9WY7LvOpdkSRM=; b=LxJ1hL8Bc3irdKR+Bb2FXigE99fzRykJle0B32qnAuEarZJC3JnvsPWJIfcKrxKG7c xoZ8+youpmioY65/Pq1Ntf95MpnVESQ3O8acSpfIduVftMDErHGirB0CBc01vY4uTZrf qz5IMlS/y1l/BBJc2VHrajHpk90l8ggBj8qbP4S0NjxOzpMHEaX/aXh8W1j5DGpwivzi xmiKbrs7dOm+NIF9LKbrH+tdneMvKiHteFMUur6ZwDkxtkoiLdP8Oqgcx8+nGOzt2XN3 c+dzN6iUfxKqbW3u+pjTFiAV7/0RyWC6FcShUa5wfVvS1yFpr81qsOaav8ZD2gzP6uav hFOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0tTQpIVyCNzXoNXHf3yDuCTrarUzSq9WY7LvOpdkSRM=; b=fiMI7XhRlZAksqXTsdnSbG2MJW01FLiRLyW209uEqj9i47aBvZxgt7U7z4GyT1hBVq b5WPMFKXutwAxO0LUzrJEiedmQxNzCreLdOXrNGhCmf/6Q3TiyN6j6BkuPohE8MVNLt+ mRPjZG2W/hZIh6KNcW2CerIcVyV0Nt5C1AucT3ULjcGN2/tsEX4EQmkIteOpD3JWBhQs P1cOyLbyu4OC7z3UMSAx4uaa4Iga++9cefJpoOcuYxMwqQqKZWVQP8HEq0beZNheIxY9 A5njdex8Kk/Oxc0zALxut4uRCwZdVX/SUA8eihI4Fud0OYcyTn1EF0JxikXHuBRU1ozE cdBQ== X-Gm-Message-State: AElRT7EQeIx+N/U+9o3FYmzfxLjN0hD49yk9WbxaQGnJiOij+vAJmrAV BVkg9jSl4oG9uacL/DWM25xWzrwA X-Google-Smtp-Source: AG47ELtmJ+D10x5VusHbqItT57kfsHcjPomAPjWdE1oGExCzDwVoXYQIaOIAYrikdGMaf7fUwW3pEA== X-Received: by 2002:a17:902:6b82:: with SMTP id p2-v6mr4300319plk.326.1520754399149; Sat, 10 Mar 2018 23:46:39 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:13 +0800 Message-Id: <1520754314-5969-18-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH V6 16/17] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index f3d588e..1a59baa 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -87,6 +87,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -796,6 +801,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); --=20 2.7.4 From nobody Wed May 1 08:38:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520755326008446.5963388751703; Sun, 11 Mar 2018 00:02:06 -0800 (PST) Received: from localhost ([::1]:53432 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvvh-00026W-9W for importer@patchew.org; Sun, 11 Mar 2018 04:02:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59450) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euvgu-0006tN-7o for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euvgr-0001BX-19 for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:48 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:46063) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1euvgq-0001B1-Qv for qemu-devel@nongnu.org; Sun, 11 Mar 2018 03:46:44 -0400 Received: by mail-pg0-x244.google.com with SMTP id s13so2337868pgn.12 for ; Sat, 10 Mar 2018 23:46:44 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id t16sm10917037pfm.69.2018.03.10.23.46.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 23:46:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=618m6dLFPqsd/G6u07N4YC+vNq5frrRpcsIx6mQsyjs=; b=bmz1d4Mah0xQ/MP70Tmy2X/bP1nDKusjYfWpJhsYjAnnCudyisq4JJpWGH9/ZAOnmg E7iZwAkUWARsW+GlBxr7B0RLR2dTfv0HXt0aN+x6aETo/iQOYCqekDFE/sLlA6EF5Q/+ 4WkpaoRtgC31zRrJ1ELWPffZfrsZPxxtNEIRyxvDvnzHNMA9B3GoMsIjXlCWp3ayGwLI r9jrjzMRKvBIOpL14aBT0zZCvM1k67WCd+8yW5WDSksG1UzDJNuyPdCrrwKCpsLZD2O9 gWopaSmCEFtU+pewoksSzI/3J4HkDRF4Gfyr6G2Ypf59XiS/3xzAUDJAY8VNbnMWWH+4 t/Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=618m6dLFPqsd/G6u07N4YC+vNq5frrRpcsIx6mQsyjs=; b=K7wKZUX+0JSlmNNVZ9E6QNZd1fHKEUDJpOGcBDN7to8MQAV6IyIoQP4u9BNBbzVJQS Q+AkAIU/QQf1k4zo5AELmVG3dhlVBWHNxmqi4am0XBEeei373aWtyvenRRe6uKTQj4iP zbZExdTwvr/BLLwY8JJkMNaPX9DPuMcVBtleA+mr3HRq4hE2Nt0OgpL6EtSly3rBAeZO n4L6Nfv7vcQZ6aP99F4bkXsrO31sajIk5aNnzr4G20CbRjtwHQXAFBOZyxT2Hz8BxQzV Qlmml/KMrvzJAPEZs7RHWpq/pY6+AQkLPj7C334bpLLAd986O5EL9Hp0K+isMAPrUnpn OvWw== X-Gm-Message-State: AElRT7EAHCL2pkDzg0t2BUduAGZVIHASrYkRsbE5fl4H9WMu7YCCa8BP fT0xSu7bk9s506s72h1QGjDO4MSY X-Google-Smtp-Source: AG47ELtV50vy5TDTBm9laW2RzrVAJhCYQ45+RBV+IvrbL//wE/Rr86Jh5lXqo368fI+C2kNYcWVing== X-Received: by 10.99.109.136 with SMTP id i130mr3452268pgc.380.1520754403570; Sat, 10 Mar 2018 23:46:43 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 15:45:14 +0800 Message-Id: <1520754314-5969-19-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> References: <1520754314-5969-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V6 17/17] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 1a59baa..1bc5f02 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -134,6 +134,11 @@ static void primary_vm_do_failover(void) =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); =20 /* * Wake up COLO thread which may blocked in recv() or send(), @@ -549,6 +554,9 @@ static void colo_process_checkpoint(MigrationState *s) =20 qemu_sem_wait(&s->colo_checkpoint_sem); =20 + if (s->state !=3D MIGRATION_STATUS_COLO) { + goto out; + } ret =3D colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; --=20 2.7.4