From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516369719413117.34773703041355; Fri, 19 Jan 2018 05:48:39 -0800 (PST) Received: from localhost ([::1]:50332 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX22-0004sx-0K for importer@patchew.org; Fri, 19 Jan 2018 08:48:34 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46548) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecWzm-0003X3-Ea for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecWzh-0002bt-Gd for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:14 -0500 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:44423) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecWzh-0002bZ-8Y for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:09 -0500 Received: by mail-pf0-x243.google.com with SMTP id m26so1386297pfj.11 for ; Fri, 19 Jan 2018 05:46:09 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wkt/Ox6pwMzQEiOeOtqTem016VC/LjxH2opuj5feFeQ=; b=aSu5hXvIVTU7WzpMZeoItcWrVLCQIdRekXJ6omJBSr3/iKQpPh63vHacqmb6osD0UY bVvGn8/y8TdyKbrCcHzaLeK6rlPcw/Owswke5n9UTTTvLOHn82rD/wM8NppUNVQGB3gN DjxBd2qi1Lozy/Ap4wCET4lt3o6BQBgVkju4lappCJFXQdmMZ/HmYZzwVik3/n+oxMzv aENEv2KolVT3QJR9LDYqAAi23/B9HolQCDtjxSr2Ii8gBs2J6Mfp/4kdekgND6Bw9jyx XeM5y7ZF1wnnPO8Eatav41nG2YQH2t3zIDrtKNPjZk7wSpMT1Kl0PGyugVhKcMbk6U09 RvvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Wkt/Ox6pwMzQEiOeOtqTem016VC/LjxH2opuj5feFeQ=; b=QCa6UzEs+HwDT8A2bOx8STXnnHRG/hiZofmI0qPju0jk14ASa6OdpOwtTaMi/aJett 9j1ksbQY1nO3PRvd1Eh5DDDn0b3GA0CQd2QmkH4XuTHeVsFw6tqllwdSlVr401dUcicm Ye6FXcRHVDFIoikkGcdQ8ql1u1yMP1anqe/KDscErDBGDe4wTZyC1rNy9BJ0ZVUvpjXp kr7IypuFO2QEclA6jCp7wXkk8/eg4v2o1yYD3I1bqCDrKELhjeKTRkouJ90xDfLwxqJy 8MC2Y+pYyy1XIwvHi4YFGydhwFZDSwzvYkYqLcI6WHierVZb1phsoUiMJuMVy3TzXel+ 3skg== X-Gm-Message-State: AKwxytdGCbuvTVOkGo0HxD1SbfVg4gIKidu6bgBO46FGuPXCL8xN53rS 1CQfehTLlB81NDyvRjb5rEg2b5Dd X-Google-Smtp-Source: ACJfBos+LjY5KMHVXru6yD1incvkA8+vSJ/F+SR8AFs2V1ZztPYKBVV/hiLHlBNTn9pIL2AL3L6+fA== X-Received: by 2002:a17:902:b288:: with SMTP id u8-v6mr1553901plr.291.1516369567602; Fri, 19 Jan 2018 05:46:07 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:30 +0800 Message-Id: <1516369485-5374-2-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V4 01/16] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 +++++++++++++++++++++++++++++++++++++++++++++--= ---- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index 0658e86..0193935 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" =20 #define HASHTABLE_MAX_SIZE 16384 =20 @@ -71,6 +72,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=3D1,ack=3D1' */ } Connection; =20 uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 2be388f..a58310a 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -62,9 +62,9 @@ static int is_tcp_packet(Packet *pkt) } =20 /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -102,15 +102,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=3D1' packets from client side, it acks 'fin=3D1, ac= k=3D1' + * packet from server side. From this point, we can ensure that th= ere + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this ra= re + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_ta= ble. + * + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from client side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D htonl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } =20 return 0; } =20 /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -142,8 +171,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=3D1' packets from server side, it acks 'fin=3D1, ac= k=3D1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=3D1,ack=3D1'= packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from server side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D ntohl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } - return 0; } =20 @@ -193,7 +248,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, =20 if (sender =3D=3D nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -206,7 +261,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516369715785750.0216498494306; Fri, 19 Jan 2018 05:48:35 -0800 (PST) Received: from localhost ([::1]:50331 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX20-0004sm-1D for importer@patchew.org; Fri, 19 Jan 2018 08:48:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46549) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecWzm-0003X4-Ek for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecWzl-0002dc-04 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:14 -0500 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:41983) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecWzk-0002dH-O9 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:12 -0500 Received: by mail-pg0-x244.google.com with SMTP id 136so1458621pgd.8 for ; Fri, 19 Jan 2018 05:46:12 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Hu9Feno9vRYER8WMKcaLr1DBR5y+XFmncbgvti/KUhs=; b=meoRyNcAG2CEYg7+pSAMffurB+HME9fceh0eeiEVxpVk7y7zaiReXJWrLJN2kSiFeE jsugzJQgm11B99bdk6tbOlOg0cdXIZVlJWfsoT1FQp4ZLprHWuz2qrFtaCIKsPRCwnrs 7xnW1QcpRUOGA2oPg7roaq4NYdnEh2vJuAslTd6fm1LVsKZKpn4U0fNRntDpZoJbjRr8 CuFb2vD0KbxiRj/+suwJiSTPdTaqcE4xBcW5hrTxDjAqk+lTyN0iwHmyyW0ysyzGwsX0 uWQ72K1i1pKcUM0phgUxKhxU/J6tL34gEE1iSI0B02X267LdI7ozmxHWUcV8Ouda117p dpow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Hu9Feno9vRYER8WMKcaLr1DBR5y+XFmncbgvti/KUhs=; b=CyOJ2WmA2y2BnyNo4/RK1VS22A6TgRE9HpxTlML/DJCcpzkqaRTKeTN40/RjOPt1dv 1S/oYaenLysEhylcww3HChwY1SGmi1LmxU17jZ6MBJu1QUa/Ydb8Q06ZhFYJEc2QeqAC FB0dXvkXZu+TQk9mYA8AiviJcILe1ptYxnBAy7B0IKrX4hKgMrKoM/Yn2P36f/ObZPj3 5fq9h/ueHbQxXA6bLJlM8ONRQes75pw7OdBr9B8rK1VvagkakbdxPTLxteebmpSgMmum X7ZF268PMpkh8M2I02nu1hbILk3QcZe1Q3Vjz9QwwFSuoUj4c8HPuISuR00bgBA0KtsX kHjw== X-Gm-Message-State: AKwxytfUqcSWaUoNAkhorVbnyONUTbVn22KVWWJk2RmRZfDC+vDUh20M eOZNy3CsNGUUU9nTHaO18qb6P7DP X-Google-Smtp-Source: ACJfBosLH8ZyXcSiMLQJCmP2VUkY5hPIVbt/iXvDI0+yOKkbDva8z+sCeAm/1xzDPB5RC5C1c3OwTw== X-Received: by 10.99.125.74 with SMTP id m10mr26448995pgn.354.1516369571389; Fri, 19 Jan 2018 05:46:11 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:31 +0800 Message-Id: <1516369485-5374-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V4 02/16] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++++++= ++++ net/colo-compare.h | 22 ++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index ff9874e..6adf3a5 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -15,6 +15,12 @@ =20 #include "qemu-common.h" =20 +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); =20 void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 0ebdec9..4bceca8 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,17 +29,26 @@ #include "qapi-visit.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) =20 +static QTAILQ_HEAD(, CompareState) net_compares =3D + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 =20 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -86,6 +95,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; =20 typedef struct CompareClass { @@ -631,6 +645,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } =20 +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event =3D event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx =3D iothread_get_aio_context(s->iothread); @@ -651,6 +684,28 @@ static void colo_compare_timer_del(CompareState *s) } } =20 +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s =3D opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -664,6 +719,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); =20 colo_compare_timer_init(s); + s->event_bh =3D qemu_bh_new(colo_compare_handle_event, s); } =20 static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -821,8 +877,13 @@ static void colo_compare_complete(UserCreatable *uc, E= rror **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); =20 + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); =20 + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table =3D g_hash_table_new_full(connection_key_has= h, connection_key_equal, g_free, @@ -885,6 +946,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s =3D COLO_COMPARE(obj); + CompareState *tmp =3D NULL; =20 qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -892,6 +954,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); =20 @@ -904,6 +976,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516369719435817.9486366486914; Fri, 19 Jan 2018 05:48:39 -0800 (PST) Received: from localhost ([::1]:50333 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX21-0004tz-Q2 for importer@patchew.org; Fri, 19 Jan 2018 08:48:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46567) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecWzp-0003Z6-Pa for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:18 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecWzo-0002jI-Kb for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:17 -0500 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:33239) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecWzo-0002iF-D9 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:16 -0500 Received: by mail-pg0-x243.google.com with SMTP id u1so1474293pgr.0 for ; Fri, 19 Jan 2018 05:46:16 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9sXkzsd0pgyAySExmQdvVGMI0fVmQGsy2pOngFUzHK4=; b=aP+XtrluK5P4E3C8CGihVNtUuXQDV7sxacEbYN0kTCfZG+jEWQ1G/rKC1bzuBfnT8K Pj07boFI0sxsTnien3axScCyFLychncuXwIsF8IitpWsd1d9s3vQmhetSeTJUNjyOMd1 doF1GnN97OqVMvDbhQAKLeAAI2fxtkl/4e8Sm2Lih6Stxglkt4BCyccO8JCqnHG1bjbe Du4Edav63xqmTX6djq0ZsmtCaC305YRzOWjhKmvses3pbS7bAV+KQocG8QEm4VgfAvIW aaai+m0u1l5k5jAzqvlxI3buGNZrByOIcrlENZhCyaUa85TlqWZo+sNhYde/+dXhlvnr TXzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9sXkzsd0pgyAySExmQdvVGMI0fVmQGsy2pOngFUzHK4=; b=UYZ/ODH++thQj6DrMIMdPileJl4n8XDecghx+z3rvSuwTSjE+BxYgkie7Tn4qnew6z q/k2eVU/M+qLdzShC0wMgVaeEBxJ8IhMky1NqBqZBTnn0bJvJjBNeJagL9/BOq/OaH8f alFFVZBW5d8SRTEp+rAbyrNFy7ZRDRMATMuJTfN09IZ2HEsyDXvV/+dZT6fi0bcQo1LL BZyuhDCQXT818Ga0pmIQ0ONllzSojHq9ylF/PpxHeuTIjGDd9CbUlBr5ZAT0TpnexJLq TcYJRTxiWPHxkFkdrxUX/mrnK2j+wnOgZiZeXuHM2anWzBoSgiz+JBneNMWOXtomQ1vG Xr0w== X-Gm-Message-State: AKwxytfR6XU6RvJGZ78MJDd1xBOqWMh3W76CmwmIZOsgq++YCWZpSA3S uRnTYKZfBGyYn16TjzMV2Z5zci+v X-Google-Smtp-Source: ACJfBov7+FwQRGC4THKK2+A6X0t0s01OjkwQBVkkfiA0nnWs40MBo6MlBWonh6SYSwMY5Q/NzytXvQ== X-Received: by 10.101.82.1 with SMTP id o1mr30346642pgp.259.1516369575117; Fri, 19 Jan 2018 05:46:15 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:32 +0800 Message-Id: <1516369485-5374-4-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V4 03/16] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 4bceca8..ba9bc71 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -31,6 +31,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -39,6 +40,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =3D QTAILQ_HEAD_INITIALIZER(net_compares); =20 +static NotifierList colo_compare_notifiers =3D + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -452,8 +456,24 @@ static int colo_old_packet_check_one(Packet *pkt, int6= 4_t *check_time) } } =20 +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result =3D NULL; int64_t check_time =3D REGULAR_PACKET_CHECK_MS; @@ -464,10 +484,7 @@ static int colo_old_packet_check_one_conn(Connection *= conn, =20 if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } =20 @@ -542,11 +559,12 @@ static void colo_compare_connection(void *opaque, voi= d *user_data) /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76..22ddd51 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H =20 void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); =20 #endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516369982825236.98152711211708; Fri, 19 Jan 2018 05:53:02 -0800 (PST) Received: from localhost ([::1]:50450 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX6J-0000Et-TA for importer@patchew.org; Fri, 19 Jan 2018 08:52:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46598) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecWzx-0003gI-8r for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecWzs-0002u6-9m for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:25 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:38419) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecWzs-0002rl-1J for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:20 -0500 Received: by mail-pg0-x242.google.com with SMTP id y27so1468824pgc.5 for ; Fri, 19 Jan 2018 05:46:19 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QwDha8YO8xZLJ4b5TryFJdTCDrZKxP98aoQBqEbiu8s=; b=BxFzULVVoFIseXkaZCj4F8JHFXFaWiM61NUSdCro/iHVWf5ck/UlhOBPynV+LZ3Rn3 glfz5dY+K5JVcSA5wX47JmkRPZRZt5/Tdz1Iei9qTUk53lZJm7UQNXtFphXoFr8CagHe B/U0Z1f09cVq+c5BT43PsB4vwhWFM+lPJYa9vGJDDydjgXiohs+oz4e1rZ/zG+rmTeUA Zpka8M8QgOBBnLiaJ9xIl6ONJSVCLwd0qp1K9IClwlopox1zP41MCiBRTVWcW1p+/0tO X1kv8zpILrZBuL3YWCcj7RhVdGEkvKz+bOv/ZTS728uw9c9P+mGR0NhU/emqk7jmhOoY VDNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=QwDha8YO8xZLJ4b5TryFJdTCDrZKxP98aoQBqEbiu8s=; b=CPQbVKSW0wC+CXFjjcKTIyL1S2xf/Ki5KR4iQsMAiG7mU+FBqLSbUrfZiFkRMvwV3f nMftd5ClEVppTCI4jKLBIDeXTLZu5EjQSLhCUfe9MCe7So86MAgCAaoPsD8e/2Ns+2M5 0sDeYwfRitVlXJsr0CkgG2WCV5toBLDjwJzFxeblO8eknKveAn0+YV8d67ivn0b0g5rM cvSbReKuqG+BJhWEAWVtklTvYFc6RbDYrEyWNYgw2fZhmtGnrkYimKnai8FoML9aDmoM rX6QT8APvPUFrUTtEIummRexwDhWVZ7biJdBSRD1O5Ssn10R0iFfOEIC3pjOYEg5IBfx EzWQ== X-Gm-Message-State: AKGB3mJ2gsP47xPxi1xa0NGXw2TMRvZqp6v+vtUubHzbxmPzgLlxDBWo NLVSOX93at+ffFC8nRfcc38uD2/j X-Google-Smtp-Source: ACJfBotZ6dA9TIpbS4LY6ZZ+8045CucHNxdtdeZCmsn1JqFATH9NRjnC+5mdWF9JasxTpf9/SCCIMQ== X-Received: by 10.98.178.17 with SMTP id x17mr42463855pfe.57.1516369578785; Fri, 19 Jan 2018 05:46:18 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:33 +0800 Message-Id: <1516369485-5374-5-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V4 04/16] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index dee3aa8..c513805 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -24,8 +24,11 @@ #include "migration/failover.h" #include "replication.h" #include "qmp-commands.h" +#include "net/colo-compare.h" +#include "net/colo.h" =20 static bool vmstate_loading; +static Notifier packets_compare_notifier; =20 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) =20 @@ -342,6 +345,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -399,6 +407,11 @@ out: return ret; } =20 +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -415,6 +428,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 + packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -460,11 +476,21 @@ out: qemu_fclose(fb); } =20 - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -557,6 +583,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -576,6 +607,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -629,6 +665,8 @@ void *colo_process_incoming_thread(void *opaque) } =20 vmstate_loading =3D false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); =20 if (failover_get_state() =3D=3D FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index d3a1c49..5f8c2de 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -74,9 +74,9 @@ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) =20 /* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid = mode. + * Note: Please change this default value to 20000 when we support hybrid = mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 =20 --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516369903346795.092718685747; Fri, 19 Jan 2018 05:51:43 -0800 (PST) Received: from localhost ([::1]:50360 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX54-0007YX-HV for importer@patchew.org; Fri, 19 Jan 2018 08:51:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46607) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecWzx-0003gn-T5 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecWzw-00036n-H3 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:25 -0500 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:38219) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecWzw-000342-8Q for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:24 -0500 Received: by mail-pf0-x244.google.com with SMTP id k19so1395640pfj.5 for ; Fri, 19 Jan 2018 05:46:23 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AJoBEHA0u5SSBqFQSfh76BU8EwRpu3kZjAEqZLZZI/s=; b=RfnO0aK+Nzw4/c8V8bdQGbSwBphC0yVnPTxXMSlP1h696oNqzrqNdlIg9dGLY0Lz/S J60SCiuCebwkOAv7rBIBVDm7MxzD471C8Aqm84C+b8/9yaBC3+aPj2n3kIGj/N8szcEu 176OyyUkjyEH1woVvSlxybfwob5aJ6VbzyRTvqh/2TwFbGMS9Lu+mgyDds6pvpDfkC46 LonhWaOehLnlXw5BJNBT2bo9PfcZGRUboR5lAN9HUtaL3bh1pFCa/5muIzM9tpuH0KOe s1fdVL1JVLCpEkjddGHERfaFyJSCItO3ix8PPX4xNotw+Lnoxr1z+yhcEf61z+xsibRZ sBng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AJoBEHA0u5SSBqFQSfh76BU8EwRpu3kZjAEqZLZZI/s=; b=jxiFqB3Gb5UWJbX2xlnf6DTcO26+Jk71+Cv+ZzaswAav9WZNw9502aXOfTGWQBGzPc whkFlfXUIjyetNmkGBc77nCYn9wOSw/dat4+/jUMZoEMp2aABRR5cmzvWBGskPehLcSx AeAcVhpxeqLYYF7pQ5fiNFfbmKQ0fht9D5fHWcGt3hC5lBooSCRoU6w7gkJnBIj6p4bl aY4x+IcYQynuEcVcyTCfJTamvFK9GceKi+nHNHFHwhV261nT/E6H07JmHy13IlmxyJgi xXE/WzaCzCZy5Xw6moZbh3+oIWPSZT9h1m8QIQcRcdWyfZSGfB+ty5hLwW2Kv9FTA+aY NdAA== X-Gm-Message-State: AKwxyteABPORFfEKgBqVYNrNh5Uxxs2ojIHryTGazJPkoDwK6NPPcBA3 2MPzCfrssCtpbaNM7hqTkoYXFiaH X-Google-Smtp-Source: ACJfBos3nMXV/4dvDcfBqpSsE0YWc6rkw3/lV+jBQxtKm68eOXH2NoYrX7UwnbpqvY/WOUv0ZcKptQ== X-Received: by 2002:a17:902:20c8:: with SMTP id v8-v6mr1623026plg.226.1516369582672; Fri, 19 Jan 2018 05:46:22 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:34 +0800 Message-Id: <1516369485-5374-6-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH V4 05/16] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 55 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c513805..0e689df 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -26,6 +26,9 @@ #include "qmp-commands.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "qapi-event.h" +#include "block/block.h" +#include "replication.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -55,6 +58,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; =20 /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -72,6 +76,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); =20 + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); /* recover runstate to normal migration finish state */ @@ -109,6 +118,7 @@ static void primary_vm_do_failover(void) { MigrationState *s =3D migrate_get_current(); int old_state; + Error *local_err =3D NULL; =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -132,6 +142,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err =3D NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -355,6 +372,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); =20 @@ -396,6 +418,7 @@ static int colo_do_checkpoint_transaction(MigrationStat= e *s, ret =3D 0; =20 qemu_mutex_lock_iothread(); + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -445,6 +468,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -584,6 +613,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -664,6 +698,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 5f8c2de..23b3cff 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -323,6 +323,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis =3D migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err =3D NULL; =20 assert(mis->from_src_file); mis->largest_page_size =3D qemu_ram_pagesize_largest(); @@ -354,6 +355,14 @@ static void process_incoming_migration_co(void *opaque) =20 /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15163698966191019.4229716902418; Fri, 19 Jan 2018 05:51:36 -0800 (PST) Received: from localhost ([::1]:50359 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX4x-0007V9-EA for importer@patchew.org; Fri, 19 Jan 2018 08:51:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46647) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX02-0003ki-6M for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX00-0003ES-Ba for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:30 -0500 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:36717) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX00-0003E4-2w for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:28 -0500 Received: by mail-pf0-x242.google.com with SMTP id 23so1397863pfp.3 for ; Fri, 19 Jan 2018 05:46:27 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ReE54vLNK+fu/KOjsjII5rN8XmpENjXBPPgn3EYo1Fw=; b=H26Noq2Zg+aj9rQtBLQX6P3PQIM2ihhb2d9S6SdRnVv/5XATIZZQPYYj5tNAe1B71O 3jsv2eZnqh3n/kHN0I9BUiVZN28GpAfSoi/K/j+A8GaXBFcmpOkH3wRCAU8rIfdOCWbR zc1J9Hg1WLlY7WzfTQDhK+mEBdHEefAws6XYdU6rL83GrKKfx9yls1wgnbGkXVRxv6WC 7HKB1uDrifokcdB+pACOSqgTytt5E3PmmH+7hfFtlo+xw1cVhORhKDWMAvZR6C65tPdP 9+8kdacMIhYJ5qhNdqNl70kftb0HPOErqrZwPsLNtvDHOBDfENqDJ9PS8L/UCWLyMzhh 8qBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ReE54vLNK+fu/KOjsjII5rN8XmpENjXBPPgn3EYo1Fw=; b=M+C7Jmci9JfpRI67wpWAxE9EjViNXWr/RFFjs7TcGmi8pkm1GT1yUOcrPz4t/tw9Mx jc/u5CA+X2f2lWMOE/+HlQp2UT012M2GYIuC9mdJgCi6xkiXOGOEHd7oSg39u/LzTDJl 6zsV8moxJrI6W0TE5phJCnTdOQeM/2KG+Z1xy35zhpEr76DcjuPZnmWAlcldEOeUY7vL +BANZ/0ZWv0VKIPgteediiGEJLiFw8dgBZHySxCAo3jwH05fvxYgnLB+hmZi6bn4nFxP f8xSW0KNBOS51MH0MZwLYYZ9iJQFm2zsSwtNxonyvAb9qUWqRwRlDjEzhlC82R217Vjm E+Wg== X-Gm-Message-State: AKGB3mIYZd4EGpmisDqbwqcm31ZRWWqGttmBV7J2BoqOUxyOIbTiE+PT tla+8cdV+8pzCReZUv17jXWCOjg3 X-Google-Smtp-Source: ACJfBothMefDxrauUZ9e5Idfr57yA4UhU+U08SVQjBuFVyCys3qRA+gA+7GsKC1YDLAZOa7kAzcAyg== X-Received: by 10.98.196.75 with SMTP id y72mr40369811pff.73.1516369586731; Fri, 19 Jan 2018 05:46:26 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:35 +0800 Message-Id: <1516369485-5374-7-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V4 06/16] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 --------------------------------------------= ---- migration/colo.c | 13 ++++++++- migration/migration.c | 23 ++++++++++++++- migration/savevm.c | 19 ++++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 59 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index 6adf3a5..546cb9a 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -27,8 +27,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); =20 /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); =20 diff --git a/migration/Makefile.objs b/migration/Makefile.objs index 99e0380..3099eec 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y +=3D migration.o socket.o fd.o exec.o common-obj-y +=3D tls.o channel.o savevm.o -common-obj-y +=3D colo-comm.o colo.o colo-failover.o +common-obj-y +=3D colo.o colo-failover.o common-obj-y +=3D vmstate.o vmstate-types.o page_cache.o common-obj-y +=3D qemu-file.o global_state.o common-obj-y +=3D qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4d..0000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s =3D opaque; - - s->colo_requested =3D migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state =3D { - .name =3D "COLOState", - .version_id =3D 1, - .minimum_version_id =3D 1, - .pre_save =3D colo_info_pre_save, - .needed =3D colo_info_need, - .fields =3D (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested =3D false; -} diff --git a/migration/colo.c b/migration/colo.c index 0e689df..8d2e3f8 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -153,6 +153,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } =20 +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -747,7 +758,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); =20 return NULL; } diff --git a/migration/migration.c b/migration/migration.c index 23b3cff..6042ee3 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -245,6 +245,22 @@ void migrate_send_rp_req_pages(MigrationIncomingState = *mis, const char *rbname, } } =20 +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled =3D false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled =3D true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -354,7 +370,7 @@ static void process_incoming_migration_co(void *opaque) } =20 /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2383,6 +2399,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } =20 + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); =20 s->setup_time =3D qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index b7908f6..cd753c4 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -52,6 +52,7 @@ #include "qemu/cutils.h" #include "io/channel-buffer.h" #include "io/channel-file.h" +#include "migration/colo.h" =20 #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -78,6 +79,9 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream= */ + + MIG_CMD_ENABLE_COLO, /* Enable COLO */ + MIG_CMD_MAX }; =20 @@ -833,6 +837,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } =20 +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1749,6 +1759,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncom= ingState *mis) return ret; } =20 +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1817,6 +1833,9 @@ static int loadvm_process_command(QEMUFile *f) =20 case MIG_CMD_POSTCOPY_RAM_DISCARD: return loadvm_postcopy_ram_handle_discard(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } =20 return 0; diff --git a/migration/savevm.h b/migration/savevm.h index 295c4a1..041d23c 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -51,6 +51,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, c= onst char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 141e773..59c7e3e 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -34,6 +34,7 @@ savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" +savevm_send_colo_enable(void) "" savevm_state_setup(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" diff --git a/vl.c b/vl.c index 2586f25..0184c7d 100644 --- a/vl.c +++ b/vl.c @@ -4499,8 +4499,6 @@ int main(int argc, char **argv, char **envp) #endif } =20 - colo_info_init(); - if (net_init_clients() < 0) { exit(1); } --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370126634950.0425937696923; Fri, 19 Jan 2018 05:55:26 -0800 (PST) Received: from localhost ([::1]:50493 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX8a-0002Fh-TU for importer@patchew.org; Fri, 19 Jan 2018 08:55:20 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46690) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX08-0003pD-HW for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX04-0003GM-4v for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:36 -0500 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:44427) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX03-0003Fv-T2 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:32 -0500 Received: by mail-pf0-x244.google.com with SMTP id m26so1386892pfj.11 for ; Fri, 19 Jan 2018 05:46:31 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sGihzN6AqmRwDgyPk7qcLYr2oDEPUv2W9jiaot/+gyc=; b=Ymsztuf8mDRfZKY6FI6DrFsoc1ffLC5rKm23x8yABFvE1Kg682qS3oW0oytZm8y55C BmLHIiUwjWkXPy6fT1ffj4B0N11cBo46nP47UdjPwlhmom3rU87xUqEqWJ9gFTFiMOnN co11W9k/kzsBVxqdAFiGgIuIqezikPa0+pF4MRc9OO0rcZhyRiSA2hekKccW4lqCZcHI 30f9uwW0vifhqyXiu3tTl+c+4kmZvzGa40zQGKR0oToDWk4fYvBZwJNibEokc4WxV1rO 7d+dDjcJhm3sOhDsZ9/HgGcVmhOpAKCUfnVh71pABoqAi3OPmEj5wFhbzlf7NYihaUWd 21MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sGihzN6AqmRwDgyPk7qcLYr2oDEPUv2W9jiaot/+gyc=; b=TTHcsvuS5xeyR/NYRm3Vn59bUonGq+AWltWOk29wdw2ZRm2fIU/neCfqYcxlkK5s2j sNg8Y3esMR1/URrkP+llbk0cAmsD5wohMIAYm7ObcMgD73gXKAXMHDCrM1Onq+XgAMxH M5ZVsCCgFap5NoDuqwsINTifDedBVZEmG0RSDmgw0caoLcnRixCjnuhx5J4IePFSrENi YSJk8XMLUhyDRD4qm9ixoX6AjOzETrW7EXvSKGQT4s2AAkjsAVJD+GxTudAjXFKcK51/ 08RAXHQfVMWGYoY9UiFEzaphUsMqTDBVcSs6hYcjDojVmlqUCcBLaiS5FiSBsJ2U4xdP VSCQ== X-Gm-Message-State: AKwxytcBGAYZL6SyRXdCwfqUVoA3pFQTtbv5+t2JiVgjzi+YGYgGgWVu FrhCpl3JQispyh2pWUyrLiprxzWB X-Google-Smtp-Source: ACJfBosDn9HtAg+/IgSKHNC9qK6nS9QGo+fdWxDf5q4TDRORcULrcV9SeS/4/62VmBpA1c1l6vv2UQ== X-Received: by 10.98.206.1 with SMTP id y1mr35689403pfg.136.1516369590586; Fri, 19 Jan 2018 05:46:30 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:36 +0800 Message-Id: <1516369485-5374-8-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH V4 07/16] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary= side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram ca= che always the same as PVM's memory at every checkpoint, then we flush this cac= hed ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 97 +++++++++++++++++++++++++++++++++++++++++++++= ++-- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 102 insertions(+), 4 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index 7633ef6..15e2474 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 6042ee3..be8defd 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -387,6 +387,8 @@ static void process_incoming_migration_co(void *opaque) =20 /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } =20 if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index cb1950f..6460777 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2467,6 +2467,20 @@ static inline void *host_from_ram_block_offset(RAMBl= ock *block, return block->host + offset; } =20 +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2620,6 +2634,55 @@ static void decompress_data_with_multi_threads(QEMUF= ile *f, qemu_mutex_unlock(&decomp_done_lock); } =20 +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =3D qemu_anon_ram_alloc(block->used_length, NULL= ); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %= s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -2633,6 +2696,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); compress_threads_load_setup(); ramblock_recv_map_init(); + return 0; } =20 @@ -2646,6 +2710,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap =3D NULL; } + return 0; } =20 @@ -2846,7 +2911,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) =20 while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host =3D NULL; + void *host =3D NULL, *host_bak =3D NULL; uint8_t ch; =20 addr =3D qemu_get_be64(f); @@ -2866,13 +2931,36 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_ca= che + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Privously, we copied all these memory in preparing stage of= COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects = the + * speed of the migration, but it obviously reduce the downtim= e of + * back-up all SVM'S memory in COLO preparing stage. + */ + if (migration_incoming_in_colo_state()) { + host =3D colo_cache_from_block_offset(block, addr); + /* After goes into COLO state, don't backup it any more */ + if (!migration_incoming_in_colo_state()) { + host_bak =3D host; + } + } + if (!migration_incoming_in_colo_state()) { + host =3D host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } =20 @@ -2967,6 +3055,9 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) if (!ret) { ret =3D qemu_file_get_error(f); } + if (!ret && host_bak && host) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } =20 wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index 64d81e9..07abf71 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -62,4 +62,8 @@ int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_ad= dr); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t = nr); =20 +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index cd753c4..c582716 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1762,7 +1762,7 @@ static int loadvm_handle_cmd_packaged(MigrationIncomi= ngState *mis) static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } =20 /* --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370047358713.386259591305; Fri, 19 Jan 2018 05:54:07 -0800 (PST) Received: from localhost ([::1]:50465 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX7M-0001CQ-JV for importer@patchew.org; Fri, 19 Jan 2018 08:54:04 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46700) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX09-0003pg-4E for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX07-0003Hx-NN for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:37 -0500 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:38930) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX07-0003Hb-I7 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:35 -0500 Received: by mail-pf0-x241.google.com with SMTP id e11so1392661pff.6 for ; Fri, 19 Jan 2018 05:46:35 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=KBf9N7x+ZU7xkYOlauplRvR3IwUKA84kyNXw2mc4Z0M=; b=WXLAS3JInljX/R5BgxI7xOs8mKYldTOsqqMITVV7ErXmdXTY3o//vwQ/SuSx311nFK Oq6hnwHjH4meo6oA6ozMPhXl+jIs7rD9SGXWFDLp91mVL9ZgiGXxe1rhsTuKhgILHwJs SsAANHbHMJjKNkfPMdwYhP07BohGbgBd443bYxOJYQIDr6dJ4NLZrnWtmsjRWyPMuKl+ dae4bV23EwF1NPJmUKarybySW1vqzKfsqcuL6C8M8ltfjDEo2utLIKVoqmexMYpEhwXR uluAUaYASAnKBg4pRPDw5VpZ8n2MCdqSUaWZsYM9UtAab4p17EygTELuLjsDagUKYcv4 Xq+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=KBf9N7x+ZU7xkYOlauplRvR3IwUKA84kyNXw2mc4Z0M=; b=kAYG15eXGHHC49vV1TnEn4AQUiAC5IvlR0nNcuiuiJsOEuJAezAcV8ocuYzwg0HdZn fl0vkh07uxPJABQb5wNw/jzHMvmBerV3l3aSukqlKtYl94MBo+kzKFXVaNG0Lo4hMKJ4 qcsjWtFaJb7kTMXy18tBnpDTPi5AwtYB8lp6i9K4pkgHzckMcZ4vzeme4EtP4xBP0tS6 txEk4t37nKtXcz/eoqZ183LZyo7h/RMAr8XraGl27whTtKNQIzcU0+XfLMHp6OiIHPhe 7GtvHbw83CXNNFuQYN5Rvd8avMQqn73D2UgCcIQG3auMsVAE28he9yMk16EVVqWVzqse sRJA== X-Gm-Message-State: AKGB3mIE4taEGcpy6Dh4/DoqpbgTL2nyRoRP8HTW8psmKZLK8btZBFN/ AJcWWDPjaMduN8eUij0TM8tSx/ef X-Google-Smtp-Source: ACJfBoubw0Sw7mJZmpIfGcx1MY+XAz0vXtdl5t7SmplZzMwzObqIW7Gcj3/vUwLux3ukTl2o+bby4g== X-Received: by 10.98.60.67 with SMTP id j64mr40083211pfa.217.1516369594227; Fri, 19 Jan 2018 05:46:34 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:37 +0800 Message-Id: <1516369485-5374-9-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V4 08/16] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 6460777..d916da0 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2478,6 +2478,15 @@ static inline void *colo_cache_from_block_offset(RAM= Block *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } =20 @@ -2654,6 +2663,24 @@ int colo_init_ram_cache(void) } } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap to= gether + * with to decide which page in cache should be flushed into SVM's RAM.= Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; + + block->bmap =3D bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state =3D g_new0(RAMState, 1); + ram_state->migration_dirty_pages =3D 0; + return 0; =20 out_locked: @@ -2673,6 +2700,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap =3D NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2681,6 +2712,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state =3D NULL; } =20 /** --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370185645841.097560755867; Fri, 19 Jan 2018 05:56:25 -0800 (PST) Received: from localhost ([::1]:50587 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX9c-0003ET-SZ for importer@patchew.org; Fri, 19 Jan 2018 08:56:24 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46743) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0C-0003s5-Fb for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0B-0003Js-BF for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:40 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:34461) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0B-0003Ja-61 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:39 -0500 Received: by mail-pg0-x242.google.com with SMTP id r19so1478952pgn.1 for ; Fri, 19 Jan 2018 05:46:39 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uF3bxFIb4dK4MypnNIwsKgS8xIuTCoFdUWnL9+BgNGo=; b=LMtQMFCSCDLVEMc1py5MLoNeJn8Q5l/v8inhQpQ7bBTH+cG2b+6uQDAHmdsn8IHz4v pTvbZGwlMlQTvrpE8mopDm5ml/gRZ7IKJBktYyKGYoYv6s+lZQjJoy0Rdynz9/d8E0oR t30kPGcg/PMJK1TadSMgq05mUFzv7HP/rdLK97PhVRpyld/5ZIEFE2RP94i7/S4MWOCo C3dsPF2LQRYi1ORTlptPwbetpmOJj0fldwKnRWk01j4bBfDXDAFByHPZooNEX5FBOenx GP1ddFMjSqk6Vx4eb/hw+eQZLwXuF18CQKMk2aToE8hZUeSc4gUJ6+gnXoI9+O1llP5A U9vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uF3bxFIb4dK4MypnNIwsKgS8xIuTCoFdUWnL9+BgNGo=; b=cm5Dem5vcB9zXofPQ+RnCBrIeOiNzSKL+ZC5qG1cXxSy2TRKVrZ5v/b5N72FxmrFoQ 2pSeBhodhmf5Nex0MYDJH33Lv+fIyKn+xvw7PW04XWIeNd6TdhjkC1539U4PrO4awJlt Gy9XEP9bgVXPHmC0ZHdyVv8MTWgsgKuc7hQnhawVHAI4ESrXCLYUuTFCZgaJrbEnjtcn Oyvv8iLowZVkwaOTVinK22SoR1GdlsX6TyxOsuERB+wHKHZPsosBmVlqXvejiVUMcv5N cyZ5rUN43RPnqRkQBPinx8m84tPFrEZrdWuoOOUMwSx92ZrRgXdUnSR+nKUhuiDtKp0O 4FCA== X-Gm-Message-State: AKwxytd+I99Om1SktgtFYMf/LXYXGn8IYpYz8VBFMOHOxA1XR6tmqO7x sF5G+jSee1M99zUo8YJiOsSgrDUm X-Google-Smtp-Source: ACJfBos24Irmb1O+fpayUIoUAW8C3/ZPhKzqvuF1+oc+YiPQBi0a/zJXPwVrTljVTnx7REETuRTc4Q== X-Received: by 10.99.177.4 with SMTP id r4mr16407539pgf.245.1516369597952; Fri, 19 Jan 2018 05:46:37 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:38 +0800 Message-Id: <1516369485-5374-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V4 09/16] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkp= oint time. So, the content of SVM's RAM cache will always be same with PVM's mem= ory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index d916da0..faee086 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2909,6 +2909,40 @@ static bool postcopy_is_running(void) return ps >=3D POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_E= ND; } =20 +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block =3D NULL; + void *dst_host; + void *src_host; + unsigned long offset =3D 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + dst_host =3D block->host + (offset << TARGET_PAGE_BITS); + src_host =3D block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages =3D=3D 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags =3D 0, ret =3D 0, invalid_flags =3D 0; @@ -2921,6 +2955,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) bool postcopy_running =3D postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability = on */ bool postcopy_advised =3D postcopy_is_advised(); + bool need_flush =3D false; =20 seq_iter++; =20 @@ -3096,6 +3131,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } =20 diff --git a/migration/trace-events b/migration/trace-events index 59c7e3e..eb56cc6 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRI= x64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset= : 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370332700812.1517074501573; Fri, 19 Jan 2018 05:58:52 -0800 (PST) Received: from localhost ([::1]:50655 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecXBu-0005Iu-VM for importer@patchew.org; Fri, 19 Jan 2018 08:58:47 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46783) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0J-0003xh-Dc for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0F-0003La-2E for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:47 -0500 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:42798) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0E-0003LK-TC for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:42 -0500 Received: by mail-pg0-x244.google.com with SMTP id q67so1458676pga.9 for ; Fri, 19 Jan 2018 05:46:42 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lSCwC99Noxssn3VatGN4GXPj6exgWJyE0rMb01p67aE=; b=jb8CaERNgxFtnfrPW+1ZjBlPqYZU0A6BLljhkmfYDq8+D99FP6UYyZMqmonZ9QTtpO OgQZuNoGStZVjX0WGobM/eWxkYkx4Hnmy6As2qUMfjFVDPdBMMgqQPwXDOvkUgW2+xIm DnnT4tqE3H5Kg5100pAxnZH0Eq0liZlN526kRMHGd2jLhfYU9UEMEvRGF+kci6vJqGc3 jeSttunL0TzHfqp4Eg15HZSbBYP3RATZWhdXjh52le6phBcT8Nf67JKYzHL7pbVzlztb S2HRzyvzK+YDmQTswTJ8cdDOaPekGOezRlenTBx6xon2MGKIY8b2XapiQTZobZ6EtLfv FGbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lSCwC99Noxssn3VatGN4GXPj6exgWJyE0rMb01p67aE=; b=XGluJ1C6748a+AEcj4l0Llf50pTH7yNVkQ2iN5s45w6WJUTjKkfwcw6Ng41pwOOpyj AOo+asOHGN/KMDCbPyKAVZbdH65Q6YcZ9N16tr8mWlxssr8ocz5kIc2pHx/pjau2vyGm +Vn38bzQ7Y8R5XIAHnj7X/XiWX7dFrR4hqIQlRCGu+jAKO8trsWRx3bDNXwm6cswVw/3 zppu4tnIVMsKi8tfvNkhGZWfO9mfaveDhmdiDhFOdRhbPIlIKr8prrcItBoyLymeiheG p9jTO0VzrBRGQ2px/ZF+u9oZd++sB3VlJPv+E08N+EYmQWsWmFnyykyuZi7a/9WceILh BySA== X-Gm-Message-State: AKGB3mKfb9NTdR5N81BzF8PdVlk6qttQSe9ahGei5784N9bbE2xmh2i1 GL0rxJr9oDX6YkimNrBRAp9/q/Gb X-Google-Smtp-Source: ACJfBouuoYEzw7z+yGFvu2mwTzTAIdb6IYCD136WvJaAuMwuqdtjotKcvHh9YQKl85XByAowJSIfQg== X-Received: by 10.99.126.86 with SMTP id o22mr37188684pgn.364.1516369601661; Fri, 19 Jan 2018 05:46:41 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:39 +0800 Message-Id: <1516369485-5374-11-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V4 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c | 19 +++++++++++++++++++ qapi/migration.json | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8d2e3f8..790b122 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -516,6 +516,18 @@ out: qemu_fclose(fb); } =20 + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -746,6 +758,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } =20 if (fb) { qemu_fclose(fb); diff --git a/qapi/migration.json b/qapi/migration.json index 70e7b67..6fc95b7 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -869,6 +869,41 @@ 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } =20 ## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.12 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "reques= t" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.12 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'request', 'error' ] } + +## # @x-colo-lost-heartbeat: # # Tell qemu that heartbeat is lost, request it to do takeover procedures. --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 151637049018211.208309683282437; Fri, 19 Jan 2018 06:01:30 -0800 (PST) Received: from localhost ([::1]:50699 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecXEW-0007hH-5Z for importer@patchew.org; Fri, 19 Jan 2018 09:01:28 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46805) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0K-0003z4-RN for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0J-0003Nk-5a for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:48 -0500 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:40967) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0I-0003NR-TC for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:47 -0500 Received: by mail-pg0-x243.google.com with SMTP id 136so1459775pgd.8 for ; Fri, 19 Jan 2018 05:46:46 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vSLl4gjxYp66G4eOCYOmqxEnncB7nyB9Sur4LRAW3nc=; b=HtdysEdj8HpEVzcJ6yUWCtKOpUiFHJX1Gg4fwpU3emPVOkWk/DaXBUfbMBKZ0Qu+Oj GlgGcwBqkEAqt03cSP3dbGGClULu7hczqZu340xXAGp5PDROcD7jcIVxPCEWpbWeo3SY EkOeZvqroarAMYDPK+VqHD+R0HIIDsLKMKZJNizR44SOXVOazC5Zr6jTzvc6gYb6kshX AtWVY39t1BjZf926bvaJyViPqXDx3PfZeL2nAvgiVld7eTO8IU0oTj+O7VRxVtlPGv7F s6EOZa6B5zbUBi84f//biGf8h7bMUevSMuff58xdVdbF/n+NdKLX+VgrPLtGyxUHEYsq dcHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vSLl4gjxYp66G4eOCYOmqxEnncB7nyB9Sur4LRAW3nc=; b=pydBgKX3GYlbYZK8+XMywalTlCCYWlPjvlOV6yRmGSiHXOHtUcw6Ctf4v7+mii3HeY LRdGcA9SiE0Bo53CJUGrW0BK0WMQk1sh5gz7MUXvVvIS7+67BwJV4FAzB3U6pEPxwLIW HAful+oTsaoZu5e2B4GhQU3XNBxINbYsv768oQs45KNMGzIc2X4ynVpTZzXFD0a+5Wqb 9N26uYXWIzq6lFSDqXklgeYAzydT9lYMBouV41STYyLC2XoN3Zjk4AZHBRP7+/wCNgzy WGKbbyj3DPDpotf7cqcsxkyO2XNcQJjoWPsPJURSOnDcp/zGXWNcnStiFQpOfcXDO3mN FL6g== X-Gm-Message-State: AKwxytf82alLiOVImCLuIjZrTqqmjW23R9zoQvsAsj4s2VNdKwPjelkm TN7WuVQZL9c7BBTamKGG176GTXWO X-Google-Smtp-Source: ACJfBotTGZC4pYM+pElAOUHAW5yYgr3+jptXGtY0X+PuW0WppQas5uGCaOtHHe+j7u9xv0v0ykNHZg== X-Received: by 10.98.150.20 with SMTP id c20mr24285648pfe.200.1516369605548; Fri, 19 Jan 2018 05:46:45 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:40 +0800 Message-Id: <1516369485-5374-12-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V4 11/16] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 37 +++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 61 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 790b122..a931ff2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "qapi-event.h" #include "block/block.h" #include "replication.h" +#include "sysemu/cpus.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -380,24 +381,31 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, =20 /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); =20 colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret =3D qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -610,6 +618,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err =3D NULL; + int ret; =20 qemu_sem_init(&mis->colo_incoming_sem, 0); =20 @@ -682,6 +691,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value =3D colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -715,8 +734,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading =3D true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret =3D qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } @@ -777,6 +797,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } + qemu_loadvm_state_cleanup(); migration_incoming_disable_colo(); =20 return NULL; diff --git a/migration/savevm.c b/migration/savevm.c index c582716..30a3c77 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1317,13 +1317,20 @@ done: return ret; } =20 -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} =20 - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; =20 + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1379,8 +1386,6 @@ enum LoadVMExitCodes { LOADVM_QUIT =3D 1, }; =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis= ); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2003,7 +2008,7 @@ void qemu_loadvm_state_cleanup(void) } } =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret =3D 0; @@ -2148,6 +2153,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } =20 +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 041d23c..8d463fd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,8 +52,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, = const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); =20 #endif --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370260723221.94667752442854; Fri, 19 Jan 2018 05:57:40 -0800 (PST) Received: from localhost ([::1]:50608 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecXAq-0004DI-3M for importer@patchew.org; Fri, 19 Jan 2018 08:57:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46834) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0N-00042g-LO for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0M-0003T6-Px for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:51 -0500 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:38425) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0M-0003RV-LQ for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:50 -0500 Received: by mail-pg0-x243.google.com with SMTP id y27so1469857pgc.5 for ; Fri, 19 Jan 2018 05:46:50 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=01E7Obz5WZ86oldwAeTnu2Mgcuou9cRrEtdosWSVXT8=; b=MQKD1QMGG1DqTD0oB8hck46Zn4/Jm9JLxLLE0jAgvQxqEyQTG33MgoS2Ke3gVC/1rI HyrMZdSB6WX+8ZOqH2YsRmxsdKOAjQKzWXx7MhNNivXOi1coLwyZXHimrEAUwx7npZMf JE6HO7oYh5O0JUIwpWmJxLsawfQVAwnBH3Gj8o8ijCDUNyiLdusP5vaJxAGnSU+ddBXt mKcxR6+KGyZ4ry3gA4UBMHTGg8hQW8ip4XbbfSWeGbvbjHkQeg0PV3hTsQYQK3pxe8lp AC9aOW/MGZGNDOiSmZD16NEv9aZX7//6aBzfqR91dxbNvg4Ejsw9aF72T6dWGd3k4sfJ pM9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=01E7Obz5WZ86oldwAeTnu2Mgcuou9cRrEtdosWSVXT8=; b=lTI459GLZ//X2/Ho2hFs0ZPwjEqmndWsPZ1+tsAdupP/3wlGz8P/g3O5PEVRGf5apk dyibJ7XT8E5A7sXzM3KgrIuEPOq1X58sCev6hFv+ZZ4Le86WD5bDYJmmFWmhk8QL/jun ioKyR/ArkQhHVlSslzIRKbWq+oXzB0nAFOJ4rOCm6CjoxWG/7TpPvPH6ide4uCQy6RXp 7hrGrPPxc8p8LmTT8nQqpIsy+6HDryogMefKX+y301Ic1MgKKg3nNZXDlmRAAi8CbXJT 8av8KLvxU/lV1fXkYrqnge+lFjuROXwBhetMfvbEQcQj1rxUoNFBdz7tJrS/4QJIklMC T/8A== X-Gm-Message-State: AKwxytevkr90oPkUhKQwx5d06gjpGgWEv2xMFJEfqu3CjA0D7XoFhUJF eHrpVm4S6mIIa7chZXstK+1c+8FG X-Google-Smtp-Source: ACJfBovQq5RLusYkEhPHq6NkBgbgm0zbIuJcq15UuietVfkKfUZJ1EfBX/mB+7TjaKbx3qADwU1bAA== X-Received: by 10.98.102.4 with SMTP id a4mr30156498pfc.210.1516369609451; Fri, 19 Jan 2018 05:46:49 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:41 +0800 Message-Id: <1516369485-5374-13-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V4 12/16] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index faee086..7f9ce60 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2680,6 +2680,7 @@ int colo_init_ram_cache(void) } ram_state =3D g_new0(RAMState, 1); ram_state->migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); =20 return 0; =20 @@ -2700,10 +2701,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap =3D NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2920,6 +2923,15 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_lengt= h); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state->bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block =3D QLIST_FIRST_RCU(&ram_list.blocks); --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370400251919.8943718923376; Fri, 19 Jan 2018 06:00:00 -0800 (PST) Received: from localhost ([::1]:50673 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecXD2-0006OC-HN for importer@patchew.org; Fri, 19 Jan 2018 08:59:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46902) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0V-0004Ab-Gu for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0Q-0003bI-KI for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:59 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:42798) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0Q-0003aE-EV for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:54 -0500 Received: by mail-pg0-x242.google.com with SMTP id q67so1459034pga.9 for ; Fri, 19 Jan 2018 05:46:54 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MmtdVD5J5uBdRDEP+kxjmLnMZS7z1Fh9r9xNvavOdso=; b=fEPCD5gaAETZ+MYlYv3MTPoA+U3xgYJEkT3dYEvPkFkJK8+q/6oDLDPPJhVvmXQC7L wTs3IP4jtdnnkvI42R/gVXoLQMkx17gtNNP1hLpfpkiyqpwCe1nidp6MfvBOR1QurSL6 wbdF23LdF3rL0Y9FQxSTPMeog6lFgIL1zTmEfc0nu5hoZBWc272OIRnQGfNkVyqToshr /AOA56hLK0A02tKV1psKqHn8z4wD9YXVqQVir0WJnQhL/tVkbseB6yz7cTa+xrgjkpjA Oe844t0hFn6OnMrokHbOwr7aPgqKYqE85+0SuqWZGJer7w+ZkPkt7e0PxJmzUfsK74Tr YHQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MmtdVD5J5uBdRDEP+kxjmLnMZS7z1Fh9r9xNvavOdso=; b=H6QUJW5Xtubp6YcLhed49IAFnnXP55LOsD6GqdaDF9gYjduByHjpBmHaWOeQrHKx/O TOHrAYOwSi4d7ojtmE3ejcUYXG1hLO+ivO/i7mc2WAPYaDup2gK1XOS4VLqInmtrRowB MOwKLeFcjVtj+8di+Lqq9n4+No3xVEvSgSjUqXHsHws5UdAvKAlO4coa2Eo8xImefejW 4m7yh6PA5D8r6L70tZJC1o9QtYZ3oaYdGP9QViIHxkmXVgvhO+e2IJJD//gLGJwreKvN M06hkkEhgHig9V9WCV41Y8yNaLhD4NR8BZN5QuAprdyCITQ/XSt1Uo5qkYcQkUJLaPjI 1AWA== X-Gm-Message-State: AKwxytdHWhE9nQG4xGXyyDprMwoMTP6p+27eUCO+N140cwPcy/H7c4pc UY3IGfJ7sOrnTaGIjNI1GzW8DQln X-Google-Smtp-Source: ACJfBotojPvMB//ExumiHVwNfawg9gza6xxTbj6JpduLJKmqLetpgH6pdpxrLZcIbmm2k8Folllxfg== X-Received: by 2002:a17:902:b687:: with SMTP id c7-v6mr1662829pls.138.1516369613055; Fri, 19 Jan 2018 05:46:53 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:42 +0800 Message-Id: <1516369485-5374-14-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V4 13/16] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 0c4a2ea..df4510d 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -37,6 +37,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, =20 typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); =20 +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **e= rrp); + typedef struct NetFilterClass { ObjectClass parent_class; =20 @@ -44,6 +46,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -76,4 +79,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sende= r, int iovcnt, void *opaque); =20 +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d..0f17eba 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" =20 static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } =20 +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc =3D NETFILTER_CLASS(oc); =20 ucc->complete =3D netfilter_complete; + nfc->handle_event =3D dummy_handle_event; } =20 static const TypeInfo netfilter_info =3D { diff --git a/net/net.c b/net/net.c index 2b81c93..56a54e7 100644 --- a/net/net.c +++ b/net/net.c @@ -1399,6 +1399,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdi= ct) } } =20 +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc =3D NULL; + Error *local_err =3D NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer =3D nc->peer; + type =3D nc->info->type; + if (!peer || type !=3D NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =3D NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370055068566.9417696936401; Fri, 19 Jan 2018 05:54:15 -0800 (PST) Received: from localhost ([::1]:50467 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX7R-0001FP-Et for importer@patchew.org; Fri, 19 Jan 2018 08:54:09 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46904) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0V-0004Ad-Ha for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0U-0003ha-2T for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:59 -0500 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:33272) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0T-0003gC-TJ for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:46:58 -0500 Received: by mail-pf0-x241.google.com with SMTP id t5so1404695pfi.0 for ; Fri, 19 Jan 2018 05:46:57 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cK46XxlC3JqpIkAPYMPP76QgEA5Yp9efAhDjKZ9G0gE=; b=hDewzd/FZe/aZt5WxauITWngspKfLLnIvG/Xq2xTmt8p5Q7teBfBs21tEyc87kVMYA AwIu2+NH8ctlqLlbTMwXO0p/YxV7kui3KxraJC1VhkpwY2aKx4NGRQ5qZdPeq2d73TVX s1oUO/b56BDFoKnjg905PkJKKrFh6TwWwwOQE6ZW5Vnbc6Wm2Mxe69nqvP/3bADU3cBs x8gOjQ+mPcOS+rKtJlbMcjFx6S6r8sCDRS8acxZ8h8/f/GyjMGB9sMuEpuA6Wey8Z6qg 9x/7TZzZAZidBF3+Cyim1goP+0r3l+7LtscLBobzRdtgMY+NWnFQz/dUT+lTXowe+1m+ 746Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cK46XxlC3JqpIkAPYMPP76QgEA5Yp9efAhDjKZ9G0gE=; b=sVmAjyYQvx3m0YtDnZJYPbhi7DxFuOXXzzno2Snz/RYDV77OuaP0n0Us1BemWlXhLW 6mEzSPz8WIRGh724iAyb64BOGm7Vmk0DXC1xETDviRlYMxdpLeBWPpC1W4usAZG7p9dX v3q4VASLoXNExqCd8CERm2w6P0XE+QQSrT1osVQbSwVQTfLpgcj1YDksfpIbndZMoS9V uBi2H3lsV0U8/qGF4VjlrFvGq4/AAnQqFjBSRIdjXbyJK2vJ68Hy6+XMw0a9c2QIOFKu yUzFwjP42+Ykgcv0Ds5IPVRK6FVl0l/geltxvcu65pm8wryktZxiLQA0zl6KY9TN1TlZ 40wg== X-Gm-Message-State: AKGB3mJwBvw8kJT+fLaNdkHD91vSgWurspxWldlI3zSco1gZO1skM5VX wDYzBiofI184377/w7ROT7d5Y+l3 X-Google-Smtp-Source: ACJfBotbtnvXVuJfdtKMgO1Zs65SPH0eCIYyjDBHJIieqcGx4hXsmvUMz4V3rWez9mYipMeJYbwnrw== X-Received: by 10.98.73.157 with SMTP id r29mr42979835pfi.41.1516369616633; Fri, 19 Jan 2018 05:46:56 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:43 +0800 Message-Id: <1516369485-5374-15-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V4 14/16] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index a931ff2..9eab4a3 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "replication.h" #include "sysemu/cpus.h" +#include "net/filter.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -81,6 +82,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -753,6 +759,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index a58310a..bd4b6cf 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -23,6 +23,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" =20 #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -280,6 +282,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterStat= e *nf, return 0; } =20 +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_d= ata) +{ + Connection *conn =3D (Connection *)value; + + conn->offset =3D 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn =3D (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs =3D FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s =3D FILTER_COLO_REWRITER(nf); @@ -335,6 +374,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, v= oid *data) nfc->setup =3D colo_rewriter_setup; nfc->cleanup =3D colo_rewriter_cleanup; nfc->receive_iov =3D colo_rewriter_receive_iov; + nfc->handle_event =3D colo_rewriter_handle_event; } =20 static const TypeInfo colo_rewriter_info =3D { --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370575149459.6895583156014; Fri, 19 Jan 2018 06:02:55 -0800 (PST) Received: from localhost ([::1]:50732 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecXFu-0000bN-EQ for importer@patchew.org; Fri, 19 Jan 2018 09:02:54 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46933) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0Y-0004D8-F6 for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0X-0003mv-Ji for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:02 -0500 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:42575) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0X-0003mi-Dk for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:01 -0500 Received: by mail-pf0-x242.google.com with SMTP id b25so1391686pfd.9 for ; Fri, 19 Jan 2018 05:47:01 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.46.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:46:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=O0b9Ao0/yakzk1nyHC+GV7uFgsNge4MHuUdbqaAyE+c=; b=QUl5hM5aPcafj6EDuRzAcGW8t42N4VAvPAXIgdw+JNHupzDacAnQvUreK95FJUWp15 hEdxwyyy4bT2F062eVtvEJODSxtvgg88dXDE0q52L5Hj6HkWlYnij/0pSTVF3EivfviY EeUHcIprdtJQEua9n6kL/hCiKFVl/LXA9W4dpLvkgJLslAWGab3eszJRFrFHXgEd9Xb7 RYZbCz0b183kg4/+k/GmD+GbvNRy8FM4NdnTN936ct5061o0GSt1w2UO9jOz0/3CS1ln ZooYWOKDLVnHBQYyujcsqpAqNDCP+/4LoDJDQtAt/3zzGCU1nnBw/lBCQyGcs8GRvtNx JFrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=O0b9Ao0/yakzk1nyHC+GV7uFgsNge4MHuUdbqaAyE+c=; b=LwKv3tbX7t8PANNmNjWEgoCXViJUAOkQKw5vIOzoHLnKOfU2yj8YWNxzXGpGOa04W6 +ZlCI+OJ3aQhPtbS3eg+rB4D4iLTqOkV1XonCdXof0jrO2TJMqVuAJx2MWNWtA1JYurN pE/SfuxsITiYJnXqHwrHhntMj7vLKhuADQN95rYkiPo77K1GeANq7aCa7tfler/9OcVL 8iuHSt+as6qIiAVHL3LFuGK5EtoX027C7KH++5ji4Lmhmyq/etRO9SUCDPPwzE6PHKSI LoTMt30zHXff+1MMT7rqR1YprMOMKQa+r1JOivqH4d72EvvvhOMdPsc8SrXhngXahvtS f9yw== X-Gm-Message-State: AKwxytd1dgdRXFTgBh8AcX83qXFBU9pnfOWJfIStKpBldJbSSh5s4D7/ FnDEwHk2rkxfCHjE/htUFfYl9OZf X-Google-Smtp-Source: ACJfBovDXDXCwdPitHCnoqCOqQSzjKsToZ1UDAV4RFfeTyIA/ASjBWNTKWeC+0WB4bVrWchxh2TOEA== X-Received: by 10.98.29.2 with SMTP id d2mr36007498pfd.204.1516369620202; Fri, 19 Jan 2018 05:47:00 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:44 +0800 Message-Id: <1516369485-5374-16-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V4 15/16] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 9eab4a3..10bc80c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -87,6 +87,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -766,6 +771,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); --=20 2.7.4 From nobody Mon Apr 29 00:36:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516370718813955.1627481044736; Fri, 19 Jan 2018 06:05:18 -0800 (PST) Received: from localhost ([::1]:50956 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecXI6-0002oF-Lr for importer@patchew.org; Fri, 19 Jan 2018 09:05:10 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46990) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ecX0g-0004Iv-9p for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ecX0b-0003pF-Cy for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:10 -0500 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:43300) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ecX0b-0003or-6j for qemu-devel@nongnu.org; Fri, 19 Jan 2018 08:47:05 -0500 Received: by mail-pf0-x242.google.com with SMTP id y26so1385630pfi.10 for ; Fri, 19 Jan 2018 05:47:05 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id r88sm14251865pfb.17.2018.01.19.05.47.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 19 Jan 2018 05:47:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ykiojIxsoXQaDwibIw/YhxXHvoNepz2NylvpnOYiDqc=; b=OQpUPQH6HjD6uyq6bEO/VS3Vk+UZGo9srO/ZVLp/dcMeCTvejpJUmI4Z2K6YDTF8HE bFkylTSYE9JbnkueLUJH7ZCm9M6HRmvN0PgQ2FlC7d1sakBYSyInErkiP2+xE67QJXZg rukZgjI93+dBtZSonAAuUq29ZLKVz9tbLLeSkNDhk1tseg/OMa0n4XzvM8qi3hVrnG+l 9SAxvMn6QnNs1728KDYCzhudAWEeHM4bagM5TekTZ8d5bS/hBlbgtIT4p9QUb4/viHa7 Rj9+pM1rK5axLHBAoJ6tzQv9dl5pGefoWrdWBPjJf2Clgw4r9ZDZVULDTExXP9+Ze+hU hkag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ykiojIxsoXQaDwibIw/YhxXHvoNepz2NylvpnOYiDqc=; b=ByD/Cgy8qSunmv2cLEFl940807ot7eVQqMeioiWdjyrwjx0tGJ+9ILxeesAF2Hx7v9 oJFtBcsWUdueT0YKyEY0BrdpLXekmPbAgdyTg1H5ovFuiNFpQO/DJybG9s00ggcStE8f 08Q1zXL4sHZtx7v+f9/AfQR0v7qhBltcNBOMBa2fFqXTREU9penP1UNFvFvsCBtHklHa CmpzTcZfbwhhGd7NrKQN0XKA8aUrvjFpRVo5Y2+LRk2+sMAjvLsdOoOICIOOM8RjKYZb f/AFSCoj8vUoDxsNOt9LPNWpY5673eltHloPKZA5uVEst7T/eeNgOYqVNkkO/X01c/w3 wgRw== X-Gm-Message-State: AKGB3mILxclEdBYGsnLwXRTu0kk1lYlmLdjNh2DJVHu6ZVeAcoyTa7JU 7S4jLAbGoqJIJjwpGkfWX/KFCtk6 X-Google-Smtp-Source: ACJfBov75np6xH59HaLFa9F0EK3BLIko5ZomtYfjMepnWf1fx6b0pb78KKGKEeirjyA4XrMgxaY7Ug== X-Received: by 10.99.105.72 with SMTP id e69mr33680047pgc.239.1516369623987; Fri, 19 Jan 2018 05:47:03 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Fri, 19 Jan 2018 21:44:45 +0800 Message-Id: <1516369485-5374-17-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> References: <1516369485-5374-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V4 16/16] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 10bc80c..cc616d9 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -134,6 +134,11 @@ static void primary_vm_do_failover(void) =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); =20 /* * Wake up COLO thread which may blocked in recv() or send(), @@ -519,6 +524,9 @@ static void colo_process_checkpoint(MigrationState *s) =20 qemu_sem_wait(&s->colo_checkpoint_sem); =20 + if (s->state !=3D MIGRATION_STATUS_COLO) { + goto out; + } ret =3D colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; --=20 2.7.4