From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717095959749.6367179161821; Sat, 10 Mar 2018 13:24:55 -0800 (PST) Received: from localhost ([::1]:51975 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulyv-0003bZ-3h for importer@patchew.org; Sat, 10 Mar 2018 16:24:45 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48365) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulwl-0002CY-JL for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulwk-0003HV-ED for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:31 -0500 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:46888) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulwk-0003HC-59 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:30 -0500 Received: by mail-pl0-x242.google.com with SMTP id y8-v6so7140209pll.13 for ; Sat, 10 Mar 2018 13:22:30 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=frxRX3jL4VVjHYV1oYmTUZX4+eV8sO0DhXbz1gEnl6M=; b=CfMOfZoEr5cZ6OlTwdB2QvFLCyjiCt/k+FwKQ37TRlbcL6Jz6IO1M10GnwwObw1OlS +I1IzEPZ7zQbVcU5UwWtPxWy9f/5KoHl4QrxPze/4zEhuo5fV2r91XeMUuTAX7K8Yx2L snJltDOyst+qYe0mQh4cNP3sfpadf3agUe8unK0dg+TC8rj2NXnM+ZWwRtldfDB4ntl4 3KrQPZRYBGQsyzSffwc7pykZNhE88BLUgkF6Gxq1PI/Kfz/daeoLge+7QoHzKZgjX1wL 8HlnbfD8WELwCDVeZAg+ImpDggaxjMRIE53Y9BLwIn9ZvYd/rR33RpG7OFedtzDLj8y7 uYnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=frxRX3jL4VVjHYV1oYmTUZX4+eV8sO0DhXbz1gEnl6M=; b=V3GmWQc9RWrBnDFOrwDn6WtCwOgszh14qUbQKAkq1djDnr/li80BHQvO0sV7QHX+tT 1u4C1YDCUuDV0LNWW7ZDfkBlcxSMVsO5jShCacXXb+g6W+D5qiK3zzSme0wKtFbRTrry PwiVGuBWL63VzgB3tdoUPNAHgook+8JKQ5MdVg8UkhTWI9CDljyh+h8r1gKfGTZlfEiX bEWe1PwWx2t5eRqpjnxK74qYP5U2F0JdHDl2N9q+OOQlG9M5t+g/8LpyfGR7lHatvTsl BA5s2HQUMzU7FALl2Cy00di79Dy2YC4iYBQLuiaI2JL0RE1IU1u95BEwqTh3xxi88tDy v6fw== X-Gm-Message-State: AElRT7Hwr51ecrwJtdUBnhGhHyLcc46iwlTfBNqGxHuM5xYRNyR2Gler tLl0nUWZg7U1zWiDMLrD+OtMiGtb X-Google-Smtp-Source: AG47ELs8NNiYL1G0sGdpkodFKcOKAbNfQB1UJ9+9yigu+8Z0SfCnXLbsWthgjossEivzOQ/Sxhzkgg== X-Received: by 2002:a17:902:1c5:: with SMTP id b63-v6mr3029149plb.311.1520716948835; Sat, 10 Mar 2018 13:22:28 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:51 +0800 Message-Id: <1520716927-17068-2-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V5 01/17] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 +++++++++++++++++++++++++++++++++++++++++++++--= ---- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index da6c36d..cd11851 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" =20 #define HASHTABLE_MAX_SIZE 16384 =20 @@ -86,6 +87,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=3D1,ack=3D1' */ } Connection; =20 uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 62dad2d..0909a9a 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } =20 /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -99,15 +99,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=3D1' packets from client side, it acks 'fin=3D1, ac= k=3D1' + * packet from server side. From this point, we can ensure that th= ere + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this ra= re + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_ta= ble. + * + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from client side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D htonl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } =20 return 0; } =20 /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -139,8 +168,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=3D1' packets from server side, it acks 'fin=3D1, ac= k=3D1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=3D1,ack=3D1'= packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from server side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D ntohl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } - return 0; } =20 @@ -190,7 +245,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, =20 if (sender =3D=3D nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +258,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717258802698.6722018734066; Sat, 10 Mar 2018 13:27:38 -0800 (PST) Received: from localhost ([::1]:51992 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum1h-00063U-UB for importer@patchew.org; Sat, 10 Mar 2018 16:27:37 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48382) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulwp-0002De-HX for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulwo-0003I8-6q for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:35 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:38263) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulwn-0003I1-UF for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:34 -0500 Received: by mail-pg0-x242.google.com with SMTP id a15so582994pgn.5 for ; Sat, 10 Mar 2018 13:22:33 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jZZL+cB7ywLrmktyU3wExkoQrmvin6i8y6BlcpE3nRg=; b=F930HI2eyiUKJQCoPkX+JBnQCeUpCjYtuSiMjifx+WXXcFhfyWY54Gfhu8DGX3BtKQ iTMuBAfpTIb9TE5YQ/mSaJlooecA2IxYiRrxs1mc1tglkbEsWlNhXX9ZC4afxgjIYG0C NIRKcP6Zz/eVEiCT0Q7oSqBZCVvJcNPcKQ07sligAdeyAuAXoHPpfER5AOUu1pH42q7V qd2Inpx2Mf0cGfWc3n0M9AyTm77+WVuJHyXw6ntCRzEIhYQPQrd/MXRQjEuZEsLZ3GPI 0XQdbemijVPaJOZgBbdSZsqf+lr6t3odZE6ppoS5keU3eAawbdmDDXVyqIlWC328qI9n F3xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jZZL+cB7ywLrmktyU3wExkoQrmvin6i8y6BlcpE3nRg=; b=JbLCJSliCwZpFDsM+u7tDnS3xrb3lZQK+X1ipfNEFeQhF5wSVrKkyhyIX7r37BaK7D eZo2vfRhkXtmK2gmFKbVwXJXwlieiiLewDEQNuKXGh4qtM7BNlRbo1x72lk5sWpZ3TTy koGOvUQz2vGC43w4JBiUkVsTTHR02+VPhaKlG4E2ARW+iUtKirPbRxsG0gQOgMeV2oaY kCrrwWe84xHA32p2Lg6+FRwQrVR63GsSARPJnYHsJv9GG5E+rMHkEBDPImkaMhDxSQ1X LhD+bZ6bZhlyI2tW31oLzNVn25GPIjZwLcTOUWYwnJJEoFk02UCzixmTn1qS99lG5QZu amyg== X-Gm-Message-State: AElRT7EaFaIBc848lXA3I0RUp/sb5XrHbkCcNdWtx5AwhuBm2b+/P0yG gACDHHXHs5uZjWTplVnHkQiu2+m8 X-Google-Smtp-Source: AG47ELui9DpPiIzEddp/Kx34X0FqX1/A7M4EJ/xuqZacDSbzveCsK4xTKMmxh9jpx5hXI2wVlGIOtQ== X-Received: by 10.99.0.19 with SMTP id 19mr2543078pga.25.1520716952669; Sat, 10 Mar 2018 13:22:32 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:52 +0800 Message-Id: <1520716927-17068-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V5 02/17] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++++++= ++++ net/colo-compare.h | 22 ++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad..fefb2fc 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" =20 +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); =20 void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 23b2d2c..7ff3ae8 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) =20 +static QTAILQ_HEAD(, CompareState) net_compares =3D + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 =20 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; =20 typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } =20 +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event =3D event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx =3D iothread_get_aio_context(s->iothread); @@ -756,6 +789,28 @@ static void colo_compare_timer_del(CompareState *s) } } =20 +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s =3D opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +824,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); =20 colo_compare_timer_init(s); + s->event_bh =3D qemu_bh_new(colo_compare_handle_event, s); } =20 static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +982,13 @@ static void colo_compare_complete(UserCreatable *uc, E= rror **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); =20 + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); =20 + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table =3D g_hash_table_new_full(connection_key_has= h, connection_key_equal, g_free, @@ -990,6 +1051,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s =3D COLO_COMPARE(obj); + CompareState *tmp =3D NULL; =20 qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1059,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); =20 @@ -1009,6 +1081,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717093759204.49779644480918; Sat, 10 Mar 2018 13:24:53 -0800 (PST) Received: from localhost ([::1]:51976 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulyy-0003eN-Im for importer@patchew.org; Sat, 10 Mar 2018 16:24:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48398) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulwt-0002EN-D8 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulwr-0003K2-T9 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:39 -0500 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:42898) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulwr-0003Jc-Mp for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:37 -0500 Received: by mail-pf0-x243.google.com with SMTP id a16so2716640pfn.9 for ; Sat, 10 Mar 2018 13:22:37 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RyVyyiWz6BVrwmCmJeVQGLsDL7nGb6Qbw2cd4DzIUuw=; b=EK29L9wAr2XDEWo/TFCwMYqniYU+1XDpr+aosmjjDOI2ik4sR3KCPzO5n8c7Vrq5ki mnA1StnHkFI3Sm1yKijK3qjWMyOHT2TrQ7YyQC5yMKWZ5rtXwbjS22+kHTsaw8Flge9n C1UpTu5vvdggA30pHCSEy9ofsxAPAqaFCED/YsjWIelYbFrgRcX6B+aHptnUnKrsaV/j 6leCpDedq+mU+YgH63hznH5O6aXO1tHDnfh5jjYF0Ohf4/tNlzz990yztwODFZvwq5Hx XiCIpSMLOffdbDbIuC76i75rbs8oT9OCuHVjpYGMqh5oJxMRftF0Hc7jtrmBQ5vwD6ZZ pmjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RyVyyiWz6BVrwmCmJeVQGLsDL7nGb6Qbw2cd4DzIUuw=; b=UeHTJH3nGFOILL85fUmGAFGgAEt9q/L9qDtLKE6BlFujF0/LXeQGan0nMbjbtbL0T1 2OG13IJq2lqIcj0PSfgdlx17sZRkS1+4oZ/FH3A7vrYaoNM5ASFAaanO/IDCvzckUkAA s0VVdGBBUvIk3qwE0rlMysXWq9irj4c/1IO9XGSqZvjxYgizAe7qzGqxm1E4svjku9G9 rqI7B02nkTI9HRkUWfuGRE8y4nt8SRJB+ZkJ+FTL0X8tPPk3Q34E8+XeEsumaiy7sYxY IW0xrimGrx347v4fhWJbKHmfvFMxDhW7y0Ucx8+RTrb0gFH3ELLZVxEzBIGNsteKepCy 4Ngw== X-Gm-Message-State: AElRT7EtpjZziVNw7mSlF5A1eWfwreedGTMSEjUU4mqQwkhdAD8PzVCl u80ShkIguzOkiIt64Wuk2RJ296kc X-Google-Smtp-Source: AG47ELsEtoe0gYXlnYAFQTOYITKL0r/P6Aqc9rKgJAGeqMTTnJDzEaD/QgmKQmF/5htaVQuO+k6pLw== X-Received: by 10.98.218.7 with SMTP id c7mr2998600pfh.162.1520716956498; Sat, 10 Mar 2018 13:22:36 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:53 +0800 Message-Id: <1520716927-17068-4-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V5 03/17] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 7ff3ae8..05061cd 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =3D QTAILQ_HEAD_INITIALIZER(net_compares); =20 +static NotifierList colo_compare_notifiers =3D + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -561,8 +565,24 @@ static int colo_old_packet_check_one(Packet *pkt, int6= 4_t *check_time) } } =20 +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result =3D NULL; int64_t check_time =3D REGULAR_PACKET_CHECK_MS; @@ -573,10 +593,7 @@ static int colo_old_packet_check_one_conn(Connection *= conn, =20 if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } =20 @@ -620,11 +637,12 @@ static void colo_compare_packet(CompareState *s, Conn= ection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76..22ddd51 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H =20 void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); =20 #endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717267159223.56877144029409; Sat, 10 Mar 2018 13:27:47 -0800 (PST) Received: from localhost ([::1]:51994 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum1q-0006Bu-8v for importer@patchew.org; Sat, 10 Mar 2018 16:27:46 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48415) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulwx-0002G3-97 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulww-0003Kd-45 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:43 -0500 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:36788) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulwv-0003KM-Rg for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:42 -0500 Received: by mail-pg0-x241.google.com with SMTP id i14so4940982pgv.3 for ; Sat, 10 Mar 2018 13:22:41 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nr70ZvGv7z5M6QMDWePDS+ONDj2QkdxJHgd5N8gmR3M=; b=VhAjaZMd9N2C9kS9419Czg0OFEcBt8zGEksjwCr7K8By9jn7VE7z9nF6kmmY9TAWOH kgH52NexuiseEozvW7c25V1hw4XQ9Z3zzuRB5XaM+5S4svZfcSZGVZJyOUMZfJAbMACg 9zB3IHjmHWUblPxGiBxx0i2gd2qfZw5tsXlqWKbYvg5hgmtrxdmebI1Tbl10m3yTcr0k gCr5+WycqIKMmJ6NUiPoCz8GKr5CmowNq9ejLTgVUw/0gm7xVdtqVHFiySDWAJLmwrxJ /+al2J3OH8iapS1a2R/GXwMagYU+8mEvp8PYjMt06UFsmIKAd3OlbiqPoWziOcX//wpG +89Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nr70ZvGv7z5M6QMDWePDS+ONDj2QkdxJHgd5N8gmR3M=; b=np1yIW7RfZmiAGmXb3ZAV9eWyNL+y5JBBnxl4riqs5IEbQ3jQXxhD7fu6Bcvf40L3y b6ufNTrJdPEX+aYxBe7sh7gsHfbWd4lHGwexOWGXqsPzcurw9Cxqi3gDc/Oj6nTRrOI4 2H7R+BFxFLGW9PM4KXMqfOWNi9lLqqsMhOvZaj/EFJeB6SIPtPcONGkboOhY8/tYg/0/ cbuMNrBESH8p2fj9kYhT/QuHT1aocP7Lm1/xC6aLltH7o7Ht4FUADUTOGv4l8xfkk4Qq wJt7k+z5jhkFUDSOb48VC3TilWg09aZ9xoMNI+u+pQ4YqAAFPmKmbpi7KI2vPd2r4MvD qQoQ== X-Gm-Message-State: AElRT7GH/F2PhPYfGHQdFJrps5iXz0kYD/6tTW7zlnL2Iifea5xz4hPd CfpcdrmMPD2qE0gsiHzDph9VIiuk X-Google-Smtp-Source: AG47ELuyClOoc1NFdoBrDKkHYzsLb2nbt3Kd3ZD3TBJCL1KQ7DDrYFdOwJDutSgQQ6V+LLZwo5hGlA== X-Received: by 10.98.220.207 with SMTP id c76mr2976832pfl.159.1520716960611; Sat, 10 Mar 2018 13:22:40 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:54 +0800 Message-Id: <1520716927-17068-5-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH V5 04/17] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067..081df18 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" =20 static bool vmstate_loading; +static Notifier packets_compare_notifier; =20 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) =20 @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } =20 +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 + packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } =20 - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } =20 vmstate_loading =3D false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); =20 if (failover_get_state() =3D=3D FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index e345d0c..7f72f78 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -76,9 +76,9 @@ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) =20 /* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid = mode. + * Note: Please change this default value to 20000 when we support hybrid = mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 =20 --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717262530748.077348417764; Sat, 10 Mar 2018 13:27:42 -0800 (PST) Received: from localhost ([::1]:51993 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum1l-00067y-L1 for importer@patchew.org; Sat, 10 Mar 2018 16:27:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48434) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulx3-0002Lj-8m for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulx0-0003L7-3A for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:49 -0500 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:32850) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulwz-0003Kw-Rr for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:46 -0500 Received: by mail-pf0-x243.google.com with SMTP id q13so2726049pff.0 for ; Sat, 10 Mar 2018 13:22:45 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dV39DM1htJcTRZ9VSV+34MRY12c//1ZBNeILrPdCi1s=; b=Htx8svTe5uz8rWJcYpI7L/nk3xO2R2Mr5Mpcbor/XU7NG1YT0NUm1D7z7J9MeRk2AV CtxsaRCXoAaU6keeLKhHtyOy9UYQHfPN8SqRtdaSBjM2rVfXP3OWEzs/MwhOxl5S8fCk AdzvZyuLOcufbZ2Sq0nN1wHUkjFeM+LmboATpMjddzvwP26e8UDPkFgJG4Wj3m1RWW86 aG3+tQ4NXRbeT82eLMHInukxVry51zJb89vjSRiDdfgvKdJ0PNwB4aaMSRSYWYWa2291 xx2jLSB62o54wLQBHwvRNuqvvsBDc7L4nsjekKgb+2c0kY5xrShIHWmIM/hZ0olpKaVR DufA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dV39DM1htJcTRZ9VSV+34MRY12c//1ZBNeILrPdCi1s=; b=dlguB0LESjazXqYTc0ZAkdmDhrL4UJ1Y6+jH0y/3vOeojN9OFgRArpPGWVgvGPBJlL 4HTCiAbg0WsWn8BzKL5c1kDb8t3e7Bu5egunfFUKmpQ7XkbNK5Paabuf8QKSS9uijYy2 GTatS/X6RdKufrOT99OG3jNVZHkDzaFZ8KLwBa8wLy3+5PHvwp1RKNIpa0oee4rH2nuo 8zt1RI5vTS05JRHzpUuX9AztcXfnCTAZSiTF7hrqzYmQiHJaDxVSpnosUbVwH88xj+G4 Y5iJIXpUCcHPySkFLVrOP/CXn3MedeeL3ayst4F/BgivT23ua+BphbgwjcNXKyE9GqW+ Bieg== X-Gm-Message-State: AElRT7HjaZH3SOFVidaWzC0kLYZb/abpV4dHL+NOiUifDFVvMvaAKMrf XWuIiMx2UQ4KD/yOnKaRtH/rIJtN X-Google-Smtp-Source: AG47ELs59oHPpFIxZrW/NKXuIJaMhFmjE145sXLfXJDQafG9OOJrm1Dp71T52iPhO7sAK8DZtj0QoQ== X-Received: by 10.98.200.80 with SMTP id z77mr2940059pff.85.1520716964698; Sat, 10 Mar 2018 13:22:44 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:55 +0800 Message-Id: <1520716927-17068-6-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V5 05/17] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df18..e06640c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; =20 /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); =20 + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s =3D migrate_get_current(); int old_state; + Error *local_err =3D NULL; =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err =3D NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); =20 @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 7f72f78..128daaa 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -346,6 +346,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis =3D migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err =3D NULL; =20 assert(mis->from_src_file); mis->largest_page_size =3D qemu_ram_pagesize_largest(); @@ -377,6 +378,14 @@ static void process_incoming_migration_co(void *opaque) =20 /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717107763193.95275995079737; Sat, 10 Mar 2018 13:25:07 -0800 (PST) Received: from localhost ([::1]:51977 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulzG-0003t9-OS for importer@patchew.org; Sat, 10 Mar 2018 16:25:06 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48460) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulx6-0002Qq-Of for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulx4-0003Lt-Qd for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:52 -0500 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:38237) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulx4-0003Ld-Hn for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:50 -0500 Received: by mail-pf0-x242.google.com with SMTP id d26so2721660pfn.5 for ; Sat, 10 Mar 2018 13:22:50 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SPBx8J3iYP9KDnmMYNDMC2qTaHhQpjQcoJU9L4s1kOY=; b=HUqEbMe31HU1QaZ3bZVF5MDhianAOgALbC38ZNk/MHWM1ncRqxjoTJYhR3+7EZabeQ +8KyXaaY2DuAL8ryQnovi0+BN6Zx5z1MQ5JRLY1ds3BaDkINtTJUNLEBJ/EGxdzx9NJ1 B7W0uAxZUed75JpFd8WqqYWYTo7GzgxfJNBVg6AYRjyZzv5ocaOyk6wDz0wyxCoKqqVj K1zRvs3kA5PHxZAzpFmebOLoZs7xjmhMy7N0kYRbRwrdXqrjquwXOify8nV7lnqrksrl CAxv2lLymvA52MefdSbbIcjDtLhg/NqSZpSXYioP9EY0aMRTUZCrjpUvW4/ZVnD57LSM QPFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SPBx8J3iYP9KDnmMYNDMC2qTaHhQpjQcoJU9L4s1kOY=; b=i/WFsmoonTNnXWnRog59JBe/5gW3ncGmyDeWnKMp0tFvk+1K4Bo1QgXHpaxqSTgWFt ZkLFJNSPSoSzzkIoIOKIVHPesDY7/BDHgQ6WrrqGnT+N7RV6fK3VC2tWMOiiunKGrGuv aR6dyDCs6yNY9hzD/aftq6LEs6Nsbf37wNVIk+HLIHP8o3uJyTIImbPRQ+fZz7XzBLXQ x8wjDFWf2Ma+bQgiwViSPViiOmRPA/s7M9OljvTcteOWkqzP1chYUMNd4btBkV3wzYbk VgetqoLsJ/3qeAKH6/lxGHJ6Jn9F5Ho1OqiROmGt5bFJaha3xLrH6iavHbLLJjz4fCqU GfKA== X-Gm-Message-State: AElRT7HHipM8HIiV0dLD2C/fud6H0bPClvURqv88pME9u6veRi01rPPY /1J9kJc/kU1SBOBo9glIVnKXsoBs X-Google-Smtp-Source: AG47ELuQIDl2esGeA730kbWCpoW7DUThLoq56+yBqnSdgn7nKzDDYEB+jsyckYk9I1n1UjtB0Lb3kw== X-Received: by 10.99.106.71 with SMTP id f68mr2530918pgc.262.1520716969258; Sat, 10 Mar 2018 13:22:49 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:56 +0800 Message-Id: <1520716927-17068-7-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V5 06/17] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 --------------------------------------------= ---- migration/colo.c | 13 ++++++++- migration/migration.c | 23 ++++++++++++++- migration/savevm.c | 19 ++++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 59 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fc..99ce17a 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); =20 /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); =20 diff --git a/migration/Makefile.objs b/migration/Makefile.objs index 99e0380..3099eec 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y +=3D migration.o socket.o fd.o exec.o common-obj-y +=3D tls.o channel.o savevm.o -common-obj-y +=3D colo-comm.o colo.o colo-failover.o +common-obj-y +=3D colo.o colo-failover.o common-obj-y +=3D vmstate.o vmstate-types.o page_cache.o common-obj-y +=3D qemu-file.o global_state.o common-obj-y +=3D qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4d..0000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s =3D opaque; - - s->colo_requested =3D migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state =3D { - .name =3D "COLOState", - .version_id =3D 1, - .minimum_version_id =3D 1, - .pre_save =3D colo_info_pre_save, - .needed =3D colo_info_need, - .fields =3D (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested =3D false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c..c083d36 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } =20 +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); =20 return NULL; } diff --git a/migration/migration.c b/migration/migration.c index 128daaa..47328df 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -268,6 +268,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *= mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } =20 +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled =3D false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled =3D true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -377,7 +393,7 @@ static void process_incoming_migration_co(void *opaque) } =20 /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2365,6 +2381,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } =20 + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); =20 s->setup_time =3D qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index 358c5b5..10a6676 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -54,6 +54,7 @@ #include "qemu/cutils.h" #include "io/channel-buffer.h" #include "io/channel-file.h" +#include "migration/colo.h" =20 #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -80,6 +81,9 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream= */ + + MIG_CMD_ENABLE_COLO, /* Enable COLO */ + MIG_CMD_MAX }; =20 @@ -835,6 +839,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } =20 +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1769,6 +1779,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncom= ingState *mis) return ret; } =20 +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1842,6 +1858,9 @@ static int loadvm_process_command(QEMUFile *f) =20 case MIG_CMD_POSTCOPY_RAM_DISCARD: return loadvm_postcopy_ram_handle_discard(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } =20 return 0; diff --git a/migration/savevm.h b/migration/savevm.h index 295c4a1..041d23c 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -51,6 +51,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, c= onst char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 93961de..b27a3a0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -34,6 +34,7 @@ savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" +savevm_send_colo_enable(void) "" savevm_state_setup(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" diff --git a/vl.c b/vl.c index dae986b..e67750d 100644 --- a/vl.c +++ b/vl.c @@ -4471,8 +4471,6 @@ int main(int argc, char **argv, char **envp) #endif } =20 - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15207174278351001.5882385699391; Sat, 10 Mar 2018 13:30:27 -0800 (PST) Received: from localhost ([::1]:52005 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum4Q-0000EG-Pa for importer@patchew.org; Sat, 10 Mar 2018 16:30:26 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48506) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxC-0002WC-Fn for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulx9-0003NB-Ae for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:58 -0500 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:32800) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulx9-0003Ml-11 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:55 -0500 Received: by mail-pl0-x244.google.com with SMTP id c11-v6so7172684plo.0 for ; Sat, 10 Mar 2018 13:22:54 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FwbfEomgswg5pfZiHLpZIZOP0KsxS2HQUOa6bKdBK5Y=; b=dKUdoMUaNf36+WwIjv0uNb19eo1xfs64fUgirmSzY7LngLOCBfb6ZXCaaLmXtsrtMX Tf5zWJyr0HFHdFMbMEtx7e8gHqe+afcuriGnPGXV3EuV6il3ILObpYtLb4OdL5UWBroT /kR+i8PMHhiMxYzzG2yA8RTybmaXM/QV/vY2JMSUXVGeWPOmseAe9OOn/9tLRNIYqPwi 8Zn5vwO10AiDaQ534r+Qw6yQHQZZS1Ga4LBdImtOv1Ael1CnDr6pSdjD/7FqcOAXBoTp g63ctyCAEcBRm8v3xByCyamdR2KAk83rkIifxorB67iFpOi3HrBYQvv6fphb5V98SEHN QZBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FwbfEomgswg5pfZiHLpZIZOP0KsxS2HQUOa6bKdBK5Y=; b=fRF2kRWBFYz+XtVApdQmOafn8FMd6OvF+xNJLyhqat96Sf0u0Ly5+BH9SuMdQZ9En4 jVXN5Et+Wv9gVqPsj608H68t3mY3MzcWblRTNp5QIQUOyCmSHgynqwauQrQ8qHTqN1XV mGb+LAy0qJXCG3v7AfYRPk+BuqdNunGMOeceNYs9bchva9P0yDg1pfxiBIKfZbMkOpoC 85ImDRw+uLoEnVEOm9jeK3HrdeuM8VTHrd3ldlTw3Yv/OBloHx0L7vkGj1y0yPyyUZJM Un2JRPGHbvCnLget2wU7V1KM3dK4QcWJNQdYkq8vHNfX8IX+THX1X0OT9QdF+EC2gXsi /NOA== X-Gm-Message-State: AElRT7HRoxlcV5ofnxspbsJRmJe5derpHA8AiQqQ9FoCHa7Qftf0VGwy QHpG6qE/2J1Z6rhsu50y+WI8LIJj X-Google-Smtp-Source: AG47ELu8TJYA5QzlvqqI+NKWeq3jeFKoGKld6opPy9/3IvUlKn2dT4WImTDHKzTclbOVPaCe9B51BQ== X-Received: by 2002:a17:902:57c6:: with SMTP id g6-v6mr3146309plj.358.1520716973413; Sat, 10 Mar 2018 13:22:53 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:57 +0800 Message-Id: <1520716927-17068-8-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V5 07/17] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary= side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram ca= che always the same as PVM's memory at every checkpoint, then we flush this cac= hed ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 99 +++++++++++++++++++++++++++++++++++++++++++++= ++-- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 104 insertions(+), 4 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf2446a..51ec153 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 47328df..4d5e188 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -410,6 +410,8 @@ static void process_incoming_migration_co(void *opaque) =20 /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } =20 if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 3b6c077..86f5c3c 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2470,6 +2470,20 @@ static inline void *host_from_ram_block_offset(RAMBl= ock *block, return block->host + offset; } =20 +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2623,6 +2637,57 @@ static void decompress_data_with_multi_threads(QEMUF= ile *f, qemu_mutex_unlock(&decomp_done_lock); } =20 +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =3D qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %= s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -2636,6 +2701,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); compress_threads_load_setup(); ramblock_recv_map_init(); + return 0; } =20 @@ -2649,6 +2715,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap =3D NULL; } + return 0; } =20 @@ -2862,7 +2929,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) =20 while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host =3D NULL; + void *host =3D NULL, *host_bak =3D NULL; uint8_t ch; =20 addr =3D qemu_get_be64(f); @@ -2882,13 +2949,36 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_ca= che + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Privously, we copied all these memory in preparing stage of= COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects = the + * speed of the migration, but it obviously reduce the downtim= e of + * back-up all SVM'S memory in COLO preparing stage. + */ + if (migration_incoming_in_colo_state()) { + host =3D colo_cache_from_block_offset(block, addr); + /* After goes into COLO state, don't backup it any more */ + if (!migration_incoming_in_colo_state()) { + host_bak =3D host; + } + } + if (!migration_incoming_in_colo_state()) { + host =3D host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } =20 @@ -2983,6 +3073,9 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) if (!ret) { ret =3D qemu_file_get_error(f); } + if (!ret && host_bak && host) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } =20 wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index 53f0021..3a8e5a6 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -63,4 +63,8 @@ int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_ad= dr); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t = nr); =20 +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index 10a6676..0d1f593 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1782,7 +1782,7 @@ static int loadvm_handle_cmd_packaged(MigrationIncomi= ngState *mis) static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } =20 /* --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717625751857.4757204501685; Sat, 10 Mar 2018 13:33:45 -0800 (PST) Received: from localhost ([::1]:52024 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum7c-0002rG-ST for importer@patchew.org; Sat, 10 Mar 2018 16:33:44 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48518) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxD-0002XR-Nt for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxC-0003Nn-Lm for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:59 -0500 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:43166) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxC-0003Nf-GD for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:22:58 -0500 Received: by mail-pg0-x241.google.com with SMTP id e9so4929753pgs.10 for ; Sat, 10 Mar 2018 13:22:58 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:22:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lxN6SpcWZNUdS22P6NJoYX6YVb5xrWvofqgkGV4t3u0=; b=Pej2Tpl1W3oGVTuEQO1ILd2uhHPriqg8OI8m5XVY6y4HnnFlva1ZD+FlGjYsF5ZoXO AaDUE3JMmK4oMKycjbqTxIrGhI7BTHfxNAqj3L+ZqrrTTTJ4X23PC/L8a6/QysW0S1Bh U+bNVxgTiOToQVlBavvnQ8fiiWVY6IuLRX3I0oUH5KiG4yYwX9/qM6oYhX6bvOQgX6NB 3XxkjxpRSAqiP9phEtdyMfCWNairK2LGT9tzlqZ3Uw/9c0UHfb2TJNTTF7FqIypOFSyr vdRK3pJolTq3JlGeXm3+nvSBcbbQUHKZWv/nB9CLRAqn65nEfAVTuiMvVkIOTNY5138v jUHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lxN6SpcWZNUdS22P6NJoYX6YVb5xrWvofqgkGV4t3u0=; b=LfuG7sVPzhX4mTrW9x6fcuukQOCrs+2iii9Hpdd6LUHy4Kf/vYBwsDxPgVjsZzqZzG epPUsrZg6FwEGtg5CAF5qRvtbklNMyoX20VHI878AOEkxZqoe5RHuRQ6oTMP2tFb+dfQ PwQq0O2fKnUO5M5GyPxBfirV/VcosyVpzjvu/sSDq0l/m4dpXeOO208RJkbmFyD9tN3z kqKoai0BA2ii49EdtsHa/6YfH4snBOJUJj6xfI6Fp9xHgoGrKMTMQUlBZIOKrp2LLQdu W9Z96vgHNs0+ObGwDuwUwAkGfpJjbEv2iZThZJABvfWtEObAP6aelzhytg9pipl2KTUu fKgA== X-Gm-Message-State: AElRT7EC3jNko44bwf9pI5OBId9cI84dbpRZDomj8dHnXqd05EZUjm61 fznhOhj8PeBmCsr5lmrDwDq0+yAg X-Google-Smtp-Source: AG47ELuZXll35dRuTfH6uDuQ1IGEOWCtXEyRldI5N9zUE95VIEdyqZYNsWMcwAEHRQyJgGomaDQikA== X-Received: by 10.98.217.76 with SMTP id s73mr2967210pfg.209.1520716977334; Sat, 10 Mar 2018 13:22:57 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:58 +0800 Message-Id: <1520716927-17068-9-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH V5 08/17] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 86f5c3c..4534c7a 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2481,6 +2481,15 @@ static inline void *colo_cache_from_block_offset(RAM= Block *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } =20 @@ -2659,6 +2668,24 @@ int colo_init_ram_cache(void) } } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap to= gether + * with to decide which page in cache should be flushed into SVM's RAM.= Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; + + block->bmap =3D bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state =3D g_new0(RAMState, 1); + ram_state->migration_dirty_pages =3D 0; + return 0; =20 out_locked: @@ -2678,6 +2705,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap =3D NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2686,6 +2717,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state =3D NULL; } =20 /** --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717828469463.3032246403054; Sat, 10 Mar 2018 13:37:08 -0800 (PST) Received: from localhost ([::1]:52041 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eumAp-0005fo-K9 for importer@patchew.org; Sat, 10 Mar 2018 16:37:03 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48546) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxH-0002bW-ND for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxG-0003Oc-L5 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:03 -0500 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:45929) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxG-0003OT-Eg for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:02 -0500 Received: by mail-pg0-x244.google.com with SMTP id s13so2025749pgn.12 for ; Sat, 10 Mar 2018 13:23:02 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.22.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XqbOQdGJaaK869FsHU8cyOLMJv+HPHjhy3tE5GWcJ+0=; b=ZyQqNN3Z4z34X+UbkuU8RAH1t+FhWGaI8r4dOu6BjOLevMk6GNN0iBtq2Uonp0GjtR HuexkButXfq1jikYzR6fSUNr/lIzb/wVZhJ99xkZdjlyfVcxXGA4koJo24lfu5RWNUjT At6bNuDcC/3WmcTh8HpxgP6Mfd9QkkA/mxY/2GwUuLcS98E4fvpVuJRGRHMUv0gJQLjv DXm03u08rftGmy6/hMy0oh+t982pSI9gzLWOS2xLxWO5VuTSpVn5jvSBIqxE4g93x+8z PJa0hEbQ7bWnjurRpnIKfr0nAZepuuVvl97nkQZtBc0q41s79/ni63ph2c7Vfg/8PLGJ ddlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XqbOQdGJaaK869FsHU8cyOLMJv+HPHjhy3tE5GWcJ+0=; b=NpwWxaVBP5lsU4jRtwPWQLMEfW5Wh7x3qVDvX6sTB0f5zsdSlKNXeyDpGccNTjLYN6 yW+BCxznL4v0MGX/p6VyawQxkBgAhoJPqLSG2SlIC+jQkpDjzUGbOqPCKdbr/o8yG0PD belViv9OKbUov3379h3rVsiKW6iKWBwPYalMcqWq6/EmN3O7M/EjYjNuHsF5qc2gfZD8 w5TF3FQXnvSjkacVODCKSjV9e2xf/QqwEHYYDlVbU9IsIl+u3FB3jgMY82Kz1cg5lBRn aLpVdqhUOy4rLPbTnKFxa+oop43vx1jyE42j/I2l8wt8HHJjuatgCmSXbHPREHi8ZErn gW1g== X-Gm-Message-State: AElRT7HIS3hOYB3R9ZhCJ0Q04sB3zdV0zkdSf5O3j/qvYX7tFDv+0DkA 1D0IesO7NTlR0e+UxgitPnK2AL+M X-Google-Smtp-Source: AG47ELtU9tOFm+WzJiz6o4vJnLECVfbQgXCmG1oV6gBgWuZ9DB/i+z08JjJIDIP2KjaUAQLdtnyfsw== X-Received: by 10.98.245.131 with SMTP id b3mr3040367pfm.20.1520716981226; Sat, 10 Mar 2018 13:23:01 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:21:59 +0800 Message-Id: <1520716927-17068-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V5 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkp= oint time. So, the content of SVM's RAM cache will always be same with PVM's mem= ory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 4534c7a..ff7265e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2927,6 +2927,40 @@ static bool postcopy_is_running(void) return ps >=3D POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_E= ND; } =20 +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block =3D NULL; + void *dst_host; + void *src_host; + unsigned long offset =3D 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + dst_host =3D block->host + (offset << TARGET_PAGE_BITS); + src_host =3D block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages =3D=3D 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags =3D 0, ret =3D 0, invalid_flags =3D 0; @@ -2939,6 +2973,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) bool postcopy_running =3D postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability = on */ bool postcopy_advised =3D postcopy_is_advised(); + bool need_flush =3D false; =20 seq_iter++; =20 @@ -3114,6 +3149,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } =20 diff --git a/migration/trace-events b/migration/trace-events index b27a3a0..a3a124b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRI= x64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset= : 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717657728494.07495602842744; Sat, 10 Mar 2018 13:34:17 -0800 (PST) Received: from localhost ([::1]:52026 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum88-0003Tg-VD for importer@patchew.org; Sat, 10 Mar 2018 16:34:17 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48564) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxN-0002hZ-RT for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxK-0003PN-NV for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:09 -0500 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:40253) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxK-0003PC-Go for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:06 -0500 Received: by mail-pg0-x243.google.com with SMTP id g8so4931473pgv.7 for ; Sat, 10 Mar 2018 13:23:06 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GD6x7tdnuKsONz6vJ1sF07NbBOLox/mgTLZpcyl+d1w=; b=hkj29JxvoUlTqwhK4d1Ikn2g/ToA09/rCUziL3SJmJDMlrqTvizX7R+6oqMslk6hJb It9wWbJe5ch8zCP+1DBBDc0r6byv2FSog8iiH8OmIBwhHSlcRzxcWSB2EsgUD2mozDBR jjpJwKDygzXcE/1mkuFVdIedx1k02deHFeJoWAi5KHGdlMGuHyFr6FF/vgnR8mJxPT6I hN13lQY8Mf220gxK94wvR9LOX7zLqqawreOLk5nHcqqpmn0DbTdnLH/i0lttrmhbBg/l pbZaPFNf9B6sfcoZdZ+a0u55qxmX0e/wU9bFu4z2ua5HPCeGgAZih0Ba6v3gYf5diYzD Mtcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GD6x7tdnuKsONz6vJ1sF07NbBOLox/mgTLZpcyl+d1w=; b=muYTAr+IGpneTNVTtoq4gnX+9wbGDiLvXDJjmovh1sdPFlVHucezLZbNrEZMHPj3ZX 8JO1GOvAwtg9oPA5r04hQHVjEWSC8aqxVeSE74caEXS77F30Q49DWw91jbtPGLMdhiOy t/Mzvqat3ScENjEh2oGeFN2cLRtcOe65+uaSC/DyUi5XQwcSu4EsPXwZetmGWjlRhLKb D4k+DgJBz5Yf2sk+5zHXec/TQzkrC+vwEKFTqtSi0ewsaxv/seW5xS2dwZLWtRjHBViW IJmJTyaSbsrooRnXiV0RPlbY7RHEwCatojTuyiX3mRJKGgmg0cj1h3koTbSNaD9thWin cwIQ== X-Gm-Message-State: AElRT7Ec7jXqdBtSzKRI38peN+kpC0mW7Sxrebiwcpd/U/vDigv8x17I dJcHx59KeeKLAyegf842ujfEA2Vs X-Google-Smtp-Source: AG47ELua7NSrTl9VWvEIQeWnpx2SOnqaS8cDRyoRic1lmwDYczVSb4RuE/Nke48/XSvZXEvByBRTRQ== X-Received: by 10.98.1.88 with SMTP id 85mr3005429pfb.226.1520716985287; Sat, 10 Mar 2018 13:23:05 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:00 +0800 Message-Id: <1520716927-17068-11-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V5 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c | 20 ++++++++++++++++++++ qapi/migration.json | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d36..8ca6381 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,18 @@ out: qemu_fclose(fb); } =20 + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -744,6 +757,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } =20 if (fb) { qemu_fclose(fb); diff --git a/qapi/migration.json b/qapi/migration.json index 7f465a1..6c6c50e 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -858,6 +858,41 @@ 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } =20 ## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.12 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "reques= t" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.12 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'request', 'error' ] } + +## # @x-colo-lost-heartbeat: # # Tell qemu that heartbeat is lost, request it to do takeover procedures. --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152071785557213.286609676104263; Sat, 10 Mar 2018 13:37:35 -0800 (PST) Received: from localhost ([::1]:52043 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eumBK-00065O-PQ for importer@patchew.org; Sat, 10 Mar 2018 16:37:34 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48575) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxQ-0002jj-2z for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:13 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxP-0003Ps-6V for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:12 -0500 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:33275) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxP-0003Pn-0b for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:11 -0500 Received: by mail-pg0-x244.google.com with SMTP id g12so4943331pgs.0 for ; Sat, 10 Mar 2018 13:23:10 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BV0Tmm+QGqTZSP4DC7KUbEzKRI1xCr0G5gQI0kcvnZM=; b=Vs0mlOXCMinWwfSwzI5cQ+wt8O7v82HtWlL0FR39hmSDK6lDJ2b3om0flNPXKzu9zi I6VuCSOPqe3wJICccnM2RzWjB5JSk7sQqlnmcH4BhE4qnxwf4xzFURi5gG9zN7JRtFE6 ISnqyaG7jP/0XBxCeJU2B6FpqmMHBDyHefncekLdkQ2NzrauzMCuXqVifu/NqtP5l+eA iXh/m08mQL9WQso4esOtORY3Sy5NlVVB4ocv1m9npnRwOOsbwhCL3TWI+Vy/aa0FK81s Z9rXtO4iZUtn1bm0mvbM/k0BOotzLLg9at6bnhrQgM4d8uKtt0vcP86nSMab+J04xcb4 ncjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BV0Tmm+QGqTZSP4DC7KUbEzKRI1xCr0G5gQI0kcvnZM=; b=bPall9NnVLMHwAZM+N7Yc2hdXmvGSuxqPuxLbCnBFjD4BQtGMdHyDJ4tLE+ZuaupJ7 gJ3hRVka4X0dOCqTGUEMOpQlD1VZsWiMmjWaguWa6UwXS/gW0XHhRkGEO42hLi/Q8YUf Ep+hQeA2K3XEk5bff5WcWs23Jc5gkbQ/dIpyWZkNHfwTBdHTOLlwHegfc+e+7R5CHufO +FvJ24adoDMoGKOMZfgi70GlozwrRvr4NyBHYxBe+2C2IuzHCS0cGe60NUNSYXKExULd aTYMKAOAnBqbNGLQXf9bwHCW/qM5fCc5ilnX1MbFrn7uGL6sSfCFM/JQOvMFBF6D7rb9 jPsg== X-Gm-Message-State: AElRT7HCRb18h0QesVcXiaqqxg3NgZpx4ulpy4a383aCgHAFxzPqlzWe 1HS9REcSVMEGYOXEH9op7gZ+kA4R X-Google-Smtp-Source: AG47ELvAFgEja2nJ+r4EsSJ9v1Xd7rAg2rT8tV0NbBTKz0mdDdd5ru3PI3RnoN/SvLliv6mS3cFMUA== X-Received: by 10.99.95.135 with SMTP id t129mr2627pgb.268.1520716989869; Sat, 10 Mar 2018 13:23:09 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:01 +0800 Message-Id: <1520716927-17068-12-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V5 11/17] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Libvirt or other high level sofware can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c | 35 +++++++++++++++++++++++++++++++++++ qapi/migration.json | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8ca6381..b4243ab 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -237,6 +237,41 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } =20 +COLOStatus *qmp_query_colo_status(Error **errp) +{ + MigrationState *m; + MigrationIncomingState *mis; + COLOStatus *s =3D g_new0(COLOStatus, 1); + + if (get_colo_mode() =3D=3D COLO_MODE_PRIMARY) { + m =3D migrate_get_current(); + + if (m->state =3D=3D MIGRATION_STATUS_COLO) { + s->colo_running =3D true; + } else { + s->colo_running =3D false; + } + s->mode =3D true; + } else { + mis =3D migration_incoming_get_current(); + + if (mis->state =3D=3D MIGRATION_STATUS_COLO) { + s->colo_running =3D true; + } else { + s->colo_running =3D false; + } + s->mode =3D false; + } + + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + s->reason =3D true; + } else { + s->reason =3D false; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index 6c6c50e..8f81f36 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1201,3 +1201,38 @@ # Since: 2.9 ## { 'command': 'xen-colo-do-checkpoint' } + +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: which COLO mode the VM was in when it exited. +# true is primary mode, false is secondary mode. +# +# @colo_running: if true means COLO running well, otherwise COLO have done. +# +# @reason: describes the reason for the COLO exit. +# true is error, false is user request. +# +# Since: 2.12 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'bool', 'colo_running': 'bool', 'reason': 'bool' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "colo_running": "true", "mode": "true", "reason": "true= " } } +# +# Since: 2.12 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717285765481.38510497932157; Sat, 10 Mar 2018 13:28:05 -0800 (PST) Received: from localhost ([::1]:51995 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum28-0006RT-P4 for importer@patchew.org; Sat, 10 Mar 2018 16:28:04 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48591) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxW-0002nk-U5 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxT-0003QM-PH for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:18 -0500 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:34367) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxT-0003QF-GF for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:15 -0500 Received: by mail-pl0-x242.google.com with SMTP id u13-v6so7168654plq.1 for ; Sat, 10 Mar 2018 13:23:15 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3NtIl+BhqScR/RvfCgbvQWP4zLV3DMAEEZeECT6rxQo=; b=vIaeiS5oJDrCsd31Sp4etbSwlICcW9X1eY9BS8osRE7VQcdcWJELzqw9UZFYet/+hL btDZXqBlPc7sr2WnlVV5CULT6tM4XhMl3OBBje9ZMH3fMXMu4cXWVXbUw+tGk1KAwm+Q u9/qbVO6/trV7I2W1UziE8HrxGaiY1XFZ32FR2iXCj0uVRep8C4m8ZfM2h4VnBtoB4t2 PbhuoDz0lZBAhMoJV4nNrMbnisCmNOS4ZLhSqbprlAR2r6qlE5iAbO0RGkVNBLXz/O3O 1At3XopGjHfRXy6z7842SfoQMsGNHg65hBIiauMtDADyPkEugOQVMw05JLu2xpt21sNP GjRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3NtIl+BhqScR/RvfCgbvQWP4zLV3DMAEEZeECT6rxQo=; b=W7foWxkBV5ORbjbxcuTVy+S8tdi9sJbHyEsXdDoQPXZhbl3x574vrtXn8+Lg1i8LqU d0yZXTfIiLjfp/CVP7V/WLZ996ATjgLR1HTlMZ6p3eng9vxS1U7I3vGG4FGhlaLEEcSF I+qc8swzWQnth3gxyfYKJ52E2jWvzF4AdvGWy6zIVdXHmj8kuid1sd0iz7M2p/sbDLk6 P2LPTfCqXNG0fAIwHwscYI19JZaYbdM25fNOeuIC9O0ClvzTotnze/LqVSRG9XvU8QEe Khr8B1rILBbQjNk2npNjhk/tqK5WzbIO4hx3TLG+xhH6mxxtpm+tdhayqplVWXh5C0kB AZfQ== X-Gm-Message-State: AElRT7GSkHyS6tNsCQHLinQm43ApVAbyp+97RgXCZ2fHMFvLxpkxsB8d x/NqqoEqFXuYnOlThLO4DQLiw5pz X-Google-Smtp-Source: AG47ELstgLiUkm5t5cUwA3xC/cwi15093vDN2UsqnFgE7g3UvHWdeZw9urEbQ091jMqWNWkRSLALZA== X-Received: by 2002:a17:902:56c:: with SMTP id 99-v6mr3030398plf.53.1520716994256; Sat, 10 Mar 2018 13:23:14 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:02 +0800 Message-Id: <1520716927-17068-13-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V5 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 36 ++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 60 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index b4243ab..1633aed 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "sysemu/cpus.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -415,24 +416,31 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, =20 /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); =20 colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret =3D qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -644,6 +652,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err =3D NULL; + int ret; =20 qemu_sem_init(&mis->colo_incoming_sem, 0); =20 @@ -716,6 +725,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value =3D colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -749,8 +768,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading =3D true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret =3D qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 0d1f593..e347f73 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1322,13 +1322,20 @@ done: return ret; } =20 -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} =20 - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; =20 + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1384,8 +1391,6 @@ enum LoadVMExitCodes { LOADVM_QUIT =3D 1, }; =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis= ); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2050,7 +2055,7 @@ void qemu_loadvm_state_cleanup(void) } } =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret =3D 0; @@ -2204,6 +2209,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } =20 +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 041d23c..8d463fd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,8 +52,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, = const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); =20 #endif --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520718018340471.4625207774915; Sat, 10 Mar 2018 13:40:18 -0800 (PST) Received: from localhost ([::1]:52057 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eumDx-0000Cs-GG for importer@patchew.org; Sat, 10 Mar 2018 16:40:17 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48609) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxY-0002uO-HQ for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxX-0003Qy-F7 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:20 -0500 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:43077) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxX-0003Qi-98 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:19 -0500 Received: by mail-pl0-x242.google.com with SMTP id f23-v6so7161962plr.10 for ; Sat, 10 Mar 2018 13:23:19 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1Ng5StD/qge1fH0589n/nBkXQnchc0g+28X6DXcdkB4=; b=ZSVPrLFpfCgP1sSBLAx9A4Z+nzozUfqOxjzonf4YOBI+Q48bQpHqjfAtNX3fBFxVGI iiZEPOBtaR6W2PzBNwFsIpI82CyUui/2kgZksMg4laNJA6bw4h4C/NULZhVVLSqq0s29 Jq7iUTS0ljufbjFvJE13wuekN70fLkgNE8JAmgtd8QXdwd/lswMWh7AIUhxMNHJykFBm ZYxovDvq/JhxzNcSXZ4dG828B1Xk2euRmUg2R4a0vH6abe7BR9VfBIHIBehXA2WIQ4Tp xqd1b76+jjzoajD2RfisKr6FEJ7JelAuGt4vlAVEk6dK88ozjervDZtlFS1nce2aIDZm AQew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1Ng5StD/qge1fH0589n/nBkXQnchc0g+28X6DXcdkB4=; b=F78wK5qmHsbEEY6u66yagTdOgozxUsm3jOPGYB2fWY86XKmBxBvQl9mY1fT+YZW8K0 GjEaos4GMF/rShihHPfP+NxFwv0MycmcU9snBuZ0HKfUTJd6IGcb6/VknBvEd5kO5tzo SKN3Gf+bSHJufnvSzP5w0BKfTH0WDyKLkPIz/isYfgfe6kHH9iNI7qEeOl8LwT3084UW 2ColHxh4hNJdQANY5fdfnG/+b+XbK/gG8s03Bnp0QsX+Jv6TC+YpVQ3azQuPT5X88eq/ 96B9SGA/JFoTwhBaqtG9IT49KSpuLM7MT4uzoUu1IdytyjfahVSK97m0jv5ADMGZwDm3 HFow== X-Gm-Message-State: AElRT7Ecvv5QoMzFwwr4irnriHA0PFd/PexAGdX3Aq9ar55p0b9vbnvg aqFrn7p2MAzHBh1TjlmeMLbaVfUm X-Google-Smtp-Source: AG47ELuEmYCQi6ySERFa4s0qO41q1eYG511tO6EwZJJoBmtriPfAuPonRHA5EEroxTZXC/PNn0mcJw== X-Received: by 2002:a17:902:4222:: with SMTP id g31-v6mr3160842pld.335.1520716998169; Sat, 10 Mar 2018 13:23:18 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:03 +0800 Message-Id: <1520716927-17068-14-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V5 13/17] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index ff7265e..003617e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2685,6 +2685,7 @@ int colo_init_ram_cache(void) } ram_state =3D g_new0(RAMState, 1); ram_state->migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); =20 return 0; =20 @@ -2705,10 +2706,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap =3D NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2938,6 +2941,15 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_lengt= h); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state->bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block =3D QLIST_FIRST_RCU(&ram_list.blocks); --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717457407344.0832605155888; Sat, 10 Mar 2018 13:30:57 -0800 (PST) Received: from localhost ([::1]:52012 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum4u-0000eq-Dv for importer@patchew.org; Sat, 10 Mar 2018 16:30:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48634) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxc-00033b-B0 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxb-0003Rf-B5 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:24 -0500 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:45434) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxb-0003Ra-5P for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:23 -0500 Received: by mail-pl0-x244.google.com with SMTP id v9-v6so7151935plp.12 for ; Sat, 10 Mar 2018 13:23:23 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xGJfOtyitB72l6WyXk30MaHec1wKPM9nbCkyT8g5f/A=; b=eUCPe70cBSWu7DAA1OzhTae9fotfXuHBfk56RSl65firKKW5sZ8wfaEQHVwSnULEcS EbHr/fWLgPMuFmKhP7yyX8aBTcqZ6He0mcQoBdbJzq+45x5r2CA8GhL/J6nR+H0RwpoA wwC/EWFk1monBPT792qB/fepa+Oib1PUn3MHcz1ss9zpQf3fScKG1BgI5OOtUmharzs8 e7Iq7Eq1Q2Ekt+1E+Bj9dnzjzsnU1fsDuCqRqPrDcH6vbMtEcsbwZgV6uq4ntvEEXD6e wFmqDmVQ1AJR7D5QU+UAiBClDt3eY3HIFBHpTUWOfXWUxIis6TwdVzR93g9ugccmq065 s9mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xGJfOtyitB72l6WyXk30MaHec1wKPM9nbCkyT8g5f/A=; b=rImKloCa1Zft2MwjkND5Qb7vxW5ysBJTZNMkrq5cSrYyETL67OsfcH5CRiZvnVtxPC DlK4ymDFIwRfWKL4j/T62/rvLfPB9W9L79UObfmlts+41kJGWZc3m/prQs2SIj5a/PRa 0mRyIfWcQL/W+oj3LQauwetmEHAAgF6KqfwVjtP14syb6QuAcjq6te15kSjQbl8vLh10 ZgOtqcnOJDKn1DUHzbSl+vbA96qJVQNC2D7+aUrN7UdP1/vh73jfxLvNneAkZJKj1E8c TpfgmY+6shN7J77TN2Kq6DeLADLRsosJMND7UM6iYwf6DlxBDODK6XRcWcDGvkVcX4uO 4c6g== X-Gm-Message-State: AElRT7HFlhaKeGlr6VGZMZb3DIrVQIRay/HM3mzTBUBuMXoo4Wt1O7on RJwoXhkiyLUcO8fk/lCq8Q/lwYHn X-Google-Smtp-Source: AG47ELtefaMORru3GhmSGLTAdsddcqPidRZduwlyiCeFGXSKTnFr8ZgYH1kjnTG3aRIlrepfiOOvZg== X-Received: by 2002:a17:902:6183:: with SMTP id u3-v6mr433947plj.390.1520717001964; Sat, 10 Mar 2018 13:23:21 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:04 +0800 Message-Id: <1520716927-17068-15-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V5 14/17] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6..49da666 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, =20 typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); =20 +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **e= rrp); + typedef struct NetFilterClass { ObjectClass parent_class; =20 @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sende= r, int iovcnt, void *opaque); =20 +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d..0f17eba 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" =20 static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } =20 +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc =3D NETFILTER_CLASS(oc); =20 ucc->complete =3D netfilter_complete; + nfc->handle_event =3D dummy_handle_event; } =20 static const TypeInfo netfilter_info =3D { diff --git a/net/net.c b/net/net.c index 5222e45..5bdbd6a 100644 --- a/net/net.c +++ b/net/net.c @@ -1336,6 +1336,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdi= ct) } } =20 +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc =3D NULL; + Error *local_err =3D NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer =3D nc->peer; + type =3D nc->info->type; + if (!peer || type !=3D NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =3D NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717657540742.3990187359668; Sat, 10 Mar 2018 13:34:17 -0800 (PST) Received: from localhost ([::1]:52025 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eum88-0003Si-Ht for importer@patchew.org; Sat, 10 Mar 2018 16:34:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48664) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxg-00037N-3b for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxf-0003Sj-4z for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:28 -0500 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:37779) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxe-0003SZ-V3 for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:27 -0500 Received: by mail-pl0-x241.google.com with SMTP id w12-v6so7169350plp.4 for ; Sat, 10 Mar 2018 13:23:26 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yijnkq2aZJzcfDOsrvzCmN7/fNn/vCG0KxlIi2wD7mQ=; b=EtcHL8sbMhFFFuRH9b0BzoneuesHyZc6jmN6Cgzj+pFx9sSc0D14x/eY85fSmkXL0U +z6H8Sjs2kNzE6zM2yV/7oWTNptq3eqKgQ081bd5XD1KIdHdCGqF9npYJ/EPgwZcbQU8 wP0F0riAAYphZZUMuyEz0gozyC7IdykWSVmjtCDf18Q5QweuNDohCfpbHJZBvrEMXOeD 8jheF7aCzlKqLuZNoZ2KfeLwQhmdkDidlxwD0S00Z9N+BUkH/vEKoRvis3OifSlBIsPw W6nWvqvEm0hanJ6J0UeBw7Bdh88Oy9sbmP040PBzPEtQKEMWa68N93ywGOJnttR51vnU sQtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yijnkq2aZJzcfDOsrvzCmN7/fNn/vCG0KxlIi2wD7mQ=; b=KHLiIVwdrsixBBlLpEowl+bhijcK/olzeD39+eZ47b2pQfAmx2RhdP/rACw14MbA/E 1hW9kvkzdHy0NV7C376hhGzfhoehX7/gIeUMPeYUOwvxPlg4AqhAVT5slgx1V6l719rY yvnC8UzD4oY+mI04LKrrSOs2Q4D3pWiwR2pBFLTUGLkVrQizC5Gq148NEPOWST8FyqIp ZBKUE+IDq1+mlyyu7WZ6hcaTeqHMvMJ6C7d+SS5w+4Fpqng6Ztl1qrKtAmP8uizyroZH yV2ZDA/nUJ9KHgEWvohhbJtIs8r+UeSWnc8QI5SipfT1EvGEhOBkolzHTzG83lVRucJg Ob5A== X-Gm-Message-State: AElRT7F/qG81vXEtvxWl84b7YMIkZUQFqvVSq0rvnuu+9l/otbb3s4ea pnfrkEZZdrPXbFqVlwLzdsmNYncT X-Google-Smtp-Source: AG47ELtJD7r0urY1LzA6o2dfIFQ1mbl0gC+7Uc7JSOJQ28VfCQKFbivTy5+YP30Y1fpRaqm1D03iLQ== X-Received: by 2002:a17:902:7598:: with SMTP id j24-v6mr3089083pll.1.1520717005736; Sat, 10 Mar 2018 13:23:25 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:05 +0800 Message-Id: <1520716927-17068-16-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V5 15/17] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 1633aed..6834f3d 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "sysemu/cpus.h" +#include "net/filter.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -81,6 +82,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -787,6 +793,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 0909a9a..f3c306c 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,6 +20,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" =20 #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -277,6 +279,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterStat= e *nf, return 0; } =20 +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_d= ata) +{ + Connection *conn =3D (Connection *)value; + + conn->offset =3D 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn =3D (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs =3D FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s =3D FILTER_COLO_REWRITER(nf); @@ -332,6 +371,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, v= oid *data) nfc->setup =3D colo_rewriter_setup; nfc->cleanup =3D colo_rewriter_cleanup; nfc->receive_iov =3D colo_rewriter_receive_iov; + nfc->handle_event =3D colo_rewriter_handle_event; } =20 static const TypeInfo colo_rewriter_info =3D { --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717995836988.5365039896773; Sat, 10 Mar 2018 13:39:55 -0800 (PST) Received: from localhost ([::1]:52054 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eumDb-0008JT-2c for importer@patchew.org; Sat, 10 Mar 2018 16:39:55 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48694) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxj-0003Aw-JV for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxi-0003Ta-MK for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:31 -0500 Received: from mail-pg0-x243.google.com ([2607:f8b0:400e:c05::243]:34051) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxi-0003TS-Fu for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:30 -0500 Received: by mail-pg0-x243.google.com with SMTP id m19so4937137pgn.1 for ; Sat, 10 Mar 2018 13:23:30 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5T7rpXvin1NnHDIeZ5/gvUFFwCa9t11nOHknSFjGTsM=; b=XRe7q1LdANSEVfGNY1n9ebz9gRJthWcH+b03OMy/ah0WqnU5GvgyKstPhUhT3poltk 8nrSM3wJYYA4q3BisAea9hshGFpy9r47jKwWMq0ha4/VsT2YxEytYWcCHxsJmOiRnr5w 0OkmMwG7JjZo9UjQA1bQJ+PabYY/3IGh7WRoqkXSDdcAJjKyUD9JF2lJk2EKgFcxCPNE AxDNUYtoP48LIS91umfsQcOqqepMzLIIYprnEbK4NimvUzpO+NgDsZnaE82e5RA82M5b jrfcTrSa+ZTvRyMboO0P5IkajBhtXhILCIEfCY0J51z4myInzvFd4MZEr558WnyTvUVh pdeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5T7rpXvin1NnHDIeZ5/gvUFFwCa9t11nOHknSFjGTsM=; b=haPlgA5qnTnMd/O5alDHbKrW9DC+yXPX7UBX+f7GZ9Pu7Rhu/EFbzmIj6C4Mhim0Fd dA+f1bYd9EX68R8GqLm8B2H5w0oge6uyaWhGFISfbA0AB9nbT8JqrJ9Gli4O+8o82NVw cQ1CeB6n/ygH+KxbvGmS1MbaVghDkCVNhT94hl1R0GhOY2pIeOvicq14opd2hLn6lmNE HwnXlcxb15AJRUvq5MFQrl7ssoMjSxMojOxN1pwahL3FGuF1E+chS4XjFruv3Qud4/DH YSKiD4wcxQJw5VHQKWuz8clMLOezgz0/8oOS9CoKhTqJq067xZ15lcnBHhf3sZhzhQt4 SDlw== X-Gm-Message-State: AElRT7FgOd/5KTGfduXqS9Clhh39cQ2hFW1td/EmGxYQf+DY1kmMqi/Q u1pMD6xlnTVa8W0sfOADvUoRYK0s X-Google-Smtp-Source: AG47ELsKdI6i06UH/t8dwiejJ5i5HReP7gz6km8XRvBIbqSOKDnS7y7VgYktxHuWzN8id61HLMb5iA== X-Received: by 10.98.218.7 with SMTP id c7mr2999818pfh.162.1520717009377; Sat, 10 Mar 2018 13:23:29 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:06 +0800 Message-Id: <1520716927-17068-17-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::243 Subject: [Qemu-devel] [PATCH V5 16/17] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 6834f3d..5fb4d8c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -87,6 +87,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -800,6 +805,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); --=20 2.7.4 From nobody Fri Oct 24 23:54:21 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520717854243313.13866935343117; Sat, 10 Mar 2018 13:37:34 -0800 (PST) Received: from localhost ([::1]:52042 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eumBJ-00064l-Fy for importer@patchew.org; Sat, 10 Mar 2018 16:37:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48710) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eulxq-0003FT-0K for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eulxm-0003U2-Sf for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:37 -0500 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:46891) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eulxm-0003Tw-Mt for qemu-devel@nongnu.org; Sat, 10 Mar 2018 16:23:34 -0500 Received: by mail-pl0-x244.google.com with SMTP id y8-v6so7140987pll.13 for ; Sat, 10 Mar 2018 13:23:34 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id q65sm10532191pfd.140.2018.03.10.13.23.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 10 Mar 2018 13:23:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pYSpfSOCMnJHb7OrZlaDcPmcRr6aYPMvPFICQCYTPDU=; b=VlUPaSKiwxTysk89OO/FkzPwbjOzV03WfV4aUHUN0mcwfZWNCvDya3WRWEb8LyXsN/ VAfcB/dP4RvyOFsL7p6CrL+RsVrnSUcbd0pxDqYw77Etx2nQjVb+Jbm5zfw5ednj8Ai6 q+vwKXxu4u9F90Xu13dU9RWv79ODIHq5M/5kJdA/2/VQD38HyXrcEfbaLBbKFf5x1nen jIiqNKEZpWS2QXZ7lyESbmqicOTy6nMSv0c0Gj4XQIJgYnECg4C9L2oaVWZW3Snomdq1 hbKO8Bfzfh31eFE8W/0p4ZVeD9fu3Ci0oP5e2gol2X0GBYn8pdhQcBvGu49sU0vbzoIs bm4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pYSpfSOCMnJHb7OrZlaDcPmcRr6aYPMvPFICQCYTPDU=; b=dV1Gk4w0s5UGxs/K6uxqWmwHaKPyDdriftEcqBFEUdu/+eaURFfzQZehzxsGxoQYpO Csqnbv7HeuV2bURvNEk8NIW6Rm2M3BNFB0Jb4HJW9Epc0xobBG3uUQi72owUuzvn/692 wlaySGVEegYZ3FUeJy2GT5OwLjh3RTQEvw9WipO4br2d61X4qzgEja7N2PQzTwrhiK7L o2ABSYLh7C244FG7iKCt/5wxfroLCiHevEZZz63HBNJn6hA9F2/0pIDRfK9L++oBOr2q d1zhV8+oz9RTDXP081iXyBp/C6jClg8/9DIn1TOQyUyf6SXQcxijW/v1T1zSOXAlQlW/ IrzQ== X-Gm-Message-State: AElRT7EWYkYShv2f9DX3gb7HWYAlZ2uB0snUrLGKSZTHDSkNq8SirNT7 urcVODKfuPm3g2EanpQPstmwNHE2 X-Google-Smtp-Source: AG47ELvAeUobPbPknejEV53lNQRoD9VtEnKFsbxU405P6Pp8gY5fkhKfY2gFtknv+w5UP9QwCPBdnA== X-Received: by 2002:a17:902:1c5:: with SMTP id b63-v6mr3030689plb.311.1520717013591; Sat, 10 Mar 2018 13:23:33 -0800 (PST) From: Zhang Chen To: qemu-devel@nongnu.org Date: Sun, 11 Mar 2018 05:22:07 +0800 Message-Id: <1520716927-17068-18-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> References: <1520716927-17068-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V5 17/17] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela , Jason Wang , "Dr . David Alan Gilbert" , Markus Armbruster , Zhang Chen , Paolo Bonzini Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 5fb4d8c..de0442e 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -134,6 +134,11 @@ static void primary_vm_do_failover(void) =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); =20 /* * Wake up COLO thread which may blocked in recv() or send(), @@ -553,6 +558,9 @@ static void colo_process_checkpoint(MigrationState *s) =20 qemu_sem_wait(&s->colo_checkpoint_sem); =20 + if (s->state !=3D MIGRATION_STATUS_COLO) { + goto out; + } ret =3D colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; --=20 2.7.4