From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 152800249858633.12748483156315; Sat, 2 Jun 2018 22:08:18 -0700 (PDT) Received: from localhost ([::1]:33636 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLFO-0000ku-GB for importer@patchew.org; Sun, 03 Jun 2018 01:08:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57724) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDP-00088R-PA for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDO-0000et-E1 for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:03 -0400 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:41226) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDO-0000ec-5V for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:02 -0400 Received: by mail-pf0-x242.google.com with SMTP id v63-v6so14351528pfk.8 for ; Sat, 02 Jun 2018 22:06:02 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.05.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:05:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FgjqhcsFRw6DGwtKJ6Bu/n7PVfPRyFTCe0am8svE4pY=; b=SSaIl+EeG6vvQHPQkE0y0x/7kTt861Ppt0keMMwrS/o765vBQ1CXQsWUarG+v41+L0 elkW5P2J7q8NdR9xEhFgtGc6VInRX8arN4ME8r2LP7zmA3N1IPZQJwI2w7izicUu9yap phEAB9TfqLxVzyBqX9UX63qY4FN168c+0oGTPAZC9WVQk+pqwVM2uedMQ/iZ6GCdxO6w SvmBRqyFkOnACTedMCNb5N/zSW0uQWXgD2RgLJKweFbhehtoeYCE70TKPWfKn0zhZdtQ 3SLQYH4SyWkqtjc0V6cD/SffwSjAnchAwyn5GRB2qcNq+NV0N8H/nNxJIX8tw0dMQkzW hVcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FgjqhcsFRw6DGwtKJ6Bu/n7PVfPRyFTCe0am8svE4pY=; b=Cqqtl+d+jd4j26BpMMtrYDKS8iyHlDZq9gkRlp4CT8PMKJEsGZQEDtLkcBCGRGD4m1 pBmK1AuHEdig6LYOsHNC+yl/2j9EkgeZ0BezWC/eL84jjpAZAgPYv2gZ0r7Aw4E+hoZC bp6hukK3T7Kb/Nh5KQ9AX2Z+nX+nwSY94vZkpSNIaz03z5pCDl21S4OkJ2tOgqjGIdfb aBm0em9GHabsiGc13uozHBFWkYXvZQuyNjs/A1oAfZkITKWJCs3D6Rt99wA1fNIQjorC CbZ7Nh/683LCDaONpUv1EG/ijAZXQgb70ciiqyPLtDsZvghH2ntuV52wy4hme0eUKeVb jC2Q== X-Gm-Message-State: APt69E1X0RgLpRuIEA7TGmKdxSQqtoWCx8Z3Kmsh94snAZSbsrx1TUtO jQWN0unaQ9qtBG09hGfTVyp/vA== X-Google-Smtp-Source: ADUXVKJeKALbW0+jaBGCsqQvihoJPAkQQmPMk9KECVaiDqEXhEcnPHawNUwaLen2yhmqHG2PHBeO7Q== X-Received: by 2002:a65:5d09:: with SMTP id e9-v6mr2329806pgr.150.1528002360916; Sat, 02 Jun 2018 22:06:00 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:30 +0800 Message-Id: <20180603050546.6827-2-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V8 01/17] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index da6c36dcf7..cd118510c5 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" =20 #define HASHTABLE_MAX_SIZE 16384 =20 @@ -86,6 +87,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=3D1,ack=3D1' */ } Connection; =20 uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 62dad2d773..0909a9a8af 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -59,9 +59,9 @@ static int is_tcp_packet(Packet *pkt) } =20 /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -99,15 +99,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=3D1' packets from client side, it acks 'fin=3D1, ac= k=3D1' + * packet from server side. From this point, we can ensure that th= ere + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this ra= re + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_ta= ble. + * + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from client side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D htonl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } =20 return 0; } =20 /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -139,8 +168,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=3D1' packets from server side, it acks 'fin=3D1, ac= k=3D1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=3D1,ack=3D1'= packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from server side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D ntohl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } - return 0; } =20 @@ -190,7 +245,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, =20 if (sender =3D=3D nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -203,7 +258,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15280026667091019.4719169088669; Sat, 2 Jun 2018 22:11:06 -0700 (PDT) Received: from localhost ([::1]:33654 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLIH-00034j-Nf for importer@patchew.org; Sun, 03 Jun 2018 01:11:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57747) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDX-0008Aw-0v for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDT-0000gF-Pd for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:11 -0400 Received: from mail-pl0-x234.google.com ([2607:f8b0:400e:c01::234]:35613) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDT-0000fs-Hj for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:07 -0400 Received: by mail-pl0-x234.google.com with SMTP id i5-v6so17657621plt.2 for ; Sat, 02 Jun 2018 22:06:07 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pUvZCAOlnw4NFtVLw+qbjIJRYnINssjK35yBI5gb4x8=; b=E51FiXmhRLpWDip6WSqw7Gsj3rjtY0jGK64YI3kk44SxigbZh44CqA6oU10f+L8Lkz GHde57fSETvysiDjflIAveb3SAMF5LN8z39cVYjUh1mCmXbIK8UB2bEEGCjQuRkwQiqI UhWZrCa+4HrEoDpT8EvHISQi4kbyumg2BAV6ieOv4iiZOwfToXSBm+uj7cpgb0yt1aYe hJrnAZvlKd6joPPYuuwDErHuYWTxv2yRCjrKRPZNpLUcnuoUOBmXcf9YD2EKsJcA05Au pjqwldSk0JwXWm38gdhiio/Rmh7uW4Y00+y1aujHW8TY94A1Ly5hjuXzUDePMsgKvcZ0 hG0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pUvZCAOlnw4NFtVLw+qbjIJRYnINssjK35yBI5gb4x8=; b=spBv2ZmyoSfP/5VwTihLdXF3CetnGWuJOXa63jueIKsuIvXTobOTHpNMZXzHxIeaEh TN5BavNbJjE4bFYhxiZAEjPpGqHCTE7YfS4sgmwP2NxdBNBcvZIr2GuFy0bOmFnyG3Yr C95ETbpSmZSINe+CjoMvdxMBSbDAW4wpsMYMk+djfHIhh1PRwT2KU4HCSz5WKBolzzqu y/8xe8H5IygckAI9tR/KUvHBMDVPYY0srdD657iykMwc6RPyIi25xQagDQeGxWWCeDc3 zOLY8zIiwk5uYD+x8DxCG0T4qhCcoqvKJRHi9pRXu7W1xKEAh4vS6uUH97bIHc7n9Wue Wrvg== X-Gm-Message-State: ALKqPwcbPuHg1Wv+fwmHga9DwQ47GvSUhzrpLP7QOijgOoNrl4ahoFtA GNTEAqDrPWDwTn2D6riEXQ2LFw== X-Google-Smtp-Source: ADUXVKKDv9XGg9nEld5EKM1UOncXeyy1F0S9ane8otfmx2sRbaKBJw+8BW0phN+GXaCjZFH8TmXYBg== X-Received: by 2002:a17:902:7248:: with SMTP id c8-v6mr10309870pll.128.1528002366316; Sat, 02 Jun 2018 22:06:06 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:31 +0800 Message-Id: <20180603050546.6827-3-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::234 Subject: [Qemu-devel] [PATCH V8 02/17] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 76 ++++++++++++++++++++++++++++++++++++++++ net/colo-compare.h | 22 ++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index 2fe48ad353..fefb2fcf4c 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -16,6 +16,12 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" =20 +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); =20 void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 23b2d2c4cc..7ff3ae8904 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -27,11 +27,16 @@ #include "qemu/sockets.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) =20 +static QTAILQ_HEAD(, CompareState) net_compares =3D + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -41,6 +46,10 @@ /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 =20 +static QemuMutex event_mtx; +static QemuCond event_complete_cond; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -87,6 +96,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; =20 typedef struct CompareClass { @@ -736,6 +750,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } =20 +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event =3D event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx =3D iothread_get_aio_context(s->iothread); @@ -756,6 +789,28 @@ static void colo_compare_timer_del(CompareState *s) } } =20 +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s =3D opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -769,6 +824,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); =20 colo_compare_timer_init(s); + s->event_bh =3D qemu_bh_new(colo_compare_handle_event, s); } =20 static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -926,8 +982,13 @@ static void colo_compare_complete(UserCreatable *uc, E= rror **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); =20 + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); =20 + qemu_mutex_init(&event_mtx); + qemu_cond_init(&event_complete_cond); + s->connection_track_table =3D g_hash_table_new_full(connection_key_has= h, connection_key_equal, g_free, @@ -990,6 +1051,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s =3D COLO_COMPARE(obj); + CompareState *tmp =3D NULL; =20 qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -997,6 +1059,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); =20 @@ -1009,6 +1081,10 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { object_unref(OBJECT(s->iothread)); } + + qemu_mutex_destroy(&event_mtx); + qemu_cond_destroy(&event_complete_cond); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev); diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000000..1b1ce76aea --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528002842571122.35394848255191; Sat, 2 Jun 2018 22:14:02 -0700 (PDT) Received: from localhost ([::1]:33666 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLL4-0005Ke-7X for importer@patchew.org; Sun, 03 Jun 2018 01:13:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57760) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDZ-0008Cb-Pe for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDY-0000hh-RV for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:13 -0400 Received: from mail-pf0-x22d.google.com ([2607:f8b0:400e:c00::22d]:37479) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDY-0000hR-Ky for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:12 -0400 Received: by mail-pf0-x22d.google.com with SMTP id e9-v6so14350529pfi.4 for ; Sat, 02 Jun 2018 22:06:12 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rI/OUn5nKZGeFVE0YBkyZT2FOjo2TGgXCiCUgS4lTAI=; b=k4rOWKKQKL/7X5Z/XXV/4FxDrOsRnpaNzEQSkmTl4dh0cbbubA3u633f32IiVKRLHL tDZO/X24lVR4ry+n4gmXD+m+Umz1BZfYXIcHON7R9PBboTY+C4wvs+zMXYxI/TKIHhRs BpNEcohoU3WvgGOOgNM9fD775vQAvHdXDHZxoPJBMpCiIR3DNEHSVS3hmVNmCxnyfkz7 PDJ3QmYRS+crYUk5K4tByafBy3scDoaG9Kv4LwAxmHaGkgb9OSe/Wd3ByTJv/D/cS6Ql EgfeIrdQmmWm7n6K7dORRxbE/AsK+sVKF1XVQ7VhmuaTTpI85xTav76uDTPs3V/alEhy vI0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rI/OUn5nKZGeFVE0YBkyZT2FOjo2TGgXCiCUgS4lTAI=; b=B48fVjGP4fzChodn6w7PTYjP3/ysV3iYKAQW0P73LdbS5rB36Xzc4+MuUqxcd3YlG+ VboLeAbRBe7l/lpsecZhS0rtQ0ci1SZmcx0IxaOnDzZnweBp2poiQVf8HzhhRx0Xy+DN EtaPO6va5NPJwDuQ3he5+CxWu4yZ8p5rHQo8quzq4DSVaXyNRfmYl8OUngoCVv1uMxns rTkhYOvg/GzRizFwZGHYM0cNxMzqCpY1+2Qvro1oe2gYezid+joO4yoPl3y/8hX+462Q Gn7nf9Wr2Z6viJ8A3wzKAeucuvq8BbHi8yoKpV+/GE3RkMF9SYVynHAAALXUs5tn/43y MSbA== X-Gm-Message-State: ALKqPwfFzOlaqoob9dAZ2KPLi/sxeOo1JD2s0lr2mqAG6k6czzQW3xKe hTIqSQJzTbncRGxQHRaZkjk0Og== X-Google-Smtp-Source: ADUXVKKVUb/wnkxCSbGOZD6thLODPhd19UY2KySLtiApL7Fva086miCdEJJlr9YPiU3E227jwz6TwA== X-Received: by 2002:a62:9e0b:: with SMTP id s11-v6mr16609614pfd.198.1528002371507; Sat, 02 Jun 2018 22:06:11 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:32 +0800 Message-Id: <20180603050546.6827-4-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::22d Subject: [Qemu-devel] [PATCH V8 03/17] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 7ff3ae8904..05061cd1c4 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,6 +29,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -37,6 +38,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =3D QTAILQ_HEAD_INITIALIZER(net_compares); =20 +static NotifierList colo_compare_notifiers =3D + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -561,8 +565,24 @@ static int colo_old_packet_check_one(Packet *pkt, int6= 4_t *check_time) } } =20 +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result =3D NULL; int64_t check_time =3D REGULAR_PACKET_CHECK_MS; @@ -573,10 +593,7 @@ static int colo_old_packet_check_one_conn(Connection *= conn, =20 if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } =20 @@ -620,11 +637,12 @@ static void colo_compare_packet(CompareState *s, Conn= ection *conn, /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76aea..22ddd512e2 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H =20 void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); =20 #endif /* QEMU_COLO_COMPARE_H */ --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528002513849975.3416903188876; Sat, 2 Jun 2018 22:08:33 -0700 (PDT) Received: from localhost ([::1]:33637 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLFl-0000z5-1Y for importer@patchew.org; Sun, 03 Jun 2018 01:08:29 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57776) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDh-0008FF-L3 for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDe-0000jB-Bq for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:21 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:36282) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDe-0000iy-3C for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:18 -0400 Received: by mail-pl0-x241.google.com with SMTP id v24-v6so17660645plo.3 for ; Sat, 02 Jun 2018 22:06:18 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZLgNRdgi4AnRM4MIxf5DTdk1oxvsEcUXIDYl9c3Dzdg=; b=VRJoyUF/5aJpDNukw4hWUWyNo/VcMJb72OtgiODjxb0QxNjYAQG9RlP9rDAQlT8Cle bhQbwypUSU/ouVXu0a26Lyk2a2nJLQy2kiZWHtZel+Wbo0i+VpFMCYEsQwuTdid2MpyT K/lvA8TJAybQrQxlMgnt9ft2GZDT1Asy0Hd3PUaJEYCLz0AEn7c9CZZA/fQpr2prnshv E8LPUUhghFpHYySXqX9fH2NTzJ0ktze23cGPBQXw1F+74ucX4Jo2gHOyDaL0tzTproiZ Lcshi9bglA/zCu8tWHlT383ylRJu6KkrIuUj78jw1rGBL1mUiQXc1qCh5Vd9En2DhamW 8IOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZLgNRdgi4AnRM4MIxf5DTdk1oxvsEcUXIDYl9c3Dzdg=; b=RBDJ/Su+csQ9g01wJ+qmXjuIt80bMehlfI+WMnT/ra0kXKh6E4h8UsxN3CsBHugvFL wjL/sE3L6LF6KDcW4hYaXhu/Q6RXGhtOMjTTEuCLERHQwVYJqN14dPSTMNtfMJFnHg3S ZDLFrO+kvSG+jlhTIuNZBsRPZDjFzFNX9GEblHNMX9nvhNQr+zGw6LWAy/j/QAbajGPZ xoD0HdsfHNuDo0mlRT8lonk8tX6JfukLkYyDzhM6hTbqC2uD1XN8CkO4nwTfeftqEBwj vjO6A0FR/9PnXqgWqsdaENIhgIKsSNjG/EAItaneix5k/R2UU8j633nGMHCNt17fLBnI doXA== X-Gm-Message-State: ALKqPwcmquHW15nKPXHuV3HcCzW6aVCft9RASGlW0472ZbLwKh+wVFi7 Z8Dba5OxBZhknu8FbNiGpxyfBQ== X-Google-Smtp-Source: ADUXVKK7RRjur5ecUKhafIFXkloZnGTuiYWEPNTmnt0PvHkq2SVgXS4+Avst/QhSAgTLqhHuhF+dgA== X-Received: by 2002:a17:902:8bc6:: with SMTP id r6-v6mr17237501plo.257.1528002376898; Sat, 02 Jun 2018 22:06:16 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:33 +0800 Message-Id: <20180603050546.6827-5-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V8 04/17] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 6 ++---- 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067ed4..081df1835f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -25,8 +25,11 @@ #include "qemu/error-report.h" #include "migration/failover.h" #include "replication.h" +#include "net/colo-compare.h" +#include "net/colo.h" =20 static bool vmstate_loading; +static Notifier packets_compare_notifier; =20 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) =20 @@ -343,6 +346,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -400,6 +408,11 @@ out: return ret; } =20 +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -416,6 +429,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 + packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -461,11 +477,21 @@ out: qemu_fclose(fb); } =20 - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -558,6 +584,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -577,6 +608,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -630,6 +666,8 @@ void *colo_process_incoming_thread(void *opaque) } =20 vmstate_loading =3D false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); =20 if (failover_get_state() =3D=3D FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 05aec2c905..59aab11f4a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -75,10 +75,8 @@ /* Migration XBZRLE default cache size */ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) =20 -/* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid = mode. - */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +/* The delay time (in ms) between two COLO checkpoints */ +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 =20 --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152800266550286.445987870968; Sat, 2 Jun 2018 22:11:05 -0700 (PDT) Received: from localhost ([::1]:33653 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLIG-00034R-J9 for importer@patchew.org; Sun, 03 Jun 2018 01:11:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57787) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDk-0008HR-TG for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDj-0000kI-KO for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:24 -0400 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:38897) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDj-0000k8-Cw for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:23 -0400 Received: by mail-pf0-x242.google.com with SMTP id b74-v6so10906796pfl.5 for ; Sat, 02 Jun 2018 22:06:23 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=oseOHTqohr5ini5PW6OwA1pG0qipaqJfOV/91DWRcyo=; b=rubOPT2LcY6JsHiTiFa3U2a0USX6bi7qu5bqbOmQH1pJQ+hanHBk3fu+R/Maxp/ydQ IyGV3YAfbLyZX/I2DuckLWAH6JXB8rRPX+6RagWFcPSPRH8SRnHjiqNhp0TUKXTnwx6r SCdTSHw4veO0ckMKjb7c8CSQydZTdN0nwHwYPfyaw5VgB/Wrmc3TToPpmE1hf1Js/kcj g5LjZWnlrmUE3oDk/uPmAlIdqnZbcrevGuvvozjKmSM1uD2a+RDhwOF4aUC0bthAMV96 HyvsWuw9DDfrCekLDHJnHSGTQ6wslu+sSI6xx4De67g+lGKvGcZ0jYwRvmAf02LQNyGX SV+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=oseOHTqohr5ini5PW6OwA1pG0qipaqJfOV/91DWRcyo=; b=GNPMmN8Ft4IEttn924DhyQiRMmDJfzitLPV+oEeYbxapMkZ+8eQtVfkljkWMk/dfqk 3UlsPCvvJsd6KuETYhdwBrVmiHrHvWfF6XjrPQe41AkBr7dP6s7rEtKkZY8S4wxeM2RZ ErqjeWhgnKC1f+lbshU6EjLt9ZRyNtF4Bq7N90WBT+XiSnujylNRd6/5PtqoU2mCqV1F a6NO7sUIRXeQUPYomQFrKfOORdkJBtH1a1d4DK7aIAem0fpsmJqau3VbkJci0IWMZs1p 925stmY4gt/7qMy0R/iBa4CtfUlQDmWiBV1lqGG3hNLbEBd8nXgS8Ipfg/+88ilkOTdD z3oA== X-Gm-Message-State: ALKqPwdJ+vtgDfbIo0EcwjTbm4JoXk1YgfAFqQECZ/4PK5baeavbHr5Y D7FxmhFiS+N7UhwJ4MWh3vkm3g== X-Google-Smtp-Source: ADUXVKJJcO7hpbY27CnIvVPG4qJnXO9w2+4b0aBRFkcvroganVnI9mgRNt+dMkbupgW2LBjUt8Je+w== X-Received: by 2002:a62:91b:: with SMTP id e27-v6mr16609340pfd.93.1528002382269; Sat, 02 Jun 2018 22:06:22 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:34 +0800 Message-Id: <20180603050546.6827-6-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH V8 05/17] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 52 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 081df1835f..e06640c3d6 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -27,6 +27,7 @@ #include "replication.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "block/block.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -56,6 +57,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; =20 /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -73,6 +75,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); =20 + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); /* recover runstate to normal migration finish state */ @@ -110,6 +117,7 @@ static void primary_vm_do_failover(void) { MigrationState *s =3D migrate_get_current(); int old_state; + Error *local_err =3D NULL; =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -133,6 +141,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err =3D NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -356,6 +371,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); =20 @@ -446,6 +466,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -585,6 +611,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -665,6 +696,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index 59aab11f4a..0dfdeecf0f 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -360,6 +360,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis =3D migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err =3D NULL; =20 assert(mis->from_src_file); mis->largest_page_size =3D qemu_ram_pagesize_largest(); @@ -391,6 +392,14 @@ static void process_incoming_migration_co(void *opaque) =20 /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152800252370376.79152958360942; Sat, 2 Jun 2018 22:08:43 -0700 (PDT) Received: from localhost ([::1]:33638 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLFy-0001BM-Md for importer@patchew.org; Sun, 03 Jun 2018 01:08:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57798) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDs-0008Pu-Rs for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDp-0000ms-I4 for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:32 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:34331) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDp-0000mf-8t for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:29 -0400 Received: by mail-pf0-x243.google.com with SMTP id a63-v6so877072pfl.1 for ; Sat, 02 Jun 2018 22:06:29 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cSvQPI0+fy4lIDWaCCXx5ewzAXc/jsSLKXCF/HWHAZE=; b=CE2XvbAwNRsH62WvaeLKHdtvuQm5PodSvroorJjO/N6tF6ZATCLZ/d1FoFZb0VzKnU XBIYWTBtymgJIFbuRGfZXMBUaJL/Tz1WU+ehiI/OvuQTcsN+EW6amP11DfWsVP0d46JT ZCxKCDgIaNvpg4qoR+1yoOj1Zh95zANZe0XdKyNwuAel66qYtrMfQBTmXWjpEbO4n/JO Pd+GQ9TZDMRWEoOipg1v9iVXCFmi8Ol4PmhDrdcNGFgyTsIDpmErPeKoCL31SYlJKrJ5 rJItXmJTqq8ks72MQCfsoyckd6OfomJ410Mhjvxa49/0WD4GgktvQQYzX6gvNB5nhltW UEGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cSvQPI0+fy4lIDWaCCXx5ewzAXc/jsSLKXCF/HWHAZE=; b=hPqK20D8Pt5Hf4BsDU4B2VsqnCM2ReWxDUg1gMINuPujBUv60XQ02by1ipwGVTY4mT ZKpQ9ZAxI1U4PrjqfLYIYFoLp1ofvCASIKVmJSRvjUxP6Bro4E9NBP/IqopSkC3Km6BG JMc3xDZxzDp62qpB9ShQlIiZSDOh7+HHUUvnZPJYUIawoUDQa9QTZQDRa0Lx3axw5HMi bNX11Z7sye/SudPgCtaqkN4a+1vNp6uADlukdavdXreM/bSUg6zlcmy0p78gXPPn8dSK vJ41lndoL6Fb7zAt4igYob/cKOnW4jlOInaENEiwGK0vCxw+0F3Egohx7cRnUghXENdA QUTQ== X-Gm-Message-State: ALKqPwd2pBpO/e10shlp/1Egbj4YNo6h5R79ZiWtJuDWSM41cXcNka9X 6q28/1XuSZP0qi09SnLzlt5F0g== X-Google-Smtp-Source: ADUXVKLtOs1VayWPis43fD1YpGpu0CVo50Oxp9yXvKStiy8wfrf4sPHfVLkKH96TvbvMuXhkjkgF3Q== X-Received: by 2002:a63:43c6:: with SMTP id q189-v6mr13876560pga.123.1528002388018; Sat, 02 Jun 2018 22:06:28 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:35 +0800 Message-Id: <20180603050546.6827-7-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V8 06/17] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- include/migration/colo.h | 5 +-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 ---------------------------------------- migration/colo.c | 13 ++++++- migration/migration.c | 23 +++++++++++- migration/savevm.c | 18 ++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 58 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index fefb2fcf4c..99ce17aca7 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -28,8 +28,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); =20 /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); =20 diff --git a/migration/Makefile.objs b/migration/Makefile.objs index c83ec47ba8..a4f3bafd86 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y +=3D migration.o socket.o fd.o exec.o common-obj-y +=3D tls.o channel.o savevm.o -common-obj-y +=3D colo-comm.o colo.o colo-failover.o +common-obj-y +=3D colo.o colo-failover.o common-obj-y +=3D vmstate.o vmstate-types.o page_cache.o common-obj-y +=3D qemu-file.o global_state.o common-obj-y +=3D qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4dfe7..0000000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s =3D opaque; - - s->colo_requested =3D migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state =3D { - .name =3D "COLOState", - .version_id =3D 1, - .minimum_version_id =3D 1, - .pre_save =3D colo_info_pre_save, - .needed =3D colo_info_need, - .fields =3D (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested =3D false; -} diff --git a/migration/colo.c b/migration/colo.c index e06640c3d6..c083d3696f 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -152,6 +152,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } =20 +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -745,7 +756,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); =20 return NULL; } diff --git a/migration/migration.c b/migration/migration.c index 0dfdeecf0f..48e183a54e 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -280,6 +280,22 @@ int migrate_send_rp_req_pages(MigrationIncomingState *= mis, const char *rbname, return migrate_send_rp_message(mis, msg_type, msglen, bufc); } =20 +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled =3D false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled =3D true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -391,7 +407,7 @@ static void process_incoming_migration_co(void *opaque) } =20 /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2847,6 +2863,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } =20 + if (migrate_colo_enabled()) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); =20 s->setup_time =3D qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index 4251125831..308f753013 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -55,6 +55,8 @@ #include "io/channel-buffer.h" #include "io/channel-file.h" #include "sysemu/replay.h" +#include "migration/colo.h" + =20 #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -82,6 +84,7 @@ enum qemu_vm_cmd { precopy but are dirty. */ MIG_CMD_POSTCOPY_RESUME, /* resume postcopy on dest */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream= */ + MIG_CMD_ENABLE_COLO, /* Enable COLO */ MIG_CMD_RECV_BITMAP, /* Request for recved bitmap on dst */ MIG_CMD_MAX }; @@ -840,6 +843,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } =20 +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1917,6 +1926,12 @@ static int loadvm_handle_recv_bitmap(MigrationIncomi= ngState *mis, return 0; } =20 +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1996,6 +2011,9 @@ static int loadvm_process_command(QEMUFile *f) =20 case MIG_CMD_RECV_BITMAP: return loadvm_handle_recv_bitmap(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } =20 return 0; diff --git a/migration/savevm.h b/migration/savevm.h index a5e65b8ae3..8373c2f6bd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -55,6 +55,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, c= onst char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 3c798ddd11..20accb5b80 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -37,6 +37,7 @@ savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" +savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" savevm_state_setup(void) "" savevm_state_resume_prepare(void) "" diff --git a/vl.c b/vl.c index 70f090c823..e00ca5b0c2 100644 --- a/vl.c +++ b/vl.c @@ -4346,8 +4346,6 @@ int main(int argc, char **argv, char **envp) #endif } =20 - colo_info_init(); - if (net_init_clients(&err) < 0) { error_report_err(err); exit(1); --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528003001386969.1683636109169; Sat, 2 Jun 2018 22:16:41 -0700 (PDT) Received: from localhost ([::1]:33684 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLNg-0007pN-Bx for importer@patchew.org; Sun, 03 Jun 2018 01:16:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57809) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLDw-0008Sm-5k for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDu-0000o2-Td for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:36 -0400 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:42083) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDu-0000nn-Ix for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:34 -0400 Received: by mail-pl0-x243.google.com with SMTP id w17-v6so1813162pll.9 for ; Sat, 02 Jun 2018 22:06:34 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=H77yPB6nw9pmv3d2ipxZMvXxdFKWha2ByddfMCRVsdo=; b=H+TWyJ8hsnaV4oX64yUWtoYijBxrcKSEADpIodZlmd9hay1h21TAYtzfTjnkAB2Hrw /Ea0eIIR7nUk0uV1DFaf7kchJrxqTknghWIHo4wmYyxRJM3rFNtfG9SaYiCTfUnHxLtX nhHXI+lJD6TPYNpzUW2CcBeYu2wYC4LyP7JvZn8a7nh1WbeqpqJzO7gHH8o8dTY5pOqn NaYGNPvRUTCzZVYSqniKiYl2ua2fathW2QVdMeqUaYNVJY/7ZPybLOH8ufolyLjiNkrR GOkGq9s4jF3tTL4nQQ25SKKVOukjsq7u7yfhD/mu0NcWswWyUis+ROZElKyZ08qrRaGK cmkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=H77yPB6nw9pmv3d2ipxZMvXxdFKWha2ByddfMCRVsdo=; b=l3aWrYO1fN53PcQVkVv6sRZK48uzbYXXOUeGLgLI068LzjZ3h64FeaZwXy3yD24dj0 WMawot8FvmgMmLhFFbdC2CKG8W2kDRkgw2Z9s8/cCmRejO0Arrn86t/sVvg173sy59/p tuG7Rap/vwDJPD0hTaP4jzQicOHfQ35wl5frKIWJdVBipYUaFL8WTsZRkdR9j9yVeOOS DBEZOg1LK62oyxhv1RXL+yMgoMzi3CRJ7uVvQcWHY3LM26s4yLQpy4mK6Vhl/G1m174k QHLmZzlif2RZuwZ2FS8arMr1OEA5o1J7QCl0lQC/GuLgvpvl4R3XQuLFSEMPdtLZvYX7 EDyg== X-Gm-Message-State: ALKqPwfF0c8AKQH3KoEENwN2NU5QvrUFwNy+Ve6j901xLpAqPqzwhZc1 qWqjA1jDUgeJsfEwJ5b7KtDlFA== X-Google-Smtp-Source: ADUXVKJ4BUyJ9xppmc9K1em6UQYiPzycYLojnN9ocifhvMOgZWb3t2e9sr1N3fnkV+vP7z9coAdX4g== X-Received: by 2002:a17:902:b110:: with SMTP id q16-v6mr16899284plr.286.1528002393381; Sat, 02 Jun 2018 22:06:33 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:36 +0800 Message-Id: <20180603050546.6827-8-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH V8 07/17] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary= side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram ca= che always the same as PVM's memory at every checkpoint, then we flush this cac= hed ram to SVM after we receive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 6 +++ migration/ram.c | 83 ++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 93 insertions(+), 3 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index cf2446a176..51ec153a57 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 48e183a54e..0d3e2e6d66 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -416,6 +416,10 @@ static void process_incoming_migration_co(void *opaque) error_report_err(local_err); exit(EXIT_FAILURE); } + if (colo_init_ram_cache() < 0) { + error_report("Init ram cache failed"); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); @@ -424,6 +428,8 @@ static void process_incoming_migration_co(void *opaque) =20 /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } =20 if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index c53e8369a3..2bcd70659f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2820,6 +2820,20 @@ static inline void *host_from_ram_block_offset(RAMBl= ock *block, return block->host + offset; } =20 +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -3024,6 +3038,58 @@ static void decompress_data_with_multi_threads(QEMUF= ile *f, qemu_mutex_unlock(&decomp_done_lock); } =20 +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =3D qemu_anon_ram_alloc(block->used_length, + NULL, + false); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %= s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + memcpy(block->colo_cache, block->host, block->used_length); + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -3040,6 +3106,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) =20 xbzrle_load_setup(); ramblock_recv_map_init(); + return 0; } =20 @@ -3053,6 +3120,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap =3D NULL; } + return 0; } =20 @@ -3286,13 +3354,24 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_ca= che. + */ + if (migration_incoming_in_colo_state()) { + host =3D colo_cache_from_block_offset(block, addr); + } else { + host =3D host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } =20 diff --git a/migration/ram.h b/migration/ram.h index d386f4d641..d5e81d4d48 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -70,4 +70,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name); int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); =20 +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index 308f753013..4a789eb4c9 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1929,7 +1929,7 @@ static int loadvm_handle_recv_bitmap(MigrationIncomin= gState *mis, static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } =20 /* --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528003133253410.7188312960932; Sat, 2 Jun 2018 22:18:53 -0700 (PDT) Received: from localhost ([::1]:33692 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLPo-0000tT-Fc for importer@patchew.org; Sun, 03 Jun 2018 01:18:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57822) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLE3-00007o-2y for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLDz-0000oz-Sv for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:43 -0400 Received: from mail-pl0-x241.google.com ([2607:f8b0:400e:c01::241]:33388) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLDz-0000om-Mp for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:39 -0400 Received: by mail-pl0-x241.google.com with SMTP id n10-v6so17652415plp.0 for ; Sat, 02 Jun 2018 22:06:39 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=q3oXufUhbNT/jJgQ13BaKw2rdL3qWs/wrQJitoG3FJI=; b=HvJhh5rh9XUtoO5cW5RDvUpZITHHne1X7urZUVnlCdrpS0NOO9Y6HnfnkZl9xpLNdU sE0Sw95cgtsVaL5IFIO+W7anXFpr7FD9CGnrmEfOHACV84n+MDzC8z9BrvBQ5ujBPxWJ aNG3z+rju23ySdrWlk0nAqpSNKBVP7kPU0zlk8u0pnF/Jt85ifchDLE1kD553yKl5G7u J1WBuh+YPBRN1UXgN1txuNGxxsdXaKFiP1PjAu5CZiOJMfM2o7raS3km3cc3rH1jXQWt TgShymMHp128s9hZCQQ6psEdQwvB5/H/10bKO9dkCbptYWRDYqE7C7bfgQ2KByKbVz8y ty6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=q3oXufUhbNT/jJgQ13BaKw2rdL3qWs/wrQJitoG3FJI=; b=KntVUgm2fgLSqHgWrvdIZalfxiTJNU3O6pAMUBNUadUmCWtQOtZIoCTrToVxHr8ORx 7gN/RV9eLC+m/evkiOcSWjDFBR8z15ttaG0v6edGySTKIh04jMKqHZraTise2Z6z+nIK BCcT6ghlxFwX9P60u0Lp6aCQ3qW0w/4FNo54UxLlhC7M0ZZasRTECXkQqyiqT7JinUpc 1zWedFMd44VfFmrfsWrHOmZCk/6+2v3KaOiZb/GmlTbXG740aqMUnAZEGojVdy0Qfnmx rssbmwFqyBCbEXHrTgeqoRv55hcVw4Mivn7lMPb9gKPzDCBWKgpMonf6cIYCMEvBNPjE bXAg== X-Gm-Message-State: APt69E1p5m+GENk0K3p6O6SYv7yKcJiUXDOw/piMO0SJz23dcgF0VwXn +5+W57iULS1AVU9tx9sX9uwtww== X-Google-Smtp-Source: ADUXVKIJ9+JulWjZZJwUDiB860tm3Nv9OuTTvZoKiXXRQFdWynL+tpzN+PklbsZYE9ndpM1VeIXweA== X-Received: by 2002:a17:902:bf43:: with SMTP id u3-v6mr4936107pls.322.1528002398559; Sat, 02 Jun 2018 22:06:38 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:37 +0800 Message-Id: <20180603050546.6827-9-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::241 Subject: [Qemu-devel] [PATCH V8 08/17] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 2bcd70659f..dd86eeba87 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2831,6 +2831,15 @@ static inline void *colo_cache_from_block_offset(RAM= Block *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } =20 @@ -3061,6 +3070,24 @@ int colo_init_ram_cache(void) memcpy(block->colo_cache, block->host, block->used_length); } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap to= gether + * with to decide which page in cache should be flushed into SVM's RAM.= Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; + + block->bmap =3D bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state =3D g_new0(RAMState, 1); + ram_state->migration_dirty_pages =3D 0; + return 0; =20 out_locked: @@ -3080,6 +3107,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap =3D NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -3088,6 +3119,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state =3D NULL; } =20 /** --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528002702053903.5315691652479; Sat, 2 Jun 2018 22:11:42 -0700 (PDT) Received: from localhost ([::1]:33655 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLIr-0003WK-5G for importer@patchew.org; Sun, 03 Jun 2018 01:11:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57837) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLE6-0000B4-2m for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLE5-0000qO-1u for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:46 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:42083) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLE4-0000qC-RC for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:44 -0400 Received: by mail-pl0-x242.google.com with SMTP id w17-v6so1813277pll.9 for ; Sat, 02 Jun 2018 22:06:44 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OAkgs56gmRWjw6ceiScAyXi376oargTexBBJEahxuwM=; b=J7oUYnvZAJSIZP261HzeomaCb2w/83dDkPWa/ZpzRmf7L+JRjwfU0jYVuEBm6NW5dK BzlKjkXE98+6w9W5GCyr/fvjD95bbTpCGf96vqWki1CX3QvpVcryZQ5xtb6mmvxDL38q hPgSAxaLLCc1/BnpyoDjmsyoaJjJN/TCN5xjrctGlQUh3nsfSdIsI443BukKroAxwIYk hO5tHm4nfcExDys8a017b70Eh7lGrk91J7pdMSN3BlgtcZzRk00IC43xMU5w/yqG/GWo AUP7jiPwBLho08PS+OBqu11PubKxUJwx9JG2VFc2iJqHAxfeSR6hpNj+J7wqYwPagM3P m10w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OAkgs56gmRWjw6ceiScAyXi376oargTexBBJEahxuwM=; b=LkZmJCmKA6/N+9BkSeoeIFNAulInQ4s+D68W44nObdyeY3KVFuvM9JqLZzzHxxVbpB JeVM1uw9lT4nI9ELbq+QrsmItkco23wa/fhJNSflNawDBFIiiWfNy3nbeFaNCr9tQjXq 5JQc3dMqiJsASCLsfApze4e/1to3cjxYI7CDtkbpjGI6N65aUVsA4ahb34ewGKftnzFW eBa0y/rRHHA2Yp5J+teZB/8G9iUVQl5VoxC6wzKkObv4XNaa6j1vj/U9qT348H44V1T9 GxoZOD38Gsf6Nec5R2WyFRnMrr7HS/9NEcvdqhdwmREu1zZCxIRgeySikpsy3Ycl2P0R 24vg== X-Gm-Message-State: ALKqPwdsA9V8iwhFjJPLlUL38Oc1TkNS5Z2SR84J+tz18EjXq/PS7vxl 586+dahb3MsM4FojFOlB4yjIxw== X-Google-Smtp-Source: ADUXVKJE64GmGlMl1VPES10tr0W2DqAHfAcb/bewQXh4OgACwD0Yxi+1IkwGV5pU12QeWXbzpHgSeQ== X-Received: by 2002:a17:902:8f8b:: with SMTP id z11-v6mr16943658plo.203.1528002403741; Sat, 02 Jun 2018 22:06:43 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:38 +0800 Message-Id: <20180603050546.6827-10-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH V8 09/17] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkp= oint time. So, the content of SVM's RAM cache will always be same with PVM's mem= ory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 39 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index dd86eeba87..927436bd12 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3332,6 +3332,39 @@ static bool postcopy_is_running(void) return ps >=3D POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_E= ND; } =20 +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block =3D NULL; + void *dst_host; + void *src_host; + unsigned long offset =3D 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + migration_bitmap_clear_dirty(ram_state, block, offset); + dst_host =3D block->host + (offset << TARGET_PAGE_BITS); + src_host =3D block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags =3D 0, ret =3D 0, invalid_flags =3D 0; @@ -3504,6 +3537,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) ret |=3D wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state()) { + colo_flush_ram_cache(); + } return ret; } =20 diff --git a/migration/trace-events b/migration/trace-events index 20accb5b80..9cc80075ca 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -90,6 +90,8 @@ ram_dirty_bitmap_sync_start(void) "" ram_dirty_bitmap_sync_wait(void) "" ram_dirty_bitmap_sync_complete(void) "" ram_state_resume_prepare(uint64_t v) "%" PRId64 +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152800324744155.126729182524286; Sat, 2 Jun 2018 22:20:47 -0700 (PDT) Received: from localhost ([::1]:33703 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLRZ-00024V-OH for importer@patchew.org; Sun, 03 Jun 2018 01:20:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57871) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLED-0000FK-Dn for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEA-0000rq-6o for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:53 -0400 Received: from mail-pl0-x234.google.com ([2607:f8b0:400e:c01::234]:38336) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEA-0000rZ-0n for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:50 -0400 Received: by mail-pl0-x234.google.com with SMTP id b14-v6so511146pls.5 for ; Sat, 02 Jun 2018 22:06:49 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=blGkwLorxtbaogbHNXwbLYDrkc5No8/I1VNcR1Md91Q=; b=qoSh+nzJTt+2fU0hdkJ0xy2X0UIuLktHkKNNdtri9U28T/vtw/GgNKjvytRJ/aTl1J 2KqGxdAF+PGOd/ZPXFV/dMtGhQAsENhGkvK5V3K4U0x4KKEd2oAkhwOwbeWZnWA/oisd YuFVsN4MUAicZYsaSSevM7y/C5/Hs50F8VI4gxdvrWDuHcnnos7+hRyy2q+FbvcgoAl9 zb0WDQw4HM2cj/8CnBv1oGCVz8p74rizR1VjuvdcM7jPLV3OJbaXIH9d9s8k93I+WFE9 NOsmlXUVZB7S2qasffpH2FNvolIrtCsC8AQr/zHKiwcmW2UcFOwXecYvlAAscTICUK1M KkAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=blGkwLorxtbaogbHNXwbLYDrkc5No8/I1VNcR1Md91Q=; b=fGSNe41fDQi1FWhgOsySF7OavpsJdBHoWAT0oEm9jU8QG+cPHTDeYWYYG4Tov45wtS SByJuHwhYg8fYlfYuA+S78Ar7WuFCsTuN5PABCjv7o1jESBHR1PXiFN7t0+MULJCHYRS +Te+tX4TbA/Nuh3dGe/IYVdEoGG2wOpUlPyoH7IllvBnnlzEk64ZmyE2Hby5M1yAkeFk 5nQeVakY3LGLhOV8j2CcOhRekYVlB+daVhuhNzb1zRmiah66lmWaEBa9siB7wTB41kiA MhUmU85zHSpqAoTaBX/gFCUny3I9KwP/XeTh6rAlNHMZ1pLmxggQtFzRc7ebdZjfucRE 9WZA== X-Gm-Message-State: ALKqPweSJi56/YmquA4br0wXF9jvM4nbh6yYU3Eu33DMstUEMKLzS1iJ OIEpqT0t/M/Bj+7t7zyAQe3K+Q== X-Google-Smtp-Source: ADUXVKK+WnEVff3R1VLrEBjdM+1zYLoHyyc2a2VsB7EYax7ojhJiV6i4HBEwgsu6ty1+REWIAYIGPg== X-Received: by 2002:a17:902:7248:: with SMTP id c8-v6mr10311128pll.128.1528002408914; Sat, 02 Jun 2018 22:06:48 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:39 +0800 Message-Id: <20180603050546.6827-11-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::234 Subject: [Qemu-devel] [PATCH V8 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x-colo-lost-heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake Reviewed-by: Markus Armbruster --- migration/colo.c | 31 +++++++++++++++++++++++++++++++ qapi/migration.json | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c083d3696f..bedb677788 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -28,6 +28,7 @@ #include "net/colo-compare.h" #include "net/colo.h" #include "block/block.h" +#include "qapi/qapi-events-migration.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -514,6 +515,23 @@ out: qemu_fclose(fb); } =20 + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -745,6 +763,19 @@ out: error_report_err(local_err); } =20 + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + break; + case FAILOVER_STATUS_REQUIRE: + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + break; + default: + abort(); + } + if (fb) { qemu_fclose(fb); } diff --git a/qapi/migration.json b/qapi/migration.json index dc9cc85545..93136ce5a0 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -880,6 +880,44 @@ { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } =20 +## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: report COLO mode when COLO exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.13 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "reques= t" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @none: no failover has ever happened, This can't occur in the COLO_EXIT = event, +# only in the result of query-colo-status. +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.13 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'none', 'request', 'error' ] } + ## # @x-colo-lost-heartbeat: # --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528002842608882.1546378899649; Sat, 2 Jun 2018 22:14:02 -0700 (PDT) Received: from localhost ([::1]:33667 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLL4-0005M9-Ns for importer@patchew.org; Sun, 03 Jun 2018 01:13:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57882) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLEG-0000Gq-DE for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEF-0000sq-Ge for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:56 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:41226) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEF-0000sg-9n for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:06:55 -0400 Received: by mail-pf0-x241.google.com with SMTP id v63-v6so14351979pfk.8 for ; Sat, 02 Jun 2018 22:06:55 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=R+RoxCRLCgvH4FQH4IMksXdM88TrTlAOFtvilkY1OM0=; b=Pm9C2GACdGmxc+k7IoH45zHVxhwfaUPvsV3Bg2FsF50Gc636i78q4SL7IqLRU8Kqtl U3+qKdCmgD7ZYzyYRqbBTBm0oidNmhVFZjoe/WZl+SwNMKlGPN9vVhcINzo0qzRQOv+w dvLpQhAHFts/3bD8FlYltlHoeefifZLgxiCoEBPOgr/4VFBy/ZJcpRlLI7YBKVZwSZJ9 rIDjjymp2qsML2lxt56lISvjTx5fxbjjmRzu81V+bbU5RexUYL6ppgHfXn4VAXBjmaT0 etnzRLEE2fHAKh+zgiDxan9IpRiq0IcuJEhu7TfYpc4R1IRBF32RNq69FBGMFSqYjveV bWtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=R+RoxCRLCgvH4FQH4IMksXdM88TrTlAOFtvilkY1OM0=; b=oX0oSQxsE3ddsPx2pcWC9QhiJ7MgjQYfWv8IYvVvtmFt0y+Q02F3kqLnqkKFlGmu/x OK0cL3MpZv1V8YKr2nMvNVG+vMA1tr0arEufQTQAlXIlBIqBYP+bd4w4BJhhvkB5unn9 X52JpqzvYJmXaxCbG+qrQXBq380kiziDL+cBH16j5appDkbk2IB07I0AWhzpSZYkezwb OoOn8Aju/FFGDBThGYkzJakX/XnAe8/GxhekK64/22biU4HzLF6jRKSwVdmil5lzqKqx V5n+cREo407+kWeuTLtlrBrozcUvgmucs4T8ARGfYftOh8W8I5LtJSdNfIBs+AxHY2Nh 2/yg== X-Gm-Message-State: ALKqPwegWbZ3nvR+0QxBAf7HiPOEiVk7OJKVQ3cZL7xQAvCOHp9t0Wob eqet2Ujh+sJuYzEA2bQPeF4YWA== X-Google-Smtp-Source: ADUXVKJXRooR0TEp6vsdIY9EcfEY73NZj4ZUgP7TJahA0c9ZtZBO84JpqiPpPhitP9RFKLOs0h8arQ== X-Received: by 2002:a63:9843:: with SMTP id l3-v6mr13723250pgo.208.1528002414116; Sat, 02 Jun 2018 22:06:54 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:40 +0800 Message-Id: <20180603050546.6827-12-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH V8 11/17] qapi: Add new command to query colo status X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c | 39 +++++++++++++++++++++++++++++++++++++++ qapi/migration.json | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index bedb677788..8c6b8e9a4e 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "net/colo.h" #include "block/block.h" #include "qapi/qapi-events-migration.h" +#include "qapi/qmp/qerror.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -237,6 +238,44 @@ void qmp_xen_colo_do_checkpoint(Error **errp) #endif } =20 +COLOStatus *qmp_query_colo_status(Error **errp) +{ + int state; + COLOStatus *s =3D g_new0(COLOStatus, 1); + + s->mode =3D get_colo_mode(); + + switch (s->mode) { + case COLO_MODE_UNKNOWN: + error_setg(errp, "COLO is disabled"); + state =3D MIGRATION_STATUS_NONE; + break; + case COLO_MODE_PRIMARY: + state =3D migrate_get_current()->state; + break; + case COLO_MODE_SECONDARY: + state =3D migration_incoming_get_current()->state; + break; + default: + abort(); + } + + s->colo_running =3D state =3D=3D MIGRATION_STATUS_COLO; + + switch (failover_get_state()) { + case FAILOVER_STATUS_NONE: + s->reason =3D COLO_EXIT_REASON_NONE; + break; + case FAILOVER_STATUS_REQUIRE: + s->reason =3D COLO_EXIT_REASON_REQUEST; + break; + default: + s->reason =3D COLO_EXIT_REASON_ERROR; + } + + return s; +} + static void colo_send_message(QEMUFile *f, COLOMessage msg, Error **errp) { diff --git a/qapi/migration.json b/qapi/migration.json index 93136ce5a0..356a370949 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1231,6 +1231,40 @@ ## { 'command': 'xen-colo-do-checkpoint' } =20 +## +# @COLOStatus: +# +# The result format for 'query-colo-status'. +# +# @mode: COLO running mode. If COLO is running, this field will return +# 'primary' or 'secodary'. +# +# @colo-running: true if COLO is running. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.13 +## +{ 'struct': 'COLOStatus', + 'data': { 'mode': 'COLOMode', 'colo-running': 'bool', 'reason': 'COLOExi= tReason' } } + +## +# @query-colo-status: +# +# Query COLO status while the vm is running. +# +# Returns: A @COLOStatus object showing the status. +# +# Example: +# +# -> { "execute": "query-colo-status" } +# <- { "return": { "mode": "primary", "colo-running": true, "reason": "req= uest" } } +# +# Since: 2.13 +## +{ 'command': 'query-colo-status', + 'returns': 'COLOStatus' } + ## # @migrate-recover: # --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528003319061499.04736058677474; Sat, 2 Jun 2018 22:21:59 -0700 (PDT) Received: from localhost ([::1]:33707 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLSo-0002mF-5j for importer@patchew.org; Sun, 03 Jun 2018 01:21:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57901) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLEP-0000NB-6s for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEL-0000u6-IG for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:04 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:41228) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEL-0000tq-AG for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:01 -0400 Received: by mail-pf0-x243.google.com with SMTP id v63-v6so14352043pfk.8 for ; Sat, 02 Jun 2018 22:07:01 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:06:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ws5LbF4GyfCxomP0a2gkQb8pgo7r8WbSGAThCAAnVj4=; b=ND5cnFfzRZv7MIbYpPBPl/4YL60TjsZSYpfaAqyrferzuKZNOy66fbz6KyEPJK5SzM NPvlzIhvRktzIX6dMrT2U6ZtoQHbsxbCwNwSayaLaJrdRNZ+kihE8gzohHdLFBHZikED 40tHn18ybddNzu54Idh1rUhJyMtqhl5MmRcAmF5OdU3QVRzmffNdjLaENz7y4NY9B9xb m1uqwwizNc0dIZPWo5p5OsRFZlwnlDJycNPhjE7fu2k/mpy+/+KH82f3ShqPgTsULujQ /uj7QfeW5OxfgqsmAqrbeIKn7iBhqVsFpp5ybCZnPKugpIqYvxP+b/ppAnqo8C0ea+ZT 1OSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ws5LbF4GyfCxomP0a2gkQb8pgo7r8WbSGAThCAAnVj4=; b=XLvWR0Pr7dAoMwO1AWN3LQtcFS0W6RzYIykqp+mILXuf7aJatOwOKUoFqRUAHMDdpV f7p2Xy7YPApbkf76JrQW/ne9vz222G5mfdWbsZsmMhnLFBejB/EJG9MQVw/CLEoH1J4K ncy6OTlSNFpSbZjznTo8PmUIqgaWSSS6rQwH+Qi7UD+v4AYr5i/UFqglpcJMVjbWtjgO yOaRvSvUtvAPQmxyO10tkqm+UY/HWZRqG5DGf1MIGBGOp7WoTzNfZXDCjZo12ewXhgGc iEwPt/PISeRUxwjCxOlbdr2tSqxZ53RHQVsTApa69cvelhHQBJ0S2wWsJMe5JpD4I9u4 xcAg== X-Gm-Message-State: APt69E3VrrYOgF4YH8BfSp67dYJ8xgE/U0m9LZIPAyy8zoGFIBD41VHa 4UnkfpLOWbKoSfj8w79RsJnxSg== X-Google-Smtp-Source: ADUXVKLLNJxnFT9gSXYOdeyXbytcFL3UBA52HOUUJVWPHZBdPPSJf1pSkigBB1FhhnujwojHFDk8Hg== X-Received: by 2002:a65:5b0a:: with SMTP id y10-v6mr5355808pgq.112.1528002419548; Sat, 02 Jun 2018 22:06:59 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:41 +0800 Message-Id: <20180603050546.6827-13-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH V8 12/17] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 40 ++++++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 64 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 8c6b8e9a4e..442471e088 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" +#include "sysemu/cpus.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -419,23 +420,34 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, =20 /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); =20 colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* Note: device state is saved into buffer */ + ret =3D qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + if (ret < 0) { goto out; } + /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + + qemu_fflush(fb); + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. @@ -653,6 +665,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err =3D NULL; + int ret; =20 qemu_sem_init(&mis->colo_incoming_sem, 0); =20 @@ -725,6 +738,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value =3D colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -758,8 +781,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading =3D true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret =3D qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } diff --git a/migration/savevm.c b/migration/savevm.c index 4a789eb4c9..24be7c75e5 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1378,13 +1378,20 @@ done: return ret; } =20 -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} =20 - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; =20 + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1440,8 +1447,6 @@ enum LoadVMExitCodes { LOADVM_QUIT =3D 1, }; =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis= ); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2241,7 +2246,7 @@ static bool postcopy_pause_incoming(MigrationIncoming= State *mis) return true; } =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret =3D 0; @@ -2414,6 +2419,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } =20 +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 8373c2f6bd..51a4b9caa8 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -56,8 +56,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, = const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); =20 #endif --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528002747715326.8387383450929; Sat, 2 Jun 2018 22:12:27 -0700 (PDT) Received: from localhost ([::1]:33656 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLJa-00049O-Mt for importer@patchew.org; Sun, 03 Jun 2018 01:12:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57912) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLER-0000Or-26 for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEQ-0000xr-2J for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:07 -0400 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:33230) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEP-0000xG-Qb for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:06 -0400 Received: by mail-pg0-x242.google.com with SMTP id e11-v6so3796530pgq.0 for ; Sat, 02 Jun 2018 22:07:05 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.06.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FIEdOS4Qpk/MgP0vNCirmcAoGj2mB4cFvrpVJXmUrvY=; b=kGERQzHX+8DtAXscdefDY9H9ZdbyS166WXpkEZ9I9m+r6nH9kjxuy2Ry1S/pvYVJff LXjeXqjL+WYoAWk6Ydc/zdJuA52IGYPfDKNUNmT5YrosmpTmK9t2lt3BjcGTC0m9TbFu oJQtVoJcBz+acVjSiOT1hdTuHrUdUq5UMyS39oemhqM0hNMaQR0NkHYcO29bblIVkIIo 6vLdnGa8F+l/j9ZStxQdkdspOCnz61IFWd6u8zkMOyx/PjVzW/m80qMwMCTECZAMjjYu NE1Sr1oirdUPD6teekSqawQ06JC3IMxAkWh4sBUvseQlfF1M0NWFgYvrmeg8jdueXiLG DTMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FIEdOS4Qpk/MgP0vNCirmcAoGj2mB4cFvrpVJXmUrvY=; b=WcTnxuvFhGimLGL2OVlZTW2Eg3QuDTti3gUyWlnBnGiXmqdE6AlGK/YjBLDtMDHbxO UeESwAJ05s4H68c+Y5+aArh9Y7Zaoe39H7lTApIiSGtlLZIipix85aSbx3C1FOuMAsag P3/4FIdUY+I30EOx7pedF4KzIOyYOOsX7ggb8rgVdC9Mp2soNhSSGKWtAUzLPfsP3Jep 1m7PEPZq4XKTHPjp3JOGQoXQiUmd8k96OehVDCAMSiq69bnj8w2eoq1ySGNvf5hIrfHv AnjXNxCL/hFRwnRHQjArUOZVDXpUy1pGWKivDmSfxFLOFFR4XxfN5WfU7RM+3sFn2dNw IW+w== X-Gm-Message-State: ALKqPwfCgrhsAZweSkcC/XwPs6zPKNmFcYGTCq6EHBWrVTRQZ2qTh9cP NTZsIIDgci4sdBjhCGnaT+PunA== X-Google-Smtp-Source: ADUXVKJy+mK/8DTnIC6yNisfrdwxyOR2RUnvWO03p355lBIcZtnaNppD5XAF3oxpMxA1PpocFgt5xg== X-Received: by 2002:a63:7b07:: with SMTP id w7-v6mr12958521pgc.199.1528002424705; Sat, 02 Jun 2018 22:07:04 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:42 +0800 Message-Id: <20180603050546.6827-14-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH V8 13/17] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 927436bd12..e34a015785 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3087,6 +3087,7 @@ int colo_init_ram_cache(void) } ram_state =3D g_new0(RAMState, 1); ram_state->migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); =20 return 0; =20 @@ -3107,10 +3108,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap =3D NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -3343,6 +3346,13 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 + memory_global_dirty_log_sync(); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_lengt= h); + } + rcu_read_unlock(); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block =3D QLIST_FIRST_RCU(&ram_list.blocks); --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152800292453392.73521427642572; Sat, 2 Jun 2018 22:15:24 -0700 (PDT) Received: from localhost ([::1]:33671 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLMM-0006be-Rx for importer@patchew.org; Sun, 03 Jun 2018 01:15:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57933) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLEY-0000Vy-GJ for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEV-00011s-9J for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:14 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:42369) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEV-00011L-38 for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:11 -0400 Received: by mail-pf0-x244.google.com with SMTP id p14-v6so14353087pfh.9 for ; Sat, 02 Jun 2018 22:07:11 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.07.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:07:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Jsc2fbiXWxMRQzqXoNeLvaFoiXBFonpX+YVx0x44eTo=; b=ju947f6wHMLFrkJ0BM+TMCoJF80siVCT+kLSs8nyRPpC4fM3HOo3oqmq8RcH3erL6l 1ELMZ9S+tTNfyOCga1EW2oeslaicgjD+4ElPwJ69HZ4ahShD0bNCzymgAYufyoPxKmWF kSj42Nfrmc0oIDo9OthhraE704oZ+xrMHAS/yHPePh9jjf/TGx/9Loejj0oRqSPYwYCx FwJ9GTbmY6i3TVJVQjHtJvZFChynVjjv4feh1tnZfIbjjn8MHG38FkHxp6KOTXUiDB1l dbQWCNUzDg3fqoGU+es2N2q9E2BGxsLGwarUJCTkJZJDyfyrfN0QckTltQL5VyCJI8m+ VE/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Jsc2fbiXWxMRQzqXoNeLvaFoiXBFonpX+YVx0x44eTo=; b=mdcuTENUQ5thC9bqFGMQdUGmzTC+5KxAsAgzjuQ4zhSyY4/Sdh8a4qp2KDq4dZXNz6 BnyOnV4NRDu494wDndzST1ZEe/xKl6fADfu7la3GzIqHZqcnSOu0IEtCYJerzLAb9/In dKcr28ZNpI2uOIsxEleUnG0EwsnViFUAnlBjVewKcEniG5iyK+TTA90iLMkvIDHtYUga +ilVoxUZydFqHFez9gVy2PC89dNJmCPDHX1Hz9kjLLpRXfgtwVgRiOvrNAv5Sthwifg2 b6eiBw9jjLM5oAYMLrVsnJbkOA+EHnrXt4Wv71QezzLVgR3HFfP3rRednqzoTZ86iI8X v5dw== X-Gm-Message-State: ALKqPwf7my2MEE+suu4d/96nDsK+NXB4kwqn07KrUx7XKudhnJRcYNvH n9Mu5uWGvm6+QoUaAbzpwjOqDA== X-Google-Smtp-Source: ADUXVKJ14BbHYxqf1lUV+rrgkIYedufwLW61jenlJ4nsWYCRkJ0tzWrbMJ+hsV17mw+KfCvTr4QiXg== X-Received: by 2002:a63:43c6:: with SMTP id q189-v6mr13877511pga.123.1528002429915; Sat, 02 Jun 2018 22:07:09 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:43 +0800 Message-Id: <20180603050546.6827-15-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH V8 14/17] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 435acd6f82..49da666ac0 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -38,6 +38,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, =20 typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); =20 +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **e= rrp); + typedef struct NetFilterClass { ObjectClass parent_class; =20 @@ -45,6 +47,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -77,4 +80,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sende= r, int iovcnt, void *opaque); =20 +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d663..0f17eba143 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" =20 static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } =20 +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc =3D NETFILTER_CLASS(oc); =20 ucc->complete =3D netfilter_complete; + nfc->handle_event =3D dummy_handle_event; } =20 static const TypeInfo netfilter_info =3D { diff --git a/net/net.c b/net/net.c index efb9eaf779..378cd2f9ec 100644 --- a/net/net.c +++ b/net/net.c @@ -1329,6 +1329,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdi= ct) } } =20 +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc =3D NULL; + Error *local_err =3D NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer =3D nc->peer; + type =3D nc->info->type; + if (!peer || type !=3D NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =3D NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528003393187431.69156561386205; Sat, 2 Jun 2018 22:23:13 -0700 (PDT) Received: from localhost ([::1]:33710 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLU0-0003Mn-E1 for importer@patchew.org; Sun, 03 Jun 2018 01:23:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57958) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLEb-0000Yq-I7 for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEa-00016I-GN for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:17 -0400 Received: from mail-pl0-x235.google.com ([2607:f8b0:400e:c01::235]:46814) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEa-00015Y-AX for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:16 -0400 Received: by mail-pl0-x235.google.com with SMTP id 30-v6so17638887pld.13 for ; Sat, 02 Jun 2018 22:07:16 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.07.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:07:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=JRTYEkNp4mEWB73celv7UEJSi6SIr5keQeQTLS1CCg0=; b=DKo9ZG9bSST0CTP8N5PGKKZAeRjg9fwZyESavdq3gZOrffQlYtRQkAhhmyUuFXPbgW NG/WRDlE67NH4HTaTKmjh9oF8F+HGTVEIJ1FZbvRBWTEASzljbOyumJK1h6fZc23HcSk 21tnjJ/AJu7x+gcvkcFSyBfF8/ZRgYgSskuULRcaV5l7ZL/FHGbiHxngrAEwBSsEwoQi BdrXzFrC9FjYqA6dYxwrkgofo4IGGnEWTzPxaAghjAmlEt9LEVH4mt6flVI59o/csWjz QOiq5sOKZMcTsXLEW/Zd0ch1spdjlXIjNB/hAnvjlRiTYn4YeSQHZed8G4jwp5rbjYC0 y0Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=JRTYEkNp4mEWB73celv7UEJSi6SIr5keQeQTLS1CCg0=; b=KB7OTw0ofBDX15KVh94nxKu30K4qvVjcj0d9/1yDMGAFvG9MhisxlNWqW9aYrxHP4M KvQidwJgBBVq0U2aE0S5wLmsKmBiqnpIfSa/HMy9Qq3oG3/cIxYWkxxRzMJutb+EdJU7 QudKg7D6MsUohIbed0mp1+ExzUGV0Cg2p/BELQS72RX3Nws5YEC7fKduZtgsT2h7l2Sq suzkLKqNc+ujjtqTIvFlJXyzPgWA9LSwMNnTOVa2v1203DYnvey3vhKSHUKew5xpm79G mS/3G5ZJA+ycHtMV0z3LIzzztl+CCukIbitcuexmQQwdZbBl3s3iVJiasvBI5ixTsqz3 Wwsw== X-Gm-Message-State: ALKqPwetT3ZLzboXuO5PA0PHW+hPhcwSHPRDOQv1ZhdzgVLqjkrLe5Zu hB3NT1xQMjI10s83hF39ktbsmg== X-Google-Smtp-Source: ADUXVKIdJD3qNd6k7t5VHhZQKctp0EpR/zGAIJrpstnEDDQhSw2gO5chORhNUFjaWPjzELfQKBAS1Q== X-Received: by 2002:a17:902:28ab:: with SMTP id f40-v6mr17057408plb.208.1528002435098; Sat, 02 Jun 2018 22:07:15 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:44 +0800 Message-Id: <20180603050546.6827-16-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::235 Subject: [Qemu-devel] [PATCH V8 15/17] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 442471e088..0bff21d9e5 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -31,6 +31,7 @@ #include "qapi/qapi-events-migration.h" #include "qapi/qmp/qerror.h" #include "sysemu/cpus.h" +#include "net/filter.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -82,6 +83,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -800,6 +806,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 0909a9a8af..f3c306cc89 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -20,6 +20,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" =20 #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -277,6 +279,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterStat= e *nf, return 0; } =20 +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_d= ata) +{ + Connection *conn =3D (Connection *)value; + + conn->offset =3D 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn =3D (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs =3D FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s =3D FILTER_COLO_REWRITER(nf); @@ -332,6 +371,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, v= oid *data) nfc->setup =3D colo_rewriter_setup; nfc->cleanup =3D colo_rewriter_cleanup; nfc->receive_iov =3D colo_rewriter_receive_iov; + nfc->handle_event =3D colo_rewriter_handle_event; } =20 static const TypeInfo colo_rewriter_info =3D { --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528003176090896.7556735066723; Sat, 2 Jun 2018 22:19:36 -0700 (PDT) Received: from localhost ([::1]:33693 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLQV-0001Ge-Bo for importer@patchew.org; Sun, 03 Jun 2018 01:19:35 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58018) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLEj-0000gF-5J for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEf-0001BM-UA for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:25 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:36919) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEf-0001Aq-OD for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:21 -0400 Received: by mail-pg0-x244.google.com with SMTP id a13-v6so12987705pgu.4 for ; Sat, 02 Jun 2018 22:07:21 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.07.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:07:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3gki9G147bx6x33sgLsCMxxLRFBF8jp0rqP+wTdMSuo=; b=pUDn9NOCXbeYkwnsLgV5Nshk8PxgzYsaLYQM6yQk8YUmxJrRbSjTGaBVPc+P8yE6mP in0U9DZ9LgIv6Mx57dohMctN3wYEfu2OCoVVdh4Yu5XKvSrSulIDEGDDrZWU7swvfbmU BboHHhwa8HRGO3tF112TOYb23CyVOroo7mafwOSt6UM+jVBX8gVOqor69C5PEukOjeE8 GUMSxd3xetZ+2HPQoLkxfLuO/p57d2SUea3Uw65TFtGXLRvl4kTcDRzNmoNL59vICn0h XnLQ2FqKOj0bB3pNPVo6w6m7aCVPDdST0vPG7xOtBsNZvMrxvr8rG+zkIWTzrl85N/lX WOLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3gki9G147bx6x33sgLsCMxxLRFBF8jp0rqP+wTdMSuo=; b=ZPmHT9pGJJTw9mIjSW5QykFwa/r7Zf5P58/MSszKvIRJSf+nkF+vF3piw1GKr9Gzbf NgnXQxs3XKbZfuVg7ROkX66oWJ5tr+WA3aLQB1i8mFkIoxW3Ek/p4++0kjKxIarp1SpU HfOjCkil4obw/zzCep2Q41UkC6Khv7aL7cDWQZOHc0pqaQe2l5WN70kyDVLIpbJA5DpO qg//Bnf2KZhi4OWK13jHv29gQMo4ULU3a/OP5Z+qUU34U88R06XP3w5oUBoKVia8Ru0I CPZHXyKPN+g5CgzOAAFTXREglVgDWJEPZmZNKsMp1yrRg7vX3mknz7PwxyZP6CL2oPFn Vh6g== X-Gm-Message-State: ALKqPwfkblttD82/GRh/agPBTklhrucvOtm54pUg8vAewPJ0knUHesfE X8TbSQbDGq5QP2dHmcA31g3LTQ== X-Google-Smtp-Source: ADUXVKKF34bcgdceJm0S6nQDEbT+wDquWHU7DiL0TuSp6yhmASWvJ6theWMo7Ep3qx81ovQGOWK+EA== X-Received: by 2002:a63:9843:: with SMTP id l3-v6mr13723870pgo.208.1528002440637; Sat, 02 Jun 2018 22:07:20 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:45 +0800 Message-Id: <20180603050546.6827-17-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH V8 16/17] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 0bff21d9e5..e3824d139a 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -88,6 +88,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -813,6 +818,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); --=20 2.17.GIT From nobody Sun Nov 2 11:43:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528002876050611.696544154415; Sat, 2 Jun 2018 22:14:36 -0700 (PDT) Received: from localhost ([::1]:33668 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLLb-0005nu-2k for importer@patchew.org; Sun, 03 Jun 2018 01:14:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58044) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fPLEm-0000kp-2B for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fPLEl-0001F6-6n for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:28 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:38413) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fPLEl-0001Ea-0y for qemu-devel@nongnu.org; Sun, 03 Jun 2018 01:07:27 -0400 Received: by mail-pl0-x244.google.com with SMTP id b14-v6so511546pls.5 for ; Sat, 02 Jun 2018 22:07:26 -0700 (PDT) Received: from localhost.localdomain (197.225.229.35.bc.googleusercontent.com. [35.229.225.197]) by smtp.gmail.com with ESMTPSA id 10-v6sm45841595pgb.40.2018.06.02.22.07.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 02 Jun 2018 22:07:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LLhwPayMmp0eMKNTbN/o7NgGKlb/J1/zJjpk0khgm0s=; b=GhOqhOuHsSKMQw2DfdcMyVSMwrPnbeMxWX44LzKoc6+K99IJgFB6KZFal6VGO/o/60 +hs+9uwHcpnILurK8iQ0k5C43hGrC60zy1CKJyOT3l46PwXrfBV4o/X7YKdlitmp5jL2 7hWdZABGJL7hjfO6HeVkzEBZn0e/dNp8ZvMjHjjhPafTwZUW+e6O6qydkghVOcfKCjMu ND4UvXWymrUw3Qe/SuSDz6m6GbMzvvuFpjBoLxxeXki1PUOM1KCWrcbiX+eAhToE5lwe OAtaAotw1trT8oEZzBfog9mEV1f0P0TO1OVybC6QzHFuAM7ug8jVXMPOfEFLBw7Gg9BJ F5KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LLhwPayMmp0eMKNTbN/o7NgGKlb/J1/zJjpk0khgm0s=; b=SeX4+L2MTnnh9SYAJefwwQl0mv4vRSR/sQ+7hZS6WTytRCU1g6Cth8WZu9QHsEXfoD NArhAW9qPNgTXc9Fpy/4RTsKM0+6V5p6+8U5e2ns8dKOLoP572PVYu/9P+PhYulm6xH4 zwe8wa99xknv4rE2dsjrLM+TgqWm64jX8LuwoaiY+DfEykWS++p0aIL37GxBA3+25ra9 EBS8jLq37u/JVbeTvp5GTmNk+jDJSXjXTBohYXSWjutnWu6kaZ4/0mdgL+oo6YhbY8g3 ITpna+oBQOEz8KgQ/9flGvbZMvIOU/cJE9yicthUwIUFyhke20aUFMfduK6n0QXt79gE NS6Q== X-Gm-Message-State: ALKqPwd4JXMhwbz+03Z3gMfGsnHlM/wuG0dstFG7nl3Q/u8gt3JlTD8L jHTkUlA+x6twHRjVorS9B4yWtA== X-Google-Smtp-Source: ADUXVKJ/fsXPFT0RxKSP0xUS7j6/DnUJKE4Qt1AcgU8tbhyAjQPZZpt9WhmlqYJslJ2Y+zwn8vLioA== X-Received: by 2002:a17:902:b110:: with SMTP id q16-v6mr16900803plr.286.1528002445835; Sat, 02 Jun 2018 22:07:25 -0700 (PDT) From: Zhang Chen To: qemu-devel@nongnu.org, Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Jason Wang , Eric Blake , Markus Armbruster Date: Sun, 3 Jun 2018 13:05:46 +0800 Message-Id: <20180603050546.6827-18-zhangckid@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180603050546.6827-1-zhangckid@gmail.com> References: <20180603050546.6827-1-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH V8 17/17] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index e3824d139a..1d01f5ba08 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -135,6 +135,11 @@ static void primary_vm_do_failover(void) =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); =20 /* * Wake up COLO thread which may blocked in recv() or send(), @@ -561,6 +566,9 @@ static void colo_process_checkpoint(MigrationState *s) =20 qemu_sem_wait(&s->colo_checkpoint_sem); =20 + if (s->state !=3D MIGRATION_STATUS_COLO) { + goto out; + } ret =3D colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; --=20 2.17.GIT