From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515045840695977.9416430291152; Wed, 3 Jan 2018 22:04:00 -0800 (PST) Received: from localhost ([::1]:47453 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyd5-0004hD-RC for importer@patchew.org; Thu, 04 Jan 2018 01:03:51 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36693) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyb5-0003SG-Ti for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWyb4-0003uP-M3 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:47 -0500 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:46939) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWyb4-0003tm-EP for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:46 -0500 Received: by mail-pf0-x243.google.com with SMTP id c204so365265pfc.13 for ; Wed, 03 Jan 2018 22:01:46 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wkt/Ox6pwMzQEiOeOtqTem016VC/LjxH2opuj5feFeQ=; b=HICCxhLB5+Wy9vXuJpza9Pg6tzFDIz8VMsnhqu03P5d+m5LmS+GpqB4487854ohi2T IjD2cyyODyrQz3CsvExI+EAw9Mjw0C5H2f3roaklLlZbAbuOdSCYBLbf2zWYLSn3uxUS cGpCbVkj2PC7qywMQ33lHySRcvrHwVCciqqlbNeiajDKE4923Io1LXAeC/wyNS7ZqL0E f92POeG7OOeZC7LzoHLyjRka4kphAYpUaER+cC5XBXpxJnULro3Eebr6netoae6zzDU9 87cw5LJsrjUj3EbZ54RbMozUATuvePdzBC60DV2cflGNZ0BXcB5f5jX661moJyRd2T10 ZRbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Wkt/Ox6pwMzQEiOeOtqTem016VC/LjxH2opuj5feFeQ=; b=CzhhP7mN4twQvvTeVsLsCMlTqYfk/rvpD5aCbmjUNC7nRzOvv/lbtqVyvTfv2Gl3jA iEa6YUEBgRFRYPmSiOTQ0HuOxIF5kVeo24B2j83IfKNO5++mJu67//DesX62kQhKoXlI wM3YbDYWn6TpxrMZmDbFLQa2ALLynw4gZSUmF5HNzvf4o3o59tmljrMWN2qHENe445C5 NcToogdeIS5UUBRHI+OAHwMRBC8aOZBXq2mSbr4MoGYC8UEuf3yoXlNiEHomKU29tGXW YcL2Afg5pyBm/ejzWdhLX7hL+5cAcAO2/4a0oEVbgsNOsin09Fvprf77Q8fWusEszBsg Y8oA== X-Gm-Message-State: AKGB3mLpDqeKNUtFjTihXa3hrOhg7EiUpgYUDSiAAagy3SebDVkvnRyB 7CCunuXa8hOrEIjFa4EH4+Vq5e4q X-Google-Smtp-Source: ACJfBovikjqUbalNSX2Bv0JbF2SsVRqNjfVvx2GLBXAASQ8cdWlCp+s9Kzl97mIJOWQ2MWXfaTOrlg== X-Received: by 10.98.41.197 with SMTP id p188mr3689025pfp.9.1515045705015; Wed, 03 Jan 2018 22:01:45 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:00 +0800 Message-Id: <1515045675-6993-2-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH RESEND V3 01/16] filter-rewriter: fix memory leak for connection in connection_track_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- net/colo.h | 4 +++ net/filter-rewriter.c | 69 +++++++++++++++++++++++++++++++++++++++++++++--= ---- 2 files changed, 66 insertions(+), 7 deletions(-) diff --git a/net/colo.h b/net/colo.h index 0658e86..0193935 100644 --- a/net/colo.h +++ b/net/colo.h @@ -18,6 +18,7 @@ #include "slirp/slirp.h" #include "qemu/jhash.h" #include "qemu/timer.h" +#include "slirp/tcp.h" =20 #define HASHTABLE_MAX_SIZE 16384 =20 @@ -71,6 +72,9 @@ typedef struct Connection { * run once in independent tcp connection */ int syn_flag; + + int tcp_state; /* TCP FSM state */ + tcp_seq fin_ack_seq; /* the seq of 'fin=3D1,ack=3D1' */ } Connection; =20 uint32_t connection_key_hash(const void *opaque); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index 2be388f..a58310a 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -62,9 +62,9 @@ static int is_tcp_packet(Packet *pkt) } =20 /* handle tcp packet from primary guest */ -static int handle_primary_tcp_pkt(NetFilterState *nf, +static int handle_primary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -102,15 +102,44 @@ static int handle_primary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'ack=3D1' packets from client side, it acks 'fin=3D1, ac= k=3D1' + * packet from server side. From this point, we can ensure that th= ere + * will be no packets in the connection, except that, some errors + * happen between the path of 'filter object' and vNIC, if this ra= re + * case really happen, we can still create a new connection, + * So it is safe to remove the connection from connection_track_ta= ble. + * + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from client side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D htonl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } =20 return 0; } =20 /* handle tcp packet from secondary guest */ -static int handle_secondary_tcp_pkt(NetFilterState *nf, +static int handle_secondary_tcp_pkt(RewriterState *rf, Connection *conn, - Packet *pkt) + Packet *pkt, ConnectionKey *key) { struct tcphdr *tcp_pkt; =20 @@ -142,8 +171,34 @@ static int handle_secondary_tcp_pkt(NetFilterState *nf, net_checksum_calculate((uint8_t *)pkt->data + pkt->vnet_hdr_le= n, pkt->size - pkt->vnet_hdr_len); } + /* + * Case 2: + * The *server* side of this connect is VM, *server* tries to close + * the connection. + * + * We got 'ack=3D1' packets from server side, it acks 'fin=3D1, ac= k=3D1' + * packet from client side. Like Case 1, there should be no packets + * in the connection from now know, But the difference here is + * if the packet is lost, We will get the resent 'fin=3D1,ack=3D1'= packet. + * TODO: Fix above case. + */ + if ((conn->tcp_state =3D=3D TCPS_LAST_ACK) && + (ntohl(tcp_pkt->th_ack) =3D=3D (conn->fin_ack_seq + 1))) { + g_hash_table_remove(rf->connection_track_table, key); + } + } + /* + * Case 1: + * The *server* side of this connect is VM, *client* tries to close + * the connection. + * + * We got 'fin=3D1, ack=3D1' packet from server side, we need to + * record the seq of 'fin=3D1, ack=3D1' packet. + */ + if ((tcp_pkt->th_flags & (TH_ACK | TH_FIN)) =3D=3D (TH_ACK | TH_FIN)) { + conn->fin_ack_seq =3D ntohl(tcp_pkt->th_seq); + conn->tcp_state =3D TCPS_LAST_ACK; } - return 0; } =20 @@ -193,7 +248,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, =20 if (sender =3D=3D nf->netdev) { /* NET_FILTER_DIRECTION_TX */ - if (!handle_primary_tcp_pkt(nf, conn, pkt)) { + if (!handle_primary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); @@ -206,7 +261,7 @@ static ssize_t colo_rewriter_receive_iov(NetFilterState= *nf, } } else { /* NET_FILTER_DIRECTION_RX */ - if (!handle_secondary_tcp_pkt(nf, conn, pkt)) { + if (!handle_secondary_tcp_pkt(s, conn, pkt, &key)) { qemu_net_queue_send(s->incoming_queue, sender, 0, (const uint8_t *)pkt->data, pkt->size, NULL); packet_destroy(pkt, NULL); --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515045993912505.1169274714423; Wed, 3 Jan 2018 22:06:33 -0800 (PST) Received: from localhost ([::1]:47471 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyfd-0007Gh-JX for importer@patchew.org; Thu, 04 Jan 2018 01:06:29 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36715) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyb9-0003U2-6v for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWyb7-0003vt-TU for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:51 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:41969) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWyb7-0003vV-LV for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:49 -0500 Received: by mail-pg0-x242.google.com with SMTP id 77so337307pgd.8 for ; Wed, 03 Jan 2018 22:01:49 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sKzd1WODysTlkeNRThruj5hxDE/MMDkcc/U2y8ce+fk=; b=hu+OpC/1Rh89/tvNAtJ8eZuzlOUKMXbqlkhJCAAH0wOJzQcsiOSfdiWBvg4vrcbJ1a D/CKP0UXnSQzPhWDoWrDTBgOWPiojN+30S1fJNsW4FlvTAHMp2EGKdhU5OAW8VdkBlCy mah1ijJ9f6hbOKea8Rbwae9lJ6HyTp9zdVcptXW/OGm4vxwDZQABDq4LSMiwmk94ypUa 49QlEBDMWg4hr+pq2f2+JcFhYa8OVFkvkpBblwhXHG2tvmsBj3xzNRJdvJJas5CxGApn wQofJ8AZTo3khGZJiajxSRZXg9dpJ5CVSH5RD/5MDQeO5++MgB7LZtZW1+q/PPOeP2cv ADzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sKzd1WODysTlkeNRThruj5hxDE/MMDkcc/U2y8ce+fk=; b=qNe3sSktrtK3OZixwK2hhnZZNjBT3J3nNnfgdNCMAzqBRU537h2R1CcSsltsEeTO1c KFxNlwyGNgu7j4k9kA6tzqsjkkjlv1AdDrBTA/yj323t+BN8Cc2S9PMyIIjpf/f95adI 4uBY5deDHoa3p4/GBetJeD/bAbBcaOuU94Y7fLh5nIVhRHKEvwqU3mVPGYXST2bCoSyW qfJaQwOLOpCAwcmh9wq9CCC3xMFhohxWhgIt2DaX7k6UAGdvCQ9a3h4mmaKWRSSNvuQh 1jKAnLOQ4PxX9Wh76PNWr01BdKP6bRv/usHT1ZUI/77CJUM74JDwZe4NvtuIfc2qjv3w XLsg== X-Gm-Message-State: AKGB3mKrzTwqG+T9wxcAxKeoPyIrNu/+1qOwwQmoNTaATdDhoBF/WJoM 2/FQevDZD3rW/eZpyeoXeF0UOOX4 X-Google-Smtp-Source: ACJfBotiyaE9XPsscHU+4sPdhd/BgUce41yLB3g2ZSNshebtx8J7MGG+oUJtGtiZ5oAkzjpDO20Trw== X-Received: by 10.98.34.85 with SMTP id i82mr3706821pfi.57.1515045708257; Wed, 03 Jan 2018 22:01:48 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:01 +0800 Message-Id: <1515045675-6993-3-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH RESEND V3 02/16] colo-compare: implement the process of checkpoint X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 6 ++++ net/colo-compare.c | 71 ++++++++++++++++++++++++++++++++++++++++++++= ++++ net/colo-compare.h | 22 +++++++++++++++ 3 files changed, 99 insertions(+) create mode 100644 net/colo-compare.h diff --git a/include/migration/colo.h b/include/migration/colo.h index ff9874e..6adf3a5 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -15,6 +15,12 @@ =20 #include "qemu-common.h" =20 +enum colo_event { + COLO_EVENT_NONE, + COLO_EVENT_CHECKPOINT, + COLO_EVENT_FAILOVER, +}; + void colo_info_init(void); =20 void migrate_start_colo_process(MigrationState *s); diff --git a/net/colo-compare.c b/net/colo-compare.c index 0ebdec9..e9cfca2 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -29,17 +29,28 @@ #include "qapi-visit.h" #include "net/colo.h" #include "sysemu/iothread.h" +#include "net/colo-compare.h" +#include "migration/colo.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) =20 +static QTAILQ_HEAD(, CompareState) net_compares =3D + QTAILQ_HEAD_INITIALIZER(net_compares); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 /* TODO: Should be configurable */ #define REGULAR_PACKET_CHECK_MS 3000 =20 +static QemuMutex event_mtx =3D { .lock =3D PTHREAD_MUTEX_INITIALIZER, + .initialized =3D true }; +static QemuCond event_complete_cond =3D { .cond =3D PTHREAD_COND_INITIALIZ= ER, + .initialized =3D true}; +static int event_unhandled_count; + /* * + CompareState ++ * | | @@ -86,6 +97,11 @@ typedef struct CompareState { IOThread *iothread; GMainContext *worker_context; QEMUTimer *packet_check_timer; + + QEMUBH *event_bh; + enum colo_event event; + + QTAILQ_ENTRY(CompareState) next; } CompareState; =20 typedef struct CompareClass { @@ -631,6 +647,25 @@ static void check_old_packet_regular(void *opaque) REGULAR_PACKET_CHECK_MS); } =20 +/* Public API, Used for COLO frame to notify compare event */ +void colo_notify_compares_event(void *opaque, int event, Error **errp) +{ + CompareState *s; + + qemu_mutex_lock(&event_mtx); + QTAILQ_FOREACH(s, &net_compares, next) { + s->event =3D event; + qemu_bh_schedule(s->event_bh); + event_unhandled_count++; + } + /* Wait all compare threads to finish handling this event */ + while (event_unhandled_count > 0) { + qemu_cond_wait(&event_complete_cond, &event_mtx); + } + + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_timer_init(CompareState *s) { AioContext *ctx =3D iothread_get_aio_context(s->iothread); @@ -651,6 +686,28 @@ static void colo_compare_timer_del(CompareState *s) } } =20 +static void colo_flush_packets(void *opaque, void *user_data); + +static void colo_compare_handle_event(void *opaque) +{ + CompareState *s =3D opaque; + + switch (s->event) { + case COLO_EVENT_CHECKPOINT: + g_queue_foreach(&s->conn_list, colo_flush_packets, s); + break; + case COLO_EVENT_FAILOVER: + break; + default: + break; + } + qemu_mutex_lock(&event_mtx); + assert(event_unhandled_count > 0); + event_unhandled_count--; + qemu_cond_broadcast(&event_complete_cond); + qemu_mutex_unlock(&event_mtx); +} + static void colo_compare_iothread(CompareState *s) { object_ref(OBJECT(s->iothread)); @@ -664,6 +721,7 @@ static void colo_compare_iothread(CompareState *s) s, s->worker_context, true); =20 colo_compare_timer_init(s); + s->event_bh =3D qemu_bh_new(colo_compare_handle_event, s); } =20 static char *compare_get_pri_indev(Object *obj, Error **errp) @@ -821,6 +879,8 @@ static void colo_compare_complete(UserCreatable *uc, Er= ror **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr); =20 + QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); =20 s->connection_track_table =3D g_hash_table_new_full(connection_key_has= h, @@ -885,6 +945,7 @@ static void colo_compare_init(Object *obj) static void colo_compare_finalize(Object *obj) { CompareState *s =3D COLO_COMPARE(obj); + CompareState *tmp =3D NULL; =20 qemu_chr_fe_deinit(&s->chr_pri_in, false); qemu_chr_fe_deinit(&s->chr_sec_in, false); @@ -892,6 +953,16 @@ static void colo_compare_finalize(Object *obj) if (s->iothread) { colo_compare_timer_del(s); } + + qemu_bh_delete(s->event_bh); + + QTAILQ_FOREACH(tmp, &net_compares, next) { + if (!strcmp(tmp->outdev, s->outdev)) { + QTAILQ_REMOVE(&net_compares, s, next); + break; + } + } + /* Release all unhandled packets after compare thead exited */ g_queue_foreach(&s->conn_list, colo_flush_packets, s); =20 diff --git a/net/colo-compare.h b/net/colo-compare.h new file mode 100644 index 0000000..1b1ce76 --- /dev/null +++ b/net/colo-compare.h @@ -0,0 +1,22 @@ +/* + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) + * (a.k.a. Fault Tolerance or Continuous Replication) + * + * Copyright (c) 2017 HUAWEI TECHNOLOGIES CO., LTD. + * Copyright (c) 2017 FUJITSU LIMITED + * Copyright (c) 2017 Intel Corporation + * + * Authors: + * zhanghailiang + * Zhang Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * later. See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_COLO_COMPARE_H +#define QEMU_COLO_COMPARE_H + +void colo_notify_compares_event(void *opaque, int event, Error **errp); + +#endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515045845141316.90288636718867; Wed, 3 Jan 2018 22:04:05 -0800 (PST) Received: from localhost ([::1]:47454 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWydD-0004lw-AW for importer@patchew.org; Thu, 04 Jan 2018 01:03:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36741) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybC-0003Va-44 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybB-0003xu-90 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:54 -0500 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:45258) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybB-0003xT-2y for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:53 -0500 Received: by mail-pl0-x243.google.com with SMTP id o2so502853plk.12 for ; Wed, 03 Jan 2018 22:01:52 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:01:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kbtB0BcNQHThGQQ1iIJPBdBIOj3/Jf8DLVXdz1dowKw=; b=mheA3wcL1QicrafnikG2pv1sqmCZ4Agd3Lf7PDoS2FXdykfD6C0/Uw7YbmLxIIgYIQ 4qyj15oj3Y//rhxgo6y7K8FHnQNy9dxxgeFJ8f67St+kZXROJdsydgSQykJF+NOGtqbk QPzGwTFTq61+w8U5ScNXApqzRT+95dB+ZXQyy/t8Kk31F87VlYYnaZDNseG25pP+WKxj AQ8MqXlhlyfAPjqY+S6cmfj3roqUB5k9ZTpVk09W8M/WFqwA0DnWeEvzqbSQ51J/bolV 7O+GNIntXXjn73FO1w+UFhwe4bO9SrTzxxvSZPm19p2V4pGi35JUdqrcfFOYGz6a5TRq NoCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kbtB0BcNQHThGQQ1iIJPBdBIOj3/Jf8DLVXdz1dowKw=; b=XvTCbs0AH9QQMoDX9dVC4/Cr26Az1qUErIWE+78bSi1LKLIROtZfEqiWw2WLMzOucH XNIXNNuoWEuFIVJvoFJLxl3XFoWL3QhMJoaoM1ZTQ51tTjBnPBKz9x3Pttg86syu0q72 EKRo0P10MzVLAigdUSAFfTlKMb6kWfmEpLLCoXiowG00UmDqIJ1SK51SzKabB+iadaf2 IZ9IUeYOudb39DB2jugKqKOybhUViD7X8hVs7RMfGqztnxuV02DCoXTPFT5UQ2Ct+AGK oHpnYx84wYLXkIV+sDEd19oBkPiIAUDi3pt5yuHwSpJmWT1fC2OfWVbfeOkSl1nvtskA 1jeA== X-Gm-Message-State: AKGB3mLpyJtoCdIDOCyCZqEahjK1MU2hhlgE6xlr/CNAfgjFVe6FdKYr EyEOdDK1O2EUsITLOQKtCztciwJW X-Google-Smtp-Source: ACJfBotpF3wqdVL+X1PNTChqwopQ3J3QNhZ4Ozp+hwBb1Low5t3pyCe8sLIMQXaWeIzeJ9wZQZx1yg== X-Received: by 10.84.211.143 with SMTP id c15mr3701855pli.100.1515045711629; Wed, 03 Jan 2018 22:01:51 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:02 +0800 Message-Id: <1515045675-6993-4-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH RESEND V3 03/16] colo-compare: use notifier to notify packets comparing result X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Cc: Jason Wang Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +++++++++++++++++++++++++------- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index e9cfca2..dfaa81f 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -31,6 +31,7 @@ #include "sysemu/iothread.h" #include "net/colo-compare.h" #include "migration/colo.h" +#include "migration/migration.h" =20 #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ @@ -39,6 +40,9 @@ static QTAILQ_HEAD(, CompareState) net_compares =3D QTAILQ_HEAD_INITIALIZER(net_compares); =20 +static NotifierList colo_compare_notifiers =3D + NOTIFIER_LIST_INITIALIZER(colo_compare_notifiers); + #define COMPARE_READ_LEN_MAX NET_BUFSIZE #define MAX_QUEUE_SIZE 1024 =20 @@ -454,8 +458,24 @@ static int colo_old_packet_check_one(Packet *pkt, int6= 4_t *check_time) } } =20 +static void colo_compare_inconsistent_notify(void) +{ + notifier_list_notify(&colo_compare_notifiers, + migrate_get_current()); +} + +void colo_compare_register_notifier(Notifier *notify) +{ + notifier_list_add(&colo_compare_notifiers, notify); +} + +void colo_compare_unregister_notifier(Notifier *notify) +{ + notifier_remove(notify); +} + static int colo_old_packet_check_one_conn(Connection *conn, - void *user_data) + void *user_data) { GList *result =3D NULL; int64_t check_time =3D REGULAR_PACKET_CHECK_MS; @@ -466,10 +486,7 @@ static int colo_old_packet_check_one_conn(Connection *= conn, =20 if (result) { /* Do checkpoint will flush old packet */ - /* - * TODO: Notify colo frame to do checkpoint. - * colo_compare_inconsistent_notify(); - */ + colo_compare_inconsistent_notify(); return 0; } =20 @@ -544,11 +561,12 @@ static void colo_compare_connection(void *opaque, voi= d *user_data) /* * If one packet arrive late, the secondary_list or * primary_list will be empty, so we can't compare it - * until next comparison. + * until next comparison. If the packets in the list are + * timeout, it will trigger a checkpoint request. */ trace_colo_compare_main("packet different"); g_queue_push_head(&conn->primary_list, pkt); - /* TODO: colo_notify_checkpoint();*/ + colo_compare_inconsistent_notify(); break; } } diff --git a/net/colo-compare.h b/net/colo-compare.h index 1b1ce76..22ddd51 100644 --- a/net/colo-compare.h +++ b/net/colo-compare.h @@ -18,5 +18,7 @@ #define QEMU_COLO_COMPARE_H =20 void colo_notify_compares_event(void *opaque, int event, Error **errp); +void colo_compare_register_notifier(Notifier *notify); +void colo_compare_unregister_notifier(Notifier *notify); =20 #endif /* QEMU_COLO_COMPARE_H */ --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046139277433.6036264549831; Wed, 3 Jan 2018 22:08:59 -0800 (PST) Received: from localhost ([::1]:47484 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyi1-0001NL-CZ for importer@patchew.org; Thu, 04 Jan 2018 01:08:57 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36763) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybF-0003ZY-3e for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybD-00042A-VY for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:57 -0500 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:34299) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybD-0003zn-OV for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:01:55 -0500 Received: by mail-pl0-x244.google.com with SMTP id d21so521160pll.1 for ; Wed, 03 Jan 2018 22:01:55 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:01:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ILcidBvCgHBg0AKQtZS6QwybvpfG8I9UBqX9Fi229v4=; b=BvAMhfUBRdgsUTCbr7ERVhB4MRipyoeqfcsar8PsN2JgeblQIPMOZxatPy5W16LnhO bsQi4gKXBxWrnAn3g4ytPJ+bmIf2CNDxP+lLRa66vcrnaAnbO1Je4eqnh9IhFagOqUM/ eA5ibnfRUme4/M9BbmgZi+lkxBwFA1puRayfhTbzVS2KdbXQz0jbZrvYwGTToKvfWfvn EaaI94gnWd6kE6G6el0DbbFiDiwukBlLIX6av9lumYCpdlIm1KIyjuru/azF9/6z9XsN Q3l8WFHgyFz5chq5yZQWUvMl1D5PKVHiGKqdeBRlM52octQBLap2vcbgpBp8QCPPd7aq N20g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ILcidBvCgHBg0AKQtZS6QwybvpfG8I9UBqX9Fi229v4=; b=b5PLMEjbhPDcWhPt9DPBgVn+xsvxvtKJITxIBj/Paqw8LY6B3Z/IzfnHL9rOZ0Ryou Qnmr5LiLtxVP56XfZlyMitiw4x8omWzsWCsYT/hlMMB/UyoMgmCFOEFUOpfh86O39WcS r4eCVxiHhuMkfXmANilgpNcQTbT4LZEsoSY2sRGa4lOWcZaDNotZ+Lnv2ckaNCIXpXXF Pf/H+WdDyuhVT5bB71GfOUr4VGr/xZU9WecKhfiwajvpWwuJtoRhc+QQbnjDvQEgFOZr zjzfy35pv9GulckIlmhEF8oUQ+djpzfckG6tbDygvKkAm/RHgw8WTxTqbb2dQ4q1ZOpf 8OyQ== X-Gm-Message-State: AKGB3mK2Ow7HjB64Q/Ex/3gIdIGNeNhfadefJEDyYraYqILRNepcVN6q f5p42sCrLulqDKTvUMFEG8pRqg+b X-Google-Smtp-Source: ACJfBouTY5aLcr3A/X9X5qOvXaAfh4SekR6pEjq+i3T/i1FEZhhSYKPxpezUBj3cH8PQ4B05pphB3A== X-Received: by 10.84.128.34 with SMTP id 31mr3563624pla.252.1515045714348; Wed, 03 Jan 2018 22:01:54 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:03 +0800 Message-Id: <1515045675-6993-5-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH RESEND V3 04/16] COLO: integrate colo compare with colo frame X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For COLO FT, both the PVM and SVM run at the same time, only sync the state while it needs. So here, let SVM runs while not doing checkpoint, change DEFAULT_MIGRATE_X_CHECKPOINT_DELAY to 200*100. Besides, we forgot to release colo_checkpoint_semd and colo_delay_timer, fix them here. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 ++++++++++++++++++++++++++++++++++++++++-- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index dee3aa8..c513805 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -24,8 +24,11 @@ #include "migration/failover.h" #include "replication.h" #include "qmp-commands.h" +#include "net/colo-compare.h" +#include "net/colo.h" =20 static bool vmstate_loading; +static Notifier packets_compare_notifier; =20 #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024) =20 @@ -342,6 +345,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, goto out; } =20 + colo_notify_compares_event(NULL, COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + goto out; + } + /* Disable block migration */ migrate_set_block_enabled(false, &local_err); qemu_savevm_state_header(fb); @@ -399,6 +407,11 @@ out: return ret; } =20 +static void colo_compare_notify_checkpoint(Notifier *notifier, void *data) +{ + colo_checkpoint_notify(data); +} + static void colo_process_checkpoint(MigrationState *s) { QIOChannelBuffer *bioc; @@ -415,6 +428,9 @@ static void colo_process_checkpoint(MigrationState *s) goto out; } =20 + packets_compare_notifier.notify =3D colo_compare_notify_checkpoint; + colo_compare_register_notifier(&packets_compare_notifier); + /* * Wait for Secondary finish loading VM states and enter COLO * restore. @@ -460,11 +476,21 @@ out: qemu_fclose(fb); } =20 - timer_del(s->colo_delay_timer); - /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); + + /* + * It is safe to unregister notifier after failover finished. + * Besides, colo_delay_timer and colo_checkpoint_sem can't be + * released befor unregister notifier, or there will be use-after-free + * error. + */ + colo_compare_unregister_notifier(&packets_compare_notifier); + timer_del(s->colo_delay_timer); + timer_free(s->colo_delay_timer); + qemu_sem_destroy(&s->colo_checkpoint_sem); + /* * Must be called after failover BH is completed, * Or the failover BH may shutdown the wrong fd that @@ -557,6 +583,11 @@ void *colo_process_incoming_thread(void *opaque) fb =3D qemu_fopen_channel_input(QIO_CHANNEL(bioc)); object_unref(OBJECT(bioc)); =20 + qemu_mutex_lock_iothread(); + vm_start(); + trace_colo_vm_state_change("stop", "run"); + qemu_mutex_unlock_iothread(); + colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_READY, &local_err); if (local_err) { @@ -576,6 +607,11 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + vm_stop_force_state(RUN_STATE_COLO); + trace_colo_vm_state_change("run", "stop"); + qemu_mutex_unlock_iothread(); + /* FIXME: This is unnecessary for periodic checkpoint mode */ colo_send_message(mis->to_src_file, COLO_MESSAGE_CHECKPOINT_REPLY, &local_err); @@ -629,6 +665,8 @@ void *colo_process_incoming_thread(void *opaque) } =20 vmstate_loading =3D false; + vm_start(); + trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); =20 if (failover_get_state() =3D=3D FAILOVER_STATUS_RELAUNCH) { diff --git a/migration/migration.c b/migration/migration.c index 4de3b55..ced463c 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -74,9 +74,9 @@ #define DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE (64 * 1024 * 1024) =20 /* The delay time (in ms) between two COLO checkpoints - * Note: Please change this default value to 10000 when we support hybrid = mode. + * Note: Please change this default value to 20000 when we support hybrid = mode. */ -#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY 200 +#define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) #define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 #define DEFAULT_MIGRATE_MULTIFD_PAGE_COUNT 16 =20 --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515045849901465.56635756698006; Wed, 3 Jan 2018 22:04:09 -0800 (PST) Received: from localhost ([::1]:47455 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWydN-0004tk-0C for importer@patchew.org; Thu, 04 Jan 2018 01:04:09 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36808) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybJ-0003cD-Eb for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybI-000470-7W for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:01 -0500 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:36426) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybH-00046f-Ut for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:00 -0500 Received: by mail-pl0-x244.google.com with SMTP id b12so521050plm.3 for ; Wed, 03 Jan 2018 22:01:59 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:01:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RYaeajl/PXeBCb3ZAtFya+2/+i1BhYWMfvQa7R4SAwQ=; b=OzfEjZRTwimWc7c6gj/3V4ikYnFa5zz0HFJf/Q1DYGC22Ps7aPC7te/PwgnVO7UJ+i bTuB8NJmLnjVEpzPlaM7c9dXavYPQhPDehYqJKAQGiDwSCWf5eh5ePSUfpX7N3iPOHRV kTFZgwKDOfynRC8BAr5UcI6PZIowpKaip0SbTvgf84JL9eWjrEkqc7PeIM9eCNgFqjfH 7u7+AGRtSfomYyKKvOIu1OGLDg1KPSyyedyCzvBLgramhW4E8LRwOw7EiLOKTQPwicwi EVYaTxkjtP+312AegF/T4C3a93MG34RPIw4jLD0TrVQuLrdLHoMTjxflBR+bAxIXMSqX NwHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RYaeajl/PXeBCb3ZAtFya+2/+i1BhYWMfvQa7R4SAwQ=; b=aIMDXdK9c0QS/anVlt/2n8EOdylR/Kc5WlVt8YHxKGuPiXkiodQ4dd0Yk8R0hNvksJ Kv/Se0atX3l6UVltnWngOpZxLJy3buFokmKXSEWPZg8eJh98FQms5CNE30MRQYuD/ZV0 D/PVovYaLeUnMXgAESo0sLoK9gDixKkwu00LRtwW0Va+dOChkuEJ0P3dLn8SywIhm/uT BbarOU+Q7m9km8h/uZz7iQnQlF1OZ5cTUsXR8zzWD8OKgwV2y2B0RF2qXN0xKrMICFYl i0wsk8Pjm/bdUkEbz5vFlChFOsBTKBkiDqVtTSi2bYx6O7F/lO2u1Sbe/gzqpj0cgcqQ eLcA== X-Gm-Message-State: AKGB3mKOy2dO7/WhD58g/QdOr033yenyUmODC/Y5MRbliYLIQrH3gqhA GUc5uIf71cbeSBa/ZkYVy36e/SQ4 X-Google-Smtp-Source: ACJfBovsQgSYXmBt9l9D4Ry10/27v2DwqsRL/H+bGRJEice/AfhrK4hYmp4xQ5TWtttGxgEe9To9bw== X-Received: by 10.159.246.129 with SMTP id c1mr3628405pls.262.1515045718621; Wed, 03 Jan 2018 22:01:58 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:04 +0800 Message-Id: <1515045675-6993-6-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH RESEND V3 05/16] COLO: Add block replication into colo process X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , zhanghailiang , Li Zhijian , Xie Changlong , Max Reitz , Stefan Hajnoczi , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Cc: Stefan Hajnoczi Cc: Kevin Wolf Cc: Max Reitz Cc: Xie Changlong --- migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ migration/migration.c | 9 +++++++++ 2 files changed, 55 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index c513805..0e689df 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -26,6 +26,9 @@ #include "qmp-commands.h" #include "net/colo-compare.h" #include "net/colo.h" +#include "qapi-event.h" +#include "block/block.h" +#include "replication.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -55,6 +58,7 @@ static void secondary_vm_do_failover(void) { int old_state; MigrationIncomingState *mis =3D migration_incoming_get_current(); + Error *local_err =3D NULL; =20 /* Can not do failover during the process of VM's loading VMstate, Or * it will break the secondary VM. @@ -72,6 +76,11 @@ static void secondary_vm_do_failover(void) migrate_set_state(&mis->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); =20 + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + } + if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); /* recover runstate to normal migration finish state */ @@ -109,6 +118,7 @@ static void primary_vm_do_failover(void) { MigrationState *s =3D migrate_get_current(); int old_state; + Error *local_err =3D NULL; =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); @@ -132,6 +142,13 @@ static void primary_vm_do_failover(void) FailoverStatus_str(old_state)); return; } + + replication_stop_all(true, &local_err); + if (local_err) { + error_report_err(local_err); + local_err =3D NULL; + } + /* Notify COLO thread that failover work is finished */ qemu_sem_post(&s->colo_exit_sem); } @@ -355,6 +372,11 @@ static int colo_do_checkpoint_transaction(MigrationSta= te *s, qemu_savevm_state_header(fb); qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } qemu_savevm_state_complete_precopy(fb, false, false); qemu_mutex_unlock_iothread(); =20 @@ -396,6 +418,7 @@ static int colo_do_checkpoint_transaction(MigrationStat= e *s, ret =3D 0; =20 qemu_mutex_lock_iothread(); + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -445,6 +468,12 @@ static void colo_process_checkpoint(MigrationState *s) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vm_start(); qemu_mutex_unlock_iothread(); trace_colo_vm_state_change("stop", "run"); @@ -584,6 +613,11 @@ void *colo_process_incoming_thread(void *opaque) object_unref(OBJECT(bioc)); =20 qemu_mutex_lock_iothread(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } vm_start(); trace_colo_vm_state_change("stop", "run"); qemu_mutex_unlock_iothread(); @@ -664,6 +698,18 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + replication_get_error_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + /* discard colo disk buffer */ + replication_do_checkpoint_all(&local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/migration/migration.c b/migration/migration.c index ced463c..3410145 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -318,6 +318,7 @@ static void process_incoming_migration_co(void *opaque) MigrationIncomingState *mis =3D migration_incoming_get_current(); PostcopyState ps; int ret; + Error *local_err =3D NULL; =20 assert(mis->from_src_file); mis->largest_page_size =3D qemu_ram_pagesize_largest(); @@ -349,6 +350,14 @@ static void process_incoming_migration_co(void *opaque) =20 /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { + /* Make sure all file formats flush their mutable metadata */ + bdrv_invalidate_cache_all(&local_err); + if (local_err) { + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, + MIGRATION_STATUS_FAILED); + error_report_err(local_err); + exit(EXIT_FAILURE); + } mis->migration_incoming_co =3D qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046066696966.8541780337384; Wed, 3 Jan 2018 22:07:46 -0800 (PST) Received: from localhost ([::1]:47474 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWygr-0008N1-PA for importer@patchew.org; Thu, 04 Jan 2018 01:07:45 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36834) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybM-0003eq-Hs for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybL-000495-5f for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:04 -0500 Received: from mail-pf0-x236.google.com ([2607:f8b0:400e:c00::236]:45419) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybK-00048Y-Sg for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:03 -0500 Received: by mail-pf0-x236.google.com with SMTP id u19so365429pfa.12 for ; Wed, 03 Jan 2018 22:02:02 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.01.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tJwMKr65P43f9hnTSIq0WX5JUEtd8NTGAL9rE8Qt9MY=; b=tAhGlXhSrny0GeZDfgAABeFsqz7EPdlwNLHbAmfzuzdL0Ie+sQJ0JTlbeyNlkqTzAK ntL58ve34eBiLfExN+y8j52YuiZPVGCF19m/AomzR0JR/V6imtB89NsfCD27icKM8HtM yDgUEN0RGj6EHW1FmE2Re8WXEk6PsdZbAnHW+TLlU5XAe/JIB9AW1MEH01J7Be6f/PrD jJFyL3vLfwRtzKW4mRIH6Be9sPtOBBskAAzx+tf84kmx7hjQSq/4joFelrERE2StYSjm MgOSWuxIvwtWZPR7s9M1MvXGikuzwovnLpjE4EmzR6mSpeaBPAXbkFyG+qtaNmORF0OP JSXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tJwMKr65P43f9hnTSIq0WX5JUEtd8NTGAL9rE8Qt9MY=; b=mvuixNI97YfQuf8YHVBE/A7DAzmhfRkJQTBVq+bo92z4KOZtfvs+0JXoS/9dVJE5iO BKGsFaoHrYt9qkreXsrMYxwf9xmnD6CNaW7M2JHprk8LSiMGI1LoHYcb9ItmhOi+6gTd IsYrlTnC+KVCIdKjxaP+0dLzQ4LL9Z0X/gGW0tqnVpbCM0zbnbURMT92u3/bUJtj5w9W A3aASsAMcHJXsa4Hf4E9vyfoVCPUlUbEmZ8mrM+8yt9C+7W9zIFRx+0UCEB+lT1qJu5w GNYutezSXbDMfTHLIogElsXtDui4Vj60G9uGP7meq88Zlrij3q5sajff9YxlV65xO945 VEbw== X-Gm-Message-State: AKGB3mJUz1KbKgV900++dDHJ6p1g99W5N5/OhSLblPTRoTQdGJrCwJ9p Ag821cXR9QUAIfNtYqyMlygPxmLC X-Google-Smtp-Source: ACJfBov3LpKFmAunbx+ph9Hpd9uF8tuKQYEaHyCbztaXHsdvUna5gM8Bl5rvCYppO1JmAv7gsXQRLQ== X-Received: by 10.99.173.3 with SMTP id g3mr3039910pgf.87.1515045721467; Wed, 03 Jan 2018 22:02:01 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:05 +0800 Message-Id: <1515045675-6993-7-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::236 Subject: [Qemu-devel] [PATCH RESEND V3 06/16] COLO: Remove colo_state migration struct X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Signed-off-by: zhanghailiang --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c | 76 --------------------------------------------= ---- migration/colo.c | 13 ++++++++- migration/migration.c | 23 ++++++++++++++- migration/savevm.c | 19 ++++++++++++ migration/savevm.h | 1 + migration/trace-events | 1 + vl.c | 2 -- 9 files changed, 59 insertions(+), 83 deletions(-) delete mode 100644 migration/colo-comm.c diff --git a/include/migration/colo.h b/include/migration/colo.h index 6adf3a5..546cb9a 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -27,8 +27,9 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); =20 /* loadvm */ -bool migration_incoming_enable_colo(void); -void migration_incoming_exit_colo(void); +void migration_incoming_enable_colo(void); +void migration_incoming_disable_colo(void); +bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); bool migration_incoming_in_colo_state(void); =20 diff --git a/migration/Makefile.objs b/migration/Makefile.objs index 99e0380..3099eec 100644 --- a/migration/Makefile.objs +++ b/migration/Makefile.objs @@ -1,6 +1,6 @@ common-obj-y +=3D migration.o socket.o fd.o exec.o common-obj-y +=3D tls.o channel.o savevm.o -common-obj-y +=3D colo-comm.o colo.o colo-failover.o +common-obj-y +=3D colo.o colo-failover.o common-obj-y +=3D vmstate.o vmstate-types.o page_cache.o common-obj-y +=3D qemu-file.o global_state.o common-obj-y +=3D qemu-file-channel.o diff --git a/migration/colo-comm.c b/migration/colo-comm.c deleted file mode 100644 index df26e4d..0000000 --- a/migration/colo-comm.c +++ /dev/null @@ -1,76 +0,0 @@ -/* - * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO) - * (a.k.a. Fault Tolerance or Continuous Replication) - * - * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. - * Copyright (c) 2016 FUJITSU LIMITED - * Copyright (c) 2016 Intel Corporation - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "migration.h" -#include "migration/colo.h" -#include "migration/vmstate.h" -#include "trace.h" - -typedef struct { - bool colo_requested; -} COLOInfo; - -static COLOInfo colo_info; - -COLOMode get_colo_mode(void) -{ - if (migration_in_colo_state()) { - return COLO_MODE_PRIMARY; - } else if (migration_incoming_in_colo_state()) { - return COLO_MODE_SECONDARY; - } else { - return COLO_MODE_UNKNOWN; - } -} - -static int colo_info_pre_save(void *opaque) -{ - COLOInfo *s =3D opaque; - - s->colo_requested =3D migrate_colo_enabled(); - - return 0; -} - -static bool colo_info_need(void *opaque) -{ - return migrate_colo_enabled(); -} - -static const VMStateDescription colo_state =3D { - .name =3D "COLOState", - .version_id =3D 1, - .minimum_version_id =3D 1, - .pre_save =3D colo_info_pre_save, - .needed =3D colo_info_need, - .fields =3D (VMStateField[]) { - VMSTATE_BOOL(colo_requested, COLOInfo), - VMSTATE_END_OF_LIST() - }, -}; - -void colo_info_init(void) -{ - vmstate_register(NULL, 0, &colo_state, &colo_info); -} - -bool migration_incoming_enable_colo(void) -{ - return colo_info.colo_requested; -} - -void migration_incoming_exit_colo(void) -{ - colo_info.colo_requested =3D false; -} diff --git a/migration/colo.c b/migration/colo.c index 0e689df..8d2e3f8 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -153,6 +153,17 @@ static void primary_vm_do_failover(void) qemu_sem_post(&s->colo_exit_sem); } =20 +COLOMode get_colo_mode(void) +{ + if (migration_in_colo_state()) { + return COLO_MODE_PRIMARY; + } else if (migration_incoming_in_colo_state()) { + return COLO_MODE_SECONDARY; + } else { + return COLO_MODE_UNKNOWN; + } +} + void colo_do_failover(MigrationState *s) { /* Make sure VM stopped while failover happened. */ @@ -747,7 +758,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } - migration_incoming_exit_colo(); + migration_incoming_disable_colo(); =20 return NULL; } diff --git a/migration/migration.c b/migration/migration.c index 3410145..8c16129 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -240,6 +240,22 @@ void migrate_send_rp_req_pages(MigrationIncomingState = *mis, const char *rbname, } } =20 +static bool migration_colo_enabled; +bool migration_incoming_colo_enabled(void) +{ + return migration_colo_enabled; +} + +void migration_incoming_disable_colo(void) +{ + migration_colo_enabled =3D false; +} + +void migration_incoming_enable_colo(void) +{ + migration_colo_enabled =3D true; +} + void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p; @@ -349,7 +365,7 @@ static void process_incoming_migration_co(void *opaque) } =20 /* we get COLO info, and know if we are in COLO mode */ - if (!ret && migration_incoming_enable_colo()) { + if (!ret && migration_incoming_colo_enabled()) { /* Make sure all file formats flush their mutable metadata */ bdrv_invalidate_cache_all(&local_err); if (local_err) { @@ -2211,6 +2227,11 @@ static void *migration_thread(void *opaque) qemu_savevm_send_postcopy_advise(s->to_dst_file); } =20 + if (enable_colo) { + /* Notify migration destination that we enable COLO */ + qemu_savevm_send_colo_enable(s->to_dst_file); + } + qemu_savevm_state_setup(s->to_dst_file); =20 s->setup_time =3D qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start; diff --git a/migration/savevm.c b/migration/savevm.c index b7908f6..cd753c4 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -52,6 +52,7 @@ #include "qemu/cutils.h" #include "io/channel-buffer.h" #include "io/channel-file.h" +#include "migration/colo.h" =20 #ifndef ETH_P_RARP #define ETH_P_RARP 0x8035 @@ -78,6 +79,9 @@ enum qemu_vm_cmd { were previously sent during precopy but are dirty. */ MIG_CMD_PACKAGED, /* Send a wrapped stream within this stream= */ + + MIG_CMD_ENABLE_COLO, /* Enable COLO */ + MIG_CMD_MAX }; =20 @@ -833,6 +837,12 @@ static void qemu_savevm_command_send(QEMUFile *f, qemu_fflush(f); } =20 +void qemu_savevm_send_colo_enable(QEMUFile *f) +{ + trace_savevm_send_colo_enable(); + qemu_savevm_command_send(f, MIG_CMD_ENABLE_COLO, 0, NULL); +} + void qemu_savevm_send_ping(QEMUFile *f, uint32_t value) { uint32_t buf; @@ -1749,6 +1759,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncom= ingState *mis) return ret; } =20 +static int loadvm_process_enable_colo(MigrationIncomingState *mis) +{ + migration_incoming_enable_colo(); + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -1817,6 +1833,9 @@ static int loadvm_process_command(QEMUFile *f) =20 case MIG_CMD_POSTCOPY_RAM_DISCARD: return loadvm_postcopy_ram_handle_discard(mis, len); + + case MIG_CMD_ENABLE_COLO: + return loadvm_process_enable_colo(mis); } =20 return 0; diff --git a/migration/savevm.h b/migration/savevm.h index 295c4a1..041d23c 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -51,6 +51,7 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, c= onst char *name, uint16_t len, uint64_t *start_list, uint64_t *length_list); +void qemu_savevm_send_colo_enable(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); diff --git a/migration/trace-events b/migration/trace-events index 6f29fcc..d4738c8 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -34,6 +34,7 @@ savevm_send_open_return_path(void) "" savevm_send_ping(uint32_t val) "0x%x" savevm_send_postcopy_listen(void) "" savevm_send_postcopy_run(void) "" +savevm_send_colo_enable(void) "" savevm_state_setup(void) "" savevm_state_header(void) "" savevm_state_iterate(void) "" diff --git a/vl.c b/vl.c index d3a5c5d..529cdc1 100644 --- a/vl.c +++ b/vl.c @@ -4578,8 +4578,6 @@ int main(int argc, char **argv, char **envp) #endif } =20 - colo_info_init(); - if (net_init_clients() < 0) { exit(1); } --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 151504602615257.083298595728365; Wed, 3 Jan 2018 22:07:06 -0800 (PST) Received: from localhost ([::1]:47473 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyg9-0007go-6o for importer@patchew.org; Thu, 04 Jan 2018 01:07:01 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36909) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybR-0003jf-R2 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybO-0004BD-N4 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:09 -0500 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:42367) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybO-0004AX-EU for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:06 -0500 Received: by mail-pf0-x243.google.com with SMTP id d23so369671pfe.9 for ; Wed, 03 Jan 2018 22:02:06 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=VAikSzTF03wKyXAuXZOfmDeIsQ3AD0B+rsnNTHsnv2c=; b=BjdUDQygF8fb6+T6NVVUkmX5RaLNu+er8A4ppzdpBaRSuUUocOew90BWhbR0e4KSxL MLmM4Lfy2oJUlmv+WgQifj/rYNlB7zrYpPHWNKKXOp0HNoCmebcMt4Uk1oysXmPYlTfq Cxrs32PMbmvS7ZL4eXhdrGX5hGhSJFaoM/7vhGG+ocLP12x7b3SqPl72GRg9rjoJoz9o +0SJeZBRBF3PtujhRl5dnCruQS64fiXXfZT/qOKl5SgleAFG8LaLjU3ZZtAROG8C60ga B4P9AWNRlkK9SS1SzLAzEcB4Qv4hqZRu2Emmu9n8DGdCRuEdk4egj2SLT61Ms+zItAFs iJ6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=VAikSzTF03wKyXAuXZOfmDeIsQ3AD0B+rsnNTHsnv2c=; b=pvX6tHP38ZS55p/0sT93r4e1AAmt3hSbWkfryWGTX7mMvjgqoIccLnXc/Vj2N3dk/b Ynq2xpTIoVJXxGcyeqnryIX6ESPJpDpB1fq1oKiicL4+UjzB8EcIz+jUq/qm7ucgVil8 wACJCYnb805u2t+wIDF5+5o7Kwpuz0wD2RRHRjJE2Z/p1GV0P54wD4sYOqBYc5bz5CeO 7sO3yfVIq/XFItFlq9W7TqK2QkgDxWsLEtTFR3oCW090EInGOGjPIa9HPr8XLre6CwZ1 snz5zw8pnu+1pr8byT+neXCC/xzQuIMKoUXCKAh8Q0zvbwD+4O3T+yi9fHWLF3FLoGMu NMwQ== X-Gm-Message-State: AKGB3mIWXWnscCkJzqc0vsNgfLH3By4wfVCANtLAXHcPSRaqBVyJbbt4 u3VGQ5KK0F8e7XR7vUlF+n5EOo7U X-Google-Smtp-Source: ACJfBosUu21ql7TYT+7aXP+M8bzTF/yCuw9k2VL/3Jqg8zqvNI712arLZfPKBpqjYlqCEnb2V6Xuyw== X-Received: by 10.98.102.74 with SMTP id a71mr3659090pfc.184.1515045725098; Wed, 03 Jan 2018 22:02:05 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:06 +0800 Message-Id: <1515045675-6993-8-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::243 Subject: [Qemu-devel] [PATCH RESEND V3 07/16] COLO: Load dirty pages into SVM's RAM cache firstly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , "Dr . David Alan Gilbert" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang We should not load PVM's state directly into SVM, because there maybe some errors happen when SVM is receving data, which will break SVM. We need to ensure receving all data before load the state into SVM. We use an extra memory to cache these data (PVM's ram). The ram cache in secondary= side is initially the same as SVM/PVM's memory. And in the process of checkpoint, we cache the dirty pages of PVM into this ram cache firstly, so this ram ca= che always the same as PVM's memory at every checkpoint, then we flush this cac= hed ram to SVM after we receive all PVM's state. Cc: Dr. David Alan Gilbert Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 97 +++++++++++++++++++++++++++++++++++++++++++++= ++-- migration/ram.h | 4 ++ migration/savevm.c | 2 +- 5 files changed, 102 insertions(+), 4 deletions(-) diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h index 6cbc02a..6b7b0dd 100644 --- a/include/exec/ram_addr.h +++ b/include/exec/ram_addr.h @@ -27,6 +27,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; ram_addr_t max_length; diff --git a/migration/migration.c b/migration/migration.c index 8c16129..315b6d4 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -382,6 +382,8 @@ static void process_incoming_migration_co(void *opaque) =20 /* Wait checkpoint incoming thread exit before free resource */ qemu_thread_join(&mis->colo_incoming_thread); + /* We hold the global iothread lock, so it is safe here */ + colo_release_ram_cache(); } =20 if (ret < 0) { diff --git a/migration/ram.c b/migration/ram.c index 021d583..0fc0aee 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2466,6 +2466,20 @@ static inline void *host_from_ram_block_offset(RAMBl= ock *block, return block->host + offset; } =20 +static inline void *colo_cache_from_block_offset(RAMBlock *block, + ram_addr_t offset) +{ + if (!offset_in_ramblock(block, offset)) { + return NULL; + } + if (!block->colo_cache) { + error_report("%s: colo_cache is NULL in block :%s", + __func__, block->idstr); + return NULL; + } + return block->colo_cache + offset; +} + /** * ram_handle_compressed: handle the zero page case * @@ -2619,6 +2633,55 @@ static void decompress_data_with_multi_threads(QEMUF= ile *f, qemu_mutex_unlock(&decomp_done_lock); } =20 +/* + * colo cache: this is for secondary VM, we cache the whole + * memory of the secondary VM, it is need to hold the global lock + * to call this helper. + */ +int colo_init_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + block->colo_cache =3D qemu_anon_ram_alloc(block->used_length, NULL= ); + if (!block->colo_cache) { + error_report("%s: Can't alloc memory for COLO cache of block %= s," + "size 0x" RAM_ADDR_FMT, __func__, block->idstr, + block->used_length); + goto out_locked; + } + } + rcu_read_unlock(); + return 0; + +out_locked: + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + + rcu_read_unlock(); + return -errno; +} + +/* It is need to hold the global lock to call this helper */ +void colo_release_ram_cache(void) +{ + RAMBlock *block; + + rcu_read_lock(); + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + if (block->colo_cache) { + qemu_anon_ram_free(block->colo_cache, block->used_length); + block->colo_cache =3D NULL; + } + } + rcu_read_unlock(); +} + /** * ram_load_setup: Setup RAM for migration incoming side * @@ -2632,6 +2695,7 @@ static int ram_load_setup(QEMUFile *f, void *opaque) xbzrle_load_setup(); compress_threads_load_setup(); ramblock_recv_map_init(); + return 0; } =20 @@ -2645,6 +2709,7 @@ static int ram_load_cleanup(void *opaque) g_free(rb->receivedmap); rb->receivedmap =3D NULL; } + return 0; } =20 @@ -2845,7 +2910,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) =20 while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr, total_ram_bytes; - void *host =3D NULL; + void *host =3D NULL, *host_bak =3D NULL; uint8_t ch; =20 addr =3D qemu_get_be64(f); @@ -2865,13 +2930,36 @@ static int ram_load(QEMUFile *f, void *opaque, int = version_id) RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block =3D ram_block_from_stream(f, flags); =20 - host =3D host_from_ram_block_offset(block, addr); + /* + * After going into COLO, we should load the Page into colo_ca= che + * NOTE: We need to keep a copy of SVM's ram in colo_cache. + * Privously, we copied all these memory in preparing stage of= COLO + * while we need to stop VM, which is a time-consuming process. + * Here we optimize it by a trick, back-up every page while in + * migration process while COLO is enabled, though it affects = the + * speed of the migration, but it obviously reduce the downtim= e of + * back-up all SVM'S memory in COLO preparing stage. + */ + if (migration_incoming_in_colo_state()) { + host =3D colo_cache_from_block_offset(block, addr); + /* After goes into COLO state, don't backup it any more */ + if (!migration_incoming_in_colo_state()) { + host_bak =3D host; + } + } + if (!migration_incoming_in_colo_state()) { + host =3D host_from_ram_block_offset(block, addr); + } if (!host) { error_report("Illegal RAM offset " RAM_ADDR_FMT, addr); ret =3D -EINVAL; break; } - ramblock_recv_bitmap_set(block, host); + + if (!migration_incoming_in_colo_state()) { + ramblock_recv_bitmap_set(block, host); + } + trace_ram_load_loop(block->idstr, (uint64_t)addr, flags, host); } =20 @@ -2966,6 +3054,9 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) if (!ret) { ret =3D qemu_file_get_error(f); } + if (!ret && host_bak && host) { + memcpy(host_bak, host, TARGET_PAGE_SIZE); + } } =20 wait_for_decompress_done(); diff --git a/migration/ram.h b/migration/ram.h index 64d81e9..07abf71 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -62,4 +62,8 @@ int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_ad= dr); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t = nr); =20 +/* ram cache */ +int colo_init_ram_cache(void); +void colo_release_ram_cache(void); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index cd753c4..c582716 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1762,7 +1762,7 @@ static int loadvm_handle_cmd_packaged(MigrationIncomi= ngState *mis) static int loadvm_process_enable_colo(MigrationIncomingState *mis) { migration_incoming_enable_colo(); - return 0; + return colo_init_ram_cache(); } =20 /* --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046253972834.9828747354918; Wed, 3 Jan 2018 22:10:53 -0800 (PST) Received: from localhost ([::1]:47501 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyjt-0003Eb-70 for importer@patchew.org; Thu, 04 Jan 2018 01:10:53 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36920) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybS-0003ji-OG for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybR-0004EI-Uf for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:10 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:41971) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybR-0004Da-PP for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:09 -0500 Received: by mail-pg0-x242.google.com with SMTP id 77so337610pgd.8 for ; Wed, 03 Jan 2018 22:02:09 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1O7DhA3M1mLMMkTBL0M4sTQjX2JumqYNE/EWLuud8p8=; b=VIKTnYpMdXbPvF5kmt4Eop+gQTDWrAssXEFI0cUYuLu84P1G/sgZSZtYYGYMzXYBA4 SbYsfNIWE/HdPby6uV/5eBHhsRuUGWEHdCDBdjkq1U2Y4ZIzbVD1/Gu7E9epNyPxbGTX DFeTXPcoxdg0zTXTLbaAqc5IxuS+EWNLMQwsTVBko3gu4c1C47xi1OKJ8LqbMB5NpL1P snOXC6qb7ao9kLZEX6KhKLqmZaPACt1bKRhZEtvXiLMsG+XgVyOiT9El/k8C7dL8IV6i OsP8rX+VE/MjnvrUY2/7eigDhe9sUOMMQ8CWmn7htyCI6JtWYHk3jfcvRlftjKJ4BwpG LjCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1O7DhA3M1mLMMkTBL0M4sTQjX2JumqYNE/EWLuud8p8=; b=RKbUrUE5A0bPh0HKhZQKssuAjkQMX2Ikwbx/zyZbGLF/DQgnQrqnyOQDzWdufjRqK6 s+f3t5LyuhlnCyIXs7XkVakjNaC5XO5wBdsHH4YH/YH8NWfwcfeOn5wb2pTLRWGeHkBR WTkm+TLl9Cedvvz0aO3WmKUP+O6xhNlnGspI4+jWZhkbFQUd2VZpj14LfpR+WdautwY7 7xcjyg+5mdVXZyu4bjCFoqtavyWuiyZZ0wIc+qt16WRBVNp3BgH8Ao2OG3oy/wfOa1DB RMiUBBNJEWhKIAyHtVua6Z1dkoQ6kAPjxnQPH0Fdazn2ePcWQtrsJRbcQCHOiZ8d1UPS zMIg== X-Gm-Message-State: AKGB3mJ60gup9hYpEbZvHSeQu+krStKWNR8w1PoPFd4LTJcCOIYHjch9 6lVaSX83oxd/9sge93NdIRb2pyUu X-Google-Smtp-Source: ACJfBovehN5VRDnkJUgNMEr86awtLEpQxRO77xfurCibx19JgdupVo9cjiF0IR1zjHZT0qsjgd3mLQ== X-Received: by 10.98.224.200 with SMTP id d69mr3770578pfm.100.1515045728417; Wed, 03 Jan 2018 22:02:08 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:07 +0800 Message-Id: <1515045675-6993-9-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH RESEND V3 08/16] ram/COLO: Record the dirty pages that SVM received X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juan Quintela , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the dirty pages caused by PVM and SVM, we only flush those dirty pages from RAM cache while do checkpoint. Cc: Juan Quintela Signed-off-by: zhanghailiang Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 0fc0aee..388333d 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2477,6 +2477,15 @@ static inline void *colo_cache_from_block_offset(RAM= Block *block, __func__, block->idstr); return NULL; } + + /* + * During colo checkpoint, we need bitmap of these migrated pages. + * It help us to decide which pages in ram cache should be flushed + * into VM's RAM later. + */ + if (!test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { + ram_state->migration_dirty_pages++; + } return block->colo_cache + offset; } =20 @@ -2653,6 +2662,24 @@ int colo_init_ram_cache(void) } } rcu_read_unlock(); + /* + * Record the dirty pages that sent by PVM, we use this dirty bitmap to= gether + * with to decide which page in cache should be flushed into SVM's RAM.= Here + * we use the same name 'ram_bitmap' as for migration. + */ + if (ram_bytes_total()) { + RAMBlock *block; + + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + unsigned long pages =3D block->max_length >> TARGET_PAGE_BITS; + + block->bmap =3D bitmap_new(pages); + bitmap_set(block->bmap, 0, pages); + } + } + ram_state =3D g_new0(RAMState, 1); + ram_state->migration_dirty_pages =3D 0; + return 0; =20 out_locked: @@ -2672,6 +2699,10 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { + g_free(block->bmap); + block->bmap =3D NULL; + } rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2680,6 +2711,8 @@ void colo_release_ram_cache(void) } } rcu_read_unlock(); + g_free(ram_state); + ram_state =3D NULL; } =20 /** --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046199567537.8133135812744; Wed, 3 Jan 2018 22:09:59 -0800 (PST) Received: from localhost ([::1]:47486 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyj0-0002IF-Pm for importer@patchew.org; Thu, 04 Jan 2018 01:09:58 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36949) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybW-0003nC-Kz for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybV-0004GV-E7 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:14 -0500 Received: from mail-pf0-x230.google.com ([2607:f8b0:400e:c00::230]:45414) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybV-0004G1-8x for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:13 -0500 Received: by mail-pf0-x230.google.com with SMTP id u19so365592pfa.12 for ; Wed, 03 Jan 2018 22:02:13 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o9YWNPx3CTT5V9kJXFl72B413c+NQAN7NhavooxbAVM=; b=ZVTidTRF/i6WHxmN/S2+LAlzpUMljHTipM97xSQqF/FJRO5i7i8Ii7udTI5lLnuaXs 0sgUhK4mdlpPbQsMjVWIC1zwu3xDFSJNmdJO1gS6lpJ2xHMKm9GwBuJn9ZS9PcURN6JQ 9DCBU95yY8P76+h0yOrgiBms6UgJRv7zg5siMQFWnLttvn2IOMM2LYctVUT9mjwbwN5I F8wsCMmpIt5Q1xUk1PBKALOP1N5kJneBr0aD3r8ivUW7RMVheEvzC6UBWBrnNYF895Yz XWaz4qewSYNOO3sgMJvOU+clVn7b118K1GW3mOrEveVSM3bItJLB31iemRFfOFcIZhVj Ql6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o9YWNPx3CTT5V9kJXFl72B413c+NQAN7NhavooxbAVM=; b=cGFZHGAzrsPV90ruPlJEQMQ/iGzQHqywukxuhrlMIzZXmoTwuh1YyynlzotiTeuLEJ MCIDq5Jsc1b9T8GAXkRm8ZivcXR13PMUqM6+U5FZsTLq2H0GveHGLpLJV7CEo0doNiEa IcNm+r6Q48sqVTmlDwZ96qU2GPoruuaEggcUXIYLrEJNgBNW+6C/2tEix697kRAoVfvz 3uygGj8ZqSRQQM6fGtSNY2uAVojTi/sK37KHTZ7AeuBrh/T51UkKqgtC3Y1frMWsotv/ QZsJygESB7pyBPFbFk9hmPmiOsiH32p3DvNCvY2L1+5vVDTzJyEZ9R/ue+lOOJgVl33A tr1w== X-Gm-Message-State: AKGB3mJ8xv223KRU2eYdMpcbYvVAG6NfYyjLfjKoWBBMKhvU2hebuwd0 UnUR2bkJbiQ9W5qK+m3NWKNqtcoQ X-Google-Smtp-Source: ACJfBovPwaVW556LZoEQaeD14hc8mtwXczhANc3qy8WQqnJDwzCQPbdGnoM2Pu00ouw8hioQpDEg0g== X-Received: by 10.101.101.65 with SMTP id a1mr3155682pgw.289.1515045731915; Wed, 03 Jan 2018 22:02:11 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:08 +0800 Message-Id: <1515045675-6993-10-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::230 Subject: [Qemu-devel] [PATCH RESEND V3 09/16] COLO: Flush memory data from ram cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang , Li Zhijian , Juan Quintela Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkp= oint time. So, the content of SVM's RAM cache will always be same with PVM's mem= ory after checkpoint. Instead of flushing all content of PVM's RAM cache into SVM's MEMORY, we do this in a more efficient way: Only flush any page that dirtied by PVM since last checkpoint. In this way, we can ensure SVM's memory same with PVM's. Besides, we must ensure flush RAM cache before load device state. Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 39 +++++++++++++++++++++++++++++++++++++++ migration/trace-events | 2 ++ 2 files changed, 41 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 388333d..23c67e0 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2908,6 +2908,40 @@ static bool postcopy_is_running(void) return ps >=3D POSTCOPY_INCOMING_LISTENING && ps < POSTCOPY_INCOMING_E= ND; } =20 +/* + * Flush content of RAM cache into SVM's memory. + * Only flush the pages that be dirtied by PVM or SVM or both. + */ +static void colo_flush_ram_cache(void) +{ + RAMBlock *block =3D NULL; + void *dst_host; + void *src_host; + unsigned long offset =3D 0; + + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); + rcu_read_lock(); + block =3D QLIST_FIRST_RCU(&ram_list.blocks); + + while (block) { + offset =3D migration_bitmap_find_dirty(ram_state, block, offset); + migration_bitmap_clear_dirty(ram_state, block, offset); + + if (offset << TARGET_PAGE_BITS >=3D block->used_length) { + offset =3D 0; + block =3D QLIST_NEXT_RCU(block, next); + } else { + dst_host =3D block->host + (offset << TARGET_PAGE_BITS); + src_host =3D block->colo_cache + (offset << TARGET_PAGE_BITS); + memcpy(dst_host, src_host, TARGET_PAGE_SIZE); + } + } + + rcu_read_unlock(); + trace_colo_flush_ram_cache_end(); + assert(ram_state->migration_dirty_pages =3D=3D 0); +} + static int ram_load(QEMUFile *f, void *opaque, int version_id) { int flags =3D 0, ret =3D 0, invalid_flags =3D 0; @@ -2920,6 +2954,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) bool postcopy_running =3D postcopy_is_running(); /* ADVISE is earlier, it shows the source has the postcopy capability = on */ bool postcopy_advised =3D postcopy_is_advised(); + bool need_flush =3D false; =20 seq_iter++; =20 @@ -3095,6 +3130,10 @@ static int ram_load(QEMUFile *f, void *opaque, int v= ersion_id) wait_for_decompress_done(); rcu_read_unlock(); trace_ram_load_complete(ret, seq_iter); + + if (!ret && migration_incoming_in_colo_state() && need_flush) { + colo_flush_ram_cache(); + } return ret; } =20 diff --git a/migration/trace-events b/migration/trace-events index d4738c8..69261a0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -78,6 +78,8 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRI= x64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset= : 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: st= art: 0x%zx len: 0x%zx" +colo_flush_ram_cache_begin(uint64_t dirty_pages) "dirty_pages %" PRIu64 +colo_flush_ram_cache_end(void) "" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046307008971.084471527455; Wed, 3 Jan 2018 22:11:47 -0800 (PST) Received: from localhost ([::1]:47503 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWykg-0003qG-6z for importer@patchew.org; Thu, 04 Jan 2018 01:11:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36998) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybc-0003rI-8r for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybZ-0004IX-5c for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:20 -0500 Received: from mail-pg0-x22c.google.com ([2607:f8b0:400e:c05::22c]:33498) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybY-0004I4-WF for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:17 -0500 Received: by mail-pg0-x22c.google.com with SMTP id i196so344207pgd.0 for ; Wed, 03 Jan 2018 22:02:16 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tBa4kAZZYbqvHRI7xCyuFTa4/HARGEGxBqo/NIzoUm4=; b=A6KEN00MvsPW7hs5RGGbAwt3i+5SyekOl6F7pOXlPSX5TasVnHfZRp08YYLvgBkHiu SAT+pvqHM0nHwO3cv1DfCzNkMLTsm9Um/RqJ62hcRBHGX56cQ4xY8kJykL2QYXMiMuTp 9f8cDij1XA1rLa2HEmf6X5yyDmUf9VGl7JX5Vde/op8sjZlMcueAW38zXbhmbwCFWgNf oFZO71c1cqHdgelLSdceSnVdN30IqmOCLuoDHIdkML8r+5Mpmis80Frpt+i9XwEqGp0j BvRBQu4LmBYfLqn2tO57sjIBXawd9QuRPV+BO7A+UDXRtQEM/5g8u+rloW1+DILGO1qC Wvsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tBa4kAZZYbqvHRI7xCyuFTa4/HARGEGxBqo/NIzoUm4=; b=DKr7R6l0sRtkJTdCHajj/RiH8FzK67rAsjH74HhAbxL4KEHxX4s9E/u0qq5SW6UPJN Te0qJiDKzBQGh75mUt1WImhfNVBOihWLW/Rp5HZGm9IX48Z/+BTIoUieewV6q8epVKjm 2nQkc5EAJinxAP6Qj5xdGyviotIDF8YX1FbX21Tr4/IIXILocBvzj8iALFFpn30HeK0R xVe/EmqSSk7/bbY/gnQdNCNM3SD4oKI2eNlfz3BWSVzboLXT7uNEz+7/CdvTZx5RdYLr WcRPIGReFcKujzEKk97QDzTUoNVZcnf9HVDUycNzxPUTi97lqozjJN6BPj3uGcCazuz+ dVww== X-Gm-Message-State: AKGB3mIYisBwZX0hf639kYaJbS3YqY1eSNuCjQatMMIKMLeHt9kSLRXk V2lHBGHjjiiBbfqXwGIkvbeJsvVG X-Google-Smtp-Source: ACJfBouKIoyz7bRocFUrkkI+tmv19yJUDjgpBrIIwGrU06HTU03p6xBGFcrJLXZdH7RYYypF/aosfA== X-Received: by 10.98.76.203 with SMTP id e72mr3662956pfj.91.1515045735645; Wed, 03 Jan 2018 22:02:15 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:09 +0800 Message-Id: <1515045675-6993-11-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::22c Subject: [Qemu-devel] [PATCH RESEND V3 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Zhang Chen , zhanghailiang , Li Zhijian , Markus Armbruster Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang If some errors happen during VM's COLO FT stage, it's important to notify the users of this event. Together with 'x_colo_lost_heartbeat', Users can intervene in COLO's failover work immediately. If users don't want to get involved in COLO's failover verdict, it is still necessary to notify users that we exited COLO mode. Cc: Markus Armbruster Cc: Michael Roth Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c | 19 +++++++++++++++++++ qapi-schema.json | 21 +++++++++++++++++++++ qapi/migration.json | 13 +++++++++++++ 3 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 8d2e3f8..790b122 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -516,6 +516,18 @@ out: qemu_fclose(fb); } =20 + /* + * There are only two reasons we can go here, some error happened. + * Or the user triggered failover. + */ + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_PRIMARY, + COLO_EXIT_REASON_REQUEST, NULL); + } + /* Hope this not to be too long to wait here */ qemu_sem_wait(&s->colo_exit_sem); qemu_sem_destroy(&s->colo_exit_sem); @@ -746,6 +758,13 @@ out: if (local_err) { error_report_err(local_err); } + if (failover_get_state() =3D=3D FAILOVER_STATUS_NONE) { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_ERROR, NULL); + } else { + qapi_event_send_colo_exit(COLO_MODE_SECONDARY, + COLO_EXIT_REASON_REQUEST, NULL); + } =20 if (fb) { qemu_fclose(fb); diff --git a/qapi-schema.json b/qapi-schema.json index 5c06745..4ff6d2c 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -2921,6 +2921,27 @@ { 'command': 'query-acpi-ospm-status', 'returns': ['ACPIOSTInfo'] } =20 ## +# @COLO_EXIT: +# +# Emitted when VM finishes COLO mode due to some errors happening or +# at the request of users. +# +# @mode: which COLO mode the VM was in when it exited. +# +# @reason: describes the reason for the COLO exit. +# +# Since: 2.11 +# +# Example: +# +# <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172}, +# "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "reques= t" } } +# +## +{ 'event': 'COLO_EXIT', + 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason' } } + +## # @ACPI_DEVICE_OST: # # Emitted when guest executes ACPI _OST method. diff --git a/qapi/migration.json b/qapi/migration.json index 03f57c9..f7b2cc6 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -854,6 +854,19 @@ ## { 'enum': 'FailoverStatus', 'data': [ 'none', 'require', 'active', 'completed', 'relaunch' ] } +## +# @COLOExitReason: +# +# The reason for a COLO exit +# +# @request: COLO exit is due to an external request +# +# @error: COLO exit is due to an internal error +# +# Since: 2.11 +## +{ 'enum': 'COLOExitReason', + 'data': [ 'request', 'error' ] } =20 ## # @x-colo-lost-heartbeat: --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046016911324.34352945723435; Wed, 3 Jan 2018 22:06:56 -0800 (PST) Received: from localhost ([::1]:47472 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyg3-0007bL-Tv for importer@patchew.org; Thu, 04 Jan 2018 01:06:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37021) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybe-0003tC-Aq for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybc-0004LL-U2 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:22 -0500 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:45903) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybc-0004KX-Ln for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:20 -0500 Received: by mail-pf0-x241.google.com with SMTP id u19so365692pfa.12 for ; Wed, 03 Jan 2018 22:02:20 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TGF+erUr4pYWWOWvWTJ8t3UqbJv69xPjEW0iO6u62+Y=; b=FawBJyyiVYECkl5+clDV2Ci0f0URyHQHEINGK3F4JOzUD8LzPAIDGSc05JN8Z10yS5 nNLGx94ohahHEducd4iha/GeBMDZjps/mEPYsH/DGIGTZzhdoJpve76iYHmStg9v//C3 EKhU0Zn0S6SyGhmC0+Br22/ogNdYl/cM1/ziFrgVAS5caMPzAFv9pnBZxPqTx5KDF/Rs sq+/Ysjlgz7zmxLDaWKZibT2fsgEdxOZrG/+7s50UudZiIGVGzDDhu1xJyKFLPBhJOoq VM8yoMZMAf1xh1ubpqDtXtkNAJbebLA9qgRfLuCOiphaLaXv0tx0r7OZ1GBtRjk2MO7U PKBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TGF+erUr4pYWWOWvWTJ8t3UqbJv69xPjEW0iO6u62+Y=; b=qX0e/vxXhFC0PxIm4rAbOcd3+De/h1CA6VTxDi8vUDxti+oWCbEzQ4M2s4SqNhroxe AVBqJajJI6J0XPxXBkP+GAvI6Z+VJctdOOvEIJrm94e8lk/Y/9SxR2Ri2BUQU/JjCz6w whJzx5/meBCgVsRLlekngUNB49/fFEZK1L7FNMDOfX5yIY10kTKerMOE8D0cldY7LeG0 LieV0V7zMqpRY4SeoFGpM0CwEkTHFr/gZQBWlKMijzMl9/aNsidpyn39ZjKXjY0wQW6F HwF9imp1TUlo8EmApmVdbEpoMtiyq7XklPmWCPMfBCjoL1VWCEb8/GpXtDxMHLTQhnD6 vy8Q== X-Gm-Message-State: AKGB3mKDTmGqjqqGxghNM+n9lXPi4QhsEnNwoI3g4/fN9jyNixtb+UFk ZGoOcDpj5uEfkRByenG9a+k46WGj X-Google-Smtp-Source: ACJfBosMbtPVnB7ZFa7IGUWiKw/0tcxLODNBPyXDBHNHrOduGiLLXODsfUdnHa1pimd03Dlvb2Ek+g== X-Received: by 10.99.164.73 with SMTP id c9mr3156751pgp.435.1515045739267; Wed, 03 Jan 2018 22:02:19 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:10 +0800 Message-Id: <1515045675-6993-12-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH RESEND V3 11/16] savevm: split the process of different stages for loadvm/savevm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Juan Quintela Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang There are several stages during loadvm/savevm process. In different stage, migration incoming processes different types of sections. We want to control these stages more accuracy, it will benefit COLO performance, we don't have to save type of QEMU_VM_SECTION_START sections everytime while do checkpoint, besides, we want to separate the process of saving/loading memory and devices state. So we add three new helper functions: qemu_load_device_state() and qemu_savevm_live_state() to achieve different process during migration. Besides, we make qemu_loadvm_state_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 37 +++++++++++++++++++++++++++++-------- migration/savevm.c | 35 ++++++++++++++++++++++++++++------- migration/savevm.h | 4 ++++ 3 files changed, 61 insertions(+), 15 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 790b122..a931ff2 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -29,6 +29,7 @@ #include "qapi-event.h" #include "block/block.h" #include "replication.h" +#include "sysemu/cpus.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -380,24 +381,31 @@ static int colo_do_checkpoint_transaction(MigrationSt= ate *s, =20 /* Disable block migration */ migrate_set_block_enabled(false, &local_err); - qemu_savevm_state_header(fb); - qemu_savevm_state_setup(fb); qemu_mutex_lock_iothread(); replication_do_checkpoint_all(&local_err); if (local_err) { qemu_mutex_unlock_iothread(); goto out; } - qemu_savevm_state_complete_precopy(fb, false, false); - qemu_mutex_unlock_iothread(); - - qemu_fflush(fb); =20 colo_send_message(s->to_dst_file, COLO_MESSAGE_VMSTATE_SEND, &local_er= r); if (local_err) { goto out; } /* + * Only save VM's live state, which not including device state. + * TODO: We may need a timeout mechanism to prevent COLO process + * to be blocked here. + */ + qemu_savevm_live_state(s->to_dst_file); + /* Note: device state is saved into buffer */ + ret =3D qemu_save_device_state(fb); + + qemu_mutex_unlock_iothread(); + + qemu_fflush(fb); + + /* * We need the size of the VMstate data in Secondary side, * With which we can decide how much data should be read. */ @@ -610,6 +618,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t total_size; uint64_t value; Error *local_err =3D NULL; + int ret; =20 qemu_sem_init(&mis->colo_incoming_sem, 0); =20 @@ -682,6 +691,16 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + qemu_mutex_lock_iothread(); + cpu_synchronize_all_pre_loadvm(); + ret =3D qemu_loadvm_state_main(mis->from_src_file, mis); + qemu_mutex_unlock_iothread(); + + if (ret < 0) { + error_report("Load VM's live state (ram) error"); + goto out; + } + value =3D colo_receive_message_value(mis->from_src_file, COLO_MESSAGE_VMSTATE_SIZE, &local_err); if (local_err) { @@ -715,8 +734,9 @@ void *colo_process_incoming_thread(void *opaque) qemu_mutex_lock_iothread(); qemu_system_reset(SHUTDOWN_CAUSE_NONE); vmstate_loading =3D true; - if (qemu_loadvm_state(fb) < 0) { - error_report("COLO: loadvm failed"); + ret =3D qemu_load_device_state(fb); + if (ret < 0) { + error_report("COLO: load device state failed"); qemu_mutex_unlock_iothread(); goto out; } @@ -777,6 +797,7 @@ out: if (mis->to_src_file) { qemu_fclose(mis->to_src_file); } + qemu_loadvm_state_cleanup(); migration_incoming_disable_colo(); =20 return NULL; diff --git a/migration/savevm.c b/migration/savevm.c index c582716..30a3c77 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1317,13 +1317,20 @@ done: return ret; } =20 -static int qemu_save_device_state(QEMUFile *f) +void qemu_savevm_live_state(QEMUFile *f) { - SaveStateEntry *se; + /* save QEMU_VM_SECTION_END section */ + qemu_savevm_state_complete_precopy(f, true, false); + qemu_put_byte(f, QEMU_VM_EOF); +} =20 - qemu_put_be32(f, QEMU_VM_FILE_MAGIC); - qemu_put_be32(f, QEMU_VM_FILE_VERSION); +int qemu_save_device_state(QEMUFile *f) +{ + SaveStateEntry *se; =20 + if (!migration_in_colo_state()) { + qemu_savevm_state_header(f); + } cpu_synchronize_all_states(); =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { @@ -1379,8 +1386,6 @@ enum LoadVMExitCodes { LOADVM_QUIT =3D 1, }; =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis= ); - /* ------ incoming postcopy messages ------ */ /* 'advise' arrives before any transfers just to tell us that a postcopy * *might* happen - it might be skipped if precopy transferred everything @@ -2003,7 +2008,7 @@ void qemu_loadvm_state_cleanup(void) } } =20 -static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis) { uint8_t section_type; int ret =3D 0; @@ -2148,6 +2153,22 @@ int qemu_loadvm_state(QEMUFile *f) return ret; } =20 +int qemu_load_device_state(QEMUFile *f) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + int ret; + + /* Load QEMU_VM_SECTION_FULL section */ + ret =3D qemu_loadvm_state_main(f, mis); + if (ret < 0) { + error_report("Failed to load device state: %d", ret); + return ret; + } + + cpu_synchronize_all_post_init(); + return 0; +} + int save_snapshot(const char *name, Error **errp) { BlockDriverState *bs, *bs1; diff --git a/migration/savevm.h b/migration/savevm.h index 041d23c..8d463fd 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -52,8 +52,12 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, = const char *name, uint64_t *start_list, uint64_t *length_list); void qemu_savevm_send_colo_enable(QEMUFile *f); +void qemu_savevm_live_state(QEMUFile *f); +int qemu_save_device_state(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); void qemu_loadvm_state_cleanup(void); +int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); +int qemu_load_device_state(QEMUFile *f); =20 #endif --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 151504639966211.644196414120529; Wed, 3 Jan 2018 22:13:19 -0800 (PST) Received: from localhost ([::1]:47509 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWymA-00058t-V8 for importer@patchew.org; Thu, 04 Jan 2018 01:13:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37071) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybh-0003wX-BG for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybg-0004OX-BJ for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:25 -0500 Received: from mail-pg0-x241.google.com ([2607:f8b0:400e:c05::241]:38755) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybg-0004Nh-5I for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:24 -0500 Received: by mail-pg0-x241.google.com with SMTP id t67so340181pgc.5 for ; Wed, 03 Jan 2018 22:02:23 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wP8rHc4oFD+B8t44XYNoggnsjnE/CVGhsI5u5C2EMxU=; b=kD0SP7CvisLof87Dsy1E4KYni8mOsOV86lR4M/uJ4k4whL7olw2RGy5j6iSj9lhrj5 AKoJQOJ9PZL/NI5rghFfaPSurLpMgcnx2oT61G54L1XHS39w2Qy8d3ayuYzXe4ikOHWv nyJzIe/C4k7RPxHTJoF/S3mQMi+roijVSbZhhlPxjXB2zCRAl2CP/AAQxIkUB0ht2/Gy Kf55Hx8P5GLwVv/05v1QtSlMcmQIxQ97I77Lf1XuXhwppss9kY2/dDTyMvUIT6dCXIwp IaxPhuS0hAyt9qtlIRzxC5BxFkle5Xgvwni2hf7392xIparpNx7cuMobc2Yj35urhhy5 c2Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wP8rHc4oFD+B8t44XYNoggnsjnE/CVGhsI5u5C2EMxU=; b=gkYqRFA3GXYMKM+fQahiMp1kex3Z4VPiw1XsZBqmE4K8SfjMZwgklYQLD0WStyBKgu wz+K/EA2ZXkNb0UBug/6uppYrfdciPvfJw54Bc6xAqgAjuh8ywR5dY5bd3JFNv5nC/fL Z0D77Mc+hKdzrBwGBFA2O9pJZGqK66gLfr0fzJhiB7ryZa234x9gY3LZIMokJOxzYUSK UxBDrff7B5tkJKmcqPVVQO2NKCMRrPWkw9aAIDGSbWnwnvQiM657OB+BaXJ2zEtJk7x/ P08wbDJHevwHuM2idC3yQvWJ6J4XSOw9r/3EGHRneDz0T++KU8dbEVLI+wDEYewlCATL 9qPw== X-Gm-Message-State: AKGB3mInR4L3OcrbwLUlIWZ71HkIW1QPYZdUnb3owOCEMWi75bakepzJ VsrLji9+GPM2K+Vx7w/a4KmyBboZ X-Google-Smtp-Source: ACJfBov3cUqY7snVN2Mpb6zMrdBdr54rQBV74z4mPipNHuBpplUWp+m8+ymzQFPSI7GmG1ddxX/J8w== X-Received: by 10.99.121.207 with SMTP id u198mr3198649pgc.32.1515045742857; Wed, 03 Jan 2018 22:02:22 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:11 +0800 Message-Id: <1515045675-6993-13-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::241 Subject: [Qemu-devel] [PATCH RESEND V3 12/16] COLO: flush host dirty ram from cache X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhang Chen , zhanghailiang , Li Zhijian , Juan Quintela Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Cc: Juan Quintela Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 23c67e0..0188712 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2679,6 +2679,7 @@ int colo_init_ram_cache(void) } ram_state =3D g_new0(RAMState, 1); ram_state->migration_dirty_pages =3D 0; + memory_global_dirty_log_start(); =20 return 0; =20 @@ -2699,10 +2700,12 @@ void colo_release_ram_cache(void) { RAMBlock *block; =20 + memory_global_dirty_log_stop(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { g_free(block->bmap); block->bmap =3D NULL; } + rcu_read_lock(); QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { if (block->colo_cache) { @@ -2919,6 +2922,15 @@ static void colo_flush_ram_cache(void) void *src_host; unsigned long offset =3D 0; =20 + memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); + rcu_read_lock(); + RAMBLOCK_FOREACH(block) { + migration_bitmap_sync_range(ram_state, block, 0, block->used_lengt= h); + } + rcu_read_unlock(); + qemu_mutex_unlock(&ram_state->bitmap_mutex); + trace_colo_flush_ram_cache_begin(ram_state->migration_dirty_pages); rcu_read_lock(); block =3D QLIST_FIRST_RCU(&ram_list.blocks); --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046165770666.5166644976287; Wed, 3 Jan 2018 22:09:25 -0800 (PST) Received: from localhost ([::1]:47485 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyiP-0001lo-Qs for importer@patchew.org; Thu, 04 Jan 2018 01:09:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37137) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybm-00041e-GR for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybj-0004Ra-5E for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:30 -0500 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:38756) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybj-0004R1-0k for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:27 -0500 Received: by mail-pg0-x242.google.com with SMTP id t67so340224pgc.5 for ; Wed, 03 Jan 2018 22:02:26 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dNmSKhTDLMTyVxdEvuJCjtTojXvAAXhnbn7P5Alaofc=; b=dONUvBP5Vz4Qifc2ofkK25wruNzB9INNlWemXTx0i9cE2Vaot0BgQ39bROCsYz1e5u sWIKH9oStTf+1YqC8Pc8rsANzeFwUYengjpzFriFHVmeDxjl7R1uyIe0jHCPqk/9R6kT /lU0o0UlniiDURgsL3nY29O/8xotj6u8notkwZ7QTCxLtH3oszplpG/bszqxpQXq0toP yStbkQBK7dPMZqazzlKH0VxOYh1WGUc5yrBIO7ZtPM9rD2ANfw1MAgahlXysAGelI8gv jED8GWSu7STjEuxV6CriYTZMjOB/Tx6aT03rL7sIAXHuDl9mdxZ4nrd+Dtpslu55YpsF X09w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dNmSKhTDLMTyVxdEvuJCjtTojXvAAXhnbn7P5Alaofc=; b=qYlOiWr8iYuoA8bwBAgVg0d9cZkQ0EMsSmEly+8Sjcb7/07ngIpx5lNNxph9pnVQyI Ia845dSKKsV7pXkzD1zbCHLatmTF1+cVFWpIrYjTnWjCI4dr6I1odYMjTM9pL8Qx8+wA DB6DbTvmkU0g2y6rDRsHkLZ2lCWBj8lCnwtP8RC/R9AYkzwMzEQUVWBm2X+BbODnJvrD d+Z11eFIZPqCcdTtnGl3eNVvCK8a+6TY3u1aYECi/pMMHLUkMZDeLetRTHKy4ywgaJmf KXMYGSsG1ZIqFucGdRwauE0FrgqTsJTbjhlRS38qqEtCSI2/mC9hgWMLgJWW7wXdine3 A53w== X-Gm-Message-State: AKGB3mLoUNSVWpUUfuIzN9lH3pEiuKSgLApAfvN+xSujjoVeUjp7SUM+ 5/Btt3bOubc88radXgNWnBB94/dU X-Google-Smtp-Source: ACJfBoveSFy3jfh24ZBTHgSYdio/cUx71MBEcVtHi611FvlfgYzgjD+bxlr2mLaDCL0mMRaRXgxbIQ== X-Received: by 10.99.4.145 with SMTP id 139mr3134863pge.367.1515045745748; Wed, 03 Jan 2018 22:02:25 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:12 +0800 Message-Id: <1515045675-6993-14-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH RESEND V3 13/16] filter: Add handle_event method for NetFilterClass X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Cc: Jason Wang Signed-off-by: zhanghailiang --- include/net/filter.h | 5 +++++ net/filter.c | 17 +++++++++++++++++ net/net.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/include/net/filter.h b/include/net/filter.h index 0c4a2ea..df4510d 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -37,6 +37,8 @@ typedef ssize_t (FilterReceiveIOV)(NetFilterState *nc, =20 typedef void (FilterStatusChanged) (NetFilterState *nf, Error **errp); =20 +typedef void (FilterHandleEvent) (NetFilterState *nf, int event, Error **e= rrp); + typedef struct NetFilterClass { ObjectClass parent_class; =20 @@ -44,6 +46,7 @@ typedef struct NetFilterClass { FilterSetup *setup; FilterCleanup *cleanup; FilterStatusChanged *status_changed; + FilterHandleEvent *handle_event; /* mandatory */ FilterReceiveIOV *receive_iov; } NetFilterClass; @@ -76,4 +79,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sende= r, int iovcnt, void *opaque); =20 +void colo_notify_filters_event(int event, Error **errp); + #endif /* QEMU_NET_FILTER_H */ diff --git a/net/filter.c b/net/filter.c index 2fd7d7d..0f17eba 100644 --- a/net/filter.c +++ b/net/filter.c @@ -17,6 +17,8 @@ #include "net/vhost_net.h" #include "qom/object_interfaces.h" #include "qemu/iov.h" +#include "net/colo.h" +#include "migration/colo.h" =20 static inline bool qemu_can_skip_netfilter(NetFilterState *nf) { @@ -245,11 +247,26 @@ static void netfilter_finalize(Object *obj) g_free(nf->netdev_id); } =20 +static void dummy_handle_event(NetFilterState *nf, int event, Error **errp) +{ + switch (event) { + case COLO_EVENT_CHECKPOINT: + break; + case COLO_EVENT_FAILOVER: + object_property_set_str(OBJECT(nf), "off", "status", errp); + break; + default: + break; + } +} + static void netfilter_class_init(ObjectClass *oc, void *data) { UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(oc); + NetFilterClass *nfc =3D NETFILTER_CLASS(oc); =20 ucc->complete =3D netfilter_complete; + nfc->handle_event =3D dummy_handle_event; } =20 static const TypeInfo netfilter_info =3D { diff --git a/net/net.c b/net/net.c index 39ef546..babbfd4 100644 --- a/net/net.c +++ b/net/net.c @@ -1399,6 +1399,34 @@ void hmp_info_network(Monitor *mon, const QDict *qdi= ct) } } =20 +void colo_notify_filters_event(int event, Error **errp) +{ + NetClientState *nc, *peer; + NetClientDriver type; + NetFilterState *nf; + NetFilterClass *nfc =3D NULL; + Error *local_err =3D NULL; + + QTAILQ_FOREACH(nc, &net_clients, next) { + peer =3D nc->peer; + type =3D nc->info->type; + if (!peer || type !=3D NET_CLIENT_DRIVER_TAP) { + continue; + } + QTAILQ_FOREACH(nf, &nc->filters, next) { + nfc =3D NETFILTER_GET_CLASS(OBJECT(nf)); + if (!nfc->handle_event) { + continue; + } + nfc->handle_event(nf, event, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } +} + void qmp_set_link(const char *name, bool up, Error **errp) { NetClientState *ncs[MAX_QUEUE_NUM]; --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046465699598.5288202480843; Wed, 3 Jan 2018 22:14:25 -0800 (PST) Received: from localhost ([::1]:47514 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWynE-0005s2-MZ for importer@patchew.org; Thu, 04 Jan 2018 01:14:20 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37160) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybn-000433-Bc for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybm-0004Tl-9P for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:31 -0500 Received: from mail-pf0-x236.google.com ([2607:f8b0:400e:c00::236]:44826) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybm-0004T7-2y for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:30 -0500 Received: by mail-pf0-x236.google.com with SMTP id m26so367467pfj.11 for ; Wed, 03 Jan 2018 22:02:29 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=M02kKhFXyP39YEvXPP+NgdqkZcy5c2++jhMiomAjDl8=; b=lJStB1rUStGsdpcTWwgO6l8uSVp9/zGaeZcimxBKcmN+uC6ToTTbLoJZ6e9f3n6sIN 8oMbqD58jlBXXe2OkmPZ4CCpa66+c+TaCf/bjBvWsgs8Pvk0on8QIw2gHFWuxWxYDa/U uHDCnGnmTssUeqm4qr6q4puRGintldS56sMpL9kvGgAlGGlMGic1RDzVRpN2RUm7srN6 YTu9ZKRxVZVmXK2+Y4BU/seqA0X0BLAc52nFNAVOP6NzPxaih9KKB06tF7yX8MfhyA+U wI0Dt6QUHgqVsSQBMclZvykUNksaX6llEx37G2QHHU0Bp2buPVJBr2MZfzDJl71h2GFz frKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=M02kKhFXyP39YEvXPP+NgdqkZcy5c2++jhMiomAjDl8=; b=DMygjmm3cMeTlYbka9ygUkdh6e57ZLHRNdNpRODHz+bgirywbygKDmPxVn0aFzosqN I8DcplB/50HU9dPvVXZ2FmAi1GAgRMjFsyFphDtzbId6HRw1ght3KE0i7sWGNLbUWY2s 2bQxxomaBzEFfEfx315oemqNYq5xh3t5/1MsDkXNxHhSxoh7I+6oZnp7uFhBOmiYiECF pSygYNnjnmkCWiO9rnxUMkDqTmcGvFBX05wBgSjw9XhZJD1e68DXtr+qXJtEtg/f42Wz gvuba1wNCTIb1KXOVbE5z18jiT7cqE4YnV0FgF/WLW0wMrbWofX/uPDTUDgjNcFD8hHz KCEQ== X-Gm-Message-State: AKGB3mIHz9cj4k/rt916IjXOHDywY2kZedzOekeJSJnnyEA/WSRrnMH4 GL0vyNMLoCaqBLbs6HFyV9Ihq3D7 X-Google-Smtp-Source: ACJfBosYsR/yZYAIr32Ndc8W/y6APSOO115Qm5cPXLmhruei8KxZZXgMVGLod0cTQ7sb6V9jY7iJWw== X-Received: by 10.99.4.21 with SMTP id 21mr3117478pge.72.1515045748787; Wed, 03 Jan 2018 22:02:28 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:13 +0800 Message-Id: <1515045675-6993-15-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::236 Subject: [Qemu-devel] [PATCH RESEND V3 14/16] filter-rewriter: handle checkpoint and failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang , Zhang Chen Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" After one round of checkpoint, the states between PVM and SVM become consistent, so it is unnecessary to adjust the sequence of net packets for old connections, besides, while failover happens, filter-rewriter needs to check if it still needs to adjust sequence of net packets. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 +++++++++++++ net/filter-rewriter.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index a931ff2..9eab4a3 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -30,6 +30,7 @@ #include "block/block.h" #include "replication.h" #include "sysemu/cpus.h" +#include "net/filter.h" =20 static bool vmstate_loading; static Notifier packets_compare_notifier; @@ -81,6 +82,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -753,6 +759,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c index a58310a..bd4b6cf 100644 --- a/net/filter-rewriter.c +++ b/net/filter-rewriter.c @@ -23,6 +23,8 @@ #include "qemu/main-loop.h" #include "qemu/iov.h" #include "net/checksum.h" +#include "net/colo.h" +#include "migration/colo.h" =20 #define FILTER_COLO_REWRITER(obj) \ OBJECT_CHECK(RewriterState, (obj), TYPE_FILTER_REWRITER) @@ -280,6 +282,43 @@ static ssize_t colo_rewriter_receive_iov(NetFilterStat= e *nf, return 0; } =20 +static void reset_seq_offset(gpointer key, gpointer value, gpointer user_d= ata) +{ + Connection *conn =3D (Connection *)value; + + conn->offset =3D 0; +} + +static gboolean offset_is_nonzero(gpointer key, + gpointer value, + gpointer user_data) +{ + Connection *conn =3D (Connection *)value; + + return conn->offset ? true : false; +} + +static void colo_rewriter_handle_event(NetFilterState *nf, int event, + Error **errp) +{ + RewriterState *rs =3D FILTER_COLO_REWRITER(nf); + + switch (event) { + case COLO_EVENT_CHECKPOINT: + g_hash_table_foreach(rs->connection_track_table, + reset_seq_offset, NULL); + break; + case COLO_EVENT_FAILOVER: + if (!g_hash_table_find(rs->connection_track_table, + offset_is_nonzero, NULL)) { + object_property_set_str(OBJECT(nf), "off", "status", errp); + } + break; + default: + break; + } +} + static void colo_rewriter_cleanup(NetFilterState *nf) { RewriterState *s =3D FILTER_COLO_REWRITER(nf); @@ -335,6 +374,7 @@ static void colo_rewriter_class_init(ObjectClass *oc, v= oid *data) nfc->setup =3D colo_rewriter_setup; nfc->cleanup =3D colo_rewriter_cleanup; nfc->receive_iov =3D colo_rewriter_receive_iov; + nfc->handle_event =3D colo_rewriter_handle_event; } =20 static const TypeInfo colo_rewriter_info =3D { --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1515046354272741.0060417455543; Wed, 3 Jan 2018 22:12:34 -0800 (PST) Received: from localhost ([::1]:47508 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWylV-0004Rp-Hg for importer@patchew.org; Thu, 04 Jan 2018 01:12:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37202) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybq-00046q-1M for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybp-0004WW-5l for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:34 -0500 Received: from mail-pl0-x243.google.com ([2607:f8b0:400e:c01::243]:39500) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybp-0004Vm-0r for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:33 -0500 Received: by mail-pl0-x243.google.com with SMTP id bi12so512225plb.6 for ; Wed, 03 Jan 2018 22:02:32 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0F7ZsDCObUtJ4lonLzJ/iStY50bir8SUcn4PW5uN+xE=; b=RB/YUe0J+Yay/0D42SwZUE5yeFfD1OvMy5wRG1VM7Ql9J0xA6oYMZf8dEFBmaMeQzP eCcgVgZlX4t4ioHS74iEQ50apq0j4sYGdm3XLi4V6ztleBURQg9x3L3tqN7X0o8O/FZo gHGJgWKoh8qD9dClh/HS8mO+Q9MtL1gW74tSBdlyZVw2gUioPSiPTXO21EQk0v8DjpkG VsCrVkFhSP6SRz3M0KFf3Tku8LJ2VruTu8udg+IG7sq20xqQ5pWkzij/Z6zKpt1CrpQx RHLzg2i0ELi3hBwQoqi1n2NifNX/C2G6TtTkk0d39Ao13kj2rAXuNz/GvTC0HcvaFDF+ 4xfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0F7ZsDCObUtJ4lonLzJ/iStY50bir8SUcn4PW5uN+xE=; b=Yjl1OlIHPAoTVLDnJNhq3NkvMnLQS3387V/qpA8TemzUWn8cYd6HPyS37yJCniJRW/ 8lXYeruRF4koeQW9aWoU9HxX7/gbST4R2AA2Ywd1JYBfC+gNlnTLCRXB61v6Kf2oO0Nw Ge3GtRFt0scnrhtxPqPGuHZ56iZOYkDUwE558P2CeswSpTirklO2B90RzkfkoMcxJANK YV054d74QjooLiJu2YfNheYKbs9yhXz0xi9ltYM92fd+Ii9mrsUrRFHbk8YE1iK+WGYH GttejdYfTKO62hTi+kScPyRubyQLiS8bt7SNSbUFHs/Yeel7JBxynGa48M02dq2qIXsj zGbg== X-Gm-Message-State: AKGB3mJFDvnyXy1QMf2WA8KiQ+m3cJe9GR8aOnejG8J6k6mdi8/Mgnu/ SpdtbLxAKZCTFP93zfpqHcbF0XG/ X-Google-Smtp-Source: ACJfBotJAIqwO5q4TajLu3zjlN3OKEznhLRWZFowA0SQ9piSOVn22LQUL+gF0HUNocDrRCtOW/MQgQ== X-Received: by 10.84.210.66 with SMTP id z60mr3652526plh.359.1515045751843; Wed, 03 Jan 2018 22:02:31 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:14 +0800 Message-Id: <1515045675-6993-16-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::243 Subject: [Qemu-devel] [PATCH RESEND V3 15/16] COLO: notify net filters about checkpoint/failover event X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , zhanghailiang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang Notify all net filters about the checkpoint and failover event. Cc: Jason Wang Signed-off-by: zhanghailiang --- migration/colo.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 9eab4a3..10bc80c 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -87,6 +87,11 @@ static void secondary_vm_do_failover(void) if (local_err) { error_report_err(local_err); } + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_FAILOVER, &local_err); + if (local_err) { + error_report_err(local_err); + } =20 if (!autostart) { error_report("\"-S\" qemu option will be ignored in secondary side= "); @@ -766,6 +771,13 @@ void *colo_process_incoming_thread(void *opaque) goto out; } =20 + /* Notify all filters of all NIC to do checkpoint */ + colo_notify_filters_event(COLO_EVENT_CHECKPOINT, &local_err); + if (local_err) { + qemu_mutex_unlock_iothread(); + goto out; + } + vmstate_loading =3D false; vm_start(); trace_colo_vm_state_change("stop", "run"); --=20 2.7.4 From nobody Tue Oct 28 01:55:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 151504653576159.78719973092643; Wed, 3 Jan 2018 22:15:35 -0800 (PST) Received: from localhost ([::1]:47524 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWyoM-0006Xj-1i for importer@patchew.org; Thu, 04 Jan 2018 01:15:30 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37242) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eWybs-00048R-OF for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eWybr-0004YQ-Sq for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:36 -0500 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:35592) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eWybr-0004Xy-N4 for qemu-devel@nongnu.org; Thu, 04 Jan 2018 01:02:35 -0500 Received: by mail-pf0-x241.google.com with SMTP id j124so375294pfc.2 for ; Wed, 03 Jan 2018 22:02:35 -0800 (PST) Received: from localhost.localdomain (120.236.201.35.bc.googleusercontent.com. [35.201.236.120]) by smtp.gmail.com with ESMTPSA id w5sm6214775pfi.74.2018.01.03.22.02.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jan 2018 22:02:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ykiojIxsoXQaDwibIw/YhxXHvoNepz2NylvpnOYiDqc=; b=EA9nf+mh5mc63oTMMNK7PCIMYRPUr2gcRGuyrBFE+UxWI1cDC9lriSNTByjhYDYxjB q7qKqDVacRyl1ij6gm33K66IzEf16XJMSzwBl9qEuuRWa07EceJOWfNUANrgV+jfYgGW XDxdebsMIF1kUhXJQpl3mGd56mW8HgOtK11RcsW/WROgHlSFhezr5vn98aeiG13vEFV/ P+Wnuo90bw/4I8fLllin4IQBEhHTheG7G2hkbip8da3tr/AnSyIrfO1aG8v6AA/MAPnS GrPa780TrTp6x4jd/EdMAXEPzKMKdRB/jdnwa5F66F1cNBWe3kDiziR2GtjQMkxZYaYL 1/0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ykiojIxsoXQaDwibIw/YhxXHvoNepz2NylvpnOYiDqc=; b=Ald0D7i+T9EmjGjDXyYT+QHjcgdPaVEcjh7zjLYTbXUT2/sAES3PqWnUq4B3l7Bo73 n54WqFEn0wQnkHkOwTu0pgTXdwAO2PQWahcgWa4nW46NqUoF9pGcJ+t6Ac7i4bac2VUq JAgqacM1qtbC5gaFnoDP17YgMMCVLLFjJ9ygZrKjeXqmNKkJigpsSxPnMZwyiNLuHgzm duQ/8iG6h3vwpwcUpEGUS88IQxOlKVx8boQu+qtxqU+toOB9AR0NZs8IE2OvBDB4oWT6 qY9q5ERe7OcBZigM15OkIkdLEg9qzZZIFr6eZUWGEkkxsRcRM4nRC1uCL9dsU1BNOjTx H5jw== X-Gm-Message-State: AKGB3mL/dvgUAYwbpfy5p0f8EjjKMCNwigB0gXnnxv6M4yZm+P5LCl5Y WzlR/yxw3iUPbTn3qor8FhuYpYx0 X-Google-Smtp-Source: ACJfBou83fnkDvTiFlI7IHIb0CVTfdTDc6A6EYkkqsaQZ+h/016vZTYPPP3GFVQMyKMHLqH4kc5+gQ== X-Received: by 10.99.141.75 with SMTP id z72mr3125525pgd.290.1515045754436; Wed, 03 Jan 2018 22:02:34 -0800 (PST) From: Zhang Chen To: qemu devel Date: Thu, 4 Jan 2018 14:01:15 +0800 Message-Id: <1515045675-6993-17-git-send-email-zhangckid@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> References: <1515045675-6993-1-git-send-email-zhangckid@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH RESEND V3 16/16] COLO: quick failover process by kick COLO thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: zhanghailiang Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 10bc80c..cc616d9 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -134,6 +134,11 @@ static void primary_vm_do_failover(void) =20 migrate_set_state(&s->state, MIGRATION_STATUS_COLO, MIGRATION_STATUS_COMPLETED); + /* + * kick COLO thread which might wait at + * qemu_sem_wait(&s->colo_checkpoint_sem). + */ + colo_checkpoint_notify(migrate_get_current()); =20 /* * Wake up COLO thread which may blocked in recv() or send(), @@ -519,6 +524,9 @@ static void colo_process_checkpoint(MigrationState *s) =20 qemu_sem_wait(&s->colo_checkpoint_sem); =20 + if (s->state !=3D MIGRATION_STATUS_COLO) { + goto out; + } ret =3D colo_do_checkpoint_transaction(s, bioc, fb); if (ret < 0) { goto out; --=20 2.7.4