From nobody Wed May 8 23:51:04 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1568110835; cv=none; d=zoho.com; s=zohoarc; b=XR0mQjgY1S1Ze5KbXAayyTSplk5zqF7SQrngepOaaIToNWsvvYKTuWhALmCPrXYwitth/n81EJ6iPp9+gLHFFfLCqQU2dlSzvOuH9d0ozBdAN2oF9IwalqQlbLFPnE+Z6VMyE4ySjNyS3Vd1vJFtX6nFrSnBbnoCVk5H36GTQso= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568110835; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=StrSknx6n99IGCCLmE47Oz4F/g14X+eup5MCfJZbsrw=; b=ZeXJKbCCx2U03VM3WPP5qsEcWDZ6xag+cu5YkUjhc4EeL9YNGwVVDVH0+it0GNnb7lWYCTpYnqvpDFbjaFZq1s3mD18BNxpx8v1NHop0uWIlyy1BOMQxgQJvCarWyWlGcUxBNOjCTdfzGFcnGOr2qv6iqlyT4+rn+4lTWbFaYCs= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568110835745370.70173281758525; Tue, 10 Sep 2019 03:20:35 -0700 (PDT) Received: from localhost ([::1]:37432 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dGE-0005Vj-En for importer@patchew.org; Tue, 10 Sep 2019 06:20:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55094) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dB0-0001wj-Rg for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7dAv-0005Oh-Qk for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:10 -0400 Received: from mout.web.de ([212.227.17.11]:51701) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7dAv-0005Ku-ID for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:05 -0400 Received: from luklap ([88.130.61.217]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MS2D8-1himnN2JHG-00TAUZ; Tue, 10 Sep 2019 12:14:39 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1568110479; bh=1aMK3WcfXGtXkTJdGeILldP4LKkJ6RJ1Yf8zxuMXcY0=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=QrqcmwTUG4rlAj8ghY2MUPr/eOPbY8JYJxzw40NQU4wUtqjmNbJfrfItXTATkSZX5 HfVk/QJpJjTd6Ul7n9GTdEox9uzPLOjrw3ryrFFZ2D1qVP5VFsYz6bYdVrDOMvTYQK VIapRYKIBFpCPRPTCyAdJcWNazHSYqJ9K+8piDUY= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Date: Tue, 10 Sep 2019 12:14:35 +0200 From: Lukas Straub To: qemu-devel Message-ID: <5305f30b0f798adfc3883bbcebbfc5030661dbe9.1568110100.git.lukasstraub2@web.de> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:THYE//mNaPjYgZsIjrcjlhxOyQIUjAtvpI2E2zYA5I2YQhLiaKW 0PA/LaYK7vpFITiuhegf7vn3Z8+zAz8U5iPLyGJ+I0ZrcbiiZ+ZbgeJSDBRyJJDBkWkhXKI rYHFTwx92d7TLM6L9ILzp2RPmL14W3YjLGEGQvK9jeq1u08vPoDHRg6D3qYLg9lKmzVOoxI +1A+NSUZ9E5uKqVfeWo8Q== X-UI-Out-Filterresults: notjunk:1;V03:K0:/M2mnrYdZjw=:tHfmilzwusskYKehAav9gO LeSElDBUBK+7yBLQ4zxShvflfVUO60R5Nv4rJSvQuwtAf7zmZIXjOYC/mMp6ADhVJ/RuGWANf uPVcIgT8T/bwldqBfHec2edATAkuViUvxlM5YY52cm8r6BhOtbV6LdFGT2tLV19rEmbwtNffy 548Wx0S16jl7exuh38C7X+YfhKLUlle+bGYurkb9x54x8Kx8xftn0e/dLK4V7fu4TYPEtpExB waMm4IDnVrI20d1KyFuy1e1P9OKDiDL0GhRUZxxUVIkmsZSAvuSqg3oRbQJLEJxoX+nAEiUmi vjeLO8Xhp528HZ71NEExpF8NRriU/mZi3lFCrpmOoT7WMHcH76kbYnPaABnLD8pvfBDvxpLzH nwU1w4V7FqGh5AnTKyPnJ2nANZSWUYznMWrkDfNJzSqHkk3GA4qoDzrjbNiLXCyjTqC1AAZdV oNfFeWOw8hX8ZjRykFheWbw21ksppk0br48DRRAdin+JVYx2vR/JrJhm1wqlo44IMjjPgd3PU k52nmUlqb9887G86HKUJ//4qHV7rdv+CZMU3QeSXCcAZlnu1IWS1raZ2QPW4kvHiN0MtrogYr fgf6LAHHYHqS+nG+8j6khyKF4hwRFsEu39QaFEVCQ8OSo6tcR+B2MyBCW+9zpofchGYkSTrQh jQpj8HF9s7fEZivEyTyuNGLaKb0e22iwUiLuWTv2aEJ2jZZnkNsegx3HEeBKk/08PhviAcuil gvJc+3Y1Lih6WvbhV0+TPQ9G8opFB+hwLeseD4XxgGQIZz/sv3ismd+EeuwH0l7oLLnn9+z4q wL0tsiQSoEJnfac76PuLaPjk7M3yHN58GQlOVtnhG3gLuTegOmxJta5b8joghzAv2nNWl8y1U NbBne18/XpI8bLycEXLaN84HAxkSl+SeqPiiIK+tMaAq+FViEa9zTBqvHbv4ztu5vdwFccMO7 RgqdY4XiUwuVeCuOBGc1wcthlEewKZ0UXhuc/aQzRTlknkq+3LAx+yFT9VbjOIR/n1bAwr8aC j9RgtIuGXohV+CkdUbeL3/pSJoj+aJRocSwf6vzuxpolvbhZQE4BYuN1nB6tapZbYVygO914i M0gRXyEi5hmIG7DYTFDw94Pe/5pNzHk8CCdRqkbT14Y63xxkb7bllrV7D5EHGNzwLtqBSIIh4 Jk+NGClnk5MSz61PyMJ4ixnGEK+sj7uFnXK2xRWUntnwpFtWvF0H05nP2okfv5HMCyZmy9yQk TvEYdfog3ZN9NexQy X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.17.11 Subject: [Qemu-devel] [PATCH v4 1/4] block/replication.c: Ignore requests after failover X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Wen Congyang , Jason Wang , mreitz@redhat.com, Zhang Chen , Xie Changlong Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" After failover the Secondary side of replication shouldn't change state, be= cause it now functions as our primary disk. In replication_start, replication_do_checkpoint, replication_stop, ignore the request if current state is BLOCK_REPLICATION_DONE (sucessful failover)= or BLOCK_REPLICATION_FAILOVER (failover in progres i.e. currently merging acti= ve and hidden images into the base image). Signed-off-by: Lukas Straub --- block/replication.c | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/block/replication.c b/block/replication.c index 3d4dedddfc..97cc65c0cf 100644 --- a/block/replication.c +++ b/block/replication.c @@ -454,6 +454,17 @@ static void replication_start(ReplicationState *rs, Re= plicationMode mode, aio_context_acquire(aio_context); s =3D bs->opaque; + if (s->stage =3D=3D BLOCK_REPLICATION_DONE || + s->stage =3D=3D BLOCK_REPLICATION_FAILOVER) { + /* + * This case happens when a secondary is promoted to primary. + * Ignore the request because the secondary side of replication + * doesn't have to do anything anymore. + */ + aio_context_release(aio_context); + return; + } + if (s->stage !=3D BLOCK_REPLICATION_NONE) { error_setg(errp, "Block replication is running or done"); aio_context_release(aio_context); @@ -529,8 +540,7 @@ static void replication_start(ReplicationState *rs, Rep= licationMode mode, "Block device is in use by internal backup job"); top_bs =3D bdrv_lookup_bs(s->top_id, s->top_id, NULL); - if (!top_bs || !bdrv_is_root_node(top_bs) || - !check_top_bs(top_bs, bs)) { + if (!top_bs || !check_top_bs(top_bs, bs)) { error_setg(errp, "No top_bs or it is invalid"); reopen_backing_file(bs, false, NULL); aio_context_release(aio_context); @@ -577,6 +587,17 @@ static void replication_do_checkpoint(ReplicationState= *rs, Error **errp) aio_context_acquire(aio_context); s =3D bs->opaque; + if (s->stage =3D=3D BLOCK_REPLICATION_DONE || + s->stage =3D=3D BLOCK_REPLICATION_FAILOVER) { + /* + * This case happens when a secondary was promoted to primary. + * Ignore the request because the secondary side of replication + * doesn't have to do anything anymore. + */ + aio_context_release(aio_context); + return; + } + if (s->mode =3D=3D REPLICATION_MODE_SECONDARY) { secondary_do_checkpoint(s, errp); } @@ -593,7 +614,7 @@ static void replication_get_error(ReplicationState *rs,= Error **errp) aio_context_acquire(aio_context); s =3D bs->opaque; - if (s->stage !=3D BLOCK_REPLICATION_RUNNING) { + if (s->stage =3D=3D BLOCK_REPLICATION_NONE) { error_setg(errp, "Block replication is not running"); aio_context_release(aio_context); return; @@ -635,6 +656,17 @@ static void replication_stop(ReplicationState *rs, boo= l failover, Error **errp) aio_context_acquire(aio_context); s =3D bs->opaque; + if (s->stage =3D=3D BLOCK_REPLICATION_DONE || + s->stage =3D=3D BLOCK_REPLICATION_FAILOVER) { + /* + * This case happens when a secondary was promoted to primary. + * Ignore the request because the secondary side of replication + * doesn't have to do anything anymore. + */ + aio_context_release(aio_context); + return; + } + if (s->stage !=3D BLOCK_REPLICATION_RUNNING) { error_setg(errp, "Block replication is not running"); aio_context_release(aio_context); -- 2.20.1 From nobody Wed May 8 23:51:04 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1568110754; cv=none; d=zoho.com; s=zohoarc; b=AivChs0cvITMhoWEenAfEHtz+pvY7HGYQytlkdfq1xvTZAop2HdlxlMSo0tAVXZV8H1U3auezjmcKyG+lP2+r7/HOg39SlOYCa42UwhkrvgV6mjBXWGEdSClaC6y0V9uL2zVQwWldnocNeLP3ak1iFeoD0yNZtSnnhdxID4IYqw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568110754; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=ooFJ8Q34MhVkAe27pC/DfVn86Njemz5NDYjUtWG0kfk=; b=IReVV+zWyh3EmgqFRmA/1Y9ao/evaNdyWjEOzv1glYUlXWukXzGaewE27ZXBGAYwxGS0MDYElaT6Zi2mXrRM+xFjdiSopocyqczF+xcaaCOzj3E5S9vpZqZ6HeHXNoo+m7FswqNG/IRuuKdhjLdF0Gnm4r8K0ATE26JbQoG+0M0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568110753974562.3433131261286; Tue, 10 Sep 2019 03:19:13 -0700 (PDT) Received: from localhost ([::1]:37424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dEu-0004FT-3j for importer@patchew.org; Tue, 10 Sep 2019 06:19:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55044) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dAt-0001qE-Pw for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7dAs-0005Mp-OB for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:03 -0400 Received: from mout.web.de ([212.227.17.11]:42137) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7dAs-0005L3-FF for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:02 -0400 Received: from luklap ([88.130.61.217]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0M7bQ3-1iIwst12Jz-00xIB6; Tue, 10 Sep 2019 12:14:44 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1568110484; bh=lS0He1ZjyrE6labtN6cAGz+FTFA0a9A2XqfoJ6l9fU0=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=KmuhNGzSSrMt/P+mXLYvXYKmZbJRZEFJ3uQLEAK0vl6KfqpC8et6D5fYxYCboGlIB u10r2HsIjgPPolwEjQVoJHSfKBhVNXZIh471yKQWcItfbvEvVpaSQhdZkyT20fMT2G jsrV3g5jGDCsIKJYrbTF/iSJnWKgjdqTS9q9orJM= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Date: Tue, 10 Sep 2019 12:14:43 +0200 From: Lukas Straub To: qemu-devel Message-ID: In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:sruv98jCmnOTW47dHXGQy8xBKNBqgqYaV3jtF5ALZIWASbfo5w/ MCsusuWPSoEpnw1HNVOHUjzvsarON6tY331MfETg+Rt1GIUfFR2/Wvj1FViDbWmRobWFkMi fYfK4j658VTgf0Q8V9/WRogOZHleRtYmPKlE/RuJTIRpQ0+Goju+SLPDH6QNuQzgbjz/RQg nErt+bUXjecOcISIS3rlA== X-UI-Out-Filterresults: notjunk:1;V03:K0:5RDTty9+CSM=:2qmcC3/HMH4UBKbTw1AfIM rEp8VYC3KtN4Aphf9/frvu7wFR35MQDigzgcmfB9E5IUZH0BhrtZn2PtuyMmtE79sdrdcOezT IUA8yOCtJXaZPxAAzimVl+1/cCW46mX8S0xVQAIzlS0PmfuzBeZO99lB+CdfodPWnzsnCpvOv aaYyfocmENg11N+rikX94WG9T16Z/N4kVfIVdrTD9WZ/gnb16SCsivG98+sgfbrmFxyU9igz/ ecZoN4HiQFknrgYqNCMeMDEAX0+6jLE/vMTfsKFVwaBfhKXg7L98kR1SXy5LvrepMOw9RacCT rZUcp0QfDsKUnQr86WAdF3QIs+H9OXyLabY0RbHbEbRUglPyagi6ots5lGLb1uZnlFCIwNNrX zKr3+aWUffYtkhpdCptkj1UDdI6NZOPHrrh05rrFxgN/GL6eeknGnHD+x8qVuR+xuG3v1dW0p ZJGWoSbDFuqpQwJInAYOSxkQ5LsRLveeQr/lYUos7YQUGSgQfJ05CG86Jgyjwva29Av3Eb2HS Fk4zCErZNMpwjOWV70CR8TpaUY8dHu66EaxeSMtZDj0DhsgFdDfFjfJe/2l54quP+YFXe6nAl Po4GjzYF9AMnGII5tqDalsSFiJwmqTG9Tq3wxTNIQgs+8SQ7eIDDdMQWHIwyHy+U3XivJPr63 ReAjQksilsYyDToA/yDbuyJCEf9mufFgrTU4yMAVBBW+6lpe3BdFdMuByRRoMKgutXJmWF+sf +FYMB8k7Oszv6JCfnlKO6qqMfleNA2IV7XV3ycMYb3Ii+rdVi2+SrVW64Z31njDuCi2xWz5KR 6WWsg0w6YfGx7N34YOSkVDmoFXWnucsqvKIlkCVO3okYDBVDd1QB0iUALZLyrIw5A81jWaujl NkNtrQhxi7FpEFljIlD7J23+eAuY0GPYTSHf/q8SpFPvSzL8BenIKsKmS0CeH0lCuFUOmRjxA CgQDV3SJ8B6K36Lg1hymp9mIQoP98QriA3cRwNoJmqdtH6h8HZHYFGfAyK13hxwfbhdyzSruJ LQZpkAzQKZGWBlCl+84SgrwLUZ+CbxSkKXDsxarylSdX7k9l6Vuzn2U3yKUFLHYVl0BrnVj+e jhUFzzZRzR4V0O4aImvvp2SxrBj/htVHw31nvPoBv5PrQgYOqJGi7ipkTw/x/foI51NgSKYiu L8zFvg0cY2s5p1DY98ootyCr9kFTes3s2esr26VLv6Efx0ZEnX1J6oHSmCKIl0Kij7pymqzWl 45HR3OSxftZYhoVfV X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.17.11 Subject: [Qemu-devel] [PATCH v4 2/4] tests/test-replication.c: Add test for ignoring requests after failover X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Wen Congyang , Jason Wang , mreitz@redhat.com, Zhang Chen , Xie Changlong Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This simulates the case that happens when we resume COLO after failover. Signed-off-by: Lukas Straub --- tests/test-replication.c | 52 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/tests/test-replication.c b/tests/test-replication.c index f085d1993a..5addfc2227 100644 --- a/tests/test-replication.c +++ b/tests/test-replication.c @@ -489,6 +489,56 @@ static void test_secondary_stop(void) teardown_secondary(); } +static void test_secondary_failover_then_ignore_requests(void) +{ + BlockBackend *top_blk, *local_blk; + Error *local_err =3D NULL; + + top_blk =3D start_secondary(); + replication_start_all(REPLICATION_MODE_SECONDARY, &local_err); + g_assert(!local_err); + + /* write 0x22 to s_local_disk (IMG_SIZE / 2, IMG_SIZE) */ + local_blk =3D blk_by_name(S_LOCAL_DISK_ID); + test_blk_write(local_blk, 0x22, IMG_SIZE / 2, IMG_SIZE / 2, false); + + /* replication will backup s_local_disk to s_hidden_disk */ + test_blk_read(top_blk, 0x11, IMG_SIZE / 2, + IMG_SIZE / 2, 0, IMG_SIZE, false); + + /* write 0x33 to s_active_disk (0, IMG_SIZE / 2) */ + test_blk_write(top_blk, 0x33, 0, IMG_SIZE / 2, false); + + /* do failover (active commit) */ + replication_stop_all(true, &local_err); + g_assert(!local_err); + + /* it should ignore all requests from now on */ + + /* start after failover */ + replication_start_all(REPLICATION_MODE_PRIMARY, &local_err); + g_assert(!local_err); + + /* checkpoint */ + replication_do_checkpoint_all(&local_err); + g_assert(!local_err); + + /* stop */ + replication_stop_all(true, &local_err); + g_assert(!local_err); + + /* read from s_local_disk (0, IMG_SIZE / 2) */ + test_blk_read(top_blk, 0x33, 0, IMG_SIZE / 2, + 0, IMG_SIZE / 2, false); + + + /* read from s_local_disk (IMG_SIZE / 2, IMG_SIZE) */ + test_blk_read(top_blk, 0x22, IMG_SIZE / 2, + IMG_SIZE / 2, 0, IMG_SIZE, false); + + teardown_secondary(); +} + static void test_secondary_do_checkpoint(void) { BlockBackend *top_blk, *local_blk; @@ -584,6 +634,8 @@ int main(int argc, char **argv) g_test_add_func("/replication/secondary/write", test_secondary_write); g_test_add_func("/replication/secondary/start", test_secondary_start); g_test_add_func("/replication/secondary/stop", test_secondary_stop); + g_test_add_func("/replication/secondary/failover_then_ignore_requests", + test_secondary_failover_then_ignore_requests); g_test_add_func("/replication/secondary/do_checkpoint", test_secondary_do_checkpoint); g_test_add_func("/replication/secondary/get_error_all", -- 2.20.1 From nobody Wed May 8 23:51:04 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1568111107; cv=none; d=zoho.com; s=zohoarc; b=jOlzDbedThZvU9D3kbqRPh2klDFFAjDHLjvjx+7+M67NstZS58OMMUqg9mMXo8FDgwD7k0wmPpgmhqEJObXPf9CCimfMbwEwyAjN2DwE1izS+OT9MEWEA+9hh/XLtpx/696POr4+VjuJHyi0uWwuElV7DyDLQEAAsaCQYl0vVf4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568111107; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=61S2jXCgBUaVylH/jCg6m0a4TG+knklJl9MWCijH0Uo=; b=PoDce0LLbGi479Fy5FrCC5lLQbMo1d6l9rpWHcCPCMroZSGkf3fB0/+4IWktuuVLIpqQEWdcAvB/ztmdBK5m7oJrZQCg0MfqmQtjG7Cq51bZTRxTUAnS1RQNRuWhqxv/IvZVhyO57AMg0QPBUZ8714NWBdCiPXYpOEvo2zuJdKQ= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568111107342728.2114825684758; Tue, 10 Sep 2019 03:25:07 -0700 (PDT) Received: from localhost ([::1]:37466 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dKZ-00005H-61 for importer@patchew.org; Tue, 10 Sep 2019 06:25:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55047) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dAu-0001qd-7Z for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7dAr-0005M5-QO for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:04 -0400 Received: from mout.web.de ([212.227.17.11]:55739) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7dAr-0005LX-DL for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:01 -0400 Received: from luklap ([88.130.61.217]) by smtp.web.de (mrweb101 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MVtxM-1hevB342Ei-00X1tu; Tue, 10 Sep 2019 12:14:50 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1568110490; bh=ky24J0BV9q7LhMECzyRhDm2e8abfWgxpXvC7+9DdEXk=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=SMPpITePJ/t51tWrlWyAEVc/7wi9s0JS79FuAlmXSsFYXgESI0ei1YlXvtT23s8Qs PobSGGf4wygwYyaRbqA2doGNKRPI1bjzhKkbXZMAHdMZ/dAwUwKXEBrs0a/iJAjgYK 0d7r9xhIlX0HLBtEymPFB16uJNjRA9bl28wViB54= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Date: Tue, 10 Sep 2019 12:14:49 +0200 From: Lukas Straub To: qemu-devel Message-ID: <77d2eb7d0ce8a1887a575119e21ce0a06d4af533.1568110100.git.lukasstraub2@web.de> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:YA46yQGpgFCIO4NSjWG2eniCmLRgMb93yjByJawZbin3qssseQZ 3qeQmQbg7egW9MezSMJylGadO9s4c2ykq6E3wzbpHkOtEruYwl8w5u9zdlPYdYkTMC6j8pX i5FSjKDDAtNLqrV2JbNeFHCbchShnJTUVnfy6RP0ICsZpBiNL9fTw7Rm9ZzfZUNCB/gSM95 UdOdm6lkRMBB+5OAlI0pw== X-UI-Out-Filterresults: notjunk:1;V03:K0:QeOsxHpznPQ=:KNfaovBqlIqaXjKEX8cWF1 FOyZ1FL9qmB7Dlfa7ovHESBfe0fWvTLRWFXgMPt4Rm2mOP04J9pxLPsa7OvVzq/rZH4cQAPtN NllpbUQ+bUaz2nacf0/cZUqvD+rC7vurLYoynag0foTs6A/B0mRT5eAkcNszz0xnHo5v3wejq QEfXVI/YaDftuQE9HCfbUq4zDrNoxhJI4fY+y924XnAhn/znA7+Tk0pWRtzVnV/HDspIqvBFT zQibLF+bEZ/qfIPf8Yhe+gyDFvqsvV62ssrMF9z3IA6/8fxEOQz1H/sJecncc6babfnPCJw4G bWp7oHUlznrYU82VJk22yG1w2y9Nc39kvubdc39DGyMRSZNzpV5ucjW4FibDrOkE9TzgbOLLg 9sBy0tN2/XaDc8xoWUgsP0bj4TAqXl3Dl9MHd3ZurIPmQFTJO+qKDenFSWhD/iNeSNbiXIYqH krWqTIbtZNiUvZBWj4SX65aqA9q1fg1FGsUgsLkkUEftBfBTRFlb2JQfXtB21wVpYv31t4kzJ JxagRJMhILijdFjWrAHkmd8JUdeD06hl+kvY08eKPf6itQCE0aH2rOgAPu/jonxCcUSNhAOzs HeaG9+86HlK1jNjY1g0b+ZFm4qoxc199ElBuQrJirF3JIGsC3KfyEDC3U9KjFjkTnQHBRM27i 7HF8uviuIhKqKj78LnNMTWdCLl2jwSPmVBYo+1w+eVMoqWpCntBcGPDdK8DrBNBuM3oUH7Ful ARoQDQquuqao9r8lyS872CvHapSVzRRRH07W7t20Ck35xxztRbvBajHGZ2LoTG6tK0cnITUIR cgnDtEs9NOm/xuLhc0PZHtwNgmBy8KIDEvs2mQs1wITWEpwyQ9zMOITlMfFD1oeb2GTR2Gg8w NGIzoOzwmGAnUSGNJfifQAYuiekikWEbEAGBf7sR/GMaIBJbBAynAzbA6y82VY0J5oghwfNJz MoAocUTL+9MS1bIIWWtYBLgEYluHAbP4PUB6s8vcy+mouipWRiRz35LkpkSBQTLmWIQh8wrTI Ll/cKGkBe3gkTA1D8krrekqv7YbnddcIAy5Q6n8Tyf19cJLYFNMbKhG0FjcrIclXLGbTeG9F4 NcDu0aYmhVtNcX/0tw9482OXxDFnwlvwexVh8Cf2QSeEu2uRuwJfN1eMWcFOmF/D5tAL/MvW7 d+KwzJhyWYvpWvpiX13Q+6UrCyyyn3tPasepj9ZeitzHby+Ewdg0nLbF9rlby+eVx+sB81ItB MVTi37u83EaR1O6Lr X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.17.11 Subject: [Qemu-devel] [PATCH v4 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Wen Congyang , Jason Wang , mreitz@redhat.com, Zhang Chen , Xie Changlong Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" To switch the Secondary to Primary, we need to insert new filters before the filter-rewriter. Add the options insert=3D and position=3D to be able to insert filters anywhere in the filter list. position should be either "head", "tail" or the id of another filter. insert should be either "before" or "behind" to specify where to insert the new filter relative to the one specified with position. Signed-off-by: Lukas Straub --- include/net/filter.h | 2 ++ net/filter.c | 78 +++++++++++++++++++++++++++++++++++++++++++- qemu-options.hx | 10 +++--- 3 files changed, 84 insertions(+), 6 deletions(-) diff --git a/include/net/filter.h b/include/net/filter.h index 49da666ac0..22a723305b 100644 --- a/include/net/filter.h +++ b/include/net/filter.h @@ -62,6 +62,8 @@ struct NetFilterState { NetClientState *netdev; NetFilterDirection direction; bool on; + char *position; + bool insert_before_flag; QTAILQ_ENTRY(NetFilterState) next; }; diff --git a/net/filter.c b/net/filter.c index 28d1930db7..eb0e9849a5 100644 --- a/net/filter.c +++ b/net/filter.c @@ -171,11 +171,47 @@ static void netfilter_set_status(Object *obj, const c= har *str, Error **errp) } } +static char *netfilter_get_position(Object *obj, Error **errp) +{ + NetFilterState *nf =3D NETFILTER(obj); + + return g_strdup(nf->position); +} + +static void netfilter_set_position(Object *obj, const char *str, Error **e= rrp) +{ + NetFilterState *nf =3D NETFILTER(obj); + + nf->position =3D g_strdup(str); +} + +static char *netfilter_get_insert(Object *obj, Error **errp) +{ + NetFilterState *nf =3D NETFILTER(obj); + + return nf->insert_before_flag ? g_strdup("before") : g_strdup("behind"= ); +} + +static void netfilter_set_insert(Object *obj, const char *str, Error **err= p) +{ + NetFilterState *nf =3D NETFILTER(obj); + + if (strcmp(str, "before") && strcmp(str, "behind")) { + error_setg(errp, "Invalid value for netfilter insert, " + "should be 'before' or 'behind'"); + return; + } + + nf->insert_before_flag =3D !strcmp(str, "before"); +} + static void netfilter_init(Object *obj) { NetFilterState *nf =3D NETFILTER(obj); nf->on =3D true; + nf->insert_before_flag =3D false; + nf->position =3D g_strdup("tail"); object_property_add_str(obj, "netdev", netfilter_get_netdev_id, netfilter_set_netdev_= id, @@ -187,11 +223,18 @@ static void netfilter_init(Object *obj) object_property_add_str(obj, "status", netfilter_get_status, netfilter_set_status, NULL); + object_property_add_str(obj, "position", + netfilter_get_position, netfilter_set_position, + NULL); + object_property_add_str(obj, "insert", + netfilter_get_insert, netfilter_set_insert, + NULL); } static void netfilter_complete(UserCreatable *uc, Error **errp) { NetFilterState *nf =3D NETFILTER(uc); + NetFilterState *position =3D NULL; NetClientState *ncs[MAX_QUEUE_NUM]; NetFilterClass *nfc =3D NETFILTER_GET_CLASS(uc); int queues; @@ -219,6 +262,27 @@ static void netfilter_complete(UserCreatable *uc, Erro= r **errp) return; } + if (strcmp(nf->position, "head") && strcmp(nf->position, "tail")) { + /* Search for the position to insert before/behind */ + Object *container; + Object *obj; + + container =3D object_get_objects_root(); + obj =3D object_resolve_path_component(container, nf->position); + if (!obj) { + error_setg(errp, "filter '%s' not found", nf->position); + return; + } + + position =3D NETFILTER(obj); + + if (position->netdev !=3D ncs[0]) { + error_setg(errp, "filter '%s' belongs to a different netdev", + nf->position); + return; + } + } + nf->netdev =3D ncs[0]; if (nfc->setup) { @@ -228,7 +292,18 @@ static void netfilter_complete(UserCreatable *uc, Erro= r **errp) return; } } - QTAILQ_INSERT_TAIL(&nf->netdev->filters, nf, next); + + if (position) { + if (nf->insert_before_flag) { + QTAILQ_INSERT_BEFORE(position, nf, next); + } else { + QTAILQ_INSERT_AFTER(&nf->netdev->filters, position, nf, next); + } + } else if (!strcmp(nf->position, "head")) { + QTAILQ_INSERT_HEAD(&nf->netdev->filters, nf, next); + } else if (!strcmp(nf->position, "tail")) { + QTAILQ_INSERT_TAIL(&nf->netdev->filters, nf, next); + } } static void netfilter_finalize(Object *obj) @@ -245,6 +320,7 @@ static void netfilter_finalize(Object *obj) QTAILQ_REMOVE(&nf->netdev->filters, nf, next); } g_free(nf->netdev_id); + g_free(nf->position); } static void default_handle_event(NetFilterState *nf, int event, Error **er= rp) diff --git a/qemu-options.hx b/qemu-options.hx index 08749a3391..1fd294a10f 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -4368,7 +4368,7 @@ applications, they can do this through this parameter= . Its format is a gnutls priority string as described at @url{https://gnutls.org/manual/html_node/Priority-Strings.html}. -@item -object filter-buffer,id=3D@var{id},netdev=3D@var{netdevid},interval= =3D@var{t}[,queue=3D@var{all|rx|tx}][,status=3D@var{on|off}] +@item -object filter-buffer,id=3D@var{id},netdev=3D@var{netdevid},interval= =3D@var{t}[,queue=3D@var{all|rx|tx}][,status=3D@var{on|off}][,position=3D@v= ar{head|tail|id}][,insert=3D@var{behind|before}] Interval @var{t} can't be 0, this filter batches the packet delivery: all packets arriving in a given interval on netdev @var{netdevid} are delayed @@ -4387,11 +4387,11 @@ queue @var{all|rx|tx} is an option that can be appl= ied to any netfilter. @option{tx}: the filter is attached to the transmit queue of the netdev, where it will receive packets sent by the netdev. -@item -object filter-mirror,id=3D@var{id},netdev=3D@var{netdevid},outdev= =3D@var{chardevid},queue=3D@var{all|rx|tx}[,vnet_hdr_support] +@item -object filter-mirror,id=3D@var{id},netdev=3D@var{netdevid},outdev= =3D@var{chardevid},queue=3D@var{all|rx|tx}[,vnet_hdr_support][,position=3D@= var{head|tail|id}][,insert=3D@var{behind|before}] filter-mirror on netdev @var{netdevid},mirror net packet to chardev@var{ch= ardevid}, if it has the vnet_hdr_support flag, filter-mirror will mirror pa= cket with vnet_hdr_len. -@item -object filter-redirector,id=3D@var{id},netdev=3D@var{netdevid},inde= v=3D@var{chardevid},outdev=3D@var{chardevid},queue=3D@var{all|rx|tx}[,vnet_= hdr_support] +@item -object filter-redirector,id=3D@var{id},netdev=3D@var{netdevid},inde= v=3D@var{chardevid},outdev=3D@var{chardevid},queue=3D@var{all|rx|tx}[,vnet_= hdr_support][,position=3D@var{head|tail|id}][,insert=3D@var{behind|before}] filter-redirector on netdev @var{netdevid},redirect filter's net packet to= chardev @var{chardevid},and redirect indev's packet to filter.if it has the vnet_h= dr_support flag, @@ -4400,7 +4400,7 @@ Create a filter-redirector we need to differ outdev i= d from indev id, id can not be the same. we can just use indev or outdev, but at least one of indev or= outdev need to be specified. -@item -object filter-rewriter,id=3D@var{id},netdev=3D@var{netdevid},queue= =3D@var{all|rx|tx},[vnet_hdr_support] +@item -object filter-rewriter,id=3D@var{id},netdev=3D@var{netdevid},queue= =3D@var{all|rx|tx},[vnet_hdr_support][,position=3D@var{head|tail|id}][,inse= rt=3D@var{behind|before}] Filter-rewriter is a part of COLO project.It will rewrite tcp packet to secondary from primary to keep secondary tcp connection,and rewrite @@ -4413,7 +4413,7 @@ colo secondary: -object filter-redirector,id=3Df2,netdev=3Dhn0,queue=3Drx,outdev=3Dred1 -object filter-rewriter,id=3Drew0,netdev=3Dhn0,queue=3Dall -@item -object filter-dump,id=3D@var{id},netdev=3D@var{dev}[,file=3D@var{fi= lename}][,maxlen=3D@var{len}] +@item -object filter-dump,id=3D@var{id},netdev=3D@var{dev}[,file=3D@var{fi= lename}][,maxlen=3D@var{len}][,position=3D@var{head|tail|id}][,insert=3D@va= r{behind|before}] Dump the network traffic on netdev @var{dev} to the file specified by @var{filename}. At most @var{len} bytes (64k by default) per packet are st= ored. -- 2.20.1 From nobody Wed May 8 23:51:04 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1568110770; cv=none; d=zoho.com; s=zohoarc; b=EtOzUj1lEcMFX17SNx+R8K8XgM3GhGyborO2aIk1tCpsIGysqhloslQb7fSq4IkCZVIE9W8og8SOmdsFO+TmI2X3w3b+/DXU16Xbwq5+tO45KQEz8Bbm9axF2cmLMH6wunr4dQt2/3acRhRCxXx5dljBLUeLni/Qm5qRzKLBPyI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1568110770; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=pOooIMe94lHToVfzF/RjnC1Fq8fHMU5zDTiA1OfsrUk=; b=JwKJx698fU2WsCALizZYnqK3/2MEM1UXK61ac3jQ/rFGHpCiHcjVqBUpf9Cu/cOiXoRg0n2eyNdJcPwJEu36gqTsEJNYb88LVS+4XYNxGAHsVX+LYMYnEsAgsL33ARhAMxrfbbc6mQRvJMyo5Yztlg/Yo4FZCAqezkuigEAVDJI= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1568110770582957.9509376703691; Tue, 10 Sep 2019 03:19:30 -0700 (PDT) Received: from localhost ([::1]:37426 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dFB-0004R9-6d for importer@patchew.org; Tue, 10 Sep 2019 06:19:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55127) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i7dBA-00028d-UR for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i7dB8-0005eq-GF for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:20 -0400 Received: from mout.web.de ([212.227.17.12]:41255) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i7dB8-0005eS-74 for qemu-devel@nongnu.org; Tue, 10 Sep 2019 06:15:18 -0400 Received: from luklap ([88.130.61.217]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MBkHT-1hwrf331Tr-00ApBf; Tue, 10 Sep 2019 12:14:58 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1568110499; bh=Togv/A1WXNfzWSNATE9I0D+leIHST8xdWWCXoK2b0u0=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=gYwf8Bzp4OzeVJl6sWL2R5stw/wNJecpgBNE0Nesc/FGCaCrkWUdgIJa0UoD8sZC6 yTVGrP/YYafvP+34xFwWG3mhGxarFzwt4xaoPQAx4UDMKN17uHk+YZUFSRwMApCGMc J/VpiNI5XxTiLAPrBEvrvr0b7Pp7HlcRtZgj6hw4= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Date: Tue, 10 Sep 2019 12:14:58 +0200 From: Lukas Straub To: qemu-devel Message-ID: <6d1ae9171a46b633fd03a9f6c520da189372034a.1568110100.git.lukasstraub2@web.de> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:loG2b7js2o+phrnXmAXwwMQGd37r0h7w2HXG84p/1D9XpAgod4n lgyusjlJscxngFGn+jPnLH6nfs6+9TqHo7KUiTODo11rdrgzgMy9BwYsBzEuNcJFeAkM4/L 8Vu69DNutZ2flaMwwmIdN4h5sxTtqavX7AZ4TBhV12RV3DQjxII4mhI4/ii95o/iCCqxjZp N4OTfGkv4qSjUQHcNWpSQ== X-UI-Out-Filterresults: notjunk:1;V03:K0:7TB3FpBR4WA=:yF+ZY/GLEPqWPJWblzHf9+ L6XNr3gJqXiBmYhaqvbmxSINq5xlVZKpr8okKgUv0W5q02EVqmykjSYGmEm3F2QXTxDK0SqdE a9uCPFSMeYY7jCb259v4HVO8ef7CzJBZOeEubZX92Irhj9iEUmmBy3d8L601P5eQt+UDElgWX b56RTlf2WW/gTBy2t3CFSby/VwTdjAaoOE8bo3bn+QvRYKYNY1/1qRSjCJ3Z2pntE5Q8To/XO Tf7bw/6K2ilMU/38mxHiCYI3jy0U46AOtYl8V41AapBlNaUCIC1k5l8igXVwJWNrw0gpmOLWL 5wRURj0BfsbPSkRoSWRAYcdrV5LOqxGS50u92rZlbZaSPEOIRWYxUuaKB6H1AIcbkndH0xnui 7xWfb1CQF9DuSyS+CFNzM1wd+uJAARlb1g4+EUTkmkHeeSlUfPTi9MeIBZOJW2P/ECvHXePCr 2lHaIi7A7nCJdD7UrLW+F3Hrqlu9Dgt7/B6MeaaS9B2reWrrRYX2eCCtyFyxWStij+++cT2ul b3+lTbH/3gXLIGT+4ZJRoVQKSE7ELG/oPlwJJJoezWbAfptjxMqn0Kmo3mvb3eknMD9ltyFIt Bf/jjP95i+jlzBv/pVCf71bXF46Sbyo2+joQMXWd2DMzh5nt7KvRLocezBXNBBsypY61lQTRE X8Ka1I8nponM9AI4oVOm/eWXqWjVv6OKepB81eylQXNDZN9s8lyP8I5YrxOTQlfo13gxdhx4N HU0//mgKFw/NcGBujk/gw0v06Wyb3LSds1XOTYFri24pWk4UG21gUFS0bdVSM1XsxmEMHOOmw 0C3d4SBhVcnkreecyIDse81idP45tqMi6qCrOtYLeheoaCsJSWXgrTctkuva+wPCxEFENEcql +rkJ3PPJzD7Y8v2M4UFOglg/Z7GjnFuhi4yQn+aZfn9j7Vj+YVzwN5UYXLo72yfKUuf9+M0bl C2gZ7Xaldy18R6Xc93a8AM6r8HLpOAFpVy5Y1yaSKbJ4s2Zo8G1xQhEFExGCN1TWTn7rmNYgN /C3Kr21oU5Au75Xk8o2rKQWpBatZ+lm2a8KHCrkPEPoTNd1pDXBMZEpq+ncWNxJcMVqNZ5TjW nCWzDmhnzJhSvIRzbq2N9yPR0+/SXSEMA/CvhyqwEWs0ddcBzDSUsZl4cXUqwDsm/2gsWZ6zl ltMz8jKJGAA8eoypPdql7Xd0MbLfIueUYREIpInRPXI548vW2oeQ8eWRN9vYfRwRUPoRbFhM4 BE6m5eRHEfbk4aBNy X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.17.12 Subject: [Qemu-devel] [PATCH v4 4/4] colo: Update Documentation for continious replication X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Wen Congyang , Jason Wang , mreitz@redhat.com, Zhang Chen , Xie Changlong Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Document the qemu command-line and qmp commands for continious replication Signed-off-by: Lukas Straub --- docs/COLO-FT.txt | 212 +++++++++++++++++++++++++++---------- docs/block-replication.txt | 26 +++-- 2 files changed, 172 insertions(+), 66 deletions(-) diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index ad24680d13..30dcb45d48 100644 --- a/docs/COLO-FT.txt +++ b/docs/COLO-FT.txt @@ -145,35 +145,65 @@ The diagram just shows the main qmp command, you can = get the detail in test procedure. =20 =3D=3D Test procedure =3D=3D -1. Startup qemu -Primary: -# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name primary \ - -device piix3-usb-uhci -vnc :7 \ - -device usb-tablet -netdev tap,id=3Dhn0,vhost=3Doff \ - -device virtio-net-pci,id=3Dnet-pci0,netdev=3Dhn0 \ - -drive if=3Dvirtio,id=3Dprimary-disk0,driver=3Dquorum,read-pattern=3Dfif= o,vote-threshold=3D1,\ - children.0.file.filename=3D1.raw,\ - children.0.driver=3Draw -S -Secondary: -# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name secondary \ - -device piix3-usb-uhci -vnc :7 \ - -device usb-tablet -netdev tap,id=3Dhn0,vhost=3Doff \ - -device virtio-net-pci,id=3Dnet-pci0,netdev=3Dhn0 \ - -drive if=3Dnone,id=3Dsecondary-disk0,file.filename=3D1.raw,driver=3Draw= ,node-name=3Dnode0 \ - -drive if=3Dvirtio,id=3Dactive-disk0,driver=3Dreplication,mode=3Dseconda= ry,\ - file.driver=3Dqcow2,top-id=3Dactive-disk0,\ - file.file.filename=3D/mnt/ramfs/active_disk.img,\ - file.backing.driver=3Dqcow2,\ - file.backing.file.filename=3D/mnt/ramfs/hidden_disk.img,\ - file.backing.backing=3Dsecondary-disk0 \ - -incoming tcp:0:8888 - -2. On Secondary VM's QEMU monitor, issue command +Note: Here we are running both instances on the same Machine for testing, +change the IP Addresses if you want to run it on two Hosts + +=3D=3D Startup qemu =3D=3D +1. Primary: +# imagefolder=3D"/mnt/vms/colo-test" + +The disks for the primary and secondary need to be the same before startin= g colo +# cp --reflink=3Dauto $imagefolder/primary.qcow2 $imagefolder/primary-copy= .qcow2 + +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp = stdio \ + -vnc :0 -device piix3-usb-uhci -device usb-tablet -name primary \ + -netdev tap,id=3Dhn0,vhost=3Doff,helper=3D/usr/lib/qemu/qemu-bridge-hel= per \ + -device rtl8139,id=3De0,netdev=3Dhn0 \ + -chardev socket,id=3Dmirror0,host=3D127.0.0.1,port=3D9003,server,nowait= \ + -chardev socket,id=3Dcompare1,host=3D127.0.0.1,port=3D9004,server,wait \ + -chardev socket,id=3Dcompare0,host=3D127.0.0.1,port=3D9001,server,nowai= t \ + -chardev socket,id=3Dcompare0-0,host=3D127.0.0.1,port=3D9001 \ + -chardev socket,id=3Dcompare_out,host=3D127.0.0.1,port=3D9005,server,no= wait \ + -chardev socket,id=3Dcompare_out0,host=3D127.0.0.1,port=3D9005 \ + -object filter-mirror,id=3Dm0,netdev=3Dhn0,queue=3Dtx,outdev=3Dmirror0 \ + -object filter-redirector,netdev=3Dhn0,id=3Dredire0,queue=3Drx,indev=3D= compare_out \ + -object filter-redirector,netdev=3Dhn0,id=3Dredire1,queue=3Drx,outdev= =3Dcompare0 \ + -object iothread,id=3Diothread1 \ + -object colo-compare,id=3Dcomp0,primary_in=3Dcompare0-0,secondary_in=3D= compare1,\ +outdev=3Dcompare_out0,iothread=3Diothread1 \ + -drive if=3Dide,id=3Dcolo-disk0,driver=3Dquorum,read-pattern=3Dfifo,vot= e-threshold=3D1,\ +children.0.file.filename=3D$imagefolder/primary.qcow2,children.0.driver=3D= qcow2 -S + +2. Secondary: +# imagefolder=3D"/mnt/vms/colo-test" + +# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G + +# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G + +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp = stdio \ + -vnc :1 -device piix3-usb-uhci -device usb-tablet -name secondary \ + -netdev tap,id=3Dhn0,vhost=3Doff,helper=3D/usr/lib/qemu/qemu-bridge-hel= per \ + -device rtl8139,id=3De0,netdev=3Dhn0 \ + -chardev socket,id=3Dred0,host=3D127.0.0.1,port=3D9003,reconnect=3D1 \ + -chardev socket,id=3Dred1,host=3D127.0.0.1,port=3D9004,reconnect=3D1 \ + -object filter-redirector,id=3Df1,netdev=3Dhn0,queue=3Dtx,indev=3Dred0 \ + -object filter-redirector,id=3Df2,netdev=3Dhn0,queue=3Drx,outdev=3Dred1= \ + -object filter-rewriter,id=3Drew0,netdev=3Dhn0,queue=3Dall \ + -drive if=3Dnone,id=3Dparent0,file.filename=3D$imagefolder/primary-copy= .qcow2,driver=3Dqcow2 \ + -drive if=3Dnone,id=3Dchilds0,driver=3Dreplication,mode=3Dsecondary,fil= e.driver=3Dqcow2,\ +top-id=3Dchilds0,file.file.filename=3D$imagefolder/secondary-active.qcow2,\ +file.backing.driver=3Dqcow2,file.backing.file.filename=3D$imagefolder/seco= ndary-hidden.qcow2,\ +file.backing.backing=3Dparent0 \ + -drive if=3Dide,id=3Dcolo-disk0,driver=3Dquorum,read-pattern=3Dfifo,vot= e-threshold=3D1,\ +children.0=3Dchilds0 \ + -incoming tcp:0:9998 + + +3. On Secondary VM's QEMU monitor, issue command {'execute':'qmp_capabilities'} -{ 'execute': 'nbd-server-start', - 'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', '= port': '8889'} } } -} -{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', '= writable': true } } +{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'da= ta': {'host': '127.0.0.1', 'port': '9999'} } } } +{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0', 'writable= ': true } } =20 Note: a. The qmp command nbd-server-start and nbd-server-add must be run @@ -182,44 +212,112 @@ Note: same. c. It is better to put active disk and hidden disk in ramdisk. =20 -3. On Primary VM's QEMU monitor, issue command: +4. On Primary VM's QEMU monitor, issue command: {'execute':'qmp_capabilities'} -{ 'execute': 'human-monitor-command', - 'arguments': {'command-line': 'drive_add -n buddy driver=3Dreplication,m= ode=3Dprimary,file.driver=3Dnbd,file.host=3Dxx.xx.xx.xx,file.port=3D8889,fi= le.export=3Dsecondary-disk0,node-name=3Dnbd_client0'}} -{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', '= node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities', - 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': tr= ue } ] } } -{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } } +{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_= add -n buddy driver=3Dreplication,mode=3Dprimary,file.driver=3Dnbd,file.hos= t=3D127.0.0.1,file.port=3D9999,file.export=3Dparent0,node-name=3Dreplicatio= n0'}} +{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'nod= e': 'replication0' } } +{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'= capability': 'x-colo', 'state': true } ] } } +{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } } =20 Note: a. There should be only one NBD Client for each primary disk. - b. xx.xx.xx.xx is the secondary physical machine's hostname or IP - c. The qmp command line must be run after running qmp command line in + b. The qmp command line must be run after running qmp command line in secondary qemu. =20 -4. After the above steps, you will see, whenever you make changes to PVM, = SVM will be synced. +5. After the above steps, you will see, whenever you make changes to PVM, = SVM will be synced. You can issue command '{ "execute": "migrate-set-parameters" , "arguments"= :{ "x-checkpoint-delay": 2000 } }' -to change the checkpoint period time +to change the idle checkpoint period time + +6. Failover test +You can kill one of the VMs and Failover on the surviving VM: + +If you killed the Secondary, then follow "Primary Failover". After that, +if you want to resume the replication, follow "Primary resume replication" + +If you killed the Primary, then follow "Secondary Failover". After that, +if you want to resume the replication, follow "Secondary resume replicatio= n" + +=3D=3D Primary Failover =3D=3D +The Secondary died, resume on the Primary + +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'ch= ild': 'children.1'} } +{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_= del replication0' } } +{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } } +{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } } +{'execute': 'object-del', 'arguments':{ 'id': 'm0' } } +{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } } +{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } } +{'execute': 'x-colo-lost-heartbeat' } + +=3D=3D Secondary Failover =3D=3D +The Primary died, resume on the Secondary and prepare to become the new Pr= imary + +{'execute': 'nbd-server-stop'} +{'execute': 'x-colo-lost-heartbeat'} + +{'execute': 'object-del', 'arguments':{ 'id': 'f2' } } +{'execute': 'object-del', 'arguments':{ 'id': 'f1' } } +{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } } +{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } } + +{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend': {'typ= e': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0= .1', 'port': '9003' } }, 'server': true } } } } +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend': {'ty= pe': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.= 0.1', 'port': '9004' } }, 'server': true } } } } +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend': {'ty= pe': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.= 0.1', 'port': '9001' } }, 'server': true } } } } +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend': {'= type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.= 0.0.1', 'port': '9001' } }, 'server': false } } } } +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out', 'backend': {= 'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127= .0.0.1', 'port': '9005' } }, 'server': true } } } } +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0', 'backend': = {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '12= 7.0.0.1', 'port': '9005' } }, 'server': false } } } } + +=3D=3D Primary resume replication =3D=3D +Resume replication after new Secondary is up. + +Start the new Secondary (Steps 2 and 3 above), then: +{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id'= : 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', '= format': 'raw', 'sync': 'full'} } + +Wait until disk is synced, then: +{'execute': 'stop'} +{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} } + +{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_= add -n buddy driver=3Dreplication,mode=3Dprimary,file.driver=3Dnbd,file.hos= t=3D127.0.0.1,file.port=3D9999,file.export=3Dparent0,node-name=3Dreplicatio= n0'}} +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'no= de': 'replication0' } } + +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id':= 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', '= id': 'redire0', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compar= e_out' } } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', '= id': 'redire1', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compa= re0' } } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iot= hread1' } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': = 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1',= 'outdev': 'compare_out0', 'iothread': 'iothread1' } } } + +{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'= capability': 'x-colo', 'state': true } ] } } +{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } } + +Note: +If this Primary previously was a Secondary, then we need to insert the +filters before the filter-rewriter by using the +"'insert': 'before', 'position': 'rew0'" Options. See below. + +=3D=3D Secondary resume replication =3D=3D +Become Primary and resume replication after new Secondary is up. + +Start the new Secondary (Steps 2 and 3 above, but with different image than +primary-copy.qcow2 and different VNC port), then: +{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id'= : 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', '= format': 'raw', 'sync': 'full'} } + +Wait until disk is synced, then: +{'execute': 'stop'} +{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } } =20 -5. Failover test -You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's -monitor at the same time, then SVM will failover and client will not detec= t this -change. +{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_= add -n buddy driver=3Dreplication,mode=3Dprimary,file.driver=3Dnbd,file.hos= t=3D127.0.0.1,file.port=3D9999,file.export=3Dparent0,node-name=3Dreplicatio= n0'}} +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'no= de': 'replication0' } } =20 -Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to -issue block related command to stop block replication. -Primary: - Remove the nbd child from the quorum: - { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', = 'child': 'children.1'}} - { 'execute': 'human-monitor-command','arguments': {'command-line': 'driv= e_del blk-buddy0'}} - Note: there is no qmp command to remove the blockdev now +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id':= 'm0', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', = 'queue': 'tx', 'outdev': 'mirror0' } } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', '= id': 'redire0', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev'= : 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', '= id': 'redire1', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev'= : 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iot= hread1' } } +{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': = 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1',= 'outdev': 'compare_out0', 'iothread': 'iothread1' } } } =20 -Secondary: - The primary host is down, so we should do the following thing: - { 'execute': 'nbd-server-stop' } +{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'= capability': 'x-colo', 'state': true } ] } } +{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } } =20 =3D=3D TODO =3D=3D -1. Support continuous VM replication. -2. Support shared storage. -3. Develop the heartbeat part. -4. Reduce checkpoint VM=E2=80=99s downtime while doing checkpoint. +1. Support shared storage. +2. Develop the heartbeat part. +3. Reduce checkpoint VM=E2=80=99s downtime while doing checkpoint. diff --git a/docs/block-replication.txt b/docs/block-replication.txt index 6bde6737fb..0b46c8c768 100644 --- a/docs/block-replication.txt +++ b/docs/block-replication.txt @@ -65,12 +65,12 @@ blocks that are already in QEMU. ^ || .---------- | || | Secondary 1 Quorum || '---------- - / \ || - / \ || - Primary 2 filter - disk ^ = virtio-blk - | = ^ - 3 NBD -------> 3 NBD = | + / \ || = virtio-blk + / \ || = ^ + Primary 2 filter = | + disk ^ = 7 Quorum + | = / + 3 NBD -------> 3 NBD = / client || server = 2 filter || ^ = ^ --------. || | = | @@ -106,6 +106,10 @@ any state that would otherwise be lost by the speculat= ive write-through of the NBD server into the secondary disk. So before block replication, the primary disk and secondary disk should contain the same data. =20 +7) The secondary also has a quorum node, so after secondary failover it +can become the new primary and continiue replication. + + =3D=3D Failure Handling =3D=3D There are 7 internal errors when block replication is running: 1. I/O error on primary disk @@ -171,16 +175,18 @@ Primary: leading whitespace. 5. The qmp command line must be run after running qmp command line in secondary qemu. - 6. After failover we need remove children.1 (replication driver). + 6. After primary failover we need remove children.1 (replication driver). =20 Secondary: -drive if=3Dnone,driver=3Draw,file.filename=3D1.raw,id=3Dcolo1 \ - -drive if=3Dxxx,id=3Dtopxxx,driver=3Dreplication,mode=3Dsecondary,top-id= =3Dtopxxx\ + -drive if=3Dnone,id=3Dchilds1,driver=3Dreplication,mode=3Dsecondary,top-= id=3Dchilds1 file.file.filename=3Dactive_disk.qcow2,\ file.driver=3Dqcow2,\ file.backing.file.filename=3Dhidden_disk.qcow2,\ file.backing.driver=3Dqcow2,\ file.backing.backing=3Dcolo1 + -drive if=3Dxxx,driver=3Dquorum,read-pattern=3Dfifo,id=3Dtop-disk1,\ + vote-threshold=3D1,children.0=3Dchilds1 =20 Then run qmp command in secondary qemu: { 'execute': 'nbd-server-start', @@ -234,6 +240,8 @@ Secondary: The primary host is down, so we should do the following thing: { 'execute': 'nbd-server-stop' } =20 +Promote Secondary to Primary: + see COLO-FT.txt + TODO: -1. Continuous block replication 2. Shared disk --=20 2.20.1