include/migration/colo.h | 6 +- migration/migration.h | 52 +++++++-- migration/savevm.h | 5 +- hw/vfio/migration-multifd.c | 9 +- migration/channel.c | 7 +- migration/colo-stubs.c | 2 +- migration/colo.c | 23 +--- migration/migration.c | 62 ++++++++--- migration/ram.c | 13 +-- migration/rdma.c | 206 ++++++++---------------------------- migration/savevm.c | 85 +++++++-------- migration/trace-events | 4 +- 12 files changed, 196 insertions(+), 278 deletions(-)
[this is an early RFC, not for merge, but to collect initial feedbacks]
Background
==========
Nowadays, live migration heavily depends on threads. For example, most of
the major features that will be used nowadays in live migration (multifd,
postcopy, mapped-ram, vfio, etc.) all work with threads internally.
But still, from time to time, we'll see some coroutines floating around the
migration context. The major one is precopy's loadvm, which is internally
a coroutine. It is still a critical path that any live migration depends on.
A mixture of using both coroutines and threads is prone to issues. Some
examples can refer to commit e65cec5e5d ("migration/ram: Yield periodically
to the main loop") or commit 7afbdada7e ("migration/postcopy: ensure
preempt channel is ready before loading states").
Overview
========
This series tries to move migration further into the thread-based model, by
allowing the loadvm process to happen in a thread rather than in the main
thread with a coroutine.
Luckily, since the qio channel code is always ready for both cases, IO
paths should all be fine.
Note that loadvm for postcopy already happens in a ram load thread which is
separate. However, RAM is just the simple case here, even it has its own
challenges (on atomically update of the pgtables), its complexity lies in
the kernel.
For precopy, loadvm has quite a few operations that will need BQL. The
question is we can't take BQL for the whole process of loadvm, because
that'll block the main thread from executions (e.g. QMP hangs). Here, the
finer granule we can push BQL the better. This series so far chose
somewhere in the middle, by taking BQL on majorly these two places:
- CPU synchronizations
- Device START/FULL sections
After this series applied, most of the rest loadvm path will run without
BQL anymore. There is a more detailed discussion / todo in the commit
message of patch "migration: Thread-ify precopy vmstate load process"
explaning how to further split the BQL critical sections.
I was trying to split the patches into smaller ones if possible, but it's
still quite challenging so there's one major patch that does the work.
After the series applied, the only leftover pieces in migration/ that would
use a coroutine is snapshot save/load/delete jobs.
Tests
=====
Default CI passes.
RDMA unit tests pass as usual. I also tried out cancellation / failure
tests over RDMA channels, making sure nothing is stuck.
I also roughly measured how long it takes to run the whole 80+ migration
qtest suite, and see no measurable difference before / after this series.
Risks
=====
This series has the risk of breaking things. I would be surprised if it
didn't..
I confess I didn't test anything on COLO but only from code observations
and analysis. COLO maintainers: could you add some unit tests to QEMU's
qtests?
The current way of taking BQL during FULL section load may cause issues, it
means when the IOs are unstable we could be waiting for IO (in the new
migration incoming thread) with BQL held. This is low possibility, though,
only happens when the network halts during flushing the device states.
However still possible. One solution is to further breakdown the BQL
critical sections to smaller sections, as mentioned in TODO.
Anything more than welcomed: suggestions, questions, objections, tests..
Todo
====
- Test COLO?
- Finer grained BQL breakdown
- More..
Thanks,
Peter Xu (9):
migration/vfio: Remove BQL implication in
vfio_multifd_switchover_start()
migration/rdma: Fix wrong context in qio_channel_rdma_shutdown()
migration/rdma: Allow qemu_rdma_wait_comp_channel work with thread
migration/rdma: Change io_create_watch() to return immediately
migration: Thread-ify precopy vmstate load process
migration/rdma: Remove coroutine path in qemu_rdma_wait_comp_channel
migration/postcopy: Remove workaround on wait preempt channel
migration/ram: Remove workaround on ram yield during load
migration/rdma: Remove rdma_cm_poll_handler
include/migration/colo.h | 6 +-
migration/migration.h | 52 +++++++--
migration/savevm.h | 5 +-
hw/vfio/migration-multifd.c | 9 +-
migration/channel.c | 7 +-
migration/colo-stubs.c | 2 +-
migration/colo.c | 23 +---
migration/migration.c | 62 ++++++++---
migration/ram.c | 13 +--
migration/rdma.c | 206 ++++++++----------------------------
migration/savevm.c | 85 +++++++--------
migration/trace-events | 4 +-
12 files changed, 196 insertions(+), 278 deletions(-)
--
2.50.1
Peter Xu <peterx@redhat.com> writes:
> [this is an early RFC, not for merge, but to collect initial feedbacks]
>
> Background
> ==========
>
> Nowadays, live migration heavily depends on threads. For example, most of
> the major features that will be used nowadays in live migration (multifd,
> postcopy, mapped-ram, vfio, etc.) all work with threads internally.
>
> But still, from time to time, we'll see some coroutines floating around the
> migration context. The major one is precopy's loadvm, which is internally
> a coroutine. It is still a critical path that any live migration depends on.
>
I always wanted to be an archaeologist:
https://lists.gnu.org/archive/html/qemu-devel//2012-08/msg01136.html
I was expecting to find some complicated chain of events leading to the
choice of using a coroutine, but no.
> A mixture of using both coroutines and threads is prone to issues. Some
> examples can refer to commit e65cec5e5d ("migration/ram: Yield periodically
> to the main loop") or commit 7afbdada7e ("migration/postcopy: ensure
> preempt channel is ready before loading states").
>
> Overview
> ========
>
> This series tries to move migration further into the thread-based model, by
> allowing the loadvm process to happen in a thread rather than in the main
> thread with a coroutine.
>
> Luckily, since the qio channel code is always ready for both cases, IO
> paths should all be fine.
>
> Note that loadvm for postcopy already happens in a ram load thread which is
> separate. However, RAM is just the simple case here, even it has its own
> challenges (on atomically update of the pgtables), its complexity lies in
> the kernel.
>
> For precopy, loadvm has quite a few operations that will need BQL. The
> question is we can't take BQL for the whole process of loadvm, because
> that'll block the main thread from executions (e.g. QMP hangs). Here, the
> finer granule we can push BQL the better. This series so far chose
> somewhere in the middle, by taking BQL on majorly these two places:
>
> - CPU synchronizations
> - Device START/FULL sections
>
> After this series applied, most of the rest loadvm path will run without
> BQL anymore. There is a more detailed discussion / todo in the commit
> message of patch "migration: Thread-ify precopy vmstate load process"
> explaning how to further split the BQL critical sections.
>
> I was trying to split the patches into smaller ones if possible, but it's
> still quite challenging so there's one major patch that does the work.
>
> After the series applied, the only leftover pieces in migration/ that would
> use a coroutine is snapshot save/load/delete jobs.
>
Which are then fine because the work itself runs on the main loop,
right? So the bottom-half scheduling could be left as a coroutine.
> Tests
> =====
>
> Default CI passes.
>
> RDMA unit tests pass as usual. I also tried out cancellation / failure
> tests over RDMA channels, making sure nothing is stuck.
>
> I also roughly measured how long it takes to run the whole 80+ migration
> qtest suite, and see no measurable difference before / after this series.
>
> Risks
> =====
>
> This series has the risk of breaking things. I would be surprised if it
> didn't..
>
> I confess I didn't test anything on COLO but only from code observations
> and analysis. COLO maintainers: could you add some unit tests to QEMU's
> qtests?
>
> The current way of taking BQL during FULL section load may cause issues, it
> means when the IOs are unstable we could be waiting for IO (in the new
> migration incoming thread) with BQL held. This is low possibility, though,
> only happens when the network halts during flushing the device states.
> However still possible. One solution is to further breakdown the BQL
> critical sections to smaller sections, as mentioned in TODO.
>
> Anything more than welcomed: suggestions, questions, objections, tests..
>
> Todo
> ====
>
> - Test COLO?
> - Finer grained BQL breakdown
> - More..
>
> Thanks,
>
> Peter Xu (9):
> migration/vfio: Remove BQL implication in
> vfio_multifd_switchover_start()
> migration/rdma: Fix wrong context in qio_channel_rdma_shutdown()
> migration/rdma: Allow qemu_rdma_wait_comp_channel work with thread
> migration/rdma: Change io_create_watch() to return immediately
> migration: Thread-ify precopy vmstate load process
> migration/rdma: Remove coroutine path in qemu_rdma_wait_comp_channel
> migration/postcopy: Remove workaround on wait preempt channel
> migration/ram: Remove workaround on ram yield during load
> migration/rdma: Remove rdma_cm_poll_handler
>
> include/migration/colo.h | 6 +-
> migration/migration.h | 52 +++++++--
> migration/savevm.h | 5 +-
> hw/vfio/migration-multifd.c | 9 +-
> migration/channel.c | 7 +-
> migration/colo-stubs.c | 2 +-
> migration/colo.c | 23 +---
> migration/migration.c | 62 ++++++++---
> migration/ram.c | 13 +--
> migration/rdma.c | 206 ++++++++----------------------------
> migration/savevm.c | 85 +++++++--------
> migration/trace-events | 4 +-
> 12 files changed, 196 insertions(+), 278 deletions(-)
On Tue, Sep 16, 2025 at 06:32:59PM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
>
> > [this is an early RFC, not for merge, but to collect initial feedbacks]
> >
> > Background
> > ==========
> >
> > Nowadays, live migration heavily depends on threads. For example, most of
> > the major features that will be used nowadays in live migration (multifd,
> > postcopy, mapped-ram, vfio, etc.) all work with threads internally.
> >
> > But still, from time to time, we'll see some coroutines floating around the
> > migration context. The major one is precopy's loadvm, which is internally
> > a coroutine. It is still a critical path that any live migration depends on.
> >
>
> I always wanted to be an archaeologist:
>
> https://lists.gnu.org/archive/html/qemu-devel//2012-08/msg01136.html
>
> I was expecting to find some complicated chain of events leading to the
> choice of using a coroutine, but no.
I actually didn't see that previously.. I'll add this link into that major
patch commit message, to make future archaeology work easier.
>
> > A mixture of using both coroutines and threads is prone to issues. Some
> > examples can refer to commit e65cec5e5d ("migration/ram: Yield periodically
> > to the main loop") or commit 7afbdada7e ("migration/postcopy: ensure
> > preempt channel is ready before loading states").
> >
> > Overview
> > ========
> >
> > This series tries to move migration further into the thread-based model, by
> > allowing the loadvm process to happen in a thread rather than in the main
> > thread with a coroutine.
> >
> > Luckily, since the qio channel code is always ready for both cases, IO
> > paths should all be fine.
> >
> > Note that loadvm for postcopy already happens in a ram load thread which is
> > separate. However, RAM is just the simple case here, even it has its own
> > challenges (on atomically update of the pgtables), its complexity lies in
> > the kernel.
> >
> > For precopy, loadvm has quite a few operations that will need BQL. The
> > question is we can't take BQL for the whole process of loadvm, because
> > that'll block the main thread from executions (e.g. QMP hangs). Here, the
> > finer granule we can push BQL the better. This series so far chose
> > somewhere in the middle, by taking BQL on majorly these two places:
> >
> > - CPU synchronizations
> > - Device START/FULL sections
> >
> > After this series applied, most of the rest loadvm path will run without
> > BQL anymore. There is a more detailed discussion / todo in the commit
> > message of patch "migration: Thread-ify precopy vmstate load process"
> > explaning how to further split the BQL critical sections.
> >
> > I was trying to split the patches into smaller ones if possible, but it's
> > still quite challenging so there's one major patch that does the work.
> >
> > After the series applied, the only leftover pieces in migration/ that would
> > use a coroutine is snapshot save/load/delete jobs.
> >
>
> Which are then fine because the work itself runs on the main loop,
> right? So the bottom-half scheduling could be left as a coroutine.
Correct, iochannel works for both cases.
For coroutines, it can properly register the fd and yield like before for
snapshot save/load. It used to do the same for live loadvm, but now after
moving to a thread it will start to use qio_channel_wait() instead.
I think we could also move back to blocking mode for live migration
incoming side after make it a thread, which might be slightly more
efficient to directly block in recvmsg() rather than return+poll. But it
is trivial comparing to "moving to thread" change, and it can be done for
later even if it works.
>
> > Tests
> > =====
> >
> > Default CI passes.
> >
> > RDMA unit tests pass as usual. I also tried out cancellation / failure
> > tests over RDMA channels, making sure nothing is stuck.
> >
> > I also roughly measured how long it takes to run the whole 80+ migration
> > qtest suite, and see no measurable difference before / after this series.
> >
> > Risks
> > =====
> >
> > This series has the risk of breaking things. I would be surprised if it
> > didn't..
> >
> > I confess I didn't test anything on COLO but only from code observations
> > and analysis. COLO maintainers: could you add some unit tests to QEMU's
> > qtests?
> >
> > The current way of taking BQL during FULL section load may cause issues, it
> > means when the IOs are unstable we could be waiting for IO (in the new
> > migration incoming thread) with BQL held. This is low possibility, though,
> > only happens when the network halts during flushing the device states.
> > However still possible. One solution is to further breakdown the BQL
> > critical sections to smaller sections, as mentioned in TODO.
> >
> > Anything more than welcomed: suggestions, questions, objections, tests..
> >
> > Todo
> > ====
> >
> > - Test COLO?
> > - Finer grained BQL breakdown
> > - More..
> >
> > Thanks,
> >
> > Peter Xu (9):
> > migration/vfio: Remove BQL implication in
> > vfio_multifd_switchover_start()
> > migration/rdma: Fix wrong context in qio_channel_rdma_shutdown()
> > migration/rdma: Allow qemu_rdma_wait_comp_channel work with thread
> > migration/rdma: Change io_create_watch() to return immediately
> > migration: Thread-ify precopy vmstate load process
> > migration/rdma: Remove coroutine path in qemu_rdma_wait_comp_channel
> > migration/postcopy: Remove workaround on wait preempt channel
> > migration/ram: Remove workaround on ram yield during load
> > migration/rdma: Remove rdma_cm_poll_handler
> >
> > include/migration/colo.h | 6 +-
> > migration/migration.h | 52 +++++++--
> > migration/savevm.h | 5 +-
> > hw/vfio/migration-multifd.c | 9 +-
> > migration/channel.c | 7 +-
> > migration/colo-stubs.c | 2 +-
> > migration/colo.c | 23 +---
> > migration/migration.c | 62 ++++++++---
> > migration/ram.c | 13 +--
> > migration/rdma.c | 206 ++++++++----------------------------
> > migration/savevm.c | 85 +++++++--------
> > migration/trace-events | 4 +-
> > 12 files changed, 196 insertions(+), 278 deletions(-)
>
--
Peter Xu
On Thu, Aug 28, 2025 at 4:59 AM Peter Xu <peterx@redhat.com> wrote:
>
> [this is an early RFC, not for merge, but to collect initial feedbacks]
>
> Background
> ==========
>
> Nowadays, live migration heavily depends on threads. For example, most of
> the major features that will be used nowadays in live migration (multifd,
> postcopy, mapped-ram, vfio, etc.) all work with threads internally.
>
> But still, from time to time, we'll see some coroutines floating around the
> migration context. The major one is precopy's loadvm, which is internally
> a coroutine. It is still a critical path that any live migration depends on.
>
> A mixture of using both coroutines and threads is prone to issues. Some
> examples can refer to commit e65cec5e5d ("migration/ram: Yield periodically
> to the main loop") or commit 7afbdada7e ("migration/postcopy: ensure
> preempt channel is ready before loading states").
>
> Overview
> ========
>
> This series tries to move migration further into the thread-based model, by
> allowing the loadvm process to happen in a thread rather than in the main
> thread with a coroutine.
>
> Luckily, since the qio channel code is always ready for both cases, IO
> paths should all be fine.
>
> Note that loadvm for postcopy already happens in a ram load thread which is
> separate. However, RAM is just the simple case here, even it has its own
> challenges (on atomically update of the pgtables), its complexity lies in
> the kernel.
>
> For precopy, loadvm has quite a few operations that will need BQL. The
> question is we can't take BQL for the whole process of loadvm, because
> that'll block the main thread from executions (e.g. QMP hangs). Here, the
> finer granule we can push BQL the better. This series so far chose
> somewhere in the middle, by taking BQL on majorly these two places:
>
> - CPU synchronizations
> - Device START/FULL sections
>
> After this series applied, most of the rest loadvm path will run without
> BQL anymore. There is a more detailed discussion / todo in the commit
> message of patch "migration: Thread-ify precopy vmstate load process"
> explaning how to further split the BQL critical sections.
>
> I was trying to split the patches into smaller ones if possible, but it's
> still quite challenging so there's one major patch that does the work.
>
> After the series applied, the only leftover pieces in migration/ that would
> use a coroutine is snapshot save/load/delete jobs.
>
> Tests
> =====
>
> Default CI passes.
>
> RDMA unit tests pass as usual. I also tried out cancellation / failure
> tests over RDMA channels, making sure nothing is stuck.
>
> I also roughly measured how long it takes to run the whole 80+ migration
> qtest suite, and see no measurable difference before / after this series.
>
> Risks
> =====
>
> This series has the risk of breaking things. I would be surprised if it
> didn't..
>
> I confess I didn't test anything on COLO but only from code observations
> and analysis. COLO maintainers: could you add some unit tests to QEMU's
> qtests?
For the COLO part, I think remove the coroutines related code is OK for me.
Because the original coroutine still need to call the
"colo_process_incoming_thread".
Hi Hailiang, any comments for this part?
Thanks
Chen
>
> The current way of taking BQL during FULL section load may cause issues, it
> means when the IOs are unstable we could be waiting for IO (in the new
> migration incoming thread) with BQL held. This is low possibility, though,
> only happens when the network halts during flushing the device states.
> However still possible. One solution is to further breakdown the BQL
> critical sections to smaller sections, as mentioned in TODO.
>
> Anything more than welcomed: suggestions, questions, objections, tests..
>
> Todo
> ====
>
> - Test COLO?
> - Finer grained BQL breakdown
> - More..
>
> Thanks,
>
> Peter Xu (9):
> migration/vfio: Remove BQL implication in
> vfio_multifd_switchover_start()
> migration/rdma: Fix wrong context in qio_channel_rdma_shutdown()
> migration/rdma: Allow qemu_rdma_wait_comp_channel work with thread
> migration/rdma: Change io_create_watch() to return immediately
> migration: Thread-ify precopy vmstate load process
> migration/rdma: Remove coroutine path in qemu_rdma_wait_comp_channel
> migration/postcopy: Remove workaround on wait preempt channel
> migration/ram: Remove workaround on ram yield during load
> migration/rdma: Remove rdma_cm_poll_handler
>
> include/migration/colo.h | 6 +-
> migration/migration.h | 52 +++++++--
> migration/savevm.h | 5 +-
> hw/vfio/migration-multifd.c | 9 +-
> migration/channel.c | 7 +-
> migration/colo-stubs.c | 2 +-
> migration/colo.c | 23 +---
> migration/migration.c | 62 ++++++++---
> migration/ram.c | 13 +--
> migration/rdma.c | 206 ++++++++----------------------------
> migration/savevm.c | 85 +++++++--------
> migration/trace-events | 4 +-
> 12 files changed, 196 insertions(+), 278 deletions(-)
>
> --
> 2.50.1
>
On Thu, Sep 04, 2025 at 04:27:39PM +0800, Zhang Chen wrote: > > I confess I didn't test anything on COLO but only from code observations > > and analysis. COLO maintainers: could you add some unit tests to QEMU's > > qtests? > > For the COLO part, I think remove the coroutines related code is OK for me. > Because the original coroutine still need to call the > "colo_process_incoming_thread". Chen, thanks for the comment. It's still reassuring. > > Hi Hailiang, any comments for this part? Any further comment on this series would always be helpful. It'll be also great if anyone can come up with a selftest for COLO. Now any new migration features needs both unit test and doc to get merged. COLO was merged earlier so it doesn't need to, however these will be helpful for sure to make sure COLO won't be easily broken. Thanks, -- Peter Xu
On Wed, Oct 08, 2025 at 05:26:13PM -0400, Peter Xu wrote:
> On Thu, Sep 04, 2025 at 04:27:39PM +0800, Zhang Chen wrote:
> > > I confess I didn't test anything on COLO but only from code observations
> > > and analysis. COLO maintainers: could you add some unit tests to QEMU's
> > > qtests?
> >
> > For the COLO part, I think remove the coroutines related code is OK for me.
> > Because the original coroutine still need to call the
> > "colo_process_incoming_thread".
>
> Chen, thanks for the comment. It's still reassuring.
>
> >
> > Hi Hailiang, any comments for this part?
>
> Any further comment on this series would always be helpful.
>
> It'll be also great if anyone can come up with a selftest for COLO. Now
> any new migration features needs both unit test and doc to get merged.
> COLO was merged earlier so it doesn't need to, however these will be
> helpful for sure to make sure COLO won't be easily broken.
Chen/Hailiang:
I may use some help from COLO side.
Just now, I did give it a shot with the current docs/COLO-FT.txt and it
didn't really work for me.
The cmdlines I used almost followed the doc, however I changed a few
things. For example, on secondary VM I added "file.locking=off" for drive
"parent0" because otherwise the "nbd-server-add" command will fail taking
the lock and it won't ever boot. Meanwhile I switched to socket netdev
from tap, in my case I only plan to run the COLO main routine, I hope
that's harmless too but let me know if it is a problem.
So below are the final cmdlines I used..
For primary:
bin=~/git/qemu/bin/qemu-system-x86_64
$bin -enable-kvm -cpu qemu64,kvmclock=on \
-m 512 -smp 1 -qmp stdio \
-device piix3-usb-uhci -device usb-tablet -name primary \
-netdev socket,id=hn0,listen=127.0.0.1:10000 \
-device rtl8139,id=e0,netdev=hn0 \
-chardev socket,id=mirror0,host=0.0.0.0,port=9003,server=on,wait=off \
-chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on \
-chardev socket,id=compare0,host=127.0.0.1,port=9001,server=on,wait=off \
-chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
-chardev socket,id=compare_out,host=127.0.0.1,port=9005,server=on,wait=off \
-chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
-object iothread,id=iothread1 \
-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 \
-drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=./primary.qcow2,children.0.driver=qcow2
For secondary (testing locally, hence using 127.0.0.1 as primary_ip):
bin=~/git/qemu/bin/qemu-system-x86_64
primary_ip=127.0.0.1
$bin -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \
-device piix3-usb-uhci -device usb-tablet -name secondary \
-netdev socket,id=hn0,connect=127.0.0.1:10000 \
-device rtl8139,id=e0,netdev=hn0 \
-chardev socket,id=red0,host=$primary_ip,port=9003,reconnect-ms=1000 \
-chardev socket,id=red1,host=$primary_ip,port=9004,reconnect-ms=1000 \
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
-object filter-rewriter,id=rew0,netdev=hn0,queue=all \
-drive if=none,id=parent0,file.filename=primary.qcow2,driver=qcow2,file.locking=off \
-drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=secondary-hidden.qcow2,file.backing.backing=parent0 \
-drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0 \
-incoming tcp:0.0.0.0:9998
I started secondary, then primary, run the suggested QMP commands on
secondary first, then the bunch of QMP commands on primary. I got below
error:
x1:colo $ ./primary.sh
qemu-system-x86_64: -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on: info: QEMU waiting for connection on: disconnected:tcp:0.0.0.0:9004,server=on
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
VNC server running on ::1:5900
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute": "human-monitor-command", "arguments": {"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}}
{"return": ""}
{"execute": "x-blockdev-change", "arguments":{"parent": "colo-disk0", "node": "replication0" } }
{"return": {}}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
{"return": {}}
{"execute": "migrate", "arguments": {"uri": "tcp:127.0.0.2:9998" } }
{"return": {}}
{"timestamp": {"seconds": 1760996025, "microseconds": 483349}, "event": "STOP"}
x1:colo $ ./secondary.sh
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
VNC server running on ::1:5901
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
{"return": {}}
{"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": "0.0.0.0", "port": "9999"} } } }
{"return": {}}
{"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": true } }
{"return": {}}
{"timestamp": {"seconds": 1760996025, "microseconds": 695059}, "event": "RESUME"}
qemu-system-x86_64: Can't receive COLO message: Input/output error
{"timestamp": {"seconds": 1760996025, "microseconds": 695369}, "event": "COLO_EXIT", "data": {"mode": "secondary", "reason": "error"}}
Do you know what I missed? Or does it mean that COLO is broken?
Meanwhile, do you know if COLO still being used by anyone? I'm pretty sure
both Fabiano and myself are not looking after it.. I remember Dave used to
try it, but it might be a long time ago too.
Thanks,
--
Peter Xu
On Mon, 20 Oct 2025 17:41:30 -0400
Peter Xu <peterx@redhat.com> wrote:
> On Wed, Oct 08, 2025 at 05:26:13PM -0400, Peter Xu wrote:
> > On Thu, Sep 04, 2025 at 04:27:39PM +0800, Zhang Chen wrote:
> > > > I confess I didn't test anything on COLO but only from code observations
> > > > and analysis. COLO maintainers: could you add some unit tests to QEMU's
> > > > qtests?
> > >
> > > For the COLO part, I think remove the coroutines related code is OK for me.
> > > Because the original coroutine still need to call the
> > > "colo_process_incoming_thread".
> >
> > Chen, thanks for the comment. It's still reassuring.
> >
> > >
> > > Hi Hailiang, any comments for this part?
> >
> > Any further comment on this series would always be helpful.
> >
> > It'll be also great if anyone can come up with a selftest for COLO. Now
> > any new migration features needs both unit test and doc to get merged.
> > COLO was merged earlier so it doesn't need to, however these will be
> > helpful for sure to make sure COLO won't be easily broken.
>
> Chen/Hailiang:
>
> I may use some help from COLO side.
>
> Just now, I did give it a shot with the current docs/COLO-FT.txt and it
> didn't really work for me.
>
> The cmdlines I used almost followed the doc, however I changed a few
> things. For example, on secondary VM I added "file.locking=off" for drive
> "parent0" because otherwise the "nbd-server-add" command will fail taking
> the lock and it won't ever boot. Meanwhile I switched to socket netdev
> from tap, in my case I only plan to run the COLO main routine, I hope
> that's harmless too but let me know if it is a problem.
>
> So below are the final cmdlines I used..
>
> For primary:
>
> bin=~/git/qemu/bin/qemu-system-x86_64
> $bin -enable-kvm -cpu qemu64,kvmclock=on \
> -m 512 -smp 1 -qmp stdio \
> -device piix3-usb-uhci -device usb-tablet -name primary \
> -netdev socket,id=hn0,listen=127.0.0.1:10000 \
> -device rtl8139,id=e0,netdev=hn0 \
> -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server=on,wait=off \
> -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on \
> -chardev socket,id=compare0,host=127.0.0.1,port=9001,server=on,wait=off \
> -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
> -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server=on,wait=off \
> -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
> -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
> -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
> -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
> -object iothread,id=iothread1 \
> -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 \
> -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=./primary.qcow2,children.0.driver=qcow2
>
> For secondary (testing locally, hence using 127.0.0.1 as primary_ip):
>
> bin=~/git/qemu/bin/qemu-system-x86_64
> primary_ip=127.0.0.1
> $bin -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \
> -device piix3-usb-uhci -device usb-tablet -name secondary \
> -netdev socket,id=hn0,connect=127.0.0.1:10000 \
> -device rtl8139,id=e0,netdev=hn0 \
> -chardev socket,id=red0,host=$primary_ip,port=9003,reconnect-ms=1000 \
> -chardev socket,id=red1,host=$primary_ip,port=9004,reconnect-ms=1000 \
> -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
> -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
> -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
> -drive if=none,id=parent0,file.filename=primary.qcow2,driver=qcow2,file.locking=off \
> -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=secondary-hidden.qcow2,file.backing.backing=parent0 \
> -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0 \
> -incoming tcp:0.0.0.0:9998
>
Hi Peter,
You have to use -incoming defer and enable x-colo on the
secondary side before starting migration.
And primary.qcow2 should be a separate image (with same content) for
each qemu instance.
Regards,
Lukas
> I started secondary, then primary, run the suggested QMP commands on
> secondary first, then the bunch of QMP commands on primary. I got below
> error:
>
> x1:colo $ ./primary.sh
> qemu-system-x86_64: -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on: info: QEMU waiting for connection on: disconnected:tcp:0.0.0.0:9004,server=on
> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
> VNC server running on ::1:5900
> {"execute":"qmp_capabilities"}
> {"return": {}}
> {"execute": "human-monitor-command", "arguments": {"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}}
> {"return": ""}
> {"execute": "x-blockdev-change", "arguments":{"parent": "colo-disk0", "node": "replication0" } }
> {"return": {}}
> {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
> {"return": {}}
> {"execute": "migrate", "arguments": {"uri": "tcp:127.0.0.2:9998" } }
> {"return": {}}
> {"timestamp": {"seconds": 1760996025, "microseconds": 483349}, "event": "STOP"}
>
> x1:colo $ ./secondary.sh
> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
> VNC server running on ::1:5901
> {"execute":"qmp_capabilities"}
> {"return": {}}
> {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
> {"return": {}}
> {"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": "0.0.0.0", "port": "9999"} } } }
> {"return": {}}
> {"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": true } }
> {"return": {}}
> {"timestamp": {"seconds": 1760996025, "microseconds": 695059}, "event": "RESUME"}
> qemu-system-x86_64: Can't receive COLO message: Input/output error
> {"timestamp": {"seconds": 1760996025, "microseconds": 695369}, "event": "COLO_EXIT", "data": {"mode": "secondary", "reason": "error"}}
>
> Do you know what I missed? Or does it mean that COLO is broken?
>
> Meanwhile, do you know if COLO still being used by anyone? I'm pretty sure
> both Fabiano and myself are not looking after it.. I remember Dave used to
> try it, but it might be a long time ago too.
>
> Thanks,
>
On Tue, Oct 21, 2025 at 6:09 AM Lukas Straub <lukasstraub2@web.de> wrote:
>
> On Mon, 20 Oct 2025 17:41:30 -0400
> Peter Xu <peterx@redhat.com> wrote:
>
> > On Wed, Oct 08, 2025 at 05:26:13PM -0400, Peter Xu wrote:
> > > On Thu, Sep 04, 2025 at 04:27:39PM +0800, Zhang Chen wrote:
> > > > > I confess I didn't test anything on COLO but only from code observations
> > > > > and analysis. COLO maintainers: could you add some unit tests to QEMU's
> > > > > qtests?
> > > >
> > > > For the COLO part, I think remove the coroutines related code is OK for me.
> > > > Because the original coroutine still need to call the
> > > > "colo_process_incoming_thread".
> > >
> > > Chen, thanks for the comment. It's still reassuring.
> > >
> > > >
> > > > Hi Hailiang, any comments for this part?
> > >
> > > Any further comment on this series would always be helpful.
> > >
> > > It'll be also great if anyone can come up with a selftest for COLO. Now
> > > any new migration features needs both unit test and doc to get merged.
> > > COLO was merged earlier so it doesn't need to, however these will be
> > > helpful for sure to make sure COLO won't be easily broken.
> >
> > Chen/Hailiang:
> >
> > I may use some help from COLO side.
> >
> > Just now, I did give it a shot with the current docs/COLO-FT.txt and it
> > didn't really work for me.
> >
> > The cmdlines I used almost followed the doc, however I changed a few
> > things. For example, on secondary VM I added "file.locking=off" for drive
> > "parent0" because otherwise the "nbd-server-add" command will fail taking
> > the lock and it won't ever boot. Meanwhile I switched to socket netdev
> > from tap, in my case I only plan to run the COLO main routine, I hope
> > that's harmless too but let me know if it is a problem.
> >
> > So below are the final cmdlines I used..
> >
> > For primary:
> >
> > bin=~/git/qemu/bin/qemu-system-x86_64
> > $bin -enable-kvm -cpu qemu64,kvmclock=on \
> > -m 512 -smp 1 -qmp stdio \
> > -device piix3-usb-uhci -device usb-tablet -name primary \
> > -netdev socket,id=hn0,listen=127.0.0.1:10000 \
> > -device rtl8139,id=e0,netdev=hn0 \
> > -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server=on,wait=off \
> > -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on \
> > -chardev socket,id=compare0,host=127.0.0.1,port=9001,server=on,wait=off \
> > -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
> > -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server=on,wait=off \
> > -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
> > -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
> > -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
> > -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
> > -object iothread,id=iothread1 \
> > -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 \
> > -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=./primary.qcow2,children.0.driver=qcow2
> >
> > For secondary (testing locally, hence using 127.0.0.1 as primary_ip):
> >
> > bin=~/git/qemu/bin/qemu-system-x86_64
> > primary_ip=127.0.0.1
> > $bin -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \
> > -device piix3-usb-uhci -device usb-tablet -name secondary \
> > -netdev socket,id=hn0,connect=127.0.0.1:10000 \
> > -device rtl8139,id=e0,netdev=hn0 \
> > -chardev socket,id=red0,host=$primary_ip,port=9003,reconnect-ms=1000 \
> > -chardev socket,id=red1,host=$primary_ip,port=9004,reconnect-ms=1000 \
> > -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
> > -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
> > -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
> > -drive if=none,id=parent0,file.filename=primary.qcow2,driver=qcow2,file.locking=off \
> > -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=secondary-hidden.qcow2,file.backing.backing=parent0 \
> > -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0 \
> > -incoming tcp:0.0.0.0:9998
> >
>
> Hi Peter,
> You have to use -incoming defer and enable x-colo on the
> secondary side before starting migration.
>
> And primary.qcow2 should be a separate image (with same content) for
> each qemu instance.
Yes, Lukas is right. Qemu can't allow 2 VM touch 1 image.
So, you can try to "cp primary.qcow2 secondary.qcow2",
then change the secondary side to " -drive
if=none,id=parent0,file.filename=secondary.qcow2,driver=qcow2,file.locking=off
\"
Thanks
Chen
>
> Regards,
> Lukas
>
>
> > I started secondary, then primary, run the suggested QMP commands on
> > secondary first, then the bunch of QMP commands on primary. I got below
> > error:
> >
> > x1:colo $ ./primary.sh
> > qemu-system-x86_64: -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on: info: QEMU waiting for connection on: disconnected:tcp:0.0.0.0:9004,server=on
> > {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
> > VNC server running on ::1:5900
> > {"execute":"qmp_capabilities"}
> > {"return": {}}
> > {"execute": "human-monitor-command", "arguments": {"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}}
> > {"return": ""}
> > {"execute": "x-blockdev-change", "arguments":{"parent": "colo-disk0", "node": "replication0" } }
> > {"return": {}}
> > {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
> > {"return": {}}
> > {"execute": "migrate", "arguments": {"uri": "tcp:127.0.0.2:9998" } }
> > {"return": {}}
> > {"timestamp": {"seconds": 1760996025, "microseconds": 483349}, "event": "STOP"}
> >
> > x1:colo $ ./secondary.sh
> > {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
> > VNC server running on ::1:5901
> > {"execute":"qmp_capabilities"}
> > {"return": {}}
> > {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
> > {"return": {}}
> > {"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": "0.0.0.0", "port": "9999"} } } }
> > {"return": {}}
> > {"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": true } }
> > {"return": {}}
> > {"timestamp": {"seconds": 1760996025, "microseconds": 695059}, "event": "RESUME"}
> > qemu-system-x86_64: Can't receive COLO message: Input/output error
> > {"timestamp": {"seconds": 1760996025, "microseconds": 695369}, "event": "COLO_EXIT", "data": {"mode": "secondary", "reason": "error"}}
> >
> > Do you know what I missed? Or does it mean that COLO is broken?
> >
> > Meanwhile, do you know if COLO still being used by anyone? I'm pretty sure
> > both Fabiano and myself are not looking after it.. I remember Dave used to
> > try it, but it might be a long time ago too.
> >
> > Thanks,
> >
>
On Tue, Oct 21, 2025 at 10:31:50AM +0800, Zhang Chen wrote:
> On Tue, Oct 21, 2025 at 6:09 AM Lukas Straub <lukasstraub2@web.de> wrote:
> >
> > On Mon, 20 Oct 2025 17:41:30 -0400
> > Peter Xu <peterx@redhat.com> wrote:
> >
> > > On Wed, Oct 08, 2025 at 05:26:13PM -0400, Peter Xu wrote:
> > > > On Thu, Sep 04, 2025 at 04:27:39PM +0800, Zhang Chen wrote:
> > > > > > I confess I didn't test anything on COLO but only from code observations
> > > > > > and analysis. COLO maintainers: could you add some unit tests to QEMU's
> > > > > > qtests?
> > > > >
> > > > > For the COLO part, I think remove the coroutines related code is OK for me.
> > > > > Because the original coroutine still need to call the
> > > > > "colo_process_incoming_thread".
> > > >
> > > > Chen, thanks for the comment. It's still reassuring.
> > > >
> > > > >
> > > > > Hi Hailiang, any comments for this part?
> > > >
> > > > Any further comment on this series would always be helpful.
> > > >
> > > > It'll be also great if anyone can come up with a selftest for COLO. Now
> > > > any new migration features needs both unit test and doc to get merged.
> > > > COLO was merged earlier so it doesn't need to, however these will be
> > > > helpful for sure to make sure COLO won't be easily broken.
> > >
> > > Chen/Hailiang:
> > >
> > > I may use some help from COLO side.
> > >
> > > Just now, I did give it a shot with the current docs/COLO-FT.txt and it
> > > didn't really work for me.
> > >
> > > The cmdlines I used almost followed the doc, however I changed a few
> > > things. For example, on secondary VM I added "file.locking=off" for drive
> > > "parent0" because otherwise the "nbd-server-add" command will fail taking
> > > the lock and it won't ever boot. Meanwhile I switched to socket netdev
> > > from tap, in my case I only plan to run the COLO main routine, I hope
> > > that's harmless too but let me know if it is a problem.
> > >
> > > So below are the final cmdlines I used..
> > >
> > > For primary:
> > >
> > > bin=~/git/qemu/bin/qemu-system-x86_64
> > > $bin -enable-kvm -cpu qemu64,kvmclock=on \
> > > -m 512 -smp 1 -qmp stdio \
> > > -device piix3-usb-uhci -device usb-tablet -name primary \
> > > -netdev socket,id=hn0,listen=127.0.0.1:10000 \
> > > -device rtl8139,id=e0,netdev=hn0 \
> > > -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server=on,wait=off \
> > > -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on \
> > > -chardev socket,id=compare0,host=127.0.0.1,port=9001,server=on,wait=off \
> > > -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
> > > -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server=on,wait=off \
> > > -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
> > > -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
> > > -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
> > > -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
> > > -object iothread,id=iothread1 \
> > > -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 \
> > > -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=./primary.qcow2,children.0.driver=qcow2
> > >
> > > For secondary (testing locally, hence using 127.0.0.1 as primary_ip):
> > >
> > > bin=~/git/qemu/bin/qemu-system-x86_64
> > > primary_ip=127.0.0.1
> > > $bin -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \
> > > -device piix3-usb-uhci -device usb-tablet -name secondary \
> > > -netdev socket,id=hn0,connect=127.0.0.1:10000 \
> > > -device rtl8139,id=e0,netdev=hn0 \
> > > -chardev socket,id=red0,host=$primary_ip,port=9003,reconnect-ms=1000 \
> > > -chardev socket,id=red1,host=$primary_ip,port=9004,reconnect-ms=1000 \
> > > -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
> > > -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
> > > -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
> > > -drive if=none,id=parent0,file.filename=primary.qcow2,driver=qcow2,file.locking=off \
> > > -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=secondary-hidden.qcow2,file.backing.backing=parent0 \
> > > -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0 \
> > > -incoming tcp:0.0.0.0:9998
> > >
> >
> > Hi Peter,
> > You have to use -incoming defer and enable x-colo on the
> > secondary side before starting migration.
> >
> > And primary.qcow2 should be a separate image (with same content) for
> > each qemu instance.
>
> Yes, Lukas is right. Qemu can't allow 2 VM touch 1 image.
> So, you can try to "cp primary.qcow2 secondary.qcow2",
> then change the secondary side to " -drive
> if=none,id=parent0,file.filename=secondary.qcow2,driver=qcow2,file.locking=off
> \"
Thanks both.
I think the doc says otherwise.. do you mean the doc is wrong and needs
fixing at least?
I created the secondary.qcow2, and switched to that, still it hit the same
error.
Step 1: start primary QEMU
bin=~/git/qemu/bin/qemu-system-x86_64
$bin -enable-kvm -cpu qemu64,kvmclock=on \
-m 512 -smp 1 -qmp stdio \
-device piix3-usb-uhci -device usb-tablet -name primary \
-netdev socket,id=hn0,listen=127.0.0.1:10000 \
-device rtl8139,id=e0,netdev=hn0 \
-chardev socket,id=mirror0,host=0.0.0.0,port=9003,server=on,wait=off \
-chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on \
-chardev socket,id=compare0,host=127.0.0.1,port=9001,server=on,wait=off \
-chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
-chardev socket,id=compare_out,host=127.0.0.1,port=9005,server=on,wait=off \
-chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
-object iothread,id=iothread1 \
-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 \
-drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=./primary.qcow2,children.0.driver=qcow2
Step 2: start secondary QEMU
bin=~/git/qemu/bin/qemu-system-x86_64
primary_ip=127.0.0.1
$bin -enable-kvm -cpu qemu64,kvmclock=on -m 512 -smp 1 -qmp stdio \
-device piix3-usb-uhci -device usb-tablet -name secondary \
-netdev socket,id=hn0,connect=127.0.0.1:10000 \
-device rtl8139,id=e0,netdev=hn0 \
-chardev socket,id=red0,host=$primary_ip,port=9003,reconnect-ms=1000 \
-chardev socket,id=red1,host=$primary_ip,port=9004,reconnect-ms=1000 \
-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
-object filter-rewriter,id=rew0,netdev=hn0,queue=all \
-drive if=none,id=parent0,file.filename=secondary.qcow2,driver=qcow2,file.locking=off \
-drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,top-id=colo-disk0,file.file.filename=secondary-active.qcow2,file.backing.driver=qcow2,file.backing.file.filename=secondary-hidden.qcow2,file.backing.backing=parent0 \
-drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0=childs0 \
-incoming tcp:0.0.0.0:9998
Step 3: Run these commands on secondary QEMU
{"execute":"qmp_capabilities"}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
{"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": "0.0.0.0", "port": "9999"} } } }
{"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": true } }
Step 4: Run these commands on primary QEMU
{"execute":"qmp_capabilities"}
{"execute": "human-monitor-command", "arguments": {"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}}
{"execute": "x-blockdev-change", "arguments":{"parent": "colo-disk0", "node": "replication0" } }
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
{"execute": "migrate", "arguments": {"uri": "tcp:127.0.0.2:9998" } }
What I got:
Primary QEMU output:
qemu-system-x86_64: -chardev socket,id=compare1,host=0.0.0.0,port=9004,server=on,wait=on: info: QEMU waiting for connection on: disconnected:tcp:0.0.0.0:9004,server=on
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
VNC server running on ::1:5901
{"error": {"class": "GenericError", "desc": "JSON parse error, stray '\f'"}}
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute": "human-monitor-command", "arguments": {"command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0"}}
{"return": ""}
{"execute": "x-blockdev-change", "arguments":{"parent": "colo-disk0", "node": "replication0" } }
{"return": {}}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
{"return": {}}
{"execute": "migrate", "arguments": {"uri": "tcp:127.0.0.2:9998" } }
{"return": {}}
{"timestamp": {"seconds": 1761054720, "microseconds": 515770}, "event": "STOP"}
Secondary QEMU output:
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 10}, "package": "v10.1.0-1513-g94586867df"}, "capabilities": ["oob"]}}
VNC server running on ::1:5900
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [ {"capability": "x-colo", "state": true } ] } }
{"return": {}}
{"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": "0.0.0.0", "port": "9999"} } } }
{"return": {}}
{"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": true } }
{"return": {}}
{"timestamp": {"seconds": 1761054721, "microseconds": 188336}, "event": "RESUME"}
qemu-system-x86_64: Can't receive COLO message: Input/output error
{"timestamp": {"seconds": 1761054721, "microseconds": 188883}, "event": "COLO_EXIT", "data": {"mode": "secondary", "reason": "error"}}
Thanks,
--
Peter Xu
On 27.08.25 23:59, Peter Xu wrote: > split the patches into smaller ones if possible Support for bql_held parameter for some functions may also be moved to separate preparation patches, which will simplify the main patch. -- Best regards, Vladimir
On Fri, Aug 29, 2025 at 11:29:37AM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 27.08.25 23:59, Peter Xu wrote: > > split the patches into smaller ones if possible > > Support for bql_held parameter for some functions may also be > moved to separate preparation patches, which will simplify the > main patch. Sure, I can do that. -- Peter Xu
© 2016 - 2026 Red Hat, Inc.