On 9/22/2025 9:49 AM, Steve Sistare wrote:
> This patch series adds the live migration cpr-exec mode.
I have received Acks or RB's for most of these, thank you all!
Just a reminder, these patches still need review from Peter and/or Fabiano:
Patch 5/8: migration: cpr-exec save and load
Patch 6/8: migration: cpr-exec mode
(And many thanks to Fabiano for reviewing the related qtest patches).
- Steve
> The new user-visible interfaces are:
> * cpr-exec (MigMode migration parameter)
> * cpr-exec-command (migration parameter)
>
> cpr-exec mode is similar in most respects to cpr-transfer mode, with the
> primary difference being that old QEMU directly exec's new QEMU. The user
> specifies the command to exec new QEMU in the migration parameter
> cpr-exec-command.
>
> Why?
>
> In a containerized QEMU environment, cpr-exec reuses an existing QEMU
> container and its assigned resources. By contrast, cpr-transfer mode
> requires a new container to be created on the same host as the target of
> the CPR operation. Resources must be reserved for the new container, while
> the old container still reserves resources until the operation completes.
> Avoiding over commitment requires extra work in the management layer.
> This is one reason why a cloud provider may prefer cpr-exec. A second reason
> is that the container may include agents with their own connections to the
> outside world, and such connections remain intact if the container is reused.
>
> How?
>
> cpr-exec preserves descriptors across exec by clearing the CLOEXEC flag,
> and by sending the unique name and value of each descriptor to new QEMU
> via CPR state.
>
> CPR state cannot be sent over the normal migration channel, because devices
> and backends are created prior to reading the channel, so this mode sends
> CPR state over a second migration channel that is not visible to the user.
> New QEMU reads the second channel prior to creating devices or backends.
>
> The exec itself is trivial. After writing to the migration channels, the
> migration code calls a new main-loop hook to perform the exec.
>
> Example:
>
> In this example, we simply restart the same version of QEMU, but in
> a real scenario one would use a new QEMU binary path in cpr-exec-command.
>
> # qemu-kvm -monitor stdio
> -object memory-backend-memfd,id=ram0,size=1G
> -machine memory-backend=ram0 -machine aux-ram-share=on ...
>
> QEMU 10.1.50 monitor - type 'help' for more information
> (qemu) info status
> VM status: running
> (qemu) migrate_set_parameter mode cpr-exec
> (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming file:vm.state
> (qemu) migrate -d file:vm.state
> (qemu) QEMU 10.1.50 monitor - type 'help' for more information
> (qemu) info status
> VM status: running
>
> Changes in V2:
> * dropped patch "helper to request exec" and use a BH to exec
> * used g_shell_parse_argv for cpr-exec-command parameter
> * fixed check for channel in cpr_state_load
> * tweaked QAPI docs, developer docs, and code comments
> * fixed doc: rename cpr-exec-args -> cpr-exec-command
>
> Steve Sistare (8):
> migration: multi-mode notifier
> migration: add cpr_walk_fd
> oslib: qemu_clear_cloexec
> migration: cpr-exec-command parameter
> migration: cpr-exec save and load
> migration: cpr-exec mode
> migration: cpr-exec docs
> vfio: cpr-exec mode
>
> docs/devel/migration/CPR.rst | 106 +++++++++++++++++++++++-
> qapi/migration.json | 46 ++++++++++-
> include/migration/cpr.h | 9 +++
> include/migration/misc.h | 12 +++
> include/qemu/osdep.h | 9 +++
> hw/vfio/container.c | 3 +-
> hw/vfio/cpr-iommufd.c | 3 +-
> hw/vfio/cpr-legacy.c | 9 ++-
> hw/vfio/cpr.c | 13 +--
> migration/cpr-exec.c | 178 +++++++++++++++++++++++++++++++++++++++++
> migration/cpr.c | 41 +++++++++-
> migration/migration-hmp-cmds.c | 30 +++++++
> migration/migration.c | 70 ++++++++++++----
> migration/options.c | 14 ++++
> migration/ram.c | 1 +
> migration/vmstate-types.c | 8 ++
> system/vl.c | 4 +-
> util/oslib-posix.c | 9 +++
> util/oslib-win32.c | 4 +
> hmp-commands.hx | 2 +-
> migration/meson.build | 1 +
> migration/trace-events | 1 +
> 22 files changed, 538 insertions(+), 35 deletions(-)
> create mode 100644 migration/cpr-exec.c
>
> base-commit: e7c1e8043a69c5a8efa39d4f9d111f7c72c076e6