On 4/29/2024 11:55 AM, Steve Sistare wrote:
> This patch series adds the live migration cpr-exec mode.
Here is the text I plan to add to docs/devel/migration/CPR.rst. It is
premature for me to submit this as a patch, because it includes all
the functionality I plan to add in this and future series, but it may
help you while reviewing this series.
- Steve
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
cpr-exec mode
---------------
In this mode, QEMU stops the VM, writes VM state to the migration
URI, and directly exec's a new version of QEMU on the same host,
replacing the original process while retaining its PID. Guest RAM is
preserved in place, albeit with new virtual addresses. The user
completes the migration by specifying the ``-incoming`` option, and
by issuing the ``migrate-incoming`` command if necessary; see details
below.
This mode supports vfio devices by preserving device descriptors and
hence kernel state across the exec, even for devices that do not
support live migration, and preserves tap and vhost descriptors.
cpr-exec also preserves descriptors for a subset of chardevs,
including socket, file, parallel, pipe, serial, pty, stdio, and null.
chardevs that support cpr-exec have the QEMU_CHAR_FEATURE_CPR set in
the Chardev object. The client side of a preserved chardev sees no
loss of connectivity during cpr-exec. More chardevs could be
preserved with additional developement.
All chardevs have a ``reopen-on-cpr`` option which causes the chardev
to be closed and reopened during cpr-exec. This can be set to allow
cpr-exec when the configuration includes a chardev (such as vc) that
does not have QEMU_CHAR_FEATURE_CPR.
Because the old and new QEMU instances are not active concurrently,
the URI cannot be a type that streams data from one instance to the
other.
Usage
^^^^^
Arguments for the new QEMU process are taken from the
@cpr-exec-args parameter. The first argument should be the
path of a new QEMU binary, or a prefix command that exec's the
new QEMU binary, and the arguments should include the ''-incoming''
option.
Memory backend objects must have the ``share=on`` attribute, and
must be mmap'able in the new QEMU process. For example,
memory-backend-file is acceptable, but memory-backend-ram is
not.
The VM must be started with the ``-machine memfd-alloc=on``
option. This causes implicit RAM blocks (those not explicitly
described by a memory-backend object) to be allocated by
mmap'ing a memfd. Examples include VGA, ROM, and even guest
RAM when it is specified without without reference to a
memory-backend object.
Add the ``-only-migratable-modes cpr-exec`` option to guarantee that
the configuration supports cpr-exec. QEMU will exit at start time
if not.
Outgoing:
* Set the migration mode parameter to ``cpr-exec``.
* Set the ``cpr-exec-args`` parameter.
* Issue the ``migrate`` command. It is recommended the the URI be
a ``file`` type, but one can use other types such as ``exec``,
provided the command captures all the data from the outgoing side,
and provides all the data to the incoming side.
Incoming:
* You do not need to explicitly start new QEMU. It is started as
a side effect of the migrate command above.
* If the VM was running when the outgoing ``migrate`` command was
issued, then QEMU automatically resumes VM execution.
Example 1: incoming URI
^^^^^^^^^^^^^^^^^^^^^^^
In these examples, we simply restart the same version of QEMU, but in
a real scenario one would set a new QEMU binary path in cpr-exec-args.
::
# qemu-kvm -monitor stdio
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G
-machine memfd-alloc=on
...
QEMU 9.1.50 monitor - type 'help' for more information
(qemu) info status
VM status: running
(qemu) migrate_set_parameter mode cpr-exec
(qemu) migrate_set_parameter cpr-exec-args qemu-kvm ... -incoming
file:vm.state
(qemu) migrate -d file:vm.state
(qemu) QEMU 9.1.50 monitor - type 'help' for more information
(qemu) info status
VM status: running
Example 2: incoming defer
^^^^^^^^^^^^^^^^^^^^^^^^^
::
# qemu-kvm -monitor stdio
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G
-machine memfd-alloc=on
...
QEMU 9.1.50 monitor - type 'help' for more information
(qemu) info status
VM status: running
(qemu) migrate_set_parameter mode cpr-exec
(qemu) migrate_set_parameter cpr-exec-args qemu-kvm ... -incoming defer
(qemu) migrate -d file:vm.state
(qemu) QEMU 9.1.50 monitor - type 'help' for more information
(qemu) info status
status: paused (inmigrate)
(qemu) migrate_incoming file:vm.state
(qemu) info status
VM status: running
Caveats
^^^^^^^
cpr-exec mode may not be used with postcopy, background-snapshot,
or COLO.
cpr-exec mode requires permission to use the exec system call, which
is denied by certain sandbox options, such as spawn. Use finer
grained controls to allow exec, eg:
``-sandbox on,fork=deny,ns=deny,exec=allow``
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::