From: Nikita Lapshin <nikita.lapshin@openvz.org>
Changes v2 -> v3
* Refactored tool code to decrease duplications of migration code.
* Used sequential migration for saving which means that vmstate
will be send first and only after that ram part will be sent.
For this purpose stream-content-list paramter was used.
* Removed aio work with block driver. Should be replaced with
existing in qcow2 format in next versions.
* Removed postcopy percent. Should be added in next versions.
Changes v1 -> v2:
* Fixed CI checks
Changes v0 -> v1:
* Changed command-line format, now use blockdev specification to
define vmstate image.
* Don't deal with image creation in the tool, create externally.
* Better block layer AIO handling in the load path.
* Reduced fragmentation of the image backing file by using 'writtent-slice'
bitmaps in RAM blocks. Zero block write is issued to a never written slice
before the actual memory page write takes place.
* Improved load performance in postcopy by using 'loaded-slice' bitmaps
in RAM blocks.
* Refactored error handling/messages.
* Refactored namings.
This series is a kind of PoC for asynchronous snapshot reverting. It is
about external snapshots only and doesn't involve block devices. Thus, it's
mainly intended to be used with the new 'background-snapshot' migration
capability and otherwise standard QEMU migration mechanism.
The major ideas behind this version were:
* Make it compatible with 'exec:'-style migration - options can be create
some separate tool or integrate into qemu-system.
* Support asynchronous revert stage by using unaltered postcopy logic
at destination. To do this, we should be capable of saving RAM pages
so that any particular page can be directly addressed by it's block ID
and page offset. Possible solutions here seem to be:
use separate index (and storing it somewhere)
create sparse file on host FS and address pages with file offset
use QCOW2 (or other) image container with inherent sparsity support
* Make image file dense on the host FS so we don't depend on
copy/backup tools and how they deal with sparse files. Off course,
there's some performance cost for this choice.
* Try to keep page save latencies small while not degrading migration
bandwidth too much.
This version of snapshot-tool is the first step to integrate tool into
main QEMU. Now tool replace ram hanlers so it can call existing functions
in migration/* part to parse migration stream.
For the storage format, QCOW2 as a container and large (1MB) cluster size seem
to be an optimal choice. Larger cluster is beneficial for performance
particularly
in the case when image preallocation is disabled. Such cluster size does not
result
in too high internal fragmentation level (~10% of space waste in most cases) yet
allows to reduce significantly the number of expensive cluster allocations.
"stream-content-list"
There was no strict guarantee that there is no sections in ram part
rather than ram. So to solve this problem we decided to implement
parameters stream-content-list to provide such guarantee strictly.
This decision also helps with reusing of existed migration code.
You can see it in tool load part where tool disables all handlers except
ram using this parameter. If you have already seen it in previous patches you
can skip first 8 commits.
"sequential migration"
One problem remains unsolved. We need to run two migrations first to
save vmstate and second to save ram. We cannot run migration if VM is in
postmigrate state. But if we want to make snapshot this prohibition is
unnecessary so I changed some parts of migration and softmmu so
sequential migration become permitted. But that is not a solution. May
be new capability should be implementedi for that purpose.
Some of the upgrades were removed for now. This happened because of refactoring
and should be implemented in next versions.
How to use:
**Save:**
* > qemu-img create -f qcow2 -o size=<2_x_ram_size>,cluster_size=1M,
preallocation=off,refcount_bits=8 <image-filename>
* qemu> migrate_set_capability background-snapshot on
* #set SCL to "vmstate" only
* qemu> migrate "exec:qemu-snapshot --save-vmstate
<image-filename>,cache.direct=off,file.aio=threads"
* #set SCL to "ram" only
* qemu> migrate "exec:qemu-snapshot
<image-filename>,cache.direct=off,file.aio=threads"
**Load:**
* Use 'qemu-system-* -incoming defer'
* qemu> migrate_incoming "exec:qemu-snapshot --revert
<image-filename>,cache.direct=on,file.aio=native"
**Load with postcopy:**
* Use 'qemu-system-* -incoming defer'
* qemu> migrate_set_capability postcopy-ram on
* qemu> migrate_incoming "exec:qemu-snapshot --revert --postcopy
<image-filename>,cache.direct=on,file.aio=native"
Nikita Lapshin (17):
migration: Implemented new parameter stream_content
migration: should_skip() implemented
migration: Add vmstate part of migration stream
igration: Add dirty-bitmaps part of migration stream
Add block part of migration stream
migration: Add RAM part of migration stream
migration: analyze-migration script changed
migration: Test for RAM and vmstate parts
migration/snapshot: Introduce qemu-snapshot tool
migration/snapshot: Build changes for qemu-snapshot-tool
migration/qemu-file: Fix qemu_ftell() for non-writable file
migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h
migration/snapshot: Block layer support in qemu-snapshot
migration/snpashot: Implement API for RAMBlock
migration/snapshot: Save part implement
migration/snapshot: Precopy load implemented
migration/snapshot: Postcopy load implemented
include/qemu-snapshot.h | 94 ++
meson.build | 18 +
migration/meson.build | 4 +-
migration/migration.c | 199 ++-
migration/migration.h | 4 +
migration/qemu-file.c | 3 +-
migration/qemu-snapshot-io.c | 112 ++
migration/qemu-snapshot.c | 1126 +++++++++++++++++
migration/ram.c | 22 +-
migration/ram.h | 16 +
migration/savevm.c | 116 +-
migration/savevm.h | 8 +
qapi/migration.json | 21 +-
qemu-snapshot.c | 540 ++++++++
scripts/analyze-migration.py | 19 +-
.../tests/migrate-ram-stream-content-test | 96 ++
.../tests/migrate-ram-stream-content-test.out | 5 +
17 files changed, 2342 insertions(+), 61 deletions(-)
create mode 100644 include/qemu-snapshot.h
create mode 100644 migration/qemu-snapshot-io.c
create mode 100644 migration/qemu-snapshot.c
create mode 100644 qemu-snapshot.c
create mode 100755 tests/qemu-iotests/tests/migrate-ram-stream-content-test
create mode 100644 tests/qemu-iotests/tests/migrate-ram-stream-content-test.out
--
2.31.1