Currently the only way to make (external) live VM snapshot is using existing
dirty page logging migration mechanism. The main problem is that it tends to
produce a lot of page duplicates while running VM goes on updating already
saved pages. That leads to the fact that vmstate image size is commonly several
times bigger then non-zero part of virtual machine's RSS. Time required to
converge RAM migration and the size of snapshot image severely depend on the
guest memory write rate, sometimes resulting in unacceptably long snapshot
creation time and huge image size.

This series propose a way to solve the aforementioned problems. This is done
by using different RAM migration mechanism based on UFFD write protection
management introduced in v5.7 kernel. The migration strategy is to 'freeze'
guest RAM content using write-protection and iteratively release protection
for memory ranges that have already been saved to the migration stream.
At the same time we read in pending UFFD write fault events and save those
pages out-of-order with higher priority.

How to use:
1. Enable write-tracking migration capability
   virsh qemu-monitor-command <domain> --hmp migrate_set_capability.
track-writes-ram on

2. Start the external migration to a file
   virsh qemu-monitor-command <domain> --hmp migrate exec:'cat > ./vm_state'

3. Wait for the migration finish and check that the migration has completed.
state.

Andrey Gruzdev (7):
  Introduce 'track-writes-ram' migration capability.
  Introduced UFFD-WP low-level interface helpers. Implemented support
    for the whole RAM block memory protection/un-protection. Higher
    level ram_write_tracking_start() and ram_write_tracking_stop() to
    start/stop tracking memory writes on the whole VM memory.
  Support UFFD write fault processing in ram_save_iterate().
  Implementation of write-tracking migration thread.
  Implementation of vm_start() BH.
  The rest of write tracking migration code.
  Introduced simple linear scan rate limiting mechanism for write
    tracking migration.

 include/exec/memory.h |   7 +
 migration/migration.c | 338 +++++++++++++++++++++++++++++++-
 migration/migration.h |   4 +
 migration/ram.c       | 439 +++++++++++++++++++++++++++++++++++++++++-
 migration/ram.h       |   4 +
 migration/savevm.c    |   1 -
 migration/savevm.h    |   2 +
 qapi/migration.json   |   7 +-
 8 files changed, 790 insertions(+), 12 deletions(-)

-- 
2.25.1

Patchew URL: https://patchew.org/QEMU/20201118125449.311038-1-andrey.gruzdev@virtuozzo.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20201118125449.311038-1-andrey.gruzdev@virtuozzo.com
Subject: [PATCH v1 0/7] UFFD write-tracking migration/snapshots

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/20201118125449.311038-1-andrey.gruzdev@virtuozzo.com -> patchew/20201118125449.311038-1-andrey.gruzdev@virtuozzo.com
Switched to a new branch 'test'
2210ab4 Introduced simple linear scan rate limiting mechanism for write tracking migration.
48858b8 The rest of write tracking migration code.
2b3d804 Implementation of vm_start() BH.
60fca9a Implementation of write-tracking migration thread.
fe880bb Support UFFD write fault processing in ram_save_iterate().
c8d81aa Introduced UFFD-WP low-level interface helpers. Implemented support for the whole RAM block memory protection/un-protection. Higher level ram_write_tracking_start() and ram_write_tracking_stop() to start/stop tracking memory writes on the whole VM memory.
a80c55f Introduce 'track-writes-ram' migration capability.

=== OUTPUT BEGIN ===
1/7 Checking commit a80c55ffd302 (Introduce 'track-writes-ram' migration capability.)
WARNING: line over 80 characters
#134: FILE: migration/migration.c:3881:
+    DEFINE_PROP_MIG_CAP("x-track-writes-ram", MIGRATION_CAPABILITY_TRACK_WRITES_RAM),

total: 0 errors, 1 warnings, 146 lines checked

Patch 1/7 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
2/7 Checking commit c8d81aa74043 (Introduced UFFD-WP low-level interface helpers. Implemented support for the whole RAM block memory protection/un-protection. Higher level ram_write_tracking_start() and ram_write_tracking_stop() to start/stop tracking memory writes on the whole VM memory.)
3/7 Checking commit fe880bb9d63e (Support UFFD write fault processing in ram_save_iterate().)
WARNING: line over 80 characters
#70: FILE: migration/ram.c:1952:
+                    ((page_address - (hwaddr) pss->block->host) >> TARGET_PAGE_BITS);

WARNING: line over 80 characters
#120: FILE: migration/ram.c:2002:
+        /* In case we couldn't find respective block, just unprotect faulting page */

WARNING: line over 80 characters
#121: FILE: migration/ram.c:2003:
+        uffd_protect_memory(rs->uffdio_fd, page_address, TARGET_PAGE_SIZE, false);

WARNING: Block comments use a leading /* on a separate line
#156: FILE: migration/ram.c:2050:
+        /* In case of 'write-tracking' migration we first try

WARNING: Block comments use a trailing */ on a separate line
#157: FILE: migration/ram.c:2051:
+         * to poll UFFD and get write page fault event */

WARNING: line over 80 characters
#177: FILE: migration/ram.c:2071:
+                hwaddr run_length = (hwaddr) (page_to - page + 1) << TARGET_PAGE_BITS;

WARNING: line over 80 characters
#183: FILE: migration/ram.c:2077:
+                res = uffd_protect_memory(rs->uffdio_fd, page_address, run_length, false);

total: 0 errors, 7 warnings, 167 lines checked

Patch 3/7 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
4/7 Checking commit 60fca9aa0ae7 (Implementation of write-tracking migration thread.)
5/7 Checking commit 2b3d80494965 (Implementation of vm_start() BH.)
6/7 Checking commit 48858b8e92aa (The rest of write tracking migration code.)
ERROR: line over 90 characters
#19: FILE: migration/migration.c:3216:
+ * wt_migration_completion: Used by wt_migration_thread when after all the RAM has been saved.

WARNING: line over 80 characters
#28: FILE: migration/migration.c:3225:
+    /* Stop tracking RAM writes - un-protect memory, un-register UFFD memory ranges,

WARNING: Block comments use a leading /* on a separate line
#28: FILE: migration/migration.c:3225:
+    /* Stop tracking RAM writes - un-protect memory, un-register UFFD memory ranges,

WARNING: line over 80 characters
#29: FILE: migration/migration.c:3226:
+     * flush kernel wait queues and wake up threads waiting for write fault to be

WARNING: line over 80 characters
#30: FILE: migration/migration.c:3227:
+     * resolved. All of this is essentially done by closing UFFD file descriptor */

WARNING: Block comments use a trailing */ on a separate line
#30: FILE: migration/migration.c:3227:
+     * resolved. All of this is essentially done by closing UFFD file descriptor */

total: 1 errors, 5 warnings, 90 lines checked

Patch 6/7 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

7/7 Checking commit 2210ab4337dd (Introduced simple linear scan rate limiting mechanism for write tracking migration.)
WARNING: Block comments use a leading /* on a separate line
#78: FILE: migration/ram.c:2040:
+    /* In case last fault time was available and we have

WARNING: Block comments use a trailing */ on a separate line
#79: FILE: migration/ram.c:2041:
+     * latency value, check if it's not too high */

WARNING: Block comments use a leading /* on a separate line
#84: FILE: migration/ram.c:2046:
+    /* Delay thread execution till next write fault occures or timeout expires.

WARNING: line over 80 characters
#85: FILE: migration/ram.c:2047:
+     * Next SLOW_FAULT_SKIP_PAGES can be write fault pages only, not from pages going from

WARNING: line over 80 characters
#86: FILE: migration/ram.c:2048:
+     * linear scan logic. Thus we moderate migration stream rate to reduce latencies */

WARNING: Block comments use a trailing */ on a separate line
#86: FILE: migration/ram.c:2048:
+     * linear scan logic. Thus we moderate migration stream rate to reduce latencies */

WARNING: line over 80 characters
#118: FILE: migration/ram.c:2248:
+#define WT_MAX_WAIT 1000 /* 1000 ms, need bigger limit for 'write-tracking' migration */

total: 0 errors, 7 warnings, 103 lines checked

Patch 7/7 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20201118125449.311038-1-andrey.gruzdev@virtuozzo.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com