Changeset
Makefile.target              |    1 +
block/qcow2-snapshot.c       |  154 ++++++++++++++++++++++
block/qcow2.c                |    2 +
block/qcow2.h                |    7 +
block/snapshot.c             |   45 +++++++
exec.c                       |    7 +
hw/ppc/spapr.c               |    2 +-
hw/s390x/s390-stattrib.c     |    2 +-
include/block/block_int.h    |    9 ++
include/block/snapshot.h     |    7 +
include/exec/memory.h        |    9 ++
include/exec/ram_addr.h      |    2 +
include/migration/misc.h     |    4 +
include/migration/register.h |    2 +-
include/migration/snapshot.h |    3 +
memory.c                     |   18 ++-
migration/block.c            |    2 +-
migration/nvdimm.c           | 1033 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
migration/qemu-file.c        |   61 +++++++++
migration/qemu-file.h        |   14 ++
migration/ram.c              |   19 ++-
migration/savevm.c           |   62 ++++++++-
vl.c                         |    1 +
23 files changed, 1452 insertions(+), 14 deletions(-)
Git apply log
Switched to a new branch '1520990418-28258-1-git-send-email-junyan.he@hotmail.com'
Applying: RFC: Add save and support snapshot dependency function to block driver.
Applying: RFC: Implement qcow2's snapshot dependent saving function.
Applying: RFC: Implement save and support snapshot dependency in block driver layer.
Applying: RFC: Set memory_region_set_log available for more client.
Applying: RFC: Add memory region snapshot bitmap get function.
Applying: RFC: Add save dependency functions to qemu_file
Applying: RFC: Add get_current_snapshot_info to get the snapshot state.
Applying: RFC: Add a section_id parameter to save_live_iterate call.
Applying: RFC: Add nvdimm snapshot saving to migration.
Applying: RFC: Enable nvdimm snapshot functions.
To https://github.com/patchew-project/qemu
 * [new tag]         patchew/1520990418-28258-1-git-send-email-junyan.he@hotmail.com -> patchew/1520990418-28258-1-git-send-email-junyan.he@hotmail.com
Test failed: checkpatch

loading

Test failed: docker-mingw@fedora

loading

Test passed: docker-quick@centos6

loading

Test passed: docker-build@min-glib

loading

Test passed: s390x

loading

[Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

The nvdimm size is huge, sometimes it is more than 256G or even more.
This is a huge burden for snapshot saving. One snapshot point with
nvdimm may occupy more than 50G disk space even with compression
enabled.
We need to introduce dependent snapshot manner to solve this problem.
The first snapshot point should always be saved completely, and enable
dirty log trace after saving for nvdimm memory region. The later snapshot
point should add the reference to previous snapshot's nvdimm data and
just saving dirty pages. This can save a lot of disk and time if the
snapshot operations are triggered frequently.
We add save_snapshot_dependency functions to QCOW2 file system firstly, the
later snapshot will add reference to previous dependent snapshot's data
cluster. There is an alignment problem here, the dependent data should
always be cluster aligned. We need to add some padding data when saving
the snapshot to make it always cluster aligned.
The logic between nvdimm and ram for snapshot saving is a little confused
now, we need to exclude nvdimm kind memory region from ram list and the
dirty log tracing setting is also not very clear. Maybe we can separate the
snapshot saving from the migration logic later to make code clean.
In theory, this kind of manner can apply to any kind of memory. But because
it need to turn dirty log trace on, the performance may decline. So we just
enable it for nvdimm kind memory firstly.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
Makefile.target              |    1 +
block/qcow2-snapshot.c       |  154 ++++++++++++++++++++++
block/qcow2.c                |    2 +
block/qcow2.h                |    7 +
block/snapshot.c             |   45 +++++++
exec.c                       |    7 +
hw/ppc/spapr.c               |    2 +-
hw/s390x/s390-stattrib.c     |    2 +-
include/block/block_int.h    |    9 ++
include/block/snapshot.h     |    7 +
include/exec/memory.h        |    9 ++
include/exec/ram_addr.h      |    2 +
include/migration/misc.h     |    4 +
include/migration/register.h |    2 +-
include/migration/snapshot.h |    3 +
memory.c                     |   18 ++-
migration/block.c            |    2 +-
migration/nvdimm.c           | 1033 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
migration/qemu-file.c        |   61 +++++++++
migration/qemu-file.h        |   14 ++
migration/ram.c              |   19 ++-
migration/savevm.c           |   62 ++++++++-
vl.c                         |    1 +
23 files changed, 1452 insertions(+), 14 deletions(-)

Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.
Posted by no-reply@patchew.org, 14 weeks ago
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 1520990418-28258-1-git-send-email-junyan.he@hotmail.com
Subject: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/1520990418-28258-1-git-send-email-junyan.he@hotmail.com -> patchew/1520990418-28258-1-git-send-email-junyan.he@hotmail.com
Switched to a new branch 'test'
cf8b49f484 RFC: Enable nvdimm snapshot functions.
839b0dc7fc RFC: Add nvdimm snapshot saving to migration.
edf96a743e RFC: Add a section_id parameter to save_live_iterate call.
eaaf08deb8 RFC: Add get_current_snapshot_info to get the snapshot state.
ac55e78d2c RFC: Add save dependency functions to qemu_file
99deb40575 RFC: Add memory region snapshot bitmap get function.
eb7ccdaf28 RFC: Set memory_region_set_log available for more client.
9ee8792a3b RFC: Implement save and support snapshot dependency in block driver layer.
ebadb43e47 RFC: Implement qcow2's snapshot dependent saving function.
1a29421543 RFC: Add save and support snapshot dependency function to block driver.

=== OUTPUT BEGIN ===
Checking PATCH 1/10: RFC: Add save and support snapshot dependency function to block driver....
Checking PATCH 2/10: RFC: Implement qcow2's snapshot dependent saving function....
Checking PATCH 3/10: RFC: Implement save and support snapshot dependency in block driver layer....
Checking PATCH 4/10: RFC: Set memory_region_set_log available for more client....
Checking PATCH 5/10: RFC: Add memory region snapshot bitmap get function....
Checking PATCH 6/10: RFC: Add save dependency functions to qemu_file...
Checking PATCH 7/10: RFC: Add get_current_snapshot_info to get the snapshot state....
Checking PATCH 8/10: RFC: Add a section_id parameter to save_live_iterate call....
Checking PATCH 9/10: RFC: Add nvdimm snapshot saving to migration....
WARNING: line over 80 characters
#122: FILE: migration/nvdimm.c:70:
+------------------------------------------------------------------------------------

WARNING: line over 80 characters
#123: FILE: migration/nvdimm.c:71:
+| DIRTY_BITMAP_ID | total size | ram name size | ram name | ram size | bitmap size |

WARNING: line over 80 characters
#124: FILE: migration/nvdimm.c:72:
+------------------------------------------------------------------------------------

WARNING: line over 80 characters
#129: FILE: migration/nvdimm.c:77:
+---------------------------------------------------------------------------------------

WARNING: line over 80 characters
#130: FILE: migration/nvdimm.c:78:
+| DATA_ID | size | ram name size | ram name | ram size | data size | data... | END_ID |

WARNING: line over 80 characters
#131: FILE: migration/nvdimm.c:79:
+---------------------------------------------------------------------------------------

ERROR: do not use C99 // comments
#243: FILE: migration/nvdimm.c:191:
+    padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_PADDING_ID

ERROR: do not use C99 // comments
#244: FILE: migration/nvdimm.c:192:
+    padding_sz -= sizeof(int32_t); // NVDIMM_PADDING_BYTE size

ERROR: do not use C99 // comments
#245: FILE: migration/nvdimm.c:193:
+    padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_END_ID

ERROR: do not use C99 // comments
#344: FILE: migration/nvdimm.c:292:
+            data_sz += sizeof(int); // Zero page, just a ID

ERROR: do not use C99 // comments
#346: FILE: migration/nvdimm.c:294:
+            data_sz += ((1 << TARGET_PAGE_BITS) + sizeof(int)); // ID + page

ERROR: do not use C99 // comments
#350: FILE: migration/nvdimm.c:298:
+    total_sz = sizeof(unsigned int); // NVDIMM_SECTION_DIRTY_BITMAP_ID

ERROR: do not use C99 // comments
#351: FILE: migration/nvdimm.c:299:
+    total_sz += sizeof(uint64_t);    // the total size itself

ERROR: do not use C99 // comments
#352: FILE: migration/nvdimm.c:300:
+    total_sz += sizeof(int);         // ram name size

ERROR: do not use C99 // comments
#355: FILE: migration/nvdimm.c:303:
+    total_sz += sizeof(uint64_t); // ram size

ERROR: do not use C99 // comments
#356: FILE: migration/nvdimm.c:304:
+    total_sz += sizeof(uint64_t); // data size

ERROR: do not use C99 // comments
#358: FILE: migration/nvdimm.c:306:
+    total_sz += sizeof(unsigned int); // NVDIMM_SECTION_END_ID

WARNING: line over 80 characters
#414: FILE: migration/nvdimm.c:362:
+        memory_region_size(nvdimm_state->blocks[i]->mr) >> (TARGET_PAGE_BITS + 3);

WARNING: line over 80 characters
#421: FILE: migration/nvdimm.c:369:
+                                             snap, addr, 1 << TARGET_PAGE_BITS)) {

ERROR: do not use C99 // comments
#426: FILE: migration/nvdimm.c:374:
+    total_sz = sizeof(unsigned int); // NVDIMM_SECTION_DIRTY_BITMAP_ID

ERROR: do not use C99 // comments
#427: FILE: migration/nvdimm.c:375:
+    total_sz += sizeof(uint64_t);    // the total size itself

ERROR: do not use C99 // comments
#428: FILE: migration/nvdimm.c:376:
+    total_sz += sizeof(int);         // ram name size

ERROR: do not use C99 // comments
#431: FILE: migration/nvdimm.c:379:
+    total_sz += sizeof(uint64_t); // ram size

ERROR: do not use C99 // comments
#432: FILE: migration/nvdimm.c:380:
+    total_sz += sizeof(uint64_t); // bitmap size

ERROR: do not use C99 // comments
#434: FILE: migration/nvdimm.c:382:
+    total_sz += sizeof(uint64_t); // data size

ERROR: do not use C99 // comments
#436: FILE: migration/nvdimm.c:384:
+    total_sz += sizeof(unsigned int); // NVDIMM_SECTION_END_ID

WARNING: line over 80 characters
#453: FILE: migration/nvdimm.c:401:
+                                                 snap, addr, 1 << TARGET_PAGE_BITS)) {

ERROR: do not use C99 // comments
#552: FILE: migration/nvdimm.c:500:
+    if (QEMU_IS_ALIGNED(cur_pos, alignment)) { // Already aligned

ERROR: trailing statements should be on next line
#552: FILE: migration/nvdimm.c:500:
+    if (QEMU_IS_ALIGNED(cur_pos, alignment)) { // Already aligned

ERROR: g_free(NULL) is safe this check is probably not required
#594: FILE: migration/nvdimm.c:542:
+        if (nvdimm_state->cur_snapshot_id) {
+            g_free(nvdimm_state->cur_snapshot_id);

ERROR: g_free(NULL) is safe this check is probably not required
#597: FILE: migration/nvdimm.c:545:
+        if (nvdimm_state->blocks) {
+            g_free(nvdimm_state->blocks);

ERROR: g_free(NULL) is safe this check is probably not required
#676: FILE: migration/nvdimm.c:624:
+        if (nvdimm_state->depend_snapshot_id) {
+            g_free(nvdimm_state->depend_snapshot_id);

ERROR: do not use C99 // comments
#695: FILE: migration/nvdimm.c:643:
+                // Can not find the same block?

ERROR: g_free(NULL) is safe this check is probably not required
#896: FILE: migration/nvdimm.c:844:
+    if (bitmap_buf) {
+        g_free(bitmap_buf);

ERROR: g_free(NULL) is safe this check is probably not required
#983: FILE: migration/nvdimm.c:931:
+    if (buf) {
+        g_free(buf);

ERROR: g_free(NULL) is safe this check is probably not required
#1062: FILE: migration/nvdimm.c:1010:
+    if (buf) {
+        g_free(buf);

total: 27 errors, 9 warnings, 1050 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 10/10: RFC: Enable nvdimm snapshot functions....
ERROR: do not use C99 // comments
#47: FILE: migration/ram.c:1596:
+            // If snapshot and the block is nvdimm, let nvdimm do the job

ERROR: do not use C99 // comments
#58: FILE: migration/ram.c:2239:
+            // If snapshot and the block is nvdimm, let nvdimm do the job

total: 2 errors, 0 warnings, 51 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
Re: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.
Posted by no-reply@patchew.org, 14 weeks ago
Hi,

This series failed docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1520990418-28258-1-git-send-email-junyan.he@hotmail.com
Subject: [Qemu-devel] [PATCH 00/10] RFC: Optimize nvdimm kind memory for snapshot.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
cf8b49f484 RFC: Enable nvdimm snapshot functions.
839b0dc7fc RFC: Add nvdimm snapshot saving to migration.
edf96a743e RFC: Add a section_id parameter to save_live_iterate call.
eaaf08deb8 RFC: Add get_current_snapshot_info to get the snapshot state.
ac55e78d2c RFC: Add save dependency functions to qemu_file
99deb40575 RFC: Add memory region snapshot bitmap get function.
eb7ccdaf28 RFC: Set memory_region_set_log available for more client.
9ee8792a3b RFC: Implement save and support snapshot dependency in block driver layer.
ebadb43e47 RFC: Implement qcow2's snapshot dependent saving function.
1a29421543 RFC: Add save and support snapshot dependency function to block driver.

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-fe8degv8/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-fe8degv8/src'
  GEN     /var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar.vroot'...
done.
Checking out files:  47% (2826/6005)   
Checking out files:  48% (2883/6005)   
Checking out files:  49% (2943/6005)   
Checking out files:  50% (3003/6005)   
Checking out files:  51% (3063/6005)   
Checking out files:  52% (3123/6005)   
Checking out files:  53% (3183/6005)   
Checking out files:  54% (3243/6005)   
Checking out files:  55% (3303/6005)   
Checking out files:  56% (3363/6005)   
Checking out files:  57% (3423/6005)   
Checking out files:  58% (3483/6005)   
Checking out files:  59% (3543/6005)   
Checking out files:  60% (3603/6005)   
Checking out files:  61% (3664/6005)   
Checking out files:  62% (3724/6005)   
Checking out files:  63% (3784/6005)   
Checking out files:  64% (3844/6005)   
Checking out files:  65% (3904/6005)   
Checking out files:  66% (3964/6005)   
Checking out files:  67% (4024/6005)   
Checking out files:  68% (4084/6005)   
Checking out files:  69% (4144/6005)   
Checking out files:  70% (4204/6005)   
Checking out files:  71% (4264/6005)   
Checking out files:  72% (4324/6005)   
Checking out files:  73% (4384/6005)   
Checking out files:  74% (4444/6005)   
Checking out files:  75% (4504/6005)   
Checking out files:  76% (4564/6005)   
Checking out files:  77% (4624/6005)   
Checking out files:  78% (4684/6005)   
Checking out files:  79% (4744/6005)   
Checking out files:  80% (4804/6005)   
Checking out files:  81% (4865/6005)   
Checking out files:  82% (4925/6005)   
Checking out files:  83% (4985/6005)   
Checking out files:  84% (5045/6005)   
Checking out files:  85% (5105/6005)   
Checking out files:  86% (5165/6005)   
Checking out files:  87% (5225/6005)   
Checking out files:  88% (5285/6005)   
Checking out files:  89% (5345/6005)   
Checking out files:  90% (5405/6005)   
Checking out files:  91% (5465/6005)   
Checking out files:  92% (5525/6005)   
Checking out files:  93% (5585/6005)   
Checking out files:  94% (5645/6005)   
Checking out files:  95% (5705/6005)   
Checking out files:  96% (5765/6005)   
Checking out files:  97% (5825/6005)   
Checking out files:  98% (5885/6005)   
Checking out files:  99% (5945/6005)   
Checking out files: 100% (6005/6005)   
Checking out files: 100% (6005/6005), done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '6b3d716e2b6472eb7189d3220552280ef3d832ce'
  COPY    RUNNER
    RUN test-mingw in qemu:fedora 
Packages installed:
PyYAML-3.12-5.fc27.x86_64
SDL-devel-1.2.15-29.fc27.x86_64
bc-1.07.1-3.fc27.x86_64
bison-3.0.4-8.fc27.x86_64
bzip2-1.0.6-24.fc27.x86_64
ccache-3.3.6-1.fc27.x86_64
clang-5.0.1-3.fc27.x86_64
findutils-4.6.0-16.fc27.x86_64
flex-2.6.1-5.fc27.x86_64
gcc-7.3.1-5.fc27.x86_64
gcc-c++-7.3.1-5.fc27.x86_64
gettext-0.19.8.1-12.fc27.x86_64
git-2.14.3-3.fc27.x86_64
glib2-devel-2.54.3-2.fc27.x86_64
hostname-3.18-4.fc27.x86_64
libaio-devel-0.3.110-9.fc27.x86_64
libasan-7.3.1-5.fc27.x86_64
libfdt-devel-1.4.6-1.fc27.x86_64
libubsan-7.3.1-5.fc27.x86_64
llvm-5.0.1-3.fc27.x86_64
make-4.2.1-4.fc27.x86_64
mingw32-SDL-1.2.15-9.fc27.noarch
mingw32-bzip2-1.0.6-9.fc27.noarch
mingw32-curl-7.54.1-2.fc27.noarch
mingw32-glib2-2.54.1-1.fc27.noarch
mingw32-gmp-6.1.2-2.fc27.noarch
mingw32-gnutls-3.5.13-2.fc27.noarch
mingw32-gtk2-2.24.31-4.fc27.noarch
mingw32-gtk3-3.22.16-1.fc27.noarch
mingw32-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw32-libpng-1.6.29-2.fc27.noarch
mingw32-libssh2-1.8.0-3.fc27.noarch
mingw32-libtasn1-4.13-1.fc27.noarch
mingw32-nettle-3.3-3.fc27.noarch
mingw32-pixman-0.34.0-3.fc27.noarch
mingw32-pkg-config-0.28-9.fc27.x86_64
mingw64-SDL-1.2.15-9.fc27.noarch
mingw64-bzip2-1.0.6-9.fc27.noarch
mingw64-curl-7.54.1-2.fc27.noarch
mingw64-glib2-2.54.1-1.fc27.noarch
mingw64-gmp-6.1.2-2.fc27.noarch
mingw64-gnutls-3.5.13-2.fc27.noarch
mingw64-gtk2-2.24.31-4.fc27.noarch
mingw64-gtk3-3.22.16-1.fc27.noarch
mingw64-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw64-libpng-1.6.29-2.fc27.noarch
mingw64-libssh2-1.8.0-3.fc27.noarch
mingw64-libtasn1-4.13-1.fc27.noarch
mingw64-nettle-3.3-3.fc27.noarch
mingw64-pixman-0.34.0-3.fc27.noarch
mingw64-pkg-config-0.28-9.fc27.x86_64
nettle-devel-3.4-1.fc27.x86_64
perl-5.26.1-403.fc27.x86_64
pixman-devel-0.34.0-4.fc27.x86_64
python3-3.6.2-13.fc27.x86_64
sparse-0.5.1-2.fc27.x86_64
tar-1.29-7.fc27.x86_64
which-2.21-4.fc27.x86_64
zlib-devel-1.2.11-4.fc27.x86_64

Environment variables:
TARGET_LIST=
PACKAGES=ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname     glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel     gcc gcc-c++ llvm clang make perl which bc findutils libaio-devel     nettle-devel libasan libubsan     mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config     mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1     mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2     mingw32-bzip2     mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config     mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1     mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2     mingw64-bzip2
J=8
V=
HOSTNAME=8bdd59f5a499
DEBUG=
SHOW_ENV=1
PWD=/
HOME=/root
CCACHE_DIR=/var/tmp/ccache
DISTTAG=f27container
QEMU_CONFIGURE_OPTS=--python=/usr/bin/python3
FGC=f27
TEST_DIR=/tmp/qemu-test
SHLVL=1
FEATURES=mingw clang pyyaml asan dtc
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAKEFLAGS= -j8
EXTRA_CONFIGURE_OPTS=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --python=/usr/bin/python3 --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix    /tmp/qemu-test/install
BIOS directory    /tmp/qemu-test/install
firmware path     /tmp/qemu-test/install/share/qemu-firmware
binary directory  /tmp/qemu-test/install
library directory /tmp/qemu-test/install/lib
module directory  /tmp/qemu-test/install/lib
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory  /tmp/qemu-test/install
local state directory   queried at runtime
Windows SDK       no
Source path       /tmp/qemu-test/src
GIT binary        git
GIT submodules    
C compiler        x86_64-w64-mingw32-gcc
Host C compiler   cc
C++ compiler      x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1  -I$(SRC_PATH)/dtc/libfdt -Werror -DHAS_LIBSSH2_SFTP_FSYNC -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -I/usr/x86_64-w64-mingw32/sys-root/mingw/include   -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 
LDFLAGS           -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g 
make              make
install           install
python            /usr/bin/python3 -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (1.2.15)
GTK support       yes (2.24.31)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    no
virgl support     no
curl support      yes
mingw32 support   yes
Audio drivers     dsound
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
Multipath support no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support no
Install blobs     yes
KVM support       no
HAX support       yes
HVF support       no
WHPX support      no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support no
RDMA support      no
fdt support       yes
preadv support    no
fdatasync         no
madvise           no
posix_madvise     no
posix_memalign    no
libcap-ng support no
vhost-net support no
vhost-crypto support no
vhost-scsi support no
vhost-vsock support no
vhost-user support no
Trace backends    simple
Trace output file trace-<pid>
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info yes
QGA MSI support   no
seccomp support   no
coroutine backend win32
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
TPM emulator      no
QOM debugging     yes
Live block migration yes
lzo support       no
snappy support    no
bzip2 support     yes
NUMA host support no
libxml2           no
tcmalloc support  no
jemalloc support  no
avx2 optimization yes
replication support yes
VxHS block device no
capstone          no

WARNING: Use of GTK 2.0 is deprecated and will be removed in
WARNING: future releases. Please switch to using GTK 3.0

WARNING: Use of SDL 1.2 is deprecated and will be removed in
WARNING: future releases. Please switch to using SDL 2.0
mkdir -p dtc/libfdt
mkdir -p dtc/tests
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     config-host.h
  GEN     qemu-options.def
  GEN     qapi-gen
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     trace/generated-helpers.c
  GEN     aarch64-softmmu/config-devices.mak
  GEN     module_block.h
  GEN     ui/input-keymap-atset1-to-qcode.c
  GEN     ui/input-keymap-linux-to-qcode.c
  GEN     ui/input-keymap-qcode-to-atset1.c
  GEN     ui/input-keymap-qcode-to-atset2.c
  GEN     ui/input-keymap-qcode-to-atset3.c
  GEN     ui/input-keymap-qcode-to-linux.c
  GEN     ui/input-keymap-qcode-to-qnum.c
  GEN     ui/input-keymap-qcode-to-sun.c
  GEN     ui/input-keymap-qnum-to-qcode.c
  GEN     ui/input-keymap-usb-to-qcode.c
  GEN     ui/input-keymap-win32-to-qcode.c
  GEN     ui/input-keymap-x11-to-qcode.c
  GEN     ui/input-keymap-xorgevdev-to-qcode.c
  GEN     ui/input-keymap-xorgkbd-to-qcode.c
  GEN     ui/input-keymap-xorgxquartz-to-qcode.c
  GEN     ui/input-keymap-xorgxwin-to-qcode.c
  GEN     tests/test-qapi-gen
  GEN     trace-root.h
  GEN     util/trace.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/rdma/trace.h
  GEN     hw/rdma/vmw/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/misc/macio/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/sparc64/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/pci-host/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/hppa/trace.h
  GEN     hw/xen/trace.h
  GEN     hw/ide/trace.h
  GEN     hw/tpm/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/sparc/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     nbd/trace.h
  GEN     scsi/trace.h
  GEN     trace-root.c
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     block/trace.c
  GEN     chardev/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/rdma/trace.c
  GEN     hw/rdma/vmw/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/misc/macio/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/sparc64/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/pci-host/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     hw/hppa/trace.c
  GEN     hw/xen/trace.c
  GEN     hw/ide/trace.c
  GEN     hw/tpm/trace.c
  GEN     ui/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     nbd/trace.c
  GEN     scsi/trace.c
  GEN     config-all-devices.mak
	 DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
	 DEP /tmp/qemu-test/src/dtc/tests/trees.S
	 DEP /tmp/qemu-test/src/dtc/tests/testutils.c
	 DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
	 DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
	 DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/check_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
	 DEP /tmp/qemu-test/src/dtc/tests/overlay.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
	 DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
	 DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
	 DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
	 DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
	 DEP /tmp/qemu-test/src/dtc/tests/incbin.c
	 DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
	 DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
	 DEP /tmp/qemu-test/src/dtc/tests/path-references.c
	 DEP /tmp/qemu-test/src/dtc/tests/references.c
	 DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/set_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
	 DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
	 DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
	 DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
	 DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
	 DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
	 DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/notfound.c
	 DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
	 DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/getprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/find_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/root_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
	 DEP /tmp/qemu-test/src/dtc/fdtoverlay.c
	 DEP /tmp/qemu-test/src/dtc/util.c
	 DEP /tmp/qemu-test/src/dtc/fdtput.c
	 DEP /tmp/qemu-test/src/dtc/fdtdump.c
	 DEP /tmp/qemu-test/src/dtc/fdtget.c
	 LEX convert-dtsv0-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/srcpos.c
	 BISON dtc-parser.tab.c
	 LEX dtc-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/treesource.c
	 DEP /tmp/qemu-test/src/dtc/livetree.c
	 DEP /tmp/qemu-test/src/dtc/fstree.c
	 DEP /tmp/qemu-test/src/dtc/flattree.c
	 DEP /tmp/qemu-test/src/dtc/dtc.c
	 DEP /tmp/qemu-test/src/dtc/data.c
	 DEP /tmp/qemu-test/src/dtc/checks.c
	 DEP convert-dtsv0-lexer.lex.c
	 DEP dtc-parser.tab.c
	 DEP dtc-lexer.lex.c
	CHK version_gen.h
	UPD version_gen.h
	 DEP /tmp/qemu-test/src/dtc/util.c
	 CC libfdt/fdt.o
	 CC libfdt/fdt_wip.o
	 CC libfdt/fdt_ro.o
	 CC libfdt/fdt_sw.o
	 CC libfdt/fdt_empty_tree.o
	 CC libfdt/fdt_strerror.o
	 CC libfdt/fdt_addresses.o
	 CC libfdt/fdt_rw.o
	 CC libfdt/fdt_overlay.o
	 AR libfdt/libfdt.a
x86_64-w64-mingw32-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
  RC      version.o
mkdir -p dtc/libfdt
mkdir -p dtc/tests
  GEN     qga/qapi-generated/qapi-gen
  CC      qapi/qapi-builtin-types.o
  CC      qapi/qapi-types-char.o
  CC      qapi/qapi-types.o
  CC      qapi/qapi-types-block.o
  CC      qapi/qapi-types-block-core.o
  CC      qapi/qapi-types-common.o
  CC      qapi/qapi-types-crypto.o
  CC      qapi/qapi-types-introspect.o
  CC      qapi/qapi-types-migration.o
  CC      qapi/qapi-types-misc.o
  CC      qapi/qapi-types-net.o
  CC      qapi/qapi-types-rocker.o
  CC      qapi/qapi-types-run-state.o
  CC      qapi/qapi-types-sockets.o
  CC      qapi/qapi-types-tpm.o
  CC      qapi/qapi-types-trace.o
  CC      qapi/qapi-types-transaction.o
  CC      qapi/qapi-types-ui.o
  CC      qapi/qapi-builtin-visit.o
  CC      qapi/qapi-visit.o
  CC      qapi/qapi-visit-block-core.o
  CC      qapi/qapi-visit-block.o
  CC      qapi/qapi-visit-char.o
  CC      qapi/qapi-visit-common.o
  CC      qapi/qapi-visit-crypto.o
  CC      qapi/qapi-visit-introspect.o
  CC      qapi/qapi-visit-migration.o
  CC      qapi/qapi-visit-misc.o
  CC      qapi/qapi-visit-net.o
  CC      qapi/qapi-visit-rocker.o
  CC      qapi/qapi-visit-run-state.o
  CC      qapi/qapi-visit-sockets.o
  CC      qapi/qapi-visit-tpm.o
  CC      qapi/qapi-visit-trace.o
  CC      qapi/qapi-visit-transaction.o
  CC      qapi/qapi-visit-ui.o
  CC      qapi/qapi-events.o
  CC      qapi/qapi-events-block-core.o
  CC      qapi/qapi-events-block.o
  CC      qapi/qapi-events-char.o
  CC      qapi/qapi-events-common.o
  CC      qapi/qapi-events-crypto.o
  CC      qapi/qapi-events-introspect.o
  CC      qapi/qapi-events-migration.o
  CC      qapi/qapi-events-misc.o
  CC      qapi/qapi-events-net.o
  CC      qapi/qapi-events-rocker.o
  CC      qapi/qapi-events-run-state.o
  CC      qapi/qapi-events-sockets.o
  CC      qapi/qapi-events-tpm.o
  CC      qapi/qapi-events-trace.o
  CC      qapi/qapi-events-transaction.o
  CC      qapi/qapi-events-ui.o
  CC      qapi/qapi-introspect.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/string-output-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qnum.o
  CC      qobject/qstring.o
  CC      qobject/qdict.o
  CC      qobject/qlist.o
  CC      qobject/qbool.o
  CC      qobject/qlit.o
  CC      qobject/qobject.o
  CC      qobject/qjson.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/json-parser.o
  CC      trace/simple.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/async.o
  CC      util/aio-wait.o
  CC      util/thread-pool.o
  CC      util/qemu-timer.o
  CC      util/main-loop.o
  CC      util/iohandler.o
  CC      util/aio-win32.o
  CC      util/event_notifier-win32.o
  CC      util/oslib-win32.o
  CC      util/qemu-thread-win32.o
  CC      util/path.o
  CC      util/envlist.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/bitmap.o
  CC      util/bitops.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/uri.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-win32.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/log.o
  CC      util/pagesize.o
  CC      util/qdist.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      crypto/trace.o
  CC      io/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/rdma/trace.o
  CC      hw/rdma/vmw/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/misc/macio/trace.o
  CC      hw/usb/trace.o
  CC      hw/scsi/trace.o
  CC      hw/nvram/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sparc64/trace.o
  CC      hw/sd/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/ppc/trace.o
  CC      hw/pci/trace.o
  CC      hw/pci-host/trace.o
  CC      hw/s390x/trace.o
  CC      hw/vfio/trace.o
  CC      hw/acpi/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/hppa/trace.o
  CC      hw/xen/trace.o
  CC      hw/ide/trace.o
  CC      ui/trace.o
  CC      hw/tpm/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/arm/trace.o
  CC      target/i386/trace.o
  CC      target/mips/trace.o
  CC      target/sparc/trace.o
  CC      target/s390x/trace.o
  CC      target/ppc/trace.o
  CC      qom/trace.o
  CC      linux-user/trace.o
  CC      qapi/trace.o
  CC      accel/tcg/trace.o
  CC      accel/kvm/trace.o
  CC      nbd/trace.o
  CC      scsi/trace.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/clock-warp.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/replay.o
  CC      stubs/runstate-check.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/tpm.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/fd-register.o
  CC      stubs/qmp_pc_dimm.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      stubs/pci-host-piix.o
  CC      stubs/ram-block.o
  GEN     qemu-img-cmds.h
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-win32.o
  CC      block/win32-aio.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/create.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/accounting.o
  CC      block/dirty-bitmap.o
  CC      block/write-threshold.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/throttle.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      scsi/utils.o
  CC      block/curl.o
  CC      block/dmg-bz2.o
  CC      block/ssh.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/hmac-nettle.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o
  CC      crypto/secret.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-qcow.o
  CC      crypto/block-luks.o
  CC      io/channel.o
  CC      io/channel-buffer.o
  CC      io/channel-command.o
  CC      io/channel-file.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-websock.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/net-listener.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  CC      qemu-io.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      os-win32.o
  CC      bt-host.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qapi/qapi-commands.o
  CC      qapi/qapi-commands-block-core.o
  CC      qapi/qapi-commands-block.o
  CC      qapi/qapi-commands-char.o
  CC      qapi/qapi-commands-common.o
  CC      qapi/qapi-commands-crypto.o
  CC      qapi/qapi-commands-introspect.o
  CC      qapi/qapi-commands-migration.o
  CC      qapi/qapi-commands-misc.o
  CC      qapi/qapi-commands-net.o
  CC      qapi/qapi-commands-rocker.o
  CC      qapi/qapi-commands-run-state.o
  CC      qapi/qapi-commands-sockets.o
  CC      qapi/qapi-commands-tpm.o
  CC      qapi/qapi-commands-trace.o
  CC      qapi/qapi-commands-transaction.o
  CC      qapi/qapi-commands-ui.o
  CC      hmp.o
  CC      qmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/dsoundaudio.o
  CC      audio/audio_win_int.o
  CC      audio/wavcapture.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      backends/cryptodev-vhost.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/i386.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/debugcon.o
  CC      hw/char/imx_serial.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/qdev-fw.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/stream.o
  CC      hw/core/ptimer.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/split-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/cpu/core.o
  CC      hw/display/ads7846.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/pl110.o
  CC      hw/display/sii9022.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/tc6393xb.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/ide/ich.o
  CC      hw/ide/ahci-allwinner.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/xlnx-pmu-iomod-intc.o
  CC      hw/intc/xlnx-zynqmp-ipi.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/imx_gpcv2.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/isa-bus.o
  CC      hw/isa/apm.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/unimp.o
  CC      hw/misc/vmcoreinfo.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/xgmac.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/net/can/can_sja1000.o
  CC      hw/net/can/can_kvaser_pci.o
  CC      hw/net/can/can_pcm3680_pci.o
  CC      hw/net/can/can_mioe3680_pci.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/eeprom_at24c.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pcie_pci_bridge.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci-host/designware.o
  CC      hw/pci/pci.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msix.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdmmc-internal.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/ssi/mss-spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/i8254.o
  CC      hw/timer/pl031.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/xlnx-zynqmp-rtc.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/mss-timer.o
  CC      hw/tpm/tpm_util.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/tpm/tpm_crb.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/host-stub.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/colo-comm.o
  CC      migration/savevm.o
  CC      migration/colo.o
  CC      migration/colo-failover.o
  CC      migration/vmstate.o
  CC      migration/vmstate-types.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/dump.o
  CC      net/socket.o
  CC      net/eth.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap-win32.o
  CC      net/can/can_core.o
  CC      net/can/can_host.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/sbuf.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/sdl.o
  CC      ui/sdl_zoom.o
  CC      ui/gtk.o
  CC      chardev/char.o
  CC      chardev/char-console.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-pipe.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  CC      chardev/char-win.o
  CC      chardev/char-win-stdio.o
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/commands-win32.o
  CC      qga/channel-win32.o
  CC      qga/service-win32.o
  AS      optionrom/multiboot.o
  CC      qga/vss-win32.o
  CC      qga/qapi-generated/qga-qapi-types.o
  AS      optionrom/linuxboot.o
  CC      optionrom/linuxboot_dma.o
  CC      qga/qapi-generated/qga-qapi-visit.o
  CC      qga/qapi-generated/qga-qapi-commands.o
  AR      libqemuutil.a
  AS      optionrom/kvmvapic.o
  CC      qemu-img.o
  BUILD   optionrom/multiboot.img
  BUILD   optionrom/linuxboot.img
  BUILD   optionrom/linuxboot_dma.img
  BUILD   optionrom/kvmvapic.img
  BUILD   optionrom/multiboot.raw
  BUILD   optionrom/linuxboot.raw
  BUILD   optionrom/linuxboot_dma.raw
  BUILD   optionrom/kvmvapic.raw
  SIGN    optionrom/linuxboot.bin
  SIGN    optionrom/multiboot.bin
  SIGN    optionrom/linuxboot_dma.bin
  SIGN    optionrom/kvmvapic.bin
  LINK    qemu-ga.exe
  LINK    qemu-io.exe
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     x86_64-softmmu/config-target.h
  GEN     aarch64-softmmu/config-target.h
  LINK    qemu-img.exe
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  CC      x86_64-softmmu/tcg/tcg.o
  CC      x86_64-softmmu/exec.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      x86_64-softmmu/tcg/tcg-op-vec.o
  CC      x86_64-softmmu/tcg/tcg-op-gvec.o
  CC      aarch64-softmmu/exec.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      x86_64-softmmu/disas.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/tcg/tcg-op-vec.o
  CC      aarch64-softmmu/tcg/tcg-op-gvec.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      x86_64-softmmu/arch_init.o
  CC      x86_64-softmmu/cpus.o
  CC      x86_64-softmmu/monitor.o
  CC      x86_64-softmmu/gdbstub.o
  CC      x86_64-softmmu/balloon.o
  CC      x86_64-softmmu/ioport.o
  CC      x86_64-softmmu/numa.o
  CC      x86_64-softmmu/qtest.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/memory.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      x86_64-softmmu/memory_mapping.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/disas.o
  CC      aarch64-softmmu/arch_init.o
  CC      aarch64-softmmu/cpus.o
  CC      x86_64-softmmu/dump.o
  CC      x86_64-softmmu/migration/ram.o
  CC      x86_64-softmmu/migration/nvdimm.o
  CC      x86_64-softmmu/accel/accel.o
  CC      x86_64-softmmu/accel/stubs/hvf-stub.o
/tmp/qemu-test/src/migration/nvdimm.c: In function 'nvdimm_state_save_dependency':
/tmp/qemu-test/src/migration/nvdimm.c:244:70: error: format '%lx' expects argument of type 'long unsigned int', but argument 2 has type 'int64_t {aka long long int}' [-Werror=format=]
         error_report("save file dependency failed, depend_offset = %lx "
                                                                    ~~^
                                                                    %llx
/tmp/qemu-test/src/migration/nvdimm.c:246:22:
                      nvdimm_state->depend_offset,
                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~                      
/tmp/qemu-test/src/migration/nvdimm.c:244:22: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=]
         error_report("save file dependency failed, depend_offset = %lx "
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/migration/nvdimm.c:247:22:
                      nvdimm_state->depend_size, ret);
                      ~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/migration/nvdimm.c:245:40: note: format string is defined here
                      "depend_size is %ld, ret is %d",
                                      ~~^
                                      %lld
cc1: all warnings being treated as errors
make[1]: *** [/tmp/qemu-test/src/rules.mak:66: migration/nvdimm.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/monitor.o
  CC      aarch64-softmmu/balloon.o
  CC      aarch64-softmmu/gdbstub.o
  CC      aarch64-softmmu/ioport.o
  CC      aarch64-softmmu/numa.o
  CC      aarch64-softmmu/qtest.o
  CC      aarch64-softmmu/memory.o
make: *** [Makefile:476: subdir-x86_64-softmmu] Error 2
make: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/memory_mapping.o
  CC      aarch64-softmmu/dump.o
  CC      aarch64-softmmu/migration/ram.o
  CC      aarch64-softmmu/migration/nvdimm.o
  CC      aarch64-softmmu/accel/accel.o
  CC      aarch64-softmmu/accel/stubs/hax-stub.o
  CC      aarch64-softmmu/accel/stubs/hvf-stub.o
/tmp/qemu-test/src/migration/nvdimm.c: In function 'nvdimm_state_save_dependency':
/tmp/qemu-test/src/migration/nvdimm.c:244:70: error: format '%lx' expects argument of type 'long unsigned int', but argument 2 has type 'int64_t {aka long long int}' [-Werror=format=]
         error_report("save file dependency failed, depend_offset = %lx "
                                                                    ~~^
                                                                    %llx
/tmp/qemu-test/src/migration/nvdimm.c:246:22:
                      nvdimm_state->depend_offset,
                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~                      
/tmp/qemu-test/src/migration/nvdimm.c:244:22: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=]
         error_report("save file dependency failed, depend_offset = %lx "
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/migration/nvdimm.c:247:22:
                      nvdimm_state->depend_size, ret);
                      ~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/qemu-test/src/migration/nvdimm.c:245:40: note: format string is defined here
                      "depend_size is %ld, ret is %d",
                                      ~~^
                                      %lld
cc1: all warnings being treated as errors
make[1]: *** [/tmp/qemu-test/src/rules.mak:66: migration/nvdimm.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:476: subdir-aarch64-softmmu] Error 2
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 261, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 229, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 147, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=8a0c5a0e272711e8872452540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-fe8degv8/src/docker-src.2018-03-13-21.32.09.16131:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-fe8degv8/src'
make: *** [tests/docker/Makefile.include:163: docker-run-test-mingw@fedora] Error 2

real	4m6.219s
user	0m4.555s
sys	0m4.041s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
[Qemu-devel] [PATCH 01/10] RFC: Add save and support snapshot dependency function to block driver.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

We want to support incremental snapshot saving, this needs the file
system support dependency saving. Later snapshots may ref the dependent
snapshot's content, and most time should be cluster aligned.
Add a query function to check whether the file system support this, and
use the save_dependency function to do the real work.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 include/block/block_int.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index 64a5700..be1eca3 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -274,6 +274,15 @@ struct BlockDriver {
                                   const char *snapshot_id,
                                   const char *name,
                                   Error **errp);
+    int (*bdrv_snapshot_save_dependency)(BlockDriverState *bs,
+                                         const char *depend_snapshot_id,
+                                         int64_t depend_offset,
+                                         int64_t depend_size,
+                                         int64_t offset,
+                                         Error **errp);
+    int (*bdrv_snapshot_support_dependency)(BlockDriverState *bs,
+                                            int32_t *alignment);
+
     int (*bdrv_get_info)(BlockDriverState *bs, BlockDriverInfo *bdi);
     ImageInfoSpecific *(*bdrv_get_specific_info)(BlockDriverState *bs);
 
-- 
2.7.4


[Qemu-devel] [PATCH 02/10] RFC: Implement qcow2's snapshot dependent saving function.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

For qcow2 format, we can increase the cluster's reference count of
dependent snapshot content and link the offset to the L2 table of
the new snapshot point. This way can avoid obvious snapshot's dependent
relationship, so when we delete some snapshot point, just decrease the
cluster count and no need to check further.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 block/qcow2-snapshot.c | 154 +++++++++++++++++++++++++++++++++++++++++++++++++
 block/qcow2.c          |   2 +
 block/qcow2.h          |   7 +++
 3 files changed, 163 insertions(+)

diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index cee25f5..8e83084 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -736,3 +736,157 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,
 
     return 0;
 }
+
+int qcow2_snapshot_save_dependency(BlockDriverState *bs,
+                                   const char *depend_snapshot_id,
+                                   int64_t depend_offset,
+                                   int64_t depend_size,
+                                   int64_t offset,
+                                   Error **errp)
+{
+    int snapshot_index;
+    BDRVQcow2State *s = bs->opaque;
+    QCowSnapshot *sn;
+    int ret;
+    int64_t i;
+    int64_t total_bytes = depend_size;
+    int64_t depend_offset1, offset1;
+    uint64_t *depend_l1_table = NULL;
+    uint64_t depend_l1_bytes;
+    uint64_t *depend_l2_table = NULL;
+    uint64_t depend_l2_offset;
+    uint64_t depend_entry;
+    QCowL2Meta l2meta;
+
+    assert(bs->read_only == false);
+
+    if (depend_snapshot_id == NULL) {
+        return 0;
+    }
+
+    if (!QEMU_IS_ALIGNED(depend_offset,  s->cluster_size)) {
+        error_setg(errp, "Specified snapshot offset is not multiple of %u",
+                s->cluster_size);
+        return -EINVAL;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset,  s->cluster_size)) {
+        error_setg(errp, "Offset is not multiple of %u", s->cluster_size);
+        return -EINVAL;
+    }
+
+    if (!QEMU_IS_ALIGNED(depend_size,  s->cluster_size)) {
+        error_setg(errp, "depend_size is not multiple of %u", s->cluster_size);
+        return -EINVAL;
+    }
+
+    snapshot_index = find_snapshot_by_id_and_name(bs, NULL, depend_snapshot_id);
+    /* Search the snapshot */
+    if (snapshot_index < 0) {
+        error_setg(errp, "Can't find snapshot");
+        return -ENOENT;
+    }
+
+    sn = &s->snapshots[snapshot_index];
+    if (sn->disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) {
+        error_report("qcow2: depend on the snapshots with different disk "
+                "size is not implemented");
+        return -ENOTSUP;
+    }
+
+    /* Only can save dependency of snapshot's vmstate data */
+    depend_offset1 = depend_offset + qcow2_vm_state_offset(s);
+    offset1 = offset + qcow2_vm_state_offset(s);
+
+    depend_l1_bytes = s->l1_size * sizeof(uint64_t);
+    depend_l1_table = g_try_malloc0(depend_l1_bytes);
+    if (depend_l1_table == NULL) {
+        return -ENOMEM;
+    }
+
+    ret = bdrv_pread(bs->file, sn->l1_table_offset, depend_l1_table,
+                     depend_l1_bytes);
+    if (ret < 0) {
+        g_free(depend_l1_table);
+        goto out;
+    }
+    for (i = 0; i < depend_l1_bytes / sizeof(uint64_t); i++) {
+        be64_to_cpus(&depend_l1_table[i]);
+    }
+
+    while (total_bytes) {
+        assert(total_bytes > 0);
+        /* Find the cluster of depend */
+        depend_l2_offset =
+            depend_l1_table[depend_offset1 >> (s->l2_bits + s->cluster_bits)];
+        depend_l2_offset &= L1E_OFFSET_MASK;
+        if (depend_l2_offset == 0) {
+            ret = -EINVAL;
+            goto out;
+        }
+
+        if (offset_into_cluster(s, depend_l2_offset)) {
+            qcow2_signal_corruption(bs, true, -1, -1, "L2 table offset %#"
+                                    PRIx64 " unaligned (L1 index: %#"
+                                    PRIx64 ")",
+                                    depend_l2_offset,
+                                    depend_offset1 >>
+                                        (s->l2_bits + s->cluster_bits));
+            return -EIO;
+        }
+
+        ret = qcow2_cache_get(bs, s->l2_table_cache, depend_l2_offset,
+                              (void **)(&depend_l2_table));
+        if (ret < 0) {
+            goto out;
+        }
+
+        depend_entry =
+            be64_to_cpu(
+                depend_l2_table[offset_to_l2_index(s, depend_offset1)]);
+        if (depend_entry == 0) {
+            ret = -EINVAL;
+            qcow2_cache_put(s->l2_table_cache, (void **)(&depend_l2_table));
+            goto out;
+        }
+
+        memset(&l2meta, 0, sizeof(l2meta));
+        l2meta.offset = offset1;
+        l2meta.alloc_offset = (depend_entry & L2E_OFFSET_MASK);
+        l2meta.nb_clusters = 1;
+        /* Add a ref to this cluster */
+        ret = qcow2_update_cluster_refcount(
+                  bs, l2meta.alloc_offset >> s->cluster_bits,
+                  1, false, QCOW2_DISCARD_SNAPSHOT);
+        if (ret < 0) {
+            qcow2_cache_put(s->l2_table_cache, (void **)(&depend_l2_table));
+            goto out;
+        }
+
+        ret = qcow2_alloc_cluster_link_l2(bs, &l2meta);
+        if (ret < 0) {
+            qcow2_cache_put(s->l2_table_cache, (void **)(&depend_l2_table));
+            goto out;
+        }
+
+        total_bytes -= s->cluster_size;
+        offset1 += s->cluster_size;
+        depend_offset1 += s->cluster_size;
+
+        qcow2_cache_put(s->l2_table_cache, (void **)(&depend_l2_table));
+    }
+
+out:
+    g_free(depend_l1_table);
+    return ret;
+}
+
+int qcow2_snapshot_support_dependency(BlockDriverState *bs, int32_t *alignment)
+{
+    BDRVQcow2State *s = bs->opaque;
+    if (alignment) {
+        *alignment = s->cluster_size;
+    }
+
+    return 1;
+}
diff --git a/block/qcow2.c b/block/qcow2.c
index 071dc4d..9786ba4 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -4371,6 +4371,8 @@ BlockDriver bdrv_qcow2 = {
     .bdrv_snapshot_delete   = qcow2_snapshot_delete,
     .bdrv_snapshot_list     = qcow2_snapshot_list,
     .bdrv_snapshot_load_tmp = qcow2_snapshot_load_tmp,
+    .bdrv_snapshot_support_dependency = qcow2_snapshot_support_dependency,
+    .bdrv_snapshot_save_dependency = qcow2_snapshot_save_dependency,
     .bdrv_measure           = qcow2_measure,
     .bdrv_get_info          = qcow2_get_info,
     .bdrv_get_specific_info = qcow2_get_specific_info,
diff --git a/block/qcow2.h b/block/qcow2.h
index 1a84cc7..dc7ef45 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -640,6 +640,13 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,
 
 void qcow2_free_snapshots(BlockDriverState *bs);
 int qcow2_read_snapshots(BlockDriverState *bs);
+int qcow2_snapshot_save_dependency(BlockDriverState *bs,
+                                  const char *depend_snapshot_id,
+                                  int64_t depend_offset,
+                                  int64_t depend_size,
+                                  int64_t offset,
+                                  Error **errp);
+int qcow2_snapshot_support_dependency(BlockDriverState *bs, int32_t *alignment);
 
 /* qcow2-cache.c functions */
 Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables,
-- 
2.7.4


[Qemu-devel] [PATCH 03/10] RFC: Implement save and support snapshot dependency in block driver layer.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 block/snapshot.c         | 45 +++++++++++++++++++++++++++++++++++++++++++++
 include/block/snapshot.h |  7 +++++++
 2 files changed, 52 insertions(+)

diff --git a/block/snapshot.c b/block/snapshot.c
index eacc1f1..8cc40ac 100644
--- a/block/snapshot.c
+++ b/block/snapshot.c
@@ -401,6 +401,51 @@ int bdrv_snapshot_load_tmp_by_id_or_name(BlockDriverState *bs,
     return ret;
 }
 
+int bdrv_snapshot_save_dependency(BlockDriverState *bs,
+                                  const char *depend_snapshot_id,
+                                  int64_t depend_offset,
+                                  int64_t depend_size,
+                                  int64_t offset,
+                                  Error **errp)
+{
+    BlockDriver *drv = bs->drv;
+
+    if (!drv) {
+        return -ENOMEDIUM;
+    }
+
+    if (drv->bdrv_snapshot_save_dependency) {
+        return drv->bdrv_snapshot_save_dependency(bs, depend_snapshot_id,
+                                                  depend_offset, depend_size,
+                                                  offset, errp);
+    }
+
+    if (bs->file) {
+        return bdrv_snapshot_save_dependency(bs->file->bs, depend_snapshot_id,
+                                             depend_offset, depend_size,
+                                             offset, errp);
+    }
+
+    return -ENOTSUP;
+}
+
+int bdrv_snapshot_support_dependency(BlockDriverState *bs, int32_t *alignment)
+{
+    BlockDriver *drv = bs->drv;
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+        return 0;
+    }
+
+    if (drv->bdrv_snapshot_support_dependency) {
+        return drv->bdrv_snapshot_support_dependency(bs, alignment);
+    }
+
+    if (bs->file != NULL) {
+        return bdrv_snapshot_support_dependency(bs->file->bs, alignment);
+    }
+
+    return -ENOTSUP;
+}
 
 /* Group operations. All block drivers are involved.
  * These functions will properly handle dataplane (take aio_context_acquire
diff --git a/include/block/snapshot.h b/include/block/snapshot.h
index f73d109..e5bf06f 100644
--- a/include/block/snapshot.h
+++ b/include/block/snapshot.h
@@ -73,6 +73,13 @@ int bdrv_snapshot_load_tmp(BlockDriverState *bs,
 int bdrv_snapshot_load_tmp_by_id_or_name(BlockDriverState *bs,
                                          const char *id_or_name,
                                          Error **errp);
+int bdrv_snapshot_save_dependency(BlockDriverState *bs,
+                                  const char *depend_snapshot_id,
+                                  int64_t depend_offset,
+                                  int64_t depend_size,
+                                  int64_t offset,
+                                  Error **errp);
+int bdrv_snapshot_support_dependency(BlockDriverState *bs, int32_t *alignment);
 
 
 /* Group operations. All block drivers are involved.
-- 
2.7.4


[Qemu-devel] [PATCH 04/10] RFC: Set memory_region_set_log available for more client.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

We need to collect dirty log for nvdimm kind memory, need to enable
memory_region_set_log for more clients rather than just VGA.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 memory.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/memory.c b/memory.c
index e70b64b..4a8a2fe 100644
--- a/memory.c
+++ b/memory.c
@@ -1921,11 +1921,12 @@ void memory_region_set_log(MemoryRegion *mr, bool log, unsigned client)
     uint8_t mask = 1 << client;
     uint8_t old_logging;
 
-    assert(client == DIRTY_MEMORY_VGA);
-    old_logging = mr->vga_logging_count;
-    mr->vga_logging_count += log ? 1 : -1;
-    if (!!old_logging == !!mr->vga_logging_count) {
-        return;
+    if (client == DIRTY_MEMORY_VGA) {
+        old_logging = mr->vga_logging_count;
+        mr->vga_logging_count += log ? 1 : -1;
+        if (!!old_logging == !!mr->vga_logging_count) {
+            return;
+        }
     }
 
     memory_region_transaction_begin();
-- 
2.7.4


[Qemu-devel] [PATCH 05/10] RFC: Add memory region snapshot bitmap get function.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

We need to get the bitmap content of the snapshot when enable dirty
log trace for nvdimm.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 exec.c                  | 7 +++++++
 include/exec/memory.h   | 9 +++++++++
 include/exec/ram_addr.h | 2 ++
 memory.c                | 7 +++++++
 4 files changed, 25 insertions(+)

diff --git a/exec.c b/exec.c
index a9181e6..3d2bf0d 100644
--- a/exec.c
+++ b/exec.c
@@ -1235,6 +1235,13 @@ bool cpu_physical_memory_snapshot_get_dirty(DirtyBitmapSnapshot *snap,
     return false;
 }
 
+unsigned long *cpu_physical_memory_snapshot_get_dirty_bitmap
+     (DirtyBitmapSnapshot *snap)
+{
+    assert(snap);
+    return snap->dirty;
+}
+
 /* Called from RCU critical section */
 hwaddr memory_region_section_get_iotlb(CPUState *cpu,
                                        MemoryRegionSection *section,
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 31eae0a..f742995 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -1179,6 +1179,15 @@ bool memory_region_snapshot_get_dirty(MemoryRegion *mr,
                                       hwaddr addr, hwaddr size);
 
 /**
+ * memory_region_snapshot_get_dirty_bitmap: Get the dirty bitmap data of
+ * snapshot.
+ *
+ * @snap: the dirty bitmap snapshot
+ */
+unsigned long *memory_region_snapshot_get_dirty_bitmap
+     (DirtyBitmapSnapshot *snap);
+
+/**
  * memory_region_reset_dirty: Mark a range of pages as clean, for a specified
  *                            client.
  *
diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
index cf2446a..ce366c1 100644
--- a/include/exec/ram_addr.h
+++ b/include/exec/ram_addr.h
@@ -371,6 +371,8 @@ DirtyBitmapSnapshot *cpu_physical_memory_snapshot_and_clear_dirty
 bool cpu_physical_memory_snapshot_get_dirty(DirtyBitmapSnapshot *snap,
                                             ram_addr_t start,
                                             ram_addr_t length);
+unsigned long *cpu_physical_memory_snapshot_get_dirty_bitmap
+    (DirtyBitmapSnapshot *snap);
 
 static inline void cpu_physical_memory_clear_dirty_range(ram_addr_t start,
                                                          ram_addr_t length)
diff --git a/memory.c b/memory.c
index 4a8a2fe..68f17f0 100644
--- a/memory.c
+++ b/memory.c
@@ -1991,6 +1991,13 @@ DirtyBitmapSnapshot *memory_region_snapshot_and_clear_dirty(MemoryRegion *mr,
                 memory_region_get_ram_addr(mr) + addr, size, client);
 }
 
+unsigned long *memory_region_snapshot_get_dirty_bitmap
+     (DirtyBitmapSnapshot *snap)
+{
+    assert(snap);
+    return cpu_physical_memory_snapshot_get_dirty_bitmap(snap);
+}
+
 bool memory_region_snapshot_get_dirty(MemoryRegion *mr, DirtyBitmapSnapshot *snap,
                                       hwaddr addr, hwaddr size)
 {
-- 
2.7.4


[Qemu-devel] [PATCH 06/10] RFC: Add save dependency functions to qemu_file
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

When we save snapshot, we need qemu_file to support save dependency
operations. It should call brv_driver's save dependency functions
to implement these operations.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 migration/qemu-file.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++
 migration/qemu-file.h | 14 ++++++++++++
 migration/savevm.c    | 33 +++++++++++++++++++++++++---
 3 files changed, 105 insertions(+), 3 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 2ab2bf3..9d2a39a 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -46,10 +46,13 @@ struct QEMUFile {
     int buf_index;
     int buf_size; /* 0 when writing */
     uint8_t buf[IO_BUF_SIZE];
+    char ref_name_str[128]; /* maybe snapshot id */
 
     DECLARE_BITMAP(may_free, MAX_IOV_SIZE);
     struct iovec iov[MAX_IOV_SIZE];
     unsigned int iovcnt;
+    bool support_dependency;
+    int32_t dependency_aligment;
 
     int last_error;
 };
@@ -745,3 +748,61 @@ void qemu_file_set_blocking(QEMUFile *f, bool block)
         f->ops->set_blocking(f->opaque, block);
     }
 }
+
+void qemu_file_set_support_dependency(QEMUFile *f, int32_t alignment)
+{
+    f->dependency_aligment = alignment;
+    f->support_dependency = true;
+}
+
+bool qemu_file_is_support_dependency(QEMUFile *f, int32_t *alignment)
+{
+    if (f->support_dependency && alignment) {
+        *alignment = f->dependency_aligment;
+    }
+
+    return f->support_dependency;
+}
+
+/* This function set the reference name for snapshot usage. Sometimes it needs
+ * to depend on other snapshot's data to avoid redundance.
+ */
+bool qemu_file_set_ref_name(QEMUFile *f, const char *name)
+{
+    if (strlen(name) + 1 > sizeof(f->ref_name_str)) {
+        return false;
+    }
+
+    memcpy(f->ref_name_str, name, strlen(name) + 1);
+    return true;
+}
+
+ssize_t qemu_file_save_dependency(QEMUFile *f, int64_t depend_offset,
+                                  int64_t size)
+{
+    ssize_t ret;
+
+    if (f->support_dependency == false) {
+        return -1;
+    }
+
+    assert(f->ops->save_dependency);
+
+    if (!QEMU_IS_ALIGNED(depend_offset, f->dependency_aligment)) {
+        return -1;
+    }
+
+    qemu_fflush(f);
+
+    if (!QEMU_IS_ALIGNED(f->pos, f->dependency_aligment)) {
+        return -1;
+    }
+
+    ret = f->ops->save_dependency(f->opaque, f->ref_name_str,
+                                  depend_offset, size, f->pos);
+    if (ret > 0) {
+        f->pos += size;
+    }
+
+    return ret;
+}
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index aae4e5e..137b917 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -57,6 +57,14 @@ typedef ssize_t (QEMUFileWritevBufferFunc)(void *opaque, struct iovec *iov,
                                            int iovcnt, int64_t pos);
 
 /*
+ * This function add reference to the dependency data in snapshot specified by
+ * ref_name_str to this file's offset
+ */
+typedef ssize_t (QEMUFileSaveDependencyFunc)(void *opaque, const char *name,
+                                             int64_t depend_offset,
+                                             int64_t offset, int64_t size);
+
+/*
  * This function provides hooks around different
  * stages of RAM migration.
  * 'opaque' is the backend specific data in QEMUFile
@@ -104,6 +112,7 @@ typedef struct QEMUFileOps {
     QEMUFileWritevBufferFunc *writev_buffer;
     QEMURetPathFunc *get_return_path;
     QEMUFileShutdownFunc *shut_down;
+    QEMUFileSaveDependencyFunc *save_dependency;
 } QEMUFileOps;
 
 typedef struct QEMUFileHooks {
@@ -153,6 +162,11 @@ int qemu_file_shutdown(QEMUFile *f);
 QEMUFile *qemu_file_get_return_path(QEMUFile *f);
 void qemu_fflush(QEMUFile *f);
 void qemu_file_set_blocking(QEMUFile *f, bool block);
+bool qemu_file_set_ref_name(QEMUFile *f, const char *name);
+void qemu_file_set_support_dependency(QEMUFile *f, int32_t alignment);
+bool qemu_file_is_support_dependency(QEMUFile *f, int32_t *alignment);
+ssize_t qemu_file_save_dependency(QEMUFile *f, int64_t depend_offset,
+                                  int64_t size);
 
 size_t qemu_get_counted_string(QEMUFile *f, char buf[256]);
 
diff --git a/migration/savevm.c b/migration/savevm.c
index 358c5b5..1bbd6aa 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -196,6 +196,20 @@ static ssize_t block_writev_buffer(void *opaque, struct iovec *iov, int iovcnt,
     return qiov.size;
 }
 
+static ssize_t block_save_dependency(void *opaque, const char *id_name,
+                                     int64_t depend_offset,
+                                     int64_t offset, int64_t size)
+{
+    int ret = bdrv_snapshot_save_dependency(opaque, id_name,
+                                            depend_offset, offset,
+                                            size, NULL);
+    if (ret < 0) {
+        return ret;
+    }
+
+    return size;
+}
+
 static ssize_t block_get_buffer(void *opaque, uint8_t *buf, int64_t pos,
                                 size_t size)
 {
@@ -213,15 +227,28 @@ static const QEMUFileOps bdrv_read_ops = {
 };
 
 static const QEMUFileOps bdrv_write_ops = {
-    .writev_buffer  = block_writev_buffer,
-    .close          = bdrv_fclose
+    .writev_buffer   = block_writev_buffer,
+    .close           = bdrv_fclose,
+    .save_dependency = block_save_dependency
 };
 
 static QEMUFile *qemu_fopen_bdrv(BlockDriverState *bs, int is_writable)
 {
+    int ret = 0;
+    int32_t alignment = 0;
+    QEMUFile *f = NULL;
+
     if (is_writable) {
-        return qemu_fopen_ops(bs, &bdrv_write_ops);
+        f = qemu_fopen_ops(bs, &bdrv_write_ops);
+
+        ret = bdrv_snapshot_support_dependency(bs, &alignment);
+        if (ret > 0) {
+            qemu_file_set_support_dependency(f, alignment);
+        }
+
+        return f;
     }
+
     return qemu_fopen_ops(bs, &bdrv_read_ops);
 }
 
-- 
2.7.4


[Qemu-devel] [PATCH 07/10] RFC: Add get_current_snapshot_info to get the snapshot state.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

We need to know the snapshot saving information when we do dependent
snapshot saving, e.g the name of previous snapshot. Add this global
function to query the snapshot status is usable.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 include/migration/snapshot.h |  3 +++
 migration/savevm.c           | 27 +++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/include/migration/snapshot.h b/include/migration/snapshot.h
index c85b6ec..0b950ce 100644
--- a/include/migration/snapshot.h
+++ b/include/migration/snapshot.h
@@ -15,7 +15,10 @@
 #ifndef QEMU_MIGRATION_SNAPSHOT_H
 #define QEMU_MIGRATION_SNAPSHOT_H
 
+#include "block/snapshot.h"
+
 int save_snapshot(const char *name, Error **errp);
 int load_snapshot(const char *name, Error **errp);
+int get_current_snapshot_info(QEMUSnapshotInfo *sn);
 
 #endif
diff --git a/migration/savevm.c b/migration/savevm.c
index 1bbd6aa..3a9b904 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -2212,6 +2212,29 @@ int qemu_loadvm_state(QEMUFile *f)
     return ret;
 }
 
+static int in_snap_saving;
+static QEMUSnapshotInfo in_snap_saving_sn;
+
+int get_current_snapshot_info(QEMUSnapshotInfo *sn)
+{
+    if (in_snap_saving && sn) {
+        memcpy(sn, &in_snap_saving_sn, sizeof(QEMUSnapshotInfo));
+    }
+
+    return in_snap_saving;
+}
+
+static void set_current_snapshot_info(QEMUSnapshotInfo *sn)
+{
+    if (sn) {
+        memcpy(&in_snap_saving_sn, sn, sizeof(QEMUSnapshotInfo));
+        in_snap_saving = 1;
+    } else {
+        memset(&in_snap_saving_sn, 0, sizeof(QEMUSnapshotInfo));
+        in_snap_saving = 0;
+    }
+}
+
 int save_snapshot(const char *name, Error **errp)
 {
     BlockDriverState *bs, *bs1;
@@ -2282,6 +2305,8 @@ int save_snapshot(const char *name, Error **errp)
         strftime(sn->name, sizeof(sn->name), "vm-%Y%m%d%H%M%S", &tm);
     }
 
+    set_current_snapshot_info(sn);
+
     /* save the VM state */
     f = qemu_fopen_bdrv(bs, 1);
     if (!f) {
@@ -2313,6 +2338,8 @@ int save_snapshot(const char *name, Error **errp)
     ret = 0;
 
  the_end:
+    set_current_snapshot_info(NULL);
+
     if (aio_context) {
         aio_context_release(aio_context);
     }
-- 
2.7.4


[Qemu-devel] [PATCH 08/10] RFC: Add a section_id parameter to save_live_iterate call.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

We need to know the section_id when we do snapshot saving.
Add a parameter to save_live_iterate function call.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 hw/ppc/spapr.c               | 2 +-
 hw/s390x/s390-stattrib.c     | 2 +-
 include/migration/register.h | 2 +-
 migration/block.c            | 2 +-
 migration/ram.c              | 2 +-
 migration/savevm.c           | 2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 7e1c858..4cde4f4 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1974,7 +1974,7 @@ static int htab_save_later_pass(QEMUFile *f, sPAPRMachineState *spapr,
 #define MAX_ITERATION_NS    5000000 /* 5 ms */
 #define MAX_KVM_BUF_SIZE    2048
 
-static int htab_save_iterate(QEMUFile *f, void *opaque)
+static int htab_save_iterate(QEMUFile *f, void *opaque, int section_id)
 {
     sPAPRMachineState *spapr = opaque;
     int fd;
diff --git a/hw/s390x/s390-stattrib.c b/hw/s390x/s390-stattrib.c
index adf07ef..18ece84 100644
--- a/hw/s390x/s390-stattrib.c
+++ b/hw/s390x/s390-stattrib.c
@@ -246,7 +246,7 @@ static int cmma_save(QEMUFile *f, void *opaque, int final)
     return ret;
 }
 
-static int cmma_save_iterate(QEMUFile *f, void *opaque)
+static int cmma_save_iterate(QEMUFile *f, void *opaque, int section_id)
 {
     return cmma_save(f, opaque, 0);
 }
diff --git a/include/migration/register.h b/include/migration/register.h
index f4f7bdc..7f7df2c 100644
--- a/include/migration/register.h
+++ b/include/migration/register.h
@@ -31,7 +31,7 @@ typedef struct SaveVMHandlers {
      * use data that is local to the migration thread or protected
      * by other locks.
      */
-    int (*save_live_iterate)(QEMUFile *f, void *opaque);
+    int (*save_live_iterate)(QEMUFile *f, void *opaque, int section_id);
 
     /* This runs outside the iothread lock!  */
     int (*save_setup)(QEMUFile *f, void *opaque);
diff --git a/migration/block.c b/migration/block.c
index 1f03946..6d4c8a3 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -755,7 +755,7 @@ static int block_save_setup(QEMUFile *f, void *opaque)
     return ret;
 }
 
-static int block_save_iterate(QEMUFile *f, void *opaque)
+static int block_save_iterate(QEMUFile *f, void *opaque, int section_id)
 {
     int ret;
     int64_t last_ftell = qemu_ftell(f);
diff --git a/migration/ram.c b/migration/ram.c
index 3b6c077..d1db422 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2249,7 +2249,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
  * @f: QEMUFile where to send the data
  * @opaque: RAMState pointer
  */
-static int ram_save_iterate(QEMUFile *f, void *opaque)
+static int ram_save_iterate(QEMUFile *f, void *opaque, int section_id)
 {
     RAMState **temp = opaque;
     RAMState *rs = *temp;
diff --git a/migration/savevm.c b/migration/savevm.c
index 3a9b904..ce4133a 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1072,7 +1072,7 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy)
 
         save_section_header(f, se, QEMU_VM_SECTION_PART);
 
-        ret = se->ops->save_live_iterate(f, se->opaque);
+        ret = se->ops->save_live_iterate(f, se->opaque, se->section_id);
         trace_savevm_section_end(se->idstr, se->section_id, ret);
         save_section_footer(f, se);
 
-- 
2.7.4


[Qemu-devel] [PATCH 09/10] RFC: Add nvdimm snapshot saving to migration.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

The nvdimm size is huge, sometimes is more than 256G or even more.
This is a huge burden for snapshot saving. One snapshot point with
nvdimm may occupy more than 50G disk space even with compression
enabled.
We need to introduce dependent snapshot manner to solve this problem.
The first snapshot point should always be saved completely, and enable
dirty log trace after saving for nvdimm memory region. The later snapshot
point should add the reference to previous snapshot's nvdimm data and
just saving dirty pages. This can save a lot of disk and time if the
snapshot operations are triggered frequently.

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 Makefile.target          |    1 +
 include/migration/misc.h |    4 +
 migration/nvdimm.c       | 1033 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1038 insertions(+)
 create mode 100644 migration/nvdimm.c

diff --git a/Makefile.target b/Makefile.target
index 6549481..0259e70 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -139,6 +139,7 @@ obj-y += memory.o
 obj-y += memory_mapping.o
 obj-y += dump.o
 obj-y += migration/ram.o
+obj-y += migration/nvdimm.o
 LIBS := $(libs_softmmu) $(LIBS)
 
 # Hardware support
diff --git a/include/migration/misc.h b/include/migration/misc.h
index 77fd4f5..0c23da8 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -20,6 +20,10 @@
 
 void ram_mig_init(void);
 
+/* migration/nvdimm.c */
+void nvdimm_snapshot_init(void);
+bool ram_block_is_nvdimm_active(RAMBlock *block);
+
 /* migration/block.c */
 
 #ifdef CONFIG_LIVE_BLOCK_MIGRATION
diff --git a/migration/nvdimm.c b/migration/nvdimm.c
new file mode 100644
index 0000000..8516bb0
--- /dev/null
+++ b/migration/nvdimm.c
@@ -0,0 +1,1033 @@
+/*
+ * QEMU System Emulator
+ *
+ * Authors:
+ *  He Junyan<Junyan.he@intel.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/mem/nvdimm.h"
+#include "cpu.h"
+#include "qemu/cutils.h"
+#include "exec/ram_addr.h"
+#include "exec/target_page.h"
+#include "qemu/rcu_queue.h"
+#include "qemu/error-report.h"
+#include "migration.h"
+#include "qapi/error.h"
+#include "migration/register.h"
+#include "migration/ram.h"
+#include "migration/qemu-file.h"
+#include "migration.h"
+#include "migration/misc.h"
+#include "migration/savevm.h"
+#include "block/snapshot.h"
+#include "migration/snapshot.h"
+
+#define NVDIMM_MIG_VERSION 0x01
+
+/* PADDING data, useless */
+#define NVDIMM_PADDING_BYTE 0xce
+/* PAGE id, is all zero */
+#define NVDIMM_ZERO_PAGE_ID 0xaabc250f
+#define NVDIMM_NONZERO_PAGE_ID 0xacbc250e
+/* No usage date, for alignment only */
+#define NVDIMM_SECTION_PADDING_ID 0xaaceccea
+/* Section for dirty log kind */
+#define NVDIMM_SECTION_DIRTY_LOG_ID 0xbbcd0c1e
+/* Section for raw data, no bitmap, dump the whole mem */
+#define NVDIMM_SECTION_DATA_ID 0x76bbcae3
+/* Section for setup */
+#define NVDIMM_SECTION_SETUP 0x7ace0cfa
+/* Section for setup */
+#define NVDIMM_SECTION_COMPLETE 0x8ace0cfa
+/* Section end symbol */
+#define NVDIMM_SECTION_END_ID 0xccbe8752
+/************************  Sections** ***********************
+Padding section
+----------------------------------------------------
+| PADDING_ID | size | PADDING_BYTE ...... | END_ID |
+----------------------------------------------------
+Dirty log section
+------------------------------------------------------------------------------------
+| DIRTY_BITMAP_ID | total size | ram name size | ram name | ram size | bitmap size |
+------------------------------------------------------------------------------------
+    -----------------------------------------------------------------
+     bitmap data... | dirty page size | dirty page data... | END_ID |
+    -----------------------------------------------------------------
+Raw data section
+---------------------------------------------------------------------------------------
+| DATA_ID | size | ram name size | ram name | ram size | data size | data... | END_ID |
+---------------------------------------------------------------------------------------
+*************************************************************/
+
+/* State of NVDimm for migration */
+struct NVDimmState {
+    /* Whether the block driver support dependency
+       between snapshots */
+    char *depend_snapshot_id;
+    int64_t depend_offset;
+    int64_t depend_size;
+    char *cur_snapshot_id;
+    int64_t cur_offset;
+    int64_t cur_size;
+    RAMBlock **blocks;
+    int block_num;
+    bool dirty_logging;
+};
+typedef struct NVDimmState NVDimmState;
+
+static NVDimmState *nvdimm_state_p;
+
+static int nvdimm_device_list_append(Object *obj, void *opaque)
+{
+    GSList **list = opaque;
+
+    if (object_dynamic_cast(obj, TYPE_NVDIMM)) {
+        *list = g_slist_append(*list, DEVICE(obj));
+    }
+
+    object_child_foreach(obj, nvdimm_device_list_append, opaque);
+    return 0;
+}
+
+static bool ram_block_is_nvdimm(RAMBlock *block)
+{
+    GSList *list = NULL;
+    GSList *device_list = NULL;
+    bool ret = false;
+
+    object_child_foreach(qdev_get_machine(),
+                         nvdimm_device_list_append, &device_list);
+
+    if (device_list == NULL) {
+        return false;
+    }
+
+    for (list = device_list; list; list = list->next) {
+        NVDIMMDevice *nvd = list->data;
+        MemoryRegion *mr = &nvd->nvdimm_mr;
+        int fd = memory_region_get_fd(mr);
+
+        if (fd >= 0 && fd == block->fd) {
+            ret = true;
+            break;
+        }
+    }
+
+    g_slist_free(device_list);
+    return ret;
+}
+
+bool ram_block_is_nvdimm_active(RAMBlock *block)
+{
+    if (block == NULL) {
+        return false;
+    }
+
+    if (get_current_snapshot_info(NULL) == false) {
+        return false;
+    }
+
+    return ram_block_is_nvdimm(block);
+}
+
+/* Just support snapshot, live migration use ram's handlers */
+static bool nvdimm_is_active(void *opaque)
+{
+    RAMBlock *block;
+    int ret = get_current_snapshot_info(NULL);
+    if (ret) {
+        return true;
+    }
+
+    if (!ram_bytes_total()) {
+        return false;
+    }
+
+    rcu_read_lock();
+    RAMBLOCK_FOREACH(block)
+    {
+        if (ram_block_is_nvdimm_active(block)) {
+            rcu_read_unlock();
+            return true;
+        }
+    }
+    rcu_read_unlock();
+
+    return false;
+}
+
+static int nvdimm_padding_to_alignment(QEMUFile *f, int section_id,
+                                       int32_t alignment, bool add_footer)
+{
+    int64_t cur_pos;
+    int32_t padding_sz;
+    int ret = 0;
+
+    cur_pos = qemu_ftell(f);
+    /* We need to insert some padding section here. */
+    padding_sz = (int32_t)(QEMU_ALIGN_UP(cur_pos, alignment) - cur_pos);
+    ret = padding_sz;
+
+    padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_PADDING_ID
+    padding_sz -= sizeof(int32_t); // NVDIMM_PADDING_BYTE size
+    padding_sz -= sizeof(int32_t); // NVDIMM_SECTION_END_ID
+    if (migrate_get_current()->send_section_footer) {
+        padding_sz -= sizeof(int8_t);
+        padding_sz -= sizeof(int32_t);
+    }
+
+    if (padding_sz <= 0) {
+        padding_sz += alignment;
+        ret += alignment;
+    }
+
+    qemu_put_be32(f, NVDIMM_SECTION_PADDING_ID);
+    qemu_put_be32(f, padding_sz);
+    while (padding_sz) {
+        qemu_put_byte(f, NVDIMM_PADDING_BYTE);
+        padding_sz--;
+    }
+    qemu_put_be32(f, NVDIMM_SECTION_END_ID);
+    if (add_footer && migrate_get_current()->send_section_footer) {
+        qemu_put_byte(f, QEMU_VM_SECTION_FOOTER);
+        qemu_put_be32(f, section_id);
+    }
+
+    cur_pos = qemu_ftell(f);
+    assert(QEMU_IS_ALIGNED(cur_pos, alignment) || add_footer == false);
+    return ret;
+}
+
+static int nvdimm_state_save_dependency(QEMUFile *f, NVDimmState *nvdimm_state)
+{
+    int64_t cur_pos;
+    int ret;
+    int32_t alignment = 0;
+
+    if (qemu_file_is_support_dependency(f, &alignment) == false) {
+        error_report("Enable nvdimm dependent snapshot without"
+                     "file dependency support");
+        return -ENOTSUP;
+    }
+
+    cur_pos = qemu_ftell(f);
+    assert(QEMU_IS_ALIGNED(cur_pos, alignment));
+    assert(QEMU_IS_ALIGNED(nvdimm_state->depend_offset, alignment));
+    assert(QEMU_IS_ALIGNED(nvdimm_state->depend_size, alignment));
+
+    ret = qemu_file_set_ref_name(f, nvdimm_state->depend_snapshot_id);
+    assert(ret);
+
+    ret = qemu_file_save_dependency(f, nvdimm_state->depend_offset,
+                                    nvdimm_state->depend_size);
+    if (ret < 0) {
+        error_report("save file dependency failed, depend_offset = %lx "
+                     "depend_size is %ld, ret is %d",
+                     nvdimm_state->depend_offset,
+                     nvdimm_state->depend_size, ret);
+        return ret;
+    }
+
+    cur_pos = qemu_ftell(f);
+    assert(QEMU_IS_ALIGNED(cur_pos, alignment));
+
+    return ret;
+}
+
+static inline void *nvdimm_host_from_ram_block_offset(RAMBlock *block,
+                                                      ram_addr_t offset)
+{
+    if (!offset_in_ramblock(block, offset)) {
+        return NULL;
+    }
+
+    return block->host + offset;
+}
+
+static int nvdimm_state_save_all_pages(QEMUFile *f,
+                                       NVDimmState *nvdimm_state, int i)
+{
+    hwaddr addr;
+    uint64_t total_sz;
+    int name_sz;
+    uint64_t data_sz;
+    void *host_ptr;
+
+    if (memory_region_size(nvdimm_state->blocks[i]->mr) == 0) {
+        return 0;
+    }
+
+    data_sz = 0;
+    for (addr = 0; addr < memory_region_size(nvdimm_state->blocks[i]->mr);
+         addr += 1 << TARGET_PAGE_BITS) {
+        assert(QEMU_IS_ALIGNED(addr, 1 << TARGET_PAGE_BITS));
+        host_ptr =
+            nvdimm_host_from_ram_block_offset(nvdimm_state->blocks[i], addr);
+        if (!host_ptr) {
+            error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
+            return -EINVAL;
+        }
+
+        if (buffer_is_zero(host_ptr, 1 << TARGET_PAGE_BITS)) {
+            data_sz += sizeof(int); // Zero page, just a ID
+        } else {
+            data_sz += ((1 << TARGET_PAGE_BITS) + sizeof(int)); // ID + page
+        }
+    }
+
+    total_sz = sizeof(unsigned int); // NVDIMM_SECTION_DIRTY_BITMAP_ID
+    total_sz += sizeof(uint64_t);    // the total size itself
+    total_sz += sizeof(int);         // ram name size
+    name_sz = strlen(nvdimm_state->blocks[i]->idstr) + 1;
+    total_sz += name_sz;
+    total_sz += sizeof(uint64_t); // ram size
+    total_sz += sizeof(uint64_t); // data size
+    total_sz += data_sz;
+    total_sz += sizeof(unsigned int); // NVDIMM_SECTION_END_ID
+
+    qemu_put_be32(f, NVDIMM_SECTION_DATA_ID);
+    qemu_put_be64(f, total_sz);
+    qemu_put_be32(f, name_sz);
+    qemu_put_buffer(f, (uint8_t *)nvdimm_state->blocks[i]->idstr, name_sz);
+    qemu_put_be64(f, memory_region_size(nvdimm_state->blocks[i]->mr));
+    qemu_put_be64(f, data_sz);
+
+    for (addr = 0; addr < memory_region_size(nvdimm_state->blocks[i]->mr);
+         addr += 1 << TARGET_PAGE_BITS) {
+        host_ptr =
+            nvdimm_host_from_ram_block_offset(nvdimm_state->blocks[i], addr);
+
+        if (buffer_is_zero(host_ptr, 1 << TARGET_PAGE_BITS)) {
+            qemu_put_be32(f, NVDIMM_ZERO_PAGE_ID);
+            data_sz -= sizeof(int);
+        } else {
+            qemu_put_be32(f, NVDIMM_NONZERO_PAGE_ID);
+            data_sz -= sizeof(int);
+            qemu_put_buffer(f, host_ptr, 1 << TARGET_PAGE_BITS);
+            data_sz -= 1 << TARGET_PAGE_BITS;
+        }
+    }
+    assert(data_sz == 0);
+    qemu_put_be32(f, NVDIMM_SECTION_END_ID);
+
+    return 1;
+}
+
+static int nvdimm_state_save_dirty_pages(QEMUFile *f,
+                                         NVDimmState *nvdimm_state, int i)
+{
+    DirtyBitmapSnapshot *snap;
+    uint64_t bit_sz;
+    uint64_t total_sz;
+    int name_sz;
+    uint64_t data_sz;
+    hwaddr addr;
+
+    if (memory_region_size(nvdimm_state->blocks[i]->mr) == 0) {
+        return 0;
+    }
+
+    snap = memory_region_snapshot_and_clear_dirty(
+        nvdimm_state->blocks[i]->mr,
+        0,
+        memory_region_size(nvdimm_state->blocks[i]->mr),
+        DIRTY_MEMORY_MIGRATION);
+    if (snap == NULL) {
+        error_report("Can not create snapshot bitmap for block %s",
+                     nvdimm_state->blocks[i]->idstr);
+        return -1;
+    }
+
+    bit_sz =
+        memory_region_size(nvdimm_state->blocks[i]->mr) >> (TARGET_PAGE_BITS + 3);
+
+    data_sz = 0;
+    for (addr = 0; addr < memory_region_size(nvdimm_state->blocks[i]->mr);
+         addr += 1 << TARGET_PAGE_BITS) {
+        assert(QEMU_IS_ALIGNED(addr, 1 << TARGET_PAGE_BITS));
+        if (memory_region_snapshot_get_dirty(nvdimm_state->blocks[i]->mr,
+                                             snap, addr, 1 << TARGET_PAGE_BITS)) {
+            data_sz += 1 << TARGET_PAGE_BITS;
+        }
+    }
+
+    total_sz = sizeof(unsigned int); // NVDIMM_SECTION_DIRTY_BITMAP_ID
+    total_sz += sizeof(uint64_t);    // the total size itself
+    total_sz += sizeof(int);         // ram name size
+    name_sz = strlen(nvdimm_state->blocks[i]->idstr) + 1;
+    total_sz += name_sz;
+    total_sz += sizeof(uint64_t); // ram size
+    total_sz += sizeof(uint64_t); // bitmap size
+    total_sz += bit_sz;
+    total_sz += sizeof(uint64_t); // data size
+    total_sz += data_sz;
+    total_sz += sizeof(unsigned int); // NVDIMM_SECTION_END_ID
+
+    qemu_put_be32(f, NVDIMM_SECTION_DIRTY_LOG_ID);
+    qemu_put_be64(f, total_sz);
+    qemu_put_be32(f, name_sz);
+    qemu_put_buffer(f, (uint8_t *)nvdimm_state->blocks[i]->idstr, name_sz);
+    qemu_put_be64(f, memory_region_size(nvdimm_state->blocks[i]->mr));
+    qemu_put_be64(f, bit_sz);
+    qemu_put_buffer(f, (uint8_t *)memory_region_snapshot_get_dirty_bitmap(snap),
+                    bit_sz);
+
+    qemu_put_be64(f, data_sz);
+    if (data_sz != 0) {
+        for (addr = 0; addr < memory_region_size(nvdimm_state->blocks[i]->mr);
+             addr += 1 << TARGET_PAGE_BITS) {
+            assert(QEMU_IS_ALIGNED(addr, 1 << TARGET_PAGE_BITS));
+            if (memory_region_snapshot_get_dirty(nvdimm_state->blocks[i]->mr,
+                                                 snap, addr, 1 << TARGET_PAGE_BITS)) {
+                qemu_put_buffer(f, nvdimm_state->blocks[i]->host + addr,
+                                1 << TARGET_PAGE_BITS);
+                data_sz -= 1 << TARGET_PAGE_BITS;
+            }
+        }
+        assert(data_sz == 0);
+    }
+
+    qemu_put_be32(f, NVDIMM_SECTION_END_ID);
+    g_free(snap);
+
+    return 1;
+}
+
+/**
+ * nvdimm_save_iterate: iterative stage for migration
+ *
+ * Returns zero to indicate success and negative for error
+ *
+ * @f: QEMUFile where to send the data
+ * @opaque: NVDimmState pointer
+ */
+static int nvdimm_save_iterate(QEMUFile *f, void *opaque, int section_id)
+{
+    NVDimmState *nvdimm_state = *(void **)opaque;
+    int ret = 0;
+    int i;
+    int32_t alignment;
+    int64_t begin_pos, cur_pos;
+    bool padded = false;
+
+    /* Must support dependency */
+    ret = qemu_file_is_support_dependency(f, &alignment);
+    assert(ret == true);
+
+    cur_pos = qemu_ftell(f);
+    if (!QEMU_IS_ALIGNED(cur_pos, alignment)) {
+        ret = nvdimm_padding_to_alignment(f, section_id, alignment, true);
+        if (ret < 0) {
+            error_report("NVDIMM saving, failed to padding to aligment");
+            return ret;
+        }
+        padded = true;
+    }
+
+    begin_pos = qemu_ftell(f);
+    assert(QEMU_IS_ALIGNED(begin_pos, alignment));
+    nvdimm_state->cur_offset = begin_pos;
+
+    if (nvdimm_state->dirty_logging) {
+        ret = nvdimm_state_save_dependency(f, nvdimm_state);
+        if (ret < 0) {
+            error_report("NVDIMM saving, failed to save dependency");
+            return ret;
+        }
+
+        for (i = 0; i < nvdimm_state->block_num; i++) {
+            cpu_physical_memory_test_and_clear_dirty(
+                memory_region_get_ram_addr(nvdimm_state->blocks[i]->mr),
+                memory_region_size(nvdimm_state->blocks[i]->mr),
+                DIRTY_MEMORY_MIGRATION);
+        }
+
+        if (padded) {
+            qemu_put_byte(f, QEMU_VM_SECTION_PART);
+            qemu_put_be32(f, section_id);
+        }
+
+        for (i = 0; i < nvdimm_state->block_num; i++) {
+            ret = nvdimm_state_save_dirty_pages(f, nvdimm_state, i);
+            if (ret < 0) {
+                error_report("NVDIMM saving, failed to save dirty pages");
+                return ret;
+            }
+        }
+    } else {
+        if (padded) {
+            qemu_put_byte(f, QEMU_VM_SECTION_PART);
+            qemu_put_be32(f, section_id);
+        }
+
+        /* Save the whole content of nvdimm, no dependency needed */
+        for (i = 0; i < nvdimm_state->block_num; i++) {
+            ret = nvdimm_state_save_all_pages(f, nvdimm_state, i);
+            if (ret < 0) {
+                error_report("NVDIMM saving, failed to save all pages");
+                return ret;
+            }
+        }
+    }
+
+    /* Need to add padding to make the whole data aligned, include
+       QEMU_VM_SECTION_FOOTER and section_id */
+    cur_pos = qemu_ftell(f);
+    if (migrate_get_current()->send_section_footer) {
+        cur_pos += (1 + sizeof(int));
+    }
+
+    if (QEMU_IS_ALIGNED(cur_pos, alignment)) { // Already aligned
+        nvdimm_state->cur_size = cur_pos - begin_pos;
+        assert(QEMU_IS_ALIGNED(nvdimm_state->cur_size, alignment));
+        return ret;
+    }
+
+    /* Appending the footer if needed */
+    if (migrate_get_current()->send_section_footer) {
+        qemu_put_byte(f, QEMU_VM_SECTION_FOOTER);
+        qemu_put_be32(f, section_id);
+        qemu_put_byte(f, QEMU_VM_SECTION_PART);
+        qemu_put_be32(f, section_id);
+    }
+    ret = nvdimm_padding_to_alignment(f, section_id, alignment, false);
+    if (ret < 0) {
+        error_report("NVDIMM saving, failed to save all pages");
+        return ret;
+    }
+
+    cur_pos = qemu_ftell(f);
+    nvdimm_state->cur_size = cur_pos - begin_pos;
+    if (migrate_get_current()->send_section_footer) {
+        nvdimm_state->cur_size += (1 + sizeof(int));
+    }
+    assert(QEMU_IS_ALIGNED(nvdimm_state->cur_size, alignment));
+
+    return ret;
+}
+
+static void nvdimm_destroy_nvdimm_state(NVDimmState *nvdimm_state)
+{
+    if (nvdimm_state) {
+        /* disable all dirty log trace */
+        if (nvdimm_state->depend_snapshot_id) {
+            int i;
+            for (i = 0; i < nvdimm_state->block_num; i++) {
+                memory_region_set_log(nvdimm_state->blocks[i]->mr, false,
+                                      DIRTY_MEMORY_MIGRATION);
+            }
+        }
+
+        if (nvdimm_state->cur_snapshot_id) {
+            g_free(nvdimm_state->cur_snapshot_id);
+        }
+        if (nvdimm_state->blocks) {
+            g_free(nvdimm_state->blocks);
+        }
+        g_free(nvdimm_state);
+    }
+}
+
+static NVDimmState *nvdimm_alloc_nvdimm_state(void)
+{
+    NVDimmState *nvdimm_state = g_try_new0(NVDimmState, 1);
+    RAMBlock *block;
+
+    if (nvdimm_state == NULL) {
+        return NULL;
+    }
+
+    rcu_read_lock();
+    RAMBLOCK_FOREACH(block)
+    {
+        if (ram_block_is_nvdimm(block)) {
+            nvdimm_state->block_num++;
+            nvdimm_state->blocks =
+                g_try_renew(RAMBlock *,
+                            nvdimm_state->blocks,
+                            nvdimm_state->block_num);
+            if (nvdimm_state->blocks == NULL) {
+                rcu_read_unlock();
+                nvdimm_destroy_nvdimm_state(nvdimm_state);
+                return NULL;
+            }
+
+            nvdimm_state->blocks[nvdimm_state->block_num - 1] = block;
+        }
+    }
+    rcu_read_unlock();
+
+    return nvdimm_state;
+}
+
+/**
+ * nvdimm_save_setup: Setup nvdimm for migration
+ *
+ * Returns zero to indicate success and negative for error
+ *
+ * @f: QEMUFile where to send the data
+ * @opaque: NVDimmState pointer
+ */
+static int nvdimm_save_setup(QEMUFile *f, void *opaque)
+{
+    NVDimmState *nvdimm_state = *(void **)opaque;
+    int ret = 0;
+    QEMUSnapshotInfo sn;
+    RAMBlock *block;
+    int i;
+
+    ret = get_current_snapshot_info(&sn);
+    if (ret == 0) { /* Just enable in snapshot mode */
+        info_report("Not in snapshot saving, no nvdimm snapshot optimization");
+        return -1;
+    }
+
+    /* No dependency support, just let the ram common logic do its job */
+    if (qemu_file_is_support_dependency(f, NULL) == false) {
+        assert(nvdimm_state == NULL);
+        info_report("The drive file does not support dependent snapshot");
+        return -1;
+    }
+
+    if (nvdimm_state == NULL) { /* First time */
+        nvdimm_state = nvdimm_alloc_nvdimm_state();
+        if (nvdimm_state == NULL) {
+            error_report("Alloc the nvdimm state for snapshot saving failed");
+            goto failed;
+        }
+
+        nvdimm_state->cur_snapshot_id = g_strdup(sn.name);
+        *(void **)opaque = nvdimm_state;
+    } else {
+        assert(nvdimm_state->cur_snapshot_id);
+        if (nvdimm_state->depend_snapshot_id) {
+            g_free(nvdimm_state->depend_snapshot_id);
+        }
+        nvdimm_state->depend_snapshot_id = nvdimm_state->cur_snapshot_id;
+        nvdimm_state->depend_offset = nvdimm_state->cur_offset;
+        nvdimm_state->depend_size = nvdimm_state->cur_size;
+        nvdimm_state->cur_snapshot_id = g_strdup(sn.name);
+        nvdimm_state->cur_offset = 0;
+        nvdimm_state->cur_size = 0;
+
+        rcu_read_lock();
+        RAMBLOCK_FOREACH(block)
+        {
+            if (ram_block_is_nvdimm_active(block)) {
+                for (i = 0; i < nvdimm_state->block_num; i++) {
+                    if (block == nvdimm_state->blocks[i]) {
+                        break;
+                    }
+                }
+
+                // Can not find the same block?
+                if (i == nvdimm_state->block_num) {
+                    rcu_read_unlock();
+                    error_report("Can not find the block %s", block->idstr);
+                    goto failed;
+                }
+            }
+        }
+        rcu_read_unlock();
+    }
+
+    qemu_put_be32(f, NVDIMM_SECTION_SETUP);
+    qemu_put_be32(f, NVDIMM_SECTION_END_ID);
+
+    return ret;
+
+failed:
+    nvdimm_destroy_nvdimm_state(nvdimm_state);
+    *(void **)opaque = NULL;
+    return -1;
+}
+
+/**
+ * nvdimm_save_complete: function called to send the remaining amount of ram
+ *
+ * Returns zero to indicate success
+ *
+ * Called with iothread lock
+ *
+ * @f: QEMUFile where to send the data
+ * @opaque: NVDimmState pointer
+ */
+static int nvdimm_save_complete(QEMUFile *f, void *opaque)
+{
+    NVDimmState *nvdimm_state = *(void **)opaque;
+    int i;
+
+    for (i = 0; i < nvdimm_state->block_num; i++) {
+        memory_region_set_log(nvdimm_state->blocks[i]->mr, true,
+                              DIRTY_MEMORY_MIGRATION);
+    }
+
+    /* Enable the dirty logging for next time usage */
+    nvdimm_state->dirty_logging = true;
+
+    qemu_put_be32(f, NVDIMM_SECTION_COMPLETE);
+    qemu_put_be32(f, NVDIMM_SECTION_END_ID);
+
+    return 0;
+}
+
+static bool nvdimm_has_postcopy(void *opaque)
+{
+    return false;
+}
+
+static void nvdimm_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
+                                uint64_t *non_postcopiable_pending,
+                                uint64_t *postcopiable_pending)
+{
+}
+
+static void nvdimm_save_cleanup(void *opaque)
+{
+    return;
+}
+
+/**
+ * nvdimm_load_setup: Setup NVDimm for migration incoming side
+ *
+ * Returns zero to indicate success and negative for error
+ *
+ * @f: QEMUFile where to receive the data
+ * @opaque: NVDimmState pointer
+ */
+static int nvdimm_load_setup(QEMUFile *f, void *opaque)
+{
+    NVDimmState *nvdimm_state = *(void **)opaque;
+
+    if (nvdimm_state) {
+        nvdimm_destroy_nvdimm_state(nvdimm_state);
+        *(void **)opaque = NULL;
+    }
+
+    nvdimm_state = nvdimm_alloc_nvdimm_state();
+    if (nvdimm_state == NULL) {
+        return -1;
+    }
+
+    *(void **)opaque = nvdimm_state;
+    return 1;
+}
+
+static int nvdimm_load_cleanup(void *opaque)
+{
+    NVDimmState *nvdimm_state = *(void **)opaque;
+    nvdimm_destroy_nvdimm_state(nvdimm_state);
+    *(void **)opaque = NULL;
+
+    return 0;
+}
+
+static int nvdimm_load_dirty_pages(QEMUFile *f, NVDimmState *nvdimm_state)
+{
+    int64_t total_sz = qemu_get_be64(f);
+    int ret = 0;
+    int name_sz;
+    int64_t sz;
+    uint8_t *name_buf = NULL;
+    uint8_t *bitmap_buf = NULL;
+    RAMBlock *block;
+    int64_t ram_sz = 0;
+    int64_t bitmap_sz = 0;
+    int64_t data_sz = 0;
+    hwaddr addr;
+    void *host_ptr;
+
+    if (total_sz <= 0) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    name_sz = qemu_get_be32(f);
+    if (name_sz <= 0) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    name_buf = g_malloc(name_sz);
+    if (name_buf == NULL) {
+        ret = -ENOMEM;
+        return ret;
+    }
+
+    sz = qemu_get_buffer(f, name_buf, name_sz);
+    if (sz != name_sz) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    block = qemu_ram_block_by_name((char *)name_buf);
+    if (block == NULL || block->host == NULL) {
+        ret = -EINVAL;
+        return ret;
+    }
+    g_free(name_buf);
+    name_buf = NULL;
+
+    ram_sz = qemu_get_be64(f);
+    if (ram_sz != memory_region_size(block->mr)) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    bitmap_sz = qemu_get_be64(f);
+    if (bitmap_sz <= 0) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    bitmap_buf = g_malloc(bitmap_sz);
+    if (bitmap_buf == NULL) {
+        ret = -ENOMEM;
+        return ret;
+    }
+
+    sz = qemu_get_buffer(f, bitmap_buf, bitmap_sz);
+    if (sz != bitmap_sz) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    data_sz = qemu_get_be64(f);
+    if (data_sz < 0) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    assert(QEMU_IS_ALIGNED(data_sz, TARGET_PAGE_SIZE));
+    addr = 0;
+    while (data_sz) {
+        addr = find_next_bit((unsigned long *)bitmap_buf,
+                             ram_sz >> TARGET_PAGE_BITS, addr);
+        host_ptr = nvdimm_host_from_ram_block_offset(block, addr);
+        if (!host_ptr) {
+            error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
+            ret = -EINVAL;
+            goto out;
+        }
+
+        qemu_get_buffer(f, host_ptr, TARGET_PAGE_SIZE);
+        data_sz -= TARGET_PAGE_SIZE;
+    }
+
+    if (qemu_get_be32(f) != NVDIMM_SECTION_END_ID) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+out:
+    if (bitmap_buf) {
+        g_free(bitmap_buf);
+    }
+    return ret;
+}
+
+static int nvdimm_load_all_pages(QEMUFile *f, NVDimmState *nvdimm_state)
+{
+    int64_t total_sz = qemu_get_be64(f);
+    int64_t sz;
+    int64_t data_sz;
+    int name_sz;
+    int ret = 0;
+    uint8_t *buf = NULL;
+    RAMBlock *block;
+    hwaddr addr;
+    void *host_ptr;
+    int64_t ram_sz = 0;
+    int tag;
+
+    if (total_sz <= 0) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    name_sz = qemu_get_be32(f);
+    if (name_sz <= 0) {
+        ret = -EINVAL;
+        return ret;
+    }
+
+    buf = g_malloc(name_sz);
+    if (buf == NULL) {
+        ret = -ENOMEM;
+        return ret;
+    }
+
+    sz = qemu_get_buffer(f, buf, name_sz);
+    if (sz != name_sz) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    block = qemu_ram_block_by_name((char *)buf);
+    if (block == NULL || block->host == NULL) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    ram_sz = qemu_get_be64(f);
+    if (ram_sz != memory_region_size(block->mr)) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    data_sz = qemu_get_be64(f);
+    if (data_sz <= 0) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+    for (addr = 0; addr < ram_sz; addr += 1 << TARGET_PAGE_BITS) {
+        assert(QEMU_IS_ALIGNED(addr, 1 << TARGET_PAGE_BITS));
+        host_ptr = nvdimm_host_from_ram_block_offset(block, addr);
+        if (!host_ptr) {
+            error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
+            ret = -EINVAL;
+            goto out;
+        }
+
+        tag = qemu_get_be32(f);
+        if (tag == NVDIMM_ZERO_PAGE_ID) {
+            memset(host_ptr, 0, 1 << TARGET_PAGE_BITS);
+        } else {
+            assert(tag == NVDIMM_NONZERO_PAGE_ID);
+            qemu_get_buffer(f, host_ptr, TARGET_PAGE_SIZE);
+        }
+        host_ptr += TARGET_PAGE_SIZE;
+    }
+
+    tag = qemu_get_be32(f);
+    if (tag != NVDIMM_SECTION_END_ID) {
+        ret = -EINVAL;
+        goto out;
+    }
+
+out:
+    if (buf) {
+        g_free(buf);
+    }
+
+    return ret;
+}
+
+static int nvdimm_load(QEMUFile *f, void *opaque, int version_id)
+{
+    int ret = 0;
+    unsigned int sec_id;
+    uint8_t *buf = NULL;
+    size_t sz;
+    NVDimmState *nvdimm_state = *(void **)opaque;
+
+    if (version_id != NVDIMM_MIG_VERSION) {
+        ret = -EINVAL;
+        goto failed;
+    }
+
+    sec_id = qemu_get_be32(f);
+    if (sec_id == NVDIMM_SECTION_PADDING_ID) {
+        /* Just skip all this padding section. */
+        int padding_sz = qemu_get_be32(f);
+        unsigned int end_id;
+        buf = g_malloc(padding_sz);
+        if (buf == NULL) {
+            ret = -ENOMEM;
+            goto failed;
+        }
+
+        sz = qemu_get_buffer(f, buf, padding_sz);
+        if (sz != padding_sz) {
+            ret = -EINVAL;
+            goto failed;
+        }
+        padding_sz--;
+        while (padding_sz >= 0) {
+            if (buf[padding_sz] != NVDIMM_PADDING_BYTE) {
+                ret = -EINVAL;
+                goto failed;
+            }
+            padding_sz--;
+        }
+
+        end_id = qemu_get_be32(f);
+        if (end_id != NVDIMM_SECTION_END_ID) {
+            ret = -EINVAL;
+            goto failed;
+        }
+
+        g_free(buf);
+        buf = NULL;
+    } else if (sec_id == NVDIMM_SECTION_DIRTY_LOG_ID) {
+        ret = nvdimm_load_dirty_pages(f, nvdimm_state);
+        if (ret < 0) {
+            goto failed;
+        }
+    } else if (sec_id == NVDIMM_SECTION_DATA_ID) {
+        ret = nvdimm_load_all_pages(f, nvdimm_state);
+        if (ret < 0) {
+            goto failed;
+        }
+    } else if (sec_id == NVDIMM_SECTION_SETUP ||
+               sec_id == NVDIMM_SECTION_COMPLETE) {
+        unsigned int d = qemu_get_be32(f);
+        if (d != NVDIMM_SECTION_END_ID) {
+            ret = -EINVAL;
+            goto failed;
+        }
+    } else {
+        error_report("NVDIMM load, can not recognize SEC id %d", sec_id);
+        ret = -EINVAL;
+        goto failed;
+    }
+
+    return ret;
+
+failed:
+    if (buf) {
+        g_free(buf);
+    }
+
+    return ret;
+}
+
+static SaveVMHandlers savevm_nvdimm_handlers = {
+    .is_active = nvdimm_is_active,
+    .save_setup = nvdimm_save_setup,
+    .save_live_iterate = nvdimm_save_iterate,
+    .save_live_complete_precopy = nvdimm_save_complete,
+    .has_postcopy = nvdimm_has_postcopy,
+    .save_live_pending = nvdimm_save_pending,
+    .load_state = nvdimm_load,
+    .save_cleanup = nvdimm_save_cleanup,
+    .load_setup = nvdimm_load_setup,
+    .load_cleanup = nvdimm_load_cleanup,
+};
+
+void nvdimm_snapshot_init(void)
+{
+    register_savevm_live(NULL, "nvdimm", 0, NVDIMM_MIG_VERSION,
+                         &savevm_nvdimm_handlers, &nvdimm_state_p);
+}
-- 
2.7.4


[Qemu-devel] [PATCH 10/10] RFC: Enable nvdimm snapshot functions.
Posted by junyan.he@hotmail.com, 14 weeks ago
From: Junyan He <junyan.he@intel.com>

In snapshot saving, all nvdimm kind memory will be saved in different way
and we exclude all nvdimm kind memory region in ram.c

Signed-off-by: Junyan He <junyan.he@intel.com>
---
 migration/ram.c | 17 +++++++++++++++++
 vl.c            |  1 +
 2 files changed, 18 insertions(+)

diff --git a/migration/ram.c b/migration/ram.c
index d1db422..ad32469 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1219,9 +1219,15 @@ static bool find_dirty_block(RAMState *rs, PageSearchStatus *pss, bool *again)
         /* Didn't find anything in this RAM Block */
         pss->page = 0;
         pss->block = QLIST_NEXT_RCU(pss->block, next);
+        while (ram_block_is_nvdimm_active(pss->block)) {
+            pss->block = QLIST_NEXT_RCU(pss->block, next);
+        }
         if (!pss->block) {
             /* Hit the end of the list */
             pss->block = QLIST_FIRST_RCU(&ram_list.blocks);
+            while (ram_block_is_nvdimm_active(pss->block)) {
+                pss->block = QLIST_NEXT_RCU(pss->block, next);
+            }
             /* Flag that we've looped */
             pss->complete_round = true;
             rs->ram_bulk_stage = false;
@@ -1541,6 +1547,9 @@ static int ram_find_and_save_block(RAMState *rs, bool last_stage)
 
     if (!pss.block) {
         pss.block = QLIST_FIRST_RCU(&ram_list.blocks);
+        while (ram_block_is_nvdimm_active(pss.block)) {
+            pss.block = QLIST_NEXT_RCU(pss.block, next);
+        }
     }
 
     do {
@@ -1583,6 +1592,10 @@ uint64_t ram_bytes_total(void)
 
     rcu_read_lock();
     RAMBLOCK_FOREACH(block) {
+        if (ram_block_is_nvdimm_active(block)) {
+            // If snapshot and the block is nvdimm, let nvdimm do the job
+            continue;
+        }
         total += block->used_length;
     }
     rcu_read_unlock();
@@ -2222,6 +2235,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
     qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
 
     RAMBLOCK_FOREACH(block) {
+        if (ram_block_is_nvdimm_active(block)) {
+            // If snapshot and the block is nvdimm, let nvdimm do the job
+            continue;
+        }
         qemu_put_byte(f, strlen(block->idstr));
         qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr));
         qemu_put_be64(f, block->used_length);
diff --git a/vl.c b/vl.c
index 3ef04ce..1bd5711 100644
--- a/vl.c
+++ b/vl.c
@@ -4502,6 +4502,7 @@ int main(int argc, char **argv, char **envp)
 
     blk_mig_init();
     ram_mig_init();
+    nvdimm_snapshot_init();
 
     /* If the currently selected machine wishes to override the units-per-bus
      * property of its default HBA interface type, do so now. */
-- 
2.7.4