[Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support

Singh, Brijesh posted 13 patches 4 years, 9 months ago
Test checkpatch failed
Test s390x failed
Test asan passed
Test docker-mingw@fedora passed
Test FreeBSD passed
Test docker-clang@ubuntu failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190710202219.25939-1-brijesh.singh@amd.com
Maintainers: Eric Blake <eblake@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>, Richard Henderson <rth@twiddle.net>, Cornelia Huck <cohuck@redhat.com>, Juan Quintela <quintela@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, Markus Armbruster <armbru@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>
There is a newer version of this series
accel/kvm/kvm-all.c            | 108 ++++++++
accel/kvm/sev-stub.c           |  22 ++
accel/stubs/kvm-stub.c         |  22 ++
docs/amd-memory-encryption.txt |  44 +++-
include/exec/ram_addr.h        | 161 +++++++++++-
include/exec/ramlist.h         |   3 +-
include/sysemu/kvm.h           |  25 ++
include/sysemu/sev.h           |   6 +
linux-headers/linux/kvm.h      |  53 ++++
migration/ram.c                |  91 ++++++-
qapi/misc-target.json          |  18 ++
target/i386/monitor.c          |  10 +
target/i386/sev-stub.c         |   5 +
target/i386/sev.c              | 455 +++++++++++++++++++++++++++++++--
target/i386/sev_i386.h         |  11 +-
target/i386/trace-events       |   8 +
16 files changed, 1016 insertions(+), 26 deletions(-)
[Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support
Posted by Singh, Brijesh 4 years, 9 months ago
AMD SEV encrypts the memory of VMs and because this encryption is done using
an address tweak, the hypervisor will not be able to simply copy ciphertext
between machines to migrate a VM. Instead the AMD SEV Key Management API
provides a set of functions which the hypervisor can use to package a
guest encrypted pages for migration, while maintaining the confidentiality
provided by AMD SEV.

The patch series add the support required in Qemu to perform the SEV
guest live migration. Before initiating the live migration a user
should use newly added 'migrate-set-sev-info' command to pass the
target machines certificate chain. See the docs/amd-memory-encryption.txt
for further details.

The patch series depends on kernel patches available here:
https://marc.info/?l=kvm&m=156278967226011&w=2

The complete tree with patch is available at:
https://github.com/codomania/qemu/tree/sev-migration-v2

Changes since v1:
 - use the dirty log sync APIs to also sync the page encryption bitmap
   when SEV is active.

Brijesh Singh (13):
  linux-headers: update kernel header to include SEV migration commands
  kvm: introduce high-level API to support encrypted page migration
  migration/ram: add support to send encrypted pages
  kvm: add support to sync the page encryption state bitmap
  doc: update AMD SEV API spec web link
  doc: update AMD SEV to include Live migration flow
  target/i386: sev: do not create launch context for an incoming guest
  misc.json: add migrate-set-sev-info command
  target/i386: sev: add support to encrypt the outgoing page
  target/i386: sev: add support to load incoming encrypted page
  kvm: introduce high-level API to migrate the page encryption bitmap
  migration: add support to migrate page encryption bitmap
  target/i386: sev: remove migration blocker

 accel/kvm/kvm-all.c            | 108 ++++++++
 accel/kvm/sev-stub.c           |  22 ++
 accel/stubs/kvm-stub.c         |  22 ++
 docs/amd-memory-encryption.txt |  44 +++-
 include/exec/ram_addr.h        | 161 +++++++++++-
 include/exec/ramlist.h         |   3 +-
 include/sysemu/kvm.h           |  25 ++
 include/sysemu/sev.h           |   6 +
 linux-headers/linux/kvm.h      |  53 ++++
 migration/ram.c                |  91 ++++++-
 qapi/misc-target.json          |  18 ++
 target/i386/monitor.c          |  10 +
 target/i386/sev-stub.c         |   5 +
 target/i386/sev.c              | 455 +++++++++++++++++++++++++++++++--
 target/i386/sev_i386.h         |  11 +-
 target/i386/trace-events       |   8 +
 16 files changed, 1016 insertions(+), 26 deletions(-)

-- 
2.17.1


Re: [Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support
Posted by Dr. David Alan Gilbert 4 years, 9 months ago
* Singh, Brijesh (brijesh.singh@amd.com) wrote:
> AMD SEV encrypts the memory of VMs and because this encryption is done using
> an address tweak, the hypervisor will not be able to simply copy ciphertext
> between machines to migrate a VM. Instead the AMD SEV Key Management API
> provides a set of functions which the hypervisor can use to package a
> guest encrypted pages for migration, while maintaining the confidentiality
> provided by AMD SEV.
> 
> The patch series add the support required in Qemu to perform the SEV
> guest live migration. Before initiating the live migration a user
> should use newly added 'migrate-set-sev-info' command to pass the
> target machines certificate chain. See the docs/amd-memory-encryption.txt
> for further details.

Note the two patchew errors:
  a) Mostly formatting; 80 char lines, /* comments etc - you should
     check your patches using scripts/checkpatch.pl  to get rid of that
     lot.

  b) There are some build errors on non-x86 softmmu builds.

Dave

> The patch series depends on kernel patches available here:
> https://marc.info/?l=kvm&m=156278967226011&w=2
> 
> The complete tree with patch is available at:
> https://github.com/codomania/qemu/tree/sev-migration-v2
> 
> Changes since v1:
>  - use the dirty log sync APIs to also sync the page encryption bitmap
>    when SEV is active.
> 
> Brijesh Singh (13):
>   linux-headers: update kernel header to include SEV migration commands
>   kvm: introduce high-level API to support encrypted page migration
>   migration/ram: add support to send encrypted pages
>   kvm: add support to sync the page encryption state bitmap
>   doc: update AMD SEV API spec web link
>   doc: update AMD SEV to include Live migration flow
>   target/i386: sev: do not create launch context for an incoming guest
>   misc.json: add migrate-set-sev-info command
>   target/i386: sev: add support to encrypt the outgoing page
>   target/i386: sev: add support to load incoming encrypted page
>   kvm: introduce high-level API to migrate the page encryption bitmap
>   migration: add support to migrate page encryption bitmap
>   target/i386: sev: remove migration blocker
> 
>  accel/kvm/kvm-all.c            | 108 ++++++++
>  accel/kvm/sev-stub.c           |  22 ++
>  accel/stubs/kvm-stub.c         |  22 ++
>  docs/amd-memory-encryption.txt |  44 +++-
>  include/exec/ram_addr.h        | 161 +++++++++++-
>  include/exec/ramlist.h         |   3 +-
>  include/sysemu/kvm.h           |  25 ++
>  include/sysemu/sev.h           |   6 +
>  linux-headers/linux/kvm.h      |  53 ++++
>  migration/ram.c                |  91 ++++++-
>  qapi/misc-target.json          |  18 ++
>  target/i386/monitor.c          |  10 +
>  target/i386/sev-stub.c         |   5 +
>  target/i386/sev.c              | 455 +++++++++++++++++++++++++++++++--
>  target/i386/sev_i386.h         |  11 +-
>  target/i386/trace-events       |   8 +
>  16 files changed, 1016 insertions(+), 26 deletions(-)
> 
> -- 
> 2.17.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support
Posted by Singh, Brijesh 4 years, 9 months ago

On 7/11/19 4:59 AM, Dr. David Alan Gilbert wrote:
> * Singh, Brijesh (brijesh.singh@amd.com) wrote:
>> AMD SEV encrypts the memory of VMs and because this encryption is done using
>> an address tweak, the hypervisor will not be able to simply copy ciphertext
>> between machines to migrate a VM. Instead the AMD SEV Key Management API
>> provides a set of functions which the hypervisor can use to package a
>> guest encrypted pages for migration, while maintaining the confidentiality
>> provided by AMD SEV.
>>
>> The patch series add the support required in Qemu to perform the SEV
>> guest live migration. Before initiating the live migration a user
>> should use newly added 'migrate-set-sev-info' command to pass the
>> target machines certificate chain. See the docs/amd-memory-encryption.txt
>> for further details.
> 
> Note the two patchew errors:
>    a) Mostly formatting; 80 char lines, /* comments etc - you should
>       check your patches using scripts/checkpatch.pl  to get rid of that
>       lot.
> 
>    b) There are some build errors on non-x86 softmmu builds.
> 

Dave, thanks for reviews. I will fix these in next version.


Re: [Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support
Posted by no-reply@patchew.org 4 years, 9 months ago
Patchew URL: https://patchew.org/QEMU/20190710202219.25939-1-brijesh.singh@amd.com/



Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa

echo
echo "=== UNAME ==="
uname -a

CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

  CC      alpha-softmmu/hw/scsi/virtio-scsi-dataplane.o
/usr/bin/ld: qapi/qapi-commands-misc-target.o: in function `qmp_marshal_migrate_set_sev_info':
/var/tmp/patchew-tester-tmp-iinjo1bi/src/build/qapi/qapi-commands-misc-target.c:363: undefined reference to `qmp_migrate_set_sev_info'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:208: qemu-system-cris] Error 1
make: *** [Makefile:472: cris-softmmu/all] Error 2
make: *** Waiting for unfinished jobs....
---
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
/usr/bin/ld: qapi/qapi-commands-misc-target.o: in function `qmp_marshal_migrate_set_sev_info':
/var/tmp/patchew-tester-tmp-iinjo1bi/src/build/qapi/qapi-commands-misc-target.c:363: undefined reference to `qmp_migrate_set_sev_info'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:208: qemu-system-alpha] Error 1
make: *** [Makefile:472: alpha-softmmu/all] Error 2
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
---
  CC      arm-softmmu/target/arm/debug_helper.o
/usr/bin/ld: qapi/qapi-commands-misc-target.o: in function `qmp_marshal_migrate_set_sev_info':
/var/tmp/patchew-tester-tmp-iinjo1bi/src/build/qapi/qapi-commands-misc-target.c:363: undefined reference to `qmp_migrate_set_sev_info'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:208: qemu-system-aarch64] Error 1
make: *** [Makefile:472: aarch64-softmmu/all] Error 2
  GEN     arm-softmmu/target/arm/decode-vfp.inc.c
---
  LINK    arm-softmmu/qemu-system-arm
/usr/bin/ld: qapi/qapi-commands-misc-target.o: in function `qmp_marshal_migrate_set_sev_info':
/var/tmp/patchew-tester-tmp-iinjo1bi/src/build/qapi/qapi-commands-misc-target.c:363: undefined reference to `qmp_migrate_set_sev_info'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:208: qemu-system-arm] Error 1
make: *** [Makefile:472: arm-softmmu/all] Error 2
=== OUTPUT END ===


The full log is available at
http://patchew.org/logs/20190710202219.25939-1-brijesh.singh@amd.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support
Posted by no-reply@patchew.org 4 years, 9 months ago
Patchew URL: https://patchew.org/QEMU/20190710202219.25939-1-brijesh.singh@amd.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v2 00/13] Add SEV guest live migration support
Type: series
Message-id: 20190710202219.25939-1-brijesh.singh@amd.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
9eb9639 target/i386: sev: remove migration blocker
00c1826 migration: add support to migrate page encryption bitmap
cc5d459 kvm: introduce high-level API to migrate the page encryption bitmap
744b933 target/i386: sev: add support to load incoming encrypted page
0365253 target/i386: sev: add support to encrypt the outgoing page
b616e11 misc.json: add migrate-set-sev-info command
6dbc25d target/i386: sev: do not create launch context for an incoming guest
467cdc0 doc: update AMD SEV to include Live migration flow
a02bd6f doc: update AMD SEV API spec web link
26fee5c kvm: add support to sync the page encryption state bitmap
ec80b9c migration/ram: add support to send encrypted pages
f09bf4e kvm: introduce high-level API to support encrypted page migration
9b1e5ae linux-headers: update kernel header to include SEV migration commands

=== OUTPUT BEGIN ===
1/13 Checking commit 9b1e5aef53f4 (linux-headers: update kernel header to include SEV migration commands)
2/13 Checking commit f09bf4e4d9e0 (kvm: introduce high-level API to support encrypted page migration)
WARNING: line over 80 characters
#45: FILE: accel/kvm/kvm-all.c:177:
+        return kvm_state->memcrypt_save_outgoing_page(kvm_state->memcrypt_handle,

WARNING: line over 80 characters
#56: FILE: accel/kvm/kvm-all.c:188:
+        return kvm_state->memcrypt_load_incoming_page(kvm_state->memcrypt_handle,

total: 0 errors, 2 warnings, 96 lines checked

Patch 2/13 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
3/13 Checking commit ec80b9c1d59e (migration/ram: add support to send encrypted pages)
4/13 Checking commit 26fee5cf1cef (kvm: add support to sync the page encryption state bitmap)
WARNING: Block comments use a leading /* on a separate line
#40: FILE: accel/kvm/kvm-all.c:520:
+                 /*HOST_LONG_BITS*/ 64) / 8;

WARNING: line over 80 characters
#110: FILE: include/exec/ram_addr.h:336:
+    src = atomic_rcu_read(&ram_list.dirty_memory[DIRTY_MEMORY_ENCRYPTED])->blocks;

WARNING: line over 80 characters
#124: FILE: include/exec/ram_addr.h:350:
+static inline void cpu_physical_memory_set_dirty_enc_lebitmap(unsigned long *bitmap,

WARNING: line over 80 characters
#141: FILE: include/exec/ram_addr.h:387:
+                    atomic_xchg(&blocks[DIRTY_MEMORY_ENCRYPTED][idx][offset], temp);

WARNING: line over 80 characters
#143: FILE: include/exec/ram_addr.h:389:
+                    atomic_or(&blocks[DIRTY_MEMORY_MIGRATION][idx][offset], temp);

WARNING: line over 80 characters
#146: FILE: include/exec/ram_addr.h:392:
+                        atomic_or(&blocks[DIRTY_MEMORY_CODE][idx][offset], temp);

WARNING: Block comments use a leading /* on a separate line
#156: FILE: include/exec/ram_addr.h:414:
+            /* If its encrypted bitmap update, then we need to copy the bitmap

WARNING: line over 80 characters
#160: FILE: include/exec/ram_addr.h:418:
+                cpu_physical_memory_set_encrypted_range(start + i * TARGET_PAGE_SIZE,

WARNING: line over 80 characters
#161: FILE: include/exec/ram_addr.h:419:
+                                                        TARGET_PAGE_SIZE * hpratio,

WARNING: line over 80 characters
#174: FILE: include/exec/ram_addr.h:440:
+static inline void cpu_physical_memory_set_encrypted_lebitmap(unsigned long *bitmap,

WARNING: line over 80 characters
#178: FILE: include/exec/ram_addr.h:444:
+    return cpu_physical_memory_set_dirty_enc_lebitmap(bitmap, start, pages, true);

WARNING: line over 80 characters
#185: FILE: include/exec/ram_addr.h:451:
+    return cpu_physical_memory_set_dirty_enc_lebitmap(bitmap, start, pages, false);

WARNING: line over 80 characters
#195: FILE: include/exec/ram_addr.h:473:
+    cpu_physical_memory_test_and_clear_dirty(start, length, DIRTY_MEMORY_ENCRYPTED);

WARNING: line over 80 characters
#220: FILE: include/exec/ram_addr.h:559:
+    src = atomic_rcu_read(&ram_list.dirty_memory[DIRTY_MEMORY_ENCRYPTED])->blocks;

total: 0 errors, 14 warnings, 320 lines checked

Patch 4/13 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
5/13 Checking commit a02bd6ff047c (doc: update AMD SEV API spec web link)
6/13 Checking commit 467cdc0f381c (doc: update AMD SEV to include Live migration flow)
7/13 Checking commit 6dbc25da4473 (target/i386: sev: do not create launch context for an incoming guest)
8/13 Checking commit b616e11ebffa (misc.json: add migrate-set-sev-info command)
9/13 Checking commit 0365253b8689 (target/i386: sev: add support to encrypt the outgoing page)
WARNING: line over 80 characters
#199: FILE: target/i386/sev.c:978:
+    ret = sev_ioctl(sev_state->sev_fd, KVM_SEV_SEND_UPDATE_DATA, update, fw_err);

WARNING: Block comments use a leading /* on a separate line
#222: FILE: target/i386/sev.c:1001:
+    /* If this is first call then query the packet header bytes and allocate

total: 0 errors, 2 warnings, 274 lines checked

Patch 9/13 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
10/13 Checking commit 744b933d900c (target/i386: sev: add support to load incoming encrypted page)
WARNING: Block comments use a leading /* on a separate line
#167: FILE: target/i386/sev.c:1181:
+    /* If this is first buffer and SEV is not in recieiving state then

total: 0 errors, 1 warnings, 157 lines checked

Patch 10/13 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
11/13 Checking commit cc5d45952404 (kvm: introduce high-level API to migrate the page encryption bitmap)
12/13 Checking commit 00c1826c9c19 (migration: add support to migrate page encryption bitmap)
WARNING: line over 80 characters
#40: FILE: migration/ram.c:81:
+#define RAM_SAVE_FLAG_PAGE_ENCRYPTED_BITMAP       0x400 /* used in target/i386/sev.c */

ERROR: spaces required around that '+' (ctx:VxV)
#84: FILE: target/i386/sev.c:1193:
+#define ALIGN(x, y)  (((x)+(y)-1) & ~((y)-1))
                           ^

ERROR: spaces required around that '-' (ctx:VxV)
#84: FILE: target/i386/sev.c:1193:
+#define ALIGN(x, y)  (((x)+(y)-1) & ~((y)-1))
                               ^

ERROR: spaces required around that '-' (ctx:VxV)
#84: FILE: target/i386/sev.c:1193:
+#define ALIGN(x, y)  (((x)+(y)-1) & ~((y)-1))
                                          ^

WARNING: Block comments use a leading /* on a separate line
#126: FILE: target/i386/sev.c:1235:
+    size = ALIGN((length >> TARGET_PAGE_BITS), /*HOST_LONG_BITS*/ 64) / 8;

total: 3 errors, 2 warnings, 123 lines checked

Patch 12/13 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

13/13 Checking commit 9eb9639297de (target/i386: sev: remove migration blocker)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190710202219.25939-1-brijesh.singh@amd.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com