> -----Original Message-----
> From: Qemu-devel
> [mailto:qemu-devel-bounces+shameerali.kolothum.thodi=huawei.com@nong
> nu.org] On Behalf Of Dov Murik
> Sent: 23 August 2021 15:16
> To: qemu-devel@nongnu.org
> Cc: Tom Lendacky <thomas.lendacky@amd.com>; Ashish Kalra
> <ashish.kalra@amd.com>; Brijesh Singh <brijesh.singh@amd.com>; Michael
> S. Tsirkin <mst@redhat.com>; Steve Rutherford <srutherford@google.com>;
> James Bottomley <jejb@linux.ibm.com>; Juan Quintela
> <quintela@redhat.com>; Dr. David Alan Gilbert <dgilbert@redhat.com>; Dov
> Murik <dovmurik@linux.ibm.com>; Hubertus Franke <frankeh@us.ibm.com>;
> Tobin Feldman-Fitzthum <tobin@linux.ibm.com>; Paolo Bonzini
> <pbonzini@redhat.com>
> Subject: [RFC PATCH v2 00/12] Confidential guest-assisted live migration
>
> This is an RFC series for fast migration of confidential guests using an
> in-guest migration helper that lives in OVMF. QEMU VM live migration
> needs to read source VM's RAM and write it in the target VM; this
> mechanism doesn't work when the guest memory is encrypted or QEMU is
> prevented from reading it in another way. In order to support live
> migration in such scenarios, we introduce an in-guest migration helper
> which can securely extract RAM content from the guest in order to send
> it to the target. The migration helper is implemented as part of the
> VM's firmware in OVMF.
>
> We've implemented and tested this on AMD SEV, but expect most of the
> processes can be used with other technologies that prevent direct access
> of hypervisor to the guest's memory. Specifically, we don't use SEV's
> PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all;
> but note that the mirror VM relies on
> KVM_CAP_VM_COPY_ENC_CONTEXT_FROM
> to shared the SEV ASID with the main VM.
Hi Dov,
Sorry if I missed out, but just to check if there are any updates to or revised
one to this series? This guest-assisted method seems to be a good generic
approach for live migration and just wondering whether it is worth taking a
look for ARM CCA as well(I am not sure ARM RMM spec will have any
specific proposal for live migration or not, but couldn't find anything
public yet).
Please let me know if you plan to re-spin or there are any concerns with
this approach. Appreciate if you can point me to any relevant discussion
threads.
Thanks,
Shameer
>
> Corresponding RFC patches for OVMF have been posted by Tobin
> Feldman-Fitzthum on edk2-devel [1]. Those include the crux of the
> migration helper: a mailbox protocol over a shared memory page which
> allows communication between QEMU and the migration helper. In the
> source VM this is used to read a page and encrypt it for transport; in
> the target it is used to decrypt the incoming page and storing the
> content in the correct address in the guest memory. All encryption and
> decryption operations occur inside the trusted context in the VM, and
> therefore the VM's memory plaintext content is never accessible to the
> hosts participating in the migration.
>
> In order to allow OVMF to run the migration helper in parallel to the
> guest OS, we use a mirror VM [3], which shares the same memory mapping
> and SEV ASID as the main VM but has its own run loop. To start the
> mirror vcpu and the migration handler, we added a temporary
> start-migration-handler QMP command; this will be removed in a future
> version to run as part of the migrate QMP command.
>
> In the target VM we need the migration handler running to receive
> incoming RAM pages; to achieve that, we boot the VM into OVMF with a
> special fw_cfg value that causes OVMF to not boot the guest OS; we then
> allow QEMU to receive an incoming migration by issuing a new
> start-migrate-incoming QMP command.
>
> The confidential RAM migration requires checking whether a given guest
> RAM page is encrypted or not. This is achieved using SEV shared regions
> list tracking, which is implemented as part the SEV live migration patch
> series [2]. This feature tracks hypercalls from OVMF and guest Linux to
> report changes of page encryption status so that QEMU has an up-to-date
> view of which memory regions are shared and which are encrypted.
>
> We left a few unfinished edges in this RFC but decided to publish it to
> start the commmunity discussion. TODOs:
>
> 1. QMP commands start-migration-handler and start-migrate-incoming are
> developer tools and should be performed automatically.
> 2. The entry point address of the in-guest migration handler and its GDT
> are currently hard-coded in QEMU (patch 8); instead they should be
> discovered using pc_system_ovmf_table_find. Same applies for the
> mailbox address (patch 1).
> 3. For simplicity, this patch series forces the use of the
> guest-assisted migration instead of the SEV PSP-based migration.
> Ideally we might want the user to choose the desired mode using
> migrate-set-parameters or a similar mechanism.
> 4. There is currently no discovery protocol between QEMU and OVMF to
> verify that OVMF indeed supports in-guest migration handler.
>
>
> List of patches in this series:
>
> 1-3: introduce new confidtial RAM migration functions which communicate
> with the migration helper.
> 4-6: use the new MH communication functions when migrating encrypted
> RAM
> pages
> 7-9: allow starting migration handler on mirror vcpu with QMP command
> start-migration-handler
> 10: introduce the start-migrate-incoming QMP command to switch the
> target into accepting the incoming migration.
> 11: fix devices issues when loading state into a live VM
> 12: add documentation
>
>
> This patch series is based on top of:
>
> 1. Add SEV guest live migration support, from Ashish Kalra [2]
> 2. Support for mirror VM, from Ashish Kalra [3]
>
> [1] https://edk2.groups.io/g/devel/message/79517
> [2]
> https://lore.kernel.org/qemu-devel/cover.1628076205.git.ashish.kalra@amd
> .com/
> [3]
> https://lore.kernel.org/qemu-devel/cover.1629118207.git.ashish.kalra@amd
> .com/
>
>
> Changes from RFC v1:
> - Use the an SEV mirror VM for the migation handler (instead of
> auxilliary vcpus)
>
> RFC v1:
> https://lore.kernel.org/qemu-devel/20210302204822.81901-1-dovmurik@li
> nux.vnet.ibm.com/
>
>
> Dov Murik (12):
> migration: Add helpers to save confidential RAM
> migration: Add helpers to load confidential RAM
> migration: Introduce gpa_inside_migration_helper_shared_area
> migration: Save confidential guest RAM using migration helper
> migration: Load confidential guest RAM using migration helper
> migration: Skip ROM, non-RAM, and vga.vram memory region during RAM
> migration
> i386/kvm: Exclude mirror vcpu in kvm_synchronize_all_tsc
> migration: Allow resetting the mirror vcpu to the MH entry point
> migration: Add QMP command start-migration-handler
> migration: Add start-migrate-incoming QMP command
> hw/isa/lpc_ich9: Allow updating an already-running VM
> docs: Add confidential guest live migration documentation
>
> docs/confidential-guest-live-migration.rst | 145 +++++++++
> docs/confidential-guest-support.txt | 5 +
> docs/index.rst | 1 +
> qapi/migration.json | 38 +++
> include/sysemu/sev.h | 1 +
> migration/confidential-ram.h | 23 ++
> hw/isa/lpc_ich9.c | 3 +-
> migration/confidential-ram.c | 339
> +++++++++++++++++++++
> migration/migration.c | 29 ++
> migration/ram.c | 133 +++++++-
> target/i386/kvm/kvm.c | 4 +-
> migration/meson.build | 2 +-
> migration/trace-events | 4 +
> 13 files changed, 714 insertions(+), 13 deletions(-)
> create mode 100644 docs/confidential-guest-live-migration.rst
> create mode 100644 migration/confidential-ram.h
> create mode 100644 migration/confidential-ram.c
>
> --
> 2.20.1
>