The workflow to make a snapshot is the following:
1. Pause the vm
2. Make a snapshot of block devices using the scheme of your choice
3. Turn on background-snapshot migration capability
4. Start the migration using the destination (migration stream) of your choice.
   The migration will resume the vm execution by itself
   when it has the devices' states saved  and is ready to start ram writing
   to the migration stream.
5. Listen to the migration finish event

The bakground snapshot works with support of KVM patch:
"x86: mmu: report failed memory access to the userspace"
(not applied to the mainstream, it's in the kvm mailing list)

--
Change log:

v0 => v1:
============
the patch series has been split in smaller chunks

Denis Plotnikov (17):
  migration: add background snapshot capability
  bitops: add some atomic versions of bitmap operations
  threads: add infrastructure to process sigsegv
  background snapshot: make a dedicated type for ram block list
  ram: extend the data structures for background snapshotting
  background snapshot: add helpers to manage a copy of ram block list
  background snapshot: introduce page buffer
  migration: add helpers to change VM memory protection rights
  background snapshot: extend RAM request for holding a page copy
    pointer
  background snapshots: adapt the page queueing code for using page
    copies
  background snapshot: add a memory page copying function
  ram: add background snapshot support in ram page saving part of
    migration
  background snapshot: add write-protected page access handler function
  kvm: add failed memeory access exit reason
  kvm: add vCPU failed memeory access processing
  migration: move the device state saving logic to a separate function
  background snapshot: enable background snapshot

 include/exec/ram_addr.h   |   7 +
 include/exec/ramlist.h    |   4 +-
 include/qemu/bitops.h     |  25 +++
 include/qemu/thread.h     |   5 +
 linux-headers/linux/kvm.h |   5 +
 migration/migration.c     | 140 +++++++++++++-
 migration/migration.h     |   1 +
 migration/ram.c           | 374 ++++++++++++++++++++++++++++++++++++--
 migration/ram.h           |  17 +-
 migration/savevm.c        |  91 +++++-----
 migration/savevm.h        |   2 +
 qapi/migration.json       |   6 +-
 target/i386/kvm.c         |  17 ++
 util/qemu-thread-posix.c  |  52 ++++++
 14 files changed, 684 insertions(+), 62 deletions(-)

-- 
2.17.0

On Wed, Jul 18, 2018 at 06:41:43PM +0300, Denis Plotnikov wrote:
> The workflow to make a snapshot is the following:
> 1. Pause the vm
> 2. Make a snapshot of block devices using the scheme of your choice
> 3. Turn on background-snapshot migration capability
> 4. Start the migration using the destination (migration stream) of your choice.
>    The migration will resume the vm execution by itself
>    when it has the devices' states saved  and is ready to start ram writing
>    to the migration stream.
> 5. Listen to the migration finish event
> 
> The bakground snapshot works with support of KVM patch:
> "x86: mmu: report failed memory access to the userspace"
> (not applied to the mainstream, it's in the kvm mailing list)

Hello, Denis,

Do you mind to push your tree to an online repository in case to make
review easier?

Thanks,

-- 
Peter Xu

Hi Peter

I moved the code to the repository 
https://github.com/denis-plotnikov/qemu/tree/background-snapshot-kvm.
the current version includes fixes with respect to your comments for 
version 1.
I moved KVM related patches to the end of the branch (formerly patch 
series).
Since, the KVM patches and the other parts to modify (vhost an others) 
are needless in favor of upcoming userfaltfd,
I would ask you to review the general framework which is able to work 
with tcg.

Thanks in advance!

Denis

On 20.07.2018 12:27, Peter Xu wrote:
> On Wed, Jul 18, 2018 at 06:41:43PM +0300, Denis Plotnikov wrote:
>> The workflow to make a snapshot is the following:
>> 1. Pause the vm
>> 2. Make a snapshot of block devices using the scheme of your choice
>> 3. Turn on background-snapshot migration capability
>> 4. Start the migration using the destination (migration stream) of your choice.
>>     The migration will resume the vm execution by itself
>>     when it has the devices' states saved  and is ready to start ram writing
>>     to the migration stream.
>> 5. Listen to the migration finish event
>>
>> The bakground snapshot works with support of KVM patch:
>> "x86: mmu: report failed memory access to the userspace"
>> (not applied to the mainstream, it's in the kvm mailing list)
> 
> Hello, Denis,
> 
> Do you mind to push your tree to an online repository in case to make
> review easier?
> 
> Thanks,
> 

-- 
Best,
Denis

On Tue, Sep 04, 2018 at 04:00:31PM +0300, Denis Plotnikov wrote:
> Hi Peter

Hi, Denis,

> 
> I moved the code to the repository
> https://github.com/denis-plotnikov/qemu/tree/background-snapshot-kvm.
> the current version includes fixes with respect to your comments for version
> 1.
> I moved KVM related patches to the end of the branch (formerly patch
> series).
> Since, the KVM patches and the other parts to modify (vhost an others) are
> needless in favor of upcoming userfaltfd,
> I would ask you to review the general framework which is able to work with
> tcg.
> 
> Thanks in advance!

Thank you for pushing the tree.

I might have made a mistake before that I thought this work is at
least working for TCG, but I think I was wrong.  The problem is (I'm
trying to repeat Dave's question that you seems haven't yet answered):
even for TCG there could be use cases where the process might access
guest memory from the kernel space (e.g., vhost, or any system calls
that with a guest memory buffer passed in).  I'm afraid mprotect() and
the whole signal-based mechanism cannot be able to address these page
faults, then we'll encounter adhoc errors and we'll need to fix all
these places up.  Userfaultfd-wp should not have this problem.

I think the general idea of the work is good, but I'm not sure whether
we can merge the work if we don't settle these issues.

Regards,

-- 
Peter Xu

Hi Peter,

Thanks for the reply.

Ok, I understand about tcg.
So my only option is to wait for userfaultfd-wp.
Do you know if anyone is  currently working on this? And if so, then is 
there any estimations when the userfaultfd is ready?

Denis


On 05.09.2018 06:32, Peter Xu wrote:
> On Tue, Sep 04, 2018 at 04:00:31PM +0300, Denis Plotnikov wrote:
>> Hi Peter
> 
> Hi, Denis,
> 
>>
>> I moved the code to the repository
>> https://github.com/denis-plotnikov/qemu/tree/background-snapshot-kvm.
>> the current version includes fixes with respect to your comments for version
>> 1.
>> I moved KVM related patches to the end of the branch (formerly patch
>> series).
>> Since, the KVM patches and the other parts to modify (vhost an others) are
>> needless in favor of upcoming userfaltfd,
>> I would ask you to review the general framework which is able to work with
>> tcg.
>>
>> Thanks in advance!
> 
> Thank you for pushing the tree.
> 
> I might have made a mistake before that I thought this work is at
> least working for TCG, but I think I was wrong.  The problem is (I'm
> trying to repeat Dave's question that you seems haven't yet answered):
> even for TCG there could be use cases where the process might access
> guest memory from the kernel space (e.g., vhost, or any system calls
> that with a guest memory buffer passed in).  I'm afraid mprotect() and
> the whole signal-based mechanism cannot be able to address these page
> faults, then we'll encounter adhoc errors and we'll need to fix all
> these places up.  Userfaultfd-wp should not have this problem.
> 
> I think the general idea of the work is good, but I'm not sure whether
> we can merge the work if we don't settle these issues.
> 
> Regards,
> 

-- 
Best,
Denis

On Wed, Sep 05, 2018 at 12:18:09PM +0300, Denis Plotnikov wrote:
> Hi Peter,
> 
> Thanks for the reply.
> 
> Ok, I understand about tcg.
> So my only option is to wait for userfaultfd-wp.
> Do you know if anyone is  currently working on this? And if so, then is
> there any estimations when the userfaultfd is ready?

I am working on it, but it's only progressing slowly.  The latest
status is that I can run with QEMU/tcg now with Andrea's tree (will
need two more patches to be applied upon from me) however all the
changes are still unmature (e.g., one of the patch is discussing on
the linux-mm list, the other one I haven't discussed with anyone; but
these patches worked for me).

Let me know if want to play with the tree, at least I can share with
you on what I got (so maybe at least you can try to run your live
snapshot with QEMU/tcg with my userfaultfd tree; if error occurred we
can investigate).  Again, it is always welcomed if you want to
participate in the kernel work too.

Let me know what you think about it.

Regards,

-- 
Peter Xu