include/system/ramblock.h | 8 ++ migration/migration.c | 10 +- migration/migration.h | 5 + migration/options.c | 11 +- migration/options.h | 1 + migration/postcopy-ram.c | 167 ++++++++++++++++++++++++++--- migration/postcopy-ram.h | 2 + migration/qemu-file.c | 10 +- migration/ram.c | 61 +++++++++-- migration/savevm.c | 52 ++++++++- migration/savevm.h | 2 + migration/trace-events | 2 + tests/qtest/migration/misc-tests.c | 52 --------- 13 files changed, 283 insertions(+), 100 deletions(-)
This RFC implements a "fast snapshot load" mechanism to significantly
reduce the perceived resume time of a VM from a snapshot file.
Currently, resuming a VM from a snapshot file requires loading all RAM
pages into the QEMU instance before execution begins. This extension
allows the user to run the VM nearly instantly by loading only the
required device states up front and loading RAM pages lazily, by
trapping access to pages that have not yet been loaded.
Using the Linux userfaultfd syscall, a fault thread catches all page
faults caused by the guest and loads in the pages required to keep
the VM running. Concurrently, an eager background thread iteratively
loads all remaining pages into RAM so the guest does not have to
depend on the fault thread indefinitely.
Much of code is reused from postcopy for fault handling and precopy
for reading mapped ram file. Implementation revolves around two
threads named the fault thread and eager load thread. Fault thread as
name suggests catches page faults by the guest and serves them using
userfaultfd. Postcopy fault thread is reused but instead of requesting
source for a page it loads the page directly by reading form file. In
order to remove the dependency of guest on fault thread indefinitely
the eager load thread loads in the entire RAM sequentially, and after
iterating through the entire RAM signals fault thread to exit and
calls cleanup.
In order to prevent the case of a page being loaded twice(in the
case when eager load thread is loading it and fault thread also
tries to serve fault on same page) a bitmap called pending_bmap is
used to track pages which are pending and not being loaded by any
thread. Atomic operations on this bitmap allows coordination between
threads to prevent any unwanted behaviours
This patch was tested using a Debian 13 bare minimum system and Fedora
44 KDE, snapshots for both are loaded successfully with no error.
Next Steps:
- Add testing framework, in qtest and unit tests
- Add support for postcopy-blocktime
- Update documentation
Future direction:
- Add support for hugepages
- Add support for multifd
- Add support for vhost-user
Aadeshveer Singh (5):
migration: add RAM Block fields and helpers for fast snapshot load
migration: add support for fault thread to load pages from disk
migration: add eager load thread for fast snapshot load
migration: write up code to run fast snapshot load in
qemu_loadvm_state
migration/tests: remove capability conflict test
postcopy-ram+mapped-ram
include/system/ramblock.h | 8 ++
migration/migration.c | 10 +-
migration/migration.h | 5 +
migration/options.c | 11 +-
migration/options.h | 1 +
migration/postcopy-ram.c | 167 ++++++++++++++++++++++++++---
migration/postcopy-ram.h | 2 +
migration/qemu-file.c | 10 +-
migration/ram.c | 61 +++++++++--
migration/savevm.c | 52 ++++++++-
migration/savevm.h | 2 +
migration/trace-events | 2 +
tests/qtest/migration/misc-tests.c | 52 ---------
13 files changed, 283 insertions(+), 100 deletions(-)
--
2.54.0
Hi Everyone,
Adding this small patch on top to enable blocktime support. I will include
it directly when I repost the v2 RFC.
Running cold cache tests most faults lie in the [64 us - 128 us] range(most
likely pages loaded from disk) and a secondary mean is around [4 us - 8
us](likely for directly served zero pages).
Per CPU blocktime is a fraction of a second and offers a visible ergonomic
improvement over precopy migration for snapshot loads, which previously
required a few seconds to load the guest.
Test context:
- Guest: Fedora 44 (4 core, 16 GB RAM)
- Host: Fedora 43
- Host CPU: Intel ultra9 185H(22 cores)
- Host RAM: 32 GB DDR5x
- Host Drive: Gen 4 NVMe SSD
- Acceleration: KVM enabled
Cold Cache Output:
Postcopy Blocktime (ms): 0
Postcopy vCPU Blocktime (ms):
[128, 117, 111, 97]
Postcopy Latency (ns): 153571
Postcopy non-vCPU Latencies (ns): 149885
Postcopy vCPU Latencies (ns):
[234229, 225562, 195604, 209678]
Postcopy Latency Distribution:
[ 1 us - 2 us ]: 25
[ 2 us - 4 us ]: 370
[ 4 us - 8 us ]: 3938
[ 8 us - 16 us ]: 3599
[ 16 us - 32 us ]: 520
[ 32 us - 64 us ]: 192
[ 64 us - 128 us ]: 18643
[ 128 us - 256 us ]: 6199
[ 256 us - 512 us ]: 2768
[ 512 us - 1 ms ]: 1096
[ 1 ms - 2 ms ]: 486
[ 2 ms - 4 ms ]: 78
[ 4 ms - 8 ms ]: 5
[ 8 ms - 16 ms ]: 0
[ 16 ms - 32 ms ]: 0
[ 32 ms - 65 ms ]: 0
[ 65 ms - 131 ms ]: 0
[ 131 ms - 262 ms ]: 0
[ 262 ms - 524 ms ]: 0
[ 524 ms - 1 sec ]: 0
[ 1 sec - 2 sec ]: 0
[ 2 sec - 4 sec ]: 0
[ 4 sec - 8 sec ]: 0
[ 8 sec - 16 sec ]: 0
Thank you,
Aadeshveer
On Thu, Jun 18, 2026 at 8:50 AM Aadeshveer Singh <aadeshveer07@gmail.com>
wrote:
> This RFC implements a "fast snapshot load" mechanism to significantly
> reduce the perceived resume time of a VM from a snapshot file.
>
> Currently, resuming a VM from a snapshot file requires loading all RAM
> pages into the QEMU instance before execution begins. This extension
> allows the user to run the VM nearly instantly by loading only the
> required device states up front and loading RAM pages lazily, by
> trapping access to pages that have not yet been loaded.
>
> Using the Linux userfaultfd syscall, a fault thread catches all page
> faults caused by the guest and loads in the pages required to keep
> the VM running. Concurrently, an eager background thread iteratively
> loads all remaining pages into RAM so the guest does not have to
> depend on the fault thread indefinitely.
>
> Much of code is reused from postcopy for fault handling and precopy
> for reading mapped ram file. Implementation revolves around two
> threads named the fault thread and eager load thread. Fault thread as
> name suggests catches page faults by the guest and serves them using
> userfaultfd. Postcopy fault thread is reused but instead of requesting
> source for a page it loads the page directly by reading form file. In
> order to remove the dependency of guest on fault thread indefinitely
> the eager load thread loads in the entire RAM sequentially, and after
> iterating through the entire RAM signals fault thread to exit and
> calls cleanup.
>
> In order to prevent the case of a page being loaded twice(in the
> case when eager load thread is loading it and fault thread also
> tries to serve fault on same page) a bitmap called pending_bmap is
> used to track pages which are pending and not being loaded by any
> thread. Atomic operations on this bitmap allows coordination between
> threads to prevent any unwanted behaviours
>
> This patch was tested using a Debian 13 bare minimum system and Fedora
> 44 KDE, snapshots for both are loaded successfully with no error.
>
> Next Steps:
> - Add testing framework, in qtest and unit tests
> - Add support for postcopy-blocktime
> - Update documentation
>
> Future direction:
> - Add support for hugepages
> - Add support for multifd
> - Add support for vhost-user
>
> Aadeshveer Singh (5):
> migration: add RAM Block fields and helpers for fast snapshot load
> migration: add support for fault thread to load pages from disk
> migration: add eager load thread for fast snapshot load
> migration: write up code to run fast snapshot load in
> qemu_loadvm_state
> migration/tests: remove capability conflict test
> postcopy-ram+mapped-ram
>
> include/system/ramblock.h | 8 ++
> migration/migration.c | 10 +-
> migration/migration.h | 5 +
> migration/options.c | 11 +-
> migration/options.h | 1 +
> migration/postcopy-ram.c | 167 ++++++++++++++++++++++++++---
> migration/postcopy-ram.h | 2 +
> migration/qemu-file.c | 10 +-
> migration/ram.c | 61 +++++++++--
> migration/savevm.c | 52 ++++++++-
> migration/savevm.h | 2 +
> migration/trace-events | 2 +
> tests/qtest/migration/misc-tests.c | 52 ---------
> 13 files changed, 283 insertions(+), 100 deletions(-)
>
> --
> 2.54.0
>
>
On Fri, Jun 19, 2026 at 06:48:57PM +0530, Aadeshveer Singh wrote:
> From 70ab2949ef99968c2fc16e6a0d9860a993514367 Mon Sep 17 00:00:00 2001
> From: Aadeshveer Singh <aadeshveer07@gmail.com>
> Date: Fri, 19 Jun 2026 18:12:36 +0530
> Subject: [PATCH] migration: postcopy-blocktime support for fast snapshot load
>
> Add postcopy-blocktime support to fast snapshot load by calling
> mark_postcopy_blocktime_begin(), on all page faults intercepted by fault
> thread.
>
> There is no need to call mark_postcopy_blocktime_end(), as
> postcopy_mapped_ram_load_page() calls postcopy_place_page() and
> postcopy_place_page_zero() which call the end marking internally.
>
> Signed-off-by: Aadeshveer Singh <aadeshveer07@gmail.com>
> ---
> migration/postcopy-ram.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> index 0ee294a381..2f4698fbed 100644
> --- a/migration/postcopy-ram.c
> +++ b/migration/postcopy-ram.c
> @@ -1436,6 +1436,12 @@ static void *postcopy_ram_fault_thread(void *opaque)
> msg.arg.pagefault.feat.ptid);
>
> if (migrate_fast_snapshot_load()) {
> + WITH_QEMU_LOCK_GUARD(&mis->page_request_mutex)
> + {
> + mark_postcopy_blocktime_begin(msg.arg.pagefault.address,
> + msg.arg.pagefault.feat.ptid,
> + rb);
> + }
> if (postcopy_mapped_ram_load_page(
> mis, rb, rb_offset, msg.arg.pagefault.address, 1)) {
> break;
Let's squash this directly to your core patch 4, then mention it in the
commit log.
Even if I left quite some comments, most of them are small nitpicks. It's
good to know there're only a few postcopy functions need some touch, and
most logics can be reused.
The RFC series looks a great start, thank you!
--
Peter Xu
Hi Peter,
Thank you for the detailed review of the RFC.
I agree with the feedback across the series. For v2, I will:
- Squash the blocktime patch directly into Patch 4 as you suggested.
- Apply the QEMU styling fixes (reverse christmas tree,
g_clear_pointer, macro braces).
- Replace the g_assert() calls with standard Error errp propagation to
gracefully handle disk read failures.
- Move the capability conflict test removal to Patch 1.
I have one architectural clarification regarding the bitmap usage on
Patch 1, which I will reply to inline on that specific thread. I will
get to work on v2.
Thank you,
Aadeshveer Singh
On Tue, Jun 23, 2026 at 12:49 AM Peter Xu <peterx@redhat.com> wrote:
>
> On Fri, Jun 19, 2026 at 06:48:57PM +0530, Aadeshveer Singh wrote:
> > From 70ab2949ef99968c2fc16e6a0d9860a993514367 Mon Sep 17 00:00:00 2001
> > From: Aadeshveer Singh <aadeshveer07@gmail.com>
> > Date: Fri, 19 Jun 2026 18:12:36 +0530
> > Subject: [PATCH] migration: postcopy-blocktime support for fast snapshot load
> >
> > Add postcopy-blocktime support to fast snapshot load by calling
> > mark_postcopy_blocktime_begin(), on all page faults intercepted by fault
> > thread.
> >
> > There is no need to call mark_postcopy_blocktime_end(), as
> > postcopy_mapped_ram_load_page() calls postcopy_place_page() and
> > postcopy_place_page_zero() which call the end marking internally.
> >
> > Signed-off-by: Aadeshveer Singh <aadeshveer07@gmail.com>
> > ---
> > migration/postcopy-ram.c | 6 ++++++
> > 1 file changed, 6 insertions(+)
> >
> > diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> > index 0ee294a381..2f4698fbed 100644
> > --- a/migration/postcopy-ram.c
> > +++ b/migration/postcopy-ram.c
> > @@ -1436,6 +1436,12 @@ static void *postcopy_ram_fault_thread(void *opaque)
> > msg.arg.pagefault.feat.ptid);
> >
> > if (migrate_fast_snapshot_load()) {
> > + WITH_QEMU_LOCK_GUARD(&mis->page_request_mutex)
> > + {
> > + mark_postcopy_blocktime_begin(msg.arg.pagefault.address,
> > + msg.arg.pagefault.feat.ptid,
> > + rb);
> > + }
> > if (postcopy_mapped_ram_load_page(
> > mis, rb, rb_offset, msg.arg.pagefault.address, 1)) {
> > break;
>
> Let's squash this directly to your core patch 4, then mention it in the
> commit log.
>
> Even if I left quite some comments, most of them are small nitpicks. It's
> good to know there're only a few postcopy functions need some touch, and
> most logics can be reused.
>
> The RFC series looks a great start, thank you!
>
> --
> Peter Xu
>
Aadeshveer Singh <aadeshveer07@gmail.com> writes: > This RFC implements a "fast snapshot load" mechanism to significantly > reduce the perceived resume time of a VM from a snapshot file. > > Currently, resuming a VM from a snapshot file requires loading all RAM > pages into the QEMU instance before execution begins. This extension > allows the user to run the VM nearly instantly by loading only the > required device states up front and loading RAM pages lazily, by > trapping access to pages that have not yet been loaded. > > Using the Linux userfaultfd syscall, a fault thread catches all page > faults caused by the guest and loads in the pages required to keep > the VM running. Concurrently, an eager background thread iteratively > loads all remaining pages into RAM so the guest does not have to > depend on the fault thread indefinitely. > > Much of code is reused from postcopy for fault handling and precopy > for reading mapped ram file. Implementation revolves around two > threads named the fault thread and eager load thread. Fault thread as > name suggests catches page faults by the guest and serves them using > userfaultfd. Postcopy fault thread is reused but instead of requesting > source for a page it loads the page directly by reading form file. In > order to remove the dependency of guest on fault thread indefinitely > the eager load thread loads in the entire RAM sequentially, and after > iterating through the entire RAM signals fault thread to exit and > calls cleanup. > > In order to prevent the case of a page being loaded twice(in the > case when eager load thread is loading it and fault thread also > tries to serve fault on same page) a bitmap called pending_bmap is > used to track pages which are pending and not being loaded by any > thread. Atomic operations on this bitmap allows coordination between > threads to prevent any unwanted behaviours > How does it work if a second savevm happens while the RAM has not been yet entirely loaded? > This patch was tested using a Debian 13 bare minimum system and Fedora > 44 KDE, snapshots for both are loaded successfully with no error. > > Next Steps: > - Add testing framework, in qtest and unit tests > - Add support for postcopy-blocktime > - Update documentation > > Future direction: > - Add support for hugepages > - Add support for multifd > - Add support for vhost-user > > Aadeshveer Singh (5): > migration: add RAM Block fields and helpers for fast snapshot load > migration: add support for fault thread to load pages from disk > migration: add eager load thread for fast snapshot load > migration: write up code to run fast snapshot load in > qemu_loadvm_state > migration/tests: remove capability conflict test > postcopy-ram+mapped-ram > > include/system/ramblock.h | 8 ++ > migration/migration.c | 10 +- > migration/migration.h | 5 + > migration/options.c | 11 +- > migration/options.h | 1 + > migration/postcopy-ram.c | 167 ++++++++++++++++++++++++++--- > migration/postcopy-ram.h | 2 + > migration/qemu-file.c | 10 +- > migration/ram.c | 61 +++++++++-- > migration/savevm.c | 52 ++++++++- > migration/savevm.h | 2 + > migration/trace-events | 2 + > tests/qtest/migration/misc-tests.c | 52 --------- > 13 files changed, 283 insertions(+), 100 deletions(-)
On Wed, Jun 24, 2026 at 7:43 PM Fabiano Rosas <farosas@suse.de> wrote: > > Aadeshveer Singh <aadeshveer07@gmail.com> writes: > > > This RFC implements a "fast snapshot load" mechanism to significantly > > reduce the perceived resume time of a VM from a snapshot file. > > > > Currently, resuming a VM from a snapshot file requires loading all RAM > > pages into the QEMU instance before execution begins. This extension > > allows the user to run the VM nearly instantly by loading only the > > required device states up front and loading RAM pages lazily, by > > trapping access to pages that have not yet been loaded. > > > > Using the Linux userfaultfd syscall, a fault thread catches all page > > faults caused by the guest and loads in the pages required to keep > > the VM running. Concurrently, an eager background thread iteratively > > loads all remaining pages into RAM so the guest does not have to > > depend on the fault thread indefinitely. > > > > Much of code is reused from postcopy for fault handling and precopy > > for reading mapped ram file. Implementation revolves around two > > threads named the fault thread and eager load thread. Fault thread as > > name suggests catches page faults by the guest and serves them using > > userfaultfd. Postcopy fault thread is reused but instead of requesting > > source for a page it loads the page directly by reading form file. In > > order to remove the dependency of guest on fault thread indefinitely > > the eager load thread loads in the entire RAM sequentially, and after > > iterating through the entire RAM signals fault thread to exit and > > calls cleanup. > > > > In order to prevent the case of a page being loaded twice(in the > > case when eager load thread is loading it and fault thread also > > tries to serve fault on same page) a bitmap called pending_bmap is > > used to track pages which are pending and not being loaded by any > > thread. Atomic operations on this bitmap allows coordination between > > threads to prevent any unwanted behaviours > > > > How does it work if a second savevm happens while the RAM has not been > yet entirely loaded? Hi Fabiano, Thanks for the review across the series. To answer this: If a second savevm is issued before the RAM is fully loaded, it will be blocked. Because the migration state remains MIGRATION_STATUS_POSTCOPY_ACTIVE until the eager thread finishes reading the last page from disk, standard QEMU migration checks will reject any new savevm or migration requests. > > > This patch was tested using a Debian 13 bare minimum system and Fedora > > 44 KDE, snapshots for both are loaded successfully with no error. > > > > Next Steps: > > - Add testing framework, in qtest and unit tests > > - Add support for postcopy-blocktime > > - Update documentation > > > > Future direction: > > - Add support for hugepages > > - Add support for multifd > > - Add support for vhost-user > > > > Aadeshveer Singh (5): > > migration: add RAM Block fields and helpers for fast snapshot load > > migration: add support for fault thread to load pages from disk > > migration: add eager load thread for fast snapshot load > > migration: write up code to run fast snapshot load in > > qemu_loadvm_state > > migration/tests: remove capability conflict test > > postcopy-ram+mapped-ram > > > > include/system/ramblock.h | 8 ++ > > migration/migration.c | 10 +- > > migration/migration.h | 5 + > > migration/options.c | 11 +- > > migration/options.h | 1 + > > migration/postcopy-ram.c | 167 ++++++++++++++++++++++++++--- > > migration/postcopy-ram.h | 2 + > > migration/qemu-file.c | 10 +- > > migration/ram.c | 61 +++++++++-- > > migration/savevm.c | 52 ++++++++- > > migration/savevm.h | 2 + > > migration/trace-events | 2 + > > tests/qtest/migration/misc-tests.c | 52 --------- > > 13 files changed, 283 insertions(+), 100 deletions(-)
© 2016 - 2026 Red Hat, Inc.