From: Trieu Huynh <vikingtc4@gmail.com>

When query-migrate is called on the *destination* QEMU after a precopy
migration completes it returns only {"status": "completed"} — no timing,
no RAM statistics.  The source correctly returns total-time, downtime,
setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in
  MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI
  schema so the destination can populate info->ram with only the fields
  that are meaningful on the receive side.  Fields computed from tracked
  counters (transferred, normal, duplicate, mbps, pages-per-second) are
  shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
  absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are
not set are simply absent from the output.  Source-side callers are
unaffected because populate_ram_info() sets all optional fields via
has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:
{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
  migration: track timing and received pages in MigrationIncomingState
  migration: expose RAM stats and timing on destination via
    query-migrate

 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)

-- 
2.43.0

Trieu Huynh <vikingtc4@gmail.com> writes:

> From: Trieu Huynh <vikingtc4@gmail.com>
>
> When query-migrate is called on the *destination* QEMU after a precopy
> migration completes it returns only {"status": "completed"} — no timing,
> no RAM statistics.  The source correctly returns total-time, downtime,
> setup-time, and full ram stats.
>
> This series fixes the gap in two commits:
>
> * commit 1: Track start/end timestamps and per-page-type counters in
>   MigrationIncomingState, recorded in the receive path.
>
> * commit 2: Make source-only MigrationStats fields optional in the QAPI
>   schema so the destination can populate info->ram with only the fields
>   that are meaningful on the receive side.  Fields computed from tracked
>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
>   absent.
>
> The QAPI change (commit 2) is ABI-compatible: optional fields that are
> not set are simply absent from the output.  Source-side callers are
> unaffected because populate_ram_info() sets all optional fields via
> has_* setters, so the source output is identical to before.
>
> On dst, {"execute":"query-migrate"}
> * As-is:
> {
>     "return": {
>         "status": "completed"
> }
> * To-be:
> {
>     "return": {
>         "status": "completed",
>         "total-time": 94,
>         "ram": {
>             "total": 554508288,
>             "pages-per-second": 1440234,
>             "page-size": 4096,
>             "remaining": 0,
>             "mbps": 97.955404255319152,
>             "transferred": 1150976,
>             "duplicate": 135101,
>             "normal-bytes": 1150976,
>             "normal": 281
>         }
>     }
> }
>
> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
>
> Trieu Huynh (2):
>   migration: track timing and received pages in MigrationIncomingState
>   migration: expose RAM stats and timing on destination via
>     query-migrate
>
>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
>  migration/migration.h | 17 +++++++++++++++++
>  migration/ram.c       |  3 +++
>  qapi/migration.json   | 14 +++++++-------
>  4 files changed, 64 insertions(+), 7 deletions(-)

Hi, remember to copy the interested people in your series. Using
get_maintainers is correct, but when there's already a discussion about
the topic it's good to add some CCs manually.

+CC Daniel and Claudio

On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
> Trieu Huynh <vikingtc4@gmail.com> writes:
> 
> > From: Trieu Huynh <vikingtc4@gmail.com>
> >
> > When query-migrate is called on the *destination* QEMU after a precopy
> > migration completes it returns only {"status": "completed"} — no timing,
> > no RAM statistics.  The source correctly returns total-time, downtime,
> > setup-time, and full ram stats.
> >
> > This series fixes the gap in two commits:
> >
> > * commit 1: Track start/end timestamps and per-page-type counters in
> >   MigrationIncomingState, recorded in the receive path.
> >
> > * commit 2: Make source-only MigrationStats fields optional in the QAPI
> >   schema so the destination can populate info->ram with only the fields
> >   that are meaningful on the receive side.  Fields computed from tracked
> >   counters (transferred, normal, duplicate, mbps, pages-per-second) are
> >   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
> >   absent.
> >
> > The QAPI change (commit 2) is ABI-compatible: optional fields that are
> > not set are simply absent from the output.  Source-side callers are
> > unaffected because populate_ram_info() sets all optional fields via
> > has_* setters, so the source output is identical to before.
> >
> > On dst, {"execute":"query-migrate"}
> > * As-is:
> > {
> >     "return": {
> >         "status": "completed"
> > }
> > * To-be:
> > {
> >     "return": {
> >         "status": "completed",
> >         "total-time": 94,
> >         "ram": {
> >             "total": 554508288,
> >             "pages-per-second": 1440234,
> >             "page-size": 4096,
> >             "remaining": 0,
> >             "mbps": 97.955404255319152,
> >             "transferred": 1150976,
> >             "duplicate": 135101,
> >             "normal-bytes": 1150976,
> >             "normal": 281
> >         }
> >     }
> > }
> >
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
> >
> > Trieu Huynh (2):
> >   migration: track timing and received pages in MigrationIncomingState
> >   migration: expose RAM stats and timing on destination via
> >     query-migrate
> >
> >  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
> >  migration/migration.h | 17 +++++++++++++++++
> >  migration/ram.c       |  3 +++
> >  qapi/migration.json   | 14 +++++++-------
> >  4 files changed, 64 insertions(+), 7 deletions(-)
> 
> Hi, remember to copy the interested people in your series. Using
> get_maintainers is correct, but when there's already a discussion about
> the topic it's good to add some CCs manually.
> 
> +CC Daniel and Claudio

The request is a valid one.  Said that, redo accounting on both sides seem
to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies
and report it somewhere?  I recall these info were captured somewhere, do
we at least dump them into logs?

Thanks,

-- 
Peter Xu

On 4/8/26 22:04, Peter Xu wrote:
> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>
>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>
>>> When query-migrate is called on the *destination* QEMU after a precopy
>>> migration completes it returns only {"status": "completed"} — no timing,
>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>> setup-time, and full ram stats.
>>>
>>> This series fixes the gap in two commits:
>>>
>>> * commit 1: Track start/end timestamps and per-page-type counters in
>>>   MigrationIncomingState, recorded in the receive path.
>>>
>>> * commit 2: Make source-only MigrationStats fields optional in the QAPI
>>>   schema so the destination can populate info->ram with only the fields
>>>   that are meaningful on the receive side.  Fields computed from tracked
>>>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
>>>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
>>>   absent.
>>>
>>> The QAPI change (commit 2) is ABI-compatible: optional fields that are
>>> not set are simply absent from the output.  Source-side callers are
>>> unaffected because populate_ram_info() sets all optional fields via
>>> has_* setters, so the source output is identical to before.
>>>
>>> On dst, {"execute":"query-migrate"}
>>> * As-is:
>>> {
>>>     "return": {
>>>         "status": "completed"
>>> }
>>> * To-be:
>>> {
>>>     "return": {
>>>         "status": "completed",
>>>         "total-time": 94,
>>>         "ram": {
>>>             "total": 554508288,
>>>             "pages-per-second": 1440234,
>>>             "page-size": 4096,
>>>             "remaining": 0,
>>>             "mbps": 97.955404255319152,
>>>             "transferred": 1150976,
>>>             "duplicate": 135101,
>>>             "normal-bytes": 1150976,
>>>             "normal": 281
>>>         }
>>>     }
>>> }
>>>
>>> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
>>>
>>> Trieu Huynh (2):
>>>   migration: track timing and received pages in MigrationIncomingState
>>>   migration: expose RAM stats and timing on destination via
>>>     query-migrate
>>>
>>>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
>>>  migration/migration.h | 17 +++++++++++++++++
>>>  migration/ram.c       |  3 +++
>>>  qapi/migration.json   | 14 +++++++-------
>>>  4 files changed, 64 insertions(+), 7 deletions(-)
>>
>> Hi, remember to copy the interested people in your series. Using
>> get_maintainers is correct, but when there's already a discussion about
>> the topic it's good to add some CCs manually.
>>
>> +CC Daniel and Claudio
> 
> The request is a valid one.  Said that, redo accounting on both sides seem
> to be a duplicated work and overkill to me, if src has everything.
> 
> Is there a way we can make Libvirt always query the results before src dies
> and report it somewhere?  I recall these info were captured somewhere, do
> we at least dump them into logs?
> 
> Thanks,
> 

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer,
then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

Thanks,

Claudio

On April 9, 2026 12:08 pm, Claudio Fontana wrote:
> On 4/8/26 22:04, Peter Xu wrote:
>> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>>
>>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>>
>>>> When query-migrate is called on the *destination* QEMU after a precopy
>>>> migration completes it returns only {"status": "completed"} — no timing,
>>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>>> setup-time, and full ram stats.

>>>> [..]
>>>>
>>>
>>> Hi, remember to copy the interested people in your series. Using
>>> get_maintainers is correct, but when there's already a discussion about
>>> the topic it's good to add some CCs manually.
>>>
>>> +CC Daniel and Claudio
>> 
>> The request is a valid one.  Said that, redo accounting on both sides seem
>> to be a duplicated work and overkill to me, if src has everything.
>> 
>> Is there a way we can make Libvirt always query the results before src dies
>> and report it somewhere?  I recall these info were captured somewhere, do
>> we at least dump them into logs?
>> 
>> Thanks,
>> 
> 
> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
> 
> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

that is correct, we call query-migrate in a loop while migrating both to
log progress and to manage migration parameters, and manually switch
execution and ownership over to the target node once the migration has
converged:

https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

> So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
> so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
> It might go beyond querying for stats using QEMU and libvirt.
> 
> Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
> 
> VIR_MIGRATE_NO_SOURCE_SHUTOFF
> 
> So one might say:
> 
> flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
> new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
> 
> and then we could do everything that needs to be done from the platform layer,
> then issue a virDomainResume to start the domain on destination,
> and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
> 
> But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
> 
> Thanks,
> 
> Claudio
> 
> 
> 

On 4/9/26 15:08, Fabian Grünbichler wrote:
> On April 9, 2026 12:08 pm, Claudio Fontana wrote:
>> On 4/8/26 22:04, Peter Xu wrote:
>>> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>>>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>>>
>>>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>>>
>>>>> When query-migrate is called on the *destination* QEMU after a precopy
>>>>> migration completes it returns only {"status": "completed"} — no timing,
>>>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>>>> setup-time, and full ram stats.
> 
>>>>> [..]
>>>>>
>>>>
>>>> Hi, remember to copy the interested people in your series. Using
>>>> get_maintainers is correct, but when there's already a discussion about
>>>> the topic it's good to add some CCs manually.
>>>>
>>>> +CC Daniel and Claudio
>>>
>>> The request is a valid one.  Said that, redo accounting on both sides seem
>>> to be a duplicated work and overkill to me, if src has everything.
>>>
>>> Is there a way we can make Libvirt always query the results before src dies
>>> and report it somewhere?  I recall these info were captured somewhere, do
>>> we at least dump them into logs?
>>>
>>> Thanks,
>>>
>>
>> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
>>
>> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
> 
> that is correct, we call query-migrate in a loop while migrating both to
> log progress and to manage migration parameters, and manually switch
> execution and ownership over to the target node once the migration has
> converged:
> 
> https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

Hello Fabian, thanks for the confirmation,
would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?

I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?

> 
>> So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
>> so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
>> It might go beyond querying for stats using QEMU and libvirt.
>>
>> Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
>>
>> VIR_MIGRATE_NO_SOURCE_SHUTOFF
>>
>> So one might say:
>>
>> flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
>> new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
>>
>> and then we could do everything that needs to be done from the platform layer,
>> then issue a virDomainResume to start the domain on destination,
>> and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
>>
>> But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
>>
>> Thanks,
>>
>> Claudio
>>
>>
>>
> 

On April 9, 2026 3:17 pm, Claudio Fontana wrote:
> On 4/9/26 15:08, Fabian Grünbichler wrote:
>> On April 9, 2026 12:08 pm, Claudio Fontana wrote:
>>> On 4/8/26 22:04, Peter Xu wrote:
>>>> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>>>>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>>>>
>>>>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>>>>
>>>>>> When query-migrate is called on the *destination* QEMU after a precopy
>>>>>> migration completes it returns only {"status": "completed"} — no timing,
>>>>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>>>>> setup-time, and full ram stats.
>> 
>>>>>> [..]
>>>>>>
>>>>>
>>>>> Hi, remember to copy the interested people in your series. Using
>>>>> get_maintainers is correct, but when there's already a discussion about
>>>>> the topic it's good to add some CCs manually.
>>>>>
>>>>> +CC Daniel and Claudio
>>>>
>>>> The request is a valid one.  Said that, redo accounting on both sides seem
>>>> to be a duplicated work and overkill to me, if src has everything.
>>>>
>>>> Is there a way we can make Libvirt always query the results before src dies
>>>> and report it somewhere?  I recall these info were captured somewhere, do
>>>> we at least dump them into logs?
>>>>
>>>> Thanks,
>>>>
>>>
>>> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
>>>
>>> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
>> 
>> that is correct, we call query-migrate in a loop while migrating both to
>> log progress and to manage migration parameters, and manually switch
>> execution and ownership over to the target node once the migration has
>> converged:
>> 
>> https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423
> 
> Hello Fabian, thanks for the confirmation,
> would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?

not really - the source node is driving the migration anyway, so it can
also do the polling + logging. there is no management task running on
the target node/side, after the initial spawning of the VM process.

> I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?