[PATCH v1 0/8] migration: introduce dirtylimit capability

huangy81@chinatelecom.cn posted 8 patches 3 years, 5 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/cover.1662052189.git.huangy81@chinatelecom.cn
Maintainers: Juan Quintela <quintela@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>, Eric Blake <eblake@redhat.com>, Markus Armbruster <armbru@redhat.com>, Thomas Huth <thuth@redhat.com>, Laurent Vivier <lvivier@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>
There is a newer version of this series
include/sysemu/dirtylimit.h             |   2 +
migration/migration.c                   |  51 +++++++++++
migration/migration.h                   |   1 +
migration/ram.c                         |  53 ++++++++---
migration/trace-events                  |   1 +
monitor/hmp-cmds.c                      |  26 ++++++
qapi/migration.json                     |  57 ++++++++++--
softmmu/dirtylimit.c                    |  33 ++++++-
tests/migration/guestperf/comparison.py |  24 +++++
tests/migration/guestperf/engine.py     |  33 ++++++-
tests/migration/guestperf/hardware.py   |   8 +-
tests/migration/guestperf/progress.py   |  17 +++-
tests/migration/guestperf/scenario.py   |  11 ++-
tests/migration/guestperf/shell.py      |  25 +++++-
tests/qtest/migration-test.c            | 154 ++++++++++++++++++++++++++++++++
15 files changed, 465 insertions(+), 31 deletions(-)
[PATCH v1 0/8] migration: introduce dirtylimit capability
Posted by huangy81@chinatelecom.cn 3 years, 5 months ago
From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>

v1:
- make parameter vcpu-dirty-limit experimental 
- switch dirty limit off when cancel migrate
- add cancel logic in migration test 

Please review, thanks,

Yong 

Abstract
========

This series added a new migration capability called "dirtylimit".  It can
be enabled when dirty ring is enabled, and it'll improve the vCPU performance
during the process of migration. It is based on the previous patchset:
https://lore.kernel.org/qemu-devel/cover.1656177590.git.huangy81@chinatelecom.cn/

As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of
migration can make the read-process not be penalized. This series wires up the
vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce
two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup 
of dirtylimit during live migration.

To validate the implementation, i tested a 32 vCPU vm live migration with such 
model:
Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus
untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as
the following command:
taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}

The following are results:

host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s
  |---------------------+--------+------------+---------------|
  | UnixBench test item | Normal | Dirtylimit | Auto-converge |
  |---------------------+--------+------------+---------------|
  | dhry2reg            | 32800  | 32786      | 25292         |
  | whetstone-double    | 10326  | 10315      | 9847          |
  | pipe                | 15442  | 15271      | 14506         |
  | context1            | 7260   | 6235       | 4514          |
  | spawn               | 3663   | 3317       | 3249          |
  | syscall             | 4669   | 4667       | 3841          |
  |---------------------+--------+------------+---------------|
From the data above we can draw a conclusion that vcpus that do not dirty memory
in vm are almost unaffected during the dirtylimit migration, but the auto converge
way does. 

I also tested the total time of dirtylimit migration with variable dirty memory
size in vm.

senario 1:
host cpu: Intel(R) Xeon(R) Platinum 8378A
host interface speed: 1000Mb/s
  |-----------------------+----------------+-------------------|
  | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
  |-----------------------+----------------+-------------------|
  | 60                    | 2014           | 2131              |
  | 70                    | 5381           | 12590             |
  | 90                    | 6037           | 33545             |
  | 110                   | 7660           | [*]               |
  |-----------------------+----------------+-------------------|
  [*]: This case means migration is not convergent. 

senario 2:
host cpu: Intel(R) Xeon(R) CPU E5-2650
host interface speed: 10000Mb/s
  |-----------------------+----------------+-------------------|
  | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
  |-----------------------+----------------+-------------------|
  | 1600                  | 15842          | 27548             |
  | 2000                  | 19026          | 38447             |
  | 2400                  | 19897          | 46381             |
  | 2800                  | 22338          | 57149             |
  |-----------------------+----------------+-------------------|
Above data shows that dirtylimit way of migration can also reduce the total
time of migration and it achieves convergence more easily in some case.

In addition to implement dirtylimit capability itself, this series
add 3 tests for migration, aiming at playing around for developer simply: 
 1. qtest for dirty limit migration
 2. support dirty ring way of migration for guestperf tool
 3. support dirty limit migration for guestperf tool

Please review, thanks !

Hyman Huang (8):
  qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
  qapi/migration: Introduce x-vcpu-dirty-limit parameters
  migration: Introduce dirty-limit capability
  migration: Implement dirty-limit convergence algo
  migration: Export dirty-limit time info
  tests: Add migration dirty-limit capability test
  tests/migration: Introduce dirty-ring-size option into guestperf
  tests/migration: Introduce dirty-limit into guestperf

 include/sysemu/dirtylimit.h             |   2 +
 migration/migration.c                   |  51 +++++++++++
 migration/migration.h                   |   1 +
 migration/ram.c                         |  53 ++++++++---
 migration/trace-events                  |   1 +
 monitor/hmp-cmds.c                      |  26 ++++++
 qapi/migration.json                     |  57 ++++++++++--
 softmmu/dirtylimit.c                    |  33 ++++++-
 tests/migration/guestperf/comparison.py |  24 +++++
 tests/migration/guestperf/engine.py     |  33 ++++++-
 tests/migration/guestperf/hardware.py   |   8 +-
 tests/migration/guestperf/progress.py   |  17 +++-
 tests/migration/guestperf/scenario.py   |  11 ++-
 tests/migration/guestperf/shell.py      |  25 +++++-
 tests/qtest/migration-test.c            | 154 ++++++++++++++++++++++++++++++++
 15 files changed, 465 insertions(+), 31 deletions(-)

-- 
1.8.3.1


Re: [PATCH v1 0/8] migration: introduce dirtylimit capability
Posted by Markus Armbruster 3 years, 4 months ago
huangy81@chinatelecom.cn writes:

> From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
>
> v1:
> - make parameter vcpu-dirty-limit experimental 
> - switch dirty limit off when cancel migrate
> - add cancel logic in migration test 
>
> Please review, thanks,
>
> Yong 

Are you still pursuing this feature?

> Abstract
> ========
>
> This series added a new migration capability called "dirtylimit".  It can
> be enabled when dirty ring is enabled, and it'll improve the vCPU performance
> during the process of migration. It is based on the previous patchset:
> https://lore.kernel.org/qemu-devel/cover.1656177590.git.huangy81@chinatelecom.cn/
>
> As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of
> migration can make the read-process not be penalized. This series wires up the
> vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce
> two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup 
> of dirtylimit during live migration.
>
> To validate the implementation, i tested a 32 vCPU vm live migration with such 
> model:
> Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus
> untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as
> the following command:
> taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}
>
> The following are results:
>
> host cpu: Intel(R) Xeon(R) Platinum 8378A
> host interface speed: 1000Mb/s
>   |---------------------+--------+------------+---------------|
>   | UnixBench test item | Normal | Dirtylimit | Auto-converge |
>   |---------------------+--------+------------+---------------|
>   | dhry2reg            | 32800  | 32786      | 25292         |
>   | whetstone-double    | 10326  | 10315      | 9847          |
>   | pipe                | 15442  | 15271      | 14506         |
>   | context1            | 7260   | 6235       | 4514          |
>   | spawn               | 3663   | 3317       | 3249          |
>   | syscall             | 4669   | 4667       | 3841          |
>   |---------------------+--------+------------+---------------|
>>From the data above we can draw a conclusion that vcpus that do not dirty memory
> in vm are almost unaffected during the dirtylimit migration, but the auto converge
> way does. 
>
> I also tested the total time of dirtylimit migration with variable dirty memory
> size in vm.
>
> senario 1:
> host cpu: Intel(R) Xeon(R) Platinum 8378A
> host interface speed: 1000Mb/s
>   |-----------------------+----------------+-------------------|
>   | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>   |-----------------------+----------------+-------------------|
>   | 60                    | 2014           | 2131              |
>   | 70                    | 5381           | 12590             |
>   | 90                    | 6037           | 33545             |
>   | 110                   | 7660           | [*]               |
>   |-----------------------+----------------+-------------------|
>   [*]: This case means migration is not convergent. 
>
> senario 2:
> host cpu: Intel(R) Xeon(R) CPU E5-2650
> host interface speed: 10000Mb/s
>   |-----------------------+----------------+-------------------|
>   | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>   |-----------------------+----------------+-------------------|
>   | 1600                  | 15842          | 27548             |
>   | 2000                  | 19026          | 38447             |
>   | 2400                  | 19897          | 46381             |
>   | 2800                  | 22338          | 57149             |
>   |-----------------------+----------------+-------------------|
> Above data shows that dirtylimit way of migration can also reduce the total
> time of migration and it achieves convergence more easily in some case.
>
> In addition to implement dirtylimit capability itself, this series
> add 3 tests for migration, aiming at playing around for developer simply: 
>  1. qtest for dirty limit migration
>  2. support dirty ring way of migration for guestperf tool
>  3. support dirty limit migration for guestperf tool
>
> Please review, thanks !
>
> Hyman Huang (8):
>   qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
>   qapi/migration: Introduce x-vcpu-dirty-limit parameters
>   migration: Introduce dirty-limit capability
>   migration: Implement dirty-limit convergence algo
>   migration: Export dirty-limit time info
>   tests: Add migration dirty-limit capability test
>   tests/migration: Introduce dirty-ring-size option into guestperf
>   tests/migration: Introduce dirty-limit into guestperf
>
>  include/sysemu/dirtylimit.h             |   2 +
>  migration/migration.c                   |  51 +++++++++++
>  migration/migration.h                   |   1 +
>  migration/ram.c                         |  53 ++++++++---
>  migration/trace-events                  |   1 +
>  monitor/hmp-cmds.c                      |  26 ++++++
>  qapi/migration.json                     |  57 ++++++++++--
>  softmmu/dirtylimit.c                    |  33 ++++++-
>  tests/migration/guestperf/comparison.py |  24 +++++
>  tests/migration/guestperf/engine.py     |  33 ++++++-
>  tests/migration/guestperf/hardware.py   |   8 +-
>  tests/migration/guestperf/progress.py   |  17 +++-
>  tests/migration/guestperf/scenario.py   |  11 ++-
>  tests/migration/guestperf/shell.py      |  25 +++++-
>  tests/qtest/migration-test.c            | 154 ++++++++++++++++++++++++++++++++
>  15 files changed, 465 insertions(+), 31 deletions(-)
Re: [PATCH v1 0/8] migration: introduce dirtylimit capability
Posted by Hyman Huang 3 years, 4 months ago

在 2022/10/1 22:37, Markus Armbruster 写道:
> huangy81@chinatelecom.cn writes:
> 
>> From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
>>
>> v1:
>> - make parameter vcpu-dirty-limit experimental
>> - switch dirty limit off when cancel migrate
>> - add cancel logic in migration test
>>
>> Please review, thanks,
>>
>> Yong
> 
> Are you still pursuing this feature?
Yes, of course, but the detailed test report has not been prepared, and 
the last 3 commits of this patchset has not been commentted, i'm waiting 
for the it and the next version can be improved hugely.

Ping to Daniel and David, what do you think of these 3 test patches?
I would be very pleased if you could help me with the review.  :)

Thanks

Yong
> 
>> Abstract
>> ========
>>
>> This series added a new migration capability called "dirtylimit".  It can
>> be enabled when dirty ring is enabled, and it'll improve the vCPU performance
>> during the process of migration. It is based on the previous patchset:
>> https://lore.kernel.org/qemu-devel/cover.1656177590.git.huangy81@chinatelecom.cn/
>>
>> As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of
>> migration can make the read-process not be penalized. This series wires up the
>> vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce
>> two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup
>> of dirtylimit during live migration.
>>
>> To validate the implementation, i tested a 32 vCPU vm live migration with such
>> model:
>> Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus
>> untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as
>> the following command:
>> taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}
>>
>> The following are results:
>>
>> host cpu: Intel(R) Xeon(R) Platinum 8378A
>> host interface speed: 1000Mb/s
>>    |---------------------+--------+------------+---------------|
>>    | UnixBench test item | Normal | Dirtylimit | Auto-converge |
>>    |---------------------+--------+------------+---------------|
>>    | dhry2reg            | 32800  | 32786      | 25292         |
>>    | whetstone-double    | 10326  | 10315      | 9847          |
>>    | pipe                | 15442  | 15271      | 14506         |
>>    | context1            | 7260   | 6235       | 4514          |
>>    | spawn               | 3663   | 3317       | 3249          |
>>    | syscall             | 4669   | 4667       | 3841          |
>>    |---------------------+--------+------------+---------------|
>> >From the data above we can draw a conclusion that vcpus that do not dirty memory
>> in vm are almost unaffected during the dirtylimit migration, but the auto converge
>> way does.
>>
>> I also tested the total time of dirtylimit migration with variable dirty memory
>> size in vm.
>>
>> senario 1:
>> host cpu: Intel(R) Xeon(R) Platinum 8378A
>> host interface speed: 1000Mb/s
>>    |-----------------------+----------------+-------------------|
>>    | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>>    |-----------------------+----------------+-------------------|
>>    | 60                    | 2014           | 2131              |
>>    | 70                    | 5381           | 12590             |
>>    | 90                    | 6037           | 33545             |
>>    | 110                   | 7660           | [*]               |
>>    |-----------------------+----------------+-------------------|
>>    [*]: This case means migration is not convergent.
>>
>> senario 2:
>> host cpu: Intel(R) Xeon(R) CPU E5-2650
>> host interface speed: 10000Mb/s
>>    |-----------------------+----------------+-------------------|
>>    | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>>    |-----------------------+----------------+-------------------|
>>    | 1600                  | 15842          | 27548             |
>>    | 2000                  | 19026          | 38447             |
>>    | 2400                  | 19897          | 46381             |
>>    | 2800                  | 22338          | 57149             |
>>    |-----------------------+----------------+-------------------|
>> Above data shows that dirtylimit way of migration can also reduce the total
>> time of migration and it achieves convergence more easily in some case.
>>
>> In addition to implement dirtylimit capability itself, this series
>> add 3 tests for migration, aiming at playing around for developer simply:
>>   1. qtest for dirty limit migration
>>   2. support dirty ring way of migration for guestperf tool
>>   3. support dirty limit migration for guestperf tool
>>
>> Please review, thanks !
>>
>> Hyman Huang (8):
>>    qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
>>    qapi/migration: Introduce x-vcpu-dirty-limit parameters
>>    migration: Introduce dirty-limit capability
>>    migration: Implement dirty-limit convergence algo
>>    migration: Export dirty-limit time info
>>    tests: Add migration dirty-limit capability test
>>    tests/migration: Introduce dirty-ring-size option into guestperf
>>    tests/migration: Introduce dirty-limit into guestperf
>>
>>   include/sysemu/dirtylimit.h             |   2 +
>>   migration/migration.c                   |  51 +++++++++++
>>   migration/migration.h                   |   1 +
>>   migration/ram.c                         |  53 ++++++++---
>>   migration/trace-events                  |   1 +
>>   monitor/hmp-cmds.c                      |  26 ++++++
>>   qapi/migration.json                     |  57 ++++++++++--
>>   softmmu/dirtylimit.c                    |  33 ++++++-
>>   tests/migration/guestperf/comparison.py |  24 +++++
>>   tests/migration/guestperf/engine.py     |  33 ++++++-
>>   tests/migration/guestperf/hardware.py   |   8 +-
>>   tests/migration/guestperf/progress.py   |  17 +++-
>>   tests/migration/guestperf/scenario.py   |  11 ++-
>>   tests/migration/guestperf/shell.py      |  25 +++++-
>>   tests/qtest/migration-test.c            | 154 ++++++++++++++++++++++++++++++++
>>   15 files changed, 465 insertions(+), 31 deletions(-)
> 

-- 
Best regard

Hyman Huang(黄勇)

Re: [PATCH v1 0/8] migration: introduce dirtylimit capability
Posted by Peter Xu 3 years, 5 months ago
On Fri, Sep 02, 2022 at 01:22:28AM +0800, huangy81@chinatelecom.cn wrote:
> From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
> 
> v1:
> - make parameter vcpu-dirty-limit experimental 
> - switch dirty limit off when cancel migrate
> - add cancel logic in migration test 
> 
> Please review, thanks,
> 
> Yong 
> 
> Abstract
> ========
> 
> This series added a new migration capability called "dirtylimit".  It can
> be enabled when dirty ring is enabled, and it'll improve the vCPU performance
> during the process of migration. It is based on the previous patchset:
> https://lore.kernel.org/qemu-devel/cover.1656177590.git.huangy81@chinatelecom.cn/
> 
> As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of
> migration can make the read-process not be penalized. This series wires up the
> vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce
> two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup 
> of dirtylimit during live migration.
> 
> To validate the implementation, i tested a 32 vCPU vm live migration with such 
> model:
> Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus
> untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as
> the following command:
> taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}
> 
> The following are results:
> 
> host cpu: Intel(R) Xeon(R) Platinum 8378A
> host interface speed: 1000Mb/s
>   |---------------------+--------+------------+---------------|
>   | UnixBench test item | Normal | Dirtylimit | Auto-converge |
>   |---------------------+--------+------------+---------------|
>   | dhry2reg            | 32800  | 32786      | 25292         |
>   | whetstone-double    | 10326  | 10315      | 9847          |
>   | pipe                | 15442  | 15271      | 14506         |
>   | context1            | 7260   | 6235       | 4514          |
>   | spawn               | 3663   | 3317       | 3249          |
>   | syscall             | 4669   | 4667       | 3841          |
>   |---------------------+--------+------------+---------------|
> From the data above we can draw a conclusion that vcpus that do not dirty memory
> in vm are almost unaffected during the dirtylimit migration, but the auto converge
> way does. 
> 
> I also tested the total time of dirtylimit migration with variable dirty memory
> size in vm.
> 
> senario 1:
> host cpu: Intel(R) Xeon(R) Platinum 8378A
> host interface speed: 1000Mb/s
>   |-----------------------+----------------+-------------------|
>   | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>   |-----------------------+----------------+-------------------|
>   | 60                    | 2014           | 2131              |
>   | 70                    | 5381           | 12590             |
>   | 90                    | 6037           | 33545             |
>   | 110                   | 7660           | [*]               |
>   |-----------------------+----------------+-------------------|
>   [*]: This case means migration is not convergent. 
> 
> senario 2:
> host cpu: Intel(R) Xeon(R) CPU E5-2650
> host interface speed: 10000Mb/s
>   |-----------------------+----------------+-------------------|
>   | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>   |-----------------------+----------------+-------------------|
>   | 1600                  | 15842          | 27548             |
>   | 2000                  | 19026          | 38447             |
>   | 2400                  | 19897          | 46381             |
>   | 2800                  | 22338          | 57149             |
>   |-----------------------+----------------+-------------------|
> Above data shows that dirtylimit way of migration can also reduce the total
> time of migration and it achieves convergence more easily in some case.
> 
> In addition to implement dirtylimit capability itself, this series
> add 3 tests for migration, aiming at playing around for developer simply: 
>  1. qtest for dirty limit migration
>  2. support dirty ring way of migration for guestperf tool
>  3. support dirty limit migration for guestperf tool

Yong,

I should have asked even earlier - just curious whether you have started
using this in production systems?  It's definitely not required for any
patchset to be merged, but it'll be very useful (and supportive)
information to have if there's proper testing beds applied already.

Thanks,

-- 
Peter Xu


Re: [PATCH v1 0/8] migration: introduce dirtylimit capability
Posted by Hyman 3 years, 5 months ago

在 2022/9/7 4:46, Peter Xu 写道:
> On Fri, Sep 02, 2022 at 01:22:28AM +0800, huangy81@chinatelecom.cn wrote:
>> From: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
>>
>> v1:
>> - make parameter vcpu-dirty-limit experimental
>> - switch dirty limit off when cancel migrate
>> - add cancel logic in migration test
>>
>> Please review, thanks,
>>
>> Yong
>>
>> Abstract
>> ========
>>
>> This series added a new migration capability called "dirtylimit".  It can
>> be enabled when dirty ring is enabled, and it'll improve the vCPU performance
>> during the process of migration. It is based on the previous patchset:
>> https://lore.kernel.org/qemu-devel/cover.1656177590.git.huangy81@chinatelecom.cn/
>>
>> As mentioned in patchset "support dirty restraint on vCPU", dirtylimit way of
>> migration can make the read-process not be penalized. This series wires up the
>> vcpu dirty limit and wrappers as dirtylimit capability of migration. I introduce
>> two parameters vcpu-dirtylimit-period and vcpu-dirtylimit to implement the setup
>> of dirtylimit during live migration.
>>
>> To validate the implementation, i tested a 32 vCPU vm live migration with such
>> model:
>> Only dirty vcpu0, vcpu1 with heavy memory workoad and leave the rest vcpus
>> untouched, running unixbench on the vpcu8-vcpu15 by setup the cpu affinity as
>> the following command:
>> taskset -c 8-15 ./Run -i 2 -c 8 {unixbench test item}
>>
>> The following are results:
>>
>> host cpu: Intel(R) Xeon(R) Platinum 8378A
>> host interface speed: 1000Mb/s
>>    |---------------------+--------+------------+---------------|
>>    | UnixBench test item | Normal | Dirtylimit | Auto-converge |
>>    |---------------------+--------+------------+---------------|
>>    | dhry2reg            | 32800  | 32786      | 25292         |
>>    | whetstone-double    | 10326  | 10315      | 9847          |
>>    | pipe                | 15442  | 15271      | 14506         |
>>    | context1            | 7260   | 6235       | 4514          |
>>    | spawn               | 3663   | 3317       | 3249          |
>>    | syscall             | 4669   | 4667       | 3841          |
>>    |---------------------+--------+------------+---------------|
>>  From the data above we can draw a conclusion that vcpus that do not dirty memory
>> in vm are almost unaffected during the dirtylimit migration, but the auto converge
>> way does.
>>
>> I also tested the total time of dirtylimit migration with variable dirty memory
>> size in vm.
>>
>> senario 1:
>> host cpu: Intel(R) Xeon(R) Platinum 8378A
>> host interface speed: 1000Mb/s
>>    |-----------------------+----------------+-------------------|
>>    | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>>    |-----------------------+----------------+-------------------|
>>    | 60                    | 2014           | 2131              |
>>    | 70                    | 5381           | 12590             |
>>    | 90                    | 6037           | 33545             |
>>    | 110                   | 7660           | [*]               |
>>    |-----------------------+----------------+-------------------|
>>    [*]: This case means migration is not convergent.
>>
>> senario 2:
>> host cpu: Intel(R) Xeon(R) CPU E5-2650
>> host interface speed: 10000Mb/s
>>    |-----------------------+----------------+-------------------|
>>    | dirty memory size(MB) | Dirtylimit(ms) | Auto-converge(ms) |
>>    |-----------------------+----------------+-------------------|
>>    | 1600                  | 15842          | 27548             |
>>    | 2000                  | 19026          | 38447             |
>>    | 2400                  | 19897          | 46381             |
>>    | 2800                  | 22338          | 57149             |
>>    |-----------------------+----------------+-------------------|
>> Above data shows that dirtylimit way of migration can also reduce the total
>> time of migration and it achieves convergence more easily in some case.
>>
>> In addition to implement dirtylimit capability itself, this series
>> add 3 tests for migration, aiming at playing around for developer simply:
>>   1. qtest for dirty limit migration
>>   2. support dirty ring way of migration for guestperf tool
>>   3. support dirty limit migration for guestperf tool
> 
> Yong,
> 
> I should have asked even earlier - just curious whether you have started
> using this in production systems?  It's definitely not required for any
> patchset to be merged, but it'll be very useful (and supportive)
> information to have if there's proper testing beds applied already.
> 
Actually no when i posted the cover letter above, the qemu version in 
our production is much lower than upstream, and the patchset is 
different from here, i built test mode and did the test on my own in the 
first time. But this feature is in the process of test conducted by 
another professional test team, so once report is ready, i'll post it. :)
> Thanks,
>